On the Design of KL-Regularized Policy Gradient Algorithms for LLM Reasoning
Authors: Yifan Zhang
Yifeng Liu
Huizhuo Yuan
Yang Yuan
Quanquan Gu
Andrew C Yao
Venue: ICLR 2026 Poster (2026)
Keywords: LLM reasoning
reinforcement learning
Primary area: reinforcement learning

Abstract: Policy gradient algorithms have been successfully applied to enhance the reasoning capabilities of large language models (LLMs). KL regularization is ubiquitous, yet the design surface, choice of KL direction (forward vs. reverse), normalization (normalized vs. unnormalized), and estimator ($k_1/k_2/k_3$), is scattered across the literature and often intertwined with off-policy estimation. We ask a focused question: under the off-policy setting, what weighting is required for each KL variant so that the surrogate we optimize yields the exact gradient of the intended KL-regularized objective? We answer this with a compact, unified derivation we call the Regularized Policy Gradient (\textbf{RPG}) view. RPG (i) unifies normalized and unnormalized KL variants and shows that the widely-used $k_3$ penalty is exactly the unnormalized KL; (ii) specifies conditions under which REINFORCE-style losses with stop-gradient are gradient-equivalent to fully differentiable surrogates; (iii) identifies and corrects an off-policy importance-weighting mismatch in GRPO's KL term; and (iv) introduces RPG-Style Clip, a truncated-importance-sampling step within RPG-REINFORCE that enables stable, off-policy policy-gradient training at scale. On mathematical reasoning benchmarks (AIME24, AIME25), RPG-REINFORCE with RPG-Style Clip improves accuracy by up to $+6$ absolute percentage points over DAPO. Notably, RPG is a \emph{stable and scalable} RL algorithm for LLM reasoning, realized via (a) a KL-correct objective, (b) truncated importance sampling, and (c) an iterative reference-policy update scheme.

Published as a conference paper at ICLR 2026
ON THE DESIGN OF KL-REGULARIZED POLICY GRA- DIENT ALGORITHMM FOR LLM REASONING
Yifan Zhang*o.1.2 Yifeng Liu Huizhuo Yuan'
Yang Yuan
Quanquan Gu
Andrew Chi-Chih Yao
University of California, Los Angeles
Prrinceton University
TTinggua University
yifchang@pricceton.co liuyifengêcs www.leeeenees.claeedeeeeeee
ucla.
qguêcs. edu, andrewcyaolttsinghen edu.cn
ABSTRACT
Policy gradient algorithms have been successfully applied to enhance the reasoning capabilities of large language models (LLMs). KL regularization is ubiquitous, yet the design surface, choice of KL direction (forward vs. reverse), normalization (normalized vs. unnormalized), and estimator (ka/k2/ks)) is scattered across the literature and often intertwined with off-policy estimation. We ask focused question: under the off-policy setting, what weighting is required for each KI. variant so that the surrogate we optimize yields the exact gradient of the intended KL-regularized objective? We answer this with a compact, unified derivation we call the Regularized Policy Gradient (RPG) view. RPG (i) unifies normalized and unnormalized KL. variants and shows that the widely-use ky penalty is exactly the unnormalized KL; (ii) specifies conditions under which REINFORCE-style losses with stop-gradien poopggradeent are gradient-equivalent to fully differentiable surrogates; (iii) identifies and corrects an off-policy
cerrrance-weighting mismatch in GRPO's seapeed-ipottanc-sammaee
KL term; and (iv) introduces RPG-Style Clip, step within RPG-REINFORCE that enables stable, off off-policy policy-gradien training scale. On mathematical reasoning benchmarks (A (AIME24, AIME25), RPG-REINFORCE with RPG-Style Clip improves accuracy by up to +6 absolute percentage points over DAPO. We extend our experiments to 8K context length, and RPG-REINFORCE with RPG-Style Clip achieves 52% accuracy on AIME25, surpassing the official Diven--4BIInstruu model (47%). Notably. RPG is a stable and scalable RL algorithm for LLM reasoning, realized via (a) KL-correct objective, (b) clipped importance sampling, and (c) an iterative reference-policy update scheme. Project Page:
INTRODUCTION
Reinforcement learning (RL), particularly policy gradient (PG) methods, provides powerful frame- work for solving sequential decision-making problems complex environments. These methods have been successfully applied in diverse domains, ranging from robotics to game playing. and have recently become instrumental in fine-tuning large language models (LLMs) to align with human preferences and instructions (Ouyang et al., 2022) and enhancing the reasoning capabilities of LLMs (Shao et al., 2024; Guo et al., 2025). Classical PG algorithms like REINFORCE (Williams, 1992) optimize policies directly but often suffer from high gradient variance. Advanced methods like Proximal Policy Optimization (PPO) (Schulman et al., 2017) improve stability and sample efficiency, enabling large-scale applications, often by operating in an off-policy manner and employing tech- niques like training critic models for the estimation of value functions. Our theme in this paper is stability and scalability: which design choices KL-regularized PG matter for robustness under off- off-policy sampling, and practical throughput on modern LLM stacks?
crucial technique for stabilizing policy optimization, especially when deviating from strictly on- policy updates or aiming to control policy complexity. is regularization. Kullback-Led Leibler (KL) divergence a commonly used regularizer, penalizing the deviation of the learned policy no from
"Equal contribution; 'Corresponding authors; Work done during visit to UCL.A.

---

Published as a conference paper at ICLR 2026
Goal: Stable and Scalable LLM Reasoning
RPG Core Engine
Inputs (Iteration :)
Outputs
Construct J(0 3.go (R)
Current Policy 6
Updated Policy [f+1) "g
Reference ()
Derive 1(01))
Rewards R(a)
Formulate Surogaae E(
Optimize get %
(1+1)
Key Design Configuration
Estimator:
KL Normaiized
3. KL Divergence: KL (Te) Reverse KT,(wx (ites)
Fully Difforentable
REINFORCE-dtyle
Unnormaized (UKL ks)
Update Reference: Told
Figure 1: Overview of the iterative Regularized Policy Gradient (RPG) framework proposed in this work. At each iteration t, the central RPG Core Engine processes inputs: the current policy =(t) a reference policy Told and associated rewards R(x). The engine's operation encompasses four main steps: (1) constructing the Kl-rregularized objective [(("), which combines the expected reward with KL divergence term; (2) deriving the off off-policy policy gradient Joco) (3) formulating a corresponding surrogate loss function L(B(t)): ); and (4) optimizing the policy parameters to yield
(t)
an updated policy xe (t+1) aimed at enhancing LLM reasoning capabilities. The specific behavior of the RPG Core Engine configured by three key design choices: (i) the Loss Estimator type (Fully Differentiable or REINFORCE-style with Stop-Gradient;; (ii) the KL Form (Normalized or Un-normalized, e.g., using UKL / ki estimators); and (iii) the KL Divergence Type (Forward KL Te)) Reverse (L(T Tota)). The framework operates iteratively, with the updated policy (++1) from one iteration informing the inputs for the next, including the update of the reference (I+1)
T
facilitate continuous learning and performance improvement. policy oldd
a reference policy मet (c.g., policy from previous iteration Te.ia or a fixed prior policy π SFTT). KL regularization helps prevent destructive policy updates, encourages exploration around known good policies, and can prevent catastrophic forgetting or overly confident outputs (Ouyang et al., 2022).
Despite the widespread use of KL regularization in methods such PPO (often implicitly through reward penalties) and explicit formulations like GRPO (Shao et al., 2024), there exists a considerable variety in how the KL divergence formulated and estimated. Different choices include Forward KL and Reverse KI, handling potentially unnormalized distributions (Minka et al., 2005) (leading to unnormalized KL (UKL) and unnormalized reverse KL (URKL) formulations). and the use of various estimators like the 22 and K3 estimators (Schulman, 2020) designed to potentially reduce variance or offer different properties compared the standard log-ratio (k,) ) estimator. Furthermore, the interplay between the choice of KL formulation, the policy optimization setting (on-policy vs. off-policy), and the derivation of appropriate surrogate loss functions (fully differentiable vs. REINFORCE-styll gradient estimators) can lead to subtle differences.
This paper provides systematic derivations and unifying treatment of KL-regularized policy gradient methods, and revisits classical REINFORCE CE through the lens of clipped importance sampling. Our main contributions are summarized as follows:
We derive policy gradients and corresponding surrogate losses for Forward/Reverse KL, in normal- ized (KI.) and unnormalized (UKL.) forms, under off-policy sampling with importance weights.
2

---

Published as a conference paper at ICLR 2026
We give both fully differentiable surrogates and REINFORCE-style losses (with siop-gradient and prove their pendix L).
to the intended regularized objective (Proposition 4.1, Ap-
We introduce RPG-Style Clip, clipeed-mmprttane--wiggge stantially improves stability and variance control while preserving the RPG gradients.
REINFORCE estimator that sub-
We reveal the equality between the A3 estimator and unnormalized KL (Appendix D), and show that GRPO's KL penalty omits an essential importance weight under off-policy sampling. We provide corrected estimator and loss consistent with the intended objective.
We present an iterative training framework that periodically updates the reference model to satisfy KL constraints while allowing the policy to depart meaningfully from the initial checkpoint.
On math reasoning, RPG-REINFORCE (with RPG-Style Clip) yields stable and scalable training and outperforms DAPO by up to +6 absolute points on AIME24/25.
We extend our experiments to 8K context length and find that RPG-REINFORCE with RPG-Style Clip achieves 52% accuracy on AIME25, surpassing the official Qwen3-4B -wwen--B-Instrrut model (47%) and outperforming strong baselines.
PRELIMINARIES
Policy gradient (PG) methods are comerrtone modern reinforcement learning (RL), optimizing pa- rameterized policies To by estimating the gradient of an expected objective function J(0) with respect to the policy parameters 0. Typically, J(Ø) represents the expected cumulative discounted reward over trajectories = (sp, agg To: 81, , ST, AT, (T) generated by the policy: J(0) E [G(T)), ST, where G(T) Σi= yr,, is the trajectory return (with discount factor 7), and the expectation taken over the trajectories sampled according to the policy (a(ss) s) and the environment dynamics p(s'|s, a). The Generalized Policy Gradient Theorem (GPPT) provides a foundation for deriving these gradients (see Appendix for the proof).
Proposition (Generalized Policy Gradient Theorem). Let ()( be a probability density or mass function parameterized by 0, representing the probability of sampling item Let faa 0) be a valued function associated with x, potentially depending on 0. Under suitable regularity conditions, the gradient of the expectation Ez~x, [f 0)] with respect 0 is:
V go (z, ) Egano [f 0) V log Th (x) + Vef(x, 0)]-
(2.1)
The term E log || reflects how changes in 0 affect the probability of sampling while E[V reflects how changes in f directly affect the function value f.
The classic REINFORCE algorithm (Williams, 1992) applies the GPPT to the standard RL objective (0) = ET [G()). In this case, f(T, 0) = G((T). ) the total trajectory return, which does not depend directly on 0 (i.e., VoG(T) 0). The theorem simplifies, and the gradient be expressed using per-timestep contributions (Sutton et al., 1998):
Σ G.V log
( = Eroxo
where G, = -I= t "r is the returnto-go from timestep t. Due to space limit, we defer the detailed introduction of REINFORCE to Appendix C.I.
KL REGULARIZATION IN POLICY GRADIENTS
common technique stabilize policy optimization, especially off off-policy settings or when fine-tuning large models, is regularization. The Kullback-L.eibler (KI.) divergence is frequently used to penalize the deviation of the learned policy ny from reference policy Tyef (which could be me an initial supervised fine-tuned model, or another prior). KL(P Q) > 0 with equality iff P Q Minimizing the forward KL [T || TT) encourages To cover the support of meef (eroifforiigg)... while minimizing the reverse KL KL( Tret) encourages To to be concentrated where Nre has high probability mass (mode-seeking).
almost everywhere. It is asymmetric (i.c., [L Q) * KL(QI]
Adding a KL penalty to the RL objective, such as J(Ø) = Ex, [R| 3 (L(TT]| (rer)) helps control the policy update size, prevents large deviations from meef, encourages exploration near known good
3

---

Published as a conference paper at ICLR 2026
policies, and can mitigate issues like catastrophic forgetting or overly confident outputs, particularly relevant LLM fine-tuning (Ouyang et al.. 2022). For PPO (see Appendix C.2), this penalty can be incorporated implicitly via reward shaping: r can be added explicitly to the objective function, as in GRPO. The specific form of the KI. divergence (forward/reverse,, whether distributions are normalized (KI. vs. UKI.), and the choice of estimator (e.g., standard log-ratio vs. K3 estimator (Schulman, 2020)) can vary, leading to different properties (mode seeking v.s. zero-foreing) and gradient estimators, as explored later in this paper (Sections and 4).
3 log( T ( ( [ (a,]s;)) Alternatively, it
2.2
GROUP RELATIVE POLICY OPTIMIZATION (GRPO)
Group Relative Policy Optimization (GRPO) (Shao et al., 2024) adapts the PPO framework for training LLMs, notably by eliminating the need for a learned value function (critic). Instead using GAE, GRPO estimates the advantage As, at token I of output o, based on the relative rewards within group of G outputs {0i, :ol, sampled from the old policy Toes for the same prompt q.
Crucially, GRPO modifies the PPO objective by explicitly adding a KI. regularzzation term directly the objective function. Its objective (simplified notation) is:
locl
JERRO P())
J (0) B KL (TO(( (1B
Os
where hi, = (g: 01,ct) ) is the history, [lip (0) represents the PPO-Clip term from Eq. (C.3) applied using the group-relative advantage estimate Ai,, and Tref is reference model (e.g., the initial SFT model). For the KI. penalty, GRPO employs the k3 estimator form (Schulman, 2020), evaluated per token o
Troe(Os. heee) (o(e haa,)) To (o(0 hi,e))
Tref(or.e hea)
Tref Tree(or, hi,e)) 1.
(Lea(Tell so k3
log
(o(, ha,))
This the functional form ky( = y-
yy-logg log 1 as discussed Schulman (2020), applied with (o,, (o,,, T/ (o., This form is related to the unnormalized reverse KL divergence, UKL Fref) (see Section 3.2 and Appendix D for a detailed discussion). However, a key ob- servation regarding GRPO's 's KL penalty is its estimation. If the KL penalty in GRPO is intended to approximate B UKL(TTL (To [[s,.)][[=ee(( ()),,)) its off-policy estimation (sampling Or,t from Tata) /l(o)/lk..)) multiplying the k3 term. The would generally involve importance weight Wi,r Toia((t,
direct subtraction without this weight means the gradient derived from GRPO's objective does not, in general, correspond to the gradient of the intended off- off-policy objective JClipp 3 UKL(TT (ee)). For clarity, a corrected off-policy estimator for the GRPO KL component history hi, is
(ree((s,,lluee)) nO (oc,elhhh)) )]
KTcarro-omeeeaahpa 0) = Eo m (hha,)) Waet kis
which is consistent with URKL/UKL depending direction (see Section and Appendix D). Our results in Section 3 provide derivations for KL-regularized objectives that explicitly account for off- off-policy sampling via importance weights. Related work is detailed in Appendix A.
3 REGULARIZED POLICY GRADIENTS
In this section, we start from the KL regularized objective J(Ø) = E[R] - 3 KL and we treat this as the exact target for training. Then we derive its true gradient under off off-policy sampling. The derivation shows that we need precise importance weighting so that the gradient of the surrogate loss matches the gradient of this objective. The weights are different for Forward vs. Reverse KL, as summarized in Table 1. This viewpoint unifies many existing estimators within single framework and clarifies why the KL term in GRPO can lead to unstable updates when its weighting is chosen improperly. In the main text, we focus the unnormalized objectives (UFKL/URKL), while the normalized formulations (FKL/RKL) and their losses are deferred to Appendix E (see also Table 4). All proofs provided in Appendix K.
4

---

Published as a conference paper at ICLR 2026
3.1 UNNORMALIZED FORWARD KL REGULARIZATION
In scenarios where distributions might not be normalized (i.e., J2 <<))dx * 1), the standard KI. divergence might not fully capture the dissimilarity. The unnormalized forward KI. divergence addresses this by adding a mass correction term. Let Told (x) a potentially unnormalized reference measure with total mass Zola fz oota (x)dx. Let fota Tola (z) be the corresponding normalized probability distribution. such that J (z)d())dz 1.
Table 1: Summary of fully differentiable surrogate loss functions (()) for unnormalized KL- regularized objectives (main text). Minimizing ((6) corresponds to maximizing J(Ø) E,, [R(=)]
Divergence. Samples x are drawn from Vold / These losses yield - (0) via dif-
ferentiation. Notation: t((x) no (2:) (x), R(x)) is reward, the regularization strength, and Zold Told- Normalized counterparts are in Appendix E (Table 4).
Regularization (Unnormalized)
Surrogate loss (expectation w.r.t. Tola)
Forward (UFKLL)
Zola T3 w()) ((w(x) log wn(x) 1)
Reverse (URKL)
Zota 1 (() (w((x) log w(x) - w(w))
Definition 3.1 (Unnormalized Forward KI.). The unnormalized forward KL divergence (Minka et al., 2005; Zhu and Rohwer, 1995) between the measure Told and the density To is defined as:
/ Told (x) log
(old(r) dx (Fo(()) (x)) de. (I)
(KTaaT =
Generalized KL
Correction
This form is particularly relevant when dealing with reference measures that may not be perfectly normalized or when connecting to certain KL estimators like k3 (see Remark 3.5).
Consider the objective using UKL regularization as follows:
JUFKL( Ex~w, [f(x)]] 8 UKL(Toua|ITe)
(3.1)
To estimate this off- aff-policy using samples from the normalized reference (o)a(x) = Tota(x) (Zo//
define the importance weight w(xx) To T2(x)/ (x) (using the unnormalized Fold). The gradient and corresponding loss function, incorporating the total mass Zold of the reference measure, are given in Proposition 3.2.
Proposition 3.2 (Policy Gradient and Differentiable Loss for Unnormalized Forward KL). Consider the unnormalized KL regularized objective function in Eq. (3.1). The gradient of JuFKL( is:
[w(())(()) (w<(x) 1)) Vo log (0)
V VoJUrKL(O) ZoldE
The corresponding surrogate loss for gradient descent optimization, estimated using samples {x:} ~ Told is:
LUPKI.(0) ZoraBowF (=(xx))())) - B(w(x) - wog((x) - 1)]
satisfying VoLUFKI. (f) = (OJUFKI,())..
Remark 3.3 (Interpretation UFKL Loss and Gradient). The regularization component of the surrogate loss (UFKL(!)), specifically ZolaBz~Hiut (2(w) log w(z) 1)], corresponds to off-policy estimate of the unnormalized forward KL divergence term 3 UKL (Toiall present in the objective (f). This connection is established via the ka estimator (see Remark 3.5 and Appendix D). Furthermore, the gradient term -8(u(()) 1) effectively modifies the reward, guiding Ty to match not only the shape of Vold but also its overall mass Zold. due the mass correction component in UKL(Tola TT).
3.2 UNNORMALIZED REVERSE KL. REGULARIZATION
Similar to the forward case, we can define an unnormalized reverse KL divergence, relaxing the normalization constraint on the reference distribution Told- Let Tota(z)) be potentially unnormal- ized reference measure with total mass Z,Id = (z) Let Tata (x) = Faid (x) (z)/Zola be the corresponding normalized probability distribution.
5

---

Published as a conference paper at ICLR 2026
Definition 3.4 (Unnormalized Reverse KL). The unnormalized reverse KL divergence between the density Te and the measure Told is defined as:
(x) dx Told (2)
/ (Tola(z)) Te TT(x)) dx.
(TL(TT||Fooa) = Te log
Generalized KL
Correction
The mass correction term simplifies Zold no (x))xx
Remark 3.5. (Equivalence to k3 estimator) The ka estimator (Schulman, 2020), often used for its empirical properties (e.g., in GRPO (Shao al., 2024)), is defined for a density ratio y(x) as:
kz) := y
log y-
(3.2)
As shown in Appendix D, this functional form directly relates to unnormalized KL divergences. For instance, KLag (Ty Tota) := Enwnr (T) (TTaa(z)/TT((() [r)((r))] is equivalent to UKL(TT) Tata). This equivalence relationship justifies the exploration of UKL/URKL formulations within our framework.
Consider the objective using URKL:
JURKI [R(=) 3 UKL(ma Fota)
(3.3)
where UKL defined above. As with UFKL, we derive the gradient and loss using expectations over the normalized reference oold and the importance weight w(x) = (x(2)) (x) (with unnormalized Vold). The results are summarized in Proposition 3.6.
Proposition 3.6 (Policy Gradient and Differentiable Loss for Unnormalized Reverse KL). Consider the reverse unnormalized KL regularized objective function Eq. (3.3). The gradient of JURKI,
is:
V JoJURKL(0) = Zola Ra (w((=( (R(x) (olz))~Blogaa Blog V log mo(x))
corresponding surrogate loss for gradient descent optimization, estimated using samples {a:} Told. is:
CURKL (0) = Zold Bew [-w(x)R(z) + (w(x) log w(x) w(x))]
satisfying VoLURKI. (0) VoJURKI.(0) The constant Zoid scales the loss and gradient and may
be omitted in practice.
Remark 3.7 (URKL Loss and Mass Correction). The surrogate loss LURKL is designed such Specifically, the term Zota [3(w(x)) log w(x) - w(z)) the loss directly relates to the off-policy estimation of the unnormalized reverse KL divergence [ omitting a constant related to the total mass Zola which does not affect the gradient. The policy gradient's effective reward scaling factor, (R(x) - 3 log wxx)), ), simpler than its normalized RKL counterpart.
that its gradient is - VUJJURKI.((
Remark 3.8. In Appendix B, we show the connection between RPG and the Natural Policy Gradient (NPG) (Kakade, 2001; Schulman et al., 2015). In particular, the NPG update is special case of the RPG update, which uses linear approximation for the expected return and a quadratic approximation for the KL regularization. This transforms the problem from simple first-order gradient ascent in PG (REINFORCE) into sccond-order-lln update: RPG.
3.3 RPG-STYLE CLIP: DUAL-CLIP TRUNCATION OF IMPORTANCE RATIOS
Large importance ratios w<x) = (a(x)
Fal(z] induce high variance and destabilize off-policy updates.
Our RPG-Style Clip follows the dual-clip method implemented in Algorithm I in the Appendix: we clip w into [1 é1, + 62] and additionally impose a lower bound for negative advantages.
Let A(x; 0) denote the regularized advantage analogue determined by the chosen objective (e.g.. AURKI. (R - b) - log ARKI. (R-b) -300g + +1). The loss used in our implementation
max - m(x) w((x) Ā(x; 8), -clip(((x) 1 - +22 A(r; 0)) min max( - m((:) A(x 0), -lip(u(())) 1 o. (2) A(s; -c -(x A(r; <.0.
A(x; 2 0, with
CRPG-CllD
cl. and The choice of À for each divergence (URKL/UFKLIRKK/FFL) matches the gradients in Section 3 and is instantiated in Algorithm I.

---

Published as a conference paper at ICLR 2026
Case A(x)= 2 0 (c.g.. A = =)
Case A(x) < (e.g. == -1)
0
2.5
Loss term Duukiin (ar. .)
Loss term DuuaClip (a: 0)
-0.5
2
-1
1.5
-1.5
Grad via
Grad = (w.c1 w)
Grad (wirt
Grad via
-2
0.5
5
1
8
1.5
2.5
0.5
1.5
2.5
w(x)) (a( (x)
w(x) = (e(l)/ Te(ss//on(())
Figure 2: Visualization of the Dual-Clip loss term CDualClip (x,, 0) vs. importance weight w(x), as described in Section G.1 and Algorithm 1. This formulation is typically implemented as fully differentiable w.r.t 0 (via wxx) and potentially A(x) if Å depends on e.g., via log w(x)), unlike REINFORCE-style implementations that use SG(A) or SG((, within the loss. For visualization, A(æ) is treated as constant (Á = 1 left, À = -1 right) to isolate the effect of w. Solid blue: Loss depends linearly on w, gradient VgL flows via w(x). Dotted magenta: Loss is constant w.r.t w, gradient V does not flow via wxx) in this segment (though it might flow via depends on 0).
Left: Case Å < O. Right: Case Á 2 0.
4 REINFORCE-STYLE REGULARIZED POLICY GRADIENTS
Table 2: REINFORCE-style surrogate losses C(0) for unnormalized KL-regularized objectives using the gop-gradient operator (SG). These losses yield the target gradient via automatic differentiation. Compare with the fully differentiable losses in Table 1. Normalized versions are given in Appendix F.
Regularization (Unnormalizedd)
REINFORCE-style loss (sampling x Told)
Forward (UFKL)
-E
SG(Zoia(w ((w(x) L))) log (x( 3 log w(x))) log To (z
Reverse (URKL)
E (z) -
In Section 3, we derived policy gradient estimators and corresponding fully differentiable surrogate losses L(0) for KL-regularized objectives. Those losses were constructed such that VoC(O) (0) directly, typically by setting L(0) = -Js (0) (where JIs is the importance-sampled objective) up to constants. Notice that the gradients derived in Section (Theorems 3.2 through 3.6)
share a structural similarity with the REINFORCE estimator:
VoJ(0) = Ex--Taunlu [Weight(x, 0)Vo log (x))
where sampling is Vold or its normalized version Wolds and Weight(a, 0) encapsulates the reward and KL regularization terms, differing for each specific objective.
Proposition 4.1 Prradient-Equivalenne of Surrogates). For each KL-rregllarzedd objective J(0) derived in Section 3. the corresponding REIsFORCE-style losses in Table satisfy V&L(0) (O) under the standard regularity assumptions used in the policy-gradient theorem. In particular, the stop-ge gradient operator ensures that dependence of the weight on 6 (through importance ratios) does not leak unintended gradients. A proof sketch follows directly from the policy-gradics theorem and is completed in Appendix L.
This structural similarity motivates alternative REINFORCE-Ste implementation using the stop- gradient operator SG. The general form of such losses and the detailed rationale for how they yield the target gradient via automatic differentiation are presented in Appendix F.1 (see Eq. (F.1)).
7

---

Published as a conference paper at ICLR 2026
We explore these REINFORCE-styl estimators part of our framework, they offer an alternative implementation path and demonstrate stronger or competitive empirical performance (Section 5). Proofs in Appendix L. In the main text, we tabulate the unnormalized REINFORCE-style losses; normalized counterparts are deferred to Appendix F.
4.1 RPG-STYLE DUAL CLIP FOR REINFORCE- -TYLEE ESTIMATORS
Directly optimizing the REINFORCE-style losses in Table 2 can be unstable due to high variance in the importance weights w(x). To mitigate this, introduce RPG-Style Clip, which adapts the Dual-Clip technique (Ye et al., 2020) to our regularized setting.
The key strategy is decompose the weight term inside the stop-gradient operator into the importance ratio w(x) and a regularized advantage Ang (z). For example, for the URKL objective, we identify Ang (z) = Zoia( (R(x) 3 log w(x)) such that the weight is wxx) ((r)))r)())) We then replace this linear weighting with clipped version C(w(xx), (G(eeg(())) The resulting loss is:
CRPG-CliP = Eaz [c (xx). SGGe)) log TT(
(4.1)
where Cww, A) applies dual-clip bounds, clipping w to [] f1, 1 + 62] for positive A, and enforcing lower bound for negative A, as detailed Appendix 1.I. This ensures stable off aff-policy updates while preserving the correct gradient direction derived from the KL-regularized objective.
Case g/r
Case th <
Grad via
Grad 0
Loss coefficient C,
1.5
Loss coefficient L
via E
-1.5
-2
0.5 0.5
-2.5
1
8
1.5
2.5
0.5
1.5
2
2.5
Wi To(x:)//ng (x.)
(a:)/ (xr(
Figure 3: Visualization of the loss coefficient L. vs. importance weight wy based on the specific implementation in Algorithm 2 as RPG-REINFORCE. This version swaps the main branching condi- tion compared to previous versions (branches on 1 0). The plot assumes E - llog Ta (2x) = for visualizing the value of L, The line styles indicate the nature of the gradient V,C,; Solid blue: Gradient exists, flowing only via & The coefficient multiplying Vol, depends on SG(w;). Dotted magenta: Gradient is zero. This occurs when L is detached via SG in the loss calculation. Left:
Case vú 2 0. Right: Case 3 < O.
EXPERIMENTS
In this section, we empirically evaluate our proposed Regularized Policy Gradient (RPG) framework, including both its fully differentiable (RPG) and REINFORCE- -tyle (PP-RREINFORCE) variants. We compare their performance against established baselines on challenging mathematical reasoning tasks using large language models, including GRPO (Shao al., 2024) and DAPO (Yu et al., 2025). Our evaluation focuses task-specific accuracy, training stability, and key training dynamics such as reward, policy entropy, and response length."
Base Models and Datasets. We conduct experiments using the Qwen3-4B and Qwen2.5-7B. Beeen..5-7B-Inttreee Instructt models. For training, we utilize the DAPO-Math- 17k dataset (Yu et al., 2025) (13.9) 3. English samples). We evaluate AIME2024, AIME2025, and AMC23, and additionally report results on MinervaMath and OlympiaBBench in the Appendix. We compare against baselines including GRPO, DAPO, and REINFORCE++
8

---

Published as a conference paper at ICLR 2026
LMT
120
(a) AIME24
(b) AIME25
(c) AMC23
(d) Reward (Critic Score)
(c) Entropy
(f) Response Length
Figure 4: Training dynamics and benchmark performance for fully differentiable Regularized Policy Gradient (RPG) and REINFORCE-Style RPG (RPG-REINFORCE) compared to baselines (GRPO and DAPO) with 8k context length.
Table 3: Combined performance metrics with 8K context length on the AIME24, AIME25, and AMC23 mathematical reasoning benchmarks, showing "Last" and "Best" scores. The "Last" score is from the 400th training step, assuming the training process remained stable to that point. The highest score in each column bolded, and the second highest is underlined. RPG and RPG- REG-RRINFORCE methods are highlighted with light eyan and light green backgrounds, respectively.
AIME24
AIME25
AMC23
Method
Last
Best
Last
Best
Last
Best
GRPO DAPO
0.3750
0.4396
0.3354
0.4063
0.9109
0.9297
0.5438
0.5740
0.4469
0.4740
0.9375
0.9430
RPG-UFKL
0.5938
0.6177
0,4698
0.4865
0.9492
0.9517
RPG-URKI
0.4542
0.5260
0.5261
0.4938
0.9406
0.9539
OPG-REINFORCE-AO UFKL
0.5906
0,5958
0.4833
0.5031
0.9453
0.9469
OPG-REINFORCE- URKL
0.5708
0.5781
0,5073
0.5208
0.9398
0,9469
Implementation and Framework. Experiments are implemented using the verl framework (Sheng et al., 2025) with the vLLM engine (K woonn et al., 2023) for efficient LLM serving and inference. For practical implementation of our RPG methods, we emphasize that the probabilities (or log- probabilities) from the last iteration's 's model (Tola) for the sampled data can be pre-computed and stored. This allows the KL regularization terms to be calculated without needing to keep Told in GPU memory during the training step of the current policy Consequently, only one model (wa) needs to actively managed in GPU memory for training, which is faster and more memory-efficient compared to approaches like GRPO that typically require access to at least two models (the current policy and reference/sampling policy) during optimization.
Iterative reference updates. To further stabilize optimization, we adopt iterative reference-update scheme: we periodically set Told + ng (every optimizer steps, or when a moving average of teken-level KL exceeds target x). This realizes practical KL trust region while avoiding over- regularization toward the initial checkpoint. Further implementation details and hyperparameters (learning rate, B3, clipping) provided in Appendix H.

---

Published as a conference paper at ICLR 2026
Stabilization and Advanced RL Techniques. Our RPG implementations (both fully differentiable and REINFORCE incorporate stabilization techniques like baseline subtraction and PPO-style objective clipping (specifically, Dual-Clip (Ye et al., 2020; Schulman et al., 2017)), crucial for robust off-policy learning. Detailed algorithmic descriptions are provided in Appendix G (see Algorithm 1 for RPG with Dual-Clip and Algorithm 2 for the REINFORCE equivalent, along with Figures and 2 for visualization). Varying the clip ratios in REINFORCE-style RPG algorithms, we find that while critic scores and response lengths are similar for (e1, (Figure 8), DAPO's s nigher-and-clip-higee to underlie its performance gains, details can be found in Appendix 1.2.1. For PPO-style clipping, we set (e1, 62) = (0.2, 0.28) for RPG, RPG-REINFORCE and DAPO. For GRPO, we use (2), (0.2,0.) 0.2). Furthermore, to enhance training efficiency and data quality, we adopted techniques introduced by DAPO (Yu et al., 2025), including dynamic sampling strategy with group filtering mechanism, which oversamples challenging prompts and filters out those with near-perfect or near- zero accuracy based on initial rollouts and overlong punishment component in the reward shaping to discourage excessively verbose outputs. In addition, we enable RPG-Style Clip (Section 3.3) for the REINFORCE-sty/ estimators, which we found to be the best variant for RL training at larger scales.
(0.1, 0.1) and (0.2, 0.28) strategy substantially reduces actor entropy, which appears
Results and Discussion. We display the curves in Figure and last and best scores on AIME24 and AIME25 benchmarks in Table for the experiments with 8K context length. The results also demonstrate the superiority of our algorithms over baselines, including GRPO and DAPO. With 8k context length, RPG-REINFORCE achieves 52% accuracy on AIME25, surpassing the official Qwen3-44-Instruett baseline (47%). Tables 5 and in Appendix I summarize the performance of our RPG algorithms against baselines with 4k and 2k context lengths, reporting both the last and best scores achieved during training on these benchmarks. Figures and in Appendix complement these results by illustrating the evaluation scores and training dynamics for the fully differentiable RPG variants and baselines when training the Qwen-3-4B model using 4k and 2k context length, respectively. These figures display performance on the AIME24 and AIME25 benchmarks, alongside key training metries: reward (critic score), policy entropy. and average response length. Across settings, the RPG-REINFORCE variants with RPG-Style Clip have the strongest results. Following DAPO (Yu et al., 2025; Yue et al., 2025), we report "Mean "2"" (average accuracy of 32 sampled responses).
Moreover, these algorithms generally exhibit stable training progressions regarding reward (critic score) and policy entropy, as shown in subfigures (c) and (d) in Figures 5 and 6 in the Appendix, compared to some baselines like GRPO, which can show more volatility. This stability likely contributes to their robust benchmark performances (subfigures a-b). The response lengths (subfigure e) for RPG methods also appear well-controlled. These observations align with the strong final scores reported in Tables 5 and 6 for these variants.
CONCLUSION
We introduced RPG, a framework for deriving and organizing KL-regularized policy gradient al- gorithms for online, off- off-policy RL. We provided derivations for policy gradients and surrogate loss functions covering forward/reverse KL, normalzzedunnormalzze distributions, and both fully differentiable and REINFORCE-style estimators. Beyond derivations, we revisited the classical REINFORCE algorithm and made it viable off-policy through RPG-Style Clip and iterative refer- ence updates. On LLM reasoning. these design choices deliver stable and scalable training with competitive and superior accuracy relative to strong basclines.
ETHICS STATEMENT
The methods developed in this paper contribute to the broader effort of enhancing the reasoning capabilities of large language models. Improved reasoning in LLMs has the potential to significantly benefit various fields, including scientific discovery, education, and complex problem-solving in engineering and medicine. By providing more stable and efficient training algorithms, our work can facilitate the development of more reliable and capable AI systems.
However, as with any advancement in AI capabilities, it is crucial to consider the ethical implications and ensure responsible development and deployment of these technologies to mitigate potential
10

---

Published as a conference paper at ICLR 2026
misuse. While framework offers unified perspective KL-regularized policy gradient algo- rithms and demonstrates strong empirical performance, it has certain limitations. RPG-Style Clip introduces a controllable bias: variance trade-off through (c1, f2), developing principled schedules for clipping would be valuable. We used L.MSs as assistive tools to polish part of this paper. The roles of LLMs in this work are restricted to improving readability and presentation.
REFERENCES
Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone, arXiv preprint arXiv:2404. 0..4404441199 14219, 2024.
Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basies: Revisiting reinforce-style optimization for learning from human feedback in llms. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1724-12267) 2024.
Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learning from human preferences. In International Conference Artificial Intelligence and Statistics, pages 4447-4455. PM 2024.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324-344 1952.
Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak language models to strong language models. In Forty-firsi International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024, 2024.
Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages
2017.
Xiangxiang Chu, Hailang Huang, Xiao Zhang, Fei Wei, and Yong Wang. Gpg: simple and strong reinforcement learning baseline for model reasoning. arXiv preprint 2504v2250..02546, 2025.
Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. RAFT: reward ranked finetuning for generative foundation model alignment. Trans. Mach. Learn. Res., 2023, 2023.
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization. arXiv preprint urXiv:2402.01306, 2024.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama herd of models. arXiv preprint arXhv:2407.2178. 2024.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseck-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12945 2025.
Jian Hu. Reinforcet+: A simple and efficient approach for aligning large language models. arXiv preprint arXiv:2501.03262 2025.
Kaixuan Ji, Guanlin Liu, Ning Dai, Qingping Yang. Renjie Zheng, Zheng Wu, Chen Dun, Quanquan Gu, and Lin Yan. Enhancing multi-step reasoning abilities of language models through direct q-function optimization. arXiv preprint 2rXiv:2410.09332, 2024.

---

Published as a conference paper at ICLR 2026
Sham M Kakade. A natural policy gradient. Advances in neural information processing systems, 14, 2001.
Amirhossein Kazemnejad, Milad Aghajohari, Eva Portelance, Alessandro Sordoni, Siva Reddy, Aaron Courville, and Nicolas Le Roux. Vineppo: Unlocking rl potential for Ilm reasoning through refined credit assignment. arXiv preprint arXiv:2410.01679, 2024.
Wouter Kool, Herke van Hoof, and Max Welling. Buy 4 reinforce samples, get a baseline for free! ICLR 2019 workshop: Deep RL Meets Structured Prediction, 2019.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, pages 61 626, 2023.
Ziniu Li, Tian Xu, Yushun Zhang, Zhihang Lin, Yang Yu, Ruoyu Sun, and Zhi- Zhi-Quan Luo. Remax: A simple, effective, and efficient reinforeement learning method for aligning large language models. In Forty-first International Conference on Machine Learning, ICML. 2024, Vienna, Austria, July 21-27, 2024, 2024.
Tianqi Liu, Yao Zhao, Rishabh Joshi, Misha Khalman, Mohammad Salch, Peter J. Liu, and Jialu Liu. Statistical rejection sampling improves preference optimization. In The Twelfth Interna- tional Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024.
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding r1-zero-like training: A critical perspective. arXiv preprint arXiv:2503.20783, 2025.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, 2019.
Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with a reference- free reward. Advances in Neural Information Processing Systems, 37:124198-1244 2024.
Tom Minka et al. Divergence measures and message passing. Technical report, Microsoft Research, 2005.
Rémi Munos, Michal Valko, Daniele Calandriello, Mohammad Gheshlaghi Azar, Mark Rowland, Zhaohan Daniel Guo, Yunhao Tang, Matthieu Geist, Thomas Mesnard, Côme Fiegel, Andrea Michi, Marco Selvi, Sertan Girgin, Nikola Momchev, Olivier Bachem, Daniel J. Mankowitz, Doina Precup, and Bilal Piot. Nash learning from human feedback. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024, 2024.
OpenAL. ChatGPT, 2022. URL htLpst:::// http::////haataaeee http://htt.ppnn..ccccoo com/.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 27730-27744, 2022.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36:53728-53741, 2023.
John Schulman. Approximating kl divergence. html, March 2020. Accessed on February 19, 2026.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization, In International conference on machine learning, pages 1889-1897. PMLR, 2015.
12

---

Published as a conference paper at ICLR 2026
John Schulman, Philipp Moritz, Sergey Levine, Michael I. Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. In Yoshua Bengio and Yann LeCum, editors, 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.063777 2017.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint rxXv:2402.033300 2024.
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient RLHF framework. In Proceedings of the Twentieth European Conference on Computer Systems, EuroSys 2025, Rotterdam, The Netherlands, 30 March 2025 3 April 2025, pages 1297. ACM, 2025.
Richard S Sutton, Andrew G Barto, et al. Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998.
Kimi Team, Angang Du, Bofci Gao, Bowei Xing, Changjiu Jiang. Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1. 5: Scaling reinforcement learning with Ilms. arXiv preprint arXiv:2501. 12599, 2025.
Qwen Team. Qwen3 technical report, 2025. URL
09388.
Ronald J Williams. Simple statistical gradient- followiig algorithms for connectionist reinforcement learning. Machine learning, 8:229-255 256, 1992.
Yue Wu, Zhiqing Sun, Huizhuo Yuan, Kaixuan Ji, Yiming Yang, and Quanquan Gu. Self-play preference optimization for language model alignment. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025, 2025.
Wei Xiong, Hanze Dong, Chenlu Ye, Ziqi Wang, Han Zhong, Heng Ji, Nan Jiang, and Tong Zhang. Iterative preference learning from human feedback: bridging theory and practice for rlhf under kl-constraint. In Proceedings of the 41 International Conference Machine Learning, pages 54715-54754 2024.
Jing Xu, Andrew Lee, Sainbayar Sukhbaatar, and Jason Weston. Some things are more cringe than others: Preference optimization with the pairwise cringe loss. arXiv preprint arXiv:2312.16682, 18, 2023.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:24/2.15115, 2024.
Deheng Ye. Zhao Liu, Mingfei Sun, Bei Shi, Peilin Zhao, Hao Wu, Hongsheng Yu, Shaojie Yang, Xipeng Wu, Qingwei Guo, et al. Mastering complex control in moba games with deep reinforce- ment learning. In Proceedings of the AAAI conference artificial intelligence, volume 34, pages 6672-6679, 2020.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. Dapo: An open-source Ilm reinforcement learning system at scale. arXiv preprini 2503 14476, 2025.
Yufeng Yuan, Qiying Yu, Xiaochen Zuo, Ruofei Zhu, Wenyuan Xu, Jiaze Chen, Chengyi Wang, Tian Fan, Zhengyin Du. Xiangpeng Wei, et al. Vapo: Efficient and reliable reinforcement learning for advanced reasoning tasks. arXiv preprint arXiv:2504.05/ 18, 2025.
Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Keming Lu, Chuanqi Tan, Chang Zhou, and Jingren Zhou. Scaling relationship on learning mathematical reasoning with large language models. arXiv preprint arXiv:2308.0/88: 2023.
13

---

Published as a conference paper at ICLR 2026
Yu Yue, Yufeng Yuan, Qiying Yu, Xiaochen Zuo, Ruofei Zhu, Wenyuan Xu, Jiaze Chen, Chengyi Wang, TianTian Fan, Zhengyin Du et al. Vapo: Efficient and reliable reinforeement learning for advanced reasoning tasks. arXiv preprint arXhv:2504.05718. 2025.
Yifan Zhang, Ge Zhang, Yue Wu, Kangping Xu, and Quanquan Gu. Beyond bradley-terry models: A general preference model for language model alignment. In Proceedings of the 42nd International Conference on Machine Learning, 2025.
Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J Liu. Slic-hf: Sequence likelihood calibration with human feedback. arXiv preprint arXiv:2305.1042) 2023.
Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, et al. Group sequence policy optimization. arXiv preprint arXiv:2507.1807,, 2025.
Huaiyu Zhu and Richard Rohwer. Information geometric measurements of generalisation. Preprint. 1995.
14

---

Published as a conference paper at ICLR 2026
Appendix
A Related Work
17
Connection Between Regularized Policy Gradient and Natural Policy Gradient
18
REINFORCE and Proximal Policy Optimization (PPO)
19
C.I REINFORCE
19
C.2 Proximal Policy Optimization (PPO)
19
D Equivalence of K3 Estimator and Unnormalized KL Divergence
20
E Normalized KL Regularization
20
E.1 Forward KL Regularization
20
E.2 Reverse KL Regularization
REINFORCE-Style Regularized Policy Gradients with Various KL Regularization
Forms
21
F.
Rationale for REINFORCE-Style Loss Formulation
21
F.2 REINFORCE-SH RPG with Forward KL Regularization
22
F3
REINFORCE-Style RPG with Unnormalized Forward KL Regularization
22
F.4 REINFORCE-Style RPG with Reverse KL Regularization
22
F.5
REINFORCE-SSyle RPG with Unnormalized Reverse KI. Regularization
22
More on Algorithmic Details
22
G.1
Stabilization Techniques for Regularized Policy Gradients
22
G.2
Stabilization Techniques for REINFORCE- Regularized Policy Gradients
24
Detailed Experimental Setup
25
I Additional Experiment Results
27
I.1
The performance with 2k and 4k context length
27
1.2 Ablation Study
27
I.3 Experiments on Qwen-2.5-7B-Imm InIsructttt
29
J Proof of Theorem 2.1 (Generalized Policy Gradient Theorem)
32
K Proofs for Regularized Policy Gradients
32
K.I Proof of Proposition E. (Policy Gradient and Differentiable Loss for Normalized Forward KL)
32
K.2 Proof of Proposition 3.2 (Policy Gradient and Differentiable Loss for Unnormalized Forward KI.)
33
15

---

Published as a conference paper at ICLR 2026
K.3 Proof of Proposition E.3 (Policy Gradient and Differentiable Loss for Normalized Reverse KL)
34
K.4 Proof of Proposition 3.6 (Policy Gradient and Differentiable Loss for Unnormalized
Reverse KL)
35
Proofs for CE-yYORCE-Style Regularized Policy Gradients
37
L.1 Proof of Proposition F.I (REINFORCE-style Policy Gradient for Forward KL)
37
L,2 Proof of Proposition F33 (RREINFORCE-styl Policy Gradient for Unnormalized Forward KL)
37
L.3 Proof of Proposition F.4 (REINFORCE-Style Loss)
38
L.4 Proof of Proposition E.5 (REINFORCE-Style Loss for Unnormalized Reverse KL)
38
16

---

Published as a conference paper at ICLR 2026
A
RELATED WORK
Fine-tuning large language models (LLLMs) using human feedback has become critical step in developing capable and aligned AI systems. Broadly, methods fall into two main categories: those relying on policy optimization using an explicit reward model learned from feedback, and those directly optimizing policies based on preference data.
RLHF via Policy Optimization. The classic RLHF involves training reward model (RM) re (2r, y) to predict human preferences and then using reinforcement learning to optimize the language model policy Ty to maximize the expected reward from the RM, often regularizing against deviating too far from an initial reference policy Tyef. This approach was pioneered by Christiano et al. (2017) and gained widespread prominence with its application to LLMS like InstructGPT (Ouyang et al., 2022) and ChatGPT (OpenAI, 2022), which utilized Proximal Policy Optimization (PPO) (Schulman et al., 2017). PPO became a workhorse due to its relative stability, achieved by constraining policy updates via a clipped surrogate objective. The standard PPO setup for RLHF involves the policy TT, a value function Vp, the RM Tæ. and the reference policy Tref-
RLHF via Direct Preference Optimization. An alternative and increasingly popular approach bypasses explicit reward modeling by directly optimizing the policy xo based on preference data, typically pairwise comparisons (yur: M) indicating that response yo is preferred over y for a given prompt I. Inspired by the Bradley-Terry model (Bradley and Terry, 1952), Direct Preference Optimization (DPO) (Rafailov et al., 2023) derived simple loss function directly relating preference probabilities to policy likelihoods under Ta and reference policy Tref- DPO maximizes the relative likelihood of preferred responses using a logistic loss: LDoO x -E(log o oB ((Alogp)]] where Alogp is the difference in log- probabilities of Yu and YI between To and Tee-- DPO's 's simplicity and effectiveness led to its wide adoption in models like Llama-3 (Grattafiori et al., 2024), Qwen2 (Yang et al., 2024), and Phi-3 (Abdin et al., 2024). Numerous variants have followed: SLiC-HF (Zhao et al., 2023) uses a pairwise hinge loss for calibration; IPO (Azar et al., 2024) uses an identity link function; SimPO (Meng et al., 2024) offers a simpler objective focusing
the margin; KTO (Ethayarajh et al., 2024) handles binary (good/bad) feedback; DQO (Ji et al., 2024) incorporates direct Q-value modeling; RAFT (Dong et al., 2023), RSO (Liu et al., 2024) and RFT (Yuan et al., 2023) use rejection sampling perspective. Recognizing that preferences might evolve, iterative methods like Iterative DPO (Xiong et al., 2024), PCO (Xu et al., 2023) and SPIN (Chen al., 2024) alternate between generatiion/prfeeennc learning and policy updates, often using the current policy's 'sicy's outputs in a self-improvemmnt loop. Game theory offers another lens, with Nash Learning from Human Feedback (NLHF) (Munos et al., 2024) framing RLHF as finding Nash equilibrium between policies. Self-play ideas appear in SPPO (Wu et al., 2025) and GPO (Zhang et al., 2025), where the policy generates pairs for comparison. Methods like GPM (Zhang et al., 2025) aim to handle more general preference structures efficiently using latent embeddings beyond pairwise comparisons.
RL for Enhancing LLM Reasoning. Beyond general alignment with human preferences, RL techniques are increasingly explored to specifically enhance the multi-step reasoning capabilities of LLMs in domains like mathematics, coding, and complex instruction following. In these contexts, RL optimizes the policy to generate sequences (c.g., successful outcomes, often using rewards derived from external feedback like unit test results, execution outcomes, or correctness checks by an automated judge or specialized reward model trained on reasoning quality. For instance, the DeepSeek GRPO algorithm, a value-free PPO variant, demonstrating significant improvements in mathematical problem-solving benchmarks through RL fine-tuning. DeepSeek-RI (Guo et al., 2025) represents efforts in applying advanced techniques potentially involving RL for complex tasks, although specific methods might vary. Furthermore, preference- based methods like SPPO and GPO have been applied to maasoning-specialing observed on benchmarks involving coding and math suggest that preference-basee contribute to refining reasoning abilities, potentially by optimizing implicit properties related to logical consistency and correctness within the preference data. The need for a value function (critic model) used in PPO incurs significant computational costs, and standard PPO can face stability challenges with sparse rewards common in LLM tasks. Addressing these issues has driven recent work. Several methods aim to improve efficiency by removing the value network: RLOO (Kool et al., 2019; Ahmadian et al., 2024) shows that drawing multiple samples per input allows for a baseline based on the average reward. ReMax (Li et al., 2024) adapts REÍNFORCE (Williams, 1992)
code blocks) that lead to
model (Shao al., 2024) employed the
models such as Kimi- 1.5 (Team et al., 2025), and the resulting improvements
RL.F can also
17

---

Published as a conference paper at ICLR 2026
using Monte Carlo returns and normalization; GRPO (Shao al., 2024) uses baseline and adds a ky ba-ddeed KL penalty to the objective; and VinePPO (Kazemnejad et al., 2024) uses MC sampling from intermediate steps. Other approaches focus stability and alternative baselines, such as RLOO (Ahmadian et al., 2024), which uses leave-one-out statistics within group, and REINFORCE+ (Hu, 2025), which enhances REINFORCE with token-level KI. penalties (using the 22 estimator) and normalization. Dr. GRPO (Liu et al., 2025) identifies and corrects bias found in GRPO's advantage estimators, DAPO (Yu al., 2025) introduces strategies like Clip-Higher, reward over-sampling, and a token-level loss to handle long sequences and entropy collapse, while VAPO (Yuan al., 2025) builds upon it with length- adaptive advantage estimation. Group Policy Gradient (GPG) (Chu et al., 2025) revisits the original REINFORCE objective, using group-normalized rewards and a debiased gradient estimator. Recently, GSPO (Zheng et al., 2025) was proposed with sequence-level rewards and used in the Qwen3 model series (Team, 2025).
reward
Our contribution is to make the off- off-policy weighting and estimator equivalences explicit across normalizze//unnormalzze variants, to identify bias introduced when these weights are omitted (as in the GRPO KL term), and to provide corrected surrogates that are gradient-equivalent to the intended objectives. The design- design-space view makes transparent how several recent algorithms arise special cases.
B CONNECTION BETWEEN REGULARIZED POLICY GRADIENT AND NATURAL POLICY GRADIENT
In this section, we draw the connection between RPG and Natural Policy Gradient (NPG) (Kakade, 2001; Schulman et al., 2015). Note that NPG moves along the steepest-ascent direction defined by the Riemannian geometry induced by the Fisher information matrix, rather than by the Euclidean geometry of the raw parameters. More specifically, we demonstrate that the Natural Policy Gradient (NPG) update be recovered as special instance of the RPG update by applying a linear approximation to the expected return and a quadratic approximation to the KL regularization term.
In detail, consider the RPG objective at iteration k:
(RP J(0) (L (TO);
(B.1)
where J(Ø) is the expected return. To study the local behavior of the update, we first apply the first-order Taylor expansion to the return J(0) around the current policy parameter 0x:
(0) R= J(On) (ON 20,
(B.2)
where A0 0 - 02 denotes small change.
Then we apply the second-order Taylor expansion of the KI. divergence term 0g as follows:
(I[(oo,((T)) KL [ V KL [T
A0
AO
V3 KL (To TT)) a= A0
(B.3)
2
0 + 0+ 1
1
AOT F(Ox)A0,
(B.4)
where A0 0r. and F(Bx) is the Fisher information matrix:
F(Ox) = Eg E
Ve log To(2)) p=)) Ve log ne <e(z))
(z) l
Note that the Fisher information matrix describes the local geometry of the parameter space of the policy family. It gives a good metric inside a small neighourhood around dc.
Now insert (B.2) and (B.4) back into the RPG objective (B.1), we obtain the following quadratic surrogate JRPo (A)) around 0u:
B
Faea(A0) = J(0z) + VoJ(Oa) A0 (AOTF(0A)00
(B.5)
18

---

Published as a conference paper at ICLR 2026
We now look for the best local step A Take the gradient of (B.5) with respect to A and set equal to zero, we obtain:
V AARGGG VoJ(On) (F())) - 0.
Solving this linear system for A gives
A F(08))
The step A matches the natural policy gradient (Kakade, 2001) update direction up to the factor 1/3. This suggests that the policy gradient update of RPG in (B.1) can be approximated by NPG as follows
Ok+1 On + 3 =() (V.
In other words, the maximizer of the local KL regularized RPG approximation follows the same direction as natural policy gradient update.
C REINFORCE AND PROXIMAL POLICY OPTIMIZATION (PPO)
C.I REINFORCE
REINFORCE performs Monte Carlo (MC) updates after sampling complete trajectory, using the sampled return G2 unbiased estimate of the state-action value function QT (st. at). However, these MC estimates often exhibit high variance, lcading to slow and unstable learning.
To reduce variance, state-dependent baseline b(s:) (commonly an estimate of the state value function, T (st)) is subtracted from the return-to-go:
T Σ (G. - bos:)) Volg (o,ls()))).) = BT
T A v log tg
(oJ = BT
(C.1)
Here, A, A G,sb(s)) is an estimate of the advantage function AT* a,) = Q (s,, a V (s) Subtracting baseline that only depends the state s, does not bias the gradient estimate, since E,,-no(- (-l.) [b(s:) Vo log To (a, st)] = b(s;) E., (o(a)(s)) = b(sz) Vol = O. REINFORCE with baseline is typically implemented by minimizing the loss:
T
CREINFORCE (0) --Er~s Σ SG (G(())
(C.2)
using the stop- stop-gradient operator SG(-) prevent gradients from flowing into the advantage estimate A, As REINFORCE uses samples collected under the current policy my for gradient estimation, is an on-policy algorithm.
C.2 PROXIMAL POLICY OPTIMIZATION (PPO)
On- On-policy methods like REINFORCE be comple-inefficeent requiring trajectories for each gradient update. Proximal Policy Optimization (PPO) (Schulman et al., 2017) improves stability and sample efficiency by enabling multiple updates using the same batch of data collected under slightly older policy Taau This makes PPO effectively off-policy. PPO achieves this by optimizing a surrogate objective function that discourages large deviations between the current policy no and the The most widely used variant, PPO-Clip. employs clipped objective:
old policy Te
jPPO.Clip (0) E, min (w) clip(w.(()) - c, -,,1+ ))))
(C.3)
where the expectation E, is taken over timesteps in the collected batch sampled from Vold- Here, (() ma(a, lex). is the importance sampling ratio. A, is advantage estimate, typically computed [
19

---

Published as a conference paper at ICLR 2026
using Generalized Advantage Estimation (GAE) (Schulman al., 2016). which leverages observed rewards and learned state-value function (s)) to reduce variance.
Notably, in many practical implementations, especially in Reinforcement Learning from Human Feedback (RLHF) for large language models (Ouyang et al., 2022), KL divergence penalty against a reference policy 7ze (e.g., the initial supervised model) is often incorporated implicitly by modifying the reward signal before calculating the advantage. For example, the reward used for GAE calculation might become r/ = т log( To (:)/ss/////// / Trot (ar 8e)). When this r used within GAE compute A, the KI. penalty term is effectively folded into the advantage estimate that multiplies the importance weight wy (0) in the objective function. This approach contrasts with adding the KL penalty separate term to the final objective, as seen in GRPO (Section 2.2) or the formal derivations in Section 3.
The hyperparameter (e.g., 0.2) defines the clipping range [1 - c, c| for the importance ratio (0). This clipping limits the influence of potentially noisy importance weights when the policy changes significantly, preventing destructive updates and further stabilizing the off-policy training. PPO optimizes the policy To by maximizing JPPO-CTip (0).
D EQUIVALENCE OF K3 ESTIMATOR AND UNNORMALIZED KL DIVERGENCE
As mentioned Section 3.2, the K3 estimator for KL divergence (Schulman, 2020) is equivalent to the unnormalized KL (UKL) divergence. The hss function is defined as kz(v) 1 log
Forward KL-k3 and UKL (TO): The forward KL-k3 divergence is KL ( (Toua|]TO) := Expans (T ( Total (z))]
Tooa( ( ]
(x)
[T(x) - - -lo
To (x)
E Exnnas
hi
Exwtas
TTol(())
(x)
T0(x)) -1) dx / malu(z)) (x) (x)))o log To(x) dr [x)a((z)
Maua(r))
(x)
(Te(TT) - -Toua ddx + Tow(()) log Toua(s)) dr TT(
UKK (TToOa
Reverse KL-ka and UKL (mo (o The reverse KL-k3 divergence is KL (To Taia) = Exwnr Iks (Toia(())
Told
Tola(z)
Tota (x)
Boon
k3
E Exwro
- -log
(x)
To(x)
To (ar)
[ola(() 11) dr - Te(x)) log
Tota (x)
Te(x)
dr
To (x:)
T0(#)
(Tota (I) to(x))do + TT(x)) log To To(x) dx Tora (2r)
(JK[(To||Too)))
E NORMALIZED KL REGULARIZATION
For completeness, collect here the normalized KL formulations that were previously in the main text. Their proofs remain in Appendix K.
E. FORWARD KL REGULARIZATION
Consider the objective function with forward KL regularization:
JFKL(0) (0) E [R()] KL(Tata || n0)-
(E.1)
Proposition E.1 (Policy Gradient and Differentiable Loss for Forward KL). The gradient of JFKI, with respect to / is:
(OJFKL(() Eponas (( + B) log (x))
20

---

Published as a conference paper at ICLR 2026
Table Summary fully differentiable surrogate losses for normalized KL-regularized objectives (counterparts to Table 1). Here x ~ Wold, w(x) = xo x
Regularization (Normalized KL)
Surrogate loss (sampling x ~ Told)
Forward KL
E| B[-w(z)R(z)) log (x) E [[w(x) (- R(x) x + 3 log w (x))
Reverse KL
where w(x) = [e(x)// (x). A corresponding surrogate loss is:
LFKL(O) (f) = Egonaal w(z) ml (=())
which satisfies VoLFKL = (OJ)KL(())
Remark E.2 (Connection to Maximum Likelihood Estimation). If R(x) = 0, maximizing JFKI, (0)
reduces minimizing 33 (T TD), i.e., MLE on samples from Wold-
E.2 REVERSE KI. REGULARIZATION
Consider the reverse KL objective:
(R) Eg [R(x)] -
-[ae]]_KK[[A ( Tota).
(E.2)
Proposition E.3 (Policy Gradient and Differentiable Loss for Reverse KL). The gradient of JRKL (0)
is:
V VoJRKL = Eawstar ((xx)
(1(x)) [R(x) (loggw(z) + 1)) Vo log mo())
A corresponding surrogate loss is:
LIKI BRTWTT [w(x)(-P(x)) + 3 w()))
with V VoCRKI(()) = -V.-RKK.(().
REINFORCE-style RPG with normalized KL regularizations. FKIJRKI. appear in Appendix F (Table analogues to Table 2).
REINFORCE-style losses for
REINFORCE-STYLE REGULARIZED POLICY GRADIENTS WITH VARIOUS KL REGULARIZATION FORMS
F.1 RATIONALE FOR REINFORCE-STYE Loss FORMULATION
As noted in Section of the main text, the derived off-policy policy gradients (Theorems E.1 through 3.6) share structural similarity with the REINFORCE estimator:
J(O) Egg-saamiigg (Weight(ar, 0) log To()))
This structure suggests alternative way to implement the gradient update, analogous to the REINFORCE-style approach used in the on-policy setting. Specifically, one could define a surrogate loss of the form:
Ry (0) - -E Tumplon [SG (Weight 0)) log no (z)]
(F.I)
The rationale is that applying automatic differentiation to this loss should yield:
V CeCCEENNFORCE-spte (0)
Autodifff
[SG (Weight(z, 0)) Va log (x)
When this gradient is used for optimization, the stop-gradient SG is conceptually removed, resulting in an update aligned with - ((). This relies on SG preventing gradients from flowing through the dependence within Weight(æ, 6) (specifically, the dependence via the importance weight w(x)). The following subsections detail these REINFORCE-sth loss formulations for each KL regularization type.
21

---

Published as a conference paper at ICLR 2026
F.2 REINFORCE-STYE RPG WITH FORWARD KL REGULARIZATION
We can convert Forward KL regularization of RPG to REINFORCE-style using the stop-gradient operator:
Proposition E1 (REINFORCE-Sty)e Loss for Forward KI.). For the forward KI. regularized objec- tive function in Eq. (F.1), the corresponding REINFORCE-style surrogate loss function for gradient descent optimization via automatic differentiation is:
REINFORCE-syle (0) - Eezanaa [SG (w(x) + (3) log (x))
FKL
where w(x) = /r((i//Tola() This loss aims to produce the gradient - VVJJKL(()) via automatic
differentiation.
Remark F.2. This REINFORCE-style loss requires SG prevent backpropagation through w(xx) in the weight term. Baselines can be added to R(x) inside SG for variance reduction (see Appendix G). In practice we further apply RPG-Sryle Clip (Section 3.3) by replacing w with and, when present, log with log i inside S-(-).
F.3 REINFORCE-STEE STYLE RPG WITH UNNORMALIZED FORWARD KL REGULARIZATION
Similarly, we can also transform the Unnormalized Forward KL Regularization of RPG into REINFORCE-style follows:
Proposition F.3 (REINFORCE-Syy) Style Loss for Unnormalized Forward KL). For the objective JUFKL(() (0) = E [R(x)] 3 (TolaTill whose gradient (sampling from Tola) is
VaJUFKL (0) Ezw (Zona (w( R(x) B(w(x) - V)) log [T(r)] (Proposition 3.2), corre-
sponding REINFORCE-styy surrogate loss is:
REIINOOREE- UPKL
(0) --E [SG (Zola (w(c))((z) (ww(x) 1))) log no (x())
where Hold Told/ Zold and w(x) (otd (using unnormalized m.aa). This loss aims produce the gradient - -VeJUFKL (0) via automatic differentiation.
F.4 REINFORCE-STYLA RPG WITH REVERSE KL REGULARIZATION
Proposition F.4 (REINFORCE-Styl) Loss for Reverse KL). For the objective JRKL
Emp [[x]]] - (L Rold), whose gradient VoJRKI, (0) Ega (=(x) 3(log w(x)+
V log T (Proposition E.3), corresponding REINFORCE-styl surrogate loss is:
REINFORCE-atll (0) Eg [SG (w(x) (R(x) 3 log w(x) B)) log (0(2)
RKL
(F.2)
where m((x) ()(2)/ 3 (ota(<)) This loss aims to produce the gradient VoJuKI (0) via automatic
differentiation.
F.5 REINFORCE-STI STYLE RPG WITH UNNORMALIZED REVERSE KL REGULARIZATION
Proposition F.5 (REINFORCE Style Loss for Unnormalized Reverse KL). For the objective
JURKL. (0) Ex, [(x)) 3 UKL (To Tota), whose gradient (sampling from Roia) is
VaJURKI. (0) Ee Rold Zotaw (R(x)) 3 log w(x)) Vo log to(#)) (Proposition 3.6), a corre-
sponding REINFORCE-style surrogate loss is:
REINFORCE- (0) = -E [SG (Z))w(z) (R(x) .3 log w(x))) log (2)),
LURK
where Told = Told/ Zola and w(.) = Te
(x) (using unnormalized mata). This loss aims to produce the gradient -UJUUKI (0) via automatic differentiation.
G MORE ON ALGORITHMIC DETAILS
G.1 STABILIZATION TECHNIQUES FOR REGULARIZED POLICY GRADIENTS
Practical implementations oof-py policy gradient methods often require stabilization techniques to manage variance prevent destructively large policy updates. Common techniques include:
22

---

Published as a conference paper at ICLR 2026
Dual Objective: This method adapts the clipping mechanism from PPO, with modification for negative advantages proposed by Ye et al. (2020), to stabilize updates (Schulman et al., 2017).
The Dual Clip objective aims to maximize Duuillipp = Ex-raoa [L PuulCtipp (x, .)], where A(x) is an estimate of the advantage analogue (e.g.. R((x) b or the full term derived from the regularized gradient), w(x) =
is the importance ratio, and L DualClip (z) 0) is defined as:
-If ((x) 2 0: I.DuaClip(x, (z, 0) = min(w(()AA(t) clip(w(a)), 61, 1 +2) Axx)) - Iff A(x) < 0: (s, 0) = max(min(w(()))) ((()) clip( ( - 61,
(2) cA(x))
where 61 are clipping parameters and > 1 provides a lower bound for negative advantages. To use this with gradient descent (which minimizes a loss L), we minimize the negative of the Dual Clip objective term. Using min(a, b) max( -b) and max(a, b) min(-a. -b). b). the corresponding loss term for a single sample is:
If A(x) 2 0: DuualClip (z, 0) = max (((())((r), - clip(((x), -1, (2) e2)
-If O: Let Lelip max (-w(x)((x)) ( -cip(w(=))11 +2) Then,
CDuulClip 0) min [eeipp -cĀ(x))
Here, A(x) should represent the advantage analogous term derived from the gradient of the original neg-nee regularized objective (c.g.. Proposition E.3). The overall loss [()) = Ez [CDullCip( (zz, 0)]. This loss function is differentiable with respect to 0 (which appears in w(x) and potentially A(x) it includes terms like log w(x)).
This loss formulation ensures that updates conservative. For positive advantages, it acts like
standard PPO-Clip. For negative advantages, it prevents the objective from becoming arbitrarily large (loss becoming arbitrarily small) by introducing the lower bound cA(z) on the objective
(upper bound- (xx on the loss).
Baseline Subtraction: Used define the advantage A(z) R(z) b(z). reducing the variance
the gradient estimates. The baseline b(x) should ideally not depend strongly on A. A common choice is value function estimate V(x) simply the batch average reward b 1/2 Σ R(x,). The
definition of A(x) might also incorporate regularization terms depending on the base objective chosen (see RKL example below).
For instance, applying Dual Clip to stabilize the reverse KL objective (Proposition E.3). The gradient involves the term w(x) ((R()) - b) 3(log w(x) + 1)) Voog æT. Using this ARKL in the Dual Clip
Analogue to Auxa wx;
loss structure <RKI, (O) (0) = Ex [CRI.
DuulClip (x, 0)] where:
If ARKI. (Ir, b) 2
DualCliip (x., max -eu((//Axx -aci::--clip(((1 clip((((x), 61, (2) ARKI.
RKL
TT.RRKL (I, wi < O: Let Lelip = max (-w())ARkL ( -lpp(u(()..p___i.... +2)) ARKL)
DuulCCiip (x. 0) = min (ninLLeip:
Leup: - -cÂRKL).
where ÂRKI. (x, w; = (R(x)) b) -3og w(x) 1). Simpler approximations might use A(x) R(z) b.
Using PPO-style clipping alters the optimization objective compared to the original KL-regularized objectives, trading strict adherence for enhanced stability. The choice of base objective structure, definition of A, and stabilization techniques depends the specific application.
23

---

Published as a conference paper at ICLR 2026
Algorithm 1 RPG with Dual-Clip Stabilization
Require: Reference policy Told, Reward function R(r). Initial policy parameters Oo
Require: Base objective structure Jehssen (implies regularization type). Regularization strength 2 0 Number of epochs K 2 per iteration
Require: Learning rate a > O, Batch size N
Require: Dual Clip parameters: é1 > 0, é2 > 0, c >
Require: Baseline method (e.g., batch/group average, value function Ve)
1: Initialize policy parameters 0 4- Og
2: Initialize value function parameters (if baseline uses Voo)
3: for each training iteration do
4: Sample batch 2> = {0t) ~ Vold
D Collect data using old policy
5: Compute R: for i = I..N
6: Compute baselines by for 1..N (e.g.. E, R2 or Vo(=))
7: for = 1 to K do
b Multiple optimization epochs on the same batch
8:
Initialize batch loss Cnuch 0 NV do
9:
for ma(zz)
Wt; Toad log w: log no (I::) log (a:
Compute importance weight
11:
Define Advantage analogue Ag based on Jehosens Ri bi, Wi, B.
12:
Ex: For RKL, A, = (R, b.) B(log uy + 1). Note: A, depends on current via t0;
13:
if Dual Clip enabled then
14:
loss_terml, -0; x A,
> Negative of unclipped term, gradient flows through to;
15:
Wicelippal clip(w:,
e2)
16:
joss_term2; = cellippodd x A.
> Negative of clipped term
17:
(elip() max(loss termi;, loss poss_term,,)
18:
if A; 2 0 then
19:
Lum (i) Lelip
20:
else
Å, 0 Lower bound term
21:
loss Jowerlowonnee Liem (i) = min(Lemp()) joss_lower_bound,
A.
22:
23:
end
24:
else
25:
D Define base loss term (unclipped) based chosen objective's negative gradient structure Ex: For RKL loss (no clip): Liea( w,(-(R, b.)) + 3 log w:)
27:
Liam( Wi Ai
end if
29:
Learch Lbauch + Luma (i)
30:
end for
31:
(()) Craneh
Compute final batch loss for minimization Compute gradient (flows through u, and A.)
32:
VoL(o) OptinizerUppdatee,. g0.g. .g....)
P. Update policy parameters
34:
if luuiing a learned baseline Ve then
35:
Update value function parameters o (e.g., by minimizing (x:)) end if
R2) over the batch)
36:
end for
38:
end for
39:
return Optimized policy parameters
G.2 STABILIZATION TECHNIQUES FOR REINFORCE-STYE REGULARIZED POLICY GRADIENTS
While the REItFoRCE-style losses derived in this section (Table 2) provide theoretically grounded gradient estimators for the regularized objectives, practical implementations often benefit significantly from stabilization techniques common in policy gradient methods. These techniques aim to reduce variance and control the magnitude of policy updates, which is especially crucial in the off-policy setting where importance weights w(x) and can exacerbate instability.
Baseline Subtraction and Regularized Advantage Definition: This is a standard variance reduction technique. Critically, when combining with stabilization like PPO clipping in this REst context, the term playing the role of the advantage (Ā,) that gets clipped should
24

---

Published as a conference paper at ICLR 2026
ideally incorporate not just the baselined reward but also the regularization terms derived from the objective's 's gradient.
Recall the REINFORCE-style gradient structure VQJ(0) Eugni (Weight(r, V)Vo log mo (1))
The PPO objective involves terms like w, To align these, we define the regularized advantage A, such that A approximates the key part of Weight(a,, 0). For example:
For RKL (Proposition F.4), WeightRKL (R()( Blog w(x) + 1)). We define the regularized advantage as ABKI. (R(x) b(z)) (log w(x) + 1).
For URKL (Proposition F.5), Weight (oaw(r)(R(x)) 3 log (x))) Ignoring Zold. define AURKL (R(=) b(z)) log wxx).
- -oor FKL or UFKL, the structure might not cleanly separate into w(x) x (...). such cases, common simplification is to use A R(x) - b(x) and accept that the clipping primarily stabilizes the reward term's 's contribution.
This calculated A, (incorporating reward, baseline, and KL terms) is then treated as constant using the stop-gradient operator, (G(),,, when plugged into the clipping loss function.
RPG-Style Objective Clipping (Dual-Clip Variant): PPO (Schulman et al., 2017) introduces objective clipping limit the impact of large importance ratios w(x). The Dual-Clip variant (Ye et al., 2020) refines this, particularly for negative advantages, using a lower bound parameter c > 1.
When applied the REIsFORCE-style setting, the PPO Dual-Clip objective aims to maximize (simplified notation, expectation over Tola):
(u) = E, [L. (ILL)Ili((()))
where Ag is the regularized advantage defined above (incorporating R4, bi, and KL terms), wy(()
xo(a)ls)) mata (5)) and I, (uullClip(( (0) is defined based on the sign of SG())):
IFSG(AA) 2 0: L, DuniClip (0) min(w.(9) SG(A,), (i)((,(()) 1- e,, (,.11+)))SGAA (G) max(min((w(()) (G())), clip(wx (0), c1,
If SG(()) < 0: L DunlClip
())GG()),,
Here, 61. (2 are clipping hyperparameters, and c the lower bound factor. Note that influences this objective only through wx) as A, detached via SG.
To implement this using gradient descent (minimizing loss), we minimize the negative of the PPO Dual-Clip objective. The loss function becomes (Da E, Dualiliipp (0)) where
C. c.uulCllip (0) = LL, PumiCCip (0). Explicitly:
If SG.,)) 2 0: L. DunlCCii (0) max(- SG( -lip(wllpp))ppp (0), c1, 1 ( (G().))
If SG(A)) < 0: Let Leup max(-w,()) (Gt),,, cclpp(,, (0), en, + e2) SG (A)).
Then, c, c.natclip (0) min(Lelip- (A.))
This PPO Dual-Clip loss function [DualClip (0) replaces the simpler REINFORCE-stye losses derived earlier (like LRKL.
&REINFORCE-sllE in Eq. (F.2).. The gradient DustliCC (0) is computed via
automatic differentiation, where the gradient flows through (x) but is stopped at Aç. This approach uses the PPO objective structure with the appropriately defined regularized advantage for stabilization in an off-policy REINFORCE-style update. Algorithm 2 details this implementation.
H DETAILED EXPERIMENTAL SETUP
Hyperparameters. Unless otherwise specified, all experiments use AdamW optimizer (Loshchilov and Hutter, 2019) with learning rate of 1 x 10-6, weight decay of 0.1, and gradient clipping at 1.0. Training proceeds for 400 steps, including an initial 10 warm-up steps, after which a constant learning rate is maintained. The global training batch size is 512. For each sample in the batch, we roll out 16 responses using a temperature of 1.0. The GPP mini-ban batch size is 32, and experiments conducted on 8 NVIDIA H100 GPUs The maximum training and rollout length is set to 4,096 tokens for 2K context length and 8.192 tokens for a 4K context length, with dynamic batching enabled. The KI. regularization coefficient / is set 1 x 10-4
25

---

Published as a conference paper at ICLR 2026
Specific Clipping Parameters and Adopted Techniques. As mentioned in Section 5, set (61.62) E2) = 0.22, for RPG, RPG-REINFORCE and DAPO. For GRPO, we use (c1. = 00.1,0.1).
Algorithm 2 REINFORCE-Style RPG with Dual-Clip Stabilization
Require: Reference policy wota, Reward function R(x), Initial policy parameters 8
Require: KI. Component function Compute KL. Component (x. mata), KL Component Coefficient
Require: Learning rate a > 0, Batch size N > 0, Number of epochs K 2 per iteration
> 0 (low), E2 > 0 (high), > Require: Baseline method (e.g., batch average, value function Vτ)
Require: Dual Clip parameters:
Initialize policy parameters 8 + 0g
2: Initialize value function parameters & (if baseline uses V&)
3: for each training iteration do ) Vold
4: Sample batch D
5: Compute rewards R. for 1..N
6: Compute baselines 6,
1..N (e.g.. b:
E, R2 or Vo()))
7:
for k 1 K do Initialize batch loss Luuch 0 for = to do w; = ma(Fe)
b Multiple optimization epochs on the same batch
8:
9:
10:
> Importance weight
11:
L, = log (x.)
b Negative log probability Baseline-subtren subtracted reward
12:
Ag, RR b.
13:
CKL.1 = 33Copuue KL Component(zs, 0, (x)()))
KL component
14:
A₁ AR, SG SG(w))
Effective advantage
15:
#. = A/ x
b Branching term
16:
if th 2 then
17:
Whigh 1 4
18:
if < Whigh then
C. = th SC(w:)
Grad exists
20:
else
P Wi 2 Whigh
21:
Ahigh AR, + SG(CKL.))/ (G(wi))))
thigh Aniig x SG(),
23:
C. = thigh x (Winigh))
24:
end
25:
else
t/
S
Whow
61
if ur < Whow then
28:
A/om AR.i + SG(CCL.)) SG
29:
Mlow Alow x SG((.)
30:
C. thow (WWW)))
else w; < then
32:
C. = v; x SG(w:)
D Grad exists D 00 2
33:
else
34:
C. ARa SG(L.) x c SG(CKL.) SG(A.)
35:
end if
37:
Dzunch Liuuch + L; end for
38:
39:
C(0) Lraneh
Compute average batch loss & Compute gradient
VoL(O)
41:
OptinizerUppaate(,, g.a.
b Update policy parameters
42:
if using learned baseline Vo then
43:
Update value function parameters 6
44:
end if
45:
for
46:
end for
47:
return Optimized policy parameters 0
26

---

Published as a conference paper at ICLR 2026
2
2
0
120
(a) AIME24
(b) AIME25
(c) AMC23
(d) Reward (Critic Score)
(e) Entropy
(l) Response Length
Figure 5: Performance of RPG and REINFORCE-SO Regularized Policy Gradient (RPG-
RÉINFORCE) methods compared baselines with 4k context length.
I ADDITIONAL EXPERIMENT RESULTS
1.1 THE PERFORMANCE WITH 2K AND 4K CONTEXT LENGTH
The quantitative results in Table 5 demonstrate the competitive performance of the proposed RPG and SEINFORCE-StyLE RPG frameworks with 4k context length. On AIME24, EPG-REINFORCE variants lead, with "Last" score (0.4458), while NUG-REINFORCE-URR attain second best "Last" score (0.4281). For AIME25, "Last" score (0.4125) and RPG-REINFORCE-UFER is second only to that. Overall, RPG and RPG-REINFORCE methods rank near the top across benchmarks and metrics, while exhibiting stable training dynamics.
achieving the best "Best" score (0.4531) and the best
still achieves the top "Best" score (0.4313) and strong
Similarly, Table shows the experiment results with 2k context length. It can be observed that RPG and RPG-REINFORCE variants demonstrate robust performance, often competitive with or exceeding baselines. For example, RPG-REINFORCE--UI AIME24 (0.3625) and AIME25 (0.3083), and the top "Last" score of AIME25 (0.2927), while RPG-UFKL attains the top "Last" score AIME24 (0.3427) and the second highest "Last" score of AIME25 (0.2833). Their training curves in Figure 6 generally indicate good stability and effective learning. The consistently high performance across various RPG formulations underscores the utility of the systematically derived KL-regularized objectives explored in this work.
achieves the top "Best" scores for
1.2 ABLATION STUDY
To further investigate our algorithms, we implement an ablation study on the clip ratio and the effect of the KL regularization coefficient.
I.2.1 ABLATION ON CLIP RATIO
We first implement experiments with different clip ratios REINFORCE-styL RPG algorithms. We choose (0.1, 0.1) and 0.28) for (e1, since they are typical choices of clip ratios (Schulman et al., 2017; Yu et al., 2025), and the performance displayed in Figure similar for different settings, the actor entropy shows huge difference in trend, demonstrating that
well as key training dynamics can be observed that although the critic score and response length are
27

---

Published as a conference paper at ICLR 2026
Table 5: Combined performance metrics with 4k context length on the AIME24, AIME25 and AMC23 mathematical reasoning benchmarks, showing "Last" and "Best" scores. The "Last" score is from the 400th training step, assuming the training process remained stable that point. The highest score in each column is bolded, and the second highest is underlined. RPG and RPG-REINFORCE methods are highlighted with light cyan and light green backgrounds, respectively.
AIME24
AIME25
AMC23
Method
Last
Best
Last
Best
Last
Best
REINFORCE:++-aase Baseline
0.3854
0,4302
0.3406
0.3844
0.9234
0.9328
REINFORCE++
0.3490
0.3885
0.2822
0.3479
0.8977
0.9297
GRPO
0.3458
0.3677
0.2896
0.3042
0,9016
0.9109
DAPO
0.4063
0.4479
0.3510
0.3938
0.9297
0.9297
RPG-UFKL.
0.4031
0.4396
0.3625
0.3979
0,9477
0.9500
RPG-URKL
0.3990
0.4219
0.3438
0.3792
0.9500
0.9531
RPG-REINFORCE-UFKI
0.4281
0.4375
0.3771
0,4042
0,9023
0.9133
RPG-REINFORCE-URKLL
0.4458
0.4531
0,4125
0,4313
0.9313
0,9352
6.08
(a) AIME24
(b) AIME25
(c) AMC23
Jole (2111)
Q
(d) Reward (Critic Score)
(c) Entropy
(f) Response Length
Figure 6: Training dynamics and benchmark performance for fully differentiable Regularized Policy Gradient (RPG) and REINFORCE-Sty. RPG (RPG-REINFORCE) compared to baselines (GRPO
and DAPO) with 2k context length.
adequately higher and clip-higher strategy proposed by DAPO may greatly contribute to the increase of performance by increasing the actor entropy.
1.2.2 ABLATION ON KI. REGULARIZZATIO COFFFICIENT
We also implement ablation studies on the effect of the KL regularization coefficient. We implement experiments with REINFORCE-style RPG-UFKL (RPG-RESS REINRRECIE-ERRKKEEE with 3 = 1 x 10-3 and 1 10-4, and the results are shown in Figure 9. Figures (a) and 9(b) show that the coefficient 1 x 10 performs better than x 10-3 and the trend response length conforms to the performance, indicating that longer response length may help with the performance improvement.
We also dig into the effect of the iteratively updated reference model. We implement another experiment with no iteratively updated reference model, and display the performance and dynamics
28

---

Published as a conference paper at ICLR 2026
Table 6: Combined performance metries with 2K context length on the AIME24, and AIME25 mathematical reasoning benchmarks, showing "Last" and "Best" scores. The "Last" score is from the 400th training step, assuming the training process remained stable that point. The highest score in each column is bolded, and the second highest is underlined. RPG and APG-REINFORCE methods are highlighted with light cyan and light green backgrounds, respectively.
AIME24
ATME25
Method
Last
Best
Last
Best
GRPO
0.2563
0.2708
0.2323
0.2479
DAPO
0.3229
0.3281
0.2792
0.2844
RPG-UFKL
0.3427
0.3479
0.2833
0.2833
RPG-URKL
0.3260
0.3594
0.2677
0.2677
RPG-REINFORCE-URE
0.3396
0.3625
0.2927
0.3083
RPG- Www-RREINFORE-URE -URKL
0.3188
0.3417
0.2792
0,2938
Table Combined performance metrics with 4k context length on the Minerva-Math and Olympiad- Bench mathematical reasoning benchmarks, showing "Last" and "Best" scores. The "Last" score is from the 400th training step, assuming the training process remained stable to that point. The highest score in each column bolded, and the second highest is underlined. RPG and RPG-REINFORCE methods are highlighted with light eyan and light green backgrounds, respectively.
Minerva-Math
OlympiadBench
Method
Last
Best
Last
Best
MUINFORCE++BBaelinee
0.1250
0.1471
0.4926
0.5875
REINFORCE++
0.1691
0.1728
0.4778
0.6202
GRPO
0.1029
0.1177
0.4926
0.5594
DAPO
0.0993
0.1507
0.5163
0.5727
RPG-UFKL
0.1397
0.2059
0,4688
0.5564
RPG-URKI.
0.1654
0.1654
0.4837
0.5801
OPG-REINFORCE-O UFLLL
0.1360
0.1434
0.5074
0.5564
OPG-REINFORCE. UULL
0.1
0.1434
0.4674
0.5816
(a) AIME24
(b) AIME25
(C) AMC23
Figure 7: Pass @ 32 performances for fully differentiable Regularized Policy Gradient (RPG) and Seee RPG (RPG-REINFORCE) compared to baselines with 4k context length.
in Figure 9. can be observed that the performance recovers with longer response length and much lower actor entropy, showing that longer response length can be important factor and indicator of the performance on benchmarks.
I.3
EXPERIMENTS ON QWEN-2.5-1 2---2225-7B--STRRRE
I.3.1 REGULARIZED POLICY GRADIENT USING QWEN- 2.5-7B-INSTTTTT
29

---

Published as a conference paper at ICLR 2026
MODUEOO PARIQUE
01
(a) AIME24
(b) AIME25
(c) AMC23
(d) Reward (Critic Score)
(e) Entropy
(f) Response Length
Figure 8: Performance of REINFORCE-Style Regularized Policy Gradient (RPG-REINFORCE) methods with different clip ratios with 2k context length. Plots display accuracy on mathematical reasoning benchmarks (AIME24, AIME25) and key training dynamics (reward, policy entropy. response length).
(a) AIME24
(b) AIME25
(c) AMC23
(d) Reward (Critic Score)
(c) Entropy
(f) Response Length
Figure 9: Performance of REINFORCE- Regularized Policy Gradient (RPG-REINFORCE)
methods with different KI. coefficients with 4K context length. Here, "noiterref" indicates the model trained with no iteratively updated reference model. Plots display accuracy on mathematical reasoning benchmarks (AIME24, AIME25) and key training dynamics (reward, policy entropy, response length).
30

---

Published as a conference paper at ICLR 2026
Table 8: Combined performance metrics on the AMC23, AIME24, and AIME25 mathematical reasoning benchmarks with Qwwnn-255-7B-nssruuut model, showing "Last" and "Best" scores. The "Last" score is from the 400th training step, assuming the training process remained stable to that point. The highest score in each column is bolded, and the second highest is underlined. RPG and RPG- REINFORCE methods are highlighted with light cyan and light green backgrounds, respectively.
AMC23
A1ME24
AIME25
Method
Last
Best
Last
Best
Last
Best
GRPO
0.6266
0.7250
0.1094
0.1406
0.0281
0.0948
REINFORCE++
0.7625
0.7664
0.0521
0.1177
0.0302
0.0740
REINFORCE++BBssllnn
0.8711
0.8711
0.0990
0.1510
0.0656
0.0969
DAPO
0.8039
0.8734
0.0760
0.1240
0.0531
0.
RPG-FKL
0.8695
0.8836
0.1083
0.1490
0.0427
0.
RPG-RKL
0.8648
0.8672
0.1167
0.1469
0.0677
0.1240
RPG-UFKL
0.8703
0.8703
0.0885
0.1427
0.0927
0.1177
RPG-URKL
0.8258
0.8641
0.0875
0.1271
0.0677
0.0917
CIG-REINFORCE-HE
0.8727
0.8727
0.1208
0.1667
0.0573
0.0875
OPG-REINFORCE-RKE
0.8305
0.8516
0.1125
0.1375
0.0490
0.0875
RPG-REINTORCE--UKK.
0.8391
0.8602
0.1229
0.1458
0.0740
0.0979
RPG-REINFORCE-URKI
0.8531
0.8531
0.1208
0.1500
0.0813
0.0938
(a) AMC23
(b) AIME24
(c) AIME25
(d) Reward (Critic Score)
(e) Entropy
(f) Response Length
Figure 10: Performance of fully differentiable Regularized Policy Gradient (RPG) methods compared to baselines when using base model: Qwen-2.5-7hh I-n-nruutt Plots display accuracy on mathematical reasoning benchmarks (AMC23, AIME24, AIME25) and key training dynamics (reward, policy entropy. response length).
31

---

Published as a conference paper at ICLR 2026
J PROOF OF THEOREM 2.1 (GENERALIZED POLICY GRADIENT THEOREM)
Proof. The proof relies on the log-derivative trick, VoTo(=) = To (2)Vo log (T(), and the product rule under the integral sign:
Fo V f (or
(((T)())f(z)0) (Swap V. f)
((VoTo () dz
de (Log-derivative)
(3 Vof(,,
E Vo Vof(2,
K PROOFS FOR REGULARIZED POLICY GRADIENTS
This section provides detailed proofs for the theorems presented Section 3, demonstrating that the gradients of the proposed fully differentiable off- off-policy surrogate losses correspond to the negative gradients of the respective original objectives. The core tool used the policy gradient theorem: VolEzzwar [f 0)] [f 0) V log To(x)) Vo](x, We use the notation w(xx) = =T(x)// Tola (z) for the importance weight.
K.1
PROOF OF PROPOSITION E. (POLICY GRADIENT AND DIFFERNTIABLE LOSS FOR NORMALIZED FORWARD KL)
Proof. We start by rewriting the objective function JFKL( using expectations with respect to the fixed reference policy Told- The first term, the expected reward under Ta, can be rewritten using importance sampling:
E Exwat [R(x)] = f To(x)
Te(x) (x)a( Rold (z)R(() = Exwnntia [(x))
The second term the forward KL divergence:
Told (2)
KL(Told TT) = Epontaa
log
T
[log Rold (x)
log [x))
Exwaaa log [x))
E Ee [log tota
Substituting these into the objective function:
JFKL(() (f) = Ex [w R(x)) (E |- log To [x]] + E [lgg (x))
E Ta (() log [x BT [log Tota (z)].
Since Tota (x) does not depend on 0, the term BE [lg (x)) is a constant with respect to 0. Now compute the gradient VoJFKL Assuming (standard assumption in policy gradient methods):
swap gradient and expectation
VoJFKL(O) Vo ToEaa [w(x))(z)) + 3 log Te [z)]
Eg ExnRoad (Vo(w(() + 3 log To [z())
E Ex ((Vou(z)) + 3V log (x))
32

---

dr Published as a conference paper at ICLR 2026
We the identity for the gradient of the importance weight: TT)
Vow(x)) V
Told (x)
1
VOTT (2:)
(x)
To (x)
VOTO(z)
(o))
TT(x))
w(x) Vo log (x)).
Substituting this back into the gradient expression:
VaJFKI. = Exwwssa
+ Va
log ne (3))
E Egwstald
V Vogog
This proves the first part of the theorem.
Now, consider the surrogate loss function:
(f) = Exenoo [-w(x) - log To(x))
We compute its gradient: V VoLFKI
V VolEmowstal
(wxx) 3 log (x)]
ExwwRald
V (e(-w(z)R((z) - B log (x(x))
Ezwsool (Vow(z))B(z) Vog [T(2)]
Exwzald
-w (o) 3V [o))]
-Ezwraid
[(=()) + B) Vo log [o))
Comparing this with the gradient of the objective function, we see that VoLrkI, -0)== (0). This confirms that minimizing LFKL(() corresponds to maximizing (Fk using gradient-based methods.
K.2 PROOF OF PROPOSITION 3.2 (POLICY GRADIENT AND DIFFERENTIABLE LOss FOR UNNORMALIZED FORWARD KI.)
Proof. We start by expressing the components of JUFKL (f) using expectations over the normalized reference distribution Rata (z) mota ( (zi) The importance weight is w(x) (o)) (Tt)/))) which implies To (x) = Toi = wwxx (ol))
The expected reward term:
/ no mo(x) / (x))
Exazy [R(x)]=
f w(x) Zola (x) R(x)dx R(z)) = ZoraEara (w())
The unnormalized KI. divergence term UKL (TKaaTTTalll has two parts. Part (Generalized KL):
/ (ota(t)) log (x) dx ZosaFolaa()) log (ola(x) (T( To(x)
ZolaLewntord log w(x)
Zola |- log w(x)].
Part 2 (Mass Correction):
f (Tol(x) (ota(x))dr
[ ((w(x))ssa(a)) Tota (x))
=((u(z) (w((x) (z) -
(w(x)) - 11) Zota Total (2) (x)z Bawa (w(x)) ZolaEernten [w(x)] Zold-
33

---

Published as a conference paper at ICLR 2026
Combining these parts for the UKL term:
UKL(T (e) Zota [- log w())] + Zold [w(x)] - Zold.
Now, substitute everything into the objective JUFKI. (0):
(UFKI.()) Raaaaa (x)
(Zoia E Bew (- log (x:)] Zotd [uo(x)] Zoid) Zunita (0) [(x) + 3 log w(x) - /w((x) + 3]
To compute the gradient JuJUFK we differentiate the terms inside the expectation. The constant term BZold (arising from inside the expectation) vanishes upon differentiation.
VaJUFKI. (0)
Va Boan (w(x) R(x) log w(x) - 3wr(x))
Zola Baaaar (Vo(w(z) + VV w(x)) [V((())))
We need the gradients of wxx and w(x):
Vow(x) w (z) Vo log xo(x) (as derived in Proposition E.1 proof) log w(x) = (lolog - log aa) = V (xr).
Substituting these into the gradient expression:
VaJUFKL(() = ZoldBeeRala
[(Vo((z)) + AV, (V - [Vow(())
Zold Bew
[o (z)) a (o) Bro(x) [o)
ZoldBe-Ra
[<w[))[(x)) - Bw(x) V (o))
ZoldBenHola
(wr))((r)) (w V Tog
This proves the first part of the theorem.
Now, consider the surrogate loss function:
(UFKL( Zola Enow [-w(xx)((x)) + B(w(())) wxr) 1)]
We compute its gradient:
VoLUFkL(0) = Zold Ez
(())
log w(x) - 1)]
Zola E By
(aau())))(( V log (x))]
Zold Bzn
Vo (( log Tx(x) Vo log no (x))]
[( (-w(x))(z)+ (w(x) - 3 V log T(())
Zold By-
= - BeoiaEzzRas (w(())((()) (x)
V log Fa
Comparing this with the gradient of the objective function, we find LUFUFL)( VoJurkL This confirms the surrogate loss function. Note that the constant -1 inside the logarithm term in the loss LUPKI corresponds to the constant 3Zold in the objective JUFKI, and does not affect the gradient.
K.3 PROOF OF PROPOSITION E.3 (POLICY GRADIENT AND DIFFERENTIABLE LOSS FOR NORMALIZED REVERSE KL)
Proof. We rewrite the objective function JIKL(O) using expectations with respect to Told- The expected reward term is Ezaro [R(z)] Ezaxsss (o(x))(()) as shown previously. The reverse KL divergence term is:
To
KT Rold)
E ES log
(x)
= -E [log w(x)]
f To(x)) log wx:)dd
-f To(#))
(x)a Fold (x) log w(
Exonau (wxxx) log w(x)
34

---

Published as a conference paper at ICLR 2026
Substituting these into the objective function:
JRKL (f) Eeworar R(x) [ (w))x))))((()))))))))))))))))))) log w(x)]
E Eo [(x)))(()) - -B((x) log wx))].
Now we compute the gradient V JRIRKL((0))
VoJNKL(()
V VoL [u(z) Bw(x)) log w(x)]
Ex Ex [V(((())R(z) - VVo(w(x) (og])))))))
We need the gradient of w(x) w())
Ve(w(x) log w(x)) (Vew(z))
= (w V, m
To (z)) log w(x) + (V) (Volo
w (z) Va log me (z) (log w(x) 1).
Substituting this and Vgw(2)) = wi (x) V log (x) into the gradient expression for JRKI, (0):
(G)RKL(O) = Ezwsaad (Vow(x)) [8 V log (z) (xx) + 1)]
Bewnald
(w(x)( (2)) - Bw(x) (xx) + 1) log To (x))
Ezwxaldd
w((x) [(xx) (log w(x) + 1) Vo log log (z)
This proves the first part of the theorem.
Now, consider the surrogate loss function:
(R) Enwsata [u(z)(-R(x) + log w(x))
We compute its gradient:
VeLRKI.
V PaE
E ExwsRold
V ( (og
Exwxold
(og +1]
Exwwald
w())((V log (z) fw(l)) wxx) 1) log [=())
EZNATi
w(x)( (((x)(-R(x)) 3log w(x) 1) V log To (xx)
-g
(w(x)) [((x) 3(lo w(x) + 1) V log (x))
Comparing this with the gradient of the objective function, we confirm that VoLRKL (0) -VoJRKL(0).
K.4 PROOF OF PROPOSITION 3.6 (POLICY GRADIENT AND DIFFERENTIABLE LOSS FOR UNNORMALIZED REVERSE KI.)
Proof. We again express the objective components using expectations over the normalized reference distribution Hota (I) Tota (x)) (z)/ with wxx) =e(xz//Tal (x).
The expected reward term: Eg [R(x)) Zota Eu (w(x) R(r)).
The unnormalized reverse KI. divergence UKT. UKT.(To|laa has two parts. Part (Generalized KI.):
J no log
Tπ(x) dr / (=(x) log w(x)drr
(x))
w TTta (2)) log w(x)dr
w(x) Tota log w(x)ds
Zold Ex (w(x) log wxx)]
35

---

Published as a conference paper at ICLR 2026
Part 2 (Mass Correction):
/ (heaaxr)=oo(())) (oon())))((()da
Tata())da ro(a))de
Zold (o(2))Toll (z)
= Zold - / Zord
Zold LoldBentaaaa [w(x]]
Combining these for the UKL term:
UKL(TelI Totaa = Zola (w(x)) (x)) + Zold - ZoldEennou [u(x)]
Now, substitute into the objective JURKL
JU) = Zola Bez (m(2))((ar) ,3 (Zola Bez (w(()) log w(x))] + Zold Zold Ban (w)))))
EzwaEanRola (2xx) R(x)) (2) log w(x) - 8 (w(x)]
We compute the gradient JURKL The constant term -BZoi vanishes upon differentiation.
VOJURKI((0) = V (Z Bynaa (w(xr) R(x)) (w( ( log w(x) + (w(())
ZotaBEnwiaa (o(w(z)R(r)) - SV VV (w(x) log w(x)) + Vow(x)).
Using the previously derived gradients Vow(z) w(x Vo lo and Vo(w(x) log Vx log no (log w(x) + 1);
VoJURKI. (0) = Vo (Zota Bawiaa [w<x) R(x) Bw(xx) log + Blo()))
ZotalEa
(v(w(x)) R(x)) - (V (og + []w((r)
ZotalBawnara (Vow(z)))((r - Bw(x) Vo log To To (2) + 1) + B(Vow(z))
ZotalE Fawsaaa [(x)) Vo log (x())
(log + V) log (T())
+Bw(x)) Va log (x))
ZotalE
((x) V log m (z)
x
[(x)) 3(log wxx) 1) + 3)
Zo
Vo (z) R(z) - 3 log w(x)
=Zota Eas
w(x) (R()) B log (xx) Vo log TT(
This proves the first part of the theorem.
Now, consider the surrogate loss function:
LURKL (0) = Zold BewHora [-w(x))(xx) + (w(x) (og (og)(()) (ogg)())(())))) w(x))]
We compute its gradient:
V VoLURKI(()) Zola Bawwoord
V [
w(xx)]
Zold Ba
-
w()()) - Vow(s))
Zond Bnwaasa
[(z) log
+ w log
Zoia
log (o (w log u V mo
Zola Bawiis
<e(x)) (-R(x) + log w(x)
V log no (x)
w(x) [(xx) 3 log w(x) V log no (x)
Comparing this with the gradient of the objective function, we confirm that VoLURKL( = -VoJURKL (0). The constant term +1 (corresponding to - -Bold in the objective) that appeared in the derivation in Section 3.2 does not affect the gradient and often omitted from the final loss expression used in practice.
36

---

Published as a conference paper at ICLR 2026
L PROOFS FOR REINFORCE-STVE REGULARIZED POLICY GRADIENTS
This section provides justifications for the REINFORCE-styL surrogate loss functions presented in Section 4 (Theorems F.1 to F.5). These proofs demonstrate how automatic differentiation applied the proposed losses, utilizing the stop-gradient operator SG, yields the correct gradient direction (negative of the objective gradient derived Section 3).
The idea relies the operational definition of the stop-gradient operator SG(-) within automatic differentiation frameworks: Ve VG(((())) = 0, while the forward computation uses the value of f(0). We the notation wxx) Fold
L.1 PROOF OF PROPOSITION F.1 (REINFORCE-STYE POLICY GRADIENT FOR FORWARD
KL)
Proof.
The objective is (0) Ex, [R(x) B (L TO). From Proposition E.1, its gradient
is:
VOJFKI. Eurorood (w(x) + 3) Vo log To
Weightyxr,
The proposed REINFORCE surrogate loss is:
REINFORCE FKL
styee style - [G (w(x) B) log me(x)]
We compute the gradient of this loss as it would be computed by an automatic differentiation system. Assuming the gradient be swapped with the expectation:
V
REINFORCE-StYIC (0) -BEwTo
No (SG (w(x) log (o)))
FKL
Eewsaar
SG (w(x))(x))++ B)) log (x)
=0 by definition of 5G
+ SG
(V log [SG (w<z))((x)) + 3) Ve (x)))
This gradient expression, when used in an optimization algorithm (where SG conceptually re- moved), corresponds to applying updates proportional to:
Aa [ww(x) + B) V log (+())) VeJFKI,
RRFINFORCF.stlle (8) using gradient descent with automatic differentiation effectively
Thus, minimizing LPKL. performs gradient ascent on the original objective JFKL(0).
L.2 PROOF OF PROPOSITION F.3 (REINFORCE-STYLL POLICY GRADIENT FOR UNNORMALIZED FORWARD KL)
Proof. The objective (O) gradient is:
E.y [R(x)] - (TaaaTaa From Proposition 3.2, its
VaJUFKI Exaiiiia Zora (())))(=)-B((()) V log (x(
WeightuFFLL (x,,0)
The proposed REINFORCE-style surrogate loss is:
REINFORCE-tyle (8) - -E [SG (Zola (w(x)R(z) B(w(x) - 1))) log [o(2)) UFKL
37

---

Published as a conference paper at ICLR 2026
Computing the gradient via automatic differentiation:
RREENFORCE (0) Vo VoLUFKL
Roga [V (SG (...) .) log To (x))
(Vo (GGG(Zo ))) log mo(x) SG )) (Vo log mo()))
-BBawRu
[SG (Zold (w(x))(()) ((w(x) 1))) V log [
This gradient corresponds to update direction JUJUKKI (0) when the SG is dropped. Minimiz- ing this loss achieves gradient ascent on JUFKI. (0). If Zola is omitted, the same argument applies to the proportionally scaled objective and loss.
L.3 PROOF PROPOSITION F.4 (REINFORCE-STVV LOss)
Proof. The objective is JAKL (0) = E.,, [R(x)] KL Tola). From Proposition E.3, its gradient is:
VoJRKL(()) = Exensald w(x) R(x) B(log w(z) + 1)) Ve To(z)
WeightRKL (m.00)
The proposed REINFORCE-style surrogate loss is:
CREINFORCE-Syle (0) = -Eg-ros [SG (w(x) (R(x) - 8 log w(x) B)) log (x)]]
RKK
Computing the gradient via automatic differentiation:
RBNNOORRE- (0) Bywzos V (SG (w(x log (T()))
V RKLL
Ezwzold
(oo (G 1)) log no(x) + SG(wE) log Tx()))
--Be-Rala [SG (w(x) (R(x) B log w(x) 3)) V lo
This gradient corresponds to the update direction V JRKL( when the SG is dropped. Minimizing this loss achieves gradient ascent on (O)..(0).
L.4 PROOF OF PROPOSITION F.5 (REINFORCE-STYLE. Loss FOR UNNORMALIZED REVERSE KL)
Proof. The objective (U) E,y [R(x)] UKL(mTl Taa). mTaaa) From Proposition 3.6, its gradient is:
VoJURKL = Exwttou Zolaw(a) R(x) - log wxx) Vo log To(x)
Weight
The proposed REINFORCE-StyL surrogate loss is:
REINFORCE styl (0) = Eeranas [SG (Zoaw(x) (R(x) :3 log w(x))) log TT
LURKT.
Computing the gradient via automatic differentiation:
V. LURKI.
(0) - Egwetss Vo (SG
log To (x))]
Vo (V ml)
SG(Zouu () )x())
E Expatoo
SG (Z)ld (R(z) 3 log w(x))) Vo 1o))
38

---

Published as a conference paper at ICLR 2026
This gradient corresponds to the update direction Va-JURKI (0) when the SG is dropped. Minimiz- ing this loss achieves gradient ascent on JURKL( Zold is omitted, the same argument applies to the proportionally scaled objective and loss.
39


Reviews:
[{"review_id": "tSC2a9S7HZ", "reviewer": "Reviewer_LL8H", "summary": "The author studied the design of KL regularization in reinforcement learning. Specifically, the analysis shows that the commonly used k3 estimator is equivalent to the use of unnormalized KL. The author also shows that GRPO's original KL divergence implementation lacks an importance sampling term due to its off-policy attributes.", "questions": "* In Figure 2, is the difference between GRPO and RPG-URKL in whether having the importance weight in the kl term?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* Nice theoretical analysis between k3 estimator and UKL\n* The tackled KL divergence issue is indeed a pitfall of GRPO", "weaknesses": "The experiments in the paper are quite poorly conducted:\n* The model is tested only on math tasks, and even then, only two AIME benchmarks are used.\n* Figure 2 is presented without any analysis. It seems that in Figure 2(c), DAPO is not converging — what would it look like if we extrapolated the training curve?\n* (c) RL training often exhibits high variance; instead of showing a single curve, the authors should include a curve with a standard deviation region.\n* (d) In Figure 2(d), why is only GRPO’s actor entropy decreasing? Out of the five curves, three increase and plateau, while the REINFORCEMENT curve continues to rise — why?\n* (e) What is “mean@32”? Do you mean “pass@32”?\n\nAlso, there seem to be quite a few unanswered questions to be verified by the experiments:\n* FKL and RKL seem to be comparable in performance. Is there any evidence that such two constraints result in different properties in the post-trained model?\n* Some papers, like DAPO and Dr.GRPO, advise dropping KL divergence for RL tuning. Is there some analysis on how different designs and strengths of KL regularization change the training dynamic and the tuned model's properties?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gtVDuK3hyU", "reviewer": "Reviewer_w5SC", "summary": "This paper presents RPG, a unified theoretical and practical framework for KL-regularized policy gradient algorithms. The authors systematically analyze different KL regularization variants and corresponding estimators, clarifying their gradient properties under off-policy sampling. Their derivation shows how to obtain the exact gradients of intended KL-regularized objectives. It identifies a weighting mismatch in GRPO, and introduces a truncated-importance estimator which stabilizes training. Experiments show RPG and RPG-REINFORCE variants outperform other baselines on mathematical reasoning benchmarks and have better stability.", "questions": "1. Why the entropy grows up with training in DAPO and RPG while decreases in GRPO? I assume both DAPO and GRPO should both decrease but to different degrees. This is contradicted to the DAPO paper. \n2. More recent works have opted to remove the KL term from the RL objective and achieve good performance. In comparison, does a compromise of periodically updating the reference model offer significant and irreplaceable advantages? \n3. Do the empirical results show a significant performance difference between the corrected Normalized KL and Unnormalized KL?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper provides a unified framework connecting various KL regularization forms. The derivations offer clarity and formal grounding.\n2. The authors propose RPG-Style Clip, which can effectively balance stability and bias under off-policy training and improve scalability for LLMs\n3. The experiments demonstrate notable improvements in both accuracy and stability.", "weaknesses": "1. Some representative baselines are missing, e.g., RLOO, REINFORCE++, GPG. \n2. The experiments lack ablation studies isolating the contributions of RPG-Style Clip, reference-update frequency, and clipping thresholds.\n3. The RPG-Style Clip introduces a bias-variance trade-off but it is not explored quantitatively.\n4. Section 3 and 4 are heavy and hard to follow.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iIeji5lS0E", "reviewer": "Reviewer_jhGz", "summary": "The paper introduces RPG (Regularized Policy Gradient), a unified framework for KL-regularized policy-gradient training of LLMs under off-policy sampling. It systematizes design choices across: (i) KL direction (forward vs reverse), (ii) normalized vs unnormalized KL (UKL), and (iii) estimator/implementation (fully-differentiable vs REINFORCE-style with stop-gradient). Key theoretical results (e.g., Propositions for UFKL/URKL gradients and losses) show how to obtain exact gradients of the intended KL-regularized objective when training off-policy with importance weights, and establish gradient-equivalence between differentiable and REINFORCE-style surrogates. Practically, the paper (a) clarifies that the widely used k3 estimator corresponds exactly to the unnormalized KL, (b) identifies an importance-weighting mismatch in GRPO’s KL term under off-policy sampling and provides a corrected estimator, and (c) introduces RPG-Style Clip, a dual-clip truncated-importance step for stability. On AIME24/25 with Qwen3-4B, RPG-REINFORCE + RPG-Style Clip outperforms DAPO by up to +6pp absolute accuracy. The method also employs periodic reference-policy updates to maintain a practical KL trust region.", "questions": "Please refer to the weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- It provides a compact derivation that covers Forward/Reverse × Normalized/Unnormalized KL, with explicit off-policy corrections and both differentiable and REINFORCE-style surrogates.\n- It pinpoints the missing importance weight in GRPO’s KL when sampling off-policy, which could be an issue that many people overlook.\n- The paper introduces RPG-Style Clip (dual-clip/truncation) and iterative reference updates, which empirically stabilize off-policy PG for LLM reasoning while requiring only a single active model on GPU (log-probs from $\\pi_{\\textrm{old}}$ cached).\n- Empirically, it has strong performance with +6pp over DAPO in the best cases on AIME24/25, with stable entropy and controlled response length.", "weaknesses": "- All core results are on Qwen3-4B and math reasoning. Claims about “stable and scalable RL” would be stronger with larger backbones (e.g., 7B–14B) and diverse tasks or datasets.\n- While GRPO/DAPO are relevant, head-to-head comparisons with varying KL directions (FKL/RKL vs UFKL/URKL) would solidify the empirical story. Current ablations on clip hyper-parameters and reference-update schedules are limited in the main text.\n- RPG-Style Clip introduces controlled bias via truncation. The paper acknowledges this and suggests schedules, but there’s no systematic bias study (e.g., varying $\\epsilon_1$, $\\epsilon_2$) to quantify performance/variance trade-offs.\n\nOverall, the theoretical foundations are solid and the proposed algorithm is well-motivated. The strengths outweigh the weaknesses, and I am leaning toward acceptance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)