Group-Relative REINFORCE Is Secretly an Off-Policy Algorithm: Demystifying Some Myths About GRPO and Its Friends
Authors: Chaorui Yao
Yanxi Chen
Yuchang Sun
Yushuo Chen
Wenhao Zhang
Xuchen Pan
Yaliang Li
Bolin Ding
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Models
Reinforcement Learning
LLM Post-Training
Off-Policy RL
GRPO
Primary area: foundation or frontier models, including LLMs
TL;DR: We present a native off-policy interpretation for group-relative REINFORCE, and its broad implications.

Abstract: Off-policy reinforcement learning (RL) for large language models (LLMs) is attracting growing interest, driven by practical constraints in real-world applications, the complexity of LLM-RL infrastructure, and the need for further innovations of RL methodologies. While classic REINFORCE and its modern variants like Group Relative Policy Optimization (GRPO) are typically regarded as on-policy algorithms with limited tolerance of off-policyness, we present in this work a first-principles derivation for group-relative REINFORCE — a REINFORCE variant that uses the within-group mean reward as the baseline for advantage calculation — without assuming a specific training data distribution, showing that it admits a native off-policy interpretation. This perspective yields two general principles for adapting REINFORCE to truly off-policy settings: regularizing policy updates, and actively shaping the data distribution. Our analysis demystifies some myths about the roles of importance sampling and clipping in GRPO, unifies and reinterprets two recent algorithms — Online Policy Mirror Descent and Asymmetric REINFORCE — as regularized forms of the REINFORCE loss, and offers theoretical justification for seemingly heuristic data-weighting strategies. Our findings lead to actionable insights that are validated with extensive empirical studies, and open up new opportunities for principled algorithm design in off-policy RL for LLMs. Source code for this work is available at https://github.com/agentscope-ai/Trinity-RFT/tree/main/examples/rec_gsm8k.

Published as a conference paper at ICLR 2026
GROUP-RELATIVE REINFORCE IS SECRETLY AN OFF-POLICY ALGORITHM: DEMYSTIFYING SOME MYTHS ABOUT GRPO AND ITS FRIENDS
Chaorui Yao University of California, Los Angeles Chaorui@ucla edu
Yanxi Chen
Alibaba Group
chenyanxi. cyx@ com
Yuchang Sun, Yushuo Chen, Wenhao Zhang
Alibaba Group (sunyuchang.sy) chenyushuo.cys 2w0.3ysswhh4448888 cheerysshho...o.2222.333ff11l.l. com
Xuchen Pan, Yaliang Li, Bolin Ding
Alibaba Group
[panxuchen.pxe, yaliang. 1i,, wow.b. atw. com
ABSTRACT
Off-policy reinforcement learning (RL) for large language models (LLMs) is attracting growing interest, driven by practical constraints in real-world applications, the complexity of LLM-RL infrastructure. and the need for further innovations RL methodologies. While classic REINFORCE and its modern variants like Group Relative Policy Optimization (GRPO) typically regarded as on-policy algorithms with limited tolerance of off-policyness, we present in this work first-principles derivation for group-relative REINFORCE REINFORCE variant that uses the within-group mean reward as the baseline for advantage calculation
without assuming specific training data distribution, showing that it admits a native off-policy interpretation. This perspective yields two general principles for adapting REINFORCE truly off-policy settings: regularizing policy updates, and actively shaping the data distribution. Our analysis demystifies some myths about the roles importance sampling and clipping in GRPO, unifies and reinterprets two recent algorithms - Online Policy Mirror Descent and Asymmetric REINFORCE - as regularized forms of the REINFORCE loss, and offers theoretical justification for seemingly heuristic data- weighting strategics. Our findings lead to actionable insights that are validated with extensive empirical studies, and open up opportunities for principled algorithm design in off-cy RL for LLMs. Source code for this work is available httpp:://gitiiii http://ggittuuu.cmaggeeeeete com hatp///ggettccppppe httpss//ggtthhbb.cmmm/eegessssee
INTRODUCTION
The past few years have witnessed rapid progress in reinforcement learning (RL) for large language models (LLMs). This began with reinforcement learning from human feedback (RLHF) (Bai
et al. 2022 Ouyang al.
2022) that aligns pre-trained LLMs with human preferences, followed by reasoning-oriented RL that enables L.LMs to produce long chains of thought (OpenAl 2024
DeepSeek-Al 2025 Kimi-Team 2025b Zhang al.) 20256). More recently, agentic RI. (Kimi-
Team 2025a Gao et al. 2025 Zhang et al. (225aa) aims to train LLMs for agentic capabilities such tool use, long-horizon planning, and multi-step task execution in dynamic environments.
"Equal contribution. Part of the work done while Chaorui Yao intern Alibaba Group.

---

Published as a conference paper at ICLR 2026
Alongside these developments, off- off-policy RL has been attracting growing interest. In the "era of experience" (Silver Sutton 2025). LLM-pooered agents need to be continually updated through interaction with the environment. Practical constraints in real-world deployment and the complexity of LLM-RIL infrastructure often render on-policy training impractical (Noukhovitch et al. 2025): rollout generation and model training can proceed mismatched speeds, data might be collected from different policies, reward feedback might be irregular or delayed, and the environment may be too costly or unstable to query for fresh trajectories. Moreover, in pursuit of higher sample efficiency and model performance, it is desirable to go beyond the standard paradigm of independent rollout sampling, e.g., via replaying past experiences (Schaul et al., 2016 Rolnick et al.) 2019 An
et al. 2025), synthesizing higher-qualige experiences based on auxiliary information (Da et al. 2025
Liang et al. 2025 Guo al. 2025), or incorporating expert demonstrations into online RL Yaan)
et al. 2025 Zhhag et al. 2025c all of which incur off aff-policyness.
However, the prominent algorithms in LLM-RL- Proximal Policy Optimization (PPO) (Schulman
et al. 2017) and Group Relative Policy Optimization (GRPO) (Shao et al. (2024) are essentially on-policy methods: as modern variants of REINFORCE (Williams) 1992). their fundamental rationale is to produce unbiased estimates of the policy gradient, which requires fresh data sampled from the current policy. PPO and GRPO can handle a limited degree of off-policyness via importance sampling, but require that the current policy remains sufficiently close to the behavior policy. Truly off-polic LLM-RL. often demands ad-hoc analysis and algorithm design; worse still, as existing RL infrastructure (Sheng al.) [024] Hu et al. [024 von Werra et al. 2020 WWan al.) 2025b) Pan et al., 2025 Fu et al. 2025a) is typically optimized for REINFORC algorithms, their support for specialized off-policy RL algorithms could be limited. All these have motivated our investigation into principled and infrastructure-frinn friendly algorithm design for off-policy RL.
Core finding: native off-policy interpretation for group-relative REINFORCE. Consider a one-step RL setting and a group-relative variant of REINFORCE that, like in GRPO, assumes access to multiple responses (yi...., !/K for the same prompt x and use the group mean reward r as the baseline in advantage calculation. Each response is a sequence of tokens yi (ut .f....) and receives a response-level reward r; r(x, y.). Let ( denote an autoregressive policy parameterized by O. The update rule for each iteration of group-relative REINFORCE is O' + 1g.. where 17 is the learning rate, and g is the sum of updates from multiple prompts and their corresponding responses. For a specific prompt x, the update would be
g(0; x, 20 (yt rihsisk)
Σ ISiCK
(r, T V) log (o 1x)
(response-wise) (la)
K
1
Vo E
logg x. v.5
(token-wise) (1b)
K
ISiSK IStS|a:
Here, the response-wise and token-wise formulas are linked by the elementary decomposition
log ( (x) = E, log To( x. v,²),, where v,* denotes the first tokens of y;-
A major finding of this work is that group-relative REINFORCE admits a native -oolccy interpretation. We establish this in Section [ via a novel, first-principles derivation that makes no explicit assumption about the sampling distribution of the responses {w:}. in contrast to the standard policy gradient theory. Our derivation provides a new perspective for understanding how REINFORCE makes its way towards the optimal policy by constructing a series of surrogate objectives and taking gradient steps for the corresponding surrogate losses. Such analysis can be extended to multi-step RL settings as well, with details deferred to Appendix A
Implications: principles and concrete methods for augmenting REINFORCE. proposed off-policy interpretation does not imply that vanilla REINFORCE should converge to
While the
'For notational simplicity and consistency, we use the same normalization factor 1/K K for both response- wise and token-wise formulas in Eq. (1a) and (Ib). For practical implementation, the gradient is calculated with samples from mini-batch, and typically normalized by the total number of response tokens. This mismatch does not affect our theoretical studies in this work. Interestingly, our analysis of REINFORCE in this work provides certain justifications for calculating the token-meat loss within a mini-batch, instead of first taking the token- mean loss within each sequence and then taking the average across sequences (Shao et al. 2024): our perspective is complementary to the rationales explained in prior works like DAPO (Yu et al. 2025 although deeper understanding of this aspect is beyond our current focus.
2

---

Published as a conference paper at ICLR 2026
the optimal policy when given arbitrary training data (which is too good to be true), our analysis in Section identifies two general principles for augmenting REINFORCE off-policy settings: (1) regularize the policy update step to stabilize learning, and (2) actively shape the training data distribution to steer the policy update direction. As we will see in Section 4 this unified framework demystifies common myths about the rationales behind many recent RI. algorithms: (I) It reveals that in GRPO, clipping (as a form of regularization) plays a much more essential
role than importance sampling, and is often viable to enlarge the clipping range far beyond
conventional choices for accelerated convergence without sacrificing stability. (2) Two recent
algorithms Kimi's Online Policy Mirror Descent (OPMD) (Kimi-Team, 2025b) and Meta's
Asymmetric REINFORCE (AsymRE) (Arnal et al. 2025) can be reinterpreted as adding
regularization loss to the standard REINFORCE loss, which differs substantially from the rationales explained in their original papers. (3) Our framework justifies heuristic data-weighting strategies like discarding certain low-reward samples or up-weighting high-reward ones, even though they violate assumptions in policy gradient theory and often require ad-hoc analysis in prior works.
Extensive empirical studies Section 1 and Appendix validate these insights and demonstrate the efficacy and/or limitations of various algorithms under investigation. By revealing the off- policy nature of group-relative REINFORCE, our work opens up new opportunities for principled, fnfrastructure--riinndyy algorithm design in off-policy LLM-RL with solid theoretical foundation.
2
TWO INTERPRETATIONS FOR REINFORCE
Consider the standard reward-maximizan objective in reinforcement learning:
max J(0)= EE J(0:)),
where
J(0; By r(x,
where D is distribution over the prompts x.
We first recall the standard on-policy interpretation of REINFORCE Section 2.1 and then present our proposed off-policy interpretation in Section 2.2
2.1 RECAP: ON-POLICY INTERPRETATION VIA POLICY GRADIENT THEORY
In the classical on-policy view, REINFORCE updates policy parameters O using samples that drawn directly from Te. The policy gradient theorem (Sutton et al. 1998) tells us that
V ((:()))) Vo (yy=na(-)z) r(x, y) (- (r(
(r(z, bx)) Vo log mollory)))
where 6(x) is baseline for reducing variance when [o is estimated with finite samples. If samples are drawn from a different behavior policy Tb instead, the gradient can be rewritten
To (y xx) To
Vo = Ey (r(2: y) b(x))
Ve log To [x)
x)
While the raw estimate, it may be unstable when and diverge. Modern variants of REINFORCE address this by modifying the probability ratios (e.g., via clipping normalization), which achieves better bias-variance trade-off in the policy gradient estimate and leads to a stable learning process.
weight To
facilitates unbiased policy gradient
In the LLM context, we have Ve log Te (y x) = E, Ve log (8 [x, *t), but the response-wist probability ratio To (yly=///bb(H (b)lz)) can blow up or shrink exponentially with the sequence length. Practical implementations typically adopt token- wise probability ratio instead:
To x, <t)
Vo log To(y' |x, | (5r)))
7(0,z) x) Egnnn (-1x)) (r(x,)) - b(x))
Tb (y²
Ir, 5(
1S15
Although this becomes biased approximation of V Ve.] classical RL theory still offers policy improvement guarantees if no is sufficiently close to To (Kkkade & Langford 2002) Fragkiadaki 2018 Schulman al. 2015 2017 Achiam al. 2017

---

Published as a conference paper at ICLR 2026
2.2 A NEW OFF-POLICY INTERPRETATION FOR GROUP-RELATIVE REINFORCE
We now provide an alternative off-poliicy interpretation for group-relative REINFORCE. Let us think of policy optimization as iterative process 0,, 0z,.... and focus on the 1-th iteration that updates the policy model parameters from 0, to 0s+1- Our derivation consists of three steps: (1) define KL-regularized surrogate objective, and show that its optimal solution must satisfy certain consistency conditions; (2) define a surrogate loss (with finite samples) that enforces such consistency conditions; and (3) take one gradient step of the surrogate loss, which turns out to be equivalently the group-relative REINFORCE method.
Step 1: surrogate objective and consistency condition. Consider the following KL-regularized surrogate objective that incentivizes the policy to make stable improvement over .T.
max J(0: mo,) E (-))mo(1=)) v)] ( || no, (11=))
(3)
where T is regularization coefficient. is well-known fact that the optimal policy for this surrogate objective satisfies the following (Ziebart et al. 2008) (Nachum et al. 2017 Korbak et al.,
2022 Rafailov et al., 2023 Richemond et al.) 2024 Kimi-Team response y,
(025b):: for any prompt and
(y)ar) To, (l
where ((x, To,) = f no, (w' (c)e dy'-
(4)
7(r, no,)
Note that Eq. 44 is equivalent to the following: for any pair of responses y/1 and y/.
T (TV1 (z) To. To, (v1
r(x v1) r(z, 1/2)
π (v2 (z) To, (12/z)
exp
Taking logarithm of both sides, we have this pairwise consistency condition:
r1 - T (log x (31 (z) - log ne, (x)) = r2 T
T (log ((22x)) - log no, (V2lz))
(5)
Step 2: surrogate loss with finite samples. Given a prompt and K responses Vı. UK, define the following mean-sone meean-quarrd surrogate loss that enforces the consistency condition, as done in prior works (Gao al. 2024 Flet-Berliac et al. 2024):
(a: a²)² (1 (1+T) T)2 where a; := : log No (y)lz))
T(0:2, no.)
Σ ISKjSSK
log no, (uallr)) (6)
K2
Here, we normalize a; - 0j by 1 + T to account for the loss scale. In theory, this surrogate loss is defined by infinite samples with sufficient coverage of the action space (Song et al. 2024), then its unique minimizer is the same as the optimal policy for the surrogate objective in Eq. (3).
Step 3: one gradient step of the surrogate loss.
Let conduct further analysis for (a, - a² The trick here is that, if we take only one gradient step of this loss at 0 = O., then the values of log To ( (z)
log ne. (x)(x) and log To(y,)<)) (z)
log Te, (uylæ) are simply zero. As a result,
Vola - -,,) - (r: - Tú)) Vo log To (y:
Vo log To ( (u))1)))).))
(a, - a
Ve Σ
= =Σ (1 (1+ (re - -r Vo log logg Vo log To To(U;(x) (z))
(1+ (}+ T)2
1r
ISiCjsK
icj
-27
(ri-r,) r3) Vo log + rr) Vo
(1 T)²
-27
Σ
Σ (r Vo log To
(1+ T)²
15ick 15jSK
-2TK
Σ
(nr - - log where F==
Σ Tj.
71+
K
1gigk
ISISK
4

---

Published as a conference paper at ICLR 2026
Putting these back to the surrogate loss defined in Eq. (6) end up with this policy update step:
g(0; x, (asssss
2r
Σ (r. - F) V log (e lz).
(7)
(1+ K
ISiSK
That's it! We have just derived the group-relative REINFORCE method, but without any on on-policy assumption about the distribution of training data {x, {ui ri risisk J. The regularization coefficient
т 0 controls the update step size; larger effectively corresponds to a smaller learning rate.
Summary and remarks. Figure [ visualizes the proposed interpretation of what REINFORCE actually doing. The curve going through 0: + 0t+1 - 0,+1 + O' stands for the ideal optimization trajectory from 0, to the optimal policy model 0", if the algorithm solves each intermediate surrogate objective TT,) / surrogate loss L(0; TT.) exactly at each iteration t. In comparison, REINFORCE is effectively taking a single gradient step of the surrogate loss and immediately moving to the next iteration 0,+1 with new surrogate objective.
Two remarks are in place. (1) Our derivation of group-relative REINFORCE can be generalized to multi-step RL settings, by replacing response y in the previous analysis with full trajectory consisting of multiple turns of agent-environment interaction. For example, regarding the surrogate objective in Eq. (3) we need to replace the response-level reward and KI. divergence with their trajectory-level counterparts. Interested readers might refer to Appendix [ for the full analysis. (2) The above analysis suggests that we might interpret group-relative REINFORCE from pointwise or pairwise perspective. While the policy update in Eq. (7) is stated in a pointwise manner, we have also seen that, at each iteration, REINFORCE is implicitly enforeing the pairwise consistency condition in Eq. (5) among multiple responses. This allows us the flexibility to choose whichever perspective that offers more intuition for our analysis later this work.
3 PITFALLS AND AUGMENTATIONS
Ocs ang arg.axx / (0:1a.) ang min D (R:ma)
Although we have provided a native off- off-policy interpretation for REINFORCE, it certainly does not guarantee convergence to the optimal policy when given arbitrary training data. This section identifies pitfalls that could undermine vanilla REINFORCE, which motivate two principles for augmentations in off-policy settings.
B.n -.70-- pad;
0.
g"
Figure 1: A visualization of off-policy interpretation for group-relative REINFORCE. Here L(0; me.) = E.- L(O;,, Te)),, where Đ is the sampling distribution for prompts and [(; x, xe.) is the loss defined in Eq. (6)
Pitfalls of vanilla REINFORCE. In Figure II might expect that ideally, (1) 0.+1 O, aligns with the direction of 0;; and (2) 0 θ,
aligns with the direction of 0.+1 0,, One pitfall, however, is that even if both conditions hold, they do not necessarily imply that 0.+1 0, should align well with 0* -o. That is, (0.41 02, --
O,)
0 and (0t+1 0, 0 0,) > do not imply (0 9 0 0,) > 0. Moreover, it is possible that 0.+1 0: might not align well with 0.+1 0.. Recall from Eq. 7 that, from 0, to 0.+1, we take one gradient step for surrogate loss that enforces the pairwise consistency condition among finite number of samples. Given the enormous action space of an LLM, some implicit assumptions about the training data (e.g., balancedness and coverage) would be needed ensure that the gradient aligns well with the direction towards the optimum of the surrogate objective, namely 0 O..
In fact, without a mechanism that ensures boundedness of policy update under a sub-optimal data distribution, vanilla REINFORCE could eventually converge to a sub-optimal policy. Let us show this with minimal example in a didactic 3-arm bandit setting. Suppose that there are three actions {a, hesss with rewards (r(a,)}- Consider K training samples {ylhsis, where y/s E {a, hisss is sampled from some behavior policy Tb. Denote by μr = Σis Tb To(a,)((a) the expected reward under Tb, and F = r,(ys)/KK the average reward of training samples. We consider the softmax parameterization, i.e., Ta(ag) / e for a policy parameterized by 0 E R3 A standard fact is that Vo log Ta(a)) eg no, where e, E R33 is one-hot vector with value
5

---

Published as a conference paper at ICLR 2026
entry j. Now we examine the policy update direction of REINFORCE, as K 00:
Σ
(r(wi)
- F) mogi)
To(a))(r(a)) - M.) Vog
K
1SIS
15153
Σ Tb(a))(r(a,) - pr)(ee TT)
Σ Tb(a))(r(a,) p,)e 1953
15i53
For example, if = (r( ( 13/33 0., 1] and db = [0.3, 0.6, 0.1], then basic calculation says
H4, 0.58, H4, = 0.58, 0.22, 0.42], and finally D2 To (r(0))(((a)) μr) mb(a))(r(aa)
[4r) = 93. which implies that the policy will converge to the sub-optimal action d2.
Two principles for augmenting REINFORCE. suggest two general principles for augmenting REINFORCE in off- off-pooliy scenarios:
The identified pitfalls of vanilla REINFORCE
One is to regularize the policy update step, ensuring that the optimization trajectory remains bounded and reasonably stable when given training data from a sub-optimal distribution;
The other is to steer the policy update direction, by actively weighting the training samples rather than naively using them as is.
These principles not mutually exclusive, and might be integrated within single algorithm. We will see in the next section that many RL algorithms can be viewed as instantiations of them.
4 RETHINKING THE RATIONALES BEHIND RECENT RL ALGORITTMM
This section revisits various RL algorithms through unified lens the native off-po interpretation of group-relative REINFORCE and its augmentations - and demystifies some common myths about their working mechanisms. Our main findings are summarized as follows:
ID
Finding
Analysis & Experiments
F1 GRPO's effectiveness in off-policy settings stems from clipping regularization rather than importance sampling. wider clipping range than usual often accelerates training without harming stability.
Section 4.1
Figures
F2 Kimi's OPMD and Meta's AsymRE can be interpreted as REINFORCE loss regularization loss, complementary the rationales their original papers.
Section 4.2
Figure 10
F3 Data oriented heuristics - such as dropping excess negatives or up peightgt high-reward rollouts - fit naturally into our off- policy view and show strong empirical performance.
Section 4.3 Figures
Experimental setup. We conduct experiments with the Trinity-RFT framework (Pan al. 2025
and control off-policyness with the sync_interval (frequency of model synchronization) and sync.offset (lag between rollout generation and training) parameters. Larger values of these parameters improve efficiency (via pipeline parallelism) at the cost of off-policyness; in addition, > simulates delayed environmental feedback in practical scenarios. We also include a stress-test setting that only allows access to offline data generated by the initial policy model. Our experiments cover math reasoning tasks like GSM8k (Cobbe et al.) (021). MATH Hendrycks et al., 2021). Guru-Math (Cheng et al. 2025). and tool-use tasks like ToolACE (Liu al. 2025a). LLMS under consideration include Qwen2.5-1. I.B-Itrut,t, Dwwen.55-BB-Instmeee
[Qwen-Teee 2025a). Llama-3.1-80 1.----.-1-BB-Ittetc Llama-3.22 InIstruct (Dubey al. 2024). and
Sven3-30B-A333 (Qwen-Teaae 20256). Further details can be found in Appendix
4.1 DEMYSTIPYING MYTHS ABOUT GRPO
Recall that in GRPO, the advantage for each response y; is defined as A; = (r; r)/o,, where r and a, denote the within-group mean and standard deviation of the rewards {rihisisk respectively.

---

Published as a conference paper at ICLR 2026
inferval
offsct
0:30
025
0.00-
Tuaniig Rowod
130
Training Steps
Triiinng
Trinnnng
Uniinigg
BEC-OneclideNNi
GRPG
REC-OscHid-18 (8.20
Figure 2: Empirical results for REC algorithms GSM8k with Qwen2.5-1.5B --5-rrrututtt Training reward curves are smoothed with a running-average window of size 3. Numbers in the legend denote clipping parameters élows Ehigh- The "mixed" setting adopts sync.interval 16 and syncLoffset
We consider the practical implementation of GRPO with token-wise importance-sampling (IS) weighting and clipping, whose loss functiin for specific prompt and responses {vi
7.
Σ E
Σ min To(uf (old
I. M/,
A,, clip
( x. ve (ol (y xr, sr =t)
1 Clow 1 Chigh A
K ISISK
xr y, (
where Told denotes the older policy version that generated this group of rollout data. The gradient of this loss can be written as (Schulman et al. 2017)
g g0,a, ri)sssk))))
1 K
Σ Σ
(o A
To( v M, Told ! I, v, ()
ISISK 151siul
where Mt denotes one-side clipping mask:
To xz v,
1+ E + +
To( y. v.
M A: A 0,
2 Elow
(8)
Wold (1) x. DE
Told (yf x. yi
Ablation study with the REC series. To isolate the roles of importance sampling and clipping. (REC) algorithms. Due to space limitation, we
we consider series of CEEINFOREE-wih-Cliie defer our studies of more clipping mechanisms Appendix B.3 and focus on REC with one-side clipping in this section. More specifically, REC-ONESIDE-I removes advantage normalization in GRPO (to reduce variability), and REC- DECC-ONSSIDE-OOSS further removes IS weighting:
REC-ONESIDE-IS: g = K
1
Σ
Σ
Volog x. v. (r, -F) To (v) xr y5
M
Told (v x, y,
1SISK 15t5|ul
1
REC-ONESIDE-NOIS: g
Σ
Σ
Vo log I. | (r: M,
K
Experiments. We conduct experiments to validate Finding F1 regarding the roles clipping (with small or large clipping range) and importance sampling in GRPO. Figure 2 presents GSM8k results with Qwwen2.511.5B-Insttee
in various off-policy settings. REC-ONESSIDE-IS/NOII and GRPO (with the same élow Chigh 0.2) have nearly identical performance, indicating that importance sampling is non-essential, whereas the collapse of REINFORCE highlights the critical role of clipping. Radically enlarging (Elow: <high) (0.6, 2.0) accelerates REC-ONESIDE-NON
In our experiments with GRPO, we neglect KL regularization with respect to an extra reference model, or
entropy regularization that encourages output diversity. Recent works (Yu al. 2025 Liu et al. 2025b) have shown that these practical techniques are aften unnecessary.

---

Published as a conference paper at ICLR 2026
syne interval
syne
Figure 3: Empirical results for REC on ToolACE with Llama- 3.2-3B-Instruct. Training reward curves are smoothed with a running-average window of size 3. Details about REC-TWOSIDE and RIN-RING are provided in Appendix B.3
Trainin Reward
Clopping Fraction
10"
10"
10
0
100
200
300
400
100
200
300
400
Training Steps
Training Steps
REC-OOnSaaeeNat5 0.25)
REC-Cn-Side-NelS (0.6, 2.0)
REC-OmsSide-8 (0.2, 0.25)
RECC Tws.Stee-SeNNS (0.2, 0.25)
REC Ring Not8 (9.2, 0.25) (06,
without compromising stability all considered settings (except "offline"). Similar patterns also and other results in Appendix B As for the stress-test ("offline") setting, Figure 2 reveals an intrinsic trade-off between learning speed and stability, motivating future work toward better algorithms that achieve both. We hypothesize that sequence-leve importance sampling in GSPO (Zheng et al.J (025) could be non-essential as well; interested readers may refer to preliminary results in Appendix B.7 that support this prediction.
appear in Figure 3 (ToolAce with LJama-3.2-3B-Insttee
4.2 UNDERSTANDING KIMI'S OPMD AND META'S ASYMRE
Besides clipping, another natural method is to add regularization loss R(-) to vanilla REINFORCE:
T. (vi, ralssis))
Σ - (r) log Toly x)) B R(0; (u)rsk) KE(K)
K
and take g = Vot. We show below that Kimi's OPMD and Meta's AsymRE are indeed special cases of this unified formula, with empirical validation of their efficacy deferred Appendix B.5
Kimi's OPMD, Kimi-Team (2025)) derives an OPMD variant by taking logarithm of both sides of Eq. which leads a consistency condition and further motivates the following surrogate loss:
i- Σ 1SISK
ri - log (1x, no.) - - log (o (x) - log To, (ys (1))z))
K
With K responses generated by Rold TA. the term log Z (zx, ne,) can be approximated by finite- sample estimate 7 log(E, e//7 /K) (Brantley al. 2025), which can be further approximated by mean reward = Σ. r r is large. With these approximations, the gradient of L becomes equivalent to that of the following loss (which is the final version of Kimi's OPMD):
Î= Σ (n-r) log (o x)+ ISKS
B 2K
Σ ISICK
(oogmo mo
| =o old (x) 2
where , =
In comparison, our analysis in Sections 2 and 3 suggests that this is in itself a principled loss function for off off-policy RL, adding mean-squared regularization loss to the vanilla REINFORCE loss.
Meta's AsymRE. AsymRE (Arnal et al. 2025) modifies REINFORCE by tuning down the baseline (from F prioritizing learning from positive samples and justified by multi-arm bandit analysis in the original paper. We offer an alternative interpretation for AsymRE by rewriting its loss function:
3) in advantage calculation, which was motivated by the intuition of
-= --
1 K Σ
r-- ( log Th (us x) = K Σ (ra log To(y: (2)))
Σ log (v |z).
K
Note that the first term on the right-hand side is the REINFORCE loss, and the second term serves as regularization, enforcing imitation of responses from older version of the policy model. For the latter, we may also add a term that is independent of θ to it and take the limit + oc:
Σ 1STCK
(o
(x) +
1 K
Σ 1
log Tola xx)
1
Σ 1SiSK
Told y x)) log
K
K
no To x)
8

---

Published as a conference paper at ICLR 2026
Told (y) σ))
aa (m)
log
DRL (=x)
To ( x))
which turns out to be a finite-sample approximation of KL regularization.
4.3 UNDERSTANDINO DATA- WEIGHTTIG METHODS
We now shift our attention to the second principle for augmenting REINFORCE, i.e., actively shaping the training data distribution.
Pairwise weighting. Recall from Section that we define the surrogate loss in Eq. (6) as an unweighted sum of pairwise mean-squared losses. However, if we have certain knowledge about which pairs are more informative for RL training, we may assign higher weights to them. This motivates generalizing Eicy(a: ay)² Σiey Wiy(a; a,)², where (way} are non-negative weights. Assuming that Wi, W3, and following the steps in Section 2
we end up with
g0; Talisiss)))
1 K
Σ
Σ whs)
Vo log (16 at)
1gis
ISiSK
In the special case where wij = W;Wy,, this becomes
wi(r: Fw) Vo log (o(s) where Tw == Eg Wors
Wij
K
Eg Wj
(9)
ISiSK
Based on this, we investigate two REINFORCE withhEEE-witheeee witt-datt (RED) methods.
RED-DROP: sample dropping. The idea is to use a filtered subset S C [K] responses for training (Shang et al. 2025p. For example, the Kimi-Researcher blog (Kimi-Team 2025a) proposes to "discard some negative samples strategically". as negative gradients increase the risk of entropy collapse. This is indeed a special case of Eq. (9. by setting we = / S for E and otherwise:
1
g 0: x, (y = S
Σ (n+ - Vo) log (y6 x), where is iES
rs.
(10)
S
iES
While this is no longer unbiased estimate of policy gradient if all responses sampled from the current policy, is still well justified by our off-policy interpretation REINFORCE.
RED- WEIGHT pointwise loss weighting. Another approach for prioritizing high-reward responses is to directly up-weight their gradient terms Eq. (Ia). To better understand the working mechanism of this seemingly heuristic method, we rewrite its policy update:
Σ 1SiSK
- (oll:aloollssellllla = Σ wr - Twr + V Te
ISISK
wi(r: - Tw) Vo log (T ISiSK
Σ
log
ISEC
This is the pairwise-weighted REINFORCE gradient in Eq. 9 plus a regularization term (weighted by Tw- 1..--3 >>> that resembles the one in AsymRE but prioritizes imitating higher-reward responses, echoing the finding from offline RL literature (Hong et al. 2023a by that regularizing against high- reward trajectories can be more effective than conservatively imitating all trajectories in the dataset.
Experiments. Figure f presents GSM8k results with Qwwen.55-1.5-IInstrue which confirm the efficacy of RED-DROP and DED-WEIGHIT (details in Appendix [B.6) in on/off-policy settings, comparable to REC-ONESIDE-NE NoIS with enlarged (4ow- <high). Figure reports larger-scale experiments Guru-Math with Qwen2.5-7B-Inm Insttruct where RED- WEIGHT achieves higher rewards than GRPO, with similar KL distance to the initial policy. Figure [I] in the appendix further validates the efficacy of WFIGHTIGHT on MATH with LJama-3.1-885 1-8-a...--1---ttttcce

---

Published as a conference paper at ICLR 2026
policcy
interval
boolerion Aearrsy
Figure 4: Empirical performance of RED algorithms on GSM8K with Qwen2.5-1.5B-Instruuc, both on-policy and off-policy settings. Training reward curves smoothed with running- size
1.01
average window Implementation details about RED-WEIGHT and RED-DROP are provided Appendix B.6
193
1.2
160
Traising Stepi
Training Stops
REINFORCE
Weiahs
RBC-CondideeNol8 (.05
RED-Drup
interval = 20
interval = =0
Training Reward
0.25
KL Diverpenee
Figure 5: Empirical results on Guru-Math with Qwen2.5- 7B-Instruct. Training reward curves are smoothed with a running-average window of size 3.
to
0.20
10
0.15
500
1000
1500
500
1000
1500
Training Steps
Training Steps
GRPO
REC-OneSSie-Nol (0.2)
RED-Weight
DISCUSSIONS
Related works. Off-policy RL for LLMs has been studied from various perspectives. Importance sampling has long been considered one foundational mechanism for off off-policy RL; besides TRPO, PPO and GRPO, recent extensions include GSPO (Zheng et (005)) and GMPO (Zhao et al. 2025)
that work with sequence-wist probability ratios, CISPO (Chen al. 2025) that clips probability
ratios rather than token updates, decoupled PPO (Fu et al. 2025a that adapts PPO to asynchronous
RL, among others (Roux et al. 2025; Zheng et al. 2026 Xi et al. 2026 Wang et al. 2025a)
AsymRE (Arnal et al. 2025) offers an alternative base deseline-shiff approach (with ad- hoch analysis for discrete bandit settings), while OPMD (Kimi-Team (0256) partly overlaps with our analysis up to
REBEL (Gao et al. 2024) and CoPG (Feet-Berliac et al., 2024) overlap with our analysis up Eq. (6) before diverging, which will be elaborated in Appendix D Other perspectives for off-polic LLM-RL include learning dynamics of DPO and SFT (Ren & Sutherland] 2025), training offline loss functions with negative gradients on on-policy data (Tajwar al. 2024), improving generalization of SFT via probability-awan 2025). Another line of research integrates expert data into online RL (Yan et al.) 2025c Fu et al. 20256). Our work contributes complementary perspectives to this growing toolkit for off-policy LLM-RL. Further discussion on prior works that are most closely related to main analysis can be found in Appendix D
Eq. 4 before diverging, as discussed earlier in Section 4.2
rescaling (Wu et al.
2023 Zhang et al.]
Limitations and future work. While our work offers a new off-policy interpretation for group- relative REINFORCE and shows its broad implications for LLM-RL, several limitations remain. (1) Our current analysis covers single/multi-steg RL with response/taajectory-lee assumes access to multiple rollouts per query. Future work may expand its scope and applicability, e.g. generalizing to settings with step-level rewards or only one rollout per query. (2) Our analysis lacks formal guarantees for policy improvement convergence. Future work may identify distributional assumptions that yield provable guarantees for REINFORCE variants in off-policy settings. (3) Our experiments focus settings where training data is generated by older policy versions. Extensions to broader off-policy settings, e.g., advanced experience synthesis (Shi et al. 2026) or incorporation of expert data Yan et al. 2025 Zhang et al. 2025c), may reveal new insights. Addressing these limitations will further solidity the theoretical foundation and advance principled algorithm design for off-policy LLM-RI.
rewards, and
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
The authors would like to thank the anonymous reviewers and Area Chairs for their constructive feedback that has helped improve this work.
REPRODUCIBILITY STATEMENT
Full implementation details and hyperparameter configurations documented in Section 4 Appendix B We have released our code publicly to facilitate reproducibility.
and
ETHICS STATEMENT
All datasets used in this study (e.g., GSM8K, MATH, Guru, ToolACE) are publicly available, and no private or personally identifiable information was collected or used. Our contributions methodological, focusing improving the stability and efficiency of RL for LLM post- training. We acknowledge that LLMs may still generate biased or harmful outputs; however, our experiments are restricted benchmark evaluations and do not involve deployment in real-world systems. We believe that releasing our code and reporting detailed hyperparameter settings will foster reproducibility and responsible advancement in this field.
REFERENCES
Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 22-31. PMLR, 2017.
Chenxin An, Zhihui Xie, Xiaonan Li, Lei Li, Jun Zhang. Shansan Gong, Ming Zhong, Jingjing Xu, Xipeng Qiu, Mingxuan Wang, and Lingpeng Kong. POLARIS: A post-training recipe for scaling reinforcement learning on advanced reasoning models. http:://hhkuip..ihhhe. 1o/blogg/025/PPoorzz 2025.
Charles Arnal, Ttn Narozniak, Vivien Cabannes, Yunhao Tang, Julia Kempe, and Remi Munos. Asymmetric reinforce for off-policy reinforcement learning: Balancing positive and negative rewards. arXiv Preprint arXiv:2506.20520, 2025.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint urXiv:2204.055 058662 2022.
Kianté Brantley, Mingyu Chen, Zhaolin Gao, Jason D. Lee, Wen Sun, Wenhao Zhan, and Xuezhou Zhang. Accelerating RL for LLM reasoning with optimal advantage regression. In The Thirty- ninth Annual Conference on Neural Information Processing Systems, 2025.
Aili Chen, Aonian Li, Bangwei Gong, Binyang Jiang, Bo Fei, Yang, Boji Shan, Changqing Yu, Chao Wang, Cheng Zhu, et al. MiniMax-MI: Scaling test-time compute efficiently with lightning attention. arXiv preprint arXiv:2506. 13585, 2025.
Zhoujun Cheng, Shibo Hao, Tianyang Liu, Fan Zhou, Yutao Xie, Feng Yao, Yuexin Bian, Yonghao Zhuang, Nilabjo Dey, Yuheng Zha, Yi Gu, Kun Zhou, Yuqi Wang, Yuan Li, Richard Fan, Jianshu She, Chengqian Gao, Abulhair Saparov, Haonan Li, Taylor W. Killian, Mikhail Yurochkin, Zhengzhong Liu, Eric P. Xing, and Zhiting Hu. Revisiting reinforcement learning for LLM reasoning from cross-domain perspective. arXiv preprint urXiv:2506. 14965, 2025.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Rejichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv, 2021.
Jeff Da, Clinton Wang, Xiang Deng, Yuntao Ma, Nikhil Barhate, and Sean Hendryx. Agent-RLVRR Training software engineering agents via guidance and environment rewards. arXiv, 2025.

---

Published as a conference paper at ICLR 2026
DeepSeek-AL. DeepSeek-Ri. Incentivizing reasoning capability in LLMs via reinforcement learning. arXiv, 2025.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The Llama 3 herd of models. arXiv,, 2024.
Yannis Flet-Berliac, Nathan Grinsztajn, Florian Strub, Eugene Choi, Bill Wu, Chris Cremer, Arash Ahmadian, Yash Chandak, Mohammad Gheshlaghi Azar, Olivier Pietquin, and Matthieu Geist. Contrastive policy gradient: Aligning LLMs on sequence-level scores in supervised- friendlyy fashion. In Proceedings of the 2024 Conference Empirical Methods in Natural Language Processing, pp. 21353-21370, 2024.
Katerina Fragkiadaki. Natural policy gradients, TRPO, PPO.
https://ww..ant
cmu.
edd/ccous
pdf 2018.
Wei Fu, Jiaxuan Gao, Xujie Shen, Chen Zhu, Zhiyu Mei, Chuyi He, Shusheng Xu, Guo Wei, Jun Mei, Jiashu Wang, Tongkai Yang, Binhang Yuan, and Yi Wu. AReaL: A large-scale asynchronous reinforcement learning system for language reasoning. arXiv, 2025a.
Yugian Fu, Tinghong Chen, Jiajun Chai, Xihuai Wang, Songjun Tu, Guojun Yin, Wei Lin, Qichao Zhang, Yuanheng Zhu, and Dongbin Zhao SRFT: A single-stage method with supervised and reinforcement fine-tuning for reasoning. arXiv, 2025b.
Jiaxuan Gao, Wei Fu, Minyang Xie, Shusheng Xu, Chuyi He, Zhiyu Mei, Banghua Zhu, and Yi Wu. Beyond ten turns: Unlocking long-horizon agentic search with large-scale asynchronous rl. arXiv, 2025.
Zhaolin Gao, Jonathan D. Chang, Wenhao Zhan, Owen Oertell, Gokul Swamy, Kianté Brantley, Thorsten Joachims, J. Andrew Bagnell, Jason D. Lee, and Wen Sun. Rebel: Reinforcement learning via regressing relative rewards. In Advances in Neural Information Processing Systems, volume 37, pp. 52354-52400, 2024.
Yongxin Guo, Wenbo Deng, Zhenglin Cheng, and Xiaoying Tang. G²RPO-A: Guided group relative policy optimization with adaptive guidance. arXiv, 2025.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. NeurlPS, 2021.
Zhang-Wer Wei Hong, Pulkit Agrawal, Remi Tachet des Combes, and Romain Laroche. Harnessing mixed offline reinforcement learning datasets via trajectory weighting. In The Eleventh International Conference on Learning Representations, 2023a.
Zhang-Wa Wei Hong, Aviral Kumar, Sathwik Karnik, Abhishek Bhandwaldar, Akash Srivastava, Joni Pajarinen, Romain Laroche, Abhishek Gupta, and Pulkit Agrawal. Beyond uniform sampling: Offline reinforcement learning with imbalanced datasets. In Thirty-seventh Conference on Neural Information Processing Systems, 2023b.
Jian Hu, Xibin Wu, Zilin Zhu, Xianyu, Weixun Wang, Dehao Zhang, and Yu Cao. OpenRLHE: An easy-to-use, scalable and high-performance RLHF framework. arXiv preprint arXiv:2405. 21102.114533 11143, 2024.
Sham Kakade, A natural policy gradient. In Advances in Newral Information Processing Systems, 2001.
Sham M. Kakade and John Langford. Approximately optimal approximate reinforcement learning. In International Conference on Machine Learning, 2002.
Kimi-Team. 2025a.
Remi-Rescarcher.
https::/moonstotal..iiiiii -ow.:uu.ooooiiiieeeeeeeeao
12

---

Published as a conference paper at ICLR 2026
Kimi-Team. Kimi k1.5: Scaling reinforcement learning with LLMs. arXiv:2501.12599, 2025b.
arXiv preprint
Tomasz Korbak, Ethan Perez, and Christopher L. Buckley. RL with KL penalties is better viewed as bayesian inference. In Conference on Empirical Methods in Natural Language Processing, 2022.
Xiao Liang, Zhong-Zhi Li, Yeyun Gong, Yang Wang, Hengyuan Zhang, Yelong Shen, Ying Nian Wu, and Weizhu Chen. SwS: Self-aware weakness-driven problem synthesis in reinforcement learning for LLM reasoning. arXiv Preprint arXiv:2506 2509.08999, 2025.
Weiwen Liu, Xu Huang, Xingshan Zeng, xinlong hao, Shuai Yu, Dexun Li, Shuai Wang, Weinan Gan, Zhengying Liu, Yuanqing Yu, Zezhong WANG, Yuxian Wang, Wu Ning, Yutai Hou, Bin Wang. Chuhan Wu, Wang Xinzhi, Yong Liu, Yasheng Wang, Duyu Tang, Dandan Tu, Lifeng Shang. Xin Jiang, Ruiming Tang, Defu Lian, Qun Liu, and Enhong Chen. ToolACE: Winning the points of LLM function calling. In The Thirteenth International Conference Learning Representations, 2025a
Zihe Liu, Jiashun Liu, Yancheng He, Weixun Wang, Jiaheng Liu, Ling Pan, Xinyu Hu, Shaopan Xiong, Huang, Jian Hu, Shengyi Huang, Siran Yang. Jiamang Wang, Wenbo Su, and Bo Zheng- Part I: Tricks traps? a deep dive into RL for LLM reasoning. arXiv preprint arXiv:2508.08221, 2025b.
Ofir Nachum, Mohammad Norouzi, Kelvin Xu, and Dale Schuurmans. Bridging the gap between value and policy based reinforcement learning. In NIPS, 2017.
Michael Noukhovitch, Shengyi Huang, Sophie Xhonneux, Arian Hosseini, Rishabh Agarwal, and Aaron Courville. Asynchronous RLHF: Faster and more efficient off-policy RL for language models. In The Thirteenth International Conference on Learning Representations, 2025.
OpenAl. OpenAl ol system card, arXiv Preprint arXiv:2412.167200 2024.
Long Ouyang, Pamela Mishkin, Jeff Wu, C L Mar, Jacob Hilton, Amanda Askell, and Paul Christiano. Training language models to follow instructions with human feedback. arXiv, 2022.
Xuchen Pan, Yanxi Chen, Yushuo Chen, Yuchang Sun, Daoyuan Chen, Wenhao Zhang, Yuexiang Xie, Yilun Huang, Yilei Zhang, Dawei Gao, Weijie Shi, Yaliang Li, Bolin Ding, and Jingren Zhou. TFin A general-purpose and unified framework for reinforcement fine-tuning of large language models. arXiv Preprint arXiv:2505. 17826, 2025.
Qwen-Team. Qwen2.5 technical report. arXiv, 2025a.
Qwen-Team Qwen3 technical report. arXiv, 2025b.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In Thirty- seventh Conference on Neural Information Processing Systems, 2023.
Yi Ren and Danica J. Sutherland. Learning dynamics of LLM finetuning. In The Thirteenth International Conference Learning Representations, 2025.
Pierre Harvey Richemond, Yunhao Tang, Daniel Guo, Daniele Calandriello, Mohammad Gheshlaghi Azar, Rafael Rafailov, Bernardo Avila Pires, Eugene Tarassov, Lucas Spangher, Will Ellsworth, Aliaksei Severyn, Jonathan Mallinson, Lior Shani, Gil Shamir, Rishabh Joshi, Tianqi Liu, Remi Munos, and Bilal Piot. Offline regularised reinforcement learning for large language models alignment. arXiv Preprint arXiv:2405. 19107, 2024.
David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. Experience replay for continual learning. In Advances in Neural Information Processing Systems, volume 32, 2019.
Nicolas Le Roux, Mare G. Bellemare, Jonathan Lebensold, Araud Bergeron, Joshua Greaves, Alex Fréchette, Carolyne Pelletier, Eric Thibodcau-Laute Sándor Toth, and Sam Work. Tapered off- policy reinforce: Stable and efficient reinforcement learning for llms. arXiv, 2025.
13

---

Published as a conference paper at ICLR 2026
Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. arXiv Preprint 1rX.::551..5552,, 2016.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pp. 1889-1897. PMLR, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Ning Shang. Yifei Liu, Yi Zhu, Li Lyna Zhang, Weijiang Xu, Xinyu Guan, Buze Zhang, Bingcheng Dong, Xudong Zhou, Bowen Zhang, Ying Xin, Ziming Miao, Scarlett Li, Fan Yang, and Mao Yang. rstar2-agent: Agentic reasoning technical report. arXiv, 2025.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Junxiao Song Runxin Xu, Mingehuan Zhang, Y.K. Li, Y. Wu, and Daya Guo. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models. arXiv, 2024.
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng. Haibin Lin, and Chuan Wu. HybridFlow: A flexible and efficient RLHF framework. arXiv, 2024.
Weijie Shi, Yanxi Chen, Zexi Li, Xuchen Pan, Yuchang Sun, Jiajie Xu, Xiaofang Zhou, and Yaliang Li. Rº:: Reflect-then-retry reinforcement learning with language-guide exploration, pivotal credit, and positive amplification. arXiv, 2026.
David Silver and Richard S. Sutton. Welcome the of experience. //stoorage.coogllppiis.ccomo///p http://ppppps..eec/////mmedmeeea of Theee0000a2220000500000eeeeeeee2
https:
pdf 2025.
Yuda Song. Gokul Swamy, Aarti Singh. J. Andrew Bagnell, and Wen Sun. The importance of online data: Understanding preference fine-tuning via coverage. In Advances in Neural Information Processing Systems, volume 37, pp. 12243-12270, 2024.
Richard Sutton, Andrew G Barto, et al. Reinforcement learning: An introduction. MIT press Cambridge, 1998.
Fahim Tajwar. Anikait Singh, Archit Sharma, Rafael Rafailov, Jeff Schneider, Tengyang Xie, Stefano Ermon, Chelsea Finn, and Aviral Kumar. Preference fine-tuning of LLMs should leverage suboptimal, on-policy data. In Forty-first International Conference on Machine Learning, 2024.
Sharan Vaswani, Olivier Bachem, Simone Totaro, Robert Müller, Shivam Garg, Matthieu Geist, Marlos C. Machado, Pablo Samuel Castro, and Nicolas Le Roux. A functional mirror ascent view of policy gradient methods with function approximation. In Proceedings the 25th International Conference on Artificial Intelligence and Statistics (AISTATS), volume 151, Valencia, Spain, 2022. PMLR.
Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, Shengyi Huang. Kashif Rasul, and Quentin Gallouédec. TRL: Transformer reinforcement learning. https::/gitte
www.coo/.cc.o.eeeeeeeec http://gggngggacc/ttttee
2020.
Jiakang Wang, Runze Liu, Lei Lin, Wenping Hu, Xiu Li, Fuzheng Zhang, Guorui Zhou, and Kun Gai. Aspo: Asymmetric importance sampling policy optimization. arXiv, 2025a.
Weixun Wang, Shaopan Xiong, Gengru Chen, Wei Gao, Sheng Guo, Yancheng He, Ju Huang, Jiaheng Liu, Zhendong Li, Xiaoyang Li, al. Reinforcement learning optimization for large- scale learning: An efficient and user-friendly scaling library. arXiv preprint 0rXiv:2506.06122. 2025b.
Ronald J Williams. Simple statistical gradient- followwig algorithms for connectionist reinforcement learning, Machine learning, 8(3):229 256, 1992.
Yongliang WU, Yizhou Zhou Zhou Ziheng, Yingzhe Peng. Xinyu Ye, Xinting Hu, Wenbo Zhu, Lu Qi, Ming- Hsuan Yang, and Xu Yang. On the generalization of SFT: A reinforcement learning perspective with reward rectification. arXiv preprint arXiv:2508 05629, 2025.
14

---

Published as a conference paper at ICLR 2026
Zhiheng Xi, Xin Guo, Yang Nan, Enyu Zhou, Junrui Shen, Wenxiang Chen, Jiaqi Liu, Jixuan Huang, Xun Deng, Zhihao Zhang, Honglin Guo, Zhikai Lei, Miao Zheng. Guoteng Wang, Peng Sun, Rui Zheng, Hang Yan, Tao Gui, Qi Zhang, and Xuanjing Huang. Stabilizing off-policy reinforcement learning for LLMs via balanced policy optimization with adaptive clipping. In The Fourteenth International Conference Learning Representations, 2026.
Jianhao Yan, Yafu Li. Zican Hu, Zhi Wang, Ganqu Cui, Xiaoye Qu, Yu Cheng. and Yue Zhang. Learning to reason under oof-policy guidance. arXiv Preprint arXiv:2504. 14945, 2025.
Feng Yao, Liyunn Liu Dinghuai Zhang, Chengyu Dong, Jingbo Shang, and Jianfeng Gao.
Your efficient RL framework secretly brings you off-policy RL training. https::///// /ttgg/onngg..... 2025.
notion. sierioonsitee/off-oolcc
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang. Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu. Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu, Wei-Ying Ma, Ya-Qin Zhang, Lin Yan. Mu Qiao, Yonghui Wu, and Mingxuan Wang. DAPO: An open-source LLM reinforcement learning system at scale. arXiv preprint ar arXiv:2503. 14476, 2025.
Guibin Zhang. Hejia Geng, Xiaohang Yu, Zhenfei Yin, Zaibin Zhang, Zelin Tan, Heng Zhou, Zhongzhi Li, Xiangyuan Xue, Yijiang Li, Yifan Zhou, Yang Chen, Chen Zhang, Yutao Fan, Zihu Wang, Songtao Huang, Yue Liao, Hongru Wang, Mengyue Yang, Heng Ji, Michael Littman, Jun Wang, Shuicheng Yan, Philip Torr, and Lei Bai. The landscape of agentic reinforcement learning for LLMs: A survey, 2025a.
Kaiyan Zhang. Yuxin Zuo. Bingxiang He, Youbang Sun, Runze Liu, Che Jiang, Yuchen Fan, Kai Tian, Guoli Jia, Pengfei Li, al. A survey of reinforcement learning for large reasoning models. arXiv preprint 25xhv22509008827 2025b.
Wenhao Zhang, Yuexiang Xie, Yuchang Sun, Yanxi Chen, Guoyin Wang, Yaliang Li, Bolin Ding, and Jingren Zhou. On-policy RL meets off-policy experts: Harmonizing supervised fine-tuning and reinforcement learning via dynamic weighting, arXiv preprint arXiv:2508. 11408, 2025c.
Yuzhong Zhao, Yue Liu, Junpeng Liu, Jingye Chen, Xun Wu, Yaru Hao, Tengchao Lv, Shaohan Huang, Lei Cui, Qixiang Ye, Fang Wan, and Furu Wei. Geometric-mean policy optimization. arXiv preprint arXiv:2507.20673. 2025.
Chujie Zheng, Shixuan Liu, Mingze Li, Xiong Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, Jingren Zhou, and Junyang Lin. Group sequence policy optimization. arXiv preprint urXiv:2507. /8071, 2025.
Haizhong Zheng, Jiawei Zhao, and Beidi Chen. Prosperity before collapse: How far can off-policy RL reach with stale data LLMs? In The Fourteenth International Conference on Learning Representations, 2026.
Brian D. Ziebart, Andrew Maas, J. Andrew Bagnell, and Anind K. Dey. Maximum entropy inverse reinforcement learning. In Proceedings of the 23rd National Conference on Artificial Intelligence Volume 3, pp. 1433-1438, 2008
15

---

Published as a conference paper at ICLR 2026
LLM USAGE STATEMENT
We used large language models (LLMs) only as general-purpos writing assistants to polish the presentation and improve the clarity of the text. All research contributions and findings solely the work of the authors.
A EXTENDING SECTION 2.2 TO MULTI-STEP RL
This section extends the off offi-poliy interpretation proposed in Section 2.2 to multi-step RL settings. Let us start by introducing some notations. In multi-step RL, the initial prompt z is also regarded as the initial state s² x. A rollout trajectory consisting of multiple turns of agent-environment interaction is denoted by
as', ,.....)) , ISESITT
where of is the state and at is the action, i.e., LLM response (akin to y in Section 2.2 Let of denote the context to step f, so that a ~ (-|c²) ) for some policy æ. Throughout this section, we consider trajectory-level rewards r(z, T). Let po(-|z) denote the trajectory distribution induced by policy тe at initial state s' = I.
The following analysis focuses on the t-th iteration, updating the policy model from 0g 0s+1.
Step 1: surrogate objective and consistency condition. For the t-th iteration of policy optimization, consider the following KL-regularized objective:
plo. (1x))
max J(0; mo.) = EEEDD E (r T) - T DRc (po(1 po,
(11)
The optimal policy æ and the induced trajectory distribution satisfies the following: for any trajectory T,
P(T x:) PO. (T())er)))
((x, where
(12)
((x, po.) = f po. ()) dT' = Eyi =p (1x) (eyle"
(13)
This is equivalent the following: for any pair of trajectories Ti and T2,
p(Ti xx) Pe. (Ti xx)
p( xx) Te. (72 (72
Taking logarithm of both sides and doing some rearrangement, we have equivalently
r(æ, T1) r (log p(T) [x) - log Pa, (T]l)) = r(z, T2) T
log pe, (T2 (z).
(14)
Note that for trajectory T, we have
log pTT - log (Tx) = Σ log π (aa - Elog log Te. (a
since the state-transition probability terms log (T] and log pe, (T)z) cancel out.
Step 2: surrogate loss with finite samples. Given K trajectories from the same initial state I, we define the following mean- squared surrogate loss that enforces the consistency condition:
L(0; TT.) =
1
Σ
(a. - )
(15)
K
(1+
ISKGSK
where a a= rl, Th)
(E log To (e(a)()))
Σ log ne, (aflet).
(16)
With infinite samples and sufficient coverage of the action space, the optimum of this surrogate loss would be the same the optimal policy for the surrogate objective in Eq.. (11,
16

---

Published as a conference paper at ICLR 2026
Step 3: gradient step of the surrogate loss.
By the same trick as in Section 2.2 we have
Vo(a: - aa) e. 27 (r(2, T:) - r(z,, T7)) (v)
Σ log To (aflc)) Vo Σ log TO (a)
and
V E
(a: aay)
-2rK
Σ (r(z, TT) -(r))
mo ro moggml(afl)))
(1 T}
10 (1 71+ ISiSK
KSKjSK
where F(z) = Eisisk r(x, Tj)/K denotes the group mean reward in the last line.
In sum, the gradient of the surrogate loss in Eq. (16)
becomes:
VoL m0 T)+
27
1
Σ
(r(x, F(x)) Vo Σ log to (a (af
K
1SiSK
This motivates the following policy update step:
500:2, (T (Tarssssss)
2r
Σ
(r(z,T)) Vo Σ
log mo (a(la))))) (17)
(1+7)²
K
ISISK
ISPETT
which concludes our derivation of group-relative REINFORCE in multi-step RI. settings.
B IMPLEMENTATION DETAILS AND ADDITIONAL EXPERIMENTS
We implement all algorithms with the Trinity-RFT framework (Pan et al, 2025). and run experiments NVIDIA L20., H20, and A800 GPUs. See Tables I and 2 for detailed configurations of our experiments.
B. DATASET DETAILS
We provide additional descriptions of the datasets used in our experiments:
GSMSk (Cobbe al. 2021) is widely used benchmark with 8.5k grade-school math word
problems, designed to test arithmetic reasoning and step-by-step problem solving.
MATH (Hendrycks et al. 202T) covers algebra, geometry, probability, and number theory, containing 12.5k examples total (7.5k for training and 5k for testing); demands advanced symbolic reasoning beyond GSMSk.
Guru (Cheng et al.] 2025) is a multi-domain reasoning dataset with 91.9k examples spanning math, code, science, logic, simulation, and tabular tasks; we its math subset (around 54k samples), which introduces diverse problem formats for evaluating transfer of reasoning strategies.
ToolACE (Liu et al., (225a) is multilingual benchmark with around 11k synthetic samples designed to evaluate L.LMs' ability to solve tasks by selecting and invoking external tools via striet JSON-formatted function calls;
5k single-turn subset in our experiments.
B.2 UNDERSTANDING THE SYNCHRONIZATION PARAMETERS
We parameterize rollout-training scheduling by two configuration parameters in Trinity-RFT: the synchronization interval (sync_interval and synchronization offset (syncLoffset) Their meanings visualized in Figure and explained in the following.
The parameter sync.interval specifies the number generated rollout batches (which equals the number of gradient steps for training the policy model) between two consecutive executions of model weight synchronization. When
= 1, the rollout and policy models synchronize after each gradient step with one batch of samples, yielding strictly on-policy process (if we ignore the issue of precision mismatch between rollout and training engines (Yao et al. 20251). When syne interval > 1, syne interval rollout batches generated with stale
17

---

Published as a conference paper at ICLR 2026
Table 1: Default hyperparameters, Deviations from defaults are noted in figure captions.
GSM8K
ToolACE
Guru Qwen2.5
Guru Qwen3 30B-A3B
MATH
Qwen2.5
Llama-3.2
Llama-3.1
1.5B
3B
7B
8B
Learning rate
x 1x10-6
x 10
1 x 10 10-6
2 10
5 10
Batch size
96
96
64
72
64
8
16
16
Weight decay
0.01
0.01
0.1
0.1
0.1
Warmup steps
0
O
80
80
40
Eval temperature
1.0
N/A
N/A
N/A
0.6
Eval top-p
1.0
N/A
N/A
N/A
1.0
Figures
2
10
12
11
Table Other shared hyperparameters across all experiments.
Parameter
Value
Optimizer
Adam
(2),82)
(0.9, 0.999)
Gradient clipping
1.0
Warmup style
constant
Weight-decay increment style
constant
Auxiliary LR decay style
exponential
Training inference temperature
1.0
Training inference top-p
1.0
model weights before synchronization, which accelerates the overall RL process through pipeline parallelism but incurs off offlcolicyness.
The parameter sync offset specifies the lag between the generation and consumption of each batch. More specifically, sync of fset batches are generated and saved the buffer before training is launched, which is also useful for reducing pipeline bubbles and improving hardware utilization (Noukhovitch et al. (025). some of our experiments, we deliberately set sync_offset to large value, in order to simulate scenario where reward signals from the environment are lagged.
In general, with ) n), the off- off-policyness of consumed batch with zero-index id corresponds to its temporal distance from the most recent synchronized policy is (1 mod m) + For example, (4, 0) yields off off-policyness 0, 1, 2,2. within each interval, while (1, 4) yields constant off-policyness of
syncooffeet
Mode: interval 4, offict -
Mode: interval
offict
Rollout
Ralloct
Buffer
Buffer
Training
Training
syne
Figure A visualization of the rollout-traine -rauutng scheduling sync.interval (left) or sync_offset 4 (right) modes. Each block denotes one batch of samples for one gradient step, and the number in it denotes the corresponding batch id. Training blocks are color-coded by data freshness, with lighter color indicating increasing off- off-policyness.
18

---

Published as a conference paper at ICLR 2026
A=0
A
A-0
1-
I+ 1+thg
GRPO/ REC-OneSide
RRC-TwoSide
REC-Ring
Figure 7: A visualization of activated gradient for various REC algorithms. Here, A represents the advantage specific token, and an arrow pointing to the right and annotated with "A > 0" means there is activated gradient that incentivizes increasing To when the token advantage is positive and the probability ratio me/ Told lies in the corresponding interval.
B.3 REC WITH DIFFERENT CLIPPING MECHANISMS
In addition one-side clipping investigated in Section 4 here we compare additional clipping mechanisms for the REC series, to understand how the geometry of clipping asymmetric vs. symmetric bounds and the presence of a zero- sero-gradient band affects the learning process.
REC-TWOSIDE-IS/NON We replace the mask Mi in REC-ONESIDE-ISNNOS in Eq. ( with two-side mask
To(v x, u, <t) Toia( x. U.
M = 1 (1 flow S
< 1 + <nigh
(18)
Two-side clipping imposes weaker regularization than one-side clipping does with the same clipping parameter (Cow, Chigh). This can potentially improve training efficiency, but might also be risky when To/ Told goes far off. To compensate for this, we design REC-RING.
REC-RING. In addition to the inner band (1 Clow (high) as in Eq. we further specify outer safety margins "oo 2 flow and high 2 Chigh- The REC-RING mask is:
M 1 (1 Clow
To (vl U, 5)
S 1 (high)
(19)
Total (vf
xr 1/. <t
To y xr, v1
+ (A, and
S k(ow)
(20)
Told (y} v.
( 2. y,
+ 1 A, < 0 and
> 1 + <nigh
(21)
Tola u/ x; (
A comparison of the clipping mechanisms are visualized in Figure 7 Note that REC-ONESIDE and REC-TWOSIDE can regarded as special cases of REC-RING.
Experiments. We compare the following algorithms: REINFORCE, GRPO, REC-TWOSIDE- IS, TEC--TWSSDDE-NOS and (0.2, 0.2), and for RING additionally set (<low nighh = (0.6, 2.0).
Clipping parameters are set to (Cows Qhigh) =
Figure presee&peeentts presents the empirical results. We observe that for REC-TWOSIDE, importance sampling is non-essential all three settings, akin to the case of REC-ONESIDE, In addition, REC-TWOSIDE methods demonstrate fast policy improvement the beginning but tend collapse later on, whereas REC-RING achieves a better balance of convergence speed and stability.
B.4 ABLATION: THE IMPACT OF LEARNING RATES
Recall that in Section 4.1 we have demonstrated empirically the advantages of enlarging the clipping parameters flow: Ehigh for REC-ONESIDDENNIIS One might wonder if the relatively weak
It turns out that though derived with different rationales.
resembles the sPPO algorithm proposed by Vaswani al. 2022
19

---

Published as a conference paper at ICLR 2026
syne interval 20
syne offset =
offline
Evaluatinn
1.00-
Triinnng Reward
0.75
0.50
8.00
1.50
Divergoice
10²
10'
10
10"
Clipping
60
10
170
Training REC-TwoSid--Ndl's REC-TweSidA-SS (0.2)
Training Steps
Training
RHINFORCE
RBC-Ring-NolS (0.2,
GRPO
Figure 8: Comparison of REC variants on GSM8K with Qwen2.5-1.5B-Instree under different off-policy settings. Evaluation accuracy, training reward, KL divergence (with respect to the initial model) and clipping fraction are reported. Training reward curves are smoothed with running- average window of size 3.
20

---

Published as a conference paper at ICLR 2026
syne inferval
syn Interval
6,5
6.6
6.4
100
125
1.40
160
20
80
105
120
140
160
Triining Stepe
Tratning Stepe
GRPO 1c-5)
RBC-OnnSia-NulS
R5C-OmSidd-NulS 2.0)
(RRO (4.2) (Ir 20-6)
RBC-OnrSiio-Nii (0.2) (x- 2c-6)
SiC--OnSSid-No (0.6, 2.0 (rr 20-6)
GRPO
RBC-OmsSia-NulS.p0
REINFORCE
Figure 9: Comparison of GRPO and REC-ONESIDE-NOIS GSM8K with Qwen2.5-1.5B.
Instruct. Evaluation accuracy (left) and training reward (right) are reported for varying learning rates.
performance of GRPO or REC-ONESIDE with conventional Clow <high 0.2 is genuinely rooted the clipping mechanism itself, or simply due to the choice a small learning rate.
To answer this, enhance the experiment of Figure 2 by sweeping learning rates over (1x10-5 2x 10-6, 5 10-6). The results are illustrated Figure which confirm that simply increasing the learning rate cannot bridge the performance gap between GRPO with <low <high 0.2 and REC- ONESIDE-NOIS with élow = 0.6, Ehigh 2.0. This shows that relaxing the clipping range acts as genuine improvement of regularization, rather than merely mimicking a larger learning rate.
Remark 1. Let us elaborate on the initial motivation and rationale behind our investigation of enlarging the clipping range. In the conventional theory of importance sampling and bias-varians variance trade-off for policy gradient estimation, we need certain trust-region condition that constrains the distance between the actor policy To and behavior policy Vold- By this rationale, LLM-RL frameworks commonly adopt small values flow <high 0.2 for the default clipping range, or relax up to Ehigh 0.28 proposed by DAPO (Yu al. 2025). Our work, in contrast, proposes native off-policy interpretation for group-relative REINFORCE The essential rationale is no longer about unbiased policy gradient estimation, and importance sampling found to be non-essential. While clipping remains a valid option (among many others) for regularization, we are curious about whether the conventional choice of narrow clipping range still necessary, especially after observing that GRPO's 's learning progress slows down drastically (as the fraction of clipped tokens grows) with increasing off- policoiicssess. Conceptually, our off-policy interpretation would advocate larger clipping range than allowed by the conventional policy gradient theory. We hope that our preliminary exploration of this aspect could inspire the community to rethink the true working mechanism behind GRPO-like algorithms and the possibility of faster off- off-policy learning.
B.5 EXPERIMENTS FOR OPMD AND ASYMRE
Figure I0 presents empirical results for OPMD and AsymRE in various off-policy settings. It is worth noting that, while the analysis and experiments in their original papers (Kimi-Team) 2025b
Arnal et al., (025) focus on setting that is effectively the same as our sync_i sync_interval inteivatraall >
setting, our analysis and experiments have also validated their efficacy in syncLoffset > scenarios.
B.6 ADDITIONAL DETAILS AND RESULTS FOR RED ALGORITHMS
We present further implementation details for the RED-DROP and RED-WEIGHT algorithms investigated in Section
DRD-D When the number of negative samples in a group exceeds the number of positive ones, randomly drop the excess negatives so that positives and negatives balanced. After this subsampling step, we recompute the advantages using the remaining samples, which then fed into the loss.
21

---

Published as a conference paper at ICLR 2026
sync interval 20
syne offiet
offline
11.8
Evaluttion
0.4
0.2
0.0
20
100
130
50
100
150
50
150
150
Toining Reward
0.75
0.50
50
1:0
50
1.00
30
150
Training Steps
Training Steps
Training Steps
REINFORCE
REC-OusSide-Natt (0.6, 2.0)
OPMD
AsymRE (-0.1)
Figure 10: Empirical results for OPMD and AsymRE (ef. Section 4.2) on GSMSK with Qwen2.5- I.SB-Instruct under various off-policy settings. The regularization coefficient for OPMD and the baseline shift for AsymRE are both 0.1. Training reward curves are smoothed with a running- average window of size 3.
syne Interval 20
syne, interval 20
synte iite_iineeaallll 20
syne itterval 20
0.00
2000
Enerpyy
1500-
0.4
0.00
200
600
350
403
250
400
200
450
Treating Steps
Truhting Steps
Triinig Steps
Tainku Steps
REC Owdddd
RED
sync interval
MAT1500 Accuracy
0.50
Figure 11: Comparison BED-WEIGHT, REC- ONESIDE-NOIS, and GRPO on MATH with Llama-3.1-8B, Instruuttt Reported metrics for training include reward, KI. divergence, entropy, and response length. We also report evaluation accuracy on the MATH500 subset.
0.45
GRRP
0.40
REC-OnuSide-NatS (0.2)
RED- Weight
400
Training Steps
RED- WEIGHT: Each sample i is weighted by w; = exp(A:/T). where A, denotes its advantage estimate and T 0 is a temperature parameter controlling the sharpness of weighting. Intuitively, this scheme amplifies high-advantage samples while down- weighting low-advantage ones. We fix
for all experiments.
Additional experiments for RED- WEIGHT, and its comparison against GRPO and NoIS, can be found in Figure IT We observe that for the MATH dataset and Llama-3, 3. 3-B- 8B--Intrrcct RED-WEIGHT achieves higher rewards with lower KL divergence, while maintaining more stable entropy and response lengths.
B.7 GSPO: SEQUENCE-LEVEH IMPORTANCE SAMPLING COULD BE NON-ESSENTIAL.
proposes to replace token-wise clipping and importance sampling in GRPO with sequence- wise counterparts. Similar to Finding Fl in Section 4 for GRPO, hypothesize that GSPO's effectiveness stems from sequence-level clipping as regularization, rather than from sequence- importance sampling. We provides preliminary validation for this hypothesis, through experiments with GSPO-style REC variants.
Group Sequence Policy Optimization (GSPO) Zheng al. 2025
22

---

Published as a conference paper at ICLR 2026
yyn_interval 8
syne_ poe_interval =
0.4
Training Reward
Clipping Fraction
0.3
0.7
0.6
0.1
200
400
600
200
400
600
Training Steps
Training Steps
REC-OneSide-NoI3 (6c-4, 8c-4)
REC-OneSide-15 (6c-4, 8c-4)
Figure 12: Empirical results on Guru-Math with Qwen3-30B-AN (MoE). Training reward curves are smoothed with running-average window size
Implementations. Given prompt and K responses {uilsigk, let s;(0) denote the length- normalized sequence-level probability ratio for y::
Te x)
To x. y; () Tola y1 ()
8, (0) ==
xx) = exp
log
old (y/s
We further define the side sequence-lever clipping mask as
Mj 1 A: > (0) S enigh
A < 0, (; 2
Elow
With these notations in place, implement two GSPO-style REC variants as follows:
REC-GSPO-IS: = K
Σ
Vo log To x, (rs - F) s( M,,
ISic
REC-GSPO-NOIS: g
Ve log (o x, (r: -F) M..
ISiCK
K
Ist5lvi
One can check that REC-GSPO-IS is equivalent to GSPO (except that we use r; F as the advantage, without normalization by or), while REC-GSPO-NOIS discards the sequence-level impotiance-smmlling weights.
Experiments. We use the Guru-Math dataset and mixture-of-expprr (MoE) model Qwen3- 30B-A3B (Qwen-Team) 2025b) since stable RL for MoE models is one of the main motivations behind GSPO (Zheng et al. 2025). We set sync_inte eeval 8, flow x 10-4 and <high
x 10-4; other hyperparameters can be found in Tables and
Figure 12 shows that the learning curves of both REC-GSPO variants - with or without importance sampling - mostly overlap, indicating that importance sampling is likely a non-essential component for the effectiveness of GSPO.
C SUMMARY: A UNIFIED VIEW OF VARIOUS ALGORITHMS
For convenient reference, Table summarizes the algorithms investigated in Section
23

---

Published as a conference paper at ICLR 2026
-log Tota (3ri (=)
noel a. M
Foal!! a se Mf
Foly z.v.
E, (log no (y (x)
= R E E. Vo log no I., 25 A Felv zz, ci
- R > log me (yn (x))
Vo log lo (vila))
uy = exp(A.,T) (T
E.E, Vo log to xx,6 (r; M,
E. Vo log no To (v) a ". (ri - F)
L= E 1 ro mo (31 + 8
2R
= 1 - Ts) V log To (16
E,,,, us. ( Σ "0557
Σ m.,
Vo no (Vi [x),
Table 3: A summary of algorithms investigated in Section
1= E1s - -T) log TT Ta (3:
E, milr log
Gradient Loss
9 = K DE,,,,
= K
g K
Painwise Painwise-weiig weighted REINFORCE
REC- ONEEOOEEEEEENNIO -NIIS
NoIs
REC- ONESSDEES
RED RED-Weight Weight
Algorithm
RED-Drop
AsymRE
GRPO
OPMD
Regularize by clipping
Add regularization loss
Augmentation
Reweight data
24

---

Published as a conference paper at ICLR 2026
D ADDITIONAL RELATED WORKS
We focus our discussion on prior works that are most closely related to our core analysis in Section 2.2 In tabular setting, the surrogate objective in Eq. (3) KL-regularized reward maximization - can be regarded as an instantiation of mirror descent, whose optimum admits the closed form Eq. 4 In more general settings with parameterized policy me and large action space, it is infeasible to realize Eq. (4) directly, and one would resort to optimizing the model parameters. Various algorithms have been developed on the basis of Eq. 3 and including Kimi's OPMD
Kimi-TTeam 2025b) as explained in Section 4.2
REBEL (Gao al. 2024) has a derivation that largely overlaps with our Step and 2 analysis in Section 2.2 It then seeks to solve the squared loss in Eq. (6). which enforces the pairwise consistency condition Eq. (5). CoPG (Flet-Berliac et al., 2024) takes a similar approach, except that it uses a fixed reference policy (rather than the current iteration TH,) for KL regularization. Compared to REINFORCE-style algorithms for which enterprise-gres grade LLM-RL frameworks like verl (Sheng et al., 2024) and Trinity-RFF -FFT (Pan et al., (2025) have been heavily optimized for REBEL and CoPG could be less infrastructure-frinne data parallelism and gradient accumulation, these frameworks
or efficient. For example, in the presence of automatically divide mini-batch into multiple micro-batches (each containing multiple or just one sequence) in way that maximizes load balancing and training efficiency, while minimizing peak memory usage. However, solving the squared loss in Eq. (6) (like REBEL does) contradicts these performance optimization techniques, as it requires paired responses for the same prompt to be located within the same micro-batche This constraint increases infrastructure complexity and peak memory usage, as reported by |Brnntey et al.) (2025). Our Step 3 analysis in Section 2.2, on the other hand, proposes to take one gradient descent step for the squared loss, leading to group-relative variant of classic REINFORCE while giving it a native pollpo interpretation.
Natural Policy Gradient (NPG) (Kakade 2001) can be derived by approximating the surrogate objective in Eq. ( with first-order Taylor expansion for the max-reward term and second-order Taylor expansion for the KL term, and then setting its gradient to zero. Since NPG requires on- policy sampling, it is less relevant to our study of off-policy LLM-RL. DPO (Rafailov et al.] (023) was also derived on the basis of Eq. B 4 and ( but in substantially different setting, with pairwise preference data and the Bradley-Tergy Terry assumption.
25


Reviews:
[{"review_id": "0Noqvecg6M", "reviewer": "Reviewer_vzAn", "summary": "The authors frame a GRPO variant as an off-policy algorithm, and use it to justify and unify several recent empirical heuristics under different themes.", "questions": "(1) In https://arxiv.org/abs/2404.16767, authors analyze a similar square-loss objective and note that taking a single-step of Gauss Newton recovers the Natural Policy Gradient. They also discuss the variance-reduction benefit of using other samples in the same \"group\" as a baseline. Could you comment on the similarities and differences between your work and theirs? Also, I believe the REBEL algorithm in that paper is precisely computing the $\\tilde{\\theta}s$ in Figure 1 -- could you confirm this?\n\n(2) Can you spell out the point of Sec. 4.1's experiments more explicitly? Also, re: AsymRE, it might be good to explicitly write out how this shift effects the closed-form solution -- I think it should boil down to a change in $\\tau$.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "(+) The exposition of this paper was crystal-clear, well done! I appreciated the structuring and how clearly key results were called out. \n\n(+) It feels like every day on arXiv, we see a new GRPO variant. I appreciate the author's efforts to bring some clarity to this fast-moving space.", "weaknesses": "(-) A couple citations are missing. I'd suggest adding in one for MaxEnt IRL above Eq. 4, and to https://arxiv.org/abs/2406.01462 on line 176 re: coverage conditions for the unique minimizer to be the soft-optimal policy. Up to rescaling, I believe the square loss (Eq. 6) is the same as the loss function defined in https://arxiv.org/abs/2404.16767, so I would also add in a reference to that. I might also suggest adding in a reference to https://arxiv.org/abs/2505.20686 in Sec. 4.2.\n\n(-) I find the characterization of REINFORCE as \"inherently off-policy\" a bit strange. I believe what Sec. 2.2 actually proved is that given off-policy data, performing a single step of square-loss minimization leads to group-relative REINFORCE. Whether this data was generated on- or off-policy is an entirely orthogonal point as far as I can see. Let me know if I've misunderstood things.\n\n(-) It would be good to clarify early on in this paper that group-relative REINFORCE and GRPO refer to different things -- I believe the former doesn't include the standard deviation term in the advantage estimate?\n\n(-) I'm not sure how the theory implies that the square loss penalty in Sec. 4.2 is \"principled.\" Similarly, I don't understand how the theory implies that randomly up-weighting samples is \"well-justified.\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oKgM4V6iQk", "reviewer": "Reviewer_5sDW", "summary": "This paper argues that popular reinforcement learning (RL) algorithms for LLMs, like Group Relative Policy Optimization (GRPO), are mistakenly treated as on-policy methods. The authors provide a new theoretical derivation showing that group-relative REINFORCE (the basis for GRPO) is \"secretly an off-policy algorithm\".", "questions": "See weaknesses.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper's main contribution is a new theoretical framework that re-interprets group-relative REINFORCE algorithms, such as GRPO, as natively off-policy. While these methods are typically viewed as on-policy algorithms that can only tolerate small deviations from the data-generating policy, this work provides a \"first-principles derivation\" that does not assume a specific training data distribution.", "weaknesses": "1. The analysis in sec 2.2 is novel and interesting, but as they stated in the paper, this deduction needs the surrogate objective in (11) holds. However, this objective normally does not hold in reality since the policy only takes gradient descent for each sample and won't achieve at the optimal point. Hence, the clear conditions of when the derivation will make one gradient descent improve should be rigorously proved as in TRPO [1].\n\nSince TRPO provides rigorous proofs to discuss when importance sampling can make the algorithm converge, the work to disagree this approach should also prove convergence. Otherwise, the results are not convincing enough.\n\n[1] John Schulman, et al, Trust Region Policy Optimization\n\n2. The experimenters are not valid enough. In Figure 2, they only do experiments for 1.5b model and only one group of sync parameters. They are suggested to compare more base models and more different sync_interval and offset parameters to validate their results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EASx39N2Sr", "reviewer": "Reviewer_ZsF7", "summary": "This paper presents a novel theoretical interpretation of Group-Relative REINFORCE (GRPO), showing that it naturally admits an off-policy formulation without assuming on-policy data. The authors derive this view from first principles, unify several recent algorithms (such as Kimi’s OPMD and Meta’s AsymRE) under the same framework, and support the theory with extensive experiments on reasoning and tool-use benchmarks.", "questions": "- I think the writing style of this paper is quite different from that of typical research papers. Could the authors provide a more systematic discussion of prior work built upon this foundation and establish clearer connections to previous methods?\n- The paper mainly analyzes existing group-based algorithms from a mechanistic perspective. If the authors are unable to design more effective and efficient algorithms based on their proposed theory, then the contribution of this work appears to be overstated.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Provides a fresh and insightful theoretical reinterpretation of GRPO as an off-policy algorithm.\n- Unifies several recent algorithms (e.g., OPMD, AsymRE) under a consistent framework.\n- Writing is clear.", "weaknesses": "- Innovation lies mainly in conceptual reinterpretation, rather than proposing new algorithms.\n- The relation to DPO, Mirror Descent, and other gradient-based RL theories could be expanded.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8fgfzd7zGJ", "reviewer": "Reviewer_5aHq", "summary": "This paper presents a new perspective that REINFORCE-based algorithms, including GRPO, can be interpreted as inherently 'off-policy' algorithms with a solid theoretical foundation. Based on this off-policy interpretation, it proposes two core principles for stable learning: regularizing policy updates and actively shaping the data distribution. It frames previously disparate GRPO-like RL algorithms into a single, unified framework, proposing several new findings and using this interpretation to suggest a insight about how performance can be improved.", "questions": "1. You claim that token-level IS is \"non-essential,\" but your appendix figures demonstrate different aspects between NoIS and IS. Why do you think these phenomena have been observed?\n\n2. Your paper argues that at the token level, clipping is important for stable training of GRPO[1] while IS is \"non-essential.\" Given recent work like GSPO, which moves to a sequence-level framework, do you hypothesize that your findings(especially finding1) generalize? Would you predict that sequence-level IS can also be interpreted as non-essential in that framework under off-policy settings?\n\n[1] Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, Jingren Zhou, & Junyang Lin. (2025). Group Sequence Policy Optimization.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper's core contribution is its rigorous 'first-principles derivation' that recasts GRPO as a native off-policy algorithm. This reinterprets GRPO(and its friends) not as a on-policy method but as a principled 1-step gradient descent on a KL-regularized surrogate objective.\n\n2. The paper successfully reinterprets and unifies recent algorithms like Kimi's OPMD and Meta's AsymRE. This demonstrates the framework's utility as an analytical tool beyond just GRPO, offering a new lens through which to understand a broader class of REINFORCE-based methods.\n\n3. The paper provides a solid theoretical justification for previously heuristic data shaping techniques, such as \"dropping negatives\" (RED-DROP) or \"up-weighting positives\" (RED-WEIGHT). By showing that these established methods can be formally derived from their proposed loss, the authors successfully frame them as principled components of their unified theory, rather than just ad-hoc empirical tricks.", "weaknesses": "1. As the authors acknowledge, the paper proposes a novel interpretation but does not provide a theoretical bridge guaranteeing convergence to an optimal policy.\n\n\n2. The proposal to widen the clipping range (e.g., to (0.6, 2.0)) is presented purely as an empirical finding without theoretical backing (unlike the data shaping principle). The paper provides no detailed analysis on why this wider range is effective, whether it relates to task properties, or how one might determine an optimal \"loose\" range etc. Even though, Appendix B.4 shows that widening the clipping range is different from just controlling learning rate, it still remains a task-specific hyperparameter that requires empirical tuning, rather than a generalizable principle.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)