Revisiting Group Relative Policy Optimization: Insights into On-Policy and Off-Policy Training
Authors: Youssef Mroueh
Nicolas Dupuis
Brian Belgodere
Apoorva Nitsure
Mattia Rigotti
Kristjan Greenewald
Jiri Navratil
Jarret Ross
Jesus Rios
Venue: ICLR 2026 Poster (2026)
Keywords: GRPO
off-policy
Primary area: learning theory
TL;DR: GRPO off-policy

Abstract: We revisit Group Relative Policy Optimization (GRPO) in both on-policy and off-policy optimization regimes. Our motivation comes from recent work on off-policy Proximal Policy Optimization (PPO), which improves training stability, sampling efficiency, and memory usage. In addition, a recent analysis of GRPO suggests that estimating the advantage function with off-policy samples could be beneficial. Building on these observations, we adapt GRPO to the off-policy setting. We show that both on-policy and off-policy GRPO objectives yield an improvement in the reward. This result motivates the use of clipped surrogate objectives in the off-policy version of GRPO. We then compare the empirical performance of reinforcement learning with verifiable rewards in post-training using both GRPO variants. Our results show that off-policy GRPO either significantly outperforms or performs on par with its on-policy counterpart.

Published as a conference paper at ICLR 2026
REVISITING GROUP RELATIVE POLICY OPTIMIZA- TION: INSIGHTS INTO ON-POLICY AND PFFICY TRAINING
Youssef Mrouch, Nicolas Dupuis, Brian Belgodere, Apoorva Nitsure, Mattia Rigotti, Kristjan Greenewald, Jiri Navratil, Jerret Ross & Jesus Rios
IBM Research
ABSTRACT
We revisit Group Relative Policy Optimization (GRPO) in both on-policy and off-policy optimization regimes. Our motivation comes from recent work on off- policy Proximal Policy Optimization (PPO), which improves training stability, sampling efficiency, and memory usage. In addition, recent analysis of GRPO suggests that estimating the advantage function with off-policy samples could be beneficial. Building on these observations, we adapt GRPO to the off-policy setting. We show that both on-policy and off- off-policy GRPO objectives yield an improvement in the reward. This result motivates the use of clipped surrogate objectives in the off-policy version of GRPO. We then compare the empirical per- formance of reinforcement learning with verifiable rewards in post-training using both GRPO variants. Our results show that off-policy GRPO either significantly outperforms or performs on par with its on-policy counterpart.
INTRODUCTION
Proximal Policy Optimization (PPO) (Schulman et al., 2015; 2017) is widely used algorithm in reinforcement learning. Reinforcement learning from Human Feedback (Christiano et al., 2017; Stiennon al., 2020; Ouyang al., 2022; Bai al., 2022) and Reinforcement Learning from Verifiable Rewards (Lambert et al., 2024; Shao et al., 2024) are cornerstones in post-training of large language models align their preferences with human values and to enable reasoning and coding capabilities using verifiable rewards.
Group Relative Policy Optimization introduced in (Shao al., 2024) alleviates the need of training a critic network in PPO and uses Monte-Carlo samples referred to as "a group" to estimate the advantage function via standardized reward, where the mean and standard deviation statisties are estimated using the group. GRPO was used to train the Deepseek R1 reasoning models (Guo et al., 2025) and was adopted by the open-source community as method of choice for post-training of large language models, with open-source implementations in several libraries such as TRL of HuggingFace (von Werra et al., 2020b) and VERL (Luo al., 2025).
Several recent works analyzed the loss implemented in GRPO such as Vojnovic and Yun (2025); Mrouch (2025). The study in Mrouch (2025) suggests that the iterative GRPO of Shao et al. (2024) with sample reuse (i.e. for > 1 in Shao et al. (2024)) leads to off-policy estimation of the advantage and to a success rate amplification when using verifiable rewards. Indeed, it has been observed empirically that this off off-policy advantage estimation leads to an improved performance (HuggingFace, 2025b).
Motivated by these observations and the rich literature off-poliey PPO and RI., like work by Queeney et al. (2021); Meng et al. (2023); Gan et al. (2024); Fakoor et al. (2020) to cite a few (see related work Section for larger account this), in this paper explore the extension of GRPO to the off-policy regime where the advantage is estimated using statisties coming from different policy than the current policy.
"Correspondence mrouch www..wu.cco..

---

Published as a conference paper at ICLR 2026
The main contributions of this paper
We review in Section 2 the iterative GRPO algorithm proposed in Shao et al. (2024) and introduce in Section the off-policy GRPO.
We show in Section 3 that the on-policy and off-policy advantages provide lower bound on the policy improvement of the expected reward (Theorem and Corollary 1).
We state conditions under which optimizing the advantage leads to improvements in the off- off-policy regime, namely, given that the off-policy stays in the vicinity of the current policy and the variance of the reward under the off-policy is non zero, maximizing the regularized off-policy advantage leads to policy improvement. The regularization ensures that the updated policy stays close to the off-policy.
Finally, armed with these results, state the constrained policy optimization problem for off- off-policy GRPO in Section 3.2 and derive a clipped surrogate similar to the ones in off- policy PPO (Gan et al., 2024) and obtain on-policy GRPO clipped objective particular case.
We validate experimentally that training LLMs with off-policy GRPO leads to either im- proved par performance while potentially reducing the communication burden in serving the model in each iteration for inference.
2 ON-POLICY GRPO
Let X' be the space of inputs (prompts in the context of LLMs) and y the space of responses. We denote by Pxx the distribution inputs. We refer the policy want optimize as (-12z), which a distribution on Y conditioned on x px. For A: 2 0, let गH be the policy at the current step k.
The Group Relative Policy Optimization (GRPO) Clipped objectíve introduced in Shao et al. (2024) a variant of Proximal Policy Optimization (PPO) (Schulman et al., 2017; 2015), where the advan- tage computed as standardized reward function with mean and variances computed with respect to a group or Monte-Carke samples of size G sampled from the current policy TA (.(x) for each a independently. For c. 3 and given reference policy Tref, the clipped objective optimization in GRPO is defined as follows:
T (y|ar)
<(ylx) 1 - e, c,1cc) Ane Ax. (I. - -KKL(TT (T||) (T())z))*
max E Byy=x(((o))) (-Ix) min
Arc (I. v), clip
(z)z))
where KL is the Kullback-Leibler divergence, and A is the GRPO advantage functioon
r(x, - Emg r(r, v) A (Ir, v) VEz, (r(x, v) E,, v))
The advantage can be estimated from samples on "a group" of size G for each x, we sample y1...., Vg ~ Ta (-|x) and compute r(x, ve). l = L.....G. G. We refer to the group of reward conditioned on as (re) and the estimated GRPO advantage is therefore (Shao al., 2024):
mean((r:))
Ar, (z. v.) (sre)((rr))+t
where mean and st empirical mean and standard deviation respectively. The statisties used to normalize the reward leading to the advantage function are estimated using the current policy ma. and hence we refer to An the on on-policy advantage.
When compared with PPO, GRPO alleviates the need of training a critic network to compute the advantage and relies instead on standardized rewards that can be estimated efficiently using efficient inference frameworks such as vL.MM (Kwon et al., 2023) in the context of large language models.
GRPO with Verifiable Rewards and Success Rate Amplification The iterative GRPO (Shao et al., 2024) has two overlooked features:
The algorithm suggests to optimize the policy т for μ iterations fixing the samples from Ty, which inherently leads to an off-policy estimation of the advantage.
2

---

Published as a conference paper at ICLR 2026
The algorithm suggests to do the training in stages while changing Tret the latest opti- mized policy with GRPO.
A recent analysis of GRPO with verifiable rewards, i.e. with binary rewards (Mroueh, 2025), sug- gests that this aforementioned off-policy advantage estimation in Shao et al. (2024) leads to an implicit fixed point iteration that guarantees that the success rate of the GRPO-optimized policy is higher than the one of the reference policy. This also explains the multi-stage nature of the iterative GRPO that changes the reference along the training iterations. Motivated by these observations, we propose to take step back and analyze on-policy and off-policy GRPO. In practice, in our proposed off-policy GRPO instead just fixing the samples for i iterations from TT as suggested in Shao et al. (2024), we use the policy ak-a a to estimate the advantage for μ iterations with fresh samples each iteration, and we refer to this as off-policy advantage.
OFF-POLICY AND ON-POLICY GRPO REWARD IMPROVEMENT
We introduce in this Section off-po GRPO, and analyze conditions under which policy reward improvement is possible in both the on on-policy and off- off-policy regimes. Towards that goal, we start with some preliminary definitions. Define the expected reward of a policy given x ~ px:
(T((+zz = E (x,
(I)
For & let TTk be the policy the current step and a(-(z) be policy used for off off-policy sampling, where typically we consider a<-12:) = =T-n(-)2)) for 0 S v < k.
Define the mean and standard deviation of the off-policy reward, i.e. under policy az (I)) = Byyatt 2)r(2,)) y) and (a)(()) (1) v) Ha.r (x)), and denote for 0 << < Ca,r, (z)
The GRPO advantage function computed using the off-policy distribution a is defined as the whitened reward, follows:
An r(x, 9) - (a)
(x)
Our goal to maximize the expected advantage function using importance sampling under the policy a:
<(yl=)
(T((1))))) (y)a(-lu) a(y|z)
Ao (x. y)
If Te, we obtain the online policy objective function of GRPO, where the advantage is computed with the current policy The iec using Any, (x, y).
3.1 POLICY IMPROVEMENT GRPO
Note that goal is to optimize the expected reward under π, ( given in eq. (1), but instead use the expected advantage La (T) where the advantage is computed using a given in eq. (3). Hence, our goal in what follows is to provide lower bound on J[T(((2)) J [m (+<()())) that involves (T),, which guarantees that maximizing the expected advantage function leads to improvement in terms of expected rewards the current policy Ti.
Our lower bounds are given in Theorem and Corollary and they involve the total variation distance TV defined follows:
TV(m1, m2) = / m1 - m22--
Theorem 1 (Policy Improvement Lower Bound in Off-Policy GRPO). Assume that the reward is positive and bounded in I r < 1. Let a be the off-policy distribu- have for all (px a.s.);
tion and Thk the current policy. Then for any policy
NNte in Section 2 we referred to this Th- keep close to notation used in the original GRPO paper. We will use instead of y in the rest of the paper.
3

---

Published as a conference paper at ICLR 2026
Ja.ne(
(m(+12))- - ((T((=)))) 2 La (T((12)) -2 Ta,ne(
(V(T a(+12)) 2 (T(T((- a(-1x:)
If the reward is not bounded by 1 we can scale it by ||r|| so it becomes in [0, 1], without this impacting the overall optimization problem (for a negligble =). Note that this condition on the reward ensures that (x) S 1 which is needed in the GRPO case to get the policy improvement lower bound. Indeed for bounded random variable in [a, b] the variance is bounded by (b-a)² and hence we have (ar(()) I 1 which guarantees that the term 1- (Ta.n. S V++
0, for e S 2.
Ga.n.(2)
For on-policy GRPO i.e. setting
in Theorem we have the following corollary:
Corollary (Policy Improvement Lower Bound in On-Policy GRPO). Assume that the reward is positive and bounded, 0 S 1, and consider (negligible)s such that 0 < 3. Let Te be the current policy, then for any policy ı we have for all (px a.s.):
1 on.ne(()) TV(T(+12), mx(+=))
((T(-1z)) ( (T+((+))) 2 Cxx (T(+Ix))
Tagine (x)
Define
(1 - (x))
Mass
Exxax
(3))
Integrating Theorem on x (prompts) and applying Cauchy- Sehwarr inequality we obtain:
Ex-px (T(-Ix))) - J(TH (-1=)) 2 Ex-px La (T((1=))) 2.a.r. (Bunpx TV (1)) 2Exuxx ((TT(()
Interpreting the lower bound When compared with lower bounds for policy improvement in PPO (Theorem in TRPO (Schulman et al., 2015)) and for off-policy PPO (Lemma 3. in transduc- tive PPO (Gan et al., 2024) and Theorem in Generalized PPO (Queeney et al., 2021)), we observe similar lower bounds with crucial difference that the constants weighting total variations are ab- solute constants for PPO whereas they are policy and data dependent for GRPO. In particular, the dependency of the lower bound on:
(x.
Jo.ne (x)
of interest. We can examine this quantity for verifiable rewards, for each æ the verifiable reward is Bernoulli random variable with parameter the probability of success of the policy given (Mroueh, 2025). Hence we have:
1-0a,ne(1)
1 1
p(l - P)
(1)
Vp(1 (pp1-))- +
We observe that this quantity diverges for fully correct and incorrect answers and this can indeed hurt the lower bound, as the negative terms in the lower bound will be dominating. It was suggested DAPO (Yu et al., 2025) to filter out prompts with fully correct or incorrect answers, this will have the effect of controlling this term in the lower bound and keep that quantity finite.
Comparison To Off-Policy PPO Improvement Lower Bounds. While the statements of the the- orems are similar to off-policy PPO algorithms (GePPO/ToPPO) ((Gan et al., 2024) and(Queeney et al., 202 1)) they crucially follow different proof techniques. The main innovation is extending these results to GRPO. The proof in (Gan et al., 2024) uses the results from (Kakade and Langford, 2002) and (Achiam et al., 2017) in order to establish the lower bound. In particular, the proof uses the result from (Kakade and Langford, 2002) link the difference of expected rewards between
4

---

Published as a conference paper at ICLR 2026
two different policies the total variation between the state visitation distributions, and relies (Achiam et al., 2017) to link this quantity to the total variation between the policies. Note that this reliance the state visitation distributions restricts the analysis to Markov decision Process (MDP) setup. Our proof technique exploits the analytical form of the GRPO advantage and does need the state visitation distribution usually needed in PPO analysis (online and offline). This makes our analysis more general and does not need to be in the MDP setup.
3.2 GRPO: FROM CONSTRAINED OPTIMIZATION TO CLIPPED SURROGATE, OBJECTIVES
From Penalized to KL Constrained Optimization To maximize the lower bound in eq.(4), we that the off-policy needs to be in the vicinity of the current policy 7g i.e. for TV(a, mT)) and that Marr. < 0a (variance terms not exploding). Under these assumptions, we can solve the following penalized problem:
max Exppx (T(22)) - Mains / (TT())
By virtue of Theorem 1, maximizing this objective above leads to policy reward improvement.
We can write this as a constrained optimization, there exists A > such that the following con-
strained optimization problem is equivalent:
max Ex Co (TT(22) subject Ex-px (T(1))) s A
By Pinsker inequality for two measures m1, m2 have TV m2) S / (AKL(m,, m²) and
hence we can bound instead the KL divergence as follows:
1 max Ea subject 2 KL s A2
(5)
From Constrained Optimization to Clipped Surrogate Objectives The objective in (5) is the same in the original constrained PPO formulation (Schulman al., 2015) with two key dif- ferences: the advantage the whitened reward of GRPO where the statistics are computed using the off-policy
and the advantage objective is computed using importance sampling from the off- off-policy a, instead of Tg in both cases. This is indeed related to objectives in off-policy PPO (Queeney et al., 2021; Gan al., 2024). A practical implementation of these objectives is through clipped surrogates (Schulman et al., 2015).
For E [0, 1] following Gan et al. (2024); Queeney al. (2021) let us define:
(r i a) min(ra, clip(r, max(r' 0), r' c) a).
The clipped off-policy GRPO objective for such that TV(a, TT) - and Marro < is there- fore defined as follows: (=(((=)))) Ey fe
(xyla) ax(ulz) <x)))=)) a(vi=) Aa(
Let us unpack this, have: fe
π(yla) (u(u(a) a(ylx)) a(viæ) (
Aa y) min
*(yl=) me + atgla)) a(y(z) ((1l)) a(vlz): max (x(
r(x,y) 2 Ha, (x)
Aa y) max
a(3))) σ) - e, 0)) r(r,y) < parr (x)).
The clipping ensures that the ratio z remains bounded and a
relaxation of the KL (or the total variation distance). Since needs to satisfy closeness to F in order to ensure improvement, the clipping objective incentivizes the difference between I It to not exceed e (Gan et al., 2024).
In practice, the off-policy is F for small [0, k)). Given a small learning rate and a small v, the assumption that the policy Th-u doesn't 't deviate from क. is reasonable, and for v small we can approximate The by 1. We use this approximation in practice we found it more stable, and
5

---

Published as a conference paper at ICLR 2026
given that this approximation is in practice used in off-Policy PPO (with sample reuse) as discussed in Gan et al. (2024) (See Section 4.1 in Gan et al. (2024)).
Back to On-Policy GRPO Clipped Objective For The we obtain the clipped objective for on-policy GRPO (Shao et al., 2024):
((yla) (y|x))
C (T((11)) By-m. (1a)ff
1, An (2.8)
<[y[z)
(1ylz)) Tx 1 e, e, e) Ane (z. (1..))
= -Bymm. min
Are (I, y), clip
Finally putting together our clipped surrogate objective with the KL regularizer we obtain our final objective:
KL- Regularized RL & On-Policy / Off--Policy Algorithms
Ex-ox La (x(+l=)) (KL((T (-
(6)
We present the GRPO algorithm in Algorithm and the configurations that allow toggling between on-policy and off-policy GRPO in Table 1. Within the RL loop, the model is served for inference using vLLM (Kwon et al., 2023). The parameter v controls how often the model is updated on the vLLM server (which corresponds to off- with a Tg-u+1). The parameter i controls how many SGD iterations are applied to each batch sampled from the policy. For v = 1 and i = 1, the model is continuously served, and each batch of samples is used once in SGD. This corresponds to on-policy GRPO. For > 1 and v = 1, the model is still continuously served, but each batch is used i times in the SGD loop: this corresponds to an "off-poliey" GRPO variant, as proposed in Shao et al. (2024). For large models that require tensor parallelism and multi-GPU serving, continuous model serving incurs additional communication costs. Our off-policy GRPO mitigates these costs by serving the model every v >> 1 iterations (line 8 in Algorithm 1) and fixing i = 1. Our theory guarantees reward improvement as long as v not too large.
Computational and Communication Costs Updating the model served by vLLM during GRPO training incurs varying costs depending on the model size, update frequency (v), and parallelism settings. When the training model and vLLM instance reside on different GPUs, or when vLLM uses tensor parallelism (TP), model updates may trigger deep copies and inter-GPU communication. These involve either full weight transfers or partitioned broadcasts, which scale linearly with model size. Frequent updates (c.g.. V = 1) can dominate the runtime, especially for large models (see the recent benchmark vLLM (2025) for latencics in serving large models with tensor parallelism using vL..M). To mitigate this, we update the vI.L.M model every e > iterations. This amortizes the copy cost while maintaining reward improvement guarantees from our theory. For instance, in GRPO's TRL parallel implementation (von Werra et al., 2020b), there are two modes: a single- server dedicated for inference and co-located training and inference (Toslali et al., 2025). For instance, in GRPO's ' TRL parallel implementation (von Werra et al., 2020b), there is a server and a colocate implementation (Toslali et al.. 2025). While the single vLLM server assigns a single GPU for serving the model on single node. colocate shards the vLLM inference model using tensor parallelism (TP) on all available GPUs, colocating the inference and the training on all GPUs. In colocate mode, GRPO can benefit from off-policy updates, as the model does not need to be sharded and served at each iteration of the training when V > 1.
On-Policy Clipped Objective with Zero Variance Masking la DAPO (Yu et al., 2025) As discussed earlier in the interpretation of the lower bound in page 4, the samples with zero variance may lead to total variation terms to dominate the lower bound, hence we propose similar to DAPO (Yu et al., 2025) to mask these samples. For instance in the on policy case this would be with the following masked objective:
Exxax 1 (1) (Cs, (T((1)) - (KL (1)=)))
Remark 1. We make the following remarks on off GRPO:
(i 1,v==111 1) As Off-Policy : This setup was proposed in (Shao et al., 2024), and this form of data reuse can be traced back PPO (Schulman al., 2017). (Gan et al., 2024) analyzed this setup in PPO as an off- off-policy training regime (Section 4.1 in (Gan et al., 2024)), and we refer it off-policy following their lead.
6

---

Published as a conference paper at ICLR 2026
Method name
Update by fixed i
Update of Policy on Server
On-Policy GRPO (Shao al., 2024) Off-policy GRPO (Shao et al., 2024)
i i=
1>
Off-policy GRPO (this work)
i
Training configurations in alg. 1: (v1-i1) is on-policy GRPO and (v1-i10) is an example of off-policy GRPO in (Shao et al., 2024). Our off-polici GRPO corresponds e.g. to (v10-il).
Table I:
Zero-Varianne Variance Masking and Bias In practice the variance is estimated using a finite group size G. Masking based on that estimate introduces bias especially with small group size. (Yu et al., 2025) proposes a dynamic group size that makes sure that at least a success and failure exist within each group, this lowers the bias and caps the number of masked samples.
Algorithm Iterative GRPO with verifiable rewards, modified from Shao et al. (2024)
1: Input initial policy model Tem verifiable reward r; task prompts D;
2: Hyperparameters 6, 3, S,
3: (i, u)=[Numeer of SGD iteration by fixed batch, Model update vLLM server)
4: Policy model
Tém Tref Tha Ta ne
for 1,...,,..
6: for k k= ,, do
7:
Sample a batch D, from Px if mod 0 then Update the old policy model the vLLM server TT m
8:
10:
11:
Sample G outputs { Ta (- æz) for each question æ E D,
Compute rewards {r. k=2 for each sampled output y: by running verifiable reward
13:
a t TT
14:
Compute (z, y4 using Equation (2)
15:
i is referred to as in Original GRPO Update the policy model me by maximizing the GRPO objective with gradient ascent for end for
for GRPO iteration 1...., do
16:
18:
19:
4-
> Swap reference with the latest model
20:
end for
21:
Output RT
4 RELATED WORK
Proximal Policy Optimization (PPO) and Extensions Proximal Policy Optimization (PPO) is widely used on-policy reinforcement learning algorithm that improves training stability through clipped surrogate objectives. While PPO is effective in diverse settings. is inherently limited by its on-policy nature, which constrains sample efficiency. To address these limitations, several off-policy adaptations and extensions of PPO have been proposed. Generalized Proximal Policy Optimization (G-PPO) (Queeney al., 2021) enables sample reuse while maintaining convergence guarantees. Transductive off-Policy PPO (ToPPO) (Gan et al., 2024) builds on G-PPO by incorporating trans- ductive learning principles, bridging the gap between off- off-policy learning and theoretical guarantees of on-policy methods. Off-Policy PPO (OPPO) (Meng et al., 2023) proposes novel corrections to integrate replay buffer samples in PPO-style updates.
On-Policy and Off-Policy Actor-Critic Methods Actor-critic methods blend the strengths of policy gradients and value function estimation. Off-policy variants aim to improve sample efficiency by learning from replay buffer. The Off-Policy Actor-Critic algorithm (Degris et al., 2012) introduces importance weighting to enable stable updates from off-policy data. ACER (Wang et al., 2016) extends this with trust-region optimization and truncated importance sampling, enhancing by that the learning efficiency discrete action spaces. Mixing on-policy and off-policy methods aims to leverage the stability of on-policy updates with the efficiency of off-policy learning. P3O (Fakoor
7

---

Published as a conference paper at ICLR 2026
et al., 2020) provides a principled approach that interleaves policy updates from both on- and off- policy data.
Off-Policy RLHF and other variants of GRPO Noukhovitch et al. (2025) introduced within the iterative DPO framework an asynchronous RLHF using policy data and that ensures faster con- vergence to the optimal policy. New variants of GRPO have been proposed recently such DAPO (Yu et al., 2025) and DR-GRPO (Liu et al., 2025). DAPO proposes the zero variance masking with- theoretical backing, our work roots this in the improvement lower bound. DR-GRPO proposes to center only the reward without using the variance normalization.
5 EXPERIMENTS
5.1 ABLATION STUDIES ON GSM8K
Setup, Model, and Data In our first set of experiments, we use GSM8K dataset from Cobbe et al. (2021) (MIT license), and twen/Qwen2 5-0. IB-Instruct (Apache 2.0 license) by Yang et al. (2024). We integrate Werra et al., 2020b), and train our models on the training split of GSM8K on a node with 8 GPUs (GPU, for the vI.LM server and 7 other GPUs for distributed training). See Appendix A for the hardware specification. We use a learning rate of 5 x 10-6 for all experiments and the KI. regular- izer 3 0.1 in Equation (6). We use the correctness of the LLM output
changes in Algorithm to the GRPO implementation in TRL (von
reward. For GRPO training, the hyperparameters are the following: group size G = 16 and per-device batch size 16 (meaning each GPU processes single prompt with 16 responses). To increase the overall batch- size we use gradient accumulation of 4, ending with an effective batch size of prompts of 28. The context length used for this experiment 200, and the sampling temperature is set 0.1.
OniFiiiyy GITO
On CIPO ruskingg
642
632
631
Ectitoon
(a) On-Policy GRPO with गeee swap at end of each epoch. (v= i =3 3)
(b) On-Policy GRPO with masking of samples with
variance T 0. and with Fref swap end of each epoch. v2 = 1, 10:=1, S= 3)
oe4-aity G2RD to batch to
GF paky DAPO using Vi
09/00
645
5705
3291
262
225
320
Baxsion
Baraicn
(c) Off-Policy GRPO using e 10 (this amounts to fixing the model the vLLLM server for 10 iterations and getting fresh samples new batches), and with Teef swap. (v- 10, i 1, S = 3)
(d) Off-Policy GRPO using fixed samples from for 10 iterations. This will make epach 10x slower.
1, 10, S 1)
Figure 1: We train different variants of GRPO the train portion of GSM8K and report the Pass @ on GSM8K test set using 50 samples for each question in the test set for various variants of on-policy and off-policy GRPO. We see that as predicted by our theory, masking samples with zero variance stabilizes the training for on-policy training and leads to better performance. For off- off-policy training we see that using 10, stabilizes also the training and leads also to better performance.
8

---

Published as a conference paper at ICLR 2026
Ablations and Results We train our models with GRPO using Algorithm 1 with verifiable ward for answer correctness. We use for GRPO different configurations given in Table and report on the test split of GSM8K Pass @1 I using 50 samples (i.e. frequency of success given 50 generations for each question) using the same sampling configuration as in training. We report results in Figure 1, 1) with the objective given in Equation (6) with S = (i.e. for 4 epochs with Reef swap at end of each epoch with latest model); Fig. 1b for on-policy GRPO (i 1, U 1) with masking zero variance samples i.c. using the objective given Equation
1: Fig. Ia for on-policy GRPO
(7) with 5 = 3; Fig. Ic for our off off-policy GRPO (v 10, i 1), with S = 3 and Fig. 1d for Shao al. (2024)'s off off-policy GRPO i. (v 1, 10) for single epoch. Note that we limit here our experimentation for (u = i 10) to S = 1 for fairness as this setup is the most computationally
expensive, 10x slower than all the other setups ((i = 1, = 1) and (i = 1, 1, 10))
We see in Fig. Ia that while the on-policy GRPO converges to maximum Pass of 45% it is unstable. The masking of zero variance sampling 1b stabilizes the on-policy GRPO and leads to an improvement of the performance to 50%. This is in line with our theoretical grounding through the improvement lower bound. Our off-policy GRPO in Fig. Ic stabilizes the training also and leads to improved Pass @ of 50% the test set. In all three cases, to the latest model, GRPO amplifies the success rate above the current Tef. this concurs with the theoretical findings in Mroueh (2025). Finally, the off-policy variant in Shao et al. (2024) in Fig. Id shows a slower convergence over an epoch. Note that the standard deviation of Pass @ over three seeds across iterations ranges between 0.003 and 0.03.
that by resetting the Ref
5.2 FINETUNING QWEN DISTILL RI MODEL (1.5 B) ON DEEPSCALER DATA
In this section 2025) on DeepScall MeeSScall--Peeieeee Srrevie--Datasee from Luo et al. (2025) consisting of roughly 40K' math questions with known answers. We used math-veri fy the verifiable reward. We use learning rate of x 10 in the same distributed setting as before (GPU, for vLLM and 7 GPUs for distributed training). We use a context length of 4096, group size G 16, per-device batch size of 16, and the KL. regularizer is B 0.001. The sampling temperature used is 0.7. We compared here the on-policy GRPO (v 1, i 1) to our off-policy GRPO (v 10, i 1) and report the performance of the trained model single epoch (around 24 hours on a single node). We report in Tables 2 and Aime24 and Math500 performance using Huggingface LightEval (Habib et al., 2023). Aime24 is evaluated with Pass @1 using 32 samples, and math500 with extractive matching as recommended in lighteval with a context length of 32K (evaluation context length and all other sampling hyperparameters
GRPO to finetune DeepSeek-R1-Distte
5B (Guo et al.,
set to the default in OpenR1 for this model). Plots of evaluation as function of iterations are given in Appendix D. We see that both on-policy and off-policy GRPO improve the performance of DeepSeek-R1- wwwsstiiillgeeee
1 5B that has Aime24 of 29% to 32% at maximum (over iterations), and its math-500 from 83% to 87%. This result confirms our theoretical results that by going off-policy we don't 't lose in terms of overall performance.
Model/Aime24
Min
Max
Median
Mean
vi-il-length-4006
0.2802
0.3229
0.3021
0.3022
v10-il-length-4006
0.2781
0.3250
0.3047
0.3049
Table 2: Aime24 using lighteval with on & off-policy ( (yl--i)) and (v10-i1) GRPO.
Model/Math500
Min
Max
Median
Mean
vl-il-length-4000
0.830
0.870
0.854
0.8519
v10-il-length-4000
0.822
0.872
0.846
0.8474
Table 3: Math 500 extractive matching using lighteval (Habib et al., 2023) with on and off-policy (v1-i1) and (v10-i1) GRPO.

---

4, and a group size for GRPO Published as a conference paper at ICLR 2026
Reweadd
pop /
08
044
KL Divergence
9.10
1000
280
2000
4823
5000
3000
Figure 2: Reward and KL comparison of on-policy (blue) and off-policy (orange) GRPO with the Qwen2.5-7B model on the reasoning math dataset orz-math-57k.
5.3 SPEEDUP WITH A 7B MODEL AND COLOCATE GRPO USING TENSOR PARALLELISM
We scaling experiments the Qwen2.5-7B model the reasoning math dataset orz-math-57k (Hu et al., 2025). We used the new implementation of GRPO in TRL with colocate (Toslali et al., 2025) that colocates training and inference using tensor parallelism. On a single node with 8 GPUs, we trained using trl's GRPO with vLLM- colocate and a tensor parallel size TP 4. This means that we have two vLLM servers sharded on 4 GPUs each. In this setup inference on the two vLLM servers is colocated with the training with 70% memory for training and 30% for vLLM. We used = 0, a max completion length of 4096, a per-device-batch-sii of 8, and maximum training steps of 400.
We baselined the on-policy GRPO (v=1) and the off -policy GRPO (v=10) within this setup. We observed for on-policy (v=l) 0.02828 + 0.017 iterations/sec and for off-policy (v=10) 0.03828 + 0.016 iterations/sec. In this experiment with a B model and tensor parallelism (TP=4) on a single node therefore observe 1.35 speedup for the off off-policy with respect to the on-policy GRPO. Both on-policy and off- off-policy training result in the same reward on the orz-math- 57k (50%), this supports claims that off-policy does not compromise the training but leads reduction in wall clock and in memory traffic due to serving the model. We give in Fig. 2 a comparison of the on- policy and pof-policy GRPO in terms of the reward and the KL. of the current policy the reference model as function of the training iteration. We see that while the KL spikes at certain iterations in the off- off-policy case w-1,,il,, this does not undermine the performance that tracks well on-policy GRPO (v1-i1).
6 CONCLUSION AND DISCUSSION
We revisited (on-policy) GRPO (Shao et al., 2024) and showed that its clipping objective can be derived from first principles grounding to masking of zero variance samples suggested in DAPO (Yu et al., 2025). We introduced off-policy GRPO and laid out conditions under which leads to policy improvement. Our off- policy GRPO has the advantage of reducing communication costs in serving the model for inference within the GRPO loop at each iteration as done in the on-policy counterpart, while not sacrificing performance. We showcased that off-policy GRPO stabilizes training and leads to either on par or improved performance as the on on-policy one. The main takeaways of our paper to practitioners are: (1) Zero variance masking stabilizes on-pol GRPO's training (2) Off off-policy GRPO attains its full potential in terms of maintaining performance and lowering latencies and communication overhead in larger scale training where models are served using tensor parallelism (see vL.LM (2025)).
lower bound for reward improvement. We also gave theoretical
10

---

Published as a conference paper at ICLR 2026
REFERENCES
J. Achiam, D. Held, A. Tamar, and P. Abbeel. Constrained policy optimization. In International conference on machine learning, pages 22-31. PMLR, 2017.
Y. Bai, A. Jones, Ndousse, A. Askell, Chen, DasSarma, D. Drain, S. Fort, Ganguli, T. Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.0586. 2022.
F. Christiano, Leike, T. Brown, M. Martic, S. Legg, and D. Amodei. Deep reinforcement learning from human preferences. S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Gamett, editors, Ad- vances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https://ppocceeddgss..eee neurips cc http://cs..e.ppppppe..aea/////// file://Be2caaa600011111222200000 Papeee Paper.pdf.
In I. Guyon, U. V. Luxburg,
K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, I., Kaiser, M. Plappert, J. Tworek, Hilton, R. Nakano, C. Hesse, and J. Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14166 2021.
Degris, M. White, and R. Sutton. Off-policy actor-critic. arXiv preprint arXiv:1205.4832 2012.
R. Fakoor, P. Chaudhari, and A. J. Smola. P3o: Policy-on policy-off policy optimization. In Pro- ceedings of The 35th Uncertainty in Artificial Intelligence Conference, pages 1017-1027. PMLR, 2020.
Y. Gan, R. Yan, X. Tan, Wu, and Xing. Transductive off-policy proximal policy optimization. arXiv preprint arXiv:2406.038%/ 2024.
D. Guo, D. Yang. H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, et al. Deepseek-rl: Incentivizing reasoning capability in lims via reinforcement learning. arXiv preprint ar arXiv:2501.12944 2025.
N. Habib, Fourrier, H. Kydlfček, T. Wolf, and L. Tunstall. Lighteval: lightweight framework for Ilm evaluation, 2023. URL https: / //gttutthb///g/ cww wwwpgggiinnaaaeeeggtteeee
J. Hu, Y. Zhang, Q. Han, D. Jiang, X. Zhang, and H.-Y. Shum. Open-reasoner-zzero: An open source approach to scaling up reinforcement learning on the base model, 2025. URL https: hwwazii..rggaas////oo a25/25/3.22222222 24290..
HuggingFace. Open rl: A fully open reproduction of deepseek-rl, January 2025a. URL https: h/github.com//ugg. face open-rl. rl.
HuggingFace. Open rl: Update #3, Mar. 2025b, URL htw.:htps//hugggggffc.....///// open-r1/update-33 Accessed: 2025-05-11.
M. Kakade and J. Langford. Approximately optimal approximate reinforcement learning. Pro- ceedings of the International Conference Machine Learning (ICML), 2002.
W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, and I. Sto- ica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium Operating Systems Principles, 2023.
N. Lambert, Morrison, V. Pyatkin, S. Huang. H. Ivison, Brahman, L. J. V. Miranda. A. Liu, N. Dziri, S. Lyu, et al. Tülu 3: Pushing frontiers in open language model post-training. arXiv preprint arXiv:2411.15122 2024.
Liu, C. Chen, W. Li, P. Qi, Pang, C. Du, W. Lee, and Lin. Understanding r1 r1-zero-like training: A critical perspective, 2025. URL. https: /tt///aaiivv www.orrg/abb.. abs 2503. 207 83..
M. Luo. S. Tan, J. Wong, X. Shi, W. Y. Tang, M. Roongta, C. Cai. J. Luo. T. Zhang, L. E. Li, R. A. Popa, and Stoica. Deepscaler: Surpassing ol prepviw with a 1.5b model by scaling rl. https: /tts:///iiiyulllteee humm//e9rr3333 2025. Notion Blog.

---

Published as a conference paper at ICLR 2026
W. Meng. Q. Zheng, G. Pan, and Y. Yin. Off Off-policy proximal policy optimization. Proceedings of the AAAI Conference on Artificial Intelligence, 37(8):9162 9170, 2023.
Y. Mrouch. Reinforcement learning with verifiable rewards: Grpo's effective loss, dynamics, and 2503. 06639.
success amplification, 2025. URL https:
M. Noukhovitch, S. Huang, S. Xhonneux, A. Hosseini, R. Agarwal, and A. Courville. Faster, more efficient RLHF through off-policy asynchronous learning. In The Thirteenth International Conference Learning Reprexentations, 2025. URIhttps: WRR.httpp:///peeeee /ttp:////ppeeeeeeeeeeeeee net/ forum? id=FhTAG591Ve.
L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744, 2022.
A. Paszke, S. Gross, Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein,
L. Antiga, A. Desmaison, A. Köpf, E. Yang, Z. Vitoto M. Raison, A. Tejani, S. Chilamkurthy,
B. Steiner, L. Fang, J. Bai, and S. Chintala. PyTorch: An Imperative Style, High-Performance
Deep Learning Library, Dec. 2019.
J. Queeney, I. C. Paschalidis, and C. G. Cassandras. Generalized proximal policy optimization with sample reuse. In Advances in Neural Information Processing Systems, volume 34, 2021.
J. Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz. Trust region policy optimization. In International conference on machine learning, pages 1889-1897. PMLR, 2015.
J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: N.Kiv:1777.06333 707.0334777 2017.
Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song. X. Bi, H. Zhang, M. Zhang, Y. Li Y. Wu, et al. Deepseek- math: Pushing the limits of mathematical reasoning in open language models. arXiv preprint 2rXiv:2402.033000 2024.
N. Stiennon, L. Ouyang, J. Wu, D. Ziegler, R. Lowe, Voss, Radford, D. Amodei, and P. F. Christiano. Learning to summarize with human feedback. Advances in Neural Information Pro- cessing Systems, 33:3008-300 3021, 2020.
M. Toslali, Y. C. F. Lim, Q. Gallouédec, E. Snible, R. Ganti, and M. Srivatsa. No gpu left behind: Unlocking efficiency with co-located vllm in trl -, 06 2025. URL https: httttp::/hhuggingfac.. co/blog/vil wetps/llllmmmmmm wol.-ooooooocaccccooeeeee
mooep--ooo-----ooo - villv.m.... Blog post.
vLLM. Pytorch ci hud: vllm benchmark dashboard. https /hwd. http:://hh.ddo.e.eeeee pyt.cch..ccc.comm org/ ber.charr// 1ms Ims Nemoosareelallllll httpaaameewmmmmooooooo/eooeaaeee /v11m, 2025. Accessed: 2025-05-14.
M. Vojnovic and S.-Y. Yun. What the alignment objective of grpo?, 2025. URL https:// arxiv.org /25////2/////2222 /855./..5/.
L. von Werra, Y. Belkada, L. Tunstall, E. Beeching, Thrush, N. Lambert, S. Huang, K. Ra- sul, and Q Gallouédec. Trl: Transformer reinforcement learning. petps://githhie com/ huggingface/trrl. 2020a.
L. von Werra, Y. Belkada, L. Tunstall, E. Beeching, T. Thrush, N. Lambert, S. Huang, K. Ra- sul, and Q Gallouédec. Trl: Transformer reinforcement learning. https::///// /ttp::///ghhub.../ com/ huggingface/trl.. 2020b.
Z. Wang, V. Bapst, N. Heess, V. Mnih, R. Munos, K. Kavukcuoglu, and N. De Freitas. Sample 2016.
efficient actor-critic with experience replay. arXiv preprint
T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Fun- towicz, J. Davison, S. Shleifer, P. von Platen, C. Ma. Y. Jemite.. J. Plu, C. Xu, T. L. Scao, S. Gug- ger, M. Drame, Q. Lhoest, and A. M. Rush. HuggingFace's ' Transformers: State-of tfate-o--the-an the-art Natural Language Processing, July 2020.
12

---

Published as a conference paper at ICLR 2026
A. Yang, B. Yang, B. Hui, B. Zheng, B. Yu, C. Zhou, C. Li, C. Li, D. Liu, F. Huang. G. Dong, H. Wei, H. Lin, Tang, J. Wang, Yang, J. Tu, J. Zhang. J. Ma. J. Yang. J. Xu, Zhou, J. Bai, J. He, J. Lin, K. Dang, K. Lu, K. Chen, K. Yang, M. Li, M. Xue, N. Ni, P. Zhang, P. Wang, R. Peng, R. Men, R. Gao, R. Lin, S. Wang, S. Bai, S. Tan, T. Zhu, T. Li, T. Liu, W. Ge, X. Deng,
X. Zhou, X. Ren, X. Zhang, X. Wei, X. Ren, X. Liu, Y. Fan, Y. Yao, Y. Zhang. Y. Wan, Y. Chu,
Y. Liu, Z. Cui, Z. Zhang, Z. Guo, and Z. Fan. Qwen2 Technical Report, Sept. 2024.
Q. Yu, Z. Zhang, R. Zhu, Y. Yuan, X. Zuo. Y. Yue, T. Fan, G. Liu, L. Liu, X. Liu, H. Lin, Z. Lin, B. Ma, G. Sheng, Tong, C. Zhang, M. Zhang, W. Zhang, H. Zhu, Zhu, Chen, Chen,
C. Wang, H. Yu, W. Dai, Y. Song, X. Wei, H. Zhou, J. Liu, W.-Y. Ma, Y.QQ. Zhang, L. Yan, M. Qiao, Y. Wu, and M. Wang. Dapo: An open-source I1m reinforcement learning system at scale, 2025. URL https: / www.iii.og//eeeee
13

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY
We provide the code needed to reproduce all of our experiments in supplement.
BROADER IMPACT AND LIMITATIONS
Our work analyzes the celebrated GRPO algorithm and develops an adaptation for the off-policy setting motivated by recent efforts for PPO that demonstrated higher stability and efficiency. Our primary contributions are theoretical, providing formal conditions under which advantage optimiza- tion guarantees policy improvement for the on-policy and off policy regimes. These insights provide lower bounds on policy improvement and directly inform a practical clipped surrogate optimization objective for large language model (LLM) policy training that inherits our theoretical guarantees for both on policy and off policy regimes. gives theoretical backing to the benefits of masking samples with zero variance suggested in the DAPO paper (Yu et al., 2025). Our formulation also clarifies theoretical relationships between our newly introduced off-policy GRPO, PPO variants, and general off off-policy optimization frameworks a linkage previously underexplored validated experimentally demonstrating improved performance compared standard GRPO, while having the potential to reduce the communication overhead across devices in serving large models for sampling that is needed in GRPO. The broader impacts that we anticipate from our work (besides those directly inherited from GRPO and reinforcement fine-tuning of LLMs and the risks associated the dual use of the enabled reasoning models) are then generally positive, it enhances RL efficiency, reducing computational costs and improving stability.
the on-policy regime our lower bound sheds light and
the literature. Our derived off-policy GRPO algorithm is
A ASSETS
Hardware setup All our experiments were Xeon 8468, 2TB of RAM, 8 NVIDIA HGX H100 80GB SXM5, 8x 3.4TB Enterprise NVMe U.2 Gen4, and 10x NVIDIA Mellanox Infiniband Single port NDR adapters, running RedHat Enterprise Linux 9.5
compute node with Dual 48-core Intel
Libraries Our experiments rely on the open-source libraries PyTorch (Paszke et al., 2019) (li- cense: BSD), HuggingFace Transformers (Wolf al., 2020) (Apache 2.0 license), and Hugging ace TRL (von Werra et al., 2020a) (Apache 2.0 license). We also relied on Open- R1 (HuggingFace,, 2025a) well as LightEval (Habib al., 2023) for the evaluation of Aime24 and Math500.
Code re-use Our GRPO training code is based on the public Github repository https:/ www.huu..cc//uugggggggggeeeeeeee
(HuggingFace, 2025a).
Data and Models In our experiments, we use the following publicly available datasets: (1) GSM8K dataset from Cobbe et al. (2021) (MIT license), and (2) from Luo et al. (2025) (MIT license). The models -B-Instruct (Apache 2.0 license) by Yang et al.
DeepScale www.eee-Preeeee-attteee
that we used were Qwen/Qwen2 5--. (2024), and SeepSeee--R--isttee 11a1111---e-1-1- 5B (MIT license) by Guo et al. (2025).
B TERMINOLOGY USED IN THE PAPER
GSM8K: A standard grade-school mathematics reasoning benchmark consisting of short word problems with verifiable numeric answers.
DeepScaleR-Pres Preview-Datasedd A collection of math and reasoning datasets curated for verifiable reward modeling and fine-tune fine-tuning of cassoning-oriente LLMs.
AIME24: The 2024 American Invitational Mathematics Examination benchmark. used to assess advanced problem-solving and reasoning capabilities.
Extractive matching: Automatic evaluation that parses and matches numeric or symbolic answers from model outputs against ground-truth solutions.
14

---

Published as a conference paper at ICLR 2026
An open-source evaluation toolkit for LLMs supporting standardized bench- marking, prompt formatting, and metric computation.
LightEval:
Pass@1: The accuracy metric representing the fraction of problems solved correctly by the first generated sample.
C REWARD IMPROVEMENT LOWER BOUND
C.I PROOFS OF THEOREM
We have:
[T((-+)) = Ea
Let me be the current policy and a<+{æ) be another policy typically consider a(-|x)) TK-e(-1x)).
Define mean and variances of the off- off-policy reward, i.e policy under az
Marr (x) y) and (x) = v (eyyaa(( v) (or and denote for <
E 1' g = v (x) +
Note that we have a bounded reward 0 S r(x,8) S ||rl = 1 which implies that (x) S |1r and hence we have:
|||1
(.)
+E. 4
|| + 1
We normalize the reward so that (x)),
We denote GRPO advantage function
Aa(x,y) r(x,8) - Ma. (x)
Ja.ne(
[uyxz)
(x(T((l))) = Bywa(-ll) [xy(2r))
An (x.
a The we obtain the online policy objective function of GRPO, where the advantage is computed with the current policy i.e using An, (x, v).
We have:
Ca(T(:12)) (I) 1
(Eyun( v) (a,(
1
(
[(a(-1z))
(x)
(a)e(())
Our goal to provide an upper bound on:
(x((x=))) - (J(T(+22) - ((Fx(-1=)))
Hence we have:
(a(x(-1x)) - - -((x(())))((((( = ( (ane(()) 11 1 [(T(-12)) + [(=((112) (ans( J(((11))
J(a(- 11 -(al)))) +(a(11)))))) + [(Tx(-1)))
(a.e((z) [(((111))
(a.ee(x)
(a)ss(())
(J(T(11z) J(a(-I=))) + J(a({1)))) (a) J(a(-1z)) (a)
[a(((x)
(a))
(J(m(1z)) - J(o(+I=)) + (J(mx(+)))) - J(a(-1z)))
(x)
15

---

Published as a conference paper at ICLR 2026
Lemma 1 Seattroovic--Rubenseee duality of total variation distance). The Kantorovich-
Rubenstein duality (variational representation) of the total variation distance is as follows:
1 sup {Ez~m, 1m [g(Z)}}
TV m2)
(8)
2L gE9L
where GT = {g: - R, Il S L..
On the other hand using Lemma we have:
[(x(+[x) - J(a(-11)) ( 21 TV(T(12)). =
and
((mx(++))) J(a(-1<)) 2| (1(
By our assumption on the reward and & we have
Ca.ree
To.n(
that obtain the final bound follows:
(a(x(-1>)) - (J(+(++))) J(=+(+==)) S (ane(z)
|||l| TV(T(1))) + 21 (V)
Ozne(z)
We obtain finally lower bound on policy improvement follows:
J(T(-12)) - J(=x(=1=) La (T(++2)) - 1 (a,
|n|loo (V(((1))) - 2 |1ll (V(x)())).
(a))(())
Integrating over x (the prompts) we have:
(a)) TV(((1)).
Exppx [TT(-1))) Exxpx J (++((+1)))
Exwxx (a((((1)))) ||rll
Exuox
(ane(
- ExNpx (-(T a(+lx)
- / BBopx (1 (1-oan((z)) Exmpx (((()) 2||r|lx Ex-ox TV(
2 La - 1|
oann (x)
D EXPERIMENTS
16

---

Published as a conference paper at ICLR 2026
Math
per (with Variance)
Model
V20-11 100-11-tength-4099
vL-1I-leggth44066
Math 500 Extractive Match
0.86
0.82
0.80
0.78
teration
(a) Math500
Pass@1 over Iterations per Model (with Variance)
Model
v10-12 lengghh-ng6
ve-11/length-4066
AIME 24 Pass@
0.31
0.30
0.29
0.27
200
400
600
800
1000
1200
1400
Reration
(b) AIME24.
Figure 3: Aime 24/ Math 500
17


Reviews:
[{"review_id": "bOe9gJo2R0", "reviewer": "Reviewer_PeXH", "summary": "The paper revisits GRPO and develops a unified view of on-policy and off-policy variants by deriving a lower bound on reward improvement. From a KL-regularized objective, the authors obtain a clipped surrogate objective for GRPO and provide a theoretical rationale for masking zero-variance samples, which would otherwise worsen the bound constants. Building on this, they propose off-policy GRPO with two practical knobs: $v$ (server/model update frequency) and $i$ (SGD steps per batch). This configuration aims to reduce communication overhead in multi-GPU/tensor-parallel setups without sacrificing accuracy. Empirically, the paper reports that on-policy can be unstable on GSM8K, masking stabilizes training, and off-policy (e.g., $v=10, i=1$) attains similar or slightly better accuracy. On larger models (7B with TP), the off-policy setting yields a reported $\\sim$1.35× speedup per iteration.", "questions": "1. Can the authors add **DAPO/DR-GRPO/OPPO/ToPPO/DPO/IPO** results with identical rewarders, sampling temperature, context length, and compute budgets (ideally including a 7B+ model)?  \n2. Please report KL/TV drift curves across training for different $v$ and provide a recommended range for stable/efficient operation.  \n3. How is the zero-variance masking threshold chosen in practice? Can the authors provide a sample utilization vs. accuracy/variance?  \n4. Do the conclusions hold for code generation, safety alignment, or open-ended QA with graded or noisy reward signals?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- An explicit derivation of a reward-improvement lower bound for GRPO (on/off-policy) and a principled route from KL-constraint to a clipped surrogate objective.  \n- The $(i,v)$ design is simple and addresses a real bottleneck (communication/model updates) in multi-GPU/TP training.  \n- The analysis explains why zero-variance reward samples degrade constants and how masking can stabilize training in verifiable-reward tasks.  \n- Results on reasoning benchmarks show that the off-policy variant is competitive with the on-policy baseline while improving throughput.", "weaknesses": "- Most comparisons are within the GRPO family (on vs off; with/without masking). The paper should include controlled head-to-head comparisons with **DAPO / DR-GRPO / OPPO / ToPPO / DPO / IPO** under matched rewarders, sampling budgets, and compute. And the 7B setting reports system speed but lacks full accuracy curves and significance tests. Sensitivity to temperature, context length, $\\beta$, group size $G$, and $(i,v)$ needs more systematic coverage.  \n-  Evidence is concentrated on verifiable rewards. It remains unclear how the bound and masking behave with non-binary or hard-to-verify rewards.  \n-  The paper states that $v$ should not be too large, but lacks an empirical guideline (e.g., KL/TV drift vs. $v$ curves) or a recommended operating range.\n- Minor writing/formatting fixes for camera-ready: unify the epsilon symbol (use ε or define $\\epsilon$ and use it consistently), normalize dataset names across text/tables/figures (e.g., AIME24/Math500, no spaces around the slash), standardize the tool name to a single spelling (e.g., lighteval), correct a few misspellings/grammar (libraries, standardized, laid; learning rate $5×10^6$), and tighten math displays so TV is typeset as $TV^{2}$ and long fractions/clipping expressions aren’t broken across lines.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LtudfjFGAa", "reviewer": "Reviewer_E1KW", "summary": "This paper revisits **Group Relative Policy Optimization (GRPO)**, a reinforcement learning algorithm widely adopted for post-training large language models (LLMs) with verifiable rewards.\nWhile prior work (Shao et al., 2024) introduced GRPO as an on-policy variant of PPO using standardized rewards (thus avoiding a critic network), this paper extends it to the **off-policy regime**, theoretically analyzes **reward improvement guarantees**, and empirically validates the new formulation.\n\nThe authors:\n\n* Prove **policy improvement lower bounds** for both on-policy and off-policy GRPO (Theorem 1, Corollary 1).\n* Derive a **clipped surrogate objective** analogous to PPO but specialized for GRPO’s whitened rewards.\n* Provide a **masking strategy** for zero-variance samples (grounded theoretically).\n* Demonstrate experimentally that **off-policy GRPO** matches or exceeds the on-policy variant while reducing **communication costs** in distributed LLM training setups.\n  Experiments cover **GSM8K**, **DeepScaleR**, and scaling tests on **Qwen2.5-7B** models.", "questions": "1. How does the choice of *v* (off-policy lag) affect both convergence and stability? Are there theoretical or empirical bounds on how large *v* can be before degradation occurs?\n2. Could the policy improvement bounds be extended to **partially observable or multi-step MDPs** (where GRPO is applied iteratively on dialogue tasks)?\n3. Does masking zero-variance samples introduce bias in reward estimation over time?\n4. How does off-policy GRPO behave when the reward distribution is highly skewed or continuous rather than Bernoulli?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. **Theoretical Rigor and Novelty**\n\n   * Provides the **first formal policy improvement bound for GRPO**, including both on- and off-policy cases.\n   * The proofs elegantly extend beyond standard MDP-based analyses, avoiding dependence on state visitation distributions and instead exploiting GRPO’s analytical advantage form.\n   * The derivation of **clipped surrogate objectives** for off-policy GRPO generalizes the PPO framework in a principled way.\n\n2. **Practical Impact for LLM Training**\n\n   * Introduces a simple off-policy configuration (controlled by parameters *v* and *i*) that directly reduces communication overhead when serving models via vLLM.\n   * Empirically confirms the **stability and efficiency** of this approach, a highly relevant concern for multi-GPU and TP-based deployments.\n\n3. **Bridges Gaps Between RL Theory and LLM Post-Training**\n\n   * Clarifies the relationship between GRPO, PPO, DAPO, and off-policy PPO variants (G-PPO, ToPPO) within a unified analytical framework.\n   * Provides theoretical justification for “zero-variance masking” (previously used heuristically in DAPO) by linking it to the reward improvement bound.\n\n4. **Clear Experimental Verification**\n\n   * Ablation studies on GSM8K (Pass@1) demonstrate that both **masking** and **off-policy training** improve stability and accuracy.\n   * Finetuning experiments on DeepSeek-R1-Distill-Qwen 1.5B confirm the off-policy variant’s ability to maintain or slightly improve performance on **AIME24** and **Math500** benchmarks.\n   * Scaling experiments with **Qwen 2.5 7B** demonstrate ~1.35× speedup without loss in performance.\n\n5. **Reproducibility and Transparency**\n\n   * The codebase is based on public TRL and Open-R1 repositories, ensuring reproducibility.\n   * Detailed setup and hardware configurations are included in the appendix.", "weaknesses": "1. **Limited Experimental Diversity**\n\n   * Experiments focus primarily on math datasets (GSM8K, DeepScaleR). Broader testing on reasoning, code generation, or instruction-following datasets would strengthen generality.\n   * The number of random seeds (three) is relatively small, and there’s little reporting of variance beyond Pass@1 standard deviation.\n\n2. **Theory–Practice Gap**\n\n   * While the proofs are strong, some constants in the policy improvement bound (e.g., the variance-dependent term (1-σ_{α,r,ε}/σ_{α,r,ε})) lack empirical interpretation or sensitivity analysis.\n   * More intuition or visualization for these terms would help practitioners understand stability conditions.\n\n3. **Presentation and Readability**\n\n   * Dense notation (π, α, v, µ, ε, β, σ, Δ) makes the theoretical sections heavy.\n   * Some equations (especially Theorem 1 and derivations in Appendix B) could benefit from concise summaries or diagrams.\n   * Figures could include clearer captions and consistent scaling.\n\n4. **Minor Empirical Limitations**\n\n   * The speedup results (1.35×) are modest; while communication cost reductions are meaningful, scaling analysis beyond a single node would be more convincing.\n   * Comparison to other RLHF or off-policy RL methods (e.g., OPPO, ToPPO) is missing in quantitative form.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6Pi0PNZRot", "reviewer": "Reviewer_dUmt", "summary": "This paper proposes and analyzes an off-policy variant of GRPO (a recent policy gradient algorithm for training LLMs).\nIn particular, the authors identify two sources that render the policy gradient estimator off-policy: the optimization loop, which increases the discrepancy between the behavior policy and the target policy, and the use of past data. \n\nThe authors test the algorithm both by training from scratch and by finetuning an existing model (Deep Seek) on math problems, showing that the off-policy version does not lose performance relative to the on-policy version.", "questions": "I would ask the authors to clarify all the points above. Most importantly about:\n\n- design choices of the algorithms (outer loop and reference policy); multiple optimization steps even though the policy is not updated\n- lack of variability among seeds\n- why off-policy GRPO does not outperform the on-policy version\n- why the experiment Off-Policy GRPO uses S=1, while all the other experiments use S=3", "limitations": "", "rating": 2, "confidence": 3, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "The paper analyzes GRPO, a recent, simple, and efficient algorithm for training LLMs.\nThe authors, inspired by recent off-policy variants of PPO, propose an Off-Policy variant of GRPO. They also propose a policy improvement bound that covers both off-policy and on-policy development.", "weaknesses": "First, let me write here that I am not familiar with the LLM literature. However, I am very familiar with the reinforcement learning literature and understand GRPO and PPO well. I scored my own assessment of the paper with _confidence=3_ for this reason.\n\nClarity\n--------\n\nThe paper's presentation, in my opinion, is quite confusing. Let me state why:\n\n* As far as I understand, the central motivation of the paper revolves around the development of a decentralized update scheme of GRPO. This point remains unclear until page 6. \n* It is not clear to me why the authors use three nested loops for their policy update scheme. In particular, I am confused about a) the outer loop and the need for the \"reference policy\"; and b) the need to run the optimization (lines 15--17 of algorithm 1) several times even though the policy is not used to collect data. In particular, it appears that every optimization overrides the previous one, without having a real effect on the algorithm.\n* The authors call off-policy GRPO, GRPO with a number of optimization steps i>1. While I acknowledge that the gradient estimator becomes off-policy  after one policy update, the overall algorithm, according to the literature, is still considered on-policy (e.g., PPO is still considered on-policy, even if it applies several optimization steps on the same data)\n* The experimental section introduces acronyms and names like GSM8K, DeepScaleR-Preview-Datasets, Aime24, \"extractive matrching\", light-eval, and Pass@1 without explaining what they are. Perhaps they are clear for researchers from the LLM community, but they are not from me, and I needed to google them to understand the results. A bit of explanation in the paper wouldn't have hurt.\n* It is not clear why the experiment Off-Policy GRPO uses S=1, while all the other experiments use S=3. \n\nIn addition to the points made, the paper is really math-dense, making it hard to follow. \n\nResults\n----------\n\nGiven the paper's scope, it is entirely acceptable for the authors to use only three seeds for their LLM experiments. However, it would have been nice to see downscaled experiments with more seeds to see statistical significance, and hyperparameter sensitivity (i and \\nu). Furthermore, I am really suspicious about how the authors used the seeds: In Section 5.1, the authors report that the three seeds give such a small standard deviation (0.003--0.03) that it is not visible in the plots. I have never seen such a little deviation, and I can't understand how it is possible, since the learning curves are pretty noisy. It suggests that the policy always selects the same actions. I can't really figure out how this is possible (unless the policy is basically deterministic, which would make it very hard to train).\n\nMost importantly, it is not clear how the off-policy version does not really outperform the on-policy version. The off-policy version reuses data and should be more sample-efficient. GePPO shows a consistent improvement over \"on-policy\" PPO.\n\nSummary\n-------------\n\nThe paper does not seem ready for publication at this stage. The authors should consider writing the paper for a broader audience, clarifying details and results. In my view, the empirical section is also weak, given the low variability across results, the absence of small-scale ablation studies and hyperparameter sensitivity, and the fact that the proposed algorithms do not really improve on the on-policy version.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SJvh5BRxtS", "reviewer": "Reviewer_3Tjo", "summary": "The paper revisits Group Relative Policy Optimization (GRPO) and develops an off-policy variant alongside a theoretical analysis of policy improvement in both on- and off-policy regimes. The authors (i) prove lower bounds showing that maximizing a GRPO-style advantage (a whitened reward) improves expected reward, for both on-policy and off-policy cases under bounded rewards and distributional closeness; (ii) derive a clipped surrogate for off-policy GRPO with a KL regularizer; and (iii) empirically compare configurations that differ in sample reuse and serving/update frequency, reporting that off-policy GRPO is at least on par and sometimes better than on-policy GRPO, with potential serving-time savings.", "questions": "See Weaknesses", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.The paper extends GRPO analysis to an off-policy setting with an explicit policy-improvement lower bound, and derives an off-policy clipped surrogate analogous to PPO while exploiting GRPO’s normalized reward form.\n\n2.For practitioners training LLMs with GRPO, the paper offers a principled path to reduce serving/communication cost in colocated/tensor-parallel settings without hurting accuracy.", "weaknesses": "1.The primary practical claim hinges on setting v>1, and the paper uses v=10 in its experiments. This value seems arbitrary. The theory only vaguely suggests v should not be \"too large\". The paper would be much stronger with an ablation study analyzing the trade-off between v, wall-clock speedup, and final task performance. \n\n2.The improvement bound depends on total-variation terms and a variance factor (1−σ)/σ that explodes when reward variance is near zero (e.g., consistently correct/wrong prompts). The paper suggests masking such cases, but more discussion is needed on how often this occurs in practice for common verifiable rewards and what happens when masking is imperfect.\n\n3.Most results are on math datasets with small models (0.5B / 1.5B); evidence that the method scales or competes with strong recent baselines (e.g., DAPO/DR-GRPO, off-policy PPO variants) is thin.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)