GEPO: Group Expectation Policy Optimization for Stable Heterogeneous Reinforcement Learning
Authors: Han Zhang
RuibinZheng
ZEXUAN YI
Zhuo Zhang
Hanyang Peng
Hui Wang
Jiayin Qi
Binxing Fang
Ruifeng Xu
Yue Yu
Venue: ICLR 2026 Poster (2026)
Keywords: Mathematical Reasoning
Reinforcement Learning
Large Language Models
Decentralized Training
Heterogeneous Computing
Primary area: reinforcement learning

Abstract: As single-center computing approaches power constraints, decentralized training becomes essential. However, traditional Reinforcement Learning (RL) methods, crucial for enhancing large model post-training, cannot adapt to decentralized distributed training due to the tight coupling between parameter learning and rollout sampling. For this, we propose HeteroRL, a heterogeneous RL architecture that decouples these processes, enabling stable training across geographically distributed nodes connected via the Internet. The core component is Group Expectation Policy Optimization (GEPO), an asynchronous RL algorithm robust to latency caused by network delays or heterogeneity in computational resources. Our study reveals that high latency significantly increases KL divergence, leading to higher variance of importance weights and training instability. GEPO mitigates this issue by using group expectation weighting to exponentially reduce the variance of importance weights, with theoretical guarantees.  Experiments show GEPO achieves superior stability—only a 3\% performance drop from online to 1800s latency—and reduces the best-to-last gap by 85\% versus GSPO ($\Delta$=1.8 vs. 12.0) while attaining the highest scores, highlighting its effectiveness in decentralized, resource-heterogeneous environments.

Published as a conference paper at ICLR 2026
GEPO: GROUP EXPECTATION POLICY OPTIMIZA- TION FOR STABLE HETEROGENEOUS REINFORCE-
MENT LEARNING
Han Zhang' Ruibin Zheng2.1, Zexuan Yi', Zhuo Zhang*., Hanyang Peng", Hui Wang', Jiayin Q Binxing Fang", Ruifeng Xus.11 Yue Yu Pengcheng Laboratory, 2 Guangzhou University, 3 Harbin Institute of Technology (Shenzhen)
ABSTRACT
As single-center computing approaches power constraints, decentralized training becomes essential. However, traditional Reinforcement Learn- ing (RL) methods, crucial for enhancing large model post-training, cannot adapt to decentralized distributed training due to the tight coupling be- tween parameter learning and rollout sampling. For this, propose Het- eroRL, a heterogeneous RL architecture that decouples these processes, cnabling stable training across geographically distributed nodes connected via the Internet. The core component is Group Expectation Policy Opti- asynchronous RL algorithm robust. latency caused by network delays or helerogeneity in computational resources. Our study reveals that high latency significantly increases KL divergence, leading higher variance of importance weights and training instability. GEPO mit- igates this issue by using group expectation weighting to exponentially re- duce the variance of importance weights, with theoretical guarantees. Ex- periments show GEPO achieves superior stability-only 3% performance drop from ouline to 1800s latency and reduces the best-to-last gap by 85% versus GSPO (A-1.8 vs. 12.0) while attaining the highest scores, highlight- ing its effectiveness in decentralized, resource- hetorogennoos environments. Code available at hhttps://githei www.coo///oaa.a www/Hanlardeeseaalliaaa www.eeeeeeeeereeeeeeeettttooo RL
mization (GEPO),
GRPO (Taken Level
Latency RI)
Acs R(x.v)) mean (Re,nnR(r.)))...... ...., )) std( (.) )...., y))
0.8
0.7-
sols! x.3/.)) Ass, ciepiee
ma(sd x. to
Carro,
min
A
meuloi x.sla))
x,
0.6
CSPO
GIPPO
Bh-0
(PORPO
GEPO
GSPO (Sequnnc Level
å R(a,y") Ro,,)) ).... )3 std(R(a. ).... R(a,y")
200
400
800
500
100U
1200
Training steps
Coaro,
min
mely" A., alipsss
mola!!
0.8
Latency
Dval Accuracy
0.7-
R(x,y)) mesum ...., 1)
GEPO Lawe)
0.6
atd R(s.Ny) bo (.
0.5-
0EP0
Taly! x²) R., clipiss
naly mx)
Caxro, min (i (3)
0.1
200
400
800
800
1000
1200
Group Expectutivo
Training steps
Figure 1: Left: GEPO improves upon GRPO and GSPO by employing group-level impor- tance weights to enhance training stability. Right: In both zero-delay (online) and high-delay (up to 1800 seconds) heterogencous reinforcement learning scenarios, GEPO demonstrates superior stability and better evaluation performance.
Equal contribution. Email: zhangh040pcl. ac. cn and rrzzengge. .eaannggeegzzu.... www..gzheeeee. edu..eedd.cc cn
Ruifeng Xu and Yue Yu are Corresponding authors.

---

Published as a conference paper at ICLR 2026
1 INTRODUCTION
Training ever-larger AI models (Achiam et al., 2023; Dubey et al. 2024 Yang et al. 2025 is pushing the limits of single datacenters, making decent ralized Training across geographically distributed, heterogeneous nodes connected via the Internet an increasingly necessary pur-
suit (Team et al. 2025 Noukhovitch al., 2024]. Reinforcement Learning (RI.), crucial for
post-training LLMs is on complex tasks like mathematical reasoning (Shao et al. 2024h, faces
fundamental systemic challenge in this emerging paradigm: traditional RL frameworks [Guo et al. 2025: Stiennon et al. 2020 Bai et al. 2022, Wu et al. 2025 Dai ct al. 2024 Fu
ctt al. 2025 are architocturally incompatible with decentralized environments. Their tight
coupling between rollout sampling and parameter learning demands strict synchronization a requirement that becomes untenable under the high network latency and computational
heterogencity inherent in real-world distributed settings.
This architectural incompatibility manifests in two critical bottlenecks. First, synchronous frameworks force computational resources (e.g., GPUs) to idle while waiting for the slowest. processes such generating long reasoning chains severely constraining efficiency (Fu et al. 2025). Second, and more fundamenaally, the inevitable network latency inherent in decentralized, Internet-connected environments creates a temporal gap (policy staleness) between the sampler (generating data) and the learner (updating parameters). Most ex- isting RL algorithms, designed for homogeneous, low-latency clusters, are ill-equipped to handle this staleness. As our analysis reveals, high latency significantly inflates KL diver- gence, causing the variance of importance weights to explode ultimately leading to training instability reward collapse (Song et al. 2023). This renders conventional RL methods impractical for real-world, geographically distributed training scenarios.
To tackle these systemic bottlenccks, we introduce HeteroRL (Heterogencous Reinforce- ment Learning), novel RL framework explicitly architected for asynchronous, geographi- cally distributed, and resource- hetrrogeneero environments. HeteroRL is designed to enable elficient and stable training of large language models for complex tasks such as mathematical reasoning, even under high network latency. At its core, HeteroRL decouples the two com- putationally intensive phases of the RL pipeline rollout sampling and parameter learning and deploys them physically or logically independent nodes with potentially heteroge- neous hardware (e-g., mixing NVIDTA and Ascend chips). The sampler nodes continuously generate reasoning trajectories without interruption, while the learner node asynchronously consumes this data to update model parameters. Critically, neither component waits for the other: communication occurs infrequently and tolerates high latency, with model check- points and rollout batches exchanged over the Internet.
To address the instability arising from KL divergence under high-latency conditions, we introduce Group Expectation Policy Optimization (GEPO), novel policy gra- dient algorithm that stabilizes asynchronous RL under high latency by replacing fragile token/sann ssamppllellll importance weights with robust group-level importance weights al- lowing samplers and learners to operate independently, communicating infrequently and tolerating arbitrary delays. This shift fundamentally improves the quality gradient esti- mation- - transforming a high-variance, unstable estimator into low-variance, robust one, especially under large policy divergence. As we prove in Theorem ! GEPO exponentially reduces the variance of importance weights under high KL divergence where traditional methods like GRPO and GSPO collapse. Crucially, GEPO is not an ad hoc fix it is principled algorithmic response to the root cause of instability: variance explosion under policy divergence.
precisely the regime
In summary, our key contributions are as follows:
We propose HeteroRL, an asynchronous reinforcement. learning framework designed for beterogeneous compute networks, enabling decentralized training of large lan- guage models (LLMs) on mathematical reasoning tasks.
Framework:
Insight: We identify strong correlation between latency and the KL divergence between the rollout sampler and the learner. High latency induces high KL divergence, leading to training instability and reward collapse.
2

---

Published as a conference paper at ICLR 2026
Algorithm: We introduce Group Expectation Policy Optimization (GEPO). which im- proves upon the importance sampling mechanism in GRPO (Shao et al. 2024). We theo- retically show that GEPO exponentially reduces the variance of importance weights, and empirically demonstrate its superior stability and efficiency - not only under high-latency conditions, but also in the ideal zero-latency setting.
This work provides both algorithmic and system-level advancements for scalable LLM RL- training and establishes a practical foundation for large-scale distributed Al training in future heterogeneous compule network environments.
BACKGROUND
2.1 PROBLEM DEFINITION AND NOTATION
Consider a standard policy gradient framework. Let va denote the language model policy (i.e., the Actor) parameterized by 0, æ be an input prompt of a Dataset D (e.g., math problems), and be the output sequence generated by the model (e.g., a chain-of-thoughh solution). We define the following core notation:
Te (short for q): the policy used by the sampler at time step k to generate rollout trajectories.
Text- (short for p): the latest policy the learner at time step T, used for gradient updates.
T(2 0): policy staleness, representing the discrepancy in policy versions between the sampler and the learner, caused by network delays and computational asynchrony.
y: a trajectory sampled from the stale policy mo sf denotes the t-th token of the i-th response in a group.
r(x.y/ y): the reward for response y given input x.
A(x, y): the advantage for response y input æ, typically defined as A(x, y) r(z, y) - b(x), where b(x) is bascline reward computed for input æ. In this paper, we use the within-group average reward (Shao el al. 2024) as the baseline b(x)
à r(z,)) y²) where G is the group size.
The goal of HeteroRL to optimize the policy иy to maxinize the expected cumulative reward. To reduce gradient variance, an advantage function is used, leading to the objective:
To
(ylar)
[(0) E Ex
A(r,8)
no, x))
where 7 is a random variable. For online RL, T
3 GEPO: GROUP EXPECTATION POLICY OPTIMIZATION
Our method builds upon the group-based policy optimization paradigm of GRPO and intro- duces the group expectation importance sampling mechanism. We emphasize a paradigm shift from token-level to group-level importance weighting, which significantly reduces the variance of importance weights and alleviates gradient instability during training.
3.1 GROUP EXPECTATION IMPORTANCE WEIGHTING
To enhance the stability of importance weights, we propose the Group Expectation Importance Weight (GEIW), which replaces the individual proposal probability (yll²) in the standard importance weight p(yla) with its group-wise expected value under the currett
s(vi))
prompt x, denoted as [g(la=)))) Inspired by GRPO, for each input x, we generate group of G responses "..,...,'" ~ q(-1x) to form sampling group. Since G is typically much smaller than the full policy space and top-P P/top-t sampling leads to D ((y())) > 1,
3

---

Published as a conference paper at ICLR 2026
the vector (g(v' ........999()))))
( (x)) does not constitute valid probability distribution. Sim- ply using the arithmetic mean t Di= ((y"lz) would introduce bias due to ignoring the relative sampling probabilities. To obtain a more accurate estimate, we employ a weighted expoctation:
| E (vy1xx)
[x(((()) ~ Σ -
(2)
where ((y2)x)) gly' (a) D aiy' a) ical estimate of the sampling likelihood of each yi. We define the GEIW importance weight as:
is the within-group normalized probability, serving empir-
WGEIW (ylz) plylz)
(3)
E [x(((11
The key advantages of this mechanism are as follows:
The denominator is decoupled from any single ((ylz), avoiding extreme weight values when individual proposal probabilities ap- proach zero. Although clip(1#t) can also improve numerical stability, the gradients of the clipped tensors will be set to zero, effectively skipping this data point (ineffective gradient).
Numerically stable and gradient-effective:
Biased yet. low-variance: By leveraging within-group statistical information, GFIW pro- vides a more reliable scale estimate. Even under large divergence between p and q, E, [x)) remains well-conditioned, effectively preventing gradient explosion. Although this estimator introduces a small bias (WGEIW is a biased estimator), both thcoretical analysis (see Theo- rem [] and empirical results demonst. that it significantly reduces variance under high KT. divergence, yielding more stable gradient. directions and improved training convergence.
Variance Comparison
KL Divergence
Variance Reduction
KL vs Variance Reduction
Vorg oure van varCi
ZWar
25.0
(Sun Quar 2UR2
R200
12
52
12
02
0g
KL ODDuurgence
(a) Variance comparison of and al Bernoulli(b).
under Bernoulli distributions, where Bernoulli(a) and
Variance Camp
Divergence
Variance Reduction
KL vs Variance Reduction
Sunddrd Vargi
Dure f
FurA PURA 2012
20
Verige verii
133
05 Roo20
92
-14 11 2.
Divergence
(b) Variance comparison of and E under Gaussian distributions, where p N( 1) and
q ~ Nob,,) 1).
Figure 2: In Kigh-KL regions, Varl p(ylz) Var P(y(z)

---

Published as a conference paper at ICLR 2026
Thcorem 1. Let p.g be discrete probability distributions. Then there erists constant C such that:
p(y|z)
p(y(æ)
Var
Var
exp (DKL(())))) -.
(4)
Eylg(v|x)]]
In particular, when (KL(p)l)) > log C, it holds that Var
p(vlz) [((ale)
Var
ply [.(((rl)))
Theorem T shows that GEPO can exponentially reduce the variance importance weights, making it particularly well-suited for heterogeneous RL training under high KL divergence. The full mathematical proof is provided in Appendix A. As shown in Figure 2 visualize the relationship between KL divergence and importance weight variance when both p and 9 are Bernoulli or Gaussian distributions with varying paramcters. The results indicate that in the high-KL. regime, the group expectation approach significantly reduces the variance of importance weights, which benefits training stability under high network latency. Nevertheless, there exist regimes such as the green regions in the plots where our method incurs a slight increase in variance.
The difference across all GRPO-like algorithms lies in the computation of the importance weights, as detailed in Listing !
if self loss_type in ["grpo", "erpo","da" "drpo" "dr_grpo,, "]npo"]: # Token level
coef learner_ paenner_tokeepp_/_ sampler_token.
elif self loss_type
"gapo": Sequence level
coef learner_seq.p sampler_seq.P
5 elif self loss_ == "gepo": Group level
hat_q suaplerrsee...detee
/ (sampler_seq-p.sun () detach ())
coef_1 learner_seq.p (hat_q sampler_seq.pl sum
Listing 1: Coefficient computation for different policy optimization methods
3.2 GRADIENT COMPARISON ACROSS TOKENS
What does the GEPO update do? For mechanistic understanding of GEPO, it is useful to analyze the gradient of the loss function LGEPO. The equivalent gradient of each token in a group with respect to the parameters 0 of GRPO, GSPO and GEPO can be written as:
(0)
Pí.z (0)
(0)
(1)
pl.1(0)
pi.
91,1
91.2
Q
E,q
Egg
OC(0)
Ao
:
00
Pl.1 (0)
(0)
(0)
(0)
Pl.
@G.1
@G.T
4G
QG
Egg
E.g
GAPD
0SP0
GEPO (ourc)
(5)
where A E RGXT is token-level advantages matrix, O denotes Hadamard product, q.e qlyi x², v.c),, qi = q(8 x)), and E,9 Eylo() From the perspective of gradient stability, GSPO uses a shared denominator g(y" 2) for all tokens in sequence i, while GEPO further aggregates across the entire group by using a common denominator Rggg. This progression from token-level (GRPO) to soquence-level (GSPO) to group-level (GEPO) coefficients demonstrates that coarser importance-weight granularity significantly reduces gradient variance. Empirically, leveraging group-level statistics enhances robustness and stabilizes training, especially under high policy divergence.
EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Model, Dataset and Benchmarks We conduct reinforcement learning training and evaluation the Qwen3-1.7B/88 model. The models are trained by strong-to- weak dis- tillation YYang et al., 2025), but have not been tuned with any RL. We train the model 8,290 samples from the MATH level 3-5 dataset. (Zeg al. 2025) and evaluate it by
5

---

Published as a conference paper at ICLR 2026
Pipeline
Framework
Latency
Sync Model
rollout-6 6
Saaiin window
LogNornal
Expenential
sync
rollout-5
Train Node
Rollout Node
compler-4
rollout
Rollout Data
compler-
Pollout-3
Topology
sampler-2
rellout-2
sompler-1
rollout-3
91.ss
Nvidia
910us
Sync Delay
Ascend
Ascend
910u
Data Flow
Figure 3: The Overview of HeteroRL. By decoupling sampling and training, HeteroRL en- ables decentralized distributed RL training of LLMs across five compute nodes: one param- eter update node (learner) and four data generation nodes (sampler), forming star-shaped network topology. Network delays between the sampler and learner nodes are explicitly mod- eled and can be simulated using stochastic distributions such as the log- nor-nrmaal or Weibull distribution.
reporting average Pass over 8 sampled responses the MATH500 (Hendryeks et al. 2021 AMC23 (Li et al. 2024 AIME24 (AIME 2024), and AIME25 {AIME] 2025 benchmarks. To better evaluate the inherent stability of policy optimization algorithms, remove KL divergence constraints during training under online RL scenario, and use the same KT. coefficient under the heterogeneous RL scenario. We compare method against. baseline methods GRPO (Shao et al. 2024), GSPO (Zheng et al., 2025 BNPO (Xiao)
al. 2025 and Gr.G (Liu al. 2025 under both zero-delay and high-delay settings.
More experimental details can be found in Section B.I
Heterogeneous Computing Environment As shown in Figure we perform heteroge- neous training across five compute nodes: one learner node and four sampler nodes, forming star-shaped topology centered at the learner. During training, the sampler nodes gener- ate rollout data, which is transmitted over the network to the learner node in a streaming fashion. The learner updates the model parameters and periodically broadcasts the up- dated weights back to the sampler nodes. The learner processes incoming rollouts in the order they arrive, operating within fixed time window for data eligibility. Since data is transmitted in batch units-each containing text, generation probabilities, and rewards maximum delay of 1800 seconds sufficient for typical network conditions. Within this window, the iteration gap (in terms of gradient updates) between the learner and samplers remains within 64 steps.
4.2
MAIN EXPERIMENTAL RESULTS
Table 1: Performance comparison using Qwen3-1.7B/ under Online RL (4k limiation).
AMC2023
AIME2024
AIME2025
MATH500
Average
Method
1.7B
8B
1.7B
8B
1.7B
8B
1.7B
8B
1.7B
8B
Owen3-1.7/8B
44.6
70.6
10.9
32.4
14.0
26.1
72.4
87.1
35.5
54.1
Max Tolerable Delay
BNPO
78.8
27.7
44.1
29.3
83.7
48.6
Dr.CRPO
61.6
77.5
24.6
41.0
22.7
27.7
82.9
91.6
48.0
59.4
GRPO
60.9
81.3
30.9
42.6
24.2
31.3
83.7
92.0
49.9
61.8
GSPO
60.3
77.8
28.5
41.8
25.0
31.3
83.9
90.9
49.4
60.5
GEPO (ours)
62.2
85.6
31.6
44.1
25.8
37.5
84.7
92.6
51.1
65.0
In this section, we compare GEPO with baselines under online RL and Hetero RL settings. The experimental results in Tables and demonstrate that GEPO not only achieves

---

Published as a conference paper at ICLR 2026
superior performance but also exhibits exceptional stability across both online and Hetero RL settings. Below, we dissect these findings in depth.
Table 2: Performance comparison using Qwen3-8B under Hetero RI. (4k limiation).
AMC2023
AIME2024
AIME2025
MATH500
Average
Method
best
last
best
last
best
last
best
last
best
last
Qwen3-8B 8B
70.6
32.4
26.1
87.1
54.1
Max Tolerable Delay 64
BNPO
73.2
72.6
37.7
35.9
25.3
24.5
87.4
86.1
55.9
54.8
Dr.GRPO
73.1
72.9
38.1
37.3
24.8
22.3
88.1
86.7
56.0
54.8
GRPO
71.6
71.6
38.7
35.9
27.3
27.0
88.8
88.8
56.6
55.8
GSPO
76.2
60.0
16.4
28.9
27.3
81.9
46.4
GEPO (ours)
83.4
82.8
42.6
37.5
33.2
32.0
91.3
90.9
62.6
60.8
In the online RL setting (Table D, GEPO consistently outperforms all baselines across both model sizes and all benchmarks. On Qwen3-8B, it achieves an average score of 65.0, surpassing GRPO (61.8) and GSPO (60.5) by 3.2 and 4.1 points, respectively. The gain is 20% relative improvement).
most notable on AIME2025 (+6.2 points over ERPO/GSPOS Even on the 1.7B model, GEPO sets a new SOTA, exceeding the best baseline by 1.5 points in average, confirming that group-level importance weighting improves gradient quality even without asynchrony.
In the Hetero RL setting (Table 2,, GEPO's stability advantage becomes decisive. We find that ENPO and GR.G exhibit little performance diffcrence compared to vanilla GRPO, and both employ token-level importance weights; therefore, only analyze the performance differences among GRPO, GSPO, and GEPO. Both GEPO and GSPO improve over GRPO in best performance (+10.6% and +3.2%, respectively). However, GEPO further surpasses GSPO by 7.2% in accuracy while reducing its best-to-last degradation by 85% versus GSPO (A=1.8 vs. 12.0), achieving both higher performance and far greater stability. While GSPO's last scores collapse dramatically, GEPO maintains near-peak performance throughout training.
These results validate GEPO's core design: by replacing token sequence-level importance weights with group=level expectations, exponentially reduces importance weight variance under high KL divergence (Theorem D enabling stable, scalable decentralized RL. GEPO thus sets a new frontier both performance and stability across ideal and real-world dis- tributed settings. In the HeteroRT setting, the training process recorded in Figure shows
GIFD
Garo
OIFO jound
DEPO boun)
04
1/
17
400
1000
800
200
10.0
134
120 3.00
1000
(a) Variance of IW
(b) Gradient Norm
(c) Training Reward
Figure 4: Curves of importance sampling variance, training gradient norm, and train/eval reward under max delay 64. Compared to GRPO and GSPO, GEPO maintains more stable importance weight variance, resulting in less drastic gradient changes, more stable training, and decline in training reward.
that GRPO stably improves the reward at slower pace, while GSPO rapidly increases the reward the first 200 steps but becomes unstable between 500 and 700 steps. As seen in Figure 4a GEPO exhibits significantly lower variance in importance weights compared to

---

Published as a conference paper at ICLR 2026
GRPO and GSPO, which experience sharp spikes and fluctuations. These unstable weight variances lead to erratic gradient updates, as evidenced by the large oscillations gradient norm (Figure 4bb for GRPO and GSPO, especially during early and mid-training phases. In contrast, GEPO's gradient norms remain relatively smooth and bounded, contributing to stable learning progress. Consequently, the training reward curve (Figure 4ch shows sistent improvement for GEPO without any noticeable decline, whereas GRPO and GSPO exhibit periods of stagnation even degradation.
4.3 LATENCY ANALYSIS
275
M
18-
158
14-
25
20
30
20
30
1.0
355
30
(a) KL-divergence
(b) Variance IW
(c) Estimation error
Figure 5: KL KI-divergence, Variance of IW, and Estimation error all positively correlated with the number of delay steps.
Impact of Latency As shown in Figure 5 we analyze the changes KT. divergence between the trainer and sampler, variance of importance weights, and estimation error of the expected value of the advantage function (optimization objective) during heterogeneous RL training latency increases. We observe that latency leads to increased KL divergence (Figure 5aj, which in turn causes an increase in the variance importance weights (Figure 5b), ult ultimately resulting in increased estimation error of the expected advantage function (Figure 5ch. Since the optimization objective is to maximize the estimated expectation of the advantage function, large estimation errors will cause significant fluctuations in gradients, thereby affecting training stability and performance. To show that high latency harms training stability, we compare max delays of 8 and 64 steps. As Figure [ shows, with 64-step especallyy near step 800-the KL divergence spikes and evaluation accuracy drops sharply, confirming that latency induces instability. Although GEPO improves stability, it still sulfers performance dip around step 900, highlighting that heterogeneous RL under high latency remains challenging.
GEPO4MsDeey
GEP0-Ma02ea 94
030
0.5
031
1000
125
Trahl Strpps
Eig Stgpe
Tealiing
(a) Latency steps
(b) KL divergence
(c) Evaluation accuracy
Figure 6: Training processos under different latency conditions
Correlation and Causality. Figure 7 quantifies the pairwise correlations among KL- divergence, variance of importance weights, and estimation error of the expected advantage function. The correlation coefficients range from 0.76 to 0.96 (a 0.05), confirming a strong statistical association between these variables. This observation empirically supports our hypothesis (illustrated in Figure 5) that increased latency induces higher KL divergence, which in turn amplifies the variance importance weights and the estimation error, ulti- mately threatening training stability. However, correlation does not imply strict causation.

---

Published as a conference paper at ICLR 2026
While latency is significant contributing factor to KL divergence, it is not the sole de- terminant - the model's internal state and the statistical properties of the sampled data also play crucial roles. This explains the observed variance in training outcomes under identical latency: sometimes collapse occurs, sometimes not. Critically, what determines survival versus collapse is not latency itself, but the algorithm's capacity to mitigate the downstream instability caused by high KL divergence. As demonstrated in Figure | and Table 2 GEPO's group expectation mechanism effectively suppresses the explosion of im- portance weight variance even when KL divergence is high. This allows GEPO to maintain stable training and avoid collapse in many scenarios where bascline methods (like GRPO and GSPO) fail thereby establishing algorithmic robustness to policy divergence as core contribution of this work.
210
230
150
150-
Varience Impartancs
Varianca
(a) The correlation between im- portance weight variance and estimation error of Bp (adv(d y)] is 0.76.
(b) The correlation between im- portance weight variance and KL divergence is 0.96.
(c) The correlation between KL divergence and E, adv(x, y)] es- timation error is 0.78.
Figure 7: Correlation analysis (95% CT) of training delay steps, importance sampling variance, and estimation error of expected advantage function.
4.4
HYPERPARAMETER ANALYSIS
Group Saxe Senattadty Analysts
Eivily IL Depetatiatio Conticiit
Seastivity Network Labnary Dostibbtton
44.0
0,7
4
44
43.5
45.5
40
12
42
42
4.
25
Average Best
29.7
Avrenge
30.0
Average
Avrrage Last
Averuge Last
38
20
000530002
9.000
1000
38
Webbul
Legsermal Latency Distribution
Expaneatial
Group Store
Fei
Group size
(b) BKI.
(c) Latency distribution
Recstivity tn Terepencture
Sernecivity to Sumpling
Senstrogy top Sumpling
47.5
46
48
45.6
4
S00m
Soore
64
41.4
41.4
40.0
42
37.5
Janenge
Average Best Average
d1.4
Average Best Last
35.0
0.4
8.6
0.8
0:90
16
20
56
Temporature
top-P
top
(d) Temperature
(e) Top-p
(f) Top-k
Figure 8: Sensitivity analysis of key hyperparameters in the Hetero/Online RL setting. All plots show the A Average Best and Average Last performance across four benchmarks (AMC2023, AIME2024, AIME2025, MATH500).
We conduct comprehensive hyperparameter studies to analyze the sensitivity of GEPO to key hyperparameters under hetero RL and online RL setting, including group size G, sam- pling strategies (top-k, top-p, and temperature), the KL. regularization coefficient SKI, and
9

---

Published as a conference paper at ICLR 2026
the choice of latency simulation distribution (e.g., exponential, Weibull, or lognormal). All experiments in the hyperparameter analysis section are based on the Qwen3-1.7B model with maximum context length of 2,048 tokens. Figure g plots the best and last perfor- mance curves across six hyperparameters. More detailed experimental results are provided in Appendix [B.5
In summary, our comprehensive hyperparameter analysis reveals that group size 8 ensures optimal stability in heterogeneous RL settings with consistently high final performance (Figure Sap, while a KL regularization coefficient of θKL = 0.005 maximizes both peak and final scores (Figure Sb An excessively low KL coefficient reduces training stability, The Weibull latency distribution proves more challenging than lognormal or exponential alternatives (Figure [c). and temperature of 0.4 enhances stability by maintaining higher final performance (Figure [d).. For sampling strategies, top-p 1.0 and lop-k = 10 achieve
the best trade-off between peak and final results (Figures 8e Sip, confirming their effec- tiveness across diverse training scenarios. These findings validate our design choices in the main experiments and provide practical guidance for deploying GEPO in diverse distributed training environments.
RELATED WORK
Asynchronous Reinforcement Learning Framework. In recent. years, reinforcement learning has become central to post-training LLMs (Ziegler et al. 2019). Researchers have identified efficiency bottlenecks in traditional synchronous RL frameworks. Wu et al. (2025) first theoretically explored asynchronous RLHF, proposing to decouple generation and train- ing across GPU clusters, enabling scalable and efficient RL fine- aue-tuning of LLMs with prov- able speedup over synchronous baselines. To address practical efficiency, AREAL (Fu et al. 2025) fully decouples generation and training, using staleness thresholds and decoupled PPO objective (Hilton et al. 2022) to handle outdated samples. Prime Intellect (Team eet al.) 2025) offers a decentralized, asynchronous framework for community compute, ensur- ing trust via verifiable inference, stability via two-sided GRPO clipping, and controllable reasoning with length-awar rewards. Nevertheless, these approaches all assume centralized, low-latency infrastructure and support only limited degrees of asynchrony, making them not suited for distributed RL in high-latency, decentralized, and heterogencons compute cnvi- ronments.
Asynchronous Reinforcement Learning Algorithm. Major of asynchronous or off- policy RL. algorithms for LLMs post-training lypically adopt two primary strategies to ensure training stability: (1) Gradient Truncation, wherein updates are suppressed for tokens whose importance weight lie beyond a specified trust interval interval-Decoupled PPO (Hilton let al. 2022 exemplifies this approach; (2) Importance Weight Truncation, which tains contributions from all samples but mitigates variance by bounding or reshaping the importance brights-mehhods such as Truncated IS (Espcholt et al. 2018), CISPO (Chen ct al. 2025), and TOPR [Roux et al. 2025) fall into this category. Our mothod diverges Trom prior studies in its underlying perspective: by embracing the notion of intra-group ex- pectation, we broaden the tolerance for token importance weight ranges, thereby improving training stability.
CONCI.SSION
We propose HeteroRL, heterogeneous reinforcement learning framework designed for train- ing LLMs across geographically distributed and resource- heterogeneous nodes, paired with GEPO-a a novel policy optimization algorithm that stabilizes training under high latency. By decoupling rollout sampling from paramcter updates, HeteroRL. climinates synchroniza- tion bottlenecks inherent in traditional RL pipelines. GEPO addresses the explosion of vari- ance of importance weight caused by stale policies through group expectation importance weight, provably reducing variance exponentially, particularly under large KL divergence between the sampling and learning policies. This work establishes practical foundation for scalable, delay-tolerant decentralized RL, making it well-suited for real-world LLM post- training in heterogeneous, wide-area wetwork environments.
10

---

Published as a conference paper at ICLR 2026
AUTHOR CONTRIBUTIONS
The division of labor within the rescarch team is follows: Han Zhang was responsible for the design of the GEPO algorithm, preliminary implementation of the HeteroRL training framework, and technical report writing. Ruibin Zheng focused on optimizing the imple- mentation of the HeteroRL training framework, improving data transmission execution logic in simulated latency scenarios, luning model hyperparameterss and prompts, creating visual- izations, conducting model performance evaluations and participating in algorithm analysis. Zexuan Yi was responsible for implementing data transmission in real network environ- ments and engineering the model synchronization mechanism. Zhuo Zhang contributed to experimental analysis and experimental design.
ACKNOWLEDCMENTS
This work is supported by the Major Key Project of PCL (Grant No. PCL2025A09) and by the National Natural Science Foundation of China 62576120. We thank Xiang Li for verifying the thcoretical proofs in this paper, and Wei Li for providing technical support on hardware infrastructure and system maintenance.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXi preprint arXi::3333008774, 2023.
AIME. Aime 2024. 2024. Accessed: 2025-03-17.
AIME. Aime 2025. Accessed: 2025-03-17.
2025.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Consti- tutional ai: Harmlessness from ai feedback. ArXiv Preprint ArXiv:2212.088 080077, 2022.
Aili Chen, Aonian Li, Bangwei Gong, Binyang Jiang, Bo Fei, Bo Yang, Boji Shan, Changqing Yu, Chao Wang, Cheng Zhu, et al. Minimax-ml: Scaling test-time compute elficiently with lightning altention. arXio preprint ar arxi::2506. 13585, 2025.
Josef Dai, Xuchai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. Safe RLHF: Safe reinforcement learning from human feedback. In Proceedings of the Twelfth International Conference on Learning Representalions, 2024.
Muzhi Dai, Shixuan Liu, and Qingyi Si. Stable reinforcement learning for efficient reasoning, 2025. URL httpps///arx: org/ aww.abaaasss 18086
Abhimanyu Duboy, Abhinav Jauhri, Abhinav Pandcy, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama herd of models. ArXio Preprint ArXi::240722177 2024.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Volodymir Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning, Shane Legg, and Koray Kavukcuoglu. Impala: Scalable distributed deep-rl with importance weighted actor- learner architectures, 2018. URL pectps://arze
www./bgg////ooeooo/oo..oo.oeb/e abs/
Wei Fu, Jiaxuan Gao, Xujie Shen, Chen Zhu, Zhiyu Mei, Chuyi He, Shusheng Xu, Guo Wei, Jun Mei, Jiashu Wang, et. al. Areal: A large-scale asynchronous reinforcement learning system for language reasoning. arXiv preprint arXiv:2505.22 24298, 2025.
Daya Guo, Dejian Yang, Haowei Zhang, et al. Deepseek-rl incentivizes reasoning in Iims through reinforcement learning. Nature, 645(8081):633 6388, 2025. doi: 10.1038/ 5415866-055-09422 URL https://doo orw.oi...rg//...
11

---

Published as a conference paper at ICLR 2026
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint ar wrX:::2103.08874 2021.
Jacob Hilton, Karl Cobbe, and John Schulman. Batch size-invariance for policy optimiza- tion, 2022. URL https://arxiv. http:/awwvvv....rraaaaa/ http///b//.........
Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Huang, Kashif Rasul, Longhui Yu, Albert Q Jiang, Ziju Shen, et al. Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions. Hugging Face repository, 13(9):9, 2024.
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lec, and Min Lin. Understanding rl-zero-like training: A critical perspective. are preprint or arXiv:2503.2077 2025.
Michael Noukhovitch, Shengyi Huang, Sophie Xhonneux, Arian Hosscini, Rishabh Agar- wal, and Aaron Courville. Asynchronous rlhf: Faster and more efficient off-policy rl for language models. ar preprint or arXiv:2410. 10.1222, 2024.
Nicolas Le Roux, Mare G Bellemare, Jonathan Lebensold, Arnaud Bergeron, Joshua Greaves, Alex Fréchette, Carolyne Pelletier, Eric Thibodcau-Lanfer, Sándor Toth, and Sam Work. Tapered off-policy reinforce: Stable and efficient reinforcement learning for llms. arxi preprint Xrx::22003. 14286, 2025.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. ArXiv Preprint ArXiv:1707.0000 2017.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathe- matical reasoning in open language models. orXiv preprint ar Xiv: 002.2400,, 2024.
Vaishnavi Shrivastava, Ahmed Awadallah, Vidhisha Balachandran, Shivam Garg, Harkirat Behl, and Dimitris Papailiopoulos. Sample more to think less: Group filtered policy optimization for concise reasoning. arXiv preprint arxiv:2508 09726, 2025.
Ziang Song, Tianle Cai, Jason D. Lee, and Weijie J. Su. Reward collapse in aligning large language models, 2023. URL lhttps://aaree org/ abs/ 17608
Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feed- back. In Proceedings of the International Conference on Neural Information Processing Systems, pp. 3008 3021, 2020.
Prime Intellect Team, Sami Jaghouar, Justus Mattern, Jack Min Ong, Jannik Straube, Manveer Basra, Aaron Pazdera, Kushal Thaman, Matthew Di Ferrante, Felix Gabriel, et al. Intellect-2: reasoning model trained through globally decentralized reinforcement learning. ar Xiv preprint ar arxi::2505. 07291, 2025.
Bo Wu, Sid Wang. Yunhao Tang, Jia Ding, Eryk Helenowski, Liang Tan, Tengyu Xu, Tushar Gowda, Zhengxing Chen, Chen Zhu, et al. Llamarl: A distributed asynchronons reinforcement learning framework for efficient large-scale 1lm trainin. arXiv preprint wrXim:2505.24033 2025.
Changyi Xiao, Mengdi Zhang, and Yixin Cao. Bnpo: Beta normalization policy optimiza- tion. arXiv preprint ar Xtv:22500. 02864,, 2025.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao
12

---

Published as a conference paper at ICLR 2026
Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang. Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report. aixi preprint or 25xi::2055.000 2025.
Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild. ar preprint 2rXi:2503.18892, 2025.
Han Zhang, Yu Lei, Lin Gui, Min Yang, Yulan He, Hui Wang, and Ruifeng Xu. Cppo: Continual learning for reinforcement learning with human leedback. In The Twelfth In- ternational Conference Learning Representations, 2024.
Chujic Zheng, Shixuan Liu, Mingze Li, Xiong-Huu Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, et al. Group sequence policy optimization. arXiv preprint arXiv:2501.1807. 2025.
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Gcoffrey Irving. Fino-tuning language models from human prefer- ences. Xiv preprint Xiv: 1909. 08593, 2019.
13

---

Published as a conference paper at ICLR 2026
CONTENTS
Introduction
2
Background
2.1
Problem Definition and Notation
GEPO: Group Expectation Policy Optimization
3.1
Group Expectation Importance Weighting
3.2
Gradient Comparison Across Tokens
4
Experiments
4.1
Experimental Setup
4.2
Main Experimental Results
4.3
Latency Analysis
4.4
Hyperparameter Analysis
Related Work
10
6
Conclusion
10
A
Theoretical Extension
15
A.1 Theoretical Proof of Importance Sampling Variance
A.2
Bias and Variance Analysis
18
A.2.1 Bias Analysis
18
A.2.2 Futher Variance Analysis
20
B
Supplementary experiments: Part
22
B.1
Implementation Details
22
B.2
Baselines
................................
23
B.3
Supplementary Results of Main Experiments
23
B.4
Comparison of Think and Non-think Mode
23
B.5
Details of Hyperparameter Analysis
25
B.5.1
Heterogeneous RL Sensitivity Analysis
B.5.2
Online RL Sensitivity Analysis
Supplementary experiments: Part II
C.1
Comparison With Asynchronous RL Baselines
D From Leven to Group-Level Importance Weight
29
D.1 Ablation Study
30
E Network Latency
30
E.1 Simulated Network Latency Configuration
30
E.2 Real-World Network Scenarios
30
Engineering Optimization: Localized Reward Computation for Reduced
Communication
31
G The Comparison of Different Reinforcement Learning Paradigms
32
G.1 Core Challenge: Oll-Policy Learning and Importance Sampling
32
H Future Work
33
H.l Defensive Sampling and Smooth Denominator Mechanism
33
I Case Study
34
Large Language Model Usage
35
14

---

Published as a conference paper at ICLR 2026
A THEORETICAL EXTENSION
A.1 THEORETICAL PROOF OF IMPORTANCE SAMPLING VARIANCE
This appendix analyzes a newly proposed importance sampling weight Wnew (y|z) = p xr)
B
where Eglg] = (( and compares its variance with the standard importance sam- p(ylz)
pling weight Wata (u/a:) = ((12))
Problem Setting Tet the target distribution be p(u(=) and the proposal distribution be (lylz). We aim to estimate:
Epifi - / (x))p(l)xdd
(6)
Since direct sampling from p is difficult, we employ importance sampling by drawing samples from q.
Standard Importance Sampling
The standard weight is defined as:
p(yl=))
(7)
Its expectation under is:
p(ylz) (lylac)
plulz)
Ey
(vllx)dy
p(ulx)dy 1,
(8)
thus it is unbiased. Its variance is:
27 (2)
p 2
Var, (Wata) Byg
q
(9)
p(y [1)) 1x)²
dy 1.
a(v xx)
Denoted as:
Varsad /
p(ylz)) dy 1.
(10)
((ylar)
Group Expectation Importance Sampling The new weight is defined as:
Wnew Wnew(y/x) = p(y(xr)
where Eylgl = ((vlx))dy.
(11)
Eg[g]
Its expectation is:
[ynww]]] 1
Feyla ((yla))lll (p, g)
(12)
11al13
where (p.4) denotes the inner product p(yl=))(2)zz)dy Generally, (p.g) # 1|ql13, making this estimator biased. Its variance is:
(Wnew) = Eg [ (P(e)lz) (z)) 21 - Bg [gl
>
(P(lk)) 1² Ey [g]
(13)
(E, i0))2 ( ((tull))((ula))))))
(
((ole)alolaaaa)))
Denoted as:
1
(J
P(ole)ooouladd (f ((ole)a(ola))d)))
Varnew ( (U)(vl=))ddy)
(14)
15

---

Published as a conference paper at ICLR 2026
Variance Comparison
We compare:
p(ulæ)2 dy ((0/2z)
Varstd
(15)
1
((ooD)Doooaa (/ (Doootallol))e)))
Varnew
(S (dy)
A.1 VARIANCE COMPARISON IN DISCRETE SPACE
Since the action space of large models is discrele, this section discusses the variance difference
A = Varstd Varnew in discrete probability space. The integral expressions in the continuous
case naturally transition discrete summation forms:
Replace continuous integrals f'dy with discrete summations Σ
Replace probability density functions p(y(z), ((yla)) with probability masses pi, qii
Maintain the structural form of variance expressions unchanged.
Notation and Setting Let the sample space a finite set X m,,, where 2. Let p (P1..... Pr)) and (#1..... Qn) be two probability distributions satisfying:
P; 0, Ei=1 Pi =
q: < 0, Σ 9i =
Define the following four key quantities, corresponding to the integral terms in the contin- uous expressions:
(ʃ (cvlla)dy))
A
=
qi
1
(corresponding to
(16)
1=1
(corresponding to (Frisleatelaaja) (corresponding
B
Piqs
(17)
/
plullr) dy) <(y1x))
(18)
4
1
Σ
(corresponding to / p(ulo)/((llz))d)
(19)
t=1
Accordingly, the variance difference can be written as:
A 1 B A
(20)
A
Lemma (Range of Quantitics). Under the aboue setting, we have:
1
A
e jo, 1),
I E [1, oo),
1 e (0, 1].
Proof.
1. Range of A: By the power mean inequality, Σ 2 " and Σ of < 1, thus A E [1/n?, 1].
2. Range of B: Σ piq (0, 11, thus B E [0, 1].
3. Range of I; By the Sauchy-Schwarr inequality, 1, 2 1, and it can approach infinity.
4. Range of 12: Since 1e Pi, have T2 1, and T 0.
16

---

Published as a conference paper at ICLR 2026
Thcorem 1. Let p. q be discrete probability distributions. Then there exists a constant C such that:
Var p(y]2=)
Var
exp (DKL()llg)) C.
(21)
[gyxx)) x))
[g11(1/=)]
In particular, when (pll() log C, it holds that Var (
Var p(ul)) By (o(a(al)))
Proof. Step 1. From the fundamental inequality relationship between KL divergence and x² divergence (Pinsker's inequality):
DKL S log (1 + Dx²
(22)
where the chi-square divergence is defined
(x=(plla) = Σ
(p:
9:)
Σ
P
= 11 -111
(23)
q8
1m1
Substituting yields:
DKL. (pllg) S log(Li);
(24)
therefore:
2 exp (Pkl))
(25)
Step 2. From Lemma 1, we know that A, B, and 12 satisfy the following bounds:
1
Ae n 1].
(26)
B E [0, 1],
(27)
12 E (0, 1].
(28)
A I2
Consider the lower bound of the expression
To obtain its minimum value, we
A
take:
Minimum value of B: B 0
1 Minimum value A: A n
Maximum value of lz: l
Substituting yields:
B A - I
0 - 1/n² -1
-ni + +1.
(29)
1/n2
Step 3. Substituting inequalities (1) and (2) into the expression for A:
= 11+ A 12
A z exp (DKi(Plla)) (n² + 1).
(30)
When DKL(pllq) log(n2 1), we have:
exp (Pkl))l > n² +
(31)
thus:
A > 0,
(32)
ie., Varad Varnew-
Corollary 1. In discrete space, if DxL(pll4) > log(n2 + 1), then
Varstd Varnew:
(33)
i.c., the new estimator has smoller variance.
It should be noted that A only attains the value + when q follows uniform distribution. In practice, when large models generate responses, the distribution tends to be long-tailed, the value of A is much greater than 1 and the constant Creat & log(n" + 1). For example, we randomly generated 128 tokens using Qwen3-1.7B (n = 151936), and the standard variance and average of A 0.432 10.36
17

---

Published as a conference paper at ICLR 2026
A.2 BIAS AND VARIANCE ANALYSIS
A.2.1 BIAS ANALYSIS
The end-to-end bias of GEPO We compute the following bias of GEPO and GSPO:
((ylar)
Bias Bias(GEPP)) Ep (1, v)1
Rgg
A(x,
Edg
(34)
(((u(z)) Egq
A(x;, u))
1
p(u'(x)
By
~
G
Egg
Egg (p()(2)
Bias GBSPO) Ep [A(x, y)]
A(x; ui)
Lz(()z))
(35)
p(ular)
A(z, v)
1
p(u'
A(x, 8/)
By
T
g( [x))
End bloo
of
0.355
0.125
0.300
0.100
0.23
0.003
6000
0DD6000-00
10
50
0.094
0.306
0,000
8.000
0.012
Ladwnry Stap
D'Derrarncc
(a) Bias-GEPO vs Latency
(b) Bias- CEPO vs KL
End-to-end blss GSPO
End to and blae of GS
0.75
0.17
0.30
0.160
0.072
0.075
0.009
0.050
0.025
10
0.802
0.001
0.00s
0LECR
0.010
0.12
(c) Bias-GSPO vs Lateney
(d) Bias GSPO KL
Figure 9: The Bias of GEPO and GSPO. Over 512 global training steps, Bias(GEPP)) has mcan of 0.072 and variance of 0.001, while Bias(GSPO) has a mcan of 0.075 and variance of 0.001.
Here, the bias of GSPO serves baseline for assessing the relative magnitude of the bias in GEPO. The use of the symbol is due to our use of Monte Carlo estimation for this expected statistic. Figure plots the bias of GEPO and GSPO according latency steps and KI--divergence.
The theoretical bound of GEPO bias We derive upper bound for the bias in- troduced by GEPO's group expectation importance weighting, specifically the absolute dif- ference between the standard importance sampling (IS) estimator and GEPO's estimator in expectation, under the assumption that the advantage function A(x, y) zero-mean under the target policy p (i.e., E,(A] 0).
18

---

Published as a conference paper at ICLR 2026
Theorem 2 (Bias of GEPO). Let p(y|z) and g(y(ar) be discrete probability distributions, and let A(x. v) be bounded advantage function. The bias of GEPO can be bounded by llllaa
[p(u(=) Eglg
A(x, v)
||l|
Bias( GEPO) = Ep [A(x. - Ey
(36)
14l12"
where llple denoles the L -norm rulão of p and q. Talla
In LLM reinforcement learning, the ratio Lpla
llall which upper-bounds the bias introduced by GEPO's group expectation veighting-remane modest in practice. Although this ratio can theoretically reach vn when is one-hot and is uniform over n actions, real-world LLM policies are highly non-uniform due top-k/top-p sampling and sharp output distributions. Empirical measurements in the paper (e.g., Appendix A) show that l|qll2 & 0.432 for Qwen 1.7B, implying ||all 0.657, and p typically exhibits similar concentration. Consequently, lpll x in general, even under high policy divergence, ensuring that GEPO's bias remains lillaa
mild and controllable far from the catastrophic scale implied by worst-case theoretical bounds.
The Proof Precess
Step 1: Define Estimators We adopt the notation and meanings from Section 2 out loss of generality, we assume that 1.
With-
We define:
Standard IS estimator expectation:
((yl() [q(()>))
A(z, v)) = Σp(ulz) A(x,v) = Ep[A] 0.
P1 Egg
(37)
GEPO estimator expectation (using ④ = F = E, ((1l))2 11913):
p(yl2z) [g
#2 = E,
A(z, v) = 1
Σ p(yl))4(v1z) v)..
(38)
Thus, the bias is:
((ul))) A(z,))
Bias(GEPO) |11
p2|
=1p2|
(39)
1
Step 2: Apply Schwarz Inequality product by Cauchy-Schwarz:
We bound the absolute value of the inner
Schwarz:
Σ Mlll))((la)) A(Z, S Σ Emnlvla) A(x,
Σ a()l2))2
(40)
v
Since [A(x,y)] S 1, we have A(x, v)² S 1, so:
Ell()l A(x, Σ p()]x)) 1lel12.
(41)
The reason use < instead of < here is that the case "advantage IA(x, v)l for all possible LLM responses y" cannot occur.
Therefore:
Σ Enolx)a(lla) A(a, v) < lllll2 11
(42)
19

---

Published as a conference paper at ICLR 2026
Step 3: Combine with Denominator
Now plug into the expression for |
|pll2
||ple
|22 =
| Σ p(ul))a(((=)) A(x,v)
(43)
11
1
|1
Finally:
|pl|
Bias(
(44)
|0l12
This upper bound quantifies the bias of GEPO's estimator relative to the unbiased standard IS estimator. In large language models, both ||l|| and l|ll2 are typically the order of v0.4 0.63, their ratio remains close to 1 i-indicating mild, controllable bias.
A.2.2 FUTHER VARIANCE ANALYSIS
Variance Analysis of Advantage- Weighted Importance Sampling We propose the extended Theorem for Var
p(ulz)
A(x, 3) follows:
Theorem (Variance of GEPO). Let p(ylz) and g(ylar) be discrate probability distributions, and let A(I,y) be bounded advantage function. Assume that on the support of interest, IA(x, 1)1 2 Amin > O, and define Amax = max, |A(x, Then there exists a constant
A2
Cadv = |gg12
such that:
Var, A(x, p(ylx)]
p(u|=) [g(a(1(=]]
Var, A(x,
A exp (DR(((ll))
Cadv (45)
(y)0r)
Cady
In particular, when DRL(Plla) log A
it holds that
p(ul2)) ((((x)
p(ylx)
Varg A(r,y)
Var, A(x,y)
Ey (gy(((((x))
About Amin > 0: We first note that any sample y with A(x, = 0 contributes zero gradient to the policy update and requires no variance control, as it does not affect learning. Therefore, without loss of generality,
remove the training samples for which A(z, v) = 0. Our variance analysis focuses only on the support where A(zz y)] > 0. Since the advantage function is assumed bounded and not identically zero, to the subset where A(T, y) 2 A > 0 for some constant Amin- Within the Group Wise Advantage method like GRPO, Amiu 1/G, where G is the group size.
further restrict
The thcoretical proof as follows:
Let p(y|=), (v)la)) be discrete probability distributions and A(z, y) be bounded advantage function satisfying E,[A(a y)] = O. For notational simplicity, we omit the dependence on æ.
Step 1: Variance Expressions Following the notation in Appendix A, we define: Zi p(ylz)
A(z, y) <()()): Ze A(x,y) p(yl#) where ||g||3 = Eggal = q(ylx=)2 dy.
liol
The variances are:
(21) = Ey (Alo, p(ulz) 21 )lylx)) 9 - Ey A(x,8) v)- p(ulx)]
g(v1x))
(46)
= / A(x, y)² p(ul))) dy - (1 A(x, Al
a(vlzr)
Var, (Zz) = | [.y[A( [ 2)
(B, up(u)))))(())
(47)
AA Ale, (Alcswoollaaaolll))))
dy
| 2
20

---

Published as a conference paper at ICLR 2026
Step 2: Variance Difference
Define Aadv (arg(Z1) - (Zz):
/ A(x; A(x, p (((())
dy (/ A(x, ((xxw))(olo)))
Andv
1 / A(x, A (/
A(x, ((a.s)DDwwll)))ll)))))) (48)
Rearranging terms:
Aadv / A(x, y)² P(Ul2))2 p)2p())))² ((ylx)
dy
Alz, p(ol=)))) dy
19112
(
A(x: (a.w)l(ul))oo dy
2
(A
(49)
11912
B1
B2
For brevity, we write:
Andv J1 - J2 B1 + B2-
(50)
Step 3: Lower Bound for J1 Under the restriction that A(I, > A. 0, we have:
J1= f A(z, y) p(())z)3 ((g/r))
dy Alin /
p(ulx)2 ((ylx) dy Amain I1,
(51)
[dul)] dy as defined in Appendix A. where / p(yla)2
From the relationship between KL and x² divergences:
DKl(p)lg) S log(I1)
=
I 2 exp(Dka(xle))
(52)
and therefore:
J1 4 exp(DKi(pllg))
(53)
Step 4: Bounding the Remaining Terms For J2, since |A(x, v)1 S Amax (advantages bounded in practice; for GRPO, Amax < 1):
J2 1 p(ul))) dy T
/
A2 A
/
A2 A
p(ula))auulo)) dy
(54)
llal
where the second inequality follows from Lemma 1 on page 15 of our paper:
Σ
p. < 1. 1=1
For the bias terms Bj and B2, since Ep[A]] = 0, we have:
/ A(x, v)p()(=) dy 2
B1
0,
(55)
(/ A(x ( Alw.wwolllllallll)))) 2 0.
|B2
(56)
|Tal33
21

---

Published as a conference paper at ICLR 2026
Step 5: Combining Bounds
Combining all terms:
4 max - 0 0.
add J1 J2 B1 B2
Amia cxp(Dx<(Pll))
(57)
Note that B2 appears with positive sign in Aadv, which improves the lower bound. Since B2 non-negative and bounded, we can define a constant Cady that absorbs all bounded subtractions:
A
Amax
Cadv
(58)
In practice, as noted Appendix A, laa12 = E, se 0.432 for language model distributions and Amax max |A| < making Cady reasonable constant. Therefore:
Aadv 2 A (xp((kL((lll)) Cadv-
(599
This completes the proof. The result shows that GEPO's advantage advantage-weighteed importance sampling maintains the exponential variance reduction property of Theorem
B SUPPLEMENTARY EXPERIMENTS: PART I
B.1 IMPLEMENTATION DETAILS
All experiments use the D.wen3-1.7B/4 /BBB model with maximum input length of 768 and output length of 2048/4096 4096 tokens under both think and no-think mode, limited by com- putational constraints and low token efficiency (Shrivastava et al. (025) [reward/length) at full context. lengt Training follows a GRPO-like algorithm with a learning rate of 1 x 10-6, 3% linear warmup, per-device batch size 8, and gradient accumulation of 8, with gradient checkpointing enabled for memory efficiency. To model network latency in hetero- geneous environments, we introduce a log-normal delay simulator bounded between 60 and 1800 seconds (99.5% CT), with default delay at 60 seconds and policy staleness varied across 0-64 effective steps. For online training, KL divergence is not used, in order to better eval- uate the training stability of the algorithms. For heterogeneous setlings, CPPO-KL (Zhang et al. 2024 loss with coefficient 0.005 is applied. Another reason for using CPPO-KT. is memory efficiecc, as it does not require a separate reference model. Rollouts are gener- ated using vLLM with parallel responses per prompt, and each run lasts epochs, with metrics logged via Weights & Biases. Due to significant discrepancies often arising between the oogppobbaiiiitics computed by vLLM and those computed during the forward pass un- der PSDP-whiee can lead to training instability or divergence +
we only use vLLM for sampling. additional forward pass is then performed recompute the log-probabilities.
The system prompt used in the experiments is shown in Figure 10
[think mode] are a helpful Assistant, designed to provided well-reasoned and detailed responses. You FIRST think about the reasoning process as an internal monologue and then provide the user with the unswer. Please put your final answer within !boxed Also, indicate that it is the answer.
Ino-think model You helpful Al Assistant, designed provided well-reasoned and detailed responses. Please put your final answer within (boxed Also, indicate it answer.
Figure 10: System prompt of all trainings in our experiments.
1 This 2k/ 2k/tktooeen limit balances cost and efficiency. Longer outputs increase memory and train- ing time, making high-latency experiments impractical. Crucially, as shown in Figure 11 [[),,, "think mode" yields high "Overlength Ratio" most long outputs are truncated and wasted. Recent: studies (Shrivastava et al., 2025 Dai et al.) 2025) consistently show that reasoning does not require excessively long chains of thought; redundant thinking merely wastes resources. Thhus, 2048/ 4099 tokens ensure fair, stable, and manageable experiments.
2This blog post describes the issue of training instability (or training crashes) caused by v...M. httpa://fengyao.u notion. site/off-ppt -ffplcccr-l
22

---

Published as a conference paper at ICLR 2026
B.2 BASELINES
The baseline methods compared in our experiments are as follows:
On-Policy Baselines:
Group Relative Policy Optimization (GRPO) (hao et al. 2024 reinforcement learning algorithm that enhances mathematical reasoning in LLMs by estimating ad- vantages through group-relative reward normalization comparing responses within a group to the same query theerby eliminating the need for a separate value net- work and reducing memory overhead compared PPO.
Dr. GRPO (Liu et al. 2025 is debiased variant of GRPO that removes the per-response length normalization and per-question reward standard deviation nor- malization, thereby climinating optimization biases that artificially inflate response length and improving token efficiency while preserving reasoning performance.
Beta Normalization Policy Optimization (BNPO) (Xiao et al., 2025 is a novel rein- forcement learming algorithm that dynamically normalizes binary rewards using an adaptively parameterized Beta distribution reduce gradient variance and enhance training stability for large language models.
Group Sequence Policy Optimization (GSPO) [hhnng et al. 2025) a novel re- inforcement learning algorithm for large language models that defines importance ratios based on sequence likelihood and performs sequence-leve clipping and opti- mization, thereby achieving superior training stability, efficiency, and performance compared to token-level methods like GRPO.
B.3 SUPPLEMENTARY RESULTS OF MAIN EXPERIMENTS
As shown in Table g GEPO outperforms all baselines Jaselines-includingg GRPO and GSPO-in both best. and final performance across zero-delay and high-delay settings, demonstrating superior elfectiveness and training stability. critical observation from the results that, although GSPO's technical report claims improved stability, we only observe GSPO be more stable than GRPO under the Online RL setting.
Table 3: Performance of GEPO and bascline methods under Online RL and Hetero RL
scenarios (2k limiation).
AMC2023
ATME2024
ATME2025
MATH500
Average
Method
Best
Last
Best
Last
Best
Last
Best
Last
Best
Last
Qwen3-1.72
25.6
1.6
3.9
54.7
21.5
Max Tolerable Delay 0 (Online RL)
BNPO
34.3
0.0
18.4
0.0
19.1
0.0
78.7
0.0
42.6
0.0
Dr.RRPO
53.4
14.3
19.1
1.6
18.8
2.0
78.6
35.9
42.5
13.5
GRPO
56.3
23.4
20.7
0.4
19.9
2.3
79.8
49.7
44.2
19.0
GSPO
54.1
27.8
23.8
3.1
20.7
4.3
79.9
62.1
44.6
24.3
GEPO (ours)
56.9
56.9
21.9
16.4
20.3
80.4
78.1
44.9
41.4
Max Tolerable Delay 64
(Hetero RL)
BNPO
45.0
43.1
12.1
11.3
12.5
10.1
71.1
69.3
35.2
33.5
GR.GRPO
48.4
48.4
17.2
17.2
14.8
14.8
73.9
73.9
38.6
38.6
GRPO
46.6
46.6
19.1
14.5
14.8
14.8
74.9
74.9
38.9
37.7
GSPO
54.4
23.8
17.6
1.6
17.6
2.7
78.2
55.6
42.0
20.9
GEPO (ours)
53.8
53.8
21.9
21.9
18.8
18.8
79.6
79.6
43.5
43.5
B.4 COMPARISON THINK AND NON-THINK MODE
In mon-think mode, all methods struggle with exploration, shown by consistently lower last scores vs. best indicating late-stage instability.
Notably:
23

---

Published as a conference paper at ICLR 2026
GEPO (ours) dominates across all benchmarks and delays in both best and last, proving superior efficiency and stability - even without thinking steps.
At zero delay, GEPO beats top baseline (GSPO) by +3.7 (best) and +20.88 (last), resisting early collapse.
With dclay=64, basclines improve (e.g., Dr. Dr.GRPO: 0.0 33.4), showing slack aids stability. GEPO still leads (last 38.0 vs. 35.7).
Vanilla Qwen3-1.7B (best=33.2) is outperformed by all RL iertoodd-conffiiiin RL's value even without reasoning.
In short: while non-think mode limits reasoning, GEPO delivers unmatched stability and final performance, idcal for latency-sensitive or real-time scttings.
Table 4: Performance comparison using Qwen3-1.7B (non-think, 2k).
AMC2023
AIME2024
AIME2025
MATH500
A Avragee
Method
Best
Last
Best
Last
Best
Last
Best
Last
Best
Last
Qwen3-1.75
42.8
10.2
9.4
70.2
33.2
Max Tolerable Delay
BNPO
43.7
0.0
13.7
0.0
13.3
0.0
74.2
0.4
36.2
0.1
Dr. Dr.GRPO
45.0
0.0
14.
0.0
11.3
0.0
73.6
0.0
36.1
GRPO
50.0
28.8
16.4
7.8
13.7
7.2
77.5
59.7
39.1
25.9
GSPO
53.1
23.8
14.4
2.7
16.0
0.4
76.5
55.6
40.0
20.6
GEPO (ours)
55.0
52.5
22.3
22.3
18.4
13.7
79.2
77.1
43.7
41.4
Max Tolerable Delay 64
BNPO
42.1
28.6
11.1
4.7
10.2
6.4
68.3
36.1
32.9
19.0
Dr.GRPO
45.0
41.6
14.1
9.7
13.3
10.1
72.5
72.3
36.2
33.4
GRPO
46.3
46.3
14.5
14.5
13.6
10.1
72.6
71.7
36.8
35.7
GSPO
47.2
13.7
4.3
13.3
8.9
75.1
67.9
37.3
29.4
GEPO (ours)
52.5
52.5
14.5
10.2
14.5
12.1
77.4
77.1
39.7
38.0
A comparison of RL under think vs. non-think modes reveals key trade-offs in reasoning, stability, cost, and performance.
1) Performance and Stability (Delay = 0) GEPO in non-think achieves Best
43.7 and Last 41.4 near-perfect retention. In think mode, it reaches Best 44.9 (+1.2) but identical Last = 41.4 - no final gain. Baselines like GRPO collapse (44.2 + 19.0), showing thinking destabilizes training without proper control.
2) With Delay = 64 In think methods rebound sharply: Dr. GRPO (0.0 - 33.4),
GRPO (25.9 + 35.7) minimal slack prevents collapse. In think, only GEPO retains peak perfectly (Last Best 43.5); others degrade (e.g., GSPO: 41.9 - 20.9). = Thinking introduces instability unless explicitly regularized.
Think produces longer rollouts - deeper reasoning but with higher overlength ratios, risking wasted compute and divergence. non-think yields shorter, efficient trajectories with lower overlength yet matches or exceeds performance when paired with stable optimizers like GEPO.
3) Exploration vs. Efficiency (Fig. 11
Overall, think oflers marginal peak gains at the cost of instability and overhead. Non-think + GEPO is more stable, efficient, and often equally effective - ideal for real-time time or latency-sensitive deployment. Choose based delay tolerance, interpretability needs, and required training stability. Figure [I] compares rollout lengths GEPO under think non-think modes, with three subplots:
1) Average Length: The red curve (think) stays consistently above blue (non-think), showing longer rollouts likely due to added rcasoning or policy deliberation.
2) Terminated Length: think again produces significantly longer successful rollouts, suggesting it enables deeper, valid exploration.
24

---

Published as a conference paper at ICLR 2026
G2FO prodhiii
6270 (roneii)))
GEPO incedlink)
jain
1990
12:0
12:0
130
300
12
1000
12w
(a) A Average Length
(b) Terminated Length
(c) Overlength Ratio
Figure 11: Comparison of rollout (cliped by 2k tokens) lengths under think and non-think modes.
3) Overlength Ratio: think incurs a much higher overlength rate, especially early in training indicating that while it boosts exploration, it also risks inefficient, overly long sequences.
Takeaway: think extends useful exploration and improves completion quality, but at the cost of higher computational overhead. In practice, this trade-off can be managed via length limits or dynamic rollout control to preserve efficiency without sacrificing performance.
B.5 DETAILS OF HYPERPARAMETER ANALYSIS
B.5.1 HETEROGENEOUS RI. SENSITIVITY ANALYSIS
The default parameters for our main experiments are: group size 8, KL regularization coefficient (3Kl) 0.005, and lognormal delay distribution.
Table 5: Sensitivity to group size under Hetero RL setting (max delay 64 steps).
Group Size
AMC2023
AIME2024
AIME2025
MATH500
Average
Best
Last
Best
Last
Best
Last
Best:
Last
Best
Last
2
51.3
48.4
17.2
15.6
17.2
16.4
77.8
75.4
40.9
39.0
56.8
56.8
20.3
15.6
21.1
21.1
77.7
77.7
44.0
42.8
8
53.8
53.8
21.9
21.9
18.8
18.8
79.6
79.6
43.5
43.5
Group size analysis. Table shows the performance impact of varying group sizes. We observe that group size of achieves the highest best performance (44.0), while a group size of yields the best final performance (43.5), demonstrating superior training stability. This validates that larger group sizes provide better statistical regularization for the hetero-RI. process.
Table 6: Sensitivity to KL regularization coefficient under Hetero RL setting (max delay 64 steps).
AMC2023
AIME2024
AIME2025
MATH500
Average Best
3KL
Best
Last
Best
Last
Best
Last
Best
Last
Last
0.001
48.8
33.4
14.1
2.0
17.6
5.5
75.3
59.5
38.9
25.1
0.002
52.5
40.3
18.0
11.3
21.1
15.2
77.6
71.4
42.3
34.6
0.004
55.0
53.8
16.8
15.2
19.9
16.4
79.0
79.0
42.7
41.1
0.005
53.8
53.8
21.9
21.9
18.8
18.8
79.6
79.6
43.5
43.5
0.006
53.4
52.5
20.7
18.0
19.1
78.5
77.1
43.0
41.7
0.008
47.5
49.7
12.5
21.5
14.5
17.6
74.4
77.4
37.2
41.5
0.010
54.1
49.7
17.6
17.2
17.2
15.6
76.9
76.3
41.4
39.7
KL regularization analysis. Table demonstrates that the KL divergence coefficient is critical for stability in heterogeneous RT settings. With 3Ki 0.005, GEPO maintains
25

---

Published as a conference paper at ICLR 2026
perfect stability (no performance degradation from best to last). When 8KL is too small (0.001), the algorithm collapses dramatically (AIME2024 drops from 14.1 to 2.0). When 3kL is too large (0.010), the model becomes overly conservative, limiting its peak performance.
Sensitivity to network latency distribution (95% CI 60-1920s) under Hetero RL Table 7:
setting (max delay 64 steps).
AMC2023
AIME2024
AIME2025
MATH500
Average
Distribution
Best
Last
Best
Last
Best:
Last
Best
Last
Best
Last
Weibull
51.6
49.1
18.4
14.8
18.0
17.6
77.4
77.4
41.3
39.7
Lognormal
53.8
53.8
21.9
21.9
18.8
18.8
79.6
79.6
43.5
43.5
Exponential
53.4
47.8
19.9
17.2
16.8
15.2
79.5
79.4
42.4
39.9
Latency distribution analysis. Table compares different network latency simulation distributions. The lognormal distribution yields the best performance and stability. The Weibull and exponential distributions show slightly degraded performance, particularly on AIME2024 and AMC2023 benchmarks.
B.5.2 ONLINE RL SENSITIVITY ANALYSIS
The default parameters for our online RL experiments are: top-p=0.95, top-k=20, and temperature=066.
Table 8: Sensitivity to top-p sampling under Online RL setting (topK=20, temp-0.6).
AMC2023
AIME2024
ATME2025
MATH500
Average
top-p
Best
Last
Best
Last
Best
Last
Best
Last
Best
Last
0.95
56.9
56.9
21.9
16.4
20.3
14.1
80.4
78.1
44.9
41.4
0.98
55.9
55.9
23.8
23.8
19.5
17.6
80.3
80.3
44.9
44.4
1.00
59.7
59.7
22.3
22.3
20.7
17.6
82.0
82.0
46.2
45.4
Top-p sampling analysis. Table g reveals that top-p=1.00 achieves the highest best performance (46.2) and final performance (45.4). It suggests that total probability provides better balance between exploration and stability.
Table 9: Sensitivity to temperature under Online RL setting (1opP=0.95, topK=20).
AMC2023
AIME2024
AIME2025
MATH500
A Aveaage
Temp
Best
Last
Best
Last
Best
Last
Best
Last
Best
Last
0.4
55.9
53.4
24.6
21.9
20.3
19.5
80.9
80.3
45.4
43.8
0.6
56.9
56.9
21.9
16.4
20.3
14.1
80.4
78.1
44.9
41.4
0.8
53.4
52.5
22.3
7.8
21.1
15.6
80.6
73.5
37.4
Temperature analysis. Table shows that lower temperature (0.4) sharpens the sam-
pling distribution, favoring high-confidence samples and reducing noise. This yields bet- ter final performance (43.8 41.4), demonstrating improved convergence stability. A higher temperature (0.8) increases exploration but compromises stability, particularly on ATME2024 (drops from 22.3 to 7.8).
26

---

Published as a conference paper at ICLR 2026
Table 10: Sensitivity to top-k sampling under Online RL setting (topP=-0.95, temp=0.6).
AMC2023
AIME2024
AIME2025
MATH500
Average
top-k
Best
Last
Best
Last
Best
Last
Best
Last
Best
Last
10
58.1
58.1
22.3
18.4
21.1
15.6
80.9
78.9
45.6
42.8
20
56.9
56.9
21.9
16.4
20.3
14.1
80.4
78.1
44.9
41.4
50
55.6
55.6
20.3
18.4
18.4
18.4
80.6
80.6
43.7
43.2
demonstrates that GEPO is relatively insensitive to the top-k parameter. While top-k=20 achieves the highest best performance (44.9), top-k=50 yields the best final performance (43.2), suggesting that a larger candidate set enhances training stability without significantly compromising peak performance.
Top-k sampling analysis. Table 10
In summary, comprehensive hyperparameter analysis reveals that:
Group size is crucial for stability in heterogeneous RL settings, with size providing optimal stability.
2. KL regularization coefficient (3x1=0.0055) is critical for maintaining stability under policy divergence.
3. The Weibull distribution is more challenging compared to other distributions.
4. Lower temperature (0.4) improves stability in online RL by reducing sampling noise.
5. Top-p and top-k have a moderate impact on performance, with top-p=1.0 and top- 1= providing good balance.
These findings validate our design choices in the main experiments and provide practical guidance for deploying GEPO in diverse distributed training environments.
C SUPPLEMENTARY EXPERIMENTS: PART II
C.1 COMPARISON WITH ASYNCHRONOUS RL BASELINES
In this section, we compare GEPO with asynchronous RL baselines, including Truncated IS (IMPALA) (Espeholt et al. 2018). TOPR (Roux al., 2025) and CISPO (Chen 2025). The mathematical distinctions among these three methods are shown in Table
Table 11: Objective of Asynchronous RL Baselines, sg denotes the stop gradient operation.
Method
Objective
Truncated
(T8((Ta) = Eroxald
sg (ata(F))
) R(T) log To())
1 is
CISPO
CISPO (TO) Erwnaadd
sg
ma(T) (F)a(F)
1- low
R(()) log To (T)
TOPR
TOPPR (mO) Brazal
((ETT) + ((rer- sg ma(r) )²) RR(T) log TT)
Off-Policy Basclines:
Truncated IS IMPALA (V-trace) (Espcholt et al. 2018 relies on temporal- difference learning with step-level rewards, whereas LLM reasoning tasks typ- ically employ policy-gradient-bars mards-causing V-trace to degenerate and rendering IMPALA an unsuitable direct baseline for comparison. However, Truncated IS is the key technique of IMPALA, we direcrly compare this technique with GEPO.
optimization with sparse, episode-level re-
TOPR (Roux et al. 2025 is stable and efficient off-policy reinforcement learning algorithm for fine-tuning large language models that uses asymmetric importance sampling-applying truncation to positive examples to accelerate learning and lower-truncating negative examples at zero to ensure stability without requiring KL regularization.
27

---

Published as a conference paper at ICLR 2026
CISPO (Chen et al. 2025 is stable and efficient off-policy reinforcement learning algorithm for large language models that clips the importance sampling weights rether than token-level policy updates to preserve gradient contributions from all tokens while reducing training variance and avoiding premature entropy collapse.
We adopt the experimental settings of CISPO and TOPR as described in their original papers, both which claim that an explicit KL penalty is unnecessary for achieving stable training. However, in our empirical evaluation, we find that both methods exhibit significant fragility under heterogeneous and high-latency conditions. To ensure a fair and robust. comparison, we therefore introduce augmented variants CISPO+KL and TOPR+KL as additional baselines, where a KL regularization term is incorporated.
Table 12: Performance of GEPO and asynchronous baselines under Hetero RL. scenarios (2k limiation).
AMC2023
AIME2024
AIME2025
MATH500
Average
Method
Best
Last
Best
Last
Best
Last
Best
Last
Best
Last
Qwen3-1.7B
1.6
3.9
54.7
21.5
Max Tolerable Delay 64 (Hetero RL)
Truncated IS + KL
47.5
0.0
15.2
0.0
15.6
0.0
74.0
0.0
38.1
0.0
TOPR w/o KT.
51.3
0.0
17.2
0.0
15.6
0.0
76.4
0.0
40.1
0.0
TOPR KL
51.9
30.0
17.2
1.2
17.2
7.0
76.9
55.1
40.8
23.3
CISPO w/o KT.
44.1
20.0
10.2
0.8
12.1
0.8
70.7
43.4
34.3
16.3
CISPO + KL
44.1
21.2
9.8
2.3
12.5
2.0
71.8
41.9
34.5
16.9
GEPO (ours)
53.8
53.8
21.9
21.9
18.8
18.8
79.6
79.6
43.5
43.5
0.5
0.15
TOPR Truncated
CISPO
GEPO
0.66
0.1
0.50
Training
Training
(a) HeteroRL/ AMC2023
(b) HeteroRL/ AIME24
0.7
0,150
0.185
8,100
GEPO
TOPR
8.685
CISPO
Truncated
8.500
0.8
GEPO
200
203
1000
203
100
1003
Training Staps
Training Stepe
(c) HleteroRL/AIME2
(d) HeteroRL/MATHHSS
Figure 12: Eval accuracy curves GEPO and asynchronous baselines (BKL. 0.005).
Table 12 presents the best and final performance of GEPO alongside various asynchronous baselines, while Figure 12 shows their evaluation curves across the four benchmarks. The
28

---

Published as a conference paper at ICLR 2026
asynchronous baselines exhibit severe instability or even collapse-untee under high-latency con- ditions, indicating that merely clipping the importance weights is insufficient to ensure training stability.
D FROM TOKEN-LEVEL TO GROUP-LEVEL IMPORIANCE WEIGHT
Token-Level Importance Weight In traditional policy optimization methods, such as GRPO (Shao et al. 2024 or PPO (Schulman et al. 2017]. importance sampling is typically
performed the token-level. Specifically, for generated sequence y (y),- - ..r. the
importance weight is computed token by token:
plyn J. (,Uee) 1 1-,,1+e))
Wtoken = clip
(60)
((e [,,+(e)
and used to compute the policy gradient at cach time step. However, this per-token reweight- ing scheme suffers from two key limitations:
1) Inconsistency between optimization and reward granularity: Concurrent work such as GSPO (Zheng et al.] 2025) argues that since the reward is assigned at the sequence level-ie., ie., to the full response y the importance weighting for policy updates should also operate at the same granularity, rather than at the token level.
2) High variance in token-level probabilities: Beyond this alignment issue, provide an explanation from the perspective importance weight variance: because the final reward depends the entire sample, large local changes in token probabilities can lead to extreme values in the ratio plz. n.yee). This inflates the variance of token-level importance weights, q(ve ((.e))))))))))
causing them to frequently fall outside the clipping range. Once clipped, gradients for these tokens are effectively zeroed out due to the stop-gradient behavior of torch. clamp, preventing meaningful updates. As a result, tokens that require large corrections may be ignored, leading to inefficient optimization.
Sample-Level Importance Weight The sample-level importance weighting treats the entire response as sampling unit and computing the weight based its full conditional probability. Given a prompt I, both the larget policy p(ylz) and the behavior policy ((y(r) are defined the joint probability of the sequence:
T
T
p()x) = II p(ue x, Vea)
(v(D) = II d(v x, weer)
(61)
The sample-level importance weight is then:
p(ulr) ())
(1equ))ce(y(a)) clip
e, e,
(62)
This formulation computes the full leveen ratio before clipping, thereby avoiding premature truncation caused by high-variance individual tokens. By preserving gradient flow across the entire sequence, sample-level weighting enables more stable and effective policy updates, especially under high policy divergence induced by network latency.
Group-Level Importance Weight Group-level importance weighting represents paradigm shift in reinforcement learning optimization by recognizing that policy updates should not only align with the granularity of reward assignment but also leverage higher- order statistical relationships among multiple samples. The key insight is that individual samples should not be treated in isolation; instead, their collective behavior under the same prompt provides crucial information for stable policy updates. By considering the ex- pected value of proposal probabilities within a group of responses, we effectively smooth out the erratic fluctuations that plague token-level and even sequence-level weighting schemes. This approach acknowledges an important reality of distributed training: in heterogeneous enviromments with network latency, policy divergence is inevitable, and our optimization methods must be designed to gracefully handle- not merely tolerate- this divergence. The group-level perspective transforms what previously seen limitation (policy staleness due to latency) into an opportunity for more robust learning through statistical regulariza- tion of importance weights.
29

---

Published as a conference paper at ICLR 2026
D.1 ABLATION STUDY
Since the removal of the variance divisor term in the advantage function has already been extensively validated in prior work (Liu et al. 2025), we focus solely on ablating the Group Expectation component. Table 13 compares three importance weights (IW) in Listing L namely token-level (IW of GRPO), sequence-level (IW of GSPO), and group-level (IW of
GEPO), under max tolerable delay of 64 steps. Table 13 presents the results, the sequence- level TW proposed by GSPO does not bring significant stability improvement. Although it outperforms token-level weighting in terms of best performance, it suffers from severe performance degradation by the end of training. We also conducted an ablation study on advantage normalization, and the results show that omitting normalization yields slightly better performance.
Table 13: Ablation study of different importance weights (Hetero RL mode).
AMC2023
AIME2024
AIME2025
MATH500
Average
Ablation
Best
Last
Best
Last
Best
Last
Best
Last
Best
Last
group-lv
53.8
53.8
21.9
21.9
18.8
18.8
79.6
79.6
43.5
43.5
loken-ly
46.1
43.9
18.7
14.2
15.3
14.3
74.9
38.6
36.8
seq-ly
55.2
24.1
17.2
1.9
18.1
2.1
77.3
56.9
42.0
21.3
w/o norm
53.8
53.8
21.9
21.9
18.8
18.8
79.6
79.6
43.5
43.5
wt adv norm
54.7
49.7
21.5
21.5
18.0
17.6
77.8
77.4
43.0
41.5
E NETWORK LATENCY
We test the performance limits of GEPO and baseline methods in simulated network delay scenarios and train the model on heterogeneous computing resources connected through real-world internet connection.
E.1 SIMULATED NETWORK LATENCY CONFIGURATION
Our goal is to simulate RL training of large models over internet-connected heterogeneous compute clusters, where network delays are inherently uncertain. To evaluate algorithmic performance and training stability under extreme and variable latency conditions, employ three widely used delay distributions Pa: log-normal, Weibull, and exponential. We set a high maximum delay threshold of 1800 seconds, which is sufficient cover typical model and data transmission times. In our simulation, model parameter synchronization and rollout data transfer are implemented follows:
Learner saves model: The learner periodically saves model checkpoints to shared model file path Model Sydel_Sync_Path via torch.
torch. waveaaavveeeeeeeeeeee save.pretrained).
Sampler loads model: The sampler generates data using the current model until a delay DM, drawn from Ps, elapses. It then loads the latest model from Model Sync Path. The sampler remains active throughout- no idling occurs.
Sampler sends data: Generated rollout batches are saved Rollout Sync Path with timestamp Tynee- To simplify implementation, data transmission is assumed instantancous; its latency effectively merged into the model sync delay DM, with- out affecting simulation validity.
The learner trains on rollout data from Rollout Sync Path that falls within recent time window (e.g., no older than 1800 seconds).
Learner uses data:
E.2 REAL-WORLD NETWORK SCENARIOS
To evaluate the performance of heterogeneous reinforcement learning algorithms in realistic network environments, develop a communication toolkit based ZeroMQ, supporting TCP/IP-based transmission of inference trajectories from samplers to learners and synchro- nized model parameter updates. As shown in Figure [3| the toolkit enables multi-node communication over wide-area networks (WANs), with the following core communication logic:
30

---

Published as a conference paper at ICLR 2026
Learner-1
Sampler-1
Bidirectional
Bidirectional Endipoint
Endpoint
Training Init
rollout-1 & send
Rollout Queue
request
Campliee Loop-1
response
request
response
rollout-2 & send
Learner Loop-1
Load Rollout
Rollout Sync
Get Rewards
Model Sync
Checkpoint
update model
load mode checkpoint
send model
request
response
Learner Loop-2
Sampler Loop-2
Figure 13: Communccttion flow between a learner and sampler in a distributed reinforce- ment lcarning system, showing trajectory collection and model synchronization bidi- rectional endpoint.
The learner continuously listens for and buffers inference trajectory messages from samplers. It automatically updates its trajectory buffer upon message arrival and broadcasts the latest model parameters to all connected samplers once predefined synchronization interval is reached.
After generating inference trajectory, the sampler sends it to the learner and continuously listens for paramcter update messages. Upon recciving new model parameters, the sampler updates its local paramcters and resumes sampling before the next sampling step.
F ENGINEERING OPTIMIZATION: LOCALIZED REWARD COMPUTATION FOR REDUCED COMMUNICATION
In large-scale heterogeneous reinforcement learning (HeteroRL), where sampler and learner nodes are geographically distributed, network communication overhead becomes critical bottleneck. major source of this overhead lics the reward aggregation phase, where traditional implementations require an all gather operation across all processes to collect rewards for group-wise normalization (e.g., computing mean and standard deviation per group of generations). This global synchronization roduces significant latency, especially under high network delay or when using large numbers of distributed nodes. To address this, we introduce key engineering optimization: Localized Reward Computation. Instead of gathering rewards from all processes, we ensure that each group of generations (e.g., G responses per prompt) is entirely generated and scored within the same process or node. This design guarantees that all samples belonging to single group reside locally, enabling the calculation of group statistics (mean, std) without any cross-process communication.
This imizaaiti is particularly synergistic with GEPO's design philosophy. GEPO miti- gates training instability under high latency by reducing the variance of importance weights through group-level expectation. Our engineering improvement complements this by ducing the system's 's sensitivity to communication latency itself, creating a virtuous cycle: the algorithm is robust to policy staleness, and the system is optimized to minimize the communication that causes staleness.
The core change is implemented by removing the global gather operation in the reward calculation function. This seemingly minor change yields substantial performance benefits. The following table summarizes the key improvements:
31

---

Published as a conference paper at ICLR 2026
Rollout Batch
Global Batch (Gradient Accumulatiion
3750
3733
1%0
9200
070
0908
kmm
0000
Native
After Rovision
Rollust Batch
Global Batch (Gradient Ascumslation)
Trais Reput
Train OPUs
19122
19564
19200
1982
g+
km
20:
Group-oruurt per quary
Figure 14: Native Distributed Data Parallel and Revision.
Table 14: Comparison of Communication Overhead: Before vs. Optimized (Localized Re- ward)
Component
Before Optimization
After Optimization (Ours)
Reward Aggregation
Requires all all_gather for every batch
No quired
communication
Group Statistics
Computed globally across all GPUs
Computed locally per GPU
Communication Frequency
High (per batch)
None for reward calculation
Latency Sensitivity
High (blocked by slowest node)
Low (fully asynchronous)
Scalability
Limited by network bandwidth
Highly scalable with node count
By eliminating this frequent and costly synchronization point, our system achieves higher throughput and better resource utilization. This is especially crucial in the HeteroRL. set- ting, where network conditions are umpredictable. The optimization allows sampler nodes to operate more independently, reducing their idle time waiting for global synchronization and making the overall training pipeline more resilient to network fluctuations. This engincering refinement, combined with the GEPO algorithm, provides a holistic solution for stable and efficient training in truly heterogencous, high-latency environments.
G THE COMPARISON OF DIFFERENT REINFORCEMENT LEARNING PARADIGMS
G.1 CORE CHALLENGE: OFF-POLICY LEARNING AND IMPORTANCE SAMPLING
This formulation highlights the central challenge in asynchronous RL: the mismatch between the behavior policy (#o,) and the target policy (Tœr.r), which grows with and introduces bias and variance into the learning process. Addressing this mismatch under high and uncertain lateney is the primary focus of our work.
32

---

Published as a conference paper at ICLR 2026
Table 15: Core Characteristics of Online, Offline, and Heterogeneous Reinforcement Learn-
ing Paradigms
Aspect
Online R.L
Offline R.L
Heterogeneous R.L
Data
Online interaction: Data generated instantly by the current policy.
Fixed, pre-collected dataset: Uses static data collected by some (unknown) behavior policy.
Delayed interaction: Data generated historical policy versions (due unpredictable network delay).
Generation
Policy Version during Data Generation
Always current: Requires strict syuchronization with the learming policy.
Fixed: The
Dynamically stale:
behavior policy is fixed and inherent to the dataset.
Staleness determined by uupredictable network delay (core characteristic).
System Architecture
Tightly-coupled: Actor (environment: interaction) and Learner (parameter update) typically co-located
Single-machine simple distributed:
Geographically decoupled: Actor and Learner separated by a high-latency network, tolerant to delays.
No real-time interaction
required; training is data-driven.
low-latency network.
Handling of System Latency
Treated failure: Requires
applicable: Training process has no real-time interaction.
Algorithmic
compensation: Employs
synchronization; latency canses resource idling and training stalls.
corrective techniques
(e-g., importance sampling)
mitigate latency effects.
Core Challenge
Exploration- exploitation trade-off during learning.
Distributional shift and limited data coverage.
Algorithmic stability under dynamic delays:
High/varias variable latency causes importance weight variance explosion.
* "Resource utilization under sequential tasks.
The HeteroRL framework decouples rollout sampling from parameter learning, leading to setting where the learner updates the new policy TØg using data generated by older policy nØz- a canonical off-policy learning scenario. To correct for the distributional shift between the behavior and target policies, importance sampling (IS) is commonly employed. Under the GRPO framework, this weight is used to scale the advantage during policy gra- dient updates. However, when the delay is large, causing significant divergence between the policies (i.e., high KL divergence (KL( [T,)), )), we observe in practice that the vari- ance of the importance weights increases rapidly with т, and the estimation error of the expected reward also grows. As result, the variance of Wstd explodes, introducing high noise into gradient estimates and ultimately leading to training instability or collapse. The central challenge of this work is thus to design robust algorithm that mitigates the train- ing instability caused by the explosion of importance sampling variance under high network latency.
H FUTURE WORK
H.1 DEFENSIVE SAMPLING AND SMOOTH DENOMINATOR MECHANISM
As promising direction for future work, we propose to explore defensive sampling strategy that adaptively blends the target policy probability into the importance weight denomina- tor. This approach aims to mitigate bias introduced by approximating the denominator while improving training caabilty--particllll settings. By introducing variance- aware smoothing coefficient, the method could dynami- cally interpolate between standard importance weighting and the policy gradient objective, thereby reducing sensitivity to high-variance estimates. We hypothesize that such smooth,
in asynchronous or hetcrogencous learning
33

---

Published as a conference paper at ICLR 2026
adaptive weighting scheme would lead to more robust and stable policy updates, and plan to investigate its theoretical properties and empirical effectiveness in future studies.
CASE STUDY
A case of MATH level 3-5: GRPO v/s GSPO v/s GEPO
User:
Find the greatest integer less than (v7 + v5)". (Do not calculator!).
GRPO
<think> Okay, I need find the grealest integer less than (V(7) (5))&. Humm, that seems pretly complicated at first glance. Let me think about how to approach this. Wait, maybe ...( Omitted numerous ineffective thoughts and reflections) Alternatively, maybe I can find a recurrence relation for S.. Let me try to find a recurrence. Let me note that z and roots of the equation t 2v77 7 2 0. Therefore, for any n 2 2, we have
x" = 2V7 x= 2=
Similarly y". Therefore,
Sn x² + 2v7(2"=>>-1-- + ) 2(x= n-2 1n 3m 2v7 Sn 2Sn-2.
This is the key step solving this problem) ... But this seems to involve v(7), which complicates things. However, if consider S, - something, maybe can eliminate the v(7) term? Wait, but maybe I need another approach.
Alternatively, maybe I compute Sn for 0, 1,2, and find pattern. Let try that.
First, let's compute Su = +"+++=+++++++++ 2.
...( Omitted numerous ineffective thoughts and reflections)
Wait, but Su 2" +y" However, since x = (() + (5) and y y /(7) /5), and since r and y positive? Wait, v(7) about 2.645, V(5) about 2.236. So /(7) ((5) is about 4.881, and v(7) ((5) about 0.409. So y positive, but small. Therefore, 26 is large number, and y" is small number. Therefore, Se x + is approximately x², and the integer part of 2 would be Sa - since 3 is positive but less than 1. Therefore, the greatest integer less than x is But wait, let me check if y" is less than 1. Since v(7) /(5) 0.409, then y is (0.409)%. Let me compute that: 0.409 z 0.167, then 0.167 z 0.004
Over length here, although close to arriving at the correct answer)
GSPO
<think> Okay, so I need to find the greatest integer less than (V(7) + ((5)). Hmm, that seems a bit intimidating at first, but I'll try to break it down step by step. First, I remember that expressions like (a + 6)" often have paltern when expanded, especially and b are irrational numbers. Since the question about the greatest integer less than this value, I might need to find an expression that's just below the actual value. Let think. If cousider (v(7) ((5)) and (V(7) (5))), their integer because
might be irrational parts cancel out. That's a common trick with conjugates. So, the sum of these two would be (sgrt(7) + (5r) + (sqrt(7) - sgr)(5)) which an integer. Therefore, the integer part of the original expression would be that integer minus the (sgrt(7) sgrt(5))*, which is small positive number.
So, the answer would be boxed the boxed integer baxedlboxedboeeeebeeebooee Wait, no, the boxed answer should be the boxed boxed answer boxed. So, boxed boxed boxed but boxed the final answer boxed.
Therefore, the boxed answer is the boxed boxed boxed boxed boxed boxed boxed answer boxed. So boxed boxed boxed boxed boxed boxed boxed boxed boxed boxed boxed boxed... Numerous repetitions occur here.)
34

---

Published as a conference paper at ICLR 2026
GEPO
<think> Okay, so I need to find the greatest integer less than (V(7) + (5))a. Humm, that seems prefty big. I don't have calculator, so I have figure this out without calculating the actual value. Let me think.
But maybe I can find recurrence formula. Let's We know that x and y are of the quadratic equation 6² (ær y)t + ry 0. So
? 2-22 2/(27++ O. Therefore, the recurrence relation for S, is S, 2 -2
This is the key step to solving this problem) Therefore, the answer 13535 </think>
To find the greatest integer less than (v7+v5) of the form + vb)", the result is an integer. TTiss is because the irrational parts cancel out when the two expressions added together.
the identity that for any expression
Let: v7 + v5 y v7 v5
We compute:
First, calculate x²:
x² = (V7 v5)² = 7 +
12 2v
Then x² (1²/² (12 2v35) 144 48
3- 284 48v35
Next, x² x² x² (284 + 3 25) Expanding this:
284 12 + 284 2v
48
35 12 48 35 2/35
3408 + 568 /5 576v35
3408 + 1144v35 + 3360
6768
1544v/5
Similarly, y
6768 /144v35 Adding x and y²:
+
2 6768 13536
Since x 13536 v and y is small positive number (as v7 v5 0.4097), have:
13536 0.004 13535.996
Thus, the greatest integer less than x is:
13535
Comparison:
GRPO exhibits large number of ineffective reflections, wasting many valuable tokens; statistics show that GRPO uses words such as "wait" and "however" more than times. Due to reward collapse, GSPO begins generating extensive repetitions after brief reasoning, rendering the response unreadable. GEPO arrives at the correct answer during the thinking phase through sound reasoning and reflection. When formally responding, its thought process clear and steps are concise.
LARGE LANGUAGE MODEL USAGE
The use of LLMs in this article is limited to text polishing and code generation for plotting.
35


Reviews:
[{"review_id": "9TDxNdCPAr", "reviewer": "Reviewer_8QcJ", "summary": "The paper proposes HeteroRL, a decentralized RL framework that decouples rollout sampling from learning to tolerate Internet-scale latency and heterogeneous hardware. Its core algorithm, GEPO (Group Expectation Policy Optimization), replaces token/sequence-level importance weights with a group-level expectation to reduce the variance explosion that arises when the learner’s policy drifts from the samplers due to staleness. A theoretical result shows variance reduction grows (approximately) exponentially with KL divergence between policies, and experiments on Qwen3-1.7B/8B with math-reasoning benchmarks (AIME/MATH/AMC) report higher accuracy and markedly improved stability under simulated delays (up to 1800s).", "questions": "See above.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "**Clear systems problem + algorithmic handle**. The paper crisply diagnoses policy staleness in decentralized RL and ties it to KL growth → variance blow-ups; GEPO’s denominator uses a within-group expectation to damp variance in precisely that regime.\n\n**Compelling empirical stability**. Under Hetero RL with max delay 64, GEPO improves best accuracy vs. GRPO/GSPO and, crucially, reduces best-to-last degradation by ~85% vs. GSPO (Δ=1.8 vs. 12.0). Curves show lower IW variance and smoother gradients.\n\n**Theoretical insight**. Theorem 1 relates the variance gap (standard IS vs. group-expectation weight) to exp(D_KL), motivating why the method helps exactly when staleness is worst.\n\n**Realistic setting**. The system model explicitly injects stochastic network delay and hetero nodes (Ascend + NVIDIA), aligning with decentralized community compute setups.", "weaknesses": "**Bias–variance trade-off left under-quantified in RL objective**. GEPO’s estimator is acknowledged as biased; while lower variance can help optimization, the paper does not quantify end-to-end bias in policy gradients or returns beyond variance plots. A small-bias claim would benefit from controlled ablations where true on-policy gradients are approximated (short-horizon toy MDPs) to measure bias vs. sample efficiency. (GEIW is described as biased but stable.)\n\n**External validity beyond math-reasoning**. Results are limited to math QA on Qwen. It would help to show GEPO under non-text or mixed-modality tasks, or code/data generation tasks where rollouts have different length/entropy profiles. Current related work cites broader systems, but experiments remain narrow.\n\n**Ablations & knobs**. The method depends on group size G and sampling strategy (top-k/p). Sensitivity plots for G, the truncation strategy, and the CPPO-KL coefficient under delay would strengthen the story. (Implementation notes mention CPPO-KL and latency simulation, but tuning studies are light.)\n\n**Fairness to GSPO variants**. Since GSPO is sequence-level IS with clipping, it would be good to include GSPO + stronger clipping / trust-region or defensive mixture baselines to test whether variance spikes can be similarly tamed by tuned GSPO, not just GEPO’s denominator. The paper itself sketches “defensive sampling” as future work—great idea, but makes me wonder how close-tuned GSPO would get.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0H5McZmBsW", "reviewer": "Reviewer_aJa6", "summary": "The paper introduces GEPO, a reinforcement learning algorithm designed for heterogeneous, high-latency distributed training of large language models. It builds on a new framework that decouples sampling and learning across asynchronous nodes. To address instability caused by stale policies, GEPO replaces standard importance weights with group-level expectation weights, theoretically reducing variance when policy divergence is high. Experiments on mathematical reasoning tasks with Qwen models show that GEPO achieves higher accuracy and significantly improved training stability compared to GRPO and GSPO under simulated latency.", "questions": "See Weaknesses", "limitations": "", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The visualizations illustrate the main claims of the paper.\n2. The paper targeted on an important bottleneck problem in large-scale distributed reinforcement learning.", "weaknesses": "1. The paper lacks experimental comparisons with other asynchronous policy optimization methods [1, 2, 3, 4].\n2. Equation (1) appears very similar to PPO, except that the clipping function is removed.\n3. In line 139, the variable $G$ is undefined; in Equation (1), it is unclear how $A(x)$ is computed, and in line 146, the input of $p$ is not specified.\n4. While Theorem 1 demonstrates a reduction in the variance of the importance sampling coefficient, this result does not guarantee a corresponding reduction in the variance of the weighted term $A \\cdot w$, because $A$ itself is a random variable. In general, even if one random variable has smaller variance than another (e.g., $\\mathrm{Var}(Y)<\\mathrm{Var}(Z)$), it does **not** necessarily follow that $\\mathrm{Var}(XY)<\\mathrm{Var}(XZ)$, unless $X$ is independent of both $Y$ and $Z$ and their expectations are zero.\n5. The claim in lines 178–181 seems questionable. In extreme cases, clipping can suppress unstable tokens by assigning them zero weight, which may be beneficial since such extreme tokens usually correspond to unreliable generations. Thus, clipping might actually serve as a useful safeguard in these scenarios.\n\n [1] Rastogi, Abhinav, et al. \"Magistral.\" *arXiv preprint arXiv:2506.10910* (2025).\n\n [2] Fu, Wei, et al. \"AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning.\" *arXiv preprint arXiv:2505.24298* (2025).\n\n [3] Zhong, Yinmin, et al. \"StreamRL: Scalable, Heterogeneous, and Elastic RL for LLMs with Disaggregated Stream Generation.\" *arXiv preprint arXiv:2504.15930* (2025).\n\n [4] Han, Zhenyu, et al. \"AsyncFlow: An Asynchronous Streaming RL Framework for Efficient LLM Post-Training.\" *arXiv preprint arXiv:2507.01663* (2025).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "c3WdOj4hZP", "reviewer": "Reviewer_bBKw", "summary": "This paper presents HeteroRL, a decentralized reinforcement learning framework designed for training LLMs across distributed and resource-heterogeneous environments. The paper introduces GEPO, including a new group expectation importance weighting mechanism to mitigate the instability caused by policy staleness. Theoretical results show that GEPO exponentially reduces the variance of importance weights under large KL, thereby stabilizing training in asynchronous and high-latency settings. Extensive experiments demonstrate the method's superior performance and stability on reasoning benchmarks.", "questions": "1. Could the authors clarify whether the ablation experiments comparing GEPO, GRPO, and GSPO are conducted under the same experimental settings as the main comparison in Section 4? If the setups are identical, how do these ablation results differ conceptually from the earlier baseline comparison?\n2. Why does GEPO remove the normalization when calculating the advantage? In practice, would including this normalization affect the performance or training stability? \n3. It is unclear how the case study in section G is directly connected to the theoretical claim about variance reduction in importance weights.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea of reducing the variance of importance weights through group expectation weighting is conceptually elegant and practically powerful. It addresses the instability issue caused by large KL divergence in asynchronous or heterogeneous RL settings.\n1. The motivation, instability and variance explosion in asynchronous or heterogeneous RL due to policy staleness, is well validated by experimental results. The results across both online and heterogeneous RL settings consistently demonstrate that GEPO achieves lower variance, smoother gradients, and higher stability compared to GRPO and GSPO.", "weaknesses": "1. The experimental validation appears limited in scope. GEPO is only evaluated on mathematical reasoning datasets (MATH, AIME, AMC) and with relatively small models (up to 8B parameters).\n1. There seems an inconsistency between the formulation and the theoretical analysis. In Section 3.1, the paper explicitly states that \"the vector $(q(y_1|x), …, q(y_G|x))$ does not constitute a valid probability distribution\" since top-K/top-P sampling leads to $\\sum_i q(y_i|x) \\gg 1$. However, in Theorem 1 and its proof, the derivation assumes that $q$ is a normalized discrete probability distribution satisfying $\\sum_i q(y_i|x) = 1$. How the theoretical guarantee holds when $q$ is not a valid probability distribution in practice?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hTLazB3ujV", "reviewer": "Reviewer_5yWm", "summary": "The paper proposes HeteroRL, an asynchronous framework for RLHF that decouples rollout sampling from learning to tolerate WAN latency and heterogeneous hardware. Its key algorithm, GEPO, replaces token/sequence importance ratios with a group expectation denominator, theoretically reducing IS variance—especially at high sampler–learner KL—and stabilizing training under policy staleness. Stronger last-epoch performance and robustness vs. GRPO/GSPO are shown on math-reasoning benchmarks under simulated delays.", "questions": "As shown in weakness.\n\nBtw, why placing the related work section in the final, which is not usual. And the related works seem limited and unorganized.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Originality: It makes sense by clear shift from token/sequence to group-level weighting tied to the real failure mode.\n- Theory: It gives exponential variance reduction of importance weights in high-KL regimes; mechanistic gradient comparison clarifies why updates stabilize. \n- Empirics: GEPO maintains accuracy and avoids collapse across delays; best/last markedly stronger than GRPO/GSPO.", "weaknesses": "- Bias–variance trade-off not quantified. The authors should provide MSE curves vs. KL and group size, and analyze the green-region cases where variance may increase.\n  \n- Bound constants/tightness unclear.The theorem invokes constants (e.g., \\(C\\)) but does not estimate or validate their tightness on real training distributions; what about reporting empirical lower/upper bounds and how they vary with vocabulary/sequence length.\n  \n- Systems throughput and realism underreported.The method targets WAN latency, but the authoers do not provide wall-clock throughput (samples/s), utilization, or sensitivity to jitter/packet loss; it is better to add end-to-end performance, synchronization frequency, and utilization metrics.\n\n- Missing off-policy/staleness baselines. I am curious about the comparison omits V-trace/IMPALA-like corrections and AREAL-style staleness controls, which are only mentioned in related works; the authors should add these baselines to separate gains from grouping vs. specific normalization.\n  \n- Sensitivity analyses incomplete.The stability claim depends on grouping and sampling, but the paper does not sweep group size \\(G\\), temperature/top-p, or KL-regularization × delay.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)