Prosperity before Collapse: How Far Can Off-Policy RL Reach with Stale Data on LLMs?
Authors: Haizhong Zheng
Jiawei Zhao
Beidi Chen
Venue: ICLR 2026 Poster (2026)
Keywords: Off-policy RL; LLM; Reasoning
Primary area: foundation or frontier models, including LLMs

Abstract: Reinforcement learning has been central to recent advances in large language model reasoning, but most algorithms rely on on-policy training that demands fresh rollouts at every update, limiting efficiency and scalability. Asynchronous RL systems alleviate this by decoupling rollout generation from training, yet their effectiveness hinges on tolerating large staleness in rollout data, a setting where existing methods either degrade in performance or collapse. We revisit this challenge and uncover a \emph{prosperity-before-collapse} phenomenon: stale data can be as informative as on-policy data if exploited properly. Building on this insight, we introduce \textbf{M2PO} (Second-Moment Trust Policy Optimization), which constrains the second moment of importance weights to suppress only extreme outliers while preserving informative updates. Notably, M2PO sharply reduces the fraction of clipped tokens under high staleness (from 1.22\% to 0.06\% over training), precisely masking high-variance tokens while maintaining stable optimization. Extensive evaluation across six model scales (from 1.7B to 32B) and eight math reasoning benchmarks and one coding benchmarks shows that M2PO delivers stable off-policy training even with data stale by \underline{\emph{at least 256 model updates}} and matches on-policy performance. Our code is available at https://github.com/Infini-AI-Lab/M2PO/.

Published as a conference paper at ICLR 2026
PROSPERITY BEFORE COLLAPSE: How FAR CAN OFF- POLICY RL REACH WITH STALE DATA ON LLMs?
Haizhong Zheng' Jiawei Zhao² Beidi Chen'
CCamegic Mellon University MMeta AI (haizhonz, beidic cridiccccmu.eee.. cmm. edu, jwzhao@meta.com com
ABSTRACT
Reinforcement learning has been central to recent advances in large language model reasoning, but most algorithms rely on on-poliey training that demands fresh rollouts at every update, limiting efficiency and scalability. Asynchronous RL systems alleviate this by decoupling rollout generation from training, yet their effectiveness hinges on tolerating large staleness in rollout data, set- ting where existing methods either degrade in performance or collapse. We re- visit this challenge and uncover prosperity-beforr-collaa phenomenon: stale data can be as informative as on-policy data exploited properly. Building this insight, introduce M2PO (Second-Moment Trust Policy Optimiza- tion), which constrains the second moment of importance weights to suppress only extreme outliers while preserving informative updates. Notably, M2PO sharply reduces the fraction of clipped tokens under high staleness (from 1.22% 0.06% over training), precisely masking high-variance tokens while main- taining stable optimization. Extensive evaluation across six model scales (from 1.7B 32B) and eight math reasoning benchmarks and one coding benchmarks shows that M2PO delivers stable off aff-policy training even with data stale by least 256 model updates and matches on on-policy performance. Our code is avail- able at /ttps:://gittbb.c... IIn - -T--ATb/LaPPO
Qwen-2.5-32B
Qwen- 2.5-32B
0.2297
0.077
Clicaing Ratto
0.0257
0,8223
Accuracy
0.46
0.0022
0.8775
On policy - 256 Staleness GRPO
policy
0.252
256 Staleness GRPO
256 Staleness
GRPO (no TR)
Staleness M2PO (Ours)
0.135
04222
Updates
Updates
Figure 1: Comparison of on-policy GRPO and off- aff-pooicy training under staleness of 256 model updates on -wen-2.5-32B. Left: Standard GRPO suffers from degradation with stale rollouts, while removing the trust region (GRPO no TR) reveals clear prosperity-before-cllle contrast, M2PO achieves stable training and matches on-policy performance even under high staleness. Right: Token clipping ratio comparison shows that M2PO dramatically reduces clipping events compared to GRPO with the same staleness, while avoiding training collapse.
phenomenon.
INTRODUCTION
Reinforcement learning (RL) has been central to recent advances in large language model (LLM) reasoning, driving breakthroughs in systems like OpenAl's (OpenAl et al. 2024) and DeepSeek's R1 (DeepSeek-Al et al. 2025 Team et al. 2025). ) Most existing RL algorithms (Schulman al.
Zheng et al. 2025b Yu et al. 2025) for LLMs adopt an on an-policy design, as it provides stable training and reliable performance, but the strict requirement for fresh (or limited-stalennss) rollouts

---

Published as a conference paper at ICLR 2026
at every update. This constraint significantly limits scalability and becomes increasingly imprac- tical as we move toward harder, more complex reasoning or agentic tasks, where rollouts involve multi-step tool use, expensive program execution, and long reward computation. In such settings, rollout latency can easily range from minutes to hours, making substantial staleness not only com- mon but often unavoidable. For example, SWE-hench [Tmeeee [Jimenez et al. (2023) with OpenHands Wang]
iet al.] (2024), rollout latency can become extremely large. In a single run, even with a small batch size, the end-to-end inference time (including tool call and code execution) can exceed 100 min- utes, with more than 80 iterations in the environment, which makes on on-policy training extremely inefficient. To overcome this bottleneck, a growing line of RL systems (Fu et al.) 2025 Zhu et al.]
2025 Noukhovitch et al., 2024 Zhong et al.] [025 He et al., (025) have explored asynchronous designs that decouple rollout from training. Such approaches improve resource utilization and en- able training to scale more efficiently across large and heterogeneous clusters, but their effectiveness
fundamentally relies on the ability of RL algorithms to tolerate rollout staleness without sacrificing stability or performance.
However, under large rollout staleness, existing RL algorithms struggle to strike the right balance. Some methods (Schulman et al. 2017: Shao et al. 2024; Zheng et al. (025a) can maintain stability, but they often suffer from noticeable performance degradation. Conversely, approaches designed to maximize performance (Fu et al. 2025 Chen al. 2025 Suu et al., 2025 tend to compromise sta- bility, frequently leading to training collapse. On-policy methods provide both stability and strong performance, but their reliance on fresh or only slightly stale rollouts at every update imposes rigid constraints that hinder scalability. Consequently, an ideal off-policy RL algorithm for LLMs should enable effective reuse of trajectories collected under outdated policies to preserve strong perfor- mance under significant staleness, and ensure stable training that converges competitively with on- policy methods. Meeting these requirements is key to realizing off-policy RL as a truly scalable solution for aligning and fine-tuning large language models.
In this paper, we aim to investigate the underlying reasons for the limitations of off- off-policy RI. in LLMS and to design an effective algorithm that fully leverages stale data to unlock its potential. We begin by revealing an intriguing Prosperity before Collapse phenomenon (Yellow curve in Fig- (lef)))))): although RL training without trust region eventually collapses on stale data, it initially achieves substantially higher performance than vanilla GRPO with c-clipping. In some cases, it even matches the performance of the on-policy baseline. From this, we draw important observation: stale data can be as informative as data collected on-policy in RL for LLMs, but the key chal- lenge lies in how existing algorithms exploit it. In particular, vanilla GRPO performs poorly under staleness because stale-data training exhibits a substantially higher clipping rate, with many of the clipped updates occurring on informative high-entropy tokens (see Figure 4 This disproportionate clipping on crucial tokens hinders the full utilization of stale training data.
This pivotal token masking observation reveals that these high-entropy tokens play a dual role: they provide the most informative training signal but also introduce the greatest instability under stale- ness. Therefore, the key challenge is retain as much learning signal from these tokens as possible without risking training collapse. Motivated by this, we propose M2PO Recond-Moment Trust Policy Optimization), a novel off off-policy RL algorithm that constrains the second moment of impor- tance weights. Unlike standard e-clipping, which disproportionately suppresses high-entropy tokens and discards valuable learning signals, M2PO leverages the second- -omentt metric M2. This metric is both stable, avoiding the cancellation issues inherent to KL-based measures. By regularizing training at the batch level through M2, M2PO masks only extreme outliers while preserving the majority of informative updates. As a result, M2PO enables stable off-policy reinforcement learning with stale data, matching on-policy performance even under large staleness.
capturing instability introduced by high-entropy tokens, and statistically
As illustrated in Figure | (left), even when trained exclusively on data stale by at least 256 model updates, M2PO achieves accuracy comparable the on-policy baseline (red curve), demonstrating its ability to fully exploit stale data without sacrificing stability. M2PO achieves this through a more accurate and adaptive clipping strategy that clips substantially fewer tokens while maintaining train- ing stability. As shown in Figure I (right), M2PO dramatically reduces the fraction of clipped tokens under high staleness (from 1.22% to 0.06% over the entire training process, see Figure (d)) thereby preserving more useful training information in stale data. To further validate M2PO effectiveness, we conduct an extensive evaluation of M2PO across six model scales (ranging from 1.7B to 32B) and eight math reasoning benchmarks in Section 6 The results show that M2PO consistently deliv-
2

---

Published as a conference paper at ICLR 2026
strong performance across all training settings. M2PO also shows insensitivity to the choice of threshold, with a single value across all experiments, demonstrating its practicality and robustness.
2 RELATED WORK
RLVR. Recent advances (DeepSeek-Al al. 2025 Yu al. 2025; Team al.) 2025 Gao al. 2024 in LLM reasoning show that Reinforcement Learning with Veriñable Reward (RLVR), which relies on verifiable reward signals instead model-generated scores, can effectively improve model reasoning ability. These gains are achieved using various policy optimization methods such as 2024). Encouraged by the success of RLVR, Yuan et al. 2025b Yu et al., 2025 Liu et al., 2025| Xiong et al. 2025) has emerged to fur- ther improve reinforcement learning methods for LLM reasoning. For instance, methods such as VinePPO (Kazemnejad et al. 2024), VC-PPO (Yuan et al. 2025b), and VAPO (Yuan et al. 2025a)
PPO (Ouyang et al.) (022) and GRPO (Shao et al.
growing body of work (Kazemnejad et al. 2024
2025b, Luo et al. 2025a Zhang et al., 2025 Hu
aim to enhance LLM reasoning by optimizing the value function.
Trust Region in RLVR. While RLVR has been widely adopted for fine-tuning LLMs, a key chal- lenge lies in how to effectively constrain the trust region, only stabilize training but also to achieve better learning efficiency and overall performance. To address this, growing line of work has proposed various strategies to control the policy update, ranging from ratio clipping (Yu al.) 2025), approximate trust region (Fu et al. (2025,, sequence-level clipping (Zheng et al. 2025a). asymmetric trust region (Roux al., 2025| Armal al., 2025), and gradeent-preserving clipping (Su et al. 2025 Chen et al.) 2025). For instance, AREAL (Fu et al. 2025) uses more recent approxi- mate policy to decide the trust region rather than the behavior model. GSPO (Zheng al.) 2025a) moves from token-level to sequence-level clipping by defining importance ratios on sequence like- lihood. While these methods improve RLVR under moderate settings, most of them focus on rel- atively limited intra-iteration staleness (c.g.. 8 or 16) and have not been thoroughly studied under larger off- ool-polcc gaps, like extreme staleness. this work, our goal is to better understand the role of staleness in RLVR and seek more effective ways of constraining the trust region in RLVRR.
3 BACKGROUND
3.1 GROUP RELATIVE POLICY OPTIMIZATION (GRPO)
Group Relative Policy Optimization (GRPO) Shao et al. 2024) a variant of Proximal Policy Opti- mization (PPO) (Ouyang al. (022) tailored for language model fine-tuning. Instead of computing advantages using a value function, GRPO normalizes reward scores within groups of responses sam- pled for the same prompt, which largely improves the training efficiency, and aims to maximize the following objective:
Janro(0) =Elg P(Q), (o)) Nowi (Olg)
(Olg)
1
(ox1g)
To(0x(1)
(oslg) e, -1c,1++)) A. A:)).
(1)
min
To (0slg) A,, clip
G
Toneha
where A; is the advantage, computed using a group of rewards fri, r2.., ral corresponding to outputs within each group:
A Ti mean( (R&
(2)
std (R,(RR) (
Similar to PPO, GRPO employs clipping mechanism stabilize updates. The ratio r; mo(o, Ig) (o.44)
is clipped to 1 + e). that when A; > 0 the policy cannot increase probability mass exces- sively, and when A, < it cannot over veralizn This prevents large, unstable updates while still allowing normalized group advantages guide learning.
3

---

Published as a conference paper at ICLR 2026
3.2 PERFORMANCE DEGRADATION FROM TRAINING WITH STALE DATA
To investigate the impact of stale data on reinforcement learning for large language models, introduce Stale-k RL training, where the model is trained using data generated model updates earlier in each training iteration. More specifically, in our training setup, each training step consists of four model updates, a configuration commonly used in recent work (Zheng et al.) 2025b Wang al,, 2025b; Yu et al., 2025| Chen et al.) 2025 Zheng al., 2025a). Thus, even stale-O (s=0) training has a staleness between 0 and 3. stale-256 (s=256) training has a staleness between 256 and 259). During the first model updates, since no stale model is yet available, the model is trained on data generated by the original base model, with different training data used in each iteration. In this setup, all training data after the initial phase comes from stale models, allowing us to study how stale data affects the dynamics and effectiveness of RL training.
Stale-k RL training. In our "stale-k" setup, the training batch at step ł is collected k updates earlier, but we implement this by storing the generated data rather than storing old checkpoints. Concretely, update step ( - k we use the current policy to generate rollouts and place them into a buffer, and after & fur- ther updates, these trajectories are consumed once for training at step I. Each sampled trajectory is used exactly once and is never reused across multiple updates. More training details be found at Appendix B
Accuracy
0.44
6.4
5=128
32
5-256
220
1202
1008
2000
Model Updates
As shown in Figure 2 we train Qwen2.5-Math-7 (Yang et al.] 2024) with GRPO under varying staleness levels and report test accuracy. The results reveal a clear trend: as staleness increases, model performance degrades and convergence slows. In partic- ular, low-staleness training achieves higher accuracy, whereas high-staleness training converges more slowly to lower performance.
Figure 2: RL with stale data on Qwen2.5-Math-77B reporting avg. accuracy on all benchmarks.
4 PROSPERITY BEFORE COLLAPSE: STALE DATA CONTAIN ENOUGH TRAINING INFORMATION IN RL ON LLMS
In this section, we investigate why RL on LLM deteriorates when trained on stale data generated by earlier policies.
Qwen-2 M-tthh-78
Llama-3.2 1.mma-3.-inssrusse
0:32
0.45-
01.22
Accuracy 196)
Accuracy 1963
0.23
First, we reveal an intriguing prosperity- before-collapse phenomenon: although off- policy RL training without a trust re- gion eventually collapses stale data, it achieves substantially higher performance than GRPO with e-clipping before collapse, even matching on- on-policy results. Next, we study the causes of GRPO's inferior perfor- mance when trained with stale data.
0:43
0.18
0:35
0:18
5-0
5=0 TR
6.32
5-256
TR
5-256 TR
5-256
w/o TR
6.1
5=256 w/o TR
1000
3000
2003
2500
sce
1958 1250 3800
Model Updates
Model Updates
Figure 3: Prosperity before Collapse. Training with- out a trust region (TR) (c oc) under stale data (s 256) initially achieves higher performance than clipped training, sometimes even matching the on- policy baseline (s = 0). However, eventually col- lapses due to uncontrolled variance.
Prosperity before collapse: training with- out trust region. To disentangle whether the performance drop stems from stale data generated by highly shifted old policies or from biases introduced by the training algo- rithm, we remove the trust region entirely to remove bias from the training algorithm. Surprisingly, we observe a distinct although training without trust region eventually collapses, it achieves substantially better perfor- mance prior to collapse. In fact, under stale data (s=256), the no-clipping setting initially outper- forms clipped training, sometimes even matching on-policy baselines.
phenomenon. As shown in Figure and Figure]
Pivotal token masking by c-clipping when training with stale data. As also discussed in re- cent work (Su al. 2025 Chen et al. 2025), e-clipping may inadvertently mask important tokens,
4

---

Published as a conference paper at ICLR 2026
preventing them from contributing useful training signals. We extend this observation to the asyn- chronous setting and show that the problem becomes substantially more severe when training with stale data, since larger staleness induces greater mismatch between the behavior and target poli- cies. As illustrated in Figure 4al the clipping ratio increases sharply under large staleness (s = 256), while remaining negligible in the on-policy baseline.
To better understand this phenomenon, conduct quantitative analysis on 90 million training tokens collected during Qwen2.5-Math-7B training with staleness 256. Specifically, we gather all training tokens generated between 800 and 1200
0.006
1.0
Clipping Ratto
0.006
Entropy
0.6
0.002
model updates, ensuring the model is al- ready in stable training phase but be- fore convergence. Figure 4b shows a clear trend: as rr - increases, the average to- ken entropy also rises.
0.003
Qwenz.5 M6A1-8
0.000
1200
2200
2000
4000
0.0
0.2
0.4
0.6
Model Updates
(a)
(b)
This indicates that e-clipping dispro- portionately prunes high-entropy tokens, which are typically the most informa- tive for model improvement (Wang al.) 2025a Gao et al. 2025| Cui et al. (2025). Consequently, clipping under stale data leads degraded performance.
Figure 4: (a) Clipping ratio dynamics during RL train-
ing on the Qwen-2.5-Math- model. (b) Relation-
ship between average token entropy and the distance between the importance sampling ratio and 1.
This observation reveals a dilemma: while high-entropy tokens are crucial for learning progress, they also introduce instability the off-policy setting, which motivates our key research question:
Can a more accurate and adaptive trust region strategy preserve the benefits of stale data while ensuring stable training?
SECOND-MOMENT TRUST POLICY OPTIMIZATION
In this section, we propose Second-Moment Trust Policy Optimization (M2PO), a novel policy optimization algorithm providing more effective trust region for off-policy training with stale data. As discussed in Section 4 as high-entropy tokens are a double-edged sword, the key challenge in designing an effective trust region algorithm is how to best harness the rich information in high- entropy tokens without letting them destabilize training.
5.1 MEASURING DISTRIBUTION GAP WITH THE SECOND MOMENT
The main source instability in off-policy RL lies in the distributional mismatch between the be- havior policy that generates training data and the current policy being optimized (Schulman et al. 2013 2017). As the divergence between these two distributions grows, importance sampling cor- rections produce high-variance gradient estimates, leading to noisy and unreliable updates. Our motivation is therefore to constrain the distributional gap between Tehaa and my at the batch level, directly coupling the constraint with model updates while preventing over-constraining of token- level variations.
A natural choice to measure distribution is the batch-level KI. divergence, metric widely adopted to monitor stability in RL:
N KL
I
Σ RT. =
log =
1 N
log
To 8i) Tbeluv (o: 8;)
where is the number of tokens in a batch.
However, batch-level KL suffers from two key limitations. First, because it is computed from single- sample estimates, individual K To can be positive negative, leading cancellation effects where
5

---

Published as a conference paper at ICLR 2026
large deviations offset each other and produce deceptively small KL values. Second, tokens with large ratios (r: 1) not properly constrained, their negative KT, actually decreases the estimated KI., even though such tokens can contribute to training instability (Schulman et al. 2017)
To overcome these limitations, propose to the second moment of the log-ratio to measure the distribution gap between behavior and current policy. Formally, we define
N N
N Σ (logr:)2 N
1
log (a; 8i) 122 Tbchav (a: st)
M2
M2.1
(4)
This choice is motivated by two key advantages of M2 over the batch KL. First, each per-token estimate M2,; = (log r²) always non-negative, so the constraint can be reliably applied even when r > 1. Second, while the batch KL only measures the mean shift between policies, M2 also reflects the variance of importance weights. This makes M2 more sensitive to outliers and noisy tokens with extreme ratios r
Furthermore, Theorem shows that although M2 does not directly constrain r- like clipping, it nevertheless provides an upper bound on the Pearson chi-square divergence E[(r - 1 between the new and behavior policies. The proof is provided in Appendix [
Theorem (Bounding x² by M2). Lel Fucz
be the importance ratio and assume 1/ srs
R. Define the log-ratio second moment
M2 Bawsaccaa [1lg r(a)))]-
Let the Pearson chi-square divergence between new and Tbehav be
( nnew (a)
27
x" (Tnew Tbehav)
= EanATeehuv
(
Thohav (a) -1)
Exechar (r- -1)?
Then
(TTew (behav)
S R2 M2.
5.2 SECOND-MOMENT TRUST POLICY OPTIMIZATION
maintain training stability, M2PO applies a masking strategy that se- lectively excludes tokens until the batch-level M2 of the remaining tokens falls below a predefined thresh- old TM2- Importantly, we observe that TM₃ is not sensitive hyperparameter (see Figure 7 Across all experiments, consistently set TMs 0.04. and this single setting proved effective for stabiliz- ing training all training scenarios.
As illustrated in Algorithm
Algorithm 1: M2PO Masking
Input: {M L. for all training tokens; threshold TMs
Output: mask M
M <- True for all tokens;
T + all trust-region tokens;
/* Sort: O (NI ogN)
x/
Sort all tokens w.r.t. M2,4-
Only constrain trust-region tokens. Although the PPO loss clips the ratio on both the upper and lower sides, due to the use of the min operator, not all to- kens are actually clipped. In practice, clipping only occurs for tokens where A > O and r > 1, or A < and r 1. Following the PPO setting, we there- fore apply the M2 constraint exclusively to tokens end while that satisfy these conditions. Finally, with the result 10 return M mask M, we update the policy by maximizing the following objective?
while meanset M2, TMg do
/* Select: (1)
*/
j arg maxeT M2.i
M +- False;
T T-TV(i
'A potential alternative use N |LLL,/N. While this absolute estimate can also work empiri- cally, it is less sensitive to variance compared to Ma Moreover. M2 provides an upper bound for this absolute KT. estimate, F][r]]] < / T. Therefore, we adopt M2 in our method.
'In our loss, we average over all tokens rather than only the unmasked ones. This choice is intended to better mimic the hehavior of PPO-style clipping. However, since the masking ratio is typically very small (see Section the difference between the averaging strategies is negligible in practice.

---

Published as a conference paper at ICLR 2026
lod M
To(o; Thhahan (0i19)
[M) Σ
A,t.
M E (0,1),
(5)
1o:
where A, denotes the advantage, computed using the grouped advantage Equation
6 EXPERIMENTS
In this section, we present extensive evaluation across six models (from 1.7B to 32B) eight benchmarks. The results demonstrate that, even when trained with extremely stale data, M2PO achieves performance comparable to on-policy GRPO and significantly outperforms other baselines:
In Section 6.2 we show that M2PO achieves accuracy on par with on-policy baselines under large staleness (s 256), and outperforms baselines by up to 11.2% in average accuracy.
In Section we provide a detailed analysis of how M2PO boosts off-policy RL performance while preserving training stability. We also show that its sole threshold hyperparameter TM2 is insensitive to variation, ensuring ease of use in practice.
6.1 EXPERIMENTAL SETTINGS
Models & Datasets. To verify the effectiveness of method, extensively evaluate M2PO on six models: Qwen2.5-Math-7B (Yang et al. 2024), L (Dubey et al. 2024), Qwen3--Bss-1..78/4BB88 (Yang et al.] (025)) and Qwen2.5-32B (Yang al., (024). For Qwen2.5- Math-7B, we use context length of 4k, which is the maximum for this series, while for all other models the context length is set 16k. For training, we adopt the DeepScaleR (Luo al. 2025b)
math dataset.
Training& & Evaluation. Our method is implemented based on verl (Sheng et al. 2024) pipeline and uses vLLM (Kwon et al., 2023) for rollout. We use mix of H100 and H200 servers for training, de-
pending on resource availability. For benchmark datasets, we use eight widely used complex mathe- matical reasoning benchmarks to evaluate the performance of trained models: Math500 (Hendrycks)
al.] 201T Lightman et al. 2023), AIME24/25 (Art of Problem Solving 2024a), AMC23/24 (Art
of Problem Solving 2024b). Minerva Math (Lewkowyez et al. (022)) Gaokao (Zhang al. (023), Olympiad Bench (He et al. 2024). Similar to (Wang et al. 2025b Zheng et al. 2025b), we eval- uate models those benchmarks every 50 steps and report the performance of the checkpoint that obtains the best average performance on eight benchmarks. For GRPO, we adopt the commonly used clipping parameter 0.2, while for the other baselines, follow the recommended values reported in their respective papers. We include more detailed experimental settings in Appendix
6.2 PERFORMANCE COMPARISON ON TRAINING WITH STALENESS
Prosperity without collapse: Stable off-policy training without performance degradation using M2PO. To verify the effectiveness of M2PO, Table [ presents a comprehensive comparison of math reasoning performance across eight benchmarks using models from four different families and scales, ranging from 1.7B to 32B parameters. We evaluate multiple reinforcement learning methods under both on-policy and off-policy settings, including GRPO, GSPO, and our proposed M2PO. The results show that while both GRPO and GSPO often suf- fer significant performance drops under large staleness, M2PO consistently achieves comparable accuracy to the on-policy baseline in all training settings. Surprisingly, we notice that, in some model settings, M2PO with = 256 even achieves a better performance than M2PO with s = For instance, on the Qwen3-Base-Ie (36.6%) outperforms GRPO with s = (33.0%). A potential explanation is that small effective staleness (e.g.. = 0 corre- sponding to delays between and 3) can still adversely affect training stability. Our further analysis in Figure 6 supports this view, showing that M2PO with s = 256 exhibits an even lower clipping ratio than GRPO with 0. Overall, these results
Qwen-2.5-32
Reward
Base- 1.7B model, we observe that M2PO with s = 256
6.3
seD GRPO
5-256
5-256 M2PO (Ours)
2000 2000
Model Updites
Figure 5: Training reward on
Owen-2.5-32B.
7

---

Published as a conference paper at ICLR 2026
Table 1: Performance (%) comparison across eight math reasoning benchmarks using models from 1.7B to 32B parameters. We report results for GRPO, GSPO, and M2PO under both on-policy 0) and off ooolcy (s 256) settings. Underlined numbers denote the best average aceuracy, while bold numbers highlight the best average accuracy under stale rollouts (s 256). M2PO consistently improves stability under staleness and achieves higher average accuracy than GRPO.
Method
s
AIME24/25 AMC23/24
Math500
Gaokao
Miner.
Olymp.
Avg.
Llama-3.2-3B-Instrid
GRPO
0
11.0/
31.3/ 17.2
53.6
42.99
23.1
20.3
25.2
GRPO
256
9.6/0.4
25.0/ 13.9
52.4
42.08
17.6
18.8
22.5
GSPO
256
9.0/0.2
30.0
40.65
18.8
17.3
22.6
M2PO (Ours)
256
10.4/4.4
33.8/ 17.8
52.0
44.48
21.2
18.1
25.3
Owen2.5-Mathh:
GRPO
39.6 17.5
63.8 46.7
82.3
64.1
36.7
43.6
49.3
GRPO
256
29.4/ 12.9
39.4
63.2
33.1
43.1
45.7
GSPO
256
27.3/ 13.1
63.8/380 36.7
79.0
62.2
33.5
41.9
44.7
M2PO (Ours)
256
33.3/ 17.5
63.8/40.
84.0
66.4
38.1
47.1
48.8
Diven3-Rase-I.78
GRPO
0
7.5
40.6 26.1
26.1
67.2
55.9
28.9
30.5
33.0
GRPO
256
8.5
35.0
64.3
52.7
26.0
27.6
30.4
GSPO
256
6.9
/
19.4/1855 18.9
65.0
53.1
26.5
27.5
30.1
M2PO (Ours)
256
6.5
48.1 27.8
71.8
59.5
29.4
35.6
36.6
Qwen3-Base-4B
GRPO
0
20.2
63.8 53.9
84.6
69.8
40.2
50.5
50.7
GRPO
256
14.0/9.6
51.9 32.8
76.8
61.7
34.4
39.8
40.1
GSPO
256
17.9 15.4
55.6 38.3
76.8
62.3
35.1
44.3
43.2
M2PO (Ours)
256
26.7 21.0
64.4/499
85.8
70.3
40.5
52.3
51.3
Qwen Baxee-B
GRPO
0
26.7 / 19.4
/
87.7
71.6
41.2
52.8
53.6
GRPO
256
21.0/ 13.1
63.8 1..840.. 40.0
81.8
67.8
38.5
47.4
46.7
M2PO (Ours)
256
30.2/23.1
71.3/ 56.7
87.2
75.1
42.6
54.8
55.1
Qwen2.5-32B
GRPO GRPO
24.4/ 18.3
71.9 46.7
71.9
41.4
52.9
51.6
256
20.4/ 9.6
68.1 44..1
83.0
67.3
40.9
45.9
47.0
M2PO (Ours)
256
24.8 / 19.4
/0.0
85.7
71.7
41.5
51.7
52.6
show that M2PO remains robust and effective, sustaining stable, high performance even under ex- treme off off-policy conditions.
In addition to the final accuracy comparison in Table T we also analyze the training dynamics of accuracy and reward of Qwen-2.5-328 2.B-32B models. As shown in Figure M2PO with 256 initially falls behind the on-policy baseline but quickly catches up, eventually matching its performance, while converging much faster and achieving higher accuracy than GRPO under the same staleness. This highlights that M2PO not only maintains comparable final accuracy
also accelerates con- vergence when training with stale data. Figure showss similar trend in the reward curves. M2PO with s = 256 also starts off behind the on on-policy baseline due to the initial plateau caused by using data generated from the base model, but it quickly catches up and aligns closely with the trajectory. In contrast, GRPO with s 256 consistently underperforms across the entire training trajectory.
Performance of other baselines under staleness. A number of prior works have proposed alter- native trust region strategies beyond e-clipping, including GSPO (Zheng al.) 2025a). AREAL (Ful et al. 2025), TOPR (Roux et al., 2025), GPPO (Su et al. 2025). and CISPO (Chen et al.) 2025). Despite not being designed to handle the extreme staleness studied in this paper, we also evaluate these methods under our setting with = 256 for completeness and comparison. Among these methods, GSPO is the only one that preserves training stability, though still exhibits noticeable
8

---

Published as a conference paper at ICLR 2026
Oeen 0a0e
9013
Ancrgge O
2.3
4.0
17th
Vodel Uindow
Notel disee
Mode Upponee
Green Sase- Swer2.5.33
(a)
(b)
(c)
(d)
Figure 6: (a) Methods comparison under staleness (s 256) on (lama3.2-Instrucce (b) Perfor- mance comparison between M2PO and GRPO on coding tasks. (c) Clipping ratio dynamics during RL on the Qwen-3-Base 1.7B model. (d) Comparison of the average clipping ratio across models
and methods.
performance drop under high staleness (see Table IT. For the other methods, as shown in Figure ba most encounter substantial difficulties in maintaining training stability and tend to break down early in training. These observations suggest that while existing approaches can be effective under moder- ately stale settings, they face significant challenges when extended to larger staleness, highlighting the need for a more robust and effective solution.
Evaluation results on coding tasks. We further evaluate M2PO on coding tasks beyond the math -iN-ssee--R---iiiil---eeeee
domain. Specifically, we train
DeepSeek-Al et al. (2025) on the
code.contests s dataset (Li al. 2022) and evaluate on LiveCode Bench (Jain et al. 2024).
We follow a training setup similar to that used for math tasks, except that each training iteration samples a batch of 128 prompts for rollout and 32 prompts for each model updates. We report @vg@4 4 accuracy as the evaluation metric.
As shown in Figure 6b M2PO with staleness = 256 outperforms GRPO at the same staleness level and achieves performance comparable to GRPO with s = O, consistent with our observations on math reasoning tasks. This further demonstrates the generality of M2PO on other tasks beyond math.
6.3 ANALYSIS AND ABLATION STUDY
Stable training with reduced clipping. Figure aa)(b) illustrates the clipping dynamics of GRPO and our proposed M2PO under different staleness settings. In Figure/bc) we report results on Qwen- B-Base-1.7B. Under large staleness (s = 256), GRPO exhibits frequent clipping events, with the ratio increasing sharply and remaining high during most of the training. In contrast, M2PO under the same staleness maintains an exceptionally low clipping ratio, comparable to or even lower than the on-policy GRPO baseline (s = 0). Notably, M2PO with s = 256 exhibits less clipping than O, which explains why M2PO with 256 achieves higher accuracy than GRPO with 8 = 0 in Table I Figure [ shows the same comparison on (wen-22.532D5 A similar trend holds: GRPO with s 256 suffers from substantial clipping, whereas M2PO effectively suppresses unnecessary clipping, remaining close to the on-policy baseline.
GRPO with
Figure Baure//d]ss summarizes the average clipping ratio across the entire training process. On Qwen-3-Base-ll 1.7B, GRPO with s 256
reaches an average clipping ratio of 0.66%, compared 0.07%
3.55
for GRPO with s = 0 and only 0.02% for M2PO with s = 256. On Qwen-2.5-32B. GRPO with
3.00
Uama3.2 Instruct-
256 averages 1.22%. while GRPO with = 0 records 0.05% and M2PO with maintains similarly low ratio of 0.06%. These results show that M2PO reduces clipping by over order of magnitude compared to GRPO, thereby enabling stable and efficient training by clip- ping only when necessary.
Collapss
256
0:30
032
604
6.30
600
0.0
Robustness to the choice TM.. Figure| 7 shows that perfor- mance is not highly sensitive to the choice of TM2- Accuracy re- mains stable across a broad range, only dropping when Taz is set extremely small (overly restrictive constraint) very large (train-
Figure/ 7: Ablation study of the TM, threshold on Llama- 3.2-3B-Instruct and Qwen2.5- Math-7B.
9

---

Published as a conference paper at ICLR 2026
ing collapse). This explains why single setting of TM2 0.04 works robustly across all training settings in our paper.
Training dynamics on KL and M2. Fig-
ure g shows the impact of M2PO mask-
0.50
Oriiial
MIPD Masked
2-2 GGPO
ing on training stability. Figure 8a shows
2.30
250 HOPO (Cums)
that the average M2 without masking ex- hibits frequent spikes throughout train- ing, indicating instability in the second- moment estimates (blue curve). Apply- ing M2PO masking effectively suppresses these fluctuations and maintains consis- tently low M2 values, leading to more sta- ble updates (red curve). Figure [8b] com- pares the KL divergence across different methods. Although M2PO with 8 = 256 involves substantially less clipping than GRPO (shown in Figure [6). it maintains more stable divergence than GRPO with 8 = 256. These results indicate that M2PO performs clipping in a more precise and adaptive manner, ensuring training stability with substantially less reliance on clipping. These results demonstrate that M2PO enables more pre- cise and adaptive clipping, achieving training stability while relying on significantly fewer clipping operations, and thereby attaining better performance without risking training collapse.
0.11
0.00
2000 Updates
Model Updates
(a)
(b)
Figure 8: (a) Average M2 in each model updates with and without M2PO masking on Qwen-2.5-32B, show- ing that masking effectively suppresses spikes and sta- bilizes the M2 throughout training. (b) Average KL divergence in each model updates Qwen-2.5-32B under different methods.
Commonly clipped tokens in GRPO. Figure 9 shows the specific tokens that are most frequently clipped by c- clipping. The word cloud of commonly clipped tokens is
it conndiiet wrrtte Given
gives also should
calculate
5o
estar
Given
This
To def
highly aligned with high-entropy tokens shown in (Wang] ctt al. 2025a): these tokens are not random or unimpor- tant, but rather belong to the most semantically and struc- turally critical elements in reasoning traces. Many of them (e.g., First, simplify, determine, To def, Thus, verify, break) are precisely the high-entropy "pivotal tokens" that initiate connect, or conclude key reasoning steps. Others (e.g.. assistant, user, code markers like ### or S$) serve
because solive
Sincer
For
uuiing
Therefore
simpliff
probler $
s stepresult
user
we a
assistantienn
The
our know Let'11
Tarst
b rr
reak
cons
"wil
Figure Word clouds of frequently clipped tokens with c clipping. as structural anchors in the dialogue or mathematical formatting. This observation aligns with Fig- ure 4b: as the importance weight ratio |r 1| grows, clipped tokens tend to exhibit higher entropy.
7 CONCLUSION
In this work, we investigated why off-policy RL for LLMs often fails under stale data and uncov- ered the prosperit-beforre-ollape phenomenon: training without a trust region initially outper- forms standard methods, showing that stale data can be as informative as on-policy trajectories, but eventually collapses due to instability. Motivated by this observation, we proposed M2PO, which constrains the second moment of importance weights to provide a variance-sensitive and sta- ble trust region. This design suppresses extreme outliers while preserving informative high-entropy tokens, enabling stable training that matches on-policy performance even under extreme staleness. Extensive experiments further demonstrate that M2PO significantly reduces clipping and is highly insensitive to its threshold, highlighting its practicality and scalability for efficient RL with LLMss.
ACKNOWLEDGEMENT
We would like to thank Cheng Luo, Xinyu Yang, Ranajoy Sadhukhan, Xuesheng Liu, Yongji Wu for providing constructive feedback our paper and computing resources of NVIDIA. This work was partially supported by Google Research Award, Google ML & System Junior Faculty Award, Amazon Research Award, Fireworks AI, Intel, Li Auto, Moffett AI, and CMU CyLab Seed funding. This material is also based upon work supported by the National Science Foundation under Grant No. 2504353 and IARPA. Any opinions, findings, and conclusions or recommendations expressed are those of the authors and do not necessarily reflect the views of the National Science Foundation.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
This work focuses on developing reinforcement learning algorithms for large language models. Our research does not involve human subjects, personally identifiable information, or sensitive data. All datasets used are publicly available and widely adopted in the community. We acknowledge that more capable LLMs may have potential societal impacts, including misuse for generating misleading or harmful content. To mitigate these risks, our study is confined to controlled academic settings, and our primary goal to improve the stability and efficiency of training methods.
REPRODUCIBILITY STATEMENT
All implementation details, hyperparameters, and experimental setups are thoroughly documented in the paper and appendix, providing sufficient information for independent reproduction of our results. Our code is publicly available at
REFERENCES
Charles Arnal, GaATtan Narozniak, Vivien Cabannes, Yunhao Tang, Julia Kempe, and Remi Munos. Asymmetric reinforce for off-policy reinforcement learning: Balancing positive and neg- ative rewards. arXiv preprini wrXiv:2506.20522 2025.
Art of Problem Solving.
Aime problems and solutions.
https://
artofproblemsen httpg//cooo../////...ee//////..a Sod
2024a. Accessed: 2025-04-20.
Art of Problem Solving.
Ame
problems and solutions.
https:
/artofproblemso.lvg wwt.://wwwwwwooooooooeeeo /wi http://wwwiiiieee/ttttt.. AMC_Probl eemooeemm__ee aadmssaand_ooo Solutioss 2024b. Accessed: 2025-04-20.
Aili Chen, Aonian Li, Bangwei Gong, Binyang Jiang, Bo Fei, Bo Yang, Boji Shan, Changqing Yu, Chao Wang, Cheng Zhu, et al. Minimax-ml: Scaling test-time compute efficiently with lightning attention. arXiv preprint arXiv:2506.13588 2025.
Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang, Yuxin Zuo, Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, et al. The entropy mechanism of reinforcement learning for reasoning language models. arXiv preprint arXiv:2505. 22617, 2025.
Dee,Seek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang. Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang. Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao. Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li. Jiawei Wang. Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang. Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang. Peng Zhang. Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. Chen, R. L. Jin, Ruyi Chen. Shanghao Lu, Shangyan yn Zhou Shanhuang Chen, Shengfeng Ye, Shiyu Wang. Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye, Tao Yun, Tian Pei, Tianyu Sun, Wang. Wangding Zeng, Wanjia Zhao Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang. W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xi- aosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang. Xinxia Shan, Y. K. Li, Y. Q. Wang, Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong.
11

---

Published as a conference paper at ICLR 2026
Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. Deepseck-rl: Incentivizing reasoning capability in Ilms via reinforce- ment learning. arXiv preprint arXiv:2501. 2025.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The Ilama 3 herd of models. arXiv e-prints, pp. arXiv-2407, 2024.
Wei Fu, Jiaxuan Gao, Xujie Shen, Chen Zhu, Zhiyu Mei, Chuyi He, Shusheng Xu, Guo Wei, Jun Mei, Jiashu Wang, et al. Areal: A large-scale asynchronous reinforcement learning system for language reasoning. arXiv preprint urXiv:2505.242900 2025.
Jiaxuan Gao, Shusheng Xu, Wenjie Ye. Weilin Liu, Chuyi He, Wei Fu, Zhiyu Mci, Guangju Wang, and Yi Wu. On designing effective rl reward at training time for Ilm reasoning. arXiv preprint 11115:2400.15115, 2024.
Zitian Gao, Lynx Chen, Haoming Luo, Joey Zhou, and Bryan Dai. One-shot entropy minimization. arXiv preprint arXiv:2505 20282, 2025.
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientifie problems. arXiv preprint arXiv:2402.14008, 2024.
Jingkai He. Tianjian Li, Erhu Feng, Dong Du, Qian Liu, Tao Liu, Yubin Xia, and Haibo Chen. History rhymes: Accelerating IIm reinforcement learning with rhymerl. arXiv preprint arXiv:2508. 18588, 2025.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. NeurIPS, 2021.
Jian Hu. Reinforcet+: A simple and efficient approach for aligning large language models. arXiv preprint 2rXiv:2501.03262, 2025.
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint MXXi:2403300777 2024.
Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? arXiv preprint 1000::3300.06700 2023.
Amirhossein Kazemnejad, Milad Aghajohari, Eva Portelance, Alessandro Sordoni, Siva Reddy, Aaron Courville, and Nicolas Le Roux. Vineppo: Unlocking rl potential for Im reasoning through refined credit assignment. arXiv preprint arXiv:2410.01679, 2024.
Woosuk Kwon, Zhuohan Li. Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Prin- ciples, pp. 61 1-626, 2023.
Aitor Lewkowyez, Anders Andreassen. David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ra- masesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Sole et al. Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Systems, 35:3843-3857, 2022.
12

---

Published as a conference paper at ICLR 2026
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d' d'Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel J. Mankowitz, Esme Sutherland Robson, Push- meet Kohli, Nando de Freitas, Koray Kavukcuoglu,, and Oriol Vinyals. Competition-level code generation with alphacode. Science, 378( 102-222244)9999-9999999 2022. doi: 10.1 106/sccscieecc. 158.
www. htt.psss/ww..cciee.o..rrr///////
abs
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lec, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. arXiv preprint arXiv:2305.20055 2023.
Liyuan Liu, Feng Yao, Dinghuai Zhang, Chengyu Dong, Jingho Shang, and Jianfeng Gao. Flashrl: 8bit rollouts, full power rl, August 2025a. URL /R//nttsss///eggyooooooia wtt.://w...o...oooteeeeeeeeeeeee
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding rl-zero-like training: A critical perspective. arXiv preprint 1iXiv:2503.20883 2025b.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2019.
Michael Luo, Sijun Tan, Roy Huang, Xiaoxiang Shi, Rachel Xin, Colin Cai, Ameen Patel, Alpay Ariyak, Qingyang Wu, Ce Zhang, et al. Deepeoder: A fully open-source 14b coder at 03-mini level, 2025a.
Michael Luo, Sijun Tan, Justin Wong. Xiaoxiang Shi, William Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Tianjun Zhang, Erran Li, Raluea Ada Popa, and lon Stoica Deepscaler: Surpassing ol ol-preview with 1.5b model by scaling rl. b75:oction.stte/Depsssaee Srg.ra/eeeee..tttaaaaaarareeeeer
radio--
a a-1-5B-
2025b. Notion
Blog.
Michael Noukhovitch, Shengyi Huang, Sophie Xhonneux, Arian Hosseini, Rishabh Agarwal, and Aaron Courville, Asynchronous rlhf: Faster and more efficient off-policy for language models. arXiv preprint arXiv:2410. 18252, 2024.
OpenAl, Aaron Jacch, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, Alex Iftimie, Alex Karpenko, Alex Tachard Passos, Alexander Neitz, Alexander Prokofiev, Alexander Wei, Allison Tam, Ally Bennett, Ananya Kumar, Andre Saraiva, Andrea Vallone, Andrew Duberstein, Andrew Kondrich, Andrey Mishchenko, Andy Applebaum, Angela Jiang, Ashvin Nair, Barret Zoph, Behrooz Ghor- bani, Ben Rossen, Benjamin Sokolowsky, Boaz Barak, Bob McGrew, Borys Minaiev, Botao Hao, Bowen Baker, Brandon Houghton, Brandon McKinzie, Brydon Eastman, Camillo Lugaresi, Cary Bassin, Cary Hudson, Chak Ming Li, Charles de Bourcy, Chelsea Voss, Chen Shen, Chong Zhang, Chris Koch, Chris Orsinger, Christopher Hesse, Claudia Fischer, Clive Chan, Dan Roberts, Daniel Kappler, Daniel Levy, Daniel Selsam, David Dohan, David Farhi, David Mely, David Robinson, Dimitris Tsipras, Doug Li, Dragos Oprica, Eben Freeman, Eddie Zhang, Edmund Wong, Eliz- abeth Prochl, Enoch Cheung, Eric Mitchell, Eric Wallace, Erik Ritter, Evan Mays, Fan Wang. Felipe Petroski Such, Filippo Raso, Florencia Leoni, Foivos Tsimpourlas, Francis Song, Fred von Lohmann, Freddie Sulit, Geoff Salmon, Giambattista Parascandolo, Gildas Chabot, Grace Zhao, Greg Brockman, Guillaume Leclere, Hadi Salman. Haiming Bao, Hao Sheng, Hart An- drin, Hessam Bagherinezhad, Hongyu Ren, Hunter Lightman, Hyung Won Chung, Ian Kivlichan, Ian O'Connell, Ian Osband, Ignasi Clavera Gilaberte, Iige Akkaya, Ilya Kostrikov, Ilya Sutskever, Irina Kofman, Jakub Pachocki, James Lennon, Jason Wei, Jean Harb, Jerry Twore, Jiacheng Feng, Jiahui Yu, Jiayi Weng, Jie Tang, Jieqi Yu, Joaquin Quiñonero Candela, Joe Palermo, Joel Parish, Johannes Heidecke, John Hallman, John Rizzo, Jonathan Gordon, Jonathan Uesato, Jonathan Ward, Joost Huizinga, Julie Wang, Kai Chen, Kai Xiao, Karan Singhal, Karina Nguyen, Karl Cobbe, Katy Shi, Kayla Wood, Kendra Rimbach, Keren Gu-Lemberg, Kevin Liu, Kevin Lu, Kevin Stone, Kevin Yu, Lama Ahmad, Lauren Yang, Leo Liu, Leon Maksin, Leyton Ho, Liam
13

---

Published as a conference paper at ICLR 2026
Fedus, Lilian Weng, Linden Li, Lindsay McCallum, Lindsey Held, Lorenz Kuhn, Lukas Kon- draciuk, Lukasz Kaiser, Luke Metz, Madelaine Boyd, Maja Trebacz, Manas Joglekar, Mark Chen, Marko Tintor, Mason Meyer, Matt Jones, Matt Kaufer, Max Schwarzer, Meghan Shah, Mehmet Yatbaz, Melody Y. Guan, Mengyuan Xu, Mengyuan Yan, Mia Glaese, Mianna Chen, Michael Lampe, Michael Malek, Michele Wang, Michelle Fradin, Mike McClay, Mikhail Pavlov, Miles Wang, Mingxuan Wang, Mira Murati, Mo Bavarian, Mostafa Rohaninejad, Nat McAleese, Neil Chowdhury, Neil Chowdhury, Nick Ryder, Nikolas Tezak, Noam Brown, Ofir Nachum, Oleg Boiko, Oleg Murk, Olivia Watkins. Patrick Chao, Paul Ashbourne, Pavel Izmailov, Peter Zhokhov, Rachel Dias, Rahul Arora, Randall Lin, Rupha Gontijo Lopes, Raz Gaon, Reah Miyara, Reimar Leike, Renny Hwang. Rhythm Garg, Robin Brown, Roshan James, Rui Shu, Ryan Cheu, Ryan Greene, Saachi Jain, Sam Altman, Sam Toizer, Sam Toyer, Samuel Miserendino, Sandhini Agar- wal, Santiago Hernandez, Sasha Baker, Scott McKinney, Scottie Yan, Shengjia Zhao, Shengli Hu, Shibani Santurkar, Shraman Ray Chaudhuri, Shuyuan Zhang, Siyuan Fu, Spencer Papay, Steph Lin, Suchir Balaji. Suvansh Sanjeev, Szymon Sidor, Tal Broda, Aidan Clark, Tao Wang. Tay- lor Gordon, Ted Sanders, Tejal Patwardhan, Thibault Sottiaux, Thomas Degry, Thomas Dimson, Tianhao Zheng, Timur Garipov, Tom Stasi, Trapit Bansal, Trevor Creech, Troy Peterson, Tyna Eloundou, Valerie Qi, Vineet Kosaraju, Vinnie Monaco, Vitchyr Pong. Vlad Fomenko, Weiyi Zheng, Wenda Zhou, Wes McCabe, Wojeiech Zaremba, Yann Dubois, Yinghai Lu, Yining Chen, Young Cha, Yu Bai, Yuchen He. Yuchen Zhang, Yunyun Wang. Zheng Shao, and Zhuohan Li. Openai ol system card, 2024. URL https: http:::://axxxiogg////oo abs 00000/2422.6722
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol- low instructions with human feedback. Advances in neural information processing systems, 35: 27730-27744, 2022.
Nicolas Le Roux, Mare G Bellemare, Jonathan Lebensold, Arnaud Bergeron, Joshua Greaves, Alex Fréchette, Carolyne Pelletier, Erie Thibodeau-Laufe Sándor Toth, and Sam Work. Ta- pered off-policy reinforce: Stable and efficient reinforcement learning for llms. arXiv preprint arXiv:2503. 14286, 2025.
John Schulman. The KL divergence between two close distributions. htttp///josch.neee blog/kl-apprro.hhet 2015. Accessed: 2025-11-18.
John Schulman, Sergey Levine, Picter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pp. 1889-1897. PMLR, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song. Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300. 2024.
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng. Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv: 2409.19255 2024.
Zhenpeng Su, Leiyu Pan, Xue Bai, Dening Liu, Guanting Dong, Jiaming Huang, Wenping Hu, and Guorui Zhou. Klear-reasoner: Advancing reasoning capability via gradient-preservinn clipping policy optimization. arXiv preprint NXX1:2508.006299 2025.
Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi kl. 5: Scaling reinforcement learning with llms. arXiv preprint 1rXiv:2501.12599, 2025.
Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang, Xionghui Chen, Jianxin Yang, Zhenru Zhang, et al. Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for Ilm reasoning. arXiv preprint arXiv:2506.01930 2025a.
14

---

Published as a conference paper at ICLR 2026
Xingyao Wang, Boxuan Li, Yufan Song, Frank F Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, et al. Openhands: An open platform for ai software 2024.
developers generalist agents. arXiv preprint
Yiping Wang. Qing Yang, Zhiyuan Zeng, Liliang Ren, Lucas Liu, Baolin Peng, Hao Cheng, Xuehai He, Kuan Wang, Jianfeng Gao, et al. Reinforcement learning for reasoning in large language models with one training example. arXiv preprini urXiv:2504.2057,, 2025b.
Yongji Wu, Xueshen Liu, Haizhong Zheng, Juncheng Gu, Beidi Chen, Z. Morley Mao, Arvind Krishnamurthy, and Ion Stoica. Rlboost: Harvesting preemptible resources for cost-efficient re- inforcement learning on llms. arXiv preprint urXiv:2510.19225, 2025.
Wei Xiong, Jiarui Yao, Yuhui Xu, Bo Pang, Lei Wang, Doyen Sahoo, Junnan Li, Nan Jiang, Tong Zhang. Caiming Xiong, et al. A minimalist approach to Ilm reasoning: from rejection sampling to reinforce. arXiv preprint arXiv:2504, 1343, 2025.
Ran Yan, Youhe Jiang. Tianyuan Wu, Jiaxuan Gao, Zhiyu Mei, Wei Fu, Haohui Mai, Wei Wang. Yi Wu, and Binhang Yuan. Areal-hex: Accommodating asynchronous rl training over heteroge- neous gpus. arXiv preprint arXiv:2511.0079) 2025.
An Yang. Beichen Zhang, Binyuan Hui, Bofei Gao. Bowen Yu, Chengpeng Li, Dayiheng Liu, Jian- hong Tu, Jingren Zhou, Junyang Lin, et al. Qwen2. 5-math technical report: Toward mathematical expert model via self- self-improvement. arXiv preprint vrXiv:2409.12222 2024.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint S5t1:22005.09888. 2025.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. Dapo: An open-source I1m reinforcement learning system at scale. arXiv preprint arXiv:2503.1444
2025.
Yufeng Yuan. Qiying Yu, Xiaochen Zuo, Ruofci Zhu, Wenyuan Xu, Jiaze Chen, Chengyi Wang, TianTian Fan, Zhengyin Du, Xiangpeng Wei, et al. Vapo: Efficient and reliable reinforcement learning for advanced reasoning tasks. arXiv preprint 2rXiv:2504.05//8. 2025a.
Yufeng Yuan, Yu Yue, Ruofei Zhu, Tiantian Fan, and Lin Yan. What's behind ppo's collapse in long-cot? value optimization holds the secret. arXiv preprint arXiv:2503.0149) 2025b.
Xiaojiang Zhang, Jinghui Wang, Zifei Cheng, Wenhao Zhuang. Zheng Lin, Minglei Zhang, Shao- jie Wang, Yinghan Cui, Chao Wang, Junyi Peng, Shimiao Jiang, Shiqi Kuang, Shouyu Yin, Chaohang Wen, Haotian Zhang, Bin Chen, and Bing Yu. Srpo: A cross-domain implementa- tion of large-scale reinforcement learning on 11m, 2025. URL
2550414288
Xiaotian Zhang, Chunyang Li, Yi Zong. Zhengyu Ying, Liang He, and Xipeng Qiu. Evaluating the performance of large language models on gaokao benchmark. arXiv preprint arXiv:2305. 12474, 2023.
Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, et al. Pytorch fsdp: experiences on scaling fully sharded data parallel. arXiv preprint arXiv:2304.1122 2023.
Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, et al. Group sequence policy optimization. arXiv preprint arXiv:2507.1 18071, 2025a.
Haizhong Zheng, Yang Zhou, Brian R Bartoldson, Bhavya Kailkhura, Fan Lai. Jiawei Zhao, and Beidi Chen. Act only when it pays: Efficient reinforcement learning for Ilm reasoning via selective rollouts. arXiv preprint arXiv:2506.022 02700 2025b.
Yimmin Zhong, Zili Zhang, Xiaoniu Song, Hanpeng Hu, Chao Jin, Bingyang Wu, Nuo Chen, Yukun Chen, Yu Zhou, Changyi Wan, et al. Streamrl: Scalable, heterogeneous, and clastic rl for 1lms with disaggregated stream generation. arXiv preprint arXiv:2504. 250.:::2200...9.3.... 2025.
15

---

Published as a conference paper at ICLR 2026
Zilin Zhu, Chengxing Xie, Xin Lv, and slime Contributors. slime: An Ilm post-training framework for rl scaling, httpps://g.ttt..cmm/// www.o/////////////e.e..e slime 2025. GitHub repository. Correspond- ing author: Xin Lv.
16

---

Published as a conference paper at ICLR 2026
A OVERVIEW
In this appendix, we provide additional details to complement the main text. Appendix B de- scribes the experimental setup in full, including datasets, models, and training hyperparameters. Appendix presents theoretical proofs supporting our method and analysis.
B DETAILED EXPERIMENTAL SETTING
Models & Datasets. To verify the effectiveness of our method, we extensively evaluate M2PO six models from four model series: Owen2.5-Math-775 (Yang al., 2024), Llama-3.2-3BB Instruct (Dubey et al., al.] 2024). Qwen3-Base-1.795 1.7/4//BBB (Yang et al. 2025). and Qwen2.5-32B (Yang al. 2024). For Qwen2.5-Math-7B, we use context length of 4k, which the maximum for this series. while for all other models the context length is set to 16k. For training, we adopt the Deep-
ScaleR (Luo al. 2025b) math dataset.
Training, Our method is implemented based on verl (Sheng et al. 2024) pipeline and uses vLLM (Kwon et al. 2023) for rollout. We use a mix of HI00 and H200 servers for training, de- pending on resource availability. We set the rollout temperature to 1 for vLLM (Kwon et al. (023). The training batch size is set to 256 prompts, and the mini-batch size to prompts. We sample responses per prompt. We train all models for 1000 steps, and we optimize the actor model using the AdamW (Loshchilov & Hutter 2019) optimizer with constant learning rate le-6. We use 31 = 0.9, 32 0.999, and apply weight decay of 0.01. We use the following question template to prompt the LLM. For reward assignment, we give a score of 0.1 for successfully extracting an an- and score of 1.0 if the extracted answer correct. Similar (Yu et al. 2025). remove the Kl--divergence term. The optimization is performed on the parameters of the actor module wrapped with Fully Sharded Data Parallel (FSDP) (Zhao et al. 2023) for efficient distributed training. We set the M2PO threshold to 0.04 for all training runs.
Evaluation. For benchmark datasets, use eight widely used complex mathernatiaa reason- ing benchmarks to evaluate the performance of trained models: Math500 (Hendrycks et al.. 2021
Lightman et al. 2023), AIME24/25 (Art of Problem Solving 2024a). AMC23/24 (Art of Problem
Solving 22024b) Minerva Math (Lewkowycz et al. (022). Gaokao (Zhang et al. (023). Olympiad Bench (He et al. 2024). Same as the training setting, For Qwen2.5-
models, we use 4k as the context length. For other models, we set the context length to 16k. Similar to (Wang et al.) 2025b. Zheng et al. (2025b), we evaluate models on those benchmarks every 50 steps and report the performance of the checkpoint that obtains the best average performance on eight benchmarks. We evaluate all models with temperature = 1. For AIME24/25, we report the pass@ I (avg@16), for other benchmarks, we report the pass@1(aug@4)
Baselines. For all bascline methods, we adopt exactly the same training setup and evaluation proto- col to ensure fair comparison. For trust-region settings, follow the recommended hyperparam- eters reported in the original papers whenever available. Specifically, for GSPO, we use 3e-4 and 4c-4 as the lower and upper bounds. For CISPO, we use clipping threshold of 0.2, the original paper does not specify a particular value; we also experimented with looser bound, but found that makes training significantly more prone to collapse. For AREAL, we use a clipping threshold of 0.2. For GPPO, we set the lower and upper clipping bounds to 0.2 and 0.28, respectively. TOPR does not contain any trust-region hyperparameters.
Question Template
Please solve the following math problem: {[Question Description} The assistant first thinks about the reasoning process step by step and then provides the user with the answer. Return the final answer in (boxed{) tags, for example (boxed (1). Let's solve this step by step.
C THE USE OF LARGE LANGUAGE MODELS (LLMS)
We used large language models (LLMs) as general- purpose assistants in two limited ways: (1) for writing polish, including improving grammar, readability, and presentation of the manuscript, and
17

---

Published as a conference paper at ICLR 2026
(2) code assistants (e.g., Cursor, GitHub Copilot) accelerate routine coding tasks such as de- bugging syntax errors and refactoring simple functions. LLMs were not used for research ideation, algorithm design, experimental analysis, drawing conclusions. All conceptual and scientific con- tributions are entirely the work of the authors.
D THEORETICAL PROOF
Proof of Theorem
Proof. Let logr, so that r = = Since 1/R < S R, we have |z| S log R.
For 2 0, observe that
e² -
/ e dt f E² dt =
2
which implies (e* 1)2 < (z =2622
For S 0, set v = - 2 0. Then
(e* 1)2 (1-e S u² =² S 2
Combining both cases, for all z E R we obtain
(e - 2
Substituting Z =logr, r, this yields
(r- 1)2 S (log (logr)²e² 2e2 21 I R2 (logr)²
Taking expectation under Thehav gives
x (TTew (bebav) E behar 1(r 1) S R² Eubehaa (log [| R²2 M2.
E THEORETICAL INSIGHTS ON PROSPERITY BEFORE COLLAPSE AND M2PO
In this section, we analyze why the unclipped objective is unbiased, while the s-clipped PPO ob- jective introduces bias by construction. This explains the observed prosperitybeefore-cllaa phe- nomenon.
Unclipped gradient is unbiased. Let the importance ratio be
To 8t)
r0 Tbchav (at ss)
The true policy gradient under off- off-policy sampling from Abelar is
VoJ(f)= = E, E (0) log st))
The surrogate objective without clipping
Lunclip (0) = E, (r))A)
whose gradient matches the true policy gradient,
V Lumctip (9) E, (r Vo Tog st),,
and is therefore strictly unbiased:
E 0 Luuccipp VoJ(
PPO e-clipping introduces bias. The clipped surrogate
Tecii(() (.(m cl),(r:(0), 8, e)))
18

---

Published as a conference paper at ICLR 2026
Consider the set of samples where (A, 0, rr()) > 1 + e). For these samples PPO replaces ny rx with a constant (1 E):
min(r;A,, (1 + =)..) (1 + =)
whose gradient is zero:
Vo A
However, the true gradient contribution from these same samples is
Vo(((A)) (0) log Ty (az st) O.
Thus the gradient contribution over this region is removed. Let 92+ U Ω denote the set of clipped samples. Then,
V Lel (0) = Beg((R /r log xo),
whereas the true gradient is
=0) Ar me].
We denote by Ωt and Ω the sets of timesteps where e-clipping becomes active:
2 {t A. 0, r0) 1 ++
Ω_ = {1 A. < 0, (0) < -
Then the difference introduced by clipping is
Ag( = (it(,_.u___(r(0))444 log me st)] * O,
which shows that PPO's clipped gradient is biased by construction whenever clipping occurs.
Combining these properties yields the observed prosperity- belorr--oollapee phenomenon: When the clipping is removed, the surrogate gradient becomes the unbiased policy gradient, enabling faster early-stage improvement. However, without a trust region, the importance ratios under off-policy rollouts quickly become heavy-tailed, causing the gradient variance grow rapidly, eventually lead- ing to training collapse.
The reason M2PO achieves better trade-off between performance and stability is that its masking strategy is more adaptive. By using the second moment to quantify batch-level distribution shift, M2PO automatically adjusts the effective trust region: when variance is small, it keeps the clipping region wide; when variance grows, it tightens the region accordingly. As a result, the measure of clipped samples, [2 J. is significantly reduced compared fixed e-clipping, leading smaller bias while still preventing variance explosion.
F DISCUSSION AND ANALYSIS
F.1 WHY DOES TOLERANCE TO HIGH STALENESS MATTER FOR RL ON LLMs?
High staleness tolerance is both practical and increasingly unavoidable for future RL for LLMs and for scalable RL system designs. Here are two key reasons:
High staleness tolerance enables training on more complex tasks, Complex tasks can substantially increase rollout latency. This is especially true for coding and agent environments, where tool calls and reward computation can be expensive, as both may require executing external programs, interacting with third-party APIs, or running large numbers of test cases to verify correctness.
In realistic settings, such SWE-bench Jimenez al.) (2023) with OpenHands Wang et (2024). rollout latency can become extremely large. A single run, even with a small batch size, the end- to-end inference time (ineluding tool call and code execution) can exceed 100 minutes, with more than 80 iterations with the environment. For reward calculation, verifying correctness against tens or hundreds of test cases also requires repeatedly compiling and running programs, adding another 5 20 minutes of reward computation depending code and test-case complexity.
As a result, the combined rollout + reward latency can easily reach 120 minutes or more. In contrast, policy updates in LLM RL, especially with sufficient parallelism, can be fast 60 seconds per update. Even without any additional model updates of staleness, and the staleness can be even larger for more complex tasks.
delays, this already corresponds to up to 120
19

---

Published as a conference paper at ICLR 2026
0.44
0.42
0.40
GRPO
GRPD + TIS
M2PO
M2PO + TIS
500
1000
2009 2500 3000 3500 4000 Model Updates
1500
Figure 10: Combining TIS with GRPO and M2PO Owen2.5-Math- with 256. Combined with TIS, M2PO shows a slight performance improvement, as TIS better mitigates the distribution gap between FSDP and VLLM. However, TIS alone cannot address the off-policy caused by stale- ness.
To enable future RL to support increasingly complex LLM agents, especially those involving fre- quent tool calls and substantial code execution, training under large staleness is practically necessary.
High staleness tolerance enables flexible and scalable system design. Modern LLM RL training
systems increasingly adopt decoupled and asynchronous designs improve throughput and reduce
cost. In such architectures, the ability to tolerate high staleness directly enables more flexible and
scalable RL pipelines for LLMs Fu et al. (2025); Zhu al. (2025); Wu al. (2025) 22025,, like heterogeneous computing and disaggregated computing systems.
Yan et al.
1) Heterogeneous Computing, As demonstrated in recent work Yan al. 2025 offloading roll- outs to slower but cost-efficient machines can substantially reduce the overall training expense. However, this will significantly increase the rollout latency, leading to larger staleness.
2) Disaggregated Computing. In large-scale RL training systems, rollout generation and policy updates can run on physically separated clusters or even across different data centers. Network bandwidth and latency can become huge bottleneck: synchronizing model weights and transferring rollout trajectories can further amplify staleness as the system scales.
To sum up, these factors make high staleness realistic and sometimes unavoidable operating regime in modern RL LLMs. Tolerance to high staleness therefore crucial not only for cost-efficient and flexible system design, but also for enabling RL on complex long-latency tasks without sacrific- ing throughput.
Motivated by this, use clean and controlled stale-k abstraction isolate the effect of data staleness on RL algorithms. Within this controlled but representative setup, we show the limitation of GRPO under high staleness, while M2PO remains stable and approaches on-policy performance. We believe this work provides algorithmic understandings of RL under high staleness and lays the foundation for future system-level integration.
F.2 How IS M2PO RELATED TO TIS (TRUNCATED IMPORTANCE SAMPLING)?
Another off off-po RL technique, TIS Liu et al. 2025a). addresses different source of off- off-colicy drift: the mismatch between the inference engine used for rollouts (e.g., vLLM, SGLang, or quan- tized engines) and the training engine (e.g., FSDP or Megatron). Despite that the weights are exactly the same, the engine difference on training and inference can introduce sampling differences and cause potential mismatch issues. This form of off-policy discrepancy is orthogonal to the source we study: data staleness.
More specifically, the optimization objective for TIS in (Liu al. 2025a is:
E (rotur) min (a, Ovehar) (Tutiaaa, Obehav)
V min
Misap (a, 0) Risap(a, (a, Obehav)
A, clip
Tisppa, 1- 1, Trsdp (a, Overnv)
We can see that the TIS objective introduces a truncated (or scaled) importance ratio to prevent excessively large ratios and stabilize training under inference-engine mismatch. However, for the
20

---

Published as a conference paper at ICLR 2026
off off-policy drift caused by staleness, TIS still relies fixed c-based clipping region to form trust region. contrast, the primary goal M2PO is to provide an adaptive, variance-sensitive mechanism to determine the trust region based on the second-mon moment statistics of importance ratios, without relying on hand-chosen c. In this sense, TIS and M2PO are complementary: TIS mitigates inference-engine mismatch; M2PO stabilizes training under stale data and large-degree off-policy drift. Here is formula combining TIS and M2PO:
G lod
Tlsam Eddp (o:1g)
Mi, Tg. Fudp (0il9)
[M)
min
C)
At,s,
(6)
1od
Thetor Iil (0)(9)
Traan ladp (0:19)
=1
1=
1=1
where Mi, the mask generated by Algorithm T
In Figure 10] we evaluate the combination of TIS with both GRPO and M2PO on Qwen2.5-Mann Mathh 7B under high-staleness setting (s = 256). When integrated with TIS, M2PO achieves slight performance improvement, TIS helps reduce the distribution gap between the FSDP training engine and the VLLM rollout engine. (That said, the gain modest because both training and rollout are performed in FP16, which keeps the engine-induced distribution shift relatively small.) However, TIS alone is insufficient to address the distribution shift introduced by large staleness.
F.3 COMPARE M2PO WITH ASYMMETRIC CLIPPING
We have conducted an additional experiment using the asymmetric clipping ratio (0.2, 0.28) used in DAPO for our setting and report the results in Figure IT Despite the asymmetric clipping ratio improving the GRPO per- formance, it still underperforms M2PO.
0.68
Accuracy
0.46
The fundamental distinction between M2PO and asym- metric clipping is twofold:
0.44
GRPO (0.2,0.2)
1) M2PO adaptively determines the trust region based on the actual distribution shift of each training batch. The second-momen trust region when the shift becomes large, and relaxes it when the shift is small, thereby preserving more infor- mative gradients. This per-batch adaptivity cannot be re- produced by any static c-clipping threshold: fixed up- per/lower bound does not respond to the evolving off- policy drift during training.
0.42
GRPO (0.2.0.28)
M2PO (Ours)
statistic automatically tightens the
0.40
500
1500
2030
2500
Model Updates
Figure 11: The performanace compari- son between M2PO and GRPO on cod- ing tasks.
2) Tuning clipping ratios is highly brittle and fundamentally impractical for different settings. The optimal asymmetric bounds vary across model sizes, staleness levels, and even across different phases of training. The tuning complexity of asymmetric clipping becomes even higher because both upper and lower bounds must be selected jointly (quadratic complexity). contrast, the M2PO threshold directly reflects the current batch's is distribution shift, which is the reason that the M2PO threshold is not sensitive, and we can use the same threshold all our experiments.
We believe that these two reasons illustrate why simply adjusting clipping boundaries is not suffi- cient for stabilizing stale off- off-policy training and why the community continues to develop principled trust-region algorithms rather than relying on hand-tuned clipping thresholds.
F.4 M2PO COMPUTATION COST ANALYSIS
Although we use per-token second-moment (M2) estimates to compute the batch-level M2 statistic, we do not require logits or probabilities over the entire vocabulary. Instead, My computed only from the sampled token each position (i.e., single-sample Monte Carlo estimate). Thus, M2 estimation adds no additional forward-pas computation and no additional tensor storage, avoid- ing the O (VVocabularySize) cost.
21

---

Published as a conference paper at ICLR 2026
(Ours)
40.0
32.5
30.0
500
1000
1500
2000
2500
3000
3500
4000
Model Updates
Figure 12:
Training time (i.e., update policy) comparison between GRPO and M2PO. M2PO mask- ing does not introduce a noticeable overhead in training stage.
Table 2:
Comparison of computation time between GRPO and M2PO. Loss computation contributes a negligible portion of the total training time.
Loss Compute Time (s) Other Training Time (s)
GRPO M2PO (Ours)
0.038 (0.11%)
34.86 (99.89%)
0.065 (0.19%)
34.43 (99.81%)
Besides, for the masking procedure, our implementation sorts tokens by their M2 values once and iteratively masks those with the largest M2 until the remaining tokens' average M2 falls below the threshold TM2 This procedure has a time complexity of O(N log N), where N is the number of tokens in a batch. In practice, is typically a few thousand, and this sorting step is negligible compared to the forward/backward passes of large language models. As illustrated in Figure [2] GRPO and M2PO show a similar training time, which indicates that M2PO masking does not intro- duce a noticeable overhead in the training stage.
We further provide a breakdown of the loss-computation overhead (including masking computation) during the entire training process. Table 2 below reports the average loss-computation time and the remaining training time (including forward and backward passes) for RL training on Qwen2.5- Math-7B. As shown, M2PO introduces a slightly higher computational cost than GRPO due the additional sorting required to select tokens with the highest M2 values. However, this overhead is extremely small relative the full training step and is far from the bottleneck. For instance, M2PO spends only 0.1% of total time on loss computation and does not slow down the overall training process.
Overall, both M2 estimation and the M2PO masking procedure incur minimal computational and memory overhead, and do not slow down training or increase storage requirements in our experi- ments.
F.5 HOw IS M2PO RELATED TO KL CONSTRAINT/LOOS?
The KL-divergence loss calculated in the Deepseek RI (DeepSeck-Al et al. (025)) is a low-variance approximation of the true KI. divergence between the old and new policies (Schulman 2015). which can partially mitigate off- aff-policy drift.
In our early exploration, we also experimented with adding KL loss. While this regularizer does enhance stability, we found that a trust region is still required to prevent collapse. Once trust region introduced, however, we immediately encounter the same challenge discussed in Q3: how select adaptive and effective trust region boundary that balances stability and performance. Simply tuning a KL penalty or fixed KL target does not adequately address the dynamic distribution shift introduced by stale data.
Moreover, the KL constraint and M2PO are complementary. The KL penalty controls global devi- ation between the new and old policies, whereas M2PO adaptively masks high-shift tokens based on bat-hatch moment statistics. Combining the two straightforward and can potentially further improve stability. In this work, intentionally did not include KI. penalty in main
22

---

Published as a conference paper at ICLR 2026
0.45
0.40
Accuracy
0.45
Accuracy
$=0
5=32
5=0
5=32
$=4
5=64
5=4
5=
0.42
128
0.42
128
2-2
256
500
1000
2500 Model Updates
1500
2000
3000
3500
4000
500
2000
1500
2000
2500
3000
3500
4003
Model Updates
M2PO
GRPO
Figure 13: The performance of M2PO and GRPO under different staleness Qwen2.5-Math-75
experiments. Our goal is isolate and understand the role of the trust-region mechanism itself un- der stale off-policy training. We will explore the combination KI. regularization and M2PO as a promising future direction.
G ADDITIONAL EXPERIMENTS
G.1 M2PO UNDER DIFFERENT STALENESS
In Table we compare the performance between M2PO and other baselines under = 256. To further study the robustness of M2PO across different staleness levels, we train Swen2.5-Math- with M2PO under = 0, 32, 64, 128, 256, as shown in Figure [3] We observe that larger staleness slightly slows down the initial accuracy gain during the very early phase of training, reflecting the expected delay when using outdated rollouts. However, this effect quickly diminishes: all curves stcadily improve and converge to nearly the same accuracy. Notably, even under extreme staleness par with the on-policy case (s 0), without signs of
(s 256), M2PO achieves performance collapse or degradation. These results highlight that M2PO effectively preserves the learning signal contained in stale data, enabling stable training and strong generalization across wide range of staleness values.
G.2 TRAINING COLLAPSE ANALYSIS
M2PO
25
M220
KL Divergence
Accuracy
0.35
M2PO
W/O
522
1000
2233 2005 Model Updates
1990
2000
2000 4000
560
1828
2000 2000 Model Updates
1006
2000
2000
4220
1000
1228 2203 Model Updates
2000
2000
2028
4202
Figure 14: Qwen2.5-32D training dynamics comparison on M2PO and collapse training of GRPO without trust region. Left: Test accuracy across eight math benchmarks. Middle: KL divergence between current policy and reference policy. Right: Graident norm.
In evaluation, deem training have collapsed when observe pronounced drop in test accuracy accompanied by abnormalities in stability metrics such as KL divergence and gradient norms. In practice, accuracy degradation typically paired with sharp KL spikes and gradient- norm blow-ups. As shown in Figure [4] test accuracy drops at the same time that KL divergence and gradient norms exhibit consistent spikes.
23


Reviews:
[{"review_id": "XdEXqWeUbL", "reviewer": "Reviewer_GUjw", "summary": "The paper investigates why off‑policy RL for LLMs degrades when trained on stale data (rollouts from earlier policies). It reveals a clear “prosperity‑before‑collapse” phenomenon: removing the trust region yields substantially higher performance than standard GRPO with epsilon‑clipping for a period, sometimes matching on‑policy results, but eventually becomes unstable and collapses. The authors diagnose that GRPO performs poorly under staleness because stale‑data training exhibits a much higher clipping rate, disproportionately affecting informative, high‑entropy tokens. To address this, they propose M2PO, a variance‑sensitive trust region that constrains the second moment of importance weights. M2PO suppresses extreme outliers while preserving signal on high‑entropy tokens, achieves stable training, matches on‑policy performance even under high staleness (e.g., 256 updates on Qwen‑2.5‑32B), dramatically reduces clipping, and is reported to be insensitive to its threshold.", "questions": "How exactly is “staleness of 256 model updates” defined? Is it the number of policy updates between rollout generation and training, or another measure?\nDo you track a policy‑distance metric (e.g., KL between rollout policy and training policy) and how does M2PO perform as this distance changes?\nWhat operational criteria define “collapse”? Reward crash, divergence of losses, KL blow‑up, or mode collapse?\nWhat stability diagnostics did you monitor, and can you show trajectories (e.g., gradient norms, importance‑weight second moments) leading to collapse?\nHow do the results vary across tasks, reward models, and model sizes? Can you share performance under different staleness levels beyond 256?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "+ Clear, timely empirical finding: prosperity‑before‑collapse highlights that stale data can be as informative as on‑policy trajectories, shifting the focus from data quality to algorithm design.\n+ Concrete diagnosis of GRPO’s failure mode under staleness: elevated clipping, especially on high‑entropy tokens, plausibly explains lost training signal.\n+ M2PO’s second‑moment constraint provides a simple, principled, variance‑sensitive trust region that stabilizes off‑policy training.\n+ on Qwen‑2.5‑32B with staleness 256, M2PO matches on‑policy performance while reducing clipping events and avoiding collapse。", "weaknesses": "No formal analysis of why the unclipped regime prospers then collapses, or why second‑moment constraints deliver stability.\nIncomplete experimental detail: definitions of staleness, collapse criteria, and full baseline configurations are not specified in the provided materials, making external validity hard to assess.\nIt is unclear how results vary across model sizes, tasks, reward models, and different staleness levels beyond the highlighted case.\nCompute matching, training schedules, and hyperparameter tuning parity between on‑policy GRPO, GRPO without trust region, and M2PO are not fully documented.\nDirect token‑level gradient/credit assignment analyses are not shown here to firmly establish the masking/clipping mechanism.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nkDN8ks689", "reviewer": "Reviewer_vuPK", "summary": "This paper investigates a critical challenge in scaling RL for LLMs: performance degradation when training with stale data in off-policy settings. The authors identify a \"prosperity-before-collapse\" phenomenon, where training without a trust region initially performs well on stale data before eventual collapse, suggesting that stale data contains valuable but hard-to-harness information. To address this, they propose M2PO, a novel policy optimization algorithm that constrains the second moment $M_2$ of the importance weights. M2PO employs an adaptive masking strategy to suppress only extreme, high-variance tokens, thereby preserving informative updates. The method is evaluated extensively across 6 model scales (1.7B to 32B) and 8 math reasoning benchmarks, demonstrating that M2PO achieves performance comparable to on-policy baselines even with data stale by 256 model updates, while significantly reducing the token clipping ratio and maintaining training stability.", "questions": "- The paper convincingly shows the superiority of M2PO over standard GRPO with symmetric clipping. However, a natural alternative is to use GRPO with asymmetric clipping bounds. Could the authors discuss the theoretical and practical relationship between M2PO and such an asymmetric GRPO variant? Why is M2PO's adaptive masking fundamentally different from simply tuning the clipping boundaries?\n\n- We note that the KL divergence calculation in DeepSeek-R1's GRPO implementation, $\\frac{\\pi_{new}}{\\pi_{behav}} - \\log \\frac{\\pi_{new}}{\\pi_{behav}} - 1$, also possesses the property of being more sensitive to large ratios compared to the standard KL. Given this, what is the distinct advantage of M2PO over such an improved, large-ratio-sensitive KL divergence used within the existing GRPO framework?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The proposed M2PO algorithm is built upon a well-motivated and sound theoretical insight. It identifies the limitations of KL divergence in measuring distribution shift — specifically its insensitivity to outliers due to cancellation effects — and introduces the second moment ($M_2$) as a superior alternative. The $M_2$ metric is both an effective measure of distribution shift and inherently sensitive to the high-variance tokens that are most likely to destabilize training.\n\n- The fact that a single threshold hyperparameter $\\tau_{M_2} = 0.04$ works robustly across all model sizes and tasks is a major practical advantage, greatly enhancing the method's usability.\n\n- The empirical evaluation is extensive and compelling. The consistent success of M2PO across six model families and scales under extreme staleness $s=256$ provides strong evidence for its robustness and scalability, demonstrating performance on par with on-policy baselines.", "weaknesses": "- While the paper compellingly demonstrates M2PO's superiority under extreme staleness $s=256$, it remains unclear if this advantage holds under more common, lower-staleness regimes (e.g., $s=8$ or $s=16$). A critical comparison in a moderate-staleness setting would be highly informative.\n\n- The empirical evaluation is currently confined to mathematical reasoning benchmarks. While this domain is a valid testbed, it features verifiable rewards and potentially specific characteristics in its token-wise importance ratio $r$ distribution. The claims regarding M2PO's general superiority and the universal robustness of its threshold $\\pi_{M_2}$ would be significantly strengthened by evaluation on other domains, such as code generation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DvY2UAXAEt", "reviewer": "Reviewer_D8e7", "summary": "This paper addresses the challenge of performance degradation in Reinforcement Learning (RL) for Large Language Models (LLMs) when trained on stale, off-policy data. The authors identify a \"prosperity before collapse\" phenomenon, where training without a trust region initially performs well but eventually fails.\n\nTo harness this potential while ensuring stability, they propose M2PO (Second-Moment Trust Policy Optimization). This algorithm constrains the second moment of importance weights, which selectively masks only extreme, high-variance tokens while preserving informative updates from stale data.\n\nExtensive experiments show that M2PO enables stable training even with data stale by 256 model updates, matching on-policy performance and significantly outperforming existing methods across various model scales and reasoning benchmarks.", "questions": "- Regarding the \"stale-k\" setup: Is the model using a policy from k training steps earlier to generate the current training batch, or is it directly using data collected k steps ago? If it’s the former, I suggest using the term \"stale model\" for clarity.\n\n- Why is such a large staleness (e.g., s=256) used? While off-policy training is an important challenge in RL post-training, techniques like Truncated Importance Sampling (TIS) [1] can help maintain on-policy behavior. Could the authors compare M2PO with TIS under similar settings?\n\n- Including an ablation study with different staleness levels for M2PO would help better understand its robustness and performance across varying off-policy gaps.\n\n[1] Your Efficient RL Framework Secretly Brings You Off-Policy RL Training. https://fengyao.notion.site/off-policy-rl", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper presents an interesting observation that training without clipping on stale data can initially outperform clipped training, highlighting a \"prosperity-before-collapse\" phenomenon.\n\n- The proposed method, M2PO, is intuitive and easy to follow. It selectively excludes tokens that cause extreme increases in the batch-level second-moment metric during loss propagation.", "weaknesses": "See Questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XgWrZqBxdO", "reviewer": "Reviewer_8kZV", "summary": "This paper introduces M2PO (Second-Moment Trust Policy Optimization), a novel off-policy reinforcement learning algorithm designed to address the challenges of training large language models (LLMs) with stale data. Current on-policy RL methods, while stable, are inefficient and unscalable due to their reliance on fresh rollouts. Off-policy methods, which decouple rollout generation from training, often suffer from performance degradation or collapse when data staleness is high.\n\nThe authors identify a \"prosperity-before-collapse\" phenomenon where stale data can initially be as informative as on-policy data if properly exploited, but existing methods like GRPO fail due to excessive clipping of informative high-entropy tokens. M2PO tackles this by constraining the second moment of importance weights, effectively suppressing extreme outliers while preserving valuable informative updates. This approach dramatically reduces token clipping under high staleness, leading to stable optimization.\n\nThe paper demonstrates that M2PO achieves performance comparable to on-policy GRPO even with data stale by at least 256 model updates, across six models (1.7B to 32B) and eight math reasoning benchmarks. M2PO is shown to be robust to its threshold hyperparameter, simplifying its practical application. This work highlights the potential of stale data in RL for LLMs and provides a more stable and efficient training solution.", "questions": "In my understanding, the experiments only involves using stable data (i.e., data sample with old behavior models) but doesn't involve any reusing of sampled data (e.g., one sample prompt can be used more than once in gradient updating), am I right?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper presents several strengths that would be beneficial for reviewers to consider:\n\n- **Addresses a Critical Problem in LLM Training:** The paper tackles the significant challenge of efficiently training LLMs with reinforcement learning. The reliance of traditional RL on fresh data (on-policy) is a major bottleneck for scalability, and M2PO offers a practical solution to leverage stale data.\n- **Identifies a Novel Phenomenon:** The \"prosperity-before-collapse\" phenomenon is an intriguing observation that sheds light on the potential of stale data. This insight forms the foundation of their proposed method.\n- **Proposes a Well-Motivated Solution (M2PO):** M2PO directly addresses the identified problem of existing methods inappropriately clipping informative tokens. By constraining the second moment of importance weights, the algorithm effectively differentiates between noisy outliers and valuable high-entropy tokens, leading to more stable and efficient learning.\n- **Strong Empirical Evidence:** The extensive evaluation across six different LLMs (ranging from 1.7B to 32B parameters) and eight math reasoning benchmarks provides compelling evidence of M2PO's effectiveness. The consistent performance parity with on-policy methods, even under extreme data staleness, is a key highlight.", "weaknesses": "1. Computational Overhead of M2PO and Masking: The computational overhead associated with calculating KL divergence, which can involve operations at the scale of $VocabularySize \\times TokenNumber$, and the iterative nature of the masking algorithm (Algorithm 1) are not thoroughly detailed. More explanation is needed regarding the practical implications of these computations, including tensor storage requirements and time complexity, especially when dealing with very large language models and extensive vocabularies.\n\n2. Limited Evaluation on Small-Degree Staleness and Practical Relevance of Extreme Staleness: The paper primarily focuses on evaluating M2PO under large-degree staleness (e.g., 256 model updates), while several real-world RL training setups and recent works (e.g., DAPO with 16 gradient updates per rollout, or AREAL uses even smaller degree of staleness) demonstrate that smaller degrees of staleness (e.g., 4, 8, or 16) can be sufficient for maximizing efficiency. This raises two important questions:\n\n   - How does M2PO perform under these more commonly encountered small-degree staleness settings? A comparison in these scenarios would provide a more complete picture of its practical utility.\n   - What is the necessity of large-degree staleness (e.g., 256) in real-world RL training for LLMs? Further exploration of this aspect could clarify the specific contexts in which such high staleness is beneficial or required.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)