ExGRPO: Learning to Reason from Experience
Authors: Runzhe Zhan
Yafu Li
Zhi Wang
Xiaoye Qu
Dongrui Liu
Jing Shao
Derek F. Wong
Yu Cheng
Venue: ICLR 2026 Poster (2026)
Keywords: Reinforcement Learning
Large Reasoning Model
Reinforcement Learning with Verifiable Rewards
Primary area: foundation or frontier models, including LLMs
TL;DR: We present a systematic study of what makes reasoning experiences valuable in zero RLVR and propose a framework that leverages these insights to exploit high-value experiences for efficient RLVR.

Abstract: Reinforcement learning from verifiable rewards (RLVR) is an emerging paradigm for improving the reasoning ability of large language models. However, standard on-policy training discards rollout experiences after a single update, leading to computational inefficiency and instability. While prior work on RL has highlighted the benefits of reusing past experience, the role of experience characteristics in shaping learning dynamics of large reasoning models remains underexplored. In this paper, we are the first to investigate what makes a reasoning experience valuable and identify rollout correctness and entropy as effective indicators of experience value. Based on these insights, we propose  ExGRPO (Experiential Group Relative Policy Optimization), a framework that organizes and prioritizes valuable experiences, and employs a mixed-policy objective to balance exploration with experience exploitation. Experiments on five backbone models (1.5B-8B parameters) show that ExGRPO consistently improves reasoning performance on mathematical/general benchmarks, with an average gain of +3.5/7.6 points over on-policy RLVR. Moreover, ExGRPO stabilizes training on both stronger and weaker models where on-policy methods fail. These results highlight principled experience management as a key ingredient for efficient and scalable RLVR.

Published as a conference paper at ICLR 2026
ExGRPO: LEARNING TO REASON FROM EXPERIENCE
Runzhe Zhan 12+ Yafu Li
Zhi Wang
Xiaoye Qu
Dongrui Liu?
Jing Shao?
Derek F. Wong Yu Cheng*
University of Macau "Shanghai AI Laboratory
NNanjing University
"The Chinese University of Hong Kong
nlp2ct. runzhe@gmail. com yafuly8gmail com zhiwangenju.edu.co {quxiaoye, liudongrui, shaojing]]ppiaal org. en derekfw@um edu. chengyuθcse. wwtssseecchk.. edu....u.....
ABSTRACT
Reinforcement learning from verifiable rewards (RLVR) is an emerging paradigm for improving the reasoning ability of large language models. However, standard on on-policy training discards rollout experiences after a single update, leading to computational inefficiency and instability. While prior work RI. has highlighted the benefits of reusing past experience, the role of experience characteristics shaping learning dynamics of large reasoning models remains underexplored. In this paper, we are the first to investigate what makes a reasoning experience valuable and identify rollout correctness and entropy as effective indicators of experience value. Based on these insights, we propose ExGRPO (Experiential Group Relative Policy Optimization), a framework that organizes and prioritizes valuable experiences, and employs mixed-policy objective to balance exploration with experience exploitation. Experiments on five backbone models (1.5B-8B parameters) show that ExGRPO consistently improves reasoning performance mathematical/genera benchmarks, with an average gain of +3.5/7.6 points over on-policy RLVR. Moreover, ExGRPO stabilizes training both stronger and weaker models where on on-policy methods fail. These results highlight principled experience management key ingredient for efficient and scalable RLVR.
Code: O
ExGRPO
Models:
ExGRPO Models
INTRODUCTION
Reinforcement learning (RL) has become pivotal technique for advancing the reasoning capabilities of language models on complex tasks (Guo et al., 2025; Yang et al., 2025). RL augments language models' reasoning by modeling chain-of- ohain-of-thouuht (CoT) as action sequences optimized under verifiable rewards (RLVR), laying the groundwork for sophisticated downstream applications (Gridach et al., 2025). However, significant yet overlooked challenge persists: due to the on-policy nature of most RLVR algorithms, the valuable experience generated during the rollout phase is often discarded after single gradient update (Shao et al., 2024). This practice not only squanders substantial computational resources but also forfeits critical opportunities for the model to learn from prior successful explorations, thereby imposing a bottleneck on scaling RL for reasoning.
Experience replay (Lin, 1992) is a widely adopted technique in RL to address this issue and improve sample efficiency. As stated in Silver Sutton (2025), AI is at the cusp of a new period in which experience will become the dominant medium of improvement. This idea, often referred to as experience-base -base RI., leverages the intuition that previously collected interactions (i.e., state-action- reward tuples) contain valuable information for learning. helping models stabilize training and accelerate convergence (Mnih et al., 2013). A non-trivial challenge lies in how to exploit past experiences based on their differing "values" (Schaul al., 2016) and manage the replay process according to customized learning schedules (Sujit et al., 2023). However, efficient experience replay mechanisms remain largely underexplored in RLVR for building large reasoning models (LRMs;
Work done during Runzhe Zhan's internship at Shanghai AI Laboratory. Corresponding authors.

---

Published as a conference paper at ICLR 2026
Plaat et al. 2024). Given the vast quantity of experiences collected during rollouts, fundamental question remains: How can the reasoning model effectively exploit its own stream of experience to maximize learning toward scaling RL compute for LRMs?
To address this gap, we begin by examining what constitutes a valuable reasoning experience for RLVR optimization'. We hypothesize that experience utility varies with measurable properties. RLVR experience generally consists of a question and its corresponding trajectories. Accordingly, we study experience properties from these two components. Through systematic analysis, we identify rollout correctness (for questions) and trajectory entropy (for trajectories) as effective online proxy metrics for characterizing experience quality. Specifically, tasks of intermediate difficulty and their associated low-entropy trajectories tend to be beneficial for RLVR optimization.
Building on these insights, we introduce Experiential Group Relative Policy Optimization (ExGRPO), a novel framework designed to strategically identify, manage, and replay valuable experiences. ExGRPO maintains a replay buffer of reasoning trajectories derived from partially correct rollouts and organizes them into buckets according their correctness levels. To manage the buffer effectively, uses a sampling strategy that prioritizes experiences from the most beneficial buckets, along with the corresponding trajectory with the lowest entropy. This approach allows the model to learn more efficiently from past experiences that align best with its current capabilities, guided by the principles of valuable experience concluded in our preliminary analysis. During mini-bateh optimization, ExGRPO uses a mixed-policy optimization objective that balances between leveraging fresh exploration and reusing strategically selected past experiences, improving both sample efficiency and training stability.
Our experimental results demonstrate that ExGRPO delivers improvements over on-policy RLVR baselines. We evaluate across five backbone models, spanning the Qwen (Yang al., 2024; Qwen et al., 2024) and Llama (Grattafiori et al., 2024) families from 1.5B to 8B parameters. on both mathematical reasoning benchmarks (AIME24/25, AMC, OlympiadBench, Minerva, MATH500) and out-of-distribus reasoning benchmarks (ARC-c, GPQA, MMLU- Averaged over all back- bone models, ExGRPO achieves +3.5 and +7.6 point gains? on in-distribution and out- -fstiirriiutti benchmark performance, respectively, compared to the on-policy RLVR. Notably, ExGRPO stabilizes RLVR training on the weaker Llama-3.1 8B model (Grattafiori al., 2024) and continual learning on the stronger LUFFY model (Yan et al., 2025), where on-policy optimization collapses. Detailed anal- ysis and ablations further confirm that improvements stem from ExGRPO's experience management and optimization mechanisms, which amplify the utility of past explorations.
RELATED WORK
Reinforcement Learning with Verifiable Rewards. Reinforcement Learning with Verifiable Rewards (RLVR; Lambert al. 2024) frames the language model as a policy that generates reasoning trajectories for verifiable tasks, with rewards provided by rule-based (Guo et al., 2025; Yu et al., 2025) or model-based verifiers (Su et al., 2025; Ma et al., 2025b; Chen et al., 2025a). Most RLVR methods adopt on-policy optimization (Schulman et al., 2017; Shao et al., 2024), which ensures stability but incurs high computational cost. Recent work has explored off-policy techniques (Kallus & Uehara, 2020; Meng et al., 2023) that incorporate historical or external data. Some approaches mix expert demonstrations with on on-policy updates, e.g., off off-policy policy gradients (Yan et al., 2025), SFT loss (Zhang et al., 2025b; Ma et al., 2025a), or knowledge distillation (Xu et al., 2025); others develop direct off-policy update rules for improved sample efficiency (Cohen et al., 2025; Roux et al., 2025; Arnal et al., 2025). However, they overlook the impact of data quality within the experience replay buffer, a factor that remains underexplored in RLVR and is the central focus of our work.
Experience-basee Reinforcement Learning. Experience replay (Lin, 1992) is a classical RL technique to improve sample efficiency and stabilize training. later extended with prioritized re- play (Schaul et al., 2016; Sujit et al., 2023) to emphasize informative transitions, enabling break- throughs in control tasks (Mnih et al., 2013; 2015; Lillicrap et al., 2016). For LRMs, recent studies show that replaying successful trajectories accelerates convergence and improves reasoning capabili- ties. ReMix (Liang et al., 2025) leverages off-policy experience replay to improve training efficiency; RePO adopts online replay (Li et al., 2025), collecting early on-policy rollouts and revisiting them asynchronously; RLEP (Zhang et al., 2025a) reuses trajectories from a well-trained policy; and
In the context of RLVR, the experience refers to a state-action-rewaan trajectory during rollout the reasoning chain. We will use the terms experiencottmmjeoooryvlllll interchangeably throughout the paper. "Due space limitations, numerical results of some hackhone models are reported in Section E.3.
2

---

Published as a conference paper at ICLR 2026
RRL (Dou et al., 2025) dynamically revisits promising early states; ARPO (Lu et al., 2025) extends this idea to GUI agents, combining GRPO with a replay buffer to enhance stability and sample efficiency. Among them, the last three methods overlook importance weighting correct distribution mismatch in off-policy updates. In this work, we argue that not all stored experiences are equally valuable in zero RLVR for LRMs. We systematically analyze properties that determine the value of past experiences and design methods to better leverage high-value experiences, thereby improving efficiency and performance.
PRELIMINARIES
In this section, we briefly review the RL foundations underlying our method: RLVR and Group Relative Policy Optimization (GRPO), upon which ExGRPO is built. We then present prelimi- nary analysis of experience data, i.e., the model's past successful rollouts, and examine how their characteristics influence performance.
3.1 REINFORCEMENT LEARNING WITH VERIFIABLE REWARD
Verifiable Reward Function. The verifiable reward compares the extracted answer from the model's output with a predefined golden answer. For example, the model instructed to output the final answer in format such as \boxed from which verifier extracts the value. Formally, given a model output for question q, the reward is:
{
if o contains the correct final answer to q
r(z o)
otherwise.
Group Relative Policy Optimization (GRPO). GRPO (Shao et al., 2024) is a strong baseline within the RLVR paradigm (Guo et al., 2025; Zeng al., 2025; Liu et al., 2025), achieving effective scaling without requiring an additional value model. It estimates the advantage of each trajectory by normalizing rewards within group of N sampled solutions. Given a group of K trajectories Cy {or using the current reference policy Tew (i.e, rollout policy), GRPO estimates the advantage of each trajectory o by normalizing its reward against the empirical statisties of the group:
A. r(g. 0:) ngg
22)
OG.
where Mo, = x Ee r(g, o,) and Øg, are the mean and standard deviation of rewards in the group Gn. The on-policy objective maximizes clipped surrogate function:
(L),(,;(O),AA)) A.)
JaRRO = En (l
(3)
K
where D is the training query set and the loss term is CLIP A)
To oo min Aw, clip(w;,1 -, A). Since the trajectory is generated by the rollout policy model before update, i.e., Teau- the per-token importance weight (f) is the probability ratio be- tween the current policy He and reference policy Tew (,,((0) T0(0,,19 (1,ct)/Te, (o., <1)))) Following Dr.RPOO (Liu et al., 2025), we remove the length normalization and standard deviation normalization of GRPO loss (Eqs. 2 and 3).
3.2 PRELIMINARY STUDY ON EXPERIENCE DATA
To motivate our experience management design, identifying which questions and trajectories most valuable for experience- baased learning, we begin by analyzing rollouts from on-policy RLVR. Our study is guided by two questions: (1) Are all questions equally useful for training, do certain difficulty levels provide stronger learning signals? and (2) Does low entropy correlate with higher-quality trajectories that the model should prioritize?
3

---

Published as a conference paper at ICLR 2026
Setup. We conduct experiments using Qwen2.5-Maue 7B (Yang al., 2024) backbone model, trained with vanilla on an-poiicy RLVR following the Dr.GRPO setup on the OpenRI-Math dataset (Face, 2025). Implementation details and hyperparameters are provided in Section 5.1. The experimental setting is as follows: we train three models, each using training batches restricted to questions of a particular difficulty level, determined by rollout correctness of each question. During training, collect the generated trajectories and analyze their characteristics, such as entropy and the quality (validity) of the underlying reasoning chains. The entropy refers to the average action entropy: (o) = - [o| E, [oz q, o<t) log (or q, 9ct), indicating the uncertainty of the model generating each action (token). Under this setting, three models are trained with comparable number of samples, and detailed statistics are provided in Appendix F.I.
On-Policy RLVR w/ Questions
Trojectory Cstegorization Entropy
Or-Policy RLVR w/ Questions
Benchmark Perforanna
46-
45.3
Full Easy
Medium Hard
Cornect Reworiing Col
Easy
Wrong Roasoring
Medium
44-
42.3
Entropy
0.14
Density
7,5
Hard
42-
41.1
41.6..
0.10
40
2.5
0.08
38
0.0
Full
Easy
Medium
Hard
Easy
Medium
Hard
0.0
0.2
0.4
Question Difficulty
Rollout Conrectness Buckets
Entropy of Correct Trajectory
(a) Test Performance.
(b) Reasoning Chain Validity.
(c) Entropy Distribution.
Figure 1: Analysis of question difficulty and entropy on-policy RLVR training: (a) Test performance of models trained on different question groups; (b) Entropy comparison between correct and incorrect trajectories; (c) Entropy distributions of logically correct trajectories across question groups;
Are all questions equally useful for training? During training, we categorize each question into one of three difficulty buckets based on its online (rollout) correctness rate: Easy [75%, 100%). Medium (25%. 75% and Hard (0, 25%). This classification is computed online without additional decoding or offline annotation. We implement bucket-specific experiments by restricting each batch to questions from a single difficulty group, producing three models, along with one trained on the full set. Results in Figure la show that training on Medium questions yields the largest performance gains. In contrast, models trained exclusively other questions perform substantially worse than the full-data baseline, yet they still provide complementary signals and should not be entirely discarded.
Does lower entropy imply better reasoning? We then test which metric can as an online proxy for reasoning quality. While outcome-based rewards check the final answer, they do not capture whether the reasoning CoT is logically correct or valid. To establish reference, we follow previous work (Wen et al., 2025) to employ an LRM (Qwen3-32B: Yang et al. 2025) as an external judge, which inspects correct outputs and labels their reasoning validity (See Section F.2). As shown in Figure 1b, correct reasoning trajectories exhibit lower entropy than incorrect ones, indicating that selecting the lowest- eeveest- candidate generally yields higher CoT quality. We further examine the entropy distribution of correct trajectories across the three question groups in Figure Ic. Correct trajectories in the Medium group concentrate at lower entropy values, while Easy and Hard groups display broader distributions, with Hard problems peaking at higher entropy. This observation partly explains the marginal advantage of training on Medium- level questions compared to the other groups.
In summary, we highlight two guidelines: Medium-difficule questions provide the most valuable optimization signals, and entropy minimization is an effective heuristic for trajectory selection.
Low-entropy trajectories mitigate misguided learning. Entropy magnitude (Chen et al., 2025b) is a key signal in RLVR training. Prior work shows that aggressively minimizing entropy can cause collapse (Cui et al., 2025b), motivating a more balanced exploration eetiiiitttxooiiiiinnnnn strategy in experience management. While high-entropy policies encourage exploration, they can yield low-quality trajectories, many successes are merely lucky hits from incorrect reasoning chains.
Repeatedly sampling such misguided experiences from the replay buffer contaminates training, leading to systematic reasoning errors, we refer to snowball effect. For instance, illustrated in
4

---

Published as a conference paper at ICLR 2026
(a) Experience Management.
Collection
(2)
Optimization
Experience Ioeer
Experience Replay Experiecc Itens
Sampling
+50%
Question
Sample
Experience Metric
Action
Rollout Trajectories
[o")
Trajectory Solection
Policy Model
X
Rerioed
Online
Opdate
ExDlroo Mieed-fllicy Opimizution jlafer
(b) Experiential Policy Optimization.
Mini Botch
Rolluu Trajecterics
Policy Probability
Reploy Buffer
[q*
Og-1
o+
Policy Model
Action
Importance Weighting
Advantage
Question
(q)
Updede by fucssefft
Overview of Experiential Group Relative Policy Optimization (ExGRPO). ExGRPO operates in two phases: (a) Experience Management and (b) Policy Optimization (cf. Algorithm I).
Figure 2:
Section F.4, we observe models using invalid code blocks for math problems. traced to high-entropy trajectories in the buffer. Thus, entropy minimization should be a guiding principle for experience selection, especially early in training when stability matters most. However, balancing entropy reduction with exploration remains a non-trivial challenge, which we will detail in the methodology.
4 METHODOLOGY
Based previous findings, propose Experiential Group Relative Policy Optimization (ExGRPO) as shown in Figure 2, a method that leverages structured experience replay with entropy- based trajectory selection to enhance sample efficiency while preserving policy stability.
4.1 EXGRPO: EXPERIENCE MANAGEMENT
Problem Formulation. Given a dataset and reward model r(-), ExGRPO iteratively updates policy model mo by combining on-policy rollouts Egg Fyp=D,onwa..,.11 [r(q,o) and experience replay (-4o (o "r(g","") The Thas is the rollout policy model before gradient updates, while Tip is the previous one which generates the corresponding experiential trajectories. The buffer & is maintained as key-value structure q² + {o'), where each question is associated with set of candidate trajectories. We next describe the three-stage experience management pipeline.
Experience Collection. During training, we collect the model's successful trajectories for each question in the batch. For given question sampled from the dataset, the rollout policy Taw generates K trajectories, denoted as G {ot \ We then compute the correctness rate Aco(g") k/ K, where l is the number of successful trajectories verified by the reward model. All successful rollouts are stored in the replay buffer ε, recorded as a mapping q" + {o*}. The associated correctness rate saved together with the query and later used for experience partition.
Experience Partition. The replay buffer is partitioned into buckets by each question's 's latest correctness rate Acc(g"), with q as the indexing key. Prior work (Zhang & Zuo, 2025) reweights advantages based on correctness rates, whereas we leverage them to bucket experiences by difficulty. To prevent overfitting on trívial cases, we introduce Retired Sel: questions solved in all rollouts are removed from E, ensuring optimization focuses on partially solved questions. Xiong et (2025) also filters prompts by online correctness. In contrast, our retired-set strategy excludes mastered experience items.
Experience Selection. During optimization, ExGRPO retrieves a mini mini-batch of experiential samples from partitioned buffer & in two steps:
5

---

Published as a conference paper at ICLR 2026
(I) Question Sampling. To prioritize experiences described in Section 3.2, each bucket is sampled with probability p & [A(cc(q"); μ. = 1), where y is set to 0.5 in practice. This ensures that medium-difficule questions sampled more often than trivially easy mostly failed ones, allowing biases sampling towards the "sweet spot" questions.
(2) Trajectory Selection. For each sampled question, select the trajectory with the lowest entropy under the current policy without relying on an external judge, i.e., o < arg e (o;; To). thereby prioritizing trajectories which indicate better CoT quality discussed in Section 3.2.
After three-stage management, selected experiences are used for mixed-policy optimization.
4.2 EXGRPO: EXPERIENTIAL POLICY OPTIMIZATION
The core idea of ExGRPO optimization is unify on-policy exploration with off soff-polcy replay under a joint objective, while correcting the distribution shift. At each step, a mini-batch B is constructed from: (1) on-policy samples B.n. from dataset D, and (2) experiential samples Beep from buffer &. A hyperparameter E [0. 1) determines the experiential proportion, with Bexl min[[eB], (8)) and Bool B eeppl This ensures lower bound in cases where the experience replay buffer is depleted. The overall ExGRPO objective is a combination of two components:
The on-policy objective (0) follows GRPO in Eq. for each query q ~ B.n. K trajectories {o are sampled from policy Vow. to form an advantage group 9g. with objective JaRro Gg: f).
For the experiential off-policy objective, given q" Bexp. we form a mixed advantage estimation group Gy = {o²} U {o.) where o* a replayed trajectory from past policy Te,- and {or new rollouts from current reference policy Tow To obtain unbiased policy gradient estimate To "a(o]ln" .02+)
from the replayed trajectory, o* is reweighted by w (0) = Fopu (0" "" All trajectories in Gy- use
GRPO advantage estimates A in Eq. Finally, the unified objective is:
1
(Es)RPO( = (1 p) Eg Bun K
Σ (L), A(o,,
Expansion of On-policy objective 99:9
K-1
(LIP(WW 7 , +
Σ (L), Mor, gq-
+ p Egg BapenBBu BBp
K
Experiential Off Policy objective
(4)
Directly optimizing on replayed trajectories, which are often selected for their low entropy, can hurt the exploration in the RLVR. We introduce two mechanisms to ensure stable and effective learning:
Policy Shaping for Entropy Preservation. Inspired Yan et al. (2025), modulate the gradient from experiential data using policy shaping. We replace "CLIP" term of importance sampling for o* in Eq. 4 with (w (0)) A(o", Gy-), where f(x) z+0 is non-linear transformation with a small constant 3 = 0.1. This amplifies low- www-robability signals and dampens high-probability ones within the replayed trajectory, encouraging the model to learn from its more novel aspects from experience.
Delayed Start Mechanism. To mitigate the collection of low-quality trajectories during the initial training stages when the model's capabilities are still developing, we employ a delayed start. The on-policy RLVR process runs first, and ExGRPO is activated only after the Pass / metric for training batch surpasses a predefined threshold, ensuring experiential samples are of higher quality.
EXPERIMENTS
5.1
EXPERIMENTAL. SETUP
Data and Evaluation. We train on the OpenRI-Mate 45k subset (Face, 2025; Yan et al., 2025) and evaluate nine challenging reasoning benchmarks. Six in-distribution math benchmarks:

---

Published as a conference paper at ICLR 2026
Table 1: Overall in-distribution and out-of-distribution performance based on Owen2.5-Math-77
Bold and underline indicate the best and second-best results within comparable group, respectively.
In-Distribution Performance
Out-of-Distribution Performance
Model
AIME24
AIME25 AMC MATH-500 Minerva Olympiad Avg.
ARC-e GPQA' MMLU-Pro Avg.
Qwen- Base
11.5
31.3
43.6
7.4
15.6
19.0
18.2
11.1
16.9
15.4
Instrnnstruct
12.5
10.2
48.5
80.4
32.7
41.0
37.6
70.3
24.7
34.1
43.0
Previous Zero RLVR Methods
PRIME-Zero
32.7
Oat-Zero
33.4
11.9
61.2
78.0
34.6
43.4
43.7
70.1
23.7
41.7
45.2
GPG-Zero
29.8
12.1
67.8
80.8
30.9
44.7
44.4
70.3
40.4
50.5
41.6
RePO-Zero
19.8
10.2
54.0
76.8
34.2
40.1
39.2
73.8
24.2
42.5
46.8
Zero RLVR with ExGRPO
On-Policy ExGRPO
24.9
15.5
59.2
84.8
38.2
49.3
45.3
82.6
37.4
49.2
56.4
31.6
66.3
87.4
36.0
Off policy Learning Methods
SFT
22.2
22.3
52.8
82.6
40.8
43.7
44.1
75.2
24.7
42.7
47.5
SFT+RL
25.8
23.1
62.7
87.2
39.7
50.4
48.2
72.4
24.2
37.7
44.8
Continual RLVR with ExGRPO
LUFFY
29.4
23.1
65.6
87.6
37.5
57.2
50.1
80.5
39.9
53.0
57.8
Continual LUFFY
30.7
22.5
66.2
86.8
41.2
55.3
50.4
81.8
49.0
54.7
61.8
On-Policy
24.8
17.8
67.5
88.4
38.6
55.3
48.7
81.9
47.0
53.3
60.7
ExGRPO
32.3
25.7
65.6
87.6
40.1
$7.0
51.4
83.6
42.4
54.5
60.2
AIME 2024/2025, AMC (Li et al., 2024), MATH-500 (Hendrycks al., 2021). Minerva (Lewkowyez et al., 2022), and OlynpiadBench (He et al., 2024). To assess generalization, we include three cuttof-diistrbbution tasks: ARC-c (Clark et al., 2018). GPQA-Diamond ("GPQA""; Rein al. 2024), and MMLU-Pro (Wang et al., 2024). For AIME, AMC benchmarks with a limited number of samples, we report Avg over 32 independent runs; for others, we use Pass@ metric, representing the proportion of solved problems on single attempt. All evaluations use a sampling temperature of 0.6 and a to top-p of 1.0, with answers extracted via the Oat-evaluator toolkit (Liu et al., 2025).
RLVR Setups. We use rollout batch size 128 and an update batch size of 64. During the rollout stage, collect 8 sampled responses for on-policy questions and for questions drawn from the replay buffer. The ratio p of experience data in each mini-batch is set to 50%. For most models, the ExGRPO algorithm is activated only after the batch Pass exceeds 35%. An exception is the Llama model, where this criterion not applied due to the collapse of on-policy RLVR. Our experiments were conducted using the verl framework' either 8xH100, with Math-Verife as the reward model. More detailed training and validation settings are provided in Section E.I.
Models and Baselines. Our primary testbed for zero RLVR is the Qwen2.5-Math 7B model, and we also extend our evaluation its 1.5B variant and other models, including Llama-3.1 8B (Base and Instruct) and Qwen2.5 7B Instruct models. Details of all baselines can be found in Section E.2.
5.2 MAIN RESULTS
ExGRPO surpasses on-policy baselines complex reasoning benchmarks. Our main results in Table show that ExGRPO consistently outperforms on-policy RLVR baselines across multiple benchmarks. When applied to the Qwen2.5-Matt 7B model, ExGRPO delivers +3.0 point gain over on- on-policy RLVR on math reasoning tasks. This advantage becomes more pronounced on challenging
7

---

For each query q with K rollouts, og has no intrinsic scale, we map it to a discrete difficulty bucket using relative, batch- wise normalization. Given all variances (M, within the current batch, we ap- ply a linear min-max normalization oy when the batch variance collapses. We then discretize og E [0, 1] into {o...., Krollom round(ō, Kyollout) Each query then assigned to its corresponding bucket and up- dated using the ExGRPO mechanism originally developed for the corery-sss setting. Under this formulation, buckets still preserve the nature of discrete difficulty levels. As re- sult, ExGRPO's sampling mechanism, which was previously centered around medium suc- cess rates, now preferentially selects queries with moderate reward variance, i.c.. those that are neither overly stable nor excessively noisy. Published as a conference paper at ICLR 2026
datasets such AIME24/25, highlighting its effectiieness difficult reasoning problems, and also holds on out-of out-of-disttibution benchmarks. A detailed comparison with the relevant replay-enhance method in Section E.4 further shows that ExGRPO outperforms vanilla experiential RL baselines.
100
Model Family (Color)
In-Distribution
Diatrf--Distibbution
Llama3.1-88
80
Llamm3.1-89 Instruct
Jenchmark Score
Qwen2 1.58
60
Qwen2 Instruct
58
Variant (Shade)
47.8
Base / |Instruct
40
On-Pollcy
35.0
36,0
ExGRPO
20-
12.4
0
AIME24
AIME25
AMC
MATH--50
Minerva
Olympiad
ARC-c
CPQA*
MMLU-Pro
Figure 3: A comparison of benchmark performance for different backbone models and training variants, showing performance on both in-distribution and out-of-distribution tasks (cf. Section E.3).
ExGRPO is robust across diverse model architectures and initializations. We validate that the benefits of ExGRPO are not limited to single model in Figure 3 and Appendix Table 5. ExGRPO has consistent improvements over baselines when applied to models of varying scales, including 1.5B and 8B parameter models, as well as models initialized from instruction models, ExGRPO achieves in-distribution improvements of up to +5.3 points on Qwen2.5-Mai 1.5B Base and +4.1 points on Qwen2.5-7B Instruct. Notably, ExGRPO addresses a critical instability issue observed in on-policy RLVR, enabling successful training of Llama-3.1 basc model our data where the on- on-policy baseline method collapses. a further experiment using the LUFFY model, which was trained with external off- off-policy data, find that continual RLVR with model own experience yields greater performance gains than relying on external data. However, applying on-policy training to LUFFY still resulted in degradation, highlighting the advantage of ExGRPO. We observe that EXGRPO underperforms on certain out-of-distribution tasks under continual RLVR setting, which we attribute to reduced on-policy exposure during joint mini-batch optimization with experience replay.
checkpoints. For these
5.3 EXTENSION TO RL WITH CONTINUOUS REWARDS
Beyond the beward RLVR setup, extending ExGRPO to standard RLHF with continuous preference-modee rewards is a vital next step. In this setting, we replace rollout success rates with the within-query reward variance computed over multiple rollouts. Reward variance serves as natural signal of difficulty, and trajectory entropy remains applicable as a measure of reasoning validity.
compute the reward standard deviation Tg- Since
max(Aaas 5m where prevents degeneracy
d--0min
Table 2: Preliminary results extending ExGRPO to RL with continuous rewards.
Method
AremaHardV2
IFEval
IFBench
Avg.
GRPO
15.1
26.1
18.7
20.0
ExGRPO
16.4
27.9
20.7
21.7
These correspond to intermediate-diffieityy queries where learning gains often most substantial.
8

---

Published as a conference paper at ICLR 2026
To validate feasibility, conducted study 7k WildChat-IF samples (Zhao al., 2025; Bhaskar et al., 2025) with on-policy RL training (2 epochs), using Qwen2.5-7B- B-Bess as the base model and 3.1-8B-v0.2 (Liu et al., 2024) the reward model. Results chat and instruction-following benchmarks show a improvement of ExGRPO extension.
ANALYSIS AND DISCUSSION
To understand the effectiveness of ExGRPO, we investigate the following research questions: 1) How does experience replay influence training dynamics, especially for different models? 2) What is the impact of experience selection on overcoming performance bottlenecks? 3) How do the components of ExGRPO, e.g., experience management, contribute to overall performance?
8000
Obi Policy
0.3
EvGRPO
Respoose Length
Avg. Reward
Entrogy
6000
Fotile Exploration
0.2
Entropy
7.5
Explosion
4000
0.1
No Gains
5.0
2000
2.5
Training Steps
Training Steps
Training Steps
Figure 4: Learning dynamics of On-Policy ExGRPO during training Llama-3.1 8B. ExGRPO stabilizes training and achieves higher rewards, while on-policy suffers from training collapse.
6.1 TRAINING DYNAMICS
Experience replay helps stabilize training for weaker mod- els. Figure 4 depicts the training dynamics of the Llama3, 1-8B base model. Due to its limited capacity, this model struggles to solve most problems, resulting in low rewards under on-policy training. Consequently, the exploration signal collapses, and the model finally encountered entropy explosion challenging training data. In contrast, experience replay allows the model to exploit the "lucky hits" of correct solutions encountered early on. By replaying these successes, the model receives meaningful reward signals and is guided toward an exploration trajectory that matches its current capability, thereby stabilizing learning and preventing premature collapse.
Buffered Iistinces
43000
Ttximing Instonces
62000
moch
23000
0-
100
200
-003
500
500
700
Training Step
Figure 5: Dynamics of experi- ence replay buffer and retried set.
Experience replay improves data efficiency. For stronger model, Figure shows the evolution the replay buffer and retired set. Their which the model eventually succeeds. By the later stages, about half of the dataset achieves at least successful trajectory sP ss @), validating the promise of experien- tial learning: the model retains strong reasoning potential, and revisiting past successes enhances data utilization. The retired set further enhances efficiency: although initially small, it grows rapidly capability improves and eventu- ally matches the buffer in size. Retiring solved problems
4 Bufferred [x 105)
EAGRPO
reflects the number of examples on
Quess
p-73%
p-25%
100
200
300
400
500
600
700
Training Steps
Retiied (x 106)
ExGRPO Sel
p-78%
p-12%
shifts training toward harder ones, reducing redundancy and explaining the slower buffer growth later phases.
200
300
500
700
The efficiency of experience utilization, rather than its sheer volume, is the primary driver of model perfor- mance. Figure 6 illustrates the dynamics of the experience der different data conditions.
Training Steps
Figure 6: Dynamics of experience un-
buffer and the retired set under different data conditions. When trained on the same amount data, the absence of question selection mechanisms (e.g.. ExGRPO vs. w/o Ques. Sel.) leads to growth of the retired set and a corresponding reduction in buffer size (primarily due to repeated exposure to easy questions). As result, experience replay becomes inefficient, with high consumption of

---

Published as a conference paper at ICLR 2026
samples but low return in terms of performance gains. We further examine the scaling effects of the experience replay ratio, p. A high ratio (p 75%) leads to over-exploitation, stifling exploration and degrading performance below the on-policy baseline. Conversely, low ratio (p 25%) prioritizes fresh on-policy learning; however, while it enlarges the experience buffer, the retired set grows only marginally. This indicates that accumulating more experiences does not guarantee improved capabilities. Ultimately, effective replay relies on balancing exploration with efficient sample reuse rather than simply maximizing buffer size, with optimal performance achieved at balanced p 50% (see results in Appendix Table 6).
6.2
ABLATION STUDIES
48
Empirical ablation supports ExGRPO's selection heuristics. We conduct an ablation study to assess the contributions of core components of ExGRPO. Figure 7 shows performance dynamics of validation set (see Sec- tion E.1). For both question and trajectory selection, our analytically derived heuristics consistently outperform random selection, while random selection itself offers marginal gains over the on-policy baseline. Policy shap- ing mitigates this by ensuring exploitation does not hinder exploration. It is also noteworthy that applying shaping alone in on-policy RLVR yields no improvement (Yan et al., 2025), confirming that ExGRPO's s gains cannot be attributed to naive shaping. The above results and additional ablations on test set (see Section E.4) provide empirical support for our design of ExGRPO.
44
Validation Score
42
40
38
On-Pellry DEGRPO
36-
34-
EsGRPP w Shaping
EsCRPPO w/o Ques, Seletionn
EXGRIP w/o Ques, Traj. Selection
100
200
300
400
500
600
200
Training Steps
Figure 7: Comparison of validation per- formance of different ExGRPO variants.
Sensitivity to difficulty sampling strategies. Our method employs a probabilistic sampling scheme 0.5, o). To assess the sensitivity of ExGRPO
based on Gaussian distribution p x (Acco(g); to the width of this distribution, we vary a E (0.5, 1.5). A smaller a corresponds to a narrower near-uniform sampling distribution. We medium-difficull band, whereas larger approaches also evaluate a Bard-Biased variant by shifting the mean to μ4 = 0.25, which emphasizes harder samples (25% accuracy) while reducing the chance of revisiting easier ones (75% accuracy). As shown in Table 3, the asymmetric sampling strategies yield inferior performance compared with the symmetric -riffiuddiffiuutty focus (μ 0.5), confirming that balanced uncertainty regions provide the most effective signal.
Table 3: Sensitivity analysis of the Gaussian sampler with varying sampling focus and bias. Bold indicates the best results across settings for each column.
In-Distribution Performance
Out-of-Distrlbution Performance
Sampling Strategy
AIME24
AIME25
AMC MATH Minerva Olympiad Avg.
ARC-c
GPQA'
MMLU-P
Avg.
On-Policy
24.9
15.5
59.2
84.8
38.2
49.3
45.3
82.6
37.4
49,2
56.4
ExGRPO, (n=0.55 o=
31.6
18.7
66.3
87.4
36.0
50.1
48.4
84.7
37.4
52.9
58.3
Wider Focus (r=1.55
22.0
19.2
62.3
85.4
38.2
49.9
46.2
81.7
44.4
48.6
58.2
Narrower Focus (050.
24.9
17.4
64.9
86.6
34.2
49.0
46.2
72.9
40.4
51.1
54.8
Baad-Biased (2=0.255)
22.4
19.6
61.9
86.0
38.2
49.8
46.3
83.6
41.4
49.4
58.2
7 CONCLUSION
While experiential RL offers a promising approach to mitigate sample inefficiency in RLVR for large reasoning models, this area still lacks systematic exploration of experience value and management. In this work, we address this gap by first examining what constitutes a valuable reasoning experience, then introducing ExGRPO, framework that strategically manages, selects and replays high-qui high-quality experiences. Experiments across multiple backbones and benchmarks show that ExGRPO consistently improves performance and stabilizes training where on-policy RLVR fails.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
This research follows the ICLR Code of Ethics. We have taken care to ensure that the datasets, methodologies, and model used in our experiments ethically sound. Our work is primarily theoretical and empirical, focusing on optimizing reasoning models. As such, it does not involve sensitive personal data, or content likely to introduce societal bias, harm, or discrimination.
REPRODUCIBILITY STATEMENT
We provide the experimental setups and hyperparameters in Section 5.1, with further details in Section E. The code and model weights are open-sourced.
ACKNOWLEDGMENTS
We extend our gratitude to all the reviewers for their valuable feedback and suggestions. This work was supported by the Shanghai Artificial Intelligence Laboratory. This work was supported by the Science and Technology Development Fund of Macau SAR (Grant Nos. FDCT/0007/202//KK EF2024- 00185-FST), the UM and UMDF (Grant Nos. GRG2025-00236-FST) SHMDF-A12026/001) and the National Natural Science Foundation of China (Grant No. 62266013).
MYRG-
REFERENCES
Charles Arnal, GaÄftan Narozniak, Vivien Cabannes, Yunhao Tang, Julia Kempe, and Remi Munos. Asymmetric REINFORCE for off-policy reinforcement learning: Balancing positive and negative rewards. arXiv preprint 2rXiv:2506.20520 2025.
Adithya Bhaskar, Xi Ye, and Danqi Chen. Language models that think, chat better. arXiv preprint arXiv:2509.20357 2025.
Ding Chen, Qingchen Yu, Pengyuan Wang, Wentao Zhang, Bo Tang, Feiyu Xiong, Xinchi Li, Minchuan Yang, and Zhiyu Li. x Verify: Efficient answer verifier for reasoning model evaluations. arXiv preprint, htss/204410488.. 2025a.
Minghan Chen, Guikun Chen, Wenguan Wang, and Yi Yang. Seed-grpo: Semantic entropy enhanced grpo for uncertainty- policy optimization. arXiv preprint arXiv:2505.. 12346, 2025b.
Xiangxiang Chu, Hailang Huang, Xiao Zhang, Fei Wei, and Yong Wang- GPG: simple and strong reinforcement learning baseline for model reasoning. arXiv preprint arXiv:2504.02546, 2025.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try ARC, the A12 reasoning challenge. arXiv preprint, abs/ 803.05557, 2018.
Taco Cohen, David Zhang, Kunhao Zheng, Yunhao Tang, Remi Munos, and Gabriel Synnaeve. Soft policy optimization: Online off-policy rl for sequence models. arXiv preprint arXiv: arXiv:2503.054: 05455,, 2025.
Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, al. Process reinforcement through implicit rewards. arXiv preprint arXiv:2502.01456, 2025a.
Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang. Yuxin Zuo, Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, The entropy mechanism of reinforcement learning for reasoning language models. arXiv preprint, hts//505.22617, 2025b.
Shihan Dou, Muling Wu, Jingwen Xu, Rui Zheng, Tao Gui, Qi Zhang, and Xuanjing Huang. Improving RL exploration for LLM reasoning through retrospective replay. arXiv preprint, abs/2504. 14363, 2025.
11

---

Published as a conference paper at ICLR 2026
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In International conference machine learning, pp. 1407-1416. PMLR, 2018.
Hugging Face. Open rl: A fully open reproduction of deepseek-rl, 2025.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The Ilama 3 herd of models. arXiv preprint, abb/2240.221788 2024.
Mourad Gridach, Jay Nanavati, Khaldoun Zine El Abidine, Lenon Mendes, and Christina Mack. Agentic ai for scientific discovery: A survey of progress, challenges, and future directions. arXiv preprint, 0abbs/5033089799 2025.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint, abs/2501. .2948, 2025.
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al. Olympiadbench: A challenging benchmark for promoting agi with olympiad-lewe bilingual multimodal scientific problems. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3828-3850, 2024.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint, hass/2103.03874, 2021.
Nathan Kallus and Masatoshi Uehara. Statistically efficient off-policy policy gradients. In Proceedings of International Conference on Machine Learning, pp. 5089-5 5100, 2020.
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. Tulu 3: Pushing frontiers in open language model post-training. arXiv preprint, abs/2411.151222 2024.
Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski. Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. Solving quantitative reasoning problems with language models. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems, NeurlPS, 2022.
Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Huang, Kashif Rasul, Longhui Yu, Albert Q Jiang, Ziju Shen, et al. Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions. Hugging Face repository, 13(9):9, 2024.
Siheng Li, Zhanhui Zhou, Wai Lam, Chao Yang, and Chaochao Lu. Repo: Replay-enhanced policy optimization. arXiv preprint, 2as/2506.093440 2025.
Jing Liang, Hongyao Tang, Yi Ma, Jinyi Liu, Yan Zheng, Shuyue Hu, Lei Bai, and Jianye Hao. Squeeze the soaked sponge: Efficient off-policy reinforcement finetuning for large language model. arXiv preprint arXiv:2507.0689) 2025.
Timothy P. Lillicrap. Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In Proceedings of 4th International Conference on Learning Representations, ICLR, 2016.
Long-Ji Lin. Self-improving reactive agents based on reinforcement learning, planning and teaching.
Machine Learning, 8(3-4):293333 321, 1992.
Chris Yuhao Liu, Liang Zeng, Jiacai Liu, Rui Yan, Jujie He, Chaojie Wang. Shuicheng Yan, Yang Liu, and Yahui Zhou. Skywork-rewarce Bag of tricks for reward modeling in llms. arXiv preprint a41iv:2410. 18451, 2024.
12

---

Published as a conference paper at ICLR 2026
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding rl --zero-like training: A critical perspective. arXiv preprint, abs/2503.20783, 2025.
Fanbin Lu, Zhisheng Zhong, Shu Liu, Chi-Wing Mii-Wing Fu, and Jiaya Jia. Arpo: End-to-end policy optimization for gui agents with experience replay. arXiv preprini, abs/2505. 16282, 2025.
Lu Ma, Hao Liang, Meiyi Qiang, Lexiang Tang, Xiaochen Ma, Zhen Hao Wong, Junbo Niu, Chengyu Shen, Runming He, Bin Cui, et al. Learning what reinforcement learning can't: Interleaved online fine-tuning for hardest questions. arXiv preprini, abs/2506.07527, 2025a.
Xueguang Ma, Qian Liu, Dongfu Jiang, Ge Zhang, Zejun Ma, and Wenhu Chen. General-reasoner: Advancing IIm reasoning across all domains. arXiv preprint, abs/2505. 2025b.
Wenjia Meng, Qian Zheng, Gang Pan, and Yilong Yin. Off-policy proximal policy optimization. In Proceedings of AAAI Conference on Artificial Intelligence, volume 37, pp. 9162-9170, 2023.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprins 1r25v:13/ 2013.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin A Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):599533, 2015.
Aske Plaat, Annie Wong, Suzan Verberne, Joost Broekens, Niki van Stein, and Thomas Bäck. Reasoning with large language models, a survey. CoRR, 2024.
Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang. Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang. Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2024.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang. Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling, 2024.
Nicolas Le Roux, Marc G Bellemare, Jonathan Lebensold, Arnaud Bergeron, Joshua Greaves, Alex Fréchette, Carolyne Pelletier, Eric Thibodeau-Laufer, Sándor Toth, and Sam Work. Ta- pered policy reinforce: Stable and efficient reinforcement learning for LLMs. arXiv preprint arXiv:2503. 14286, 2025.
Tom Schaul, John Quan, loannis Antonoglou, and David Silver. Prioritized experience replay. In 4th International Conference Learning Representations, ICLR, 2016.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pp. 1889-1897. PMLR, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint, abs/1707.003477 2017.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowci Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepscekmath: Pushing the limits of mathemat- ical reasoning in open language models. arXiv preprint, abs/2402.0330m 2024.
David Silver and Richard S Sutton. Welcome to the era of experience. Google AI, 1, 2025.
Yi Su, Dian Yu, Linfeng Song, Juntao Li, Haitao Mi, Zhaopeng Tu, Min Zhang, and Dong Yu. Crossing the reward bridge: Expanding rl with verifiable rewards across diverse domains. arXiv preprint, abs/2503.23829 2025.
13

---

Published as a conference paper at ICLR 2026
Shivakanth Sujit, Somjit Nath, Pedro H. M. Braga, and Samira Ebrahimi Kahou. Prioritizing samples in reinforcement learning with reducible loss. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems, NeurIPS, 2023.
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems, NeurlPS, 2024.
Ziyu Wang, Victor Bapst, Nicolas Heess, Volodymyr Mnih, Remi Munos, Koray Kavukcuoglu. and Nando De Freitas. Sample efficient actor-critic with experience replay. arXiv preprint 16Xiv:161..0120 2016.
Xumeng Wen, Zihan Liu, Shun Zheng, Zhijian Xu, Shengyu Ye, Zhirong Wu, Xiao Liang, Yang Wang, Junjie Li, Ziming Miao, et al. Reinforcement learning with verifiable rewards implicitly incentivizes correct reasoning in base Ilms. arXiv preprint, abs/2506.14245 2025.
Wei Xiong, Jiarui Yao, Yuhui Xu, Bo Pang, Lei Wang, Doyen Sahoo, Junnan Li, Nan Jiang, Tong Zhang, Caiming Xiong, et al. A minimalist approach to IIm reasoning: from rejection sampling to reinforce. arXiv preprint arXiv:2504. 1343, 2025.
Hongling Xu, Qi Zhu, Heyuan Deng, Jinpeng Li, Lu Hou, Yasheng Wang. Lifeng Shang, Ruifeng Xu, and Fei Mi. KDRL: Post-training reasoning Ilms via unified knowledge distillation and reinforcement learning. arXiv preprint arXiv:2506. 02208, 2025.
Jianhao Yan, Yafu Li, Zican Hu, Zhi Wang, Ganqu Cui, Xiaoye Qu, Yu Cheng, and Yue Zhang. Learning to reason under off- policy guidance. arXiv preprint, abs/2504. 14945, 2025.
An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jian- hong Tu, Jingren Zhou, Junyang Lin, et al. Qwen2. 5-math technical report: Toward mathematical expert model via self-improvement. arXiv preprint, abs/2409. 12122, 2024.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint, abs/2505.09388, 2025.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source IIm reinforcement learning system at scale. arXiv preprint, abs/2503. 14476, 2025.
Lifan Yuan, Wendi Li, Huayu Chen, Ganqu Cui, Ning Ding, Kaiyan Zhang, Bowen Zhou, Zhiyuan Liu, and Hao Peng. Free process rewards without process labels. arXiv preprint arX1922412.01981, 2024.
WeiHao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl- 200: Investigating and taming zero reinforcement learning for open base models in the wild. arXiv preprint, a 18892, 2025.
Hongzhi Zhang, Jia Fu, Jingyuan Zhang, Kai Fu, Qi Wang, Fuzheng Zhang, and Guorui Zhou. Rlep: Reinforcement learning with experience replay for 1l reasoning. arXiv preprint, 1 2025a.
Jixiao Zhang and Chunsheng Zuo. Grpo-lead: A difficulty-aware reinforcement learning approach for concise mathematical reasoning in language models. arXiv preprint 250400000 2025.
Wenhao Zhang, Yuexiang Xie, Yuchang Sun, Yanxi Chen, Guoyin Wang, Yaliang Li, Bolin Ding, and Jingren Zhou On-policy meets off-policy experts: Harmonizing supervised fine-tuning and reinforcement learning via dynamic weighting. arXiv preprint, abs/2508.1 1408, 2025b.
Wenting Zhao, Xiang Ren, Jack Hessel. Claire Cardie, Yejinn Choi, and Yuntian Deng, Wildchat: Im chatgpt interaction logs in the wild. In The Twelfth International Conference on Learning Representations, 2025.
Xinyu Zhu, Mengzhou Xia, Zhepei Wei, Wei-Lin Chen, Dangi Chen, and Yu Meng. The surprising effectiveness of negative reinforcement in 1l reasoning. arXiv preprint arXiv:2506.013:2 2504.01377 2025.
14

---

Published as a conference paper at ICLR 2026
APPENDIX
APPENDIX CONTENTS
Limitations
15
The Use of Large Language Models (LLMs)
16
C ExGRPO Algorithm
16
D Theoretical Analysis of ExGRPO
18
D.I Recap and Formulation
18
D.2 Unbiasedness of Experiential Gradient
18
D.3 Variance Decomposition and Bounds
19
D.4 Final Remarks
20
Supplementary Materials of Experiments
E.1
Detailed Training Settings
21
E,2 Baselines
E.3 Results of Model Extension
................................
23
E.4
Ablation and Additional Results
E.5
Replay Memory and Computational Overhead
25
E.6
Data Utilization
26
Supplementary Materials of Preliminaries
26
F.1
Details of Masked GRPO
26
F.2
CoT Judge
26
F3 Alternative Selection Metrics
27
F.4
Snowball Effects
27
G Prompt Template
36
G.I RLVR Training and Evaluation
36
G.2 CoT Judge
36
A LIMITATIONS
While our work demonstrates the significant benefits principled experience management in RLVR. we acknowledge several limitations. First, since our research scope focused on RLVR, the evaluation tasks are verifiable problems, i.e., mathematical and general reasoning benchmarks where trajectories can be more easily verified correct or incorrect. The applicability of our correctness- based buck- cting strategy to more open-ended tasks (e.g., creative writing), where rewards are often subjective and dense, remains an open question. Second, our framework 's definition of valuable experience is based on heuristics that, while powerful, might be incomplete. ExGRPO might neglect the learning potential of "valuable failures" where some incorrect paths that contain useful training signals (Zhu et al., 2025). This focus on exploitation could risk premature convergence in some scenarios. Finally, while ExGRPO is built upon relative policy optimization objective, its interaction with other families
15

---

Published as a conference paper at ICLR 2026
of RL algorithms has not been explored. Future work includes extending method to multi- modal reasoning and agentic reinforcement learning.
B THE USE OF LARGE LANGUAGE MODELS (LLMS)
We utilize large language models to assist with proofreading, and polishing the text of this paper. The generated responses are treated as decision references and are not adopted wholesale. Importantly, all scientific contributions, conceptual developments, experimental designs, and final decisions were made by human researchers. The outputs from LLMs were treated as references or drafts, subject to human review, modification, and validation.
C EXGRPO ALGORITHM
Algorithm 1 ExGRPO: Experiential Group Relative Policy Optimization
Require: Dataset D, batch size B, experience ratio p E [o, 1]. rollout trails K, reward model R(-).
1: Initialize: Policy model my. experience replay buffer & 0, retired set S + 0.
2: for each training step do
3: Phase I: Experience Management.
4: if (EEE 0 then
5:
Partition
buckets Uy by last rollout correctness Acc Σx= 1 11.
6:
Obtain sampling probability: Pk: = (N/k/ K.0.5, a = for nonempty U%
7:
Calculate experience sampling amount: n 4 min(LoB) (8))
8:
Sample n experience: {ei,..., fn) BukcetSample(21, (pa)) for each experience E
D cf. Algorithm 2. cf. E: E: + {0'}
9:
Cnp do
10:
for each experience trajectory o do Compute trajectory entropy under Te: H(a; = [.:1 end for Select the trajectory with the lowest entropy: end for
11:
1 Σ, log To 0²
12:
arg ms H(o;)0)
14:
Experiential policy optimization data Bexp +- ,0))²
On-policy optimization data Bon + {q} B- D
17:
else
18:
Boxp + 0, Bon + {4] D end if
19:
20:
Phase 2: Experiential Policy Optimization.
21:
Construct batch data: B +- Bexp U Bon
22:
for each sample q E B do Generate rollouts: (o1,..., ,0k mo(-l)), (0r, (o1,...,oK-1} ~ Tog (0("
23:
24:
Compute rewards and success: {r
K = [R(g, , Σ
K
[r
j==
if s = K then S U else if < K then Record experience: Elg] U {0, r(o;) 1}
All successful
26:
Add to retired set
27:
> Partial success
28:
29:
end end for
30:
31:
Compute advantage and update policy model with Eq. 4.
32:
Remove well-learned samples: & : S}
33:
end for
return Optimized policy model से
Overview of ExGRPO (Algorithm 1). ExGRPO organizes training into two phases: experience management and experiential policy optimization. In first phase, the replay buffer & is partitioned into buckets based on the most recent rollout correctness Acc(q). and a Gaussian weighting centered 0.5 biases sampling toward medium-difficull problems. From these buckets,
min[[pB),,18)
16

---

Published as a conference paper at ICLR 2026
experiences are drawn, and for each, the trajectory with the lowest entropy under the current policy is selected as o*. These anchors form the experiential sub-batch Bexp- while the remaining slots are filled with on-policy samples Bon- In the second phase, each batch item generates K rollouts (or K-1 plus the past trajectory), rewards are computed, and prompts achieving full success are retired, while partially successful ones refresh &. The policy is updated using objective in Eq. 4.
Details of Bucketed Sampling (Algorithm 2). This sampler first normalizes probabilities p over non-empty buckets, then draws bucket counts с using sequential-bing -inimiall implementation of the multinomial distribution, ensuring Σ. C n where is the number of samples to be drawn. Within each bucket, C* items are sampled uniformly without replacement. This preserves the intended inter-bucket bias (e.g., Gaussian weighting by accuracy) while remaining unbiased within buckets. The method assumes S. [&4x] for feasibility. In practice, to handle rare edge cases where bucket becomes empty, clipping and redistributing the sampling counts will be applied in such scenarios.
Algorithm 2 BucketSample(). Bucketed Multinomial & Within-Bucket Uniform Sampling
Require: Valid buckets U = {(Ace, {e})}{
k= probabilities E [0, 1] sample size n N;
Ensure: probabilities p E [0, with _k=1PE= =1
1: function MULTINOMIAL(T..P p)
2: Note: sample X E Nd such that Σ X; = n
3: Initialize + (O....,0) E No, m t n
4: for i =
do
5: Draw X; Binomial m, I- Pr E
6: Update m m X;
7: end for
8: return collection of X
9: end function
10: # Note: sampling within a bucket is uniform without replacement; requires C₁ S Mace for all k.
с Multinomial(n, P)
>> Draw all bucket counts in one shot
12: sampled ttems <- 0
for < K do
14: 8 +- UnfformSample((4u ck)
Without replacement.
15:
sampleditent [u]] <
16: end for
17: return (sampled.items, (])
Complexity. Experience management in ExGRPO is efficient. Per step, ExGRPO performs O(BK) reward evaluations and O(BK) likelihood computations. Bucket partitioning and sampling O((E)
for bucketization (amortized) and O(K + n) for drawing counts and items.
Correctness and Constraints. The procedure assumes C < Ug for all nonempty buckets. In practice, this is satisfied with high probability when (i) the support of p excludes tiny buckets (ii) we clip Cs to MMl and redistribute any deficit to remaining buckets. We adopt the simple feasibility requirement stated in Algorithm 2 for clarity.
Practical Notes. We renormalize over nonempty buckets before sampling; this avoids assigning
mass to empty buckets. The scheme is streaming-friendlly and yields O(K) arithmetic plus the cost of bucket draws. When used in ExGRPO, setting p [Nk/K o) provides a tunable curriculum over the rollout correctness spectrum. In this paper, we consistently set = 0.5, a = to align with analytical findings in Section 3.
17

---

Published as a conference paper at ICLR 2026
D THEORETICAL ANALYSIS OF EXGRPO
D.1 RECAP AND FORMULATION
Let q ~ D be query. The rollout (reference) policy Taa- the current policy is nas and the experiential policy (that produced a replayed trajectory) is T8aa For each q, form a group Gg {or K with or Toao ( (g)
and use GRPO's within-group standardization
Mo, (g) = (1),0)) 1gg
K
PS, = K Σ r(g.o,), og, - ((d(f((g):))
(5)
dg,
Based on the practice we detailed in the main body of paper, we follow the simplifications advocated by Dr.GRPO (Liu et al., 2025), removing length and standard deviation normalization. For mixed- policy optimization, we also remove clipping and use policy shaping.
ExGRPO mixes (i) on-policy groups Gg and (ii) mixed groups Gg. = [o for q" sampled from experience buffer, where o* To.... (- q²) and the K fresh rollouts come from Te g²). The ExGRPO mini-batc mai-batch objective (Eq. 4) mixes on on-policy and experiential (replayed) contributions with weight p E (0, 1).
Per-token importance ratios for a trajectory comparing policy my to policy my (i.e., Tiws or T are:
we(os;0,0') To : C ci, To (Oi,e 9, (i,et)
Wo,;0,00 = II wr(os,
(6)
We will sometimes write w-() or (-)) for brevity. For the replayed item o", ExGRPO uses a bounded policy-shaping transform f(w)) -+ with B >> 0 instead of hard clipping: this is analogous to trancated/saturated importance ratios in ACER and V-trace, known control variance small bias (Wang et al., 2016; Espeholt et al., 2018). We assume bounded rewards, absolute continuity of policies (mutual support overlap), and an auxiliary KL control as in Schulman et (2015; 2017).
Assumptions. For formal statements will use the following standard assumptions where invoked:
Al. Support assumption. For any trajectory o produced by past policy used in replay, To(o; > whenever 0d (or |-) (i.e., target policy has support containing past policy).
A2. Bounded second moment. For any random variable under consideration (rewards, advantages), second moments exist and are finite. Formally, for any policy model To and query distribution DD
EypD (-)1) x²
so finiteness
[[X guarantees that Var Var(X) E[x (E[X))* is well-defined.
A3. Bounded importance ratios. Assume finite trajectory length, there exists M > 1 such that for all relevant trajectories and timesteps.
sup To(or (-) oft Tppe (or 1))
I m, and hence sup ((o; 0pus) S M == mo
(8)
D.2 UNBIASEDNESS OF EXPERIENTIAL. GRADIENT
We show that, when the replayed trajectory is corrected by the exact per-token importance weight Woo": 0, 0pas), the contribution of that trajectory yields an unbiased estimate of the corresponding on-policy term (no bias is introduced by using replay plus exact importance reweighting). even when the advantage À depends on the full mixed group g.
Theorem (Unbiasedness). Under Assumption A1, let G = {o*} U [or be a mixed group where o sampled from ai and {ox) were sampled from Te For any measurable function
18

---

Published as a conference paper at ICLR 2026
9 of a trajectory and its group (e.g. g( En Vo log To(ool9, A(o, 9)), the importance
weighted expectation equals the on on-policy expectation:
[W (o": 0, pasi) g(0" g.r) {o, {o Born [g(ō. [o.) Bot
where the right hand side is the expectation with o² replaced by sampling ~ To and forming group accordingly.
1K-1 JK-1 is implicit as the sampling Proof Sketch. Condition on the other group elements {o:)/=1) {os) of o" is independent of (ox) jo K-1 thus we have
En (o" ;0,00,,00000 Ppsa) g(o" .6-)
[-
Bo opo [W ppss)) g(0"; 9o-)
Further, consider the expectation over o* drawn from Tapa: . By the definition of expectation, sum over all possible trajectories (which we denote by o²) weighted by their probability under Tpp
B Typ0 [W 0 pus)) g(0", "0-0,,--)]]
Σ To [W() 0, Ppss) g(o", 9g-)]
Tr (0)) To(o")
TB (or) g(0 (00 g.r)
(10)
(re(o")g(o" ,
Es-ao [g(ō,)).
The key is that g(.gg) is an arbitrary measurable function of a trajectory and the group, and the algebra above does require y to be independent of the group statistics (the sum is over the trajectory variable and the group members are treated as constants in the inner sum). This yields the stated identity. Thus, we have shown that the aaportance-weighhee estimator of the on-policy term, conditioned on the rest of the group members.
experiential term is an unbiased
The theorem shows that, with exact importance weights, replay introduces no bias even when the advantage is computed using group-dependent normalization (ug, og). In practice exact per-token ratios are used (product over timesteps), and the identity holds under A1.
Remark.
D.3 VARIANCE DECOMPOSITION AND BOUNDS
Let Gexp denotes the random contribution from an experience-mixee mixed group that includes one replayed trajectory corrected by importance sampling as:
K
Gexp
W((o:; 0 0 pps) U'o'.9) + Σ U(o..9))
(11)
K
where U(0, G) denotes the per-trajectory unweighted gradient contribution (e.g. E, Vo log ( A).
Importantly, in our implementation of GRPO the group standard-deviation term omitted (we only mean-center ad advantages), so the U-,, G) terms still depend the group mean jıg but not on Øg. This reduces but does not eliminate the statistical coupling between group members.
Below we give general variance bound that does not assume independence among group members, and then a tighter bound under a ivak-inneepnddecce assumption that often approximately valid in practice (c.g., large K).
Proposition 2 (Variance upper bound for experiential term). Under Assumption A2 (finite second moments) and A3 (bounded trajectory importance ratios), the experiential variance satisfies the following bounds.
19

---

Published as a conference paper at ICLR 2026
1. (General, no independence) Without assuming independence between U(o", ,9) and the other group contributions, we have the conservative inequality
2 Var (Guxp) S K2
[ + + -) -
(A)
If in addition W is uniformly bounded by M (Assumption A3), then
(M M² EIU (K- - 11) 2(M² (K - k
2
Var (Gexp)
BIU (A))
K
2. (Tighter bound under weak/approximate independence) If the within group unweighted con- tributions (o, ( ( are pairwise uncorrelated and uncorrelated with W ('o')U(o',9) ,)) (e.g.. approximate when K is large), then
2 S K2 (K
Var (Gexp)
(B)
With the bound W S M this yields
Var (Gexp) 2(M2 + (K 1))
E[[3J.
(B))
K2
Proof Sketch. Write S = S U(o,, Then
1 E[WU SS²]
Var (Gexp)
Var (WU + S)
K2 K²
(12)
k2
using Var(X) S 3[ Expanding and applying Cauchy-Schwarz to bound cross terms yields
Var (Gexp) I
E + E[S 2 E| S(S
(13)
Using bb+b+)/ab) s b) and E[S (K (K-1)*E (which follows without independence by bounding pairwise covariances by second moments) gives the general bound (A). Plugging W22 M yields (A').
If one further assumes pairwise uncorrelatedness of the U-terms and zero covariance with WU, then E[S2] (K T) and the cross terms drop, recovering the tighter bound (B) and its specialization (B').
Practical Remark. Removing the within-group standard-deviation from GRPO reduces the coupling introduced by stochastic denominator (the og-term). This typically lowers E[U compared to the case with standard deviation normalization (because the denominator's variability can strongly amplify per-trajectory contributions), which in turn reduces all bounds above.
Based the upper bound analysis of variance, important insight is that controlling the source of experience policy is crucial for reducing the value of M, and consequently tightening the variance bound. In practice, techniques such low-entropy selection and importance sampling correction contribute to this goal. Specifically, we use the average conditional entropy of the trajectory as pickup metric, and minimizing this metric encourages trajectories from the typical set, which facilitates controlling the importance sampling term and reduces the discrepancy between Ty and Tea
D.4
FINAL REMARKS
Policy shaping via f(w) = "0 and removing clip. As mentioned in Section 4.2, we remove w+z
the CLIP term and replace it with a policy-shaping function to enable better off- policy experience exploitation. Rather than relying empirical validation from prior work (use expert trajectory as off-pooicy), we also provide a discussion of this design choice used in experiential optimization here. The policy shaping transform replaces by fww) for the replayed trajectory. Its key properties are:
20

---

Published as a conference paper at ICLR 2026
f is monotone increasing in
S f(w)) <
for all w 2 0, and f(w) - 1 as w -> 0o.
. For small w, f(w) = w/B (which can amplify very small ratios when B < 1); for large f(w) is bounded by and thus damps extremely large ratios.
Thus f reduces variance contribution of large importance weights while introducing bias (since [[f(W)] generally). Combined with group normalization, cediuum-diffiullty question sampling and low-entropy trajectory, policy shaping often suffices to control variance so that GRPO clipping can be relaxed removed in practice.
Summary and Takeaways. We have shown that exact trajectory-level importance weighting guarantees unbiasedness of experiential contributions, even when advantages are computed from mixed groups (Theorem 1). The variance experiential gradients is governed by the second moment E[ bounding or controlling importance sampling term is critical (Proposition 2).
Finally, the smooth policy-shaping transform f(w) = wht offers a principled bias- varrance tradeoff:
it suppresses extreme importance weights while preserving informative contributions that would be truncated by hard clipping. In combination with within-group normalization, this explains why ExGRPO can remain stable without clipping while still benefiting from replayed trajectories.
E SUPPLEMENTARY MATERIALS OF EXPERIMENTS
E.1 DETAILED TRAINING SETTINGS
During RLVR training, the rollout generation uses a temperature of 1.0. To reduce training time and monitor training dynamics, we used a validation set by sampling 2.2k instances from the test set. Model performance was evaluated every 10 steps to better track training progress, without applying early stopping. Qwen2.5-Matt 7B backbone was trained for 700 steps, while the models in our extension experiments were trained for 500 steps. Following the methodology of LUFFY (Yan et al., 2025), we employed a consistent prompt template across all models during training, as detailed in Section G. An exception was made for the Llama-3.1 8B base model, for which simplified prompt was used to accommodate its limited capabilities. Additional training hhper-paramene parameters are provided in Table 4.
E.2 BASELINES
For the zero RLVR using the Qwen2.5-Math 7B model (Yang et al., 2024), we compare our approach against not only the on-policy Dr.GRPO baseline but also other RLVR methods:
PRIME-Zero (Cui et al., 2025a): PRIME-Zero is trained with PRIME, an online RL framework that leverages implicit process rewards (Yuan et al., 2024) to enhance reasoning beyond imitation or distillation. from the SFT model; the policy generates rollouts, which are scored by the implicit PRM and an outcome verifier. The policy is then updated using a combination of outcome and process rewards.
PRIME, both the policy and implicit PRM are initialized
Oat-Zero (Liu et al., 2025): The original Dr.RPOO implementations discard the standard deviation in advantage computation and omit token-level normalization in the policy loss.
GPG-Zero (Chu et al., 2025): GPG directly integrates group-based decision dynamics into standard policy gradients, simplifying training and greatly reducing computation without loss of reasoning quality.
RePO-Zero (Li et al., 2025): Replay-enhanced Policy Optimization (RePO) employs an online experience replay mechanism by collecting early on- on-policy rollouts. However, revisits them asynchronously and does not incorporate experience management. We further compare ExGRPO with RePO under the same data and training settings used by RePO, as detailed in Section E.4.
21

---

Published as a conference paper at ICLR 2026
Table 4: RLVR training hyperparameters used in our experiments. All sequence lengths are measured using each model's own tokenizer.
Module
Parameter
Value
Description
Data
data.train batch size
128
Global training batch size per opti- mization step.
data.val.batch..siz
Batch size used for validation.
beeaxxxpropp.iiiee penmpt.length data.max,
1024
Maximum input length.
data.max rewaaarssssonseenggttttt
8192
Maximum response length.
Actor
actor htw.:oooooooo....aooooooooooooo aww......................
x 10
Learning rate for the actor opti- mizer.
actor cnforrrolloutrre wtw.rroooouu.rffccooooooooooo www..actoo.ppoo....t... mini_batch size
64
actor_rollout_ref wettttref.ctooo www..accorrpppp.. ppt... micro.batchh...
64
actor_rollouttreff ref.actto.entrop ccoeff
0.001
actor_rollout_ rectoooouttr..ctoo..osssss wwtpa/ooroooooooreeeoooeoeoo mean
True
Remove token-wise mean term from the loss. Disable the clipping term loss.
actor rollout ref.actor.loss remove clip
True
Rollout
actor rollout ref.rollout.engine actor rollout ref.rollout.eemperauuuee
vllm
Inference/rollout engine. Sampling temperature during train- ing rollouts.
1.0
actor.rollouu.reefrrllluu val. temperrate
0.6
Sampling temperature during vali- dation.
actor_rollout_cfffollluu..g. mew.mmy mem.mmoyuttilizationnn
0.80
Fraction of GPU memory to utilize for the rollout engine.
cctor_oolloutteefrrlllou..a
8
Number of rollouts per prompt dur- ing RLVR.
Trainer
trainer.critic.wamuup
0
Number of warmup steps (0 dis- ables warmup).
trainer.training.ster
700/500
Number of training steps. 700
steps for Qwen2.5-Math- 500
steps for other models.
22

---

Published as a conference paper at ICLR 2026
For the continual RLVR setting, apply ExGRPO strong backbone model, LUFFY (Yan et al., 2025), which integrates Deepseek-Re R1 (Guo et al., 2025) trajectories in RL and substantially boosts reasoning performance. We further compare ExGRPO against other reproduced off off-policy learning strategies, including:
SFT: Supervised fine-tuning directly on the RI trajectories from OpenRI Face (2025).
SFT+RL: A two-stage approach involving SFT on RI trajectories followed by on-policy RL on the same problem set.
Continual LUFFY: A direct continual off- optimization on the R1 trajectories using the LUFFY algorithm.
Table 5: Overall in tn-distribution and out out-of-disttnn distiiiiibttion performance based on different backbone models. Bold indicates the best results within comparable group.
In-Distribution Performance
Out-of-Distribution Performance
Model
AIME24 AIME25 AMC MATH-500 Minerva Olympiad Avg. ARC-c GPQA' MMLU-Pro Avg.
Qwen2.5-Math 1.5B
Base
7.2
3.6
26.4
28.0
9.6
21.2
16.0
3.1
3.0
2.3
2.8
On-Policy
11.8
7.7
40.2
61.8
26.8
32.0
30.0
57.5
28.3
30.7
38.9
ExGRPO
13.3
8.6
46.6
74.0
30.5
39.0
35.3
59.3
30.8
30.4
40.2
Qwen2.5-77 Instruct
Instruct
11.7
7.5
43.8
71.8
30.9
40.4
34.4
84.7
24.7
54.7
54.7
On-Policy
14.1
8.3
43.5
74.0
33.8
37.6
35.2
91.4
19.2
57.6
56.0
ExGRPO
15.2
12.4
47.8
80.0
36.0
44.4
39.3
91.1
36.9
58.6
62.2
Llama3.1 1-8B
Base
0.0
0.0
4.7
11.0
4.4
1.9
3.7
0.2
0.0
3.6
1.3
On-Policy
0.4
2.9
9.2
5.1
2.8
3.4
0.6
0.0
1.2
ExGRPO
0.7
0.1
6.6
12.0
10.7
6.5
6.1
59.0
7.1
26.2
30.8
Llama3. Instruct
Instruct
15.7
39.8
18.0
13.3
15.2
42.4
0.0
27.6
On-Policy
3.8
0.8
21.4
49.8
24.3
19.6
19.9
86.7
0.0
48.1
44.9
ExGRPO
27.1
55.2
25.7
23.1
23.0
88.9
15.2
52.1
52.0
E.3
RESULTS OF MODEL EXTENSION
The empirical results in Table 5 provide strong evidence that ExGRPO offers significant and consistent performance gains over both the original models and a strong On-Policy baseline, demonstrating its efficacy and robustness across diverse model architectures and initial tuning states. The evaluation spans multiple model families and sizes, including both base (Qwen2.5-Matt, 1.5B, Llama3. 1-8)) and instruction-tume variants (Qwen2.5-7B Instruct, L.ama3.1 Instruct), demonstrate the broad applicability of our method. The results consistently demonstrate the superiority of ExGRPO over the On-Policy method across all tested configurations. The advantage of ExGRPO is particularly pronounced when fine-tuning from base models. On Llama3. 1-B, the On-Policy method shows negligible gains over the base model, whereas ExGRPO provides a substantial boost, improving the average ID performance from 3.7 6.1 and, most notably, the average OOD performance from mere 1.3 to 30.8. This highlights ExGRPO's effectiveness in robustly enhancing model capabilities, especially in scenarios where standard on-policy fine-tuning struggles yield improvements.
23

---

Published as a conference paper at ICLR 2026
E.4 ABLATION AND ADDITIONAL RESULTS
To dissect the contribution of each key component within our proposed ExGRPO framework, we conduct series of ablation studies, with the results presented in Table 6.
Table 6: Ablation study of the key components of ExGRPO. We evaluate the impact of removing Quality (Q.) and Trajectory (T.) selection strategies, policy shaping. importance sampling (IS) correction, and the sensitivity the experience replay ratio p. The results highlight the contribution of each ExGRPO component.
AIME24
AIME25
AMC
MATH-500
Minerva
Olympiad
Avg.
On-Policy
24.9
15.5
59.2
84.8
38.2
49.3
45.3
ExGRPO w/ Selection Strategies
ExGRPO
31.6
18.7
66.3
87.4
36.0
50.1
48.3
w/o Q. Selection
26.9
17.2
63.9
86.4
38.6
47.6
46.7
w/o T. Selection
26.0
17.0
63.7
84.8
35.7
50.7
46.3
w/o Q. T. Selection
22.0
17.3
64.3
85.8
37.9
49.5
46.1
w/o Shaping
19.8
12.9
55.7
80.2
37.5
41.0
41.2
w/o IS Correction
25.9
19.8
63.2
87.0
37.9
49.0
47.1
Highest Entropy
25.0
16.0
62.9
84.6
36.4
51.0
46.0
ExGRPO w/ Experience Ratio
ExGRPO
31.6
18.7
66.3
87.4
36.0
50.1
48.3
+ w/ 25%
27.2
16.3
62.9
85,4
39,0
47.6
46.4
4 w/ 75%
22.1
17.6
61.4
84.4
36.0
47.3
44.8
ExGRPO Enhlay-Enhanced
EXGRPORE RePO-Zero
15.1
13.2
52.1
84.2
37.5
45.3
41.3
19.8
10.2
54.0
76.8
34.2
40.1
39.2
Experience Selection Strategies. First, we analyze the selection strategies. Removing only the Question Selection (w/o Q. Selection) strategy results in a performance drop to 46.7, while removing both strategy and Trajectory selection (w/o Q., T. Selection) further degrades the average score to 1..1. This indicates that both selection mechanisms are beneficial, with trajectory selection playing more substantial role. Besides, replacing our method with an alternative heuristic, w/ Highest Entropy Trajectory selection, also leads a suboptimal score of 46.0, validating our specific design choice.
Experience Ratio. Next, we investigate the impact of the experience ratio, p. which controls the proportion of past experiences used. The results show that our default configuration (Avg. 48.3) outperforms both a smaller ratio of p=25% (Avg. 46.4) and larger ratio of p=75% (Avg. 44.8). This suggests that our chosen ratio strikes an effective balance, leveraging a sufficient amount of high-quality past experience without being overwhelmed by fresh on-policy learning.
Policy Shaping Dynamics. Disabling policy shaping (w/o Shaping) causes performance drop to an average of 41.2, underscoring its essential function guiding the optimization process. Moreover, as illustrated in Figure 8, although entropy initially drops below the on-policy baseline during early training and later rises above the model still fails sustain exploration. Policy shaping alleviates this issue by ensuring that exploitation of experiences does not come at the cost of exploration.
Collectively, these ablations validate our design choices, confirming that each component ExGRPO synergistically contributes to its superior performance.
vs. RePO. To ensure a fair comparison with the most relevant baseline, RePO (Li et al., 2025), adopt its prompt template, data sources, and training configuration. We also match the number of on-policy rollouts maintain equivalent training conditions. Under this setting, we denote our the checkpoint released by RePO, apply the same
variant EXGRPORE For evaluation,
24

---

Published as a conference paper at ICLR 2026
40
On-Rolicy
ExGRPO
30
ExCRPO Shaping
Entropy
100
200
300
400
500
600
700
Training Steps
Figure 8: Dynamics policy entropy during training. ExGRPO without policy shaping even drops dramatically an early stage, performing worse than the on-policy baseline.
decoding parameters, and assess performance on our benchmark. As shown Table 6, ExGRPO outperforms RePO under identical training conditions. Beyond results in the table, the gap becomes more pronounced in out-of -iitt--diittiuutionn benchmarks, where EXGRPORF achieves an average score of 52.3 compared 46.8 for RePO.
Effects of Reintroducing Retired Queries. We investigate whether periodically replaying pre- viously solved (100%-success) queries can mitigate potential skill forgetting. Specifically, we reintroduce 10% of retired examples into the training batch every 25 steps. Contrary to intuition, this replay mechanism leads to drop in both in-distribution and out-of-distribuiion performance, as shown in Table 7, suggesting that revisiting fully solved queries injects limited learning signal while displacing more informative mid-difficulty cases.
Table 7: Ablation on reintroducing retired queries. Bold indicates the better results.
In-Distribution Performance
Out-of Duttof-Distributtion Performance
AIME24
AIME25
AMC MATH Minerva Olympiad Avg.
ARC-e GPQA"
MMLU-P Avg.
ExGRPO
18.7
66.3
36.0
48.4
84.7
37.4
sw/ Retried Review
23.2
17.7
61.0
84.0
34.9
50.7
45.3
81.7
41.4
48.0
57.1
E.5
REPLAY MEMORY AND COMPUTATIONAL OVERHEAD
Table 8: Replay memory and seee-pe--teep analysis. Times in seconds.
Model
Method
RAM
Peak Step (s)
Min. Step (s)
Avg. Step (s)
On-Policy
211.80
139.83
155.38
Qwen2.5-Mathe
ExGRPO
1.77GB
256.00
151.36
184.92
Overhead
+1.77GB
+20.9%
+8.2%
+19.0%
On-Policy
254.58
150.55
185.00
Qwen2.5 Math-1155
ExGRPO
1.05GB
228.90
163.92
188.6 85
Overhead
+1.05GB
-10.1%
+8.9%
+2.1% 1%
The replay buffer introduces additional memory usage. In our largest setup (Qwen2.5-Mathe on 8 GPUs), a buffer containing 202,011 trajectories requires approximately 1.77 GB of RAM at the end of training. This footprint is modest for modern hardware. Regarding computational overhead, ExGRPO efficient because we do not recompute entropy for the entire buffer at each step. Instead, for each query sampled in minibatch, typically a few dozen candidates. Empirically, as shown Table 8, ExGRPO introduces acceptable overhead while providing substantial gains in reasoning performance.
compute entropy only for its associated successful trajectories,
25

---

Published as a conference paper at ICLR 2026
E.6 DATA UTILIZATION
In experiments,
the data sampling ratio p 50%. This configuration implies that, for an equivalent number of training steps, our method visits 50% less fresh on-policy data per batch compared to the baselines. Notably, our approach achieves better performance despite less on-policy exploration, highlighting data efficiency of our method.
F SUPPLEMENTARY MATERIALS OF PRELIMINARIES
F.1 DETAILS OF MASKED GRPO
Formulation. To study the impact of different questions on the RLVR training process, modified version of the GRPO algorithm is used, termed as "Masked GRPO". The core idea is to selectively apply training gradients only from questions that are deemed to be of appropriate difficulty. We determine this difficulty based on the correctness of set of trajectories generated by the policy during online rollouts, as introduced in Section 3. The objective function is formalized as follows:
K
IMMACRPO = Eg mo (1)0) I(arow I Acc(g) I ahigh)
1
Σ CLIP (w,(0), (14)
K
Compared to the original GRPO objective, we introduce a crucial modification highlighted red: an indicator function I(-). Here, Acc(g) represents a function that calculates the correctness score for the set of trajectories {or} generated for given question This score is then evaluated against predefined range defined by a lower bound Clow and upper bound Ohigh-
Training Dynamics. A potential confounding factor in this analysis is that masking questions of certain difficulties might alter the total number of samples used for optimization in each group, thereby skewing the performance comparison. To rule out this possibility, we analyzed the training dynamics. Figure 9 plots the number of questions included in each mini-batch over the course of training for different groups. The visualization shows that the data throughput for all three configurations is highly similar and stable later stage of training, fluctuating around common mean. This visual evidence, further substantiated by the statistics on the average number of optimization samples (see Table 10), confirms that data quantity was significant variable. Therefore, we can attribute the observed performance in Table 9 to the distinct difficulty distributions of the training data itself.
Table 9: Performance of Qwen2.5 --tt22h---t---- -B trained under different difficulty schemes using on-policy RLVR. All results are reported on math reasoning benchmarks.
AIME24
AIME25
AMC
MATH-500
Minerva
Olympiad
Avg.
Full
24.9
15.5
59.2
84.8
38.2
49.3
45.3
On-Policy RLVR w/ Question Bucket
Easy
21.9
13.9
50.6
82.8
33.8
43.7
41.1
Medium
17.8
17.4
56.3
80.6
36.4
44,9
42.3
Hard
15.5
13.9
54.3
83.6
36.4
46.4
41.6
F.2 COT JUDGE
We adopt the same prompt template (refer to Section G.2) as prior work for evaluating the quality of reasoning COT. For the judge model, employ Qwen3-32B rather than the smaller models used in previous studies (Wen et al., 2025). This choice intended to ensure stronger judgment capability and more reliable evaluations. Decoding is performed with temperature of 0.6 and top-p of 0.95.
26

---

Published as a conference paper at ICLR 2026
Dynamics of Selected Questions (Masked GRPO)
Group
Avg. Mini-Batch
Mini-Batch
Easy
Medium
I Hand
Hard
20.79
Medium
23.87
20
Easy
21.23
100
200
300
400
500
Table 10: Average number of questions per mini-batch for
Training Steps
each difficulty-masked train- ing group, confirming similar data throughput.
Figure 9: Dynamics of the number of questions per mini- batch for the three difficulty-masked training groups. The three series demonstrate that each training regime exposed to a comparable volume of optimization data.
F.3 ALTERNATIVE SELECTION METRICS
Beyond entropy, the perplexity (PPL) of the current policy x can also serve as a lightweight metric for estimating the quality of reasoning CoT, formulated as:
PPL(o) = exp
exp
To| Σ
log T(o o2)))
(15)
In the ExGRPO method, we adopt entropy rather than PPL, with the justification detailed below. Our analysis, depicted in Figure 10, investigates the potential of using PPL metrics proxies for ovaluating the quality of reasoning trajectories. The results reveal positive correlation between these internal signals and external correctness judgments. Across all three correctness buckets, both perplexity and entropy are consistently lower for correct reasoning trajectories compared to incorrect ones. This finding suggests that the model is inherently more "confident" and "certain" when generating valid reasoning steps. More importantly, we observe that entropy exhibits superior discriminative power. The margin between correct and incorrect chains is wider for entropy than for PPL. indicating that the model's s token-level uncertainty is more sensitive indicator of flawed reasoning than overall sequence probability.
C. PPL
W. PPL
C. Entropy
W. Entrapy
1,20
0.16
0.15
1.16
1.15
0.14
0.14
1.10
1.10
0.11
0:12
0:10
1.05
0.08
1.00
Hard (0, xx25%] Medium (23%, 75%) Easy (79%, 100%)
Rollout Correctness Buckets
Figure 10: A comparison of average PPL and Entropy for correct ("C.") and wrong ("W.") reasoning trajectories (determined by an external CoT judge), grouped by online rollout correctness buckets.
F.4 SNOWBALL EFFECTS
Recap. A key challenge in learning from experience is the risk of snowball effect: the repeated sampling of trajectories with flawed reasoning from the replay buffer (generally characterised as high- entropy based in our research) reasoning errors. We observe, for instance, that models frequently generate some unnecessary code blocks when solving mathematical problems. To investigate this phenomenon, partitioned
systematically degrade the learned policy, leading to entrenched
27

---

Published as a conference paper at ICLR 2026
trajectories from our analytical experiments (refer to Section 3) based the presence of code generation (We implement a regex-based bagex-baseddd detector that flags whether a free-form text contains code in Table 11) and analyzed two key metrics: the average token-level entropy and the validity of the CoT, as assessed by the same CoT judge.
Table 11:
Rule set for code detection. The table lists high-level cues and rationale. We intentionally avoid placing regex in the table for rendering robustness, and exact patterns are provided below.
Type
High-Level Cue (plain language)
Rationale
R1
Structural (Tence)
Fenced code block explicilyy labeled as Python Unlabeled fence whose lirst emply line hegins with Python keyward (deficlassíim- hiffhonniiffo/wwhieettttwttt
Strong, explicit declaration of language Captures unlabeled lences that still look
R2
Structural (lence header)
like Python
R3
Syntax
Function detinition line
Canonical Python construct
Syntax
Class delinition line
Canonical class declaration
RS
Syntax
import statement
Frequent in code snippets
R6
Syntax
from import statement
Prequent examples/utorials
R7
Idiom
print call
Extremely common snippet
R8
Idiom
lent...> call
Common builtin call
Idiom
range
Common in loops/cxamples
R10
Syntax (loop)
"Eor in loop header
Strong loop cue
Syntax (branch)
..... header
Strong code Strong loop cue
R12
Syntax (loop)
"while .... header
R13
Syntax (exception)
"try:: header
Exception hundling comstruct
R14
Syntax (exception)
"except ...." header
Exception-handling construct
RIS
Syntax (literals)
Variable assigned to a list literal
Common quick examples
R16
Syntax (literals)
Variable assigned to dict literal
Common quick esamples
R17
Idhmm (methed call)
Dotted methood/lunction call followed by """" (e-g. obj mothodd 1..
Captures typical API usage
The results in Table 12 reveal strong correlation between code generation, model uncertainty, and reasoning quality. Across all problem difficulty levels, trajectories containing code blocks consistently exhibit higher entropy than those without (e.g., 0.14 vs. 0.07 for medium-difficuly problems). Crucially, this elevated uncertainty is coupled with a degradation in CoT quality. The proportion of logically correct CoT sequences is consistently lower in trajectories that utilize code. This performance gap is most pronounced for easy and medium problems, with an 11.6 and 10.2 point drop CoT correctness, respectively, and narrows slightly for hard problems (-8.7 point drop).
Results.
Discussion. We hypothesize that the behavior of code generation as a procedural shortcut or "black-box" execution, particularly when it struggles articulate complete, step-by-step formal proof. While this computational approach can be effective at deriving the correct final numerical answer, it often circumvents the explicit articulation of the underlying mathematical logie. This leads to higher incidence of flawed or redundant reasoning chains.
The implications for experiential RI. are direct. Using these high-entropy as experience will reinforce the model's 'ss bias toward such logically deficient trajectories, thus corrupting the reasoning process it learns. This suggests the importance of ensuring the quality of CoT in experience replay. As further evidence, our ablation studies (see Appendix Section E.4) confirm that explicitly forcing the model to learn from high-entropy experiences is harmful to its reasoning capabilities.
Table 12: Analysis of model--eneratce trajectories based the presence of code blocks. Trajectories with code exhibit higher entropy and lower rate of correct CoT, with A representing the percentage point drop in CoT correctness. Bold indicates better results of specific metric.
Average Entropy!
CoT Correctness ( (%)
o Code
Code
O Code
O Code
A
Easy
0.09
0.15
73.4
61.9
-11.5
Medium
0.07
0.14
64.7
54.5
-10.2
Hard
0.10
0.16
56.1
47.4
-8.7
Case Study. To further illustrate this behavior, we present several qualitative examples of such trajectories as follows. Case #1 presents challenging question from our training data. Although
28

---

Published as a conference paper at ICLR 2026
all trajectories ultimately arrive at the correct answer, they differ in the soundness and quality of their reasoning chains. The trajectory with the lowest entropy exhibits a shorter but coherent reasoning process. In contrast, the highest-entropy trajectory allocates more reasoning budget to code verification, producing longer responses that are logically unsound. Upon our inspection, the generated code is also invalid, and the external CoT judge also identifies this trajectory as poor reasoning. A similar phenomenon is observed in Case #3, which involves an easier problem. In Case #2, the external judge instead deems the high-entropy trajectory as a correct reasoning chain. However, it still produces unnecessary code in the final step, which we argue is unnatural and misaligned with human like reasoning behavior.
Case of Snowball Effects
QUESTION:
Given that n even, m is odd, and the system of equations
x 1988y = n {
11x + 27y
has the solution
x=p y= q
as integers, then
(A) p. q are both even.
(B) p.q both odd.
(C) p even, q odd.
(D) p is odd, is even.
("Zu Chongzhi Cup" Junior High School Mathematics Invitational Competition, 1988)
GROUND TRUTH:
C
RESPONSE: (Low Entropy)
REWARD: 1.0
ENTROPY: 0.1532 COT VALIDITY (JUDGED BY QWEN3-32B): Correct.
To determine the correct answer, let's analyze the system of equations:
x 1988y 11x + 27y m
Given that n is even, m is odd, and both a and y (i.c., p and q) are integers. Let's substitute and q the equations:
p 19884
11p 27q m
Since is even, p- 1988q must even. As 1988q is always even (since 1988 is even and any integer 4). must also be even for the sum to be even. Since m is odd, 11p + 27q must be odd. Since 1p is always odd or even (depending on p) and 27q is always odd (since 27 odd and q). the sum of 11p and 27q will be odd if and only if one of them is even and one is odd. Given is even, 11p is even, hence 27g must odd, implying g is odd. So, based on the above reasoning. is even and y is odd. Therefore, the correct answer is (C) is even, is odd. </think>
The final answer is
RESPONSE: (High Entropy)
REWARD: L.0
ENTROPY: 0.2652
CoT VALIDITY (JUDGED BY QWEN3-32B): Wrong.
29

---

Published as a conference paper at ICLR 2026
To solve the problem, need to analyze the system of linear equations and determine the parity (even or odd nature) of the variables x and y. Let's break down step-by-step.
Given the system of equations:
x 1988y
11x 27y = m
where n even and m is odd, and and y are integers, we need determine the parity of x and y.
First, let's express a and terms of n and m. We can use the method of elimination or substitution to solve the system. We can eliminate by multiplying the first equation by 11 and then subtracting the second equation from it:
11(x (xr 1988y) (11xr + 27y) 11n m
11r 21868y - 11x 27y = 11n m
-2215y 11m m
y = mn 11n
2215
Since m is odd and is even, 11n is even (because any integer multiple of an even number is even). Therefore, m 11n is odd (because odd minus even odd). Since 2215 odd, the quotient 2215
m-11n will be odd if m 11n is odd. Hence, y = 9 must be odd.
Now, we substitute y into the first equation solve for z:
+ 1988g
Since n is even and 1988q is even (because any integer multiple of an even number is even), the sum of two even numbers even. Therefore, p must be even. So we have:
even, y odd
Thus, the correct answer is (C) is even, is odd. </think>
"pyython
The reasoning above leads to the conclusion, but verify using a simple Python code snippet to ensure there are no calculation errors.
We can test with a few values of n (even) and m (odd) and verify the results. import sympy as
Define the variables x, y sp. symbols(ss y')
sp.bybb ('n', integer=True, even=True) sp.Symbol ('m', integer=True, odd=True)
Define the equations eg1 sp sp (x 1988xy, 27+y, m)
eq2 =
Solve the equations for x and y solution sp. ( (eql, eq2), (x, y)
s_scl = solution[x]
y_sol solution[y]
Check the parity of x and y
x_ritt sp. (x_sol, 2) y_scl_parity Mod (y_sol, (y_sso, 2)
print
y_sol parityy
30

---

Published as a conference paper at ICLR 2026
"output
Mod (2215+n/2215 1988+m/2215 Mod (1988+m/2215, 2)
Mod (-11#n/2215 Mod (n, 2) 2.)
The Python code confirms the parity analysis performed earlier. Here's the detailed reasoning again with the output from the code:
m-1ln Since is odd and 1 11n is even (because is even), m=lin will be odd. Therefore, y q is odd. m 11n odd. And because 2215 odd, y = 2215
1. The solution for y is y
2215
2.The solution for is x 11 22115" + 1988m.
Since (->I1"" is an integer and odd, and 11 times an odd number is odd, and 1988n is even, x= p must be because the of an odd number and an even number is odd plus even (which remains even).
Hence, the final answer is:
Case #2 of Snowball Effeets
QUESTION: 13. Find the sum:
- 100 - 99
98
1 +
2+ + 101 + 102
GrounD TRUTH: 203
RESPONSE: (Low Entropy)
REWARD: 1.0
ENTROPY: 0.2151 COT VALIDITY (JUDGED BY QWEN3-32B): Correct.
To find the sum of the arithmetic sequence given by -100 - - 98 101 + 102, let's break down the problem step-by-step. 1. Identify the arithmetic sequence: The sequence is
1
-100, -9,, 98..... -1 1,2, ..., 101, 102
We split it into separate sequences and then find the of each:
The first sequence is -100, -99, -9.....
The second sequence is 1, 2, ... 101, 102.
2. Sum of the first sequence 100, -99, -98.... 1): The sum of an arithmetic sequence from to b with common difference of can be found using the formula:
n ferm term + last term)
S = 2
where n is the number of terms.
For the first sequence:
First term (a) = -100
Last term (/)
Number of terms (n):
1
-1 (-100)
+ 100
Therefore, n 100.
Using the formula:
100 S
x (- 100 (-1)) 50 (-101) -5050
3. Sum of the second sequence (1, 2, ... 101, 102):
31

---

Published as a conference paper at ICLR 2026
First term (a)
Last term (l) = 102
Number of terms (n)):
102
102
Therefore, 102.
Using the formula:
102
S
(1 + 102) 51 x 103 5253
4. Combine the sums: The total sum is the sum of the first sequence plus the sum of the second sequence:
Sum 5050 5253 203
</think>
Final Answer The sum the arithmetic sequence 100, 99, -98, ..., 1, 1, 2, 101, 102
is:
203
RESPONSE: (High Entropy)
REWARD: 1.0 ENTROPY: 0.2967 COT VALIDITY (JUDGED BY QWEN3-32B) Correct.
To find the of the series, need to break down into manageable parts. The series is essentially two series combined together, one in descending order from -100 to 102 and the other in ascending order from to 101. We'll need to sum these series separately and then combine the results.
The series can be split as follows:
1. The series - 100 - 99 - 98
1
The series
+ ... 101 102
First, let's look at the series 1000 99 98 1. This is single decreasing arithmetic
series from -100 to -1. The sequence can rearranged to form an arithmetic series with
common difference of 11, starting from -100 to -1. The sum of arithmetic series S, is given by the formula:
S, 2 (a1 + a.,)
where n is the number of terms, aj is the first term, and an is the last term.
Here, a1 = 100, an 1, and the number terms n = 100. So, the sum of this series is:
100 - 100 + (-1)) = 50 x (-101) = -505
S = 2
Next, let's look at the series + 2 + 101 + 102. This is an arithmetic series starting from 1 with a common difference of ending at 102. The number of terms 102, so the sum of this series is:
102
$2
(1 + 102) = 51 103 5253
2
Now, combine the two results:
Total sum = S₁ S2 -505 + 5253 203
Let's verify this with Python code.
</think> Here is the Python code to verify the result:
32

---

Published as a conference paper at ICLR 2026
"python
Sum of the series -100 99 98 sum descending sum (range (-100, 0))
from -100 to -1
Sum of the series 1 + 2 sum_ascending sum (range (1, 103))
102 from to 102
Total sum
total_ sum sum sum_descending sum_ascending
print (total_sun
'ouupuu
203
...
The sum of the series 1100 98
+ ... 101 + 102 is 203
Case #3 of Snowball Effects
QUESTION:
## Auxiliary The area helps solve the problem ] Pythagorean Theorem (direct and inverse) In triangle ABC, heights AE and CD are drawn. Find the side AB, if BD 18, BC 30, AF = 20.
GROUND TRUTH: 25
RESPONSE: (Low Entropy)
REWARD: 1.0
ENTROPY: 0.0198 CoT VALIDITY (JUDGED BY QWEN3-32B): Correct.
To solve the problem, we need use the properties of right triangles and the Pythagorean theorem. Here's the step-by-step reasoning:
Identify the given information:
BD 18
BC 30
AE 20
CD is a height from to AB, so CD AB
AE is height from BC, so AE .L BC
2. Determine the position of point D: Since D is the foot of the perpendicular from C to AB,
BD = 18 and BC = 30, we can find AD using the Pythagorean theorem in triangle BCD:
CD VBC BD2 V302 182 /900 324 v 576 24
So. CD = 24.
3. Use the area of the triangle:
The area of triangle ABC can be calculated in two ways:
Using AE and BC: Area
BC x AE = x 30 x 20 = 300
Using CD and AB: Area
AB CD 1 AB 24
Equating the two expressions for the area, get:
600
AB 24 300 = AB 24 600
_ AB
25
2
So. the length of AB 25.
</think>
33

---

Published as a conference paper at ICLR 2026
The final answer is 25
RESPONSE: (High Entropy)
REWARD: 1.0
ENTROPY: 0.3002 CoT VALIDITY (JUDGED BY QWEN3-32B): Wrong.
To solve the problem, need to the properties of the triangle and the Pythagorean theorem. Here is the step-by-step thought process:
Understand the Problem:
We need to find the side AB of triangle ABC.
We know the length of BD 18 and BC 30.
We know the height AE 20.
2. Apply the Right Triangle Properties:
Since AE is height, AE is perpendicular to BC, so triangle AEB and triangle AFC are right triangles.
We the Pythagorean theorem in triangles ABD and AFC.
3. Using the Pythagorean Theorem in AABD:
In AABD, AD can be found using the Pythagorean theorem: AD VAB BD2
4. Using the Pythagorean Theorem in AAEC:
In AAEC, EC can be found using the Pythagorean theorem: EC = VAC AE
Since BC = BD + DC and D is on BC, DC = BC BD 30 - 18 = 12.
AAE,, AC = EC, because AE the height from A BC.
5. Using the Area Equivalence:
The area of the triangle can be calculated in two ways:
Using base BC' and height AE: Area = 2 x BC x AE = 2 x 30 x 20 300.
Using base AB and height CD: Area
1 x AB x CD.
6. Find CD:
From the area equivalence, x AB x CD = 300, so CD
600
AB
Using the Pythagorean Theorem in AACD:
AACD, AC = VAD= CD / (AB2 BD2) + CD2.
Let's use these steps set up the equations and solve for AB. </think>
Using the relationships and the Pythagorean theorem, find AB using the properties of the triangle and the given values. We know:
derive the value of AB. Let's first
BD 18, BC 30, AE 20
First, we calculate CD:
600
CD AB
Now, we use the Pythagorean theorem in AABD:
AD v AB2 182
And in AAEC:
AC =
12a 112
20
/144 + 400 V544 4v34
The area using base BC and height AE is:
1
Area
x 30 x 20 = 300
2
34

---

Published as a conference paper at ICLR 2026
The area using base AB and height CD is:
1
600
Area
AB CD
AB
300
2
AB
Thus,
CD 600
AB We now use the Pythagorean theorem in AABD and AAEC:
AD v AB 182
AB 324
AC / 12 + 202 4v
Since AC is the hypotenuse of AAEC:
AC AE + EC
We solve for AB by equating the areas and solving the equations.
Let's 's now calculate the value of AB using Python to find the accurate result.
'python
import sympy sp
Define the variables AB sp. Sp.Symbol( (BA'', rositive-Truel
Given values
18
BC 30
AE 20
Calculate CD using the area equivalence
600 / AB
Calculate using the Pythagorean theorem in triangle ABD AD sp.sqrt (AB**2 BD*
Calculate AC using the Pythagorean theorem in triangle ABC sp.sqrt (12**2 AE++2)
Using the area equivalence, CD AE BC AB, we a already have CD 600 / AB
Now we solve for AB using the Pythagorean theorem in triangle ABD and AEC
We know AC = sgrt (AD"2 + CD*2) and AD = sqrt (AB 2 - 324) AC_ AyDC sp. (AB**2 324) + (600 AB) ++2)
Equate AC from both triangles since they are the same equation = sp.Eq (AC, AC ff_fromAr ADDC)
Solve for AB ia_value sp.sol solve (equation, AB) print (AB_value)
...
oouuput [25]
The value of AB is 25. Therefore, the final answer is:
25
35

---

Published as a conference paper at ICLR 2026
G PROMPT TEMPLATE
G.1 RLVR TRAINING EVALUATION
We the same prompt template for most models; however, due the limited capability of the 1 8B base model, we adopt a simplified prompt to ensure that responses can be generated under zero-shot settings.
RLVR, Evaluation Prompt
Your task is to follow a systematic, thorough reasoning process before providing the final solution. This involves analyzing, summarizing, exploring, reassessing, and refining your thought process through multiple iterations. Structure your response into two sections: Thought and Solution. In the Thought section, present your reasoning using the format: "<think> An thoughts </think> \n". Each thought should include detailed analysis, brainstorming, verification, and refinement of ideas. After "< "</thing /tk An" in the Solution section, provide the final, logical, and accurate answer, clearly derived from the exploration in the Thought section. If applicable, include the answer in \boxed } for flosed-form results like multiple choices or mathematical solutions.
User: This the problem: (Question)
Assistant: <think>
RLVR, Evaluation Prompt (Llama-3.1 8B Base)
User: (Question)
Answer: Let's think step by step. An
G.2
COT JUDGE
LRM-as- Judgggge Prompt
You are an expert in mathematics and logical reasoning. Your task is to evaluate the correctness of a solution to given math problem, with a strong emphasis on the reasoning process, not just the final answer. Below is the Problem and the Solution (Provided by another Al model):
Problem: (Problem)
Solution (Provided by another AI model): (Solution]
Please perform the following tasks:
1. Analyze the solution step-by-step, paying close attention to:
Computational accuracy
Logical consistency
Conceptual understanding
Whether the reasoning is valid and complete
2. Identify any issues or errors in the reasoning, even if the final answer is correct. Classify them into the following categories (if applicable):
Calculation Error: Mistakes in arithmetic, algebraic manipulation, numerical
computation.
Logical Error: Invalid reasoning, flawed logic, or incorrect inference.
Conceptual Error: Misunderstanding or misuse of mathematical concepts or defini- tions.
36

---

Published as a conference paper at ICLR 2026
Omission / Incompleteness: Missing steps, incomplete justification, not addressing all parts the question.
Other: Any other type of error that does not into the above categories.
3. Provide final judgment on whether the solution is logically sound and free of errors in reasoning.
Please format your response as follows:
Issues Identified:
[[ssue I]: [Classification] [Brief explanation]
[[ssue IJ: [Classification) [Brief explanation]
Let's think step by step and output your final judgment within \boxed }: \boxed[yes) Aboxed }
37


Reviews:
[{"review_id": "0Y99uEdq29", "reviewer": "Reviewer_7NcB", "summary": "This paper addresses the inefficiency of on-policy RLVR for LLM reasoning, where valuable experience is wasted. The authors first identify that intermediate-difficulty problems and low-entropy or high-certainty trajectories are the most valuable training data. Based on this, they propose ExGRPO, a framework with principled experience replay. It maintains a buffer, groups experiences into buckets by real-time correctness, and prioritizes sampling medium-difficulty tasks, selecting the lowest-entropy trajectory for replay. Experiments show ExGRPO significantly outperforms on-policy baselines, especially in out-of-distribution generalization and training stability.", "questions": "1. As the replay buffer grows to millions of items, is the \"lowest-entropy\" selection (Section 4.1) recalculated for all trajectories in a sampled bucket at every step? If so, this step will become a significant computational bottleneck of this method.\n2. How sensitive is the method to the definition of the difficulty buckets? For instance, what happens to performance if the \"medium\" bucket is defined more narrowly or more broadly?\n3. The paper focuses on replaying low-entropy successes. Do the authors experiment with replaying other types of experiences, such as high-entropy successes or even \"instructive failures\"? Is there a risk of the policy becoming too narrow by only focusing on confident, correct solutions?\n4. Regarding the \"Retired Set\" (Section 4.1): questions are retired after 100% success. Does this risk \"catastrophic forgetting\" of these skills? Would it be beneficial to periodically reintroduce a small fraction of \"Easy\" or \"Retired\" problems back into the training mix to ensure the model retains these capabilities?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper's primary strength is its methodical approach. It doesn't just propose experience replay; it first runs a diagnostic study in Section 3.2 to prove that medium-difficulty questions and low-entropy trajectories are the most valuable. This provides a strong, data-driven empirical foundation for the ExGRPO framework's design.\n\n2. The average gain of +7.6 OOD points across all models is significant. It suggests that learning from a curated buffer of past successes helps the model form more robust and generalizable reasoning patterns compared to noisy on-policy exploration, which may overfit to in-distribution data.\n\n3. The paper's novelty lies in the principled application of experience replay to RLVR. The idea of dynamically bucketing experiences by online correctness and then selecting by trajectory entropy is a new and intelligent combination of ideas that goes beyond naive replay.\n\n4. The method is not based on an empirical heuristic. The appendix provides a solid theoretical analysis of the mixed-policy objective, its variance bounds, and the role of policy shaping. This adds significant depth and soundness to the proposed framework.\n\n5. The paper is well-written. It logically builds its argument from the preliminary study to the final method, and its explanations of complex concepts are intuitive. The figures and tables are clean, informative, and effectively support the main claims.", "weaknesses": "1. The method introduces a replay buffer and additional computation for entropy calculation and sampling per step. A discussion on the impact on memory usage and overall training throughput is missing, which makes it hard to assess the method's true efficiency trade-offs.\n\n2. The entire method hinges on the \"medium-difficulty\" bucket, defined as 25%-75% correctness. This range is presented without justification and is not tested in the ablations. It's unclear how sensitive the model's performance is to this definition. A sensitivity analysis, for example, showing results for 30-70 or 40-60, is a critical missing piece.\n\n3. The method exclusively replays low-entropy successes. This strategy risks two problems: a) it discards \"instructive failures\" which are known to be valuable for learning, and b) it may reduce policy exploration by only reinforcing high-confidence, correct paths, potentially leading to a less diverse policy. The authors do not sufficiently justify this \"success-only\" design choice over other possibilities.\n\n4. The reliance on a binary \"correctness\" signal for bucketing makes the method's application to standard RLHF unclear. These tasks use continuous rewards from a preference model, such as for helpfulness, and it's not obvious how to define \"difficulty\" buckets in that context. This limits the method's generalizability", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RD8KLjvZr6", "reviewer": "Reviewer_o61x", "summary": "The paper introduces ExGRPO (Experiential Group Relative Policy Optimization), a reinforcement learning framework designed to enhance the reasoning ability of large language models (LLMs) under the Reinforcement Learning with Verifiable Rewards (RLVR) paradigm. Unlike conventional on-policy RLVR methods that discard rollout experiences after each update, ExGRPO reuses past successful experiences through a structured experience management and replay mechanism. The framework partitions experiences by rollout correctness and trajectory entropy, selecting medium-difficulty, low-entropy samples for replay. Experiments across multiple backbones (1.5B–8B models, including Qwen and Llama families) demonstrate consistent improvements in mathematical reasoning performance.", "questions": "* line 258: The choice of the Gaussian distribution seems heuristic here. I am wondering if there is any theoretical justification or empirical evidence to support this choice.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Quality: The empirical dynamics of the algorithm are well investigated by the analysis in Section 6, where theoretical justifications are also provided to ensure the unbiasedness of the proposed formula in Equation 4. Experimental settings follow the standard setup in this field.\n\n2. Clarity: This paper is well written and easy to follow.\n\n3. Novelty: The proposed experience replay management and its combination with on-policy training look new to me, though the accuracy- and entropy-based approach is not new, given the existence of similar literature [1][2] in the past.", "weaknesses": "1. Significance:\n  - The selection criteria depend on the metric of correctness, which requires the reward signal to be verifiable. This introduces additional challenges when extending to more general open-domain tasks.\n  - While ExGRPO achieves better performance than the on-policy baseline, it also incurs additional time and memory costs for accuracy recomputation and experience storage. It is worth discussing whether those costs will pose practical challenges or if there are trade-offs between cost and performance.\n\n2. Quality: \n  - More baselines are encouraged to be compared with, such as [1][2][3].\n\n  - Minor issues:\n    + line 276: typo: \"a advantage group\" -> \"an advantage group\"\n\n### References\n[1]: Zhang, Jixiao, and Chunsheng Zuo. \"Grpo-lead: A difficulty-aware reinforcement learning approach for concise mathematical reasoning in language models.\" arXiv preprint arXiv:2504.09696 (2025).\n\n[2]: Chen, Minghan, et al. \"Seed-grpo: Semantic entropy enhanced grpo for uncertainty-aware policy optimization.\" arXiv preprint arXiv:2505.12346 (2025).\n\n[3]: Wei Xiong, et al. A minimalist approach to llm reasoning: from rejection\nsampling to reinforce. arXiv preprint arXiv:2504.11343, 2025.\n\n[4]: Qiying Yu, et al. Dapo: An open-source llm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476, 2025", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Nst3TTLKHt", "reviewer": "Reviewer_B8SA", "summary": "This paper addresses the computational inefficiency and instability of standard on-policy Reinforcement Learning from Verifiable Rewards (RLVR) for training large reasoning model. The authors first investigate what constitutes a valuable reasoning experience, identifying rollout correctness (favoring medium-difficulty problems) and trajectory entropy (favoring low-entropy chains) as effective indicators of quality. Based on these insights, they propose ExGRPO (Experiential Group Relative Policy Optimization), a framework that organizes and prioritizes these high-value experiences in a replay buffer. ExGRPO employs a mixed-policy objective that strategically balances fresh exploration with the exploitation of these selected past successes.", "questions": "1. It seems difficult to regard a task with 25% accuracy as having the same learning value as one with 75% accuracy. Have you considered adopting a more fine-grained categorization or another prioritization scheme?\n\n2. For each sampled query from the buffer, it is mentioned that the entropy of all stored candidate trajectories needs to be recomputed at every step with respect to the current policy ($\\pi_{\\theta}$). I’m wondering whether the computational overhead of this process is negligible.\n\n3. To validate the core heuristic of “low entropy,” we used a much more powerful model, Qwen3-32B, as an external judge compared to the 7B target model. I’m curious whether similar patterns would emerge with other models — does this imply that a significantly more capable “teacher” model is required?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Simple but Clear motivation\n\n2. Simple implementation\n\n3. Clear improvement compared to GRPO\n\n4. Various experiments", "weaknesses": "1. Depending on highly heuristic results", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bj3cUR4s8x", "reviewer": "Reviewer_wSTN", "summary": "This paper proposes ExGRPO (Experiential Group Relative Policy Optimization), a reinforcement learning method tailored for large language models (LLMs) in the RLVR (Reinforcement Learning from Verifiable Rewards) paradigm. ExGRPO addresses the inefficiencies of on-policy RLVR, which discards prior successful trajectories, by designing a principled experience management pipeline. The key innovations include:\n\nIdentifying rollout correctness and trajectory entropy as indicators of experience quality;\n\nOrganizing trajectories by correctness buckets and selecting low-entropy paths for replay;\n\nA mixed-policy optimization objective that blends off-policy experience replay with on-policy rollouts, using importance reweighting and policy shaping to control exploration–exploitation tradeoffs;\n\nA delayed start mechanism to avoid low-quality replay at early stages.\n\nExtensive experiments on multiple reasoning benchmarks (AIME, MATH-500, Olympiad, ARC-c, GPQA, etc.) and across backbone models (Qwen, LLaMA, LUFFY, etc.) demonstrate consistent performance gains (+3.5/+7.6 points in-domain/OOD) and improved stability over strong baselines like GRPO and RePO.", "questions": "How would ExGRPO perform on tasks with non-verifiable or sparse rewards, such as creative reasoning or open-ended dialogue?\n\nCould the valuable-but-incorrect trajectories (those with correct reasoning but wrong final answers) also be leveraged instead of being discarded?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "Well-motivated problem: The paper targets a real bottleneck in RLVR—wasteful discard of prior trajectories—and connects this with known issues in experience replay for RL and LLMs.\n\nPrincipled design of experience value: The use of rollout correctness buckets and entropy as quality proxy is backed by a detailed analysis (Fig.1), offering clear guidance for trajectory selection.\n\nStrong methodology: ExGRPO is conceptually elegant and practically implementable, combining well-established principles (importance weighting, entropy shaping) into a novel RLVR-specific pipeline.\n\nThorough evaluation:\n\nTested on five models (1.5B–8B);\n\nCovers both in-distribution (math) and out-of-distribution reasoning tasks;\n\nIncludes ablation studies (Fig.7), training stability analysis (Fig.4), and replay dynamics (Fig.5–6);\n\nAchieves SOTA or near-SOTA results (Table 1), including on strong baselines (Oat-Zero, PRIME-Zero, RePO-Zero).\n\nStabilizes weak models: The method prevents collapse in Llama-3.1 8B, a previously unstable model under on-policy RLVR.\n\nOpen-sourcing & reproducibility: Authors commit to releasing code and weights and provide training details and an extensible GitHub repo.", "weaknesses": "Although ExGRPO effectively improves efficiency and stability in RLVR training, it remains heuristic?. The method is also limited to verifiable reasoning tasks and has not been validated on open-ended domains like dialogue or code generation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)