EEPO: Exploration-Enhanced Policy Optimization via Sample-Then-Forget
Authors: Liang Chen
Xueting Han
Qizhou Wang
Bo Han
Jing Bai
Hinrich Schuetze
Kam-Fai Wong
Venue: ICLR 2026 Poster (2026)
Keywords: large language models
reasoning models
reinforcement learning
RLVR
exploration
unlearning
Primary area: foundation or frontier models, including LLMs
TL;DR: We propose EEPO, which enhances exploration in RLVR by temporarily suppressing sampled trajectories during rollouts, achieving 10-33% improvements across mathematical reasoning benchmarks.

Abstract: Balancing exploration and exploitation remains a central challenge in reinforcement learning with verifiable rewards (RLVR) for large language models (LLMs). Current RLVR methods often overemphasize exploitation, leading to entropy collapse, diminished exploratory capacity, and ultimately limited performance gains. Although techniques that increase policy stochasticity can promote exploration, they frequently fail to escape dominant behavioral modes. This creates a self-reinforcing loop—repeatedly sampling and rewarding dominant modes—that further erodes exploration. We introduce **E**xploration-**E**nhanced **P**olicy **O**ptimization (**EEPO**), a framework that promotes exploration via two-stage rollouts with adaptive unlearning. In the first stage, the model generates half of the trajectories; it then undergoes a lightweight unlearning step to temporarily suppress these sampled responses, forcing the second stage to explore different regions of the output space. This *sample-then-forget* mechanism disrupts the self-reinforcing loop and promotes wider exploration during rollouts. Across five reasoning benchmarks, EEPO outperforms GRPO, achieving average relative gains of 24.3\% on Qwen2.5-3B, 33.0\% on Llama3.2-3B-Instruct, and 10.4\% on Qwen3-8B-Base.

Published as a conference paper at ICLR 2026
EEPO: EXPLORATION-ENHAANE POLICY OPTIMIZA-
TION VIA THARLE-THEN-FOGGE
Liang Chen Xueting Han?
Qizhou Wang?
Bo Han
Jing Bai?
Hinrich Schütze²
Kam-Fai Wong 1.3
The Chinese University of Hong Kong
Microsoft Research Asia *LMU Munich
HHon Kong Baptist University
MoF Key Laboratory of High Confidence Software Technologies
{ichen, kfwong] wuh..edu...... hk
chr.hanemiicroootttccom
ABSTRACT
Balancing exploration and exploitation remains central challenge in reinforcement learning with verifiable rewards (RLVR) for large language models (LLMs). Cur- RLVR methods often overemphasize exploitation, leading entropy collapse, diminished exploratory capacity, and ultimately limited performance gains, Al- though techniques that increase policy stochasticity can promote exploration, they frequently fail to escape dominant behavioral modes. This creates a self- relf-reinforcing repeatedly sampling and rewarding dominant modes that further erodes exploration. We introduce Exploration- -aariice-- Policy Optimization (EEPO), a framework that promotes exploration via two-stage rollouts with adaptive unlearn- ing. In the first stage, the model generates half of the trajectories; it then undergoes a lightweight unlcarning step to temporarily suppress these sampled responses, forcing the second stage to explore different regions of the output space. This cample-then-forge mechanism disrupts the self-reinforcing loop and promotes wider exploration during rollouts. Across five reasoning benchmarks, EEPO out- performs GRPO, achieving average relative gains of 24.3% on Qwen2.5-3B, 33.0% Llama3.2-33 Insttrut, and 10.4% on Qwen3-8B-Base.
INTRODUCTION
The emergence of OpenAl's ol (OpenAl) and DeepSeek-R1 (DeepSeek-Ai et al., 2025) marks significant advance in LLM reasoning. A key driver of this progress is reinforcement learning with verifiable rewards (RLVR) (DeepSeck-AI al. 2025), powered by the Group Relative Policy Optimization (GRPO) (Shao et al. 2024). Nevertheless, RLVR continues to face the classic explo- ration-exploitation dilemma (Sutton & Barto 2018) due to the exploitative nature of its objectives. Specifically, policies tend to over-emphasize exploitation of high-reward trajectories, leading to entropy collapse and reduced final performance Yu al. 2025 Cui et al. 2025).
In this work, examine entropy collapse Qwen2.5-3B. We observe that entropy declines sharply, in-distribution test accuracy continues to rise, whereas performance on out-of-distribution benchmarks (e.g., AMC 2023) deteriorates (Figure 2 This suggests reduced exploration drives overfitting to the training distribution rather than discovering generalizable reasoning patterns. We hypothesize that, as entropy falls, the policy forms increasingly confident beliefs about solutions, yielding response distribution with multiple, imbalanced modes (Figure B3) several plausible rea- soning behaviors exist for given question, but one mode receives more probability mass. If rollouts predominantly sample this dominant mode and receive positive feedback, the policy further amplifies while suppressing alternatives (Figure (3)). This self-reinforcing loop accelerates entropy collapse. Crucially, it impedes the discovery of alternative-poteme local optima and poor generalization.
reasoning strategies, causing
Work was done during I. Liang Chen's internship at MSRA. Corresponding Xucting Han and Kam-Fai Wong.

---

Published as a conference paper at ICLR 2026
GRPO
Policy Optimization
01
F1
A
q
Policy Model
Rollout Model
O2
F2
A2
Og
FG
Ag
Trained Model
EEPO
Policy Optimization
Frazen Model
Unlearning
01
71
A
Stage
0G/2
Policy Model
Rollaut Model
F2
A2
q
Stage 2
G//2
FG
AG
GG
Figure 1: Comparison of GRPO and EEPO rollout processes. GRPO samples all trajectories from fixed rollout model, while EEPO introduces an unlearning step on the rollout model between two sampling stages to promote exploration of diverse modes.
Recent efforts to improve exploration in RLVR largely fall into two categories: objective-level modifications and indiscriminate exploration. Approaches such as increasing the sampling temper- ature (Ziegler et al. 2019) or adding entropy regularization (Hou et al. 2025) flatten the output distribution uniformly (Figure Bp). While this increases stochasticity, it fails to shift probability mass away from dominant behaviors and often yields instability degraded performance when applied aggressively (Figure 5). A widely adopted recent approach, DAPO (Yu et al. (022)., increases the upper clipping threshold to grant low- probability trajectories fewer restrictions during training, Yet these objective-level tweaks do not break the sell-reinforcing loop: during rollouts, the policy remains confined dominant modes and fails to explore beyond previously sampled high-pu high-probability regions.
To address this problem, we propose Exploration Exploration-Enbaneed Policy Optimization (EEPO), method that promotes exploration by preventing repeated sampling from dominant modes during rollout. Specifically, EEPO introduces sample-then-forge mechanism that divides the GRPO rollout into two stages, as shown in Figure [ the rollout model first generates half of the trajectories, then performs temporary unlearning step to suppress the just-sampled responses. The remaining trajectories are sampled from this updated model. Unlike objective-level approaches, this mechanism operates directly within the rollout process, explicitly encouraging subsequent samples to deviate from dominant behaviors and uncover alternative trajectories seteere steering exploration toward broader regions, illustrated in Figure
To adapt the unlearning intervention to RL exploration, we introduce three design choices that make it targeted, triggerable, and lightweight. First, impose stronger penalties dominant regions, we replace the standard negative log-liked log-likelihood with a complementary loss that penalizes high- probability tokens more than low-probability ones. Second, to trigger intervention at the onset of gating mechanism that activates unlearning only
mode collapse, we introduce an nntropy-conditions when exploration deteriorates (i.e., low entropy). Finally, keep the intervention lightweight and temporary, we apply single-step gradient update to the GRPO rollout model--Synhhoon -ynchronized from the actor each iteration and used solely for sampling thereby decoupling unlearning from policy optimization and confining its effect to the rollout phase.
To validate our approach, we evaluate EEPO on five challenging mathematical reasoning benchmarks using three distinct LLMss. The benchmarks include Minerva Math (Lewkowycz et al.) (022). OlympiadBench (He et al. 2024), and three competition-level datasets: AMC 2023, AIME 2024, and AIME 2025. EEPO consistently outperforms the baselines, yielding average relative improvements over GRPO of 24.3% on Qwen2.5-3B, 33.0% on Llama3.2-3B. I-Ittruut and 10.4% on Qwen3- -B-- Base. Furthermore, our analyses show that EEPO achieves superior performance through more effective exploration while maintaining comparable training time to standard GRPO. The code will be available at httpt::tp//... / thub. ann//EEE
2

---

Published as a conference paper at ICLR 2026
2 PRELIMINARIES
We begin by reviewing RL with Verifiblee Rewards (RLVR) (DeepSeek-Al et al. (025) and its prevalent implementation, Group Relative Policy Optimization (GRPO) (Shao et al, 2024). which has been widely adopted for training large-scale reasoning models. We then analyze its limitations related to insufficient exploration and revisit existing solutions attempted to mitigate this issues.
2.1 RL FOR TRAINING LARGE-SCALE REASONING MODELS
The success of RLVR relies on reliable reward signals (DeepSeek-Al et al. 2025), typically provided by a rule-based reward model that delivers precise feedback for tasks in mathematical. coding, and logical reasoning domains. Consider mathematical dataset D == { a)}, where is question and is ground-truth final answer The reward depends solely on the correctness of the final prediction à compared to without enforcing constraints the reasoning process:
RLVR.
r(a a) [ a].
The RLVR objective is often implemented using the large-scale policy optimization method GRPO. Compared to proximal policy optimization (PPO; |Schulmnn et al., 2017). GRPO improves computa- tional efficiency by eliminating the need for separate value function.
GRPO. As illustrated in Figure given question g and set of responses, i.e., reasoning paths, 0 = for, ..,..., ,.G sampled from the old policy model Tola, GRPO directly computes advantages to optimize the policy model x using the following objective:
lo.l
JGRPO (0)
min (r..(()A,, clip
A,] -BDKI,Tm
|| Teel]
(2)
Here, Ter denotes a reference model used to constrain policy updates via a KL divergence penalty. The score A, represents the normalized advantage of response o;, computed as A; meanm Feal)
(ri
where (ri,...,ral denotes the rewards corresponding the sampled responses in the group O.
The importance weight (0) denotes the probability ratio between current and old policies:
(os q. O..sc)
Ti.e (0)
(3)
To,0 (o gi.cc)
This importance sampling ratio crucial for obtaining unbiased gradient estimates when responses are sampled from Tota rather than the current policy mo-
2.2 REVISITING THE INSUFFICIENT EXPLORATION PROBLEM
We examine the exploration problem through entropy and performance changes on test and OOD benchmarks to char- acterize the issue and its implications. Figure 2 presents our analysis of GRPO's ' behavior during training on the MATH dataset. We observe two interconnected phenomena:
GRPO Training Dynamics
70
0.0
60
96
0.6
Entropy
(1) Rapid entropy collapse: Despite incorporating substantial entropy regularization (A x 10 3f the policy entropy decreases precipitously within the first few training steps, in- dicating rapid convergence to deterministic behaviors. This collapse stems from GRPO's inherently exploitative objec- tive function (Equation 2). which prioritizes reward maxi- mization exploration.
0.4
30
20
10
25
50
75
1000
125
Training Steps
Entropy
Testset
AMC23
(2) Deteriorating generalization: As entropy collapses, we observe divergent trend: while test accuracy continues to improve, performance on OOD benchmarks such as AMC declines. This suggests that reduced exploration causes the
Figure 2: GRPO training dynamics:
rapid entropy collapse accompanies rising Testset and decline on AMC23.
3

---

Published as a conference paper at ICLR 2026
(a) Policy Distribution
(b) self-reinforce Effect
(c) Adding Randomness
repeated sampling ling
more uniform
reward
Figure 3: Illustration of exploration challenges in GRPO. (a) Policy distribution showing imbalanced modes with dominant peak. (b) Self-reinforcement effect where the dominant mode becomes increasingly concentrated through positive feedback. (c) Effect of adding randomness (e.g., entropy regularization) which flattens the distribution but maintains the relative dominance of modes.
model to overfit to the training distribution rather than learn robust reasoning patterns that generalize to OOD benchmarks.
To explain entropy collapse, we hypothesize that when entropy begins to decline, the policy has formed partial, uncertain beliefs about the problem. this regime, its response distribution contains multiple modes multiple plausible reasoning traces cocxist for given question. These modes are often imbalanced: one dominant mode accumulates disproportionate share of probability mass, as illustrated in Figure 3 a). When responses predominantly sampled from this dominant mode and receive positive feedback, the policy reinforces it further, amplifying its probability while suppressing alternative responses. The distribution evolves toward increasing imbalance, as shown in Figure ()). This self-reinforcing dynamic creates a feedback loop that inhibits exploration and ultimately leads to entropy collapse. This is particularly problematic: once correct dominant mode emerges, it can prevent the discovery of alternative, potentially superior strategies, yielding local optima and limiting generalization to OOD benchmarks.
Current approaches to enhance exploration primarily increase randomness during optimization or sampling, such as strengthening the entropy term or raising the sampling temperature. These methods flatten the policy distribution toward a more uniform shape, as depicted in FigureBjc However, they do not disrupt the self-reinforcing loop: the dominant mode remains the most likely to be sampled even after flattening. This motivates our central question: How can we enable the policy to explore plausible behaviors beyond the dominant mode during rollout?
3 METHOD
3.1
POLICY OPTIMIZATION
To address the self self-reinforcing dynamics that lead to entropy collapse, we propose Exploration- Enhanced Policy Optimization (EEPO), which prevents the rollout model from repeatedly sampling from dominant modes by unlearning previously sampled responses during rollout generation.
Figure D illustraaes the key difference between GRPO and EEPO GRPO, the rollout model Tollont (corresponding Vold in Equation 2 samples all responses O {01, .., oG) from a fixed distribution, which are then used to compute rewards and advantages for policy optimization. EEPO introduces sample-then-forget mechanism that divides the rollout into two stages separated by an unlearning step:
Stage / sampling: Sample G/2 trajectories {0i, 02 .2. o///2) from Tolloot
Unlearning:
Update Tollout to forget the sampled trajectories.
Stage 2 sampling: Sample the remaining trajectories {oa/2+1......0 )o from the updated model.
After collecting all G trajectories across both stages, compute their rewards and apply the standard GRPO objective (Equation 2 to update the policy model. The denominator in Equation (3) uses the rollout model's probabilities, ensuring unbiased gradient estimates. Following standard GRPO practice, the rollout model is synchronized with the policy model at the beginning of each iteration, making the unlearning effect temporary and confined to the current rollout.
"This value significantly larger than the 10-4 suggested by SimpleRI. 7eng al. 2025
4

---

Published as a conference paper at ICLR 2026
This approach decouples policy optimization from exploration: while the policy model To focuses on
(a) Stage
(b) Stage
sampling
reward maximization, the rollout model actively ex-
(o [x)
sampling
plores alternative trajectory spaces by suppressing previously visited regions. As illustrated in Fig-
unlearn
ure I the unlearning step redistributes probability mass from dominant modes other plausible re- gions, encouraging Stage to sample from previ- ously underexplored areas and effectively breaking the self-reinforcing loop that causes entropy collapse.
Figure 4: Unlearning suppresses the dominant mode and enables exploration of alternative modes that would otherwise be hard reach.
3.2
ADAPTIVE UNLEARNING FOR ROLLOUT EXPLORATION
We now instantiate EEPO with an adaptive unlearning mechanism tailored to rollout-side exploration. The objective is to temporarily suppress dominant modes in Tollout when exploration begins to deteriorate. We identify three desiderata: (a) activate at the onset of entropy collapse to avoid disrupting healthy exploration, (b) penalize dominant regions more than others, and (c) remain lightweight and temporary. We realize these desiderata with three simple designs.
To meet desideratum (a), we activate unlearning only during low-entropy phases; when entropy is high, no intervention is applied. We implement this via an entropy-based indicator:
Entropy activation
I H <a]
l
(4)
where a > 0 is threshold and H!! is the m-step moving average of token-level entropy at step t:
F
H4-
Here H4 denotes the token-level entropy step t (computed from Trollou g. Oc:)). A short horizon (e.g.. m = 3) promptly detects low-entropy phases. This indicator multiplicatively gates the unlearning loss defined below.
Complementary unlearning loss with prediction probability: strong dominant regions with high probability mass and weak else- where. However, maximizing the standard negative log- ligelikl (NLL) runs counter to our goal.
To meet desideratum (b), unlcarning strength should increase
LNLL - log Frollout( Q1 ok,<t);
since it penalizes low-probability predictions than high-probabilin ones (the loss goes to 0 as probability approaches 1). We therefore use complementary loss that reverses this emphasis:
Ccmmp log( 1
Tyotiou(Ok. q, oR,<()),
which imposes stronger penalties on probabiittyy (dominant) predictions and weaker penalties on probabiioy ones.
To ensure numerical stability as Frollou(Ok..) + 1, we clip the probability before applying the loss:
Pelip = min (Tollour (ox.e q, 0k,ce) c),
(8)
where is small constant that prevents Pelip from approaching zero. The stabilized loss is:
Ccomp
log(1 Peti).
Temporary single-step updates To meet desideratum (c), we apply single-step update to optimize the unlearning objective and confine its effect the rollout model within each iteration. Let O% (Ok,1, O&,T, T. denote the k-th trajectory in the stage-1 rollout set O {or, 02, 0G/2) The entropy- condopy-c unlearning loss over O is:
Tv
((O1)
Σ
I.
Log(1 - Petip (o)))(
(10)
Oil
T.
o0E0;
5

---

Published as a conference paper at ICLR 2026
Algorithm I: ЕЕPO Exploration Enhaanced- Policy Optimization
Initialize: policy 09; learning rates JJGRPo, 'i; group size G; iteration K; entropy threshold for k = to K do
Sample q ~ D: set 0' +- gh sample query and synchronize rollout from policy
Sample {o: G/2 TTr(- 1g)
// Stage I: sample G/2 trajectories
(T))) < a then // single-step adaptive unlearning
0' - 0' n Vg. [(1o))))) G/22)
end if
Sample {oi G=/211 ~ Nep (-1g) // Stage 2: sample remaining trajectories
Form O + {o
G and compute advantages {AEO
g t- 8h + JJRFFO VoJaRrO (f%; r) // update policy with GRPO
end for
where Pelip denotes the clipped probability and Tl is the entropy-base -asee activation indicator. We then perform a single gradient ascend step without momentum to unlearn these trajectories:
- 8' V L(8').
(11)
where 0' parameterizes the rollout model, which is synchronized from the policy model (parameterized by 0), 0' - 0, as in GRPO's implementation (see Figure Consequently, the unlearning effect is cemporary-continee to the rollout model within the current iteration, without accumulation- does not alter the policy parameters optimization.
Algorithm summarizes the EEPO procedure. follows GRPO's structure but incorporates adaptive unlearning between the two rollout stages. After sampling the first G/2 trajectories (Stage 1), check if policy entropy falls below threshold a. so, we perform single gradient step to unlearn these trajectories using the complementary loss, temporarily modifying only the rollout model. We then sample the remaining G/2 trajectories (Stage 2) from the potentially modified rollout model. Finally, we update the policy with GRPO's objective all G trajectories. Note that Eq. 3 the denominator is computed using the rollout model Ta. that generated each trajectory.
4
EXPERIMENT
4.1 EXPERIMENTAL SETUP
Datasets. We train on the MATH dataset (Hendrycks et al.
2021a) using 8.5K hard problems
(difficulty levels 3-5) following SimpleRI. (Zeng et al.] 2025).
We evaluate five mathematical
reasoning benchmarks: Minerva Math (Lewkowycz et al. (022), OlympiadBench (He et al. 2024).
AMC 2023, AIME 2024. For the stronger Qwen3-8B-Bare BBaeee,,, we additionally include AIME 2025.
Models. We experiment with three LLMs: Qwen2,5-3B (Yang el al. 2024). Llama-3.2-3B-Instne
TTea 2024), and Qwen2.5-7BBInsttreeee Yang al. 2024).
Training Details. We employ binary reward (+1 for correct answer, 0 otherwise) without format constraints. All models are trained using VERL (Sheng al. 2024) with GRPO for epochs, using batch size 128, learning rate 10-? and 8 rollouts per question. For EEPO, we set entropy threshold a 0.3 and unlearning rate r = 3 x 10-3,
Further details of the experimental setup are provided in Appendix A
4.2 BASELINES.
We compare EEPO to GRPO and other methods explicitly designed to enhance exploration.
Base/Instruct Model. The base model, or its instruction-tumed variant without additional reasoning- specific training, serving as performance lower bounds.
GRPO. GRPO applied to the base instruction-tuned model using standard training settings.
With Increased Entropy Term. This variant encourages exploration by increasing the entropy weight in the objective function, prompting the actor to generate more diverse outputs.

---

Published as a conference paper at ICLR 2026
With Higher Sampling Temperature. Applies higher sampling temperature during actor's decoding process to promote exploration and reduce output determinism. With DAPO's Clip Higher. Incorporates the "clip higher" technique from DAPO encourage the selection of rare tokens during training.
With More Rollouts. Expands the exploration space by increasing the number of rollouts per training step, enabling broader trajectory sampling.
4.3 EXPERIMENTAL RESULTS
Table I: Performance of EEPO compared to baselines on Qwen2.5-3B across four math benchmarks. Baseline results report the best performance across different hyperparameter settings (refer Fig. 55 Average relative performance improvements (%) over GRPO are highlighted in blue.
Minerva Math
Olympiad Bench
Method
AMC 23
AIME 24
Average
Qwen2.5-3B
11.8
7.9
20.0
0.0
9.9
GRPO
22.4
27.9
30.3
3.3
21.0
Higher Temp.
25.0
25.2
32.5
3.3
21.5
Increased Ent.
25.0
29.6
37.5
3.3
23.9
DAAPO Clip High.
22.1
26.1
40.0
3.3
22.9
More rollouts.
21.7
26.8
37.5
6.7
23.2
ЕЕPО
23.5 (+4.9)
29.3 (+5.0)
45.0
(+50.0)
6.7 (+103.0)
26.1
(+24.3)
Overall results across three LLMs. To validate the effectiveness of our method across differ- ent models and scales, we compare EEPO with baselines on three model families Qwen2.5-3B,
Llama3.2-33 I-33-Innttruct, and Owwen-8B-Base Tables | report the results. EEPO consistently out- performs GRPO and all exploration-enhanced GRPO variants across models and scales. Relative to standard GRPO, EEPO improves average accuracy by 24.3% Qwen2.5-3B (21.0% - 26.1%) 33.0% on 1ammaa.2--3BImmaa This pattern indicates that EEPO's 's comple-then-fie scales from 3B to 8B parameters and transfers across base and instruction-tune policies, providing a robust and model-agnostic improvement for mathematical reasoning under RLVR.
(17.6% + 23.4%), ). and 10.4% on Qwen3- Bann3-BBBBr B-Base (34.7% 38.3%).
mechanism yields targeted exploration that
Table 2: Performance on Llama3.2-3B 2--33--nnttruct.
Minerva Math
Olympiad Bench
Method
AMC 23
AIME 24
Average
14.3
12.1
20.0
10.0
14.1
GRPO
19.5
17.5
20.0
13.3
17.6
Higher Temp.
20.6
19.1
22.5
10,0
18.1
Increased Ent.
20.2
18.1
30.0
10.0
19.6
DAPO Clip High.
19.1
17.3
25.0
16.7
19.5
More rollouts.
19.1
17.2
22.5
16.7
18.9
EEPO
20.6 (+5.6)
18.1 (+3.4)
35.0
(475.0)
20.0 (+50.4)
23.4 (+33.0)
Comparison with baselines. We compare EEPO four exploration strategies, each evaluated its best hyperparameter setting (Figure 5). Despite careful tuning, all baselines fail to match EEPO's performance. While these strategies can outperform GRPO, gains are modest and require brittle tuning. Temperature-based exploration exhibits clear cetoorrtioonexpliaaiiiii
trade-off: performance peaks around 1. 1.2 but degrades sharply at higher values (1.5). We also observe substantially longer training time the best temperatures exploration (Figure 8). Clip- Hiip-higher and entropy regularization likewise swing between under- and over-exploration and lag behind EEPO across all models. Increasing the number of rollouts provides benefits but plateaus quickly while computational cost also grows substantially (Figure K). In contrast, EEPO achieves larger gains by enabling targeted exploration within the rollout process.
(1.2) due to the much longer reasoning paths caused by inefficient
7

---

Published as a conference paper at ICLR 2026
(a) Higher Temperature
(b) Increased Entropy
(c) DAPO Clip Higher
(d) More Rollouts
28
25
Accuracy (16
26
26
20
24
24
24
22
22
22
20
20
20
1.0
1.2 Sampling temperature
1e-3 2e-3
Se-3
0.20
0.28
0.40
12
Entropy coofficient
Clip high ratio
Number rollouts
Baseline
EEPO (fixed)
Figure 5: Impact hyperparameter choices on baselines performance using Qwen2.5-3BB Each subplot shows the average accuracy across four math benchmarks as a function of (a) temperature, (b) entropy coefficient, (c) clip higher ratio, and (d) number of rollouts. The orange dashed line represents the EEPO with fixed hyperparameters.
Generalization to benchmarks. To assess generalization, we evaluate EEPO against baselines on five diverse math reasoning benchmarks, as shown in Tables + Our method achieves consistent improvements over GRPO across all benchmarks. Performance continues to improve on harder and distribution-shiftee splits where baselines plateau. On competition-leve benchmark with Qwen2.5-3B. EEPO reaches 45.0% compared to 30.3% for GRPO. These gains stem from EEPO's sustained exploration and superior entropy maintenance (Figures 6 and 77,, which prevent the entropy collapse that leads to overfitting on the training distribution and degraded generalization (Figure
Table 3: Performance on Qwen3-8B Beenn-8B-Baas
Minerva Math
Olympiad Bench
Method
AMC 23
AIME 24
AIME 25
Average
Qwen3-8B-
33.1
36.0
52.5
10
13.3
29.0
GRPO
41.2
45.5
50.0
20.0
16.6
34.7
Higher Temp.
40.1
44.3
55.0
16.7
20.0
35.22
Increased Ent.
40.4
42.8
60.0
16.7
20.0
35.9
DAAO Clip High.
40.1
41.6
55.0
16.7
10.0
32.7
More rollouts.
40.8
44.0
57.5
16.7
16.7
35.1
EEPO
41.5
44.3
62.5
20.0
23.3
38.3 (+10.4)
5 ANALYSIS
Exploration Enhancement and Quality Preservation. To understand the effectiveness of EEPO, we compare its training dynamics with GRPO, shown in Figure 6
Effectiveness of EEPO:
The entropy dynamics in Figure 6 aa) reveal how tamp-eethen-forget changes exploration behavior. While GRPO exhibits continuous entropy collapse indicating that responses samples increasingly concentrate on high-probability modes, EEPO maintains consistently higher entropy throughout training. Notably, EEPO's Stage 2 achieves higher entropy than Stage 1, suggesting that temporary response suppression successfully forces the model to explore low-density regions that the original actor rarely visits. This entropy gap demonstrates that our mechanism effectively prevents mode collapse by strategically sampling from diverse regions of the probability distribution.
Despite this enhanced exploration, generation quality remains preserved. Figure TigureBbb- bb-c) shows that both mean rewards and response lengths of EEPO remain stable and comparable to GRPO. These results validate our hypothesis: temporarily suppressing sampled responses can enhance exploration by steering the actor away from high-probability regions toward other plausible alternatives, while preserving the generation capabilities necessary for effective training.
Generalization and Pass @ k. Figure 7 shows that EEPO delivers better generalization dynamics and higher final performance on AMC23 (Figure 7)) with improved Pass @k scaling as the sampling budget increases (Figure 7). By mitigating entropy collapse (see Figure (b)) and maintaining higher
8

---

Published as a conference paper at ICLR 2026
Entropy
Reward
Response Langth
Mean Despoase Leapth
I
Moan Rowart
0.5
Entropy
94-
0.3
600
Training Steps
Training Steps
Training Steps
CRPO
EEPO EEPO (Stage
EEPO (Stage
Figure 6: Training dynamics comparison between EEPO and GRPO. (a) Entropy evolution shows EEPO maintains higher exploration ability throughout training, with Stage 2 exhibiting increased entropy compared to Stage 1, demonstrating effective exploration enhancement through 'sample- then-forget' mechanism. In contrast, GRPÓ exhibits monotonic entropy decay. (b) Mean reward trajectories remain comparable between methods and across EEPO stages. (c) Response length
distributions show similar patterns, indicating preserved generation quality.
(a) Accuracy
(b) Pass@kk
GRDO
80.0
Accuracy (%)
35-
75.0
67.5
125
Training Steps
Figure 7: Performance comparison GRPO and EEPO on AMC23 benchmark using Qwen2.5-3B. (a) Training accuracy dynamics; (b) Pass@k k scaling with sampling budgets. EEPO achieves higher final performance and better scaling with increased computation.
policy entropy, EEPO continues to sample non- non-dominant yet plausible modes, sustaining exploration throughout training. This stabilized exploration prevents overfitting to the training distribution and discovers reasoning patterns that generalize to the OOD benchmark AMC23, while also yielding improvements in Pass @k under larger sampling budgets.
Wall-clock Training Time
(b) Response Length During Training
GRPO
Training Time (min)
400
Reeepotte Length
T1.2
2000
Clip0.20
300-
3G0
57m
R
200
100
1000
20PO
Cliph
T1.2
Rolh1
20
50
150
120
Methods
Training Sttaps
Figure 8: Training efficiency comparison on Owen3-8B-Basee (a) Wall-clock training time for EEPO and baseline methods. (b) Mean response length during training for each method. EEPO achieves the fastest training time while maintaining stable response lengths.
Training Efficiency. We evaluate the computational efficiency of EEPO and baseline methods on Qwen3-8B-B using B200 GPUs. As shown in Figure a(),, EEPO achieves comparable train- ing time standard GRPO, demonstrating that our exploration mechanism introduces negligible computational overhead within the entire framework. Among baseline configurations, higher sam- pling temperatures significantly slow training by approximately 30%, as these methods generate substantially longer responses throughout training (Figure B)). Additional rollouts incur the high- est computational cost due increased trajectory sampling, while adjusting the clipping ratio has

---

Published as a conference paper at ICLR 2026
minimal impact efficiency. These results demonstrate that EEPO achieves superior performance through effective exploration while preserving the training efficiency of the original GRPO algorithm.
RELATED WORK
Reinforcement learning with verifiable rewards. Reinforcement learning has been widely used to improve the capabilities of language models (Bai et al., 2022, Rafailov et al.] [2033] Wang et al.) et al.) 2026b) and to align model outputs with human values under diverse scenarios (Zhang et al., 2026a Chen et al.) 2025c b), particularly through reinforcement learning from human feedback (RLHF) (Oyyagg et al. 2022). More recently, reinforcement learning with verifiable
rewards (RLVR) (Shao et al. 2024 DeepSeek-AI et al. 2025 [Team et al.) 2025) replaces subjective
preference feedback with automatically verifiable, rule-based rewards, enabling scalable optimization
in domains with reliable checkers, such as mathematics (Hendrycks et al. 2021b Chen et al. (025a)
and programming (Codeforces) 2025 [Jin et al. 2026). For example, DeepSeek-RI (DeepSeek-A) Al et al.] (2255) reports that RLVR can elicit a range of reasoning behaviors including verification and celf-correction-offee -ffenn accompanied by longer intermediate reasoning traces (Gandhi et al., 2025). However, RLVR can be unstable and may plateau early: insufficient exploration can cause premature convergence to narrow solution modes, limiting further improvement.
Exploration in RL. Exploration in RL is often promoted through policy stochasticity under the assumption that randomness broadens coverage of actions and states; however, indiscriminate randomness is insufficient, policies tend to collapse toward dearrdeterministic behavior behavior-"enntropy
collapse" (Cui et al. 2025 Yu et al. 2025) driven by exploitative objectives. Recent efforts largely fall into categories: objective-leve modifications and indiscriminate exploration. The latter increases randomness uniformly, for example via e-greedy policies (Sutton & Barto 2018), softmax temperature adjustments (Chen al. 2025d: Hou et al. (025). or entropy regularization (Hou itt al. 2025); while these methods raise stochasticity. they do not shift probability mass away from dominant behaviors and often become unstable or ineffective when applied aggressively. On the objective side, increasing the clipping threshold (e.g., DAPO (Yu et al. (025)) or concurrent work's s relaxing rewards with @ass@k (Chen et al.] (2255) admits more low- www-robability trajectories but leaves rollout dynamics unchanged, allowing the policy to repeatedly sample high-probability regions and sustain the self-reinnorcing loop (S 2.2) that drives entropy collapse. In contrast, we propose active rollout-time intervention that temporarily forgets recently sampled trajectories, explicitly discouraging revisits and stecring the model to explore alternative modes in sequence; this targeted mechanism disrupts seff-reinforcement and remains complementary to objective-level adjustments.
Machine Unlearning for LLMs Machine unlearning for LLMSs studies removing the influence of
specific data (e.g., sensitive or copyrighted content) without retraining models from scratch (Liu et al.) 2024). Typical motivations include privacy compliance and mitigating bias or harmful behaviors. Common approaches involve weight editing (Mitchell et al. 2022) gradient-based optimization (Jang et al. 2023) to forget targeted data, and inference- time strategies such as prompt manipulation. However, prior work primarily focuses on knowledge erasure, whereas EEPO repurposes and tailors unlearning for RI exploration: during rollout generation, we temporarily unlearn previously sampled trajectories to prevent the rollout model from repeatedly sampling from dominant modes.
CONCLUSION
We introduced EEPO, an exploration innloraation-ehhneeddd policy optimization framework that augments the rollout process with sample-then-fore forget mechanism. By temporarily suppressing recently sampled trajectories during rollouts, EEPO encourages exploration of alternative modes in the output distribu- tion that would otherwise remain underexplored. Our method transforms indiscriminate stochasticity into strategic exploration, breaking the self-reinforcing loop that causes insufficient exploration e
and entropy collapse. Extensive experiments across three models and five mathematical reasoning benchmarks demonstrate that EEPO consistently outperforms existing methods while maintaining comparable training efficiency. These results establish EEPO as a practical and effective approach for addressing the exploration-explootatinn trade-off in RLVR.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
We thank the anonymous reviewers for their constructive feedback and suggestions. This work was partially supported by Hong Kong RGC GRF No. 14206324. QZW and BH were supported by RGC General Research Fund No. 12200725.
REFERENCES
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer. Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Conerly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatfield-Dodde Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan. Constitutional ai: Harmlessness from ai feedback, 2022.
Liang Chen, Xueting Han, Li Shen, Jing Bai, and Kam-Fai Wong. Beyond two-stage training: Cooperative sft and rl for Ilm reasoning, 2025a. URL
06948
Liang Chen, Xueting Han, Li Shen, Jing Bai, and Kam-Fai Wong. Vulncrability alignment: Mitigating uneven forgetting in harmful fine-tuning, In Forty-second International Conference on Machine Learning, 2025b. URL
?dd EMHED4WTHT
Liang Chen, Li Shen, Yang Deng, Xiaoyan Zhao, Bin Liang, and Kam-Fai Wong. PEARL: To- wards permutation-resiiient LLMs. In The Thirteenth International Conference on Learning Representations, 2025c. URL https: http:p:s//ppeereiieeee net. http://ommmmmddxxojjie.....
Zhipeng Chen, Yingqian Min, Beichen Zhang, Jie Chen, Jinhao Jiang, Daixuan Cheng, Wayne Xin Zhao, Zheng Liu, Xu Miao, Yang Lu, Lei Fang. Zhongyuan Wang, and Ji-Rong Wen. An empirical study on eliciting and improving r1-like reasoning models, 2025d. URL https //axxiv.org/abs//55000 04548
Zhipeng Chen, Xiaobo Qin, Youbin Wu. Yue Ling, Qinghao Ye, Wayne Xin Zhao, and Guang Shi. Pass@k training for adaptively balancing exploration and exploitation of large reasoning models, 2025e. URL.hhttss///aax
www.wrgg..s...... 2508. 10751
Codeforces. Codeforces competitive programming platform, 2025.
URL https://
codefoocess com/
Accessed: 2025-03-18.
Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang, Yuxin Zuo, Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, Zhiyuan Liu, Hao Peng, Lei Bai, Wanli Ouyang, Yu Cheng, Bowen Zhou, and Ning Ding. The entropy mechanism of reinforcement learning for reasoning language models, 2025. URL https: arxiv. org/ abs/ 2505. 22617
Al, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang. Bochao Wu, Bei Feng. Chengda Lu. Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji. Erbang Li, Fangyun Lin, Fucong Dai. Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang. Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang. Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li. Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang.
11

---

Published as a conference paper at ICLR 2026
Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye. Tao Yun, Tian Pei, Tianyu Sun, T. Wang. Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wengin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang. Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang. Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang, Yukun Zha, Yuting Yan, 7. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma. Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. Deepseek-rl: Incentivizing reasoning capability in iims via reinforcement learning, 2025. 12948
URL
Kanishk Gandhi, Ayush Chakravarthy, Anikait Singh, Nathan Lile, and Noah D. Goodman. Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective stars, 2025. URL https: / org/ /ww.
Chaoqun He, Renjie Luo. Yuzhuo Bai, Shengding Hu. Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, al. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. arXiv preprint 2Xiv:2402./4008, 2024.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.038/ 2021a.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. In Thirty-fifth Conference Neural Information Processing Systems Datasets and Benchmarks Track, 2021b.
Zhenyu Hou, Xin Lv, Rui Lu, Jiajie Zhang, Yujiang Li, Zijun Yao, Juanzi Li, Jie Tang, and Yuxiao Dong. T1: Advancing language model reasoning through reinforcement learning and inference scaling, 2025. URL
11651
Joel Jang. Dongkeun Yoon, Sohee Yang, Sungmin Cha, Moontae Lee, Lajanugen Logeswaran, and Minjoon Seo. Knowledge unlearning for mitigating privacy risks in language models. In Anna Rogers, Jordan Boyd-Grabed and Naoaki Okazaki (eds.), Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 14389-14408, Toronto, Canada, July 2023, Association for Computational Linguistics. doi: 10. 1/553/1/22033 acc
acl-Iong.805. URL httttp::/ac..anntoooogg..g wrg//023/33333 acc long. 805/
Yiyang Jin, Kunzhao Xu, Hang Li, Xueting Han, Yanmin Zhou, Cheng Li, and Jing Bai. Reveal: Self-evolving code agents via reliable self-verification. In The Fourteenth International Confer- ence on Learning Representations, 2026. URL h/tp::/::///peeeeee.... wwt.eewwwwwwwwe.eeeeeee..e.eee http//ewww.e//ttortreeee htt./oorrmmiii
456211co43
Aitor Lewkowyez, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ra- masesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Systems, 35:3843-3857, 2022.
Sijia Liu. Yuanshun Yao, Jinghan Jia, Stephen Casper, Nathalie Baracaldo, Peter Hase, Yuguang Yao, Chris Yuhao Liu, Xiaojun Xu, Hang Li, Kush R. Varshney, Mohit Bansal, Sanmi Koyejo, and Yang Liu. Rethinking machine unlearning for large language models, 2024. URL https
wwtp:/wwwr///bb/////..////// 08787
12

---

Published as a conference paper at ICLR 2026
Eric Mitchell, Charles Lin, Antoine Bosselut, Christopher D. Manning, and Chelsea Finn. Memory- based model editing at scale, 2022. URL htttp:://axxxiiii org/ abs/ htt.b///2...0330
OpenAl. Learning to reason with Ilms. http:///onn..ooommmm///////annn/
Accessed: 15 March 2025.
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 27730-27744, 2022.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct Preference Optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 19-53788-53741, 2023.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707006377 2017.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300 2024.
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. arXiv preprini arXiv:2409. 19256, 2024.
Richard Sutton and Andrew Barto. Reinforcement Learning: An Introduction. MIT press, 2018.
Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi K1.5: Scaling reinforcement learning with LLMs. arXiv preprint arXiv:2501.12599 2025.
Llama Team. The llama 3 herd of models, 2024. URL
21783
Yue Wang, Qizhou Wang, Zizhuo Zhang, Ang Li, Gang Niu, Han, and Masashi Sugiyama. What is preference optimization doing, how and why? Arxiv Preprint, 2025.
An Yang. Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15111 2024.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang. Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang. Jianhong Tu, Jianwei Zhang. Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang. Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yugiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025. URL https: http//aaivvvv.. org/ abs 2505 09388
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang. Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Yuxuan Song. Xiangpeng Wei. Hao Zhou, Jingjing Liu. Wei- Ying Ma, Ya-Qin Zhang, Lin Yan, Mu Qiao, Yonghui Wu, and Mingxuan Wang. Dapo: An open-source Ilm reinforcement learning system at scale, 2025. URL. URL abs fas//200..m 14476
Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He. Zejun Ma, and Junxian He. Simplerl- zoo: Investigating and taming zero reinforcement learning for open base models in the wild, 2025. htt.hhttp///eet org htrrg//bs/25033 18892
13

---

Published as a conference paper at ICLR 2026
Zizhuo Zhang, Qizhou Wang. Shanshan Ye, Jianing Zhu, Jiangchao Yao. Bo Han, and Masashi Sugiyama. Towards understanding valuable preference data for large language model alignment. In International Conference on Learning Representations, 2026a.
Zizhuo Zhang, Jianing Zhu, Xinmu Ge, Zihua Zhao, Zhanke Zhou, Xuan Li, Xiao Feng, Jiangchao Yao, and Bo Han. Co-reward: Self-supervised reinforcement learning for large language model reasoning via contrastive agreement. In The Fourteenth International Conference on Learning Representations, 2026b.
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint wrwih:19000085900 2019.
14

---

Published as a conference paper at ICLR 2026
LLM USAGE DISCLOSURE
In our work, we mainly use GPT-5 for writing enhancements, primarily to improve grammar and text clarity,
ETHICS STATEMENT
All authors have read and adhered to the ICLR Code of Ethics. Our study relies solely publicly available datasets and models, as detailed in Appendix [A] No private or personally identifiable information was used. The work aims advance the seientific understanding of PO methods while upholding principles of transparency, fairness, and responsible research.
REPRODUCIBILITY STATEMENT
The codebase will be made publicly available upon acceptance. The base models and benchmarks used in this work are publicly accessible. All experiments were conducted using NVIDIA A100 80GB GPUs and B200 184G GPUs.
A DETAILED EXPERIMENTAL SETUP
Datasets. We use the MATH dataset (Hendrycks et al. 2021a) for RL training. Following the setup of SimpleRL (Zeng et al. 2025), we train on the hard data, which contains 8.5K problems with difficulty levels ranging from to 5. For evaluation, we adopt five challenging mathematical reasoning benchmarks: Minerva Math (Lewkowyez et al.) 2022), OlympiadBench (He al. 2024), and three recent competition-leve datasets- AMC 2023, AIME 2024, and AIME 2025. For smaller models (Qwen2.5-33) B-3B and LLaMA-3.2-311 -.-ntutt,,,, evaluation is conducted on the first four benchmarks. For the stronger Qwen3-8B -B--eBse model, additionally include AIME 2025.
Models. To demonstrate the generality of our approach, experiment with three L.MMs from different model families and scales.
Qwen2.5-3B Yangg et al. 2024): base model from the Qwen2.5 series, with stronger pretraining and support for long-context inputs.
Mlama-3223BB-Insst (Team) 2024): an instructionnfollooinng model based on Meta's Llama architecture, included to evaluate cross-family generalization.
Qwen3-8B-Bace (Yang al. 2025) larger base model from the Qwen3 family, used assess performance at larger scale.
Reward Function. We employ binary reward based on answer correctness: +1 for correct final answer and 0 otherwise. We exclude format-base rewards, which can constrain exploration and degrade performance (Zeng et al. 2025), particularly when training base models.
Implementation Details. All models trained using the VERL framework (Sheng et al. 2024), employing the GRPO algorithm. We use a batch size 128, a mini-batch size of 64, a learning rate of 5 10-7 7 and 8 rollouts, training for 2 epochs. The KI loss and entropy loss coefficient 1 x 10 and x 10-5, respectively. The maximum response length varies by model: up to 4K tokens for Qwen2.5-3B. and up 6K tokens for both LLaMA-3, 2-3B-Instruct and Qwen3-8B-Bass During evaluation, we use greedy decoding to compute pass @ accuracy. All experiments are conducted on compute clusters equipped with NVIDIA A100 GPUs (80GB) and B200 GPUs.
B BASELINES DESCRIPTION
We compare EEPO against GRPO and several variants that are explicitly designed to enhance exploration.
15

---

Published as a conference paper at ICLR 2026
Base/Instruct Model.
The base model, or its instruction-tun tuned variant without any additional ceasoning-specffer training, serves as a performance lower bound.
GRPO. Standard GRPO applied to the base or instruction model using default training settings.
Increased Entropy Regularization. This variant enhances exploration by increasing the entropy weight in the training objective, encouraging the policy to generate more diverse outputs. It represents common approach where stronger entropy regularization is used promote exploration.
Higher Sampling Temperature. This variant applies higher sampling temperature during the actor's decoding process to promote exploration and reduce output determinism. Temperature-baat softmax exploration (also known as Boltzmann exploration) is widely used method implement the e-greedy algorithm in stochastic policies. As the temperature & + O, the policy becomes nearly greedy; as t + oc, the action distribution approaches uniform, effectively increasing exploration.
Clip Higher. This variant incorporates the "clip higher" heuristic from DAPO, which encourages the selection of rare or low-probability tokens during training. It is one of the most widely used euplorationeenhaniing baselines in modern RLVR pipelines.
Increased Number of Rollouts. This baseline increases the number of rollouts per training step to expand the trajectory space and encourage broader exploration. It is designed to evaluate whether EEPO with rollouts can match or outperform GRPO with larger number of rollouts (default: 16).
C EXPERIMENTS ON LARGE-SCALE MODELS
To assess how EEPO scales with model size, we extend our experiments from 3B and 8B models to larger 14B model, Qwen3-144 14B- 14B--44BBt B-Basse The results are summarized in Table
Table 4: Results on Qwen3-14B 14B-Base across five reasoning benchmarks.
Benchmark
Method
Avg.
Minerva Math
OlympiadBench
AMC23
AIME24
AIME25
GRPO
36.8
48.6
67.5
23.3
26.7
40.6
EEPO
39.3
50.1
67.5
36.7
30.0
44.7
As shown in Table 4 EEPO continues to provide consistent improvements over GRPO on Qwen3- 14B-Base, particularly on the more challenging benchmarks (e.g., AIME24 and AIME25). This suggests that EEPO scales well with model size and remains effective in the 3B-14B range.
D ADDITIONAL COMPARISON WITH GRPO VARIANTS
To make the gain of EEPO more directly and comparable, we also provide the following fair comparisons, where EEPO is implemented on GRPO and its variants.
Method
Avg. acc.
GRPO
21.0
EEPO
26.1
GRPO Increased Entropy
23.9
EEPO Increased Entropy
27.9
GRPO + Clip High
22.9
EEPO Clip High
26.6
Table 5: Average accuracy of GRPO variants and their EEPO-enhanced counterparts. EEPO provides gains of 3.7-5.1 absolute accuracy points over already strong exploration-enhance baselines.
16

---

Published as a conference paper at ICLR 2026
Table 5 reports the average accuracy of GRPO and its variants, together with their EEPO-enhanced counterparts. EEPO consistently yields an absolute improvement of about 3.7-5.1 points over the corresponding exploration-enhameee GRPO methods.
17


Reviews:
[{"review_id": "AeKPjZk2kV", "reviewer": "Reviewer_SHnD", "summary": "The authors proposes Exploration-Enhanced Policy Optimization (EEPO), a novel framework for reinforcement learning with verifiable rewards (RLVR) that addresses policy entropy collapse.", "questions": "Please address the concerns in weakness section.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The \"sample-then-forget\" idea is a clever and novel approach to the exploration-exploitation problem. Instead of just adding indiscriminate noise (like increasing temperature), it actively and strategically steers the policy away from dominant modes it is starting to overfit on.\n\n2. The method demonstrates consistent and significant performance gains over strong GRPO baselines across five challenging mathematical reasoning benchmarks and three different language models. The average relative improvements are substantial (e.g., +33.0% on Llama3.2-3B-Instruct).", "weaknesses": "1. The method introduces at least two key new hyperparameters: the entropy threshold $\\alpha$ for activating unlearning and the unlearning rate $\\eta$. The paper uses fixed values ($\\alpha=0.3$, $\\eta=3\\times10^{-3}$) without providing an ablation study or discussion on how these values were chosen or how sensitive the model's performance is to them. This could be a point of fragility.\n\n2. The unlearning step modifies the rollout policy ($\\pi_{\\theta^{\\prime}}$) in the middle of generating a single batch of trajectories. It is unclear how the importance sampling (IS) ratio for the standard GRPO objective (Eq. 2) remains valid when trajectories in the same batch ($O$) are drawn from two different policies (pre-unlearning and post-unlearning). This potential violation of the IS assumption needs a more rigorous justification.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9Ze1iuONVQ", "reviewer": "Reviewer_zNJJ", "summary": "The authors introduce a novel method to maintain exploration while learning in LLMs with RLVR. Current approaches overemphasize exploitation, causing premature entropy policy collapse. The current method avoids this by adaptive unlearning of the dominant modes and focusing on secondary modes. Improving performance compared to baselines is shown.", "questions": "See weaknesses.", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The problem solution is simple and elegant. This is complemented by simple and useful intuitions that help us to understand the proposed algorithm.", "weaknesses": "Nice intuitions are presented, like in Fig. 2, but no theory is provided to back up those intuitions. This means that improvement can strongly depend on parameters and on a case-by-case basis. \n\nAblations studies are missing, so it is unclear what aspect of the introduced algorithm is critical: is it enough to have an entropy-conditioned activation of unlearning just based on Eq. 8, assuming a constant $p_{clip}$?\n\nIs it possible that in some problems no secondary mode exists? Then how would the method fare in discovering the most promising actions in the vast space of tokens and reasoning trajectories?\n\nWhile performance increases compared to other methods, the performance is very close to GRPO with updated parameters, so it seems that the gains are relatively small. \n\nAs far as I can see, no comparison with other methods of RL exploration is introduced (epsilon-greedy with optimal epsilon, entropy regularization, and such). Does the method outperform the obvious modifications of the algorithms with those additional exploration tricks?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PL6y6CZkAv", "reviewer": "Reviewer_WrTx", "summary": "This work introduces EEPO, a framework that addresses the exploration-exploitation dilemma in RLVR. EEPO enhances the rollout process through a sample-then-forget mechanism, which temporarily suppresses recently sampled trajectories during generation. This deliberate exclusion discourages the policy from repeatedly collapsing into dominant, high-probability modes and instead promotes the discovery of alternative, underexplored reasoning pathways. By converting passive stochasticity into purposeful, diversity-driven exploration, EEPO steers the policy toward more comprehensive coverage of the output distribution.\n\nExtensive experiments across three model families and five mathematical reasoning benchmarks show that EEPO outperforms existing methods in final performance while preserving comparable training efficiency.", "questions": "I do not know this specialized research field very well. I will adjust my score and optimize my review document based on the evaluations of other expert reviewers and my performance during the rebuttal period.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "(1) Well-written (2) Detailed experiment (3) The problems related to training efficiency that have been solved are distinctive and seem valuable to the industrial sector", "weaknesses": "N/A", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8z61x7Zwwm", "reviewer": "Reviewer_X4NZ", "summary": "This paper addresses the challenge of insufficient exploration and resulting entropy collapse in Reinforcement Learning with Verifiable Rewards (RLVR) used for training large language model (LLM) reasoning capabilities. Current methods often over-exploit dominant trajectories, and while adding randomness increases stochasticity, it fails to effectively shift the policy away from these modes. To counter this, the authors propose Exploration-Enhanced Policy Optimization (EEPO), a framework that modifies the rollout phase with a novel \"sample-then-forget\" mechanism. EEPO performs rollouts in two stages: after sampling the first half of trajectories, it applies a temporary, lightweight unlearning step to the rollout model to suppress the just-sampled responses, using a complementary loss function gated by an entropy threshold. This forces the second stage of sampling to explore different, less probable regions of the output space, actively steering away from dominant modes. The standard GRPO objective is then used to update the main policy model using the combined trajectories from both stages, decoupling the exploration enhancement from the policy optimization itself. Experiments across five reasoning benchmarks and three different LLMs show that EEPO consistently outperforms GRPO and other exploration-focused baselines, achieving significant average accuracy gains.", "questions": "Please see paper weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper investigates the critical problem of insufficient exploration and entropy collapse in Reinforcement Learning with Verifiable Rewards (RLVR), a central challenge that limits the performance gains and generalization of LLMs trained for reasoning tasks.\n2. The proposed Exploration-Enhanced Policy Optimization (EEPO) method is conceptually intuitive, employing a straightforward \"sample-then-forget\" mechanism within a two-stage rollout process to actively suppress dominant modes and encourage exploration.\n3. The effectiveness and robustness of EEPO are demonstrated through comprehensive experiments conducted across five different mathematical reasoning benchmarks and three distinct LLM architectures and sizes.", "weaknesses": "1. The comparison primarily focuses on standard GRPO and variations based on randomness or rollout count, neglecting direct benchmarks against more advanced, contemporary RLVR algorithms like the full DAPO framework.\n2. Experimental validation is confined to LLMs with fewer than 10 billion parameters (specifically 3B and 8B models), leaving the scalability and effectiveness of EEPO on larger foundation models untested.\n3. While EEPO aims to improve performance via enhanced exploration, the provided training curves show its mean reward is comparable to or even slightly lower than standard GRPO (Fig. 6), and the paper lacks a detailed explanation for why the proposed method results in slightly faster wall-clock training times despite incorporating an additional unlearning step.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)