CDE: Curiosity-Driven Exploration for Efficient Reinforcement Learning in Large Language Models
Authors: Runpeng Dai
Linfeng Song
Haolin Liu
Zhenwen Liang
Dian Yu
Haitao Mi
Zhaopeng Tu
Rui Liu
Tong Zheng
Hongtu Zhu
Dong Yu
Venue: ICLR 2026 Poster (2026)
Keywords: Large language models
Reasoning
Exploration
Primary area: foundation or frontier models, including LLMs

Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) is a powerful paradigm for enhancing the reasoning ability of Large Language Models (LLMs). Yet current RLVR methods often explore poorly, leading to premature convergence and entropy collapse. Moreover, they tend to produce poorly calibrated policies that remain confident in their generations regardless of correctness. To address this challenge, we introduce **Curiosity-Driven Exploration (CDE)**, a framework that leverages the model's intrinsic sense of curiosity to guide exploration. We formalize curiosity with signals from both the actor and the critic: for the actor, we use perplexity over its generated response, and for the critic, we use the variance of value estimates from a multi-head critic architecture. Both signals serve as an exploration bonus within the RLVR framework to guide the model. Our theoretical analysis shows that the actor-wise bonus inherently penalizes overconfident errors and promotes diversity among correct responses; moreover, we connect the critic-wise bonus to the well-established count-based exploration bonus in RL. Empirically, our method achieves an approximate **+3** point improvement over standard RLVR using GRPO/PPO on AIME benchmarks.

Published as a conference paper at ICLR 2026
CDE: CURIOSITY-DRIVEN EXPLORATION FOR EFFI- CIENT REINFORCEMENT LEARNING IN LARGE LAN- GUAGE MODELS
Runpeng Dait.s, Linfeng Song', Haolin Liu 1.4, Zhenwen Liang', Dian Yu', Haïtao Mi', Zhaopeng Tu², Rui Liu Tong Zheng!.5, Hongtu Zhu Dong Yu²
'eeneent AI Lab, Tennent Multimodal Department, Universitty of North Carolina at Chapel Hill,
Universitt Virginia, University of Maryland, College Park runpengeunc. edu, Ifsongéglobal tencent.com
ABSTRACT
Reinforcement Learning with Verifiable Rewards (RLVR) is powerful paradigm for enhancing the reasoning ability of Large Language Models (LLMs). Yet current RLVR methods often explore poorly, leading to premature convergence and entropy collapse. Moreover, they tend to produce poorly calibrated policies that remain confident in their generations regardless of correctness. To address this challenge, we introduce Curiosity Driven Exploration (CDE), framework that leverages the model's intrinsic sense of curiosity guide exploration. We formalize curiosity with signals from both the actor and the critic: for the actor, we use perplexity over generated response, and for the critic, use the variance of value estimates from a multi-head critic architecture. Both signals serve as an exploration bonus within the RLVR framework guide the model. Our theoretical analysis shows that the actor-wise bonus inherently penalizes overconfident errors and promotes diversity among correct responses; moreover, we connect the critic-wise bonus to the well-established count-based exploration bonus in RL. Empirically, our method achieves an approximate +3 point improvement over standard RLVR using GRPO/PPO on AIME benchmarks.
-Driven
INTRODUCTION
Reinforcement learning has significantly advanced the reasoning capabilities of Large Language Models (LLMs). A leading paradigm is Reinforcement Learning with Verifiable Rewards (RLVR), which directly optimizes models based on the correctness of their final answers. Despite the devel- opment of advanced RLVR algorithms like GRPO (Guo et al., 2024) and DAPO (Yu et al., 2025b), critical challenges persist. Problems such premature convergence and entropy collapse (Cui et al., 2025) are frequently observed, posing fundamental challenges to the efficiency of RI.R..
These challenges stem from the classic exploration-exploitatoo dilemma in reinforcement learning (Sutton Barto, 2018). Phenomena like entropy collapse reveal a critical flaw the training process: it is heavily biased towards exploitation, causing models converge prematurely instead of sufficiently exploring their environment for better solutions. Although the RL literature encompasses wide range of exploration strategies, these methods exhibit limitations when applied to LLMs. Simple heuristics, including e-greedy policies (Sutton & Barto, 2018) and entropy bonuses (Haarnoja encourage the policy to be more
al., 2018), either inject randomness the environment stochastic. Directly applying those approaches often demonstrates debatable effectiveness in complex environments like Deep RL (Andrychowicz et al., 2021) and LLM-based reasoning (Cui al., 2025; Shen, 2025).
'Our bonus is technically the logarithm of the perplexity (log-PPI.). For simplicity, we refer to it as the PPI. bonus throughout the paper.

---

Published as a conference paper at ICLR 2026
More principled strategies include count-based and prediction-based approaches. The former, in- cluding UCB (Lai, 1987) and LinUCB (Li et al., 2010), incentivizes visits to rarely explored states, while the latter, like ICM (Pathak et al., 2017) and RND (Burda al., 2018b), reward agent for reaching hard-to-predict states. Recent work has focused on adapting these methods for tuning L.Mss (Bai et al., 2025; Sun al., 2025; Gao al., 2025; Yu al., 2025a). However, these strategies often require training auxiliary modules and effective state representations (Burda et al., 2018a). This requirement is particularly challenging for LLMs, where efficiently represent reasoning path into fixed-size embedding remains open problem (Fu et al., 2024), and simplistic approaches such as using the last hidden state are often problematic (Barbero et al., 2024).
We first explored count- count-based methods using the SimHash technique (Tang et al., 2017), which maps the embedding of a CoT response into discrete hash code, and then uses the visitation frequency of hash cells as pseudo-counts (see Appendix D). However, this approach proved problematic, exposing the inherent challenge of representing complex reasoning trajectories. As illustrated in Fig- ure 1, most responses collapsed into small number of hash grids.
50
Number of vistaaions
This clustering creates a highly concentrated count distribution, un- dermining the effectiveness of count-based exploration. Therefore, developing efficient and scalable exploration methods for LLMS remains a critical challenge.
5200 1V40 Joo And Jkkn Hash cell index (binned)
2004
Figure 1: Distribution of num- ber of visitations across hash- cells.
In this work, we propose an intuitive approach that leverages the model's intrinsic sense of curiosity as guide for exploration. An LLM, having been trained on vast reasoning corpora, develops sophisticated internal model of what constitutes a familiar versus novel reasoning pattern. This parallels early childhood development (Chu & Schulz, 2020), where learning is not driven by an external summary and count of experiences, but is instead propelled by an intrinsic curiosity to explore novel situations. We formalize this principle in our Curiosity-Driven Exploration (CDE) framework, which considers curiosity signals from both the actor and the critic. For the actor, perplexity (PPL) over its generated response serves the curiosity measure. For the critic, measure curiosity via the variance of its posterior value distribution. We then approximate this posterior by extending the PPO framework with multi-head, bootstrapped structure (Figure The curiosity signals serve an exploration bonus, shaping the reward and advantage functions to effectively guide exploration.
Multi-head Critic
Advantage calculation
Critic Update
Head
A.
Σ (Tay By +w min
Idull
Rollouts
Subsample
D1
LLM backbone
Head
Rene (oceunn and (1V (a Puerss) [))j*])
Exploration Bonus
du- 3 Σ P/l.oca-1)
Head
Gora
Σ
(Pile.))-))
Ensemble Estimation
Figure 2: Illustration of the multi-head critic framework.
We provide theoretical foundation for our method, establishing two key properties. First, we prove that the actor's perplexity bonus intrinsically penalizes overconfident errors while encouraging diversity among correct responses (Theorem 3.1.. Second, we connect the critic's bonus to classical RL. theory by showing is equivalent to count-based bonuses in the linear MDP setting (Theorem 3.2), grounding approach in established exploration principles.
Empirically, our method delivers consistent performance gains across four widely used mathematics benchmarks (AIME25, AIME24, AMC23, and MATH), achieving notable +3 point improvement on the challenging AIME benchmarks. Beyond these performance gains, our analysis of the training dynamics uncovers critical failure mode we term calibration collapse: under a standard GRPO policy, a model's 's confidence progressively decouples from its correctness, while the proposed PPL bonus mitigates this miscalibration.
2

---

Published as a conference paper at ICLR 2026
2 PRELIMINARIES: RLVR, GRPO AND PPO
We frame the language generation process as sequential decision decision-making problem, trained using the RLVR paradigm (Lambert et al., 2024; Yu et al., 2025b). In RLVR, rule-based verifier assesses the correctness of generated response by comparing against the ground truth, providing a verifiable reward signal.
Group Relative Policy Optimization (GRPO (Shao et al., 2024)) is critic-free RL algorithm. Given prompt the policy xy generates group of G candidate responses {or, o2....,0ci. Each response 0; receives a reward rę = r(ox, q). The advantage of the i-th response calculated by normalizing its reward relative to the others in the group: A, (ra mean(ri,-. :ra))/ std(ri,....) r... Let Tha be the behavior policy used to generate the responses. The policy is updated by maximizing the GRPO objective:
G
(o) = D, Ta G
_
Σ min clip
where Fi.e (os, q, Or,e)/ mo. (0r,t q, (i,t)) is the token-level probability ratio.
Proximal Policy Optimization (PPO(Schulman et al., 2017)) is an actor-critic algorithm that uses learned value function, or critic Vg. to estimate advantages. For single response o with a final reward r, the advantage at token is:
A Σ (7 where 8, + 7Val Vo 0gi).
Here, the reward is applied only at the final token (7o) and r for / lo]). The hyperparam- eters and are the discount factor and GAE trace-decay parameter, respectively. The PPO policy objective is:
(Fro(0) = EggDD Don min - c))))
3 CDE: CURIOSITY-DRIVES EXPLORATION
This section first analyzes count- exploration, highlighting a key challenge in its application. We then introduce Curiosity- Srrivs Exploration (CDE), systematic framework that leverages curiosity signals from both the actor and the critic. We elaborate on the specific formulations for actor and critic curiosity in Sections 3.2 and 3.3, respectively.
3.1 CHALLENGE OF COUNT-BASEE EXPLORATION FOR RLVR
Count-based exploration methods use response embeddings to quantify novelty, assigning an explo- ration bonus to less frequent responses patterns. To implement this efficiently, investigated a hase-based counting method (Tang et al., 2017), which projects CoT embeddings into discrete hash grids and uses grid visitation frequency as pseudo-counts (details Appendix D).
While conceptually appealing, this approach fails in practice. We hypothesize the root cause is the poor expressiveness of embeddings. Effectively representing reasoning trajectory in single embedding is challenging, and simple heuristics such as the last token's 's hidden state, are often problematic (Barbero al., 2024). As illustrated in Figure 1, where most responses collapse into the same or neighboring hash grids, leading to highly concentrated distribution of counts and thus undermining the effectiveness of count-based exploration for RLVR..
In this work, we introduce exploration paradigm guided by the model's intrinsic curiosity, moving beyond methods that rely on explicit counts or response embeddings. Our approach inspired by early cognitive development (Chu & Schulz, 2020), where, much like children, the model is encouraged to explore what it finds novel while acting confidently on familiar patterns. Learning is therefore not driven by external count of experiences but is propelled by an intrinsic drive to explore. We formalize this principle with separate curiosity signals for the actor and the critic, beginning with the actor's implementation.
3

---

Published as a conference paper at ICLR 2026
3.2 EXPLORATION GUIDED BY ACTOR CURIOSITY
We model actor curiosity as the actor's uncertainty about its own actions. Intuitively, a response that surprising the actor-i.e.. has low probability under its current policy-likely resides in an underexplored region of its learned distribution.
natural and computationally efficient measure of this surprise is the perplexity of the actor's generation. We formalize this as response-level curiosity bonus, defined as the negative average log-probability of generated response ...,...,00 or given prompt q:
1 Σ log (orloees
Baetoe (g, 0)
where π denotes the actor policy, A higher value for Bacwer (q, 0) indicates greater surprise and thus stronger intrinsic reward signal for exploration.
However, simply adding this bonus to the original reward can be unstable and sub-optimal. Un- constrained exploration might incentivize the model to generate high- but low-quality or inacourate responses (a behavior known as reward hacking), or lead to over-exploration where the policy fails to converge to a stable, high-quality output. To address this, we integrate the bonus using an adaptive clipping mechanism. The total response-lever reward, f, is a combination the original reward signal o):
F(g. = 0) min( [((. o)
B (q, o))
(2)
This formulation promotes exploration by rewarding responses that the actor finds surprising, while constraining the bonus to remain a fraction of the original reward. The behavior of this reward function is controlled by three key hyperparameters:
The bonus weight is dynamic coefficient, typically set with annealing schedule to decrease over the course of training. This allows for aggressive exploration in the early stages and then gradually shifts focus towards exploitation of high-reward regions the policy converges.
The clipping ratio governs the maximum size of the curiosity bonus relative to the original reward. By capping the bonus at Ir(q, o)1/ K, it ensures the bonus remains a supplement and prevents it from dominating the learning signal.
The bonus scaling factor normalizes the curiosity bonus Bacor(q, o) before it is compared to the clipped reward. A higher a allows the curiosity bonus to reach the clipping threshold more easily, whereas smaller diminishes its potential impact.
Beyond exploration, we study the calibration effect of the PPL bonus. To build this intuition, we analyze responses along axes: correctness and PPI. Among these four categories, two require particular attention:
Calibration Effect
High PPL
Incorrect responses with low PPL indicate that the model is highly confident in its answer, yet the response is wrong. This reflects overfitting and should be penalized.
Correct
Incerrect
2. Correct responses with high PPL suggest that the model is less familiar with such answers, but they nevertheless turn to be successful. This reflects effective exploration and should be encouraged.
Penalize Confident Mistakes
Encourage Diverse Correctness
Figure 3: Responses by correctness and PPL.
As illustrated in Figure 3. the PPL bonus intrinsically penalizes confident mistakes while encouraging novel correct responses. For correct responses, those novel responses (with higher PPL) receive a larger positive reward. For incorrect responses, those confident responses (with lower PPI.) receive larger penalty as it receives smaller PPL. bonus. The following theorem formalizes this intuition; its precise statement and proof are deferred to Appendix F.
4

---

Published as a conference paper at ICLR 2026
Theorem 3.1. Let Th denote the policy at training step h. With PPL bonus in equation 1, the update
to M441 calibrates the policy's 's confidence as follows:
(i) Among correct responses, those with higher PPL receive larger relative probability increase.
(ii) Among incorrect responses, those with lower PPL receive a larger relative probability decrease.
The PPL bonus can be viewed ditional entropy based signals by incorporating sample-specific information. The entropy bonus is inherently sample-agnostie; as defined in Equation 3, the entropy Mr is computed over the entire next-token probability distribution, me (v q, Oet), making it independent of the token that actually sampled. As illustrated Figure 4, this means that even when the model makes high- confidence error (e.g., by sampling Token 1), the entropy bonus fails to penalize that choice.
enhancement to the tra-
Vocabulary
Figure 4: The next-token distribu- tion of the policy ne.
H4 = - Σ no (v q, g,0)) log (e 0.0)).
(3)
VEV
3.3 EXPLORATION GUIDED BY CRITIC CURIOSITY
In contrast to critic-free methods such as REINFORCE and GRPO, the critic (value function) in actor-critic frameworks provides a higher-level understanding of the prompt-response pair by esti- mating the expected reward-to-go Since this estimate is learned directly from collected trajectories, its posterior distribution conditioned on the observed data naturally reflects the degree of coverage: regions with dense data yield concentrated (low-variance) posteriors, whereas sparsely sampled re- gions result in higher uncertainty. Posterior distributions are well-established means of quantifying predictive uncertainty in deep learning models (Gal & Ghahramani, 2016; Lakshminarayanan et al., 2017; Yu et al., 2025a). As shown in Figure 5, the orange curve exhibits lower variance- variance-evidenee evidence of better data coverage whereas the other curve is more dispersed, reflecting greater uncertainty.
To approximate the posterior distribution of value estimates, we adopt the classical bootstrap method (Davison & Hinkley, 1997). widely used in statistics and increasingly recognized in the RL com- munity as an effective tool for exploration (Osband et al., 2016;
Fooidan
Pootiin
272
Irquarcy (Derskk
Ciosek et al., 2019; Bai et al., 2021). We implement this idea through multi-head critic (left subfigure in Figure 2), where K
300
critics/ { (V,..., VK} share a common LLM backbone. Each head is trained on resampled subset of the collected trajectories (right sub- figure in Figure 2), thereby producing an empirical approximation to the posterior distribution.
We use the standard deviation across the heads as a principled curiosity signal guide the policy toward under-explored regions where the heads have high disagreement. In the following theorem, we establish a surprising yet intuitive result: under Linear MDP assumption, this standard deviation is a consistent estimator of the pseudo-count bonus.
An illustration of two posterior distributions of the critic and their bootstrap approximations.
Figure 5:
Theorem 3.2. In linear MDPs, the standard deviation across multi-head critics is consistent estimator of the pseudo-count exploration bonus, v/ TT,
as used in LSVI-UCB (Jin et al., 2020) and CFPO (Cassel & Rosenberg, 2024), where Φn,t H8att an,)) denotes the feature of state-action pair and An,t - PI + XI is the coverage matrix.
Appendix provides the rigorous formulation of the linear MDP assumptions and the proof for Theorem 3.2, with further empirical support in Section 4.4. Building on this foundation, our multi- head PPO algorithm adapts the standard PPO training stages: (i) generating trajectories with the actor, (ii) updating the actor, and (iii) updating the critic. The key distinction multi-head variance as an exploration bonus. The full procedure is detailed below and illustrated in Figure 2.
incorporation of
5

---

Published as a conference paper at ICLR 2026
Actor roll-out: For clarity, we describe the process for single prompt q. The actor generates set n responses, (01...... On), each of which receives a corresponding verifiable reward ry.
Actor update: this step, the advantage calculated as:
10:1 Ars (Ga))- Be, tw min [Arial
Benii (g. 01.2t-11
Aii
The advantage consists of two components. The first term, A is modified PPO advantage estimate that leverages an ensemble of value heads instead single value function:
K E 01
K Σ Vilg,
8., - K
1 K
The second term introduces the multi-head critic bonus (Bunti), governed by the bonus weight w. clipping ratio K, and scaling factor a (see discussion following Equation 2 for interpretation). Specifically, Berite defined the standard deviation across the K value heads, encouraging exploration by assigning higher bonus to actions leading to uncertain/less visited regions:
Besui (g, 0,, = std (( 0, E و K)).
Critic update: The critic heads are updated using the collected roll-outs, which form dataset D { ( E [n],t t [[o,]]}. Each critic head is trained on a subset of the data, Dj C D. sampled with replacement. The hyperparameter Ç E (0, controls the size of this subset [Dy1 = C/D), creating trade-off: smaller Ç increases head diversity, while a larger Ç improves sample efficiency. The multi-head critic then trained by minimizing the following bootstrap loss:
K
L
Σ
(B)(1.0)-))
CKD
(g.o.))DD
4 EXPERIMENTS
4.1 DATASET AND MODEL
We evaluate our proposed method, CDE, against standard PPO and GRPO basclines on four chal- lenging mathematical reasoning benchmarks: MATH (Hendrycks et al., 2021), AMC23 (MAA, b), AIME24, and AIME25 (MAA, a). All experiments are implemented in the Verl framework, using the Qwen3-4B -B-Base model (Yang et al., 2025) and Llama-3.2 3.am--3.2-3B-... 3.2-3B-Imm Instructt (Grattafiori et al., 2024) trained on the DAPO-170 dataset (Yu et al., 2025b). Full implementation details and the prompt are provided in Appendix B.
4.2 MAIN RESULTS
We benchmark our method against the original GRPO/PPO as well as two additional baselines: the entropy bonus for traditional entropy-based exploration (Schulman et al., 2017) and i-MENTOR for curiosity-driven exploration (Gao et al., 2025). The main results are presented in Table 1 while the training dynamics presented in Figure 6. Here K Heads represents multi-head critic PPO with K heads. For all evaluations, we set the sampling temperature to 1.0. On the MATH dataset, we report accuracy from a single response (Avg@) 1). Following the protocol of (Wang et al., 2025), for the other three datasets we generate 16 independent samples and report both Pass @ 16 and the average accuracy (Avg@ 16). The key observations are as follows:
The PPL bonus enhances the mathematical reasoning ability of the GRPO method, yielding an improvement of approximately +2 points across datasets and models. In particular, method achieves notable gains on Pass@ 16, surpassing the baseline GRPO over Qwen3-4B 1aan-----BBat model by about +8 points the AIME24 dataset.

---

Published as a conference paper at ICLR 2026
Table 1: Zero-shot accuracy on the validation datasets. Avg column represents the overall accuracy across datasets, calculated by averaging the Avg@1 score from MATH with the Avg@ 16 scores (mean and standard deviation) from the remaining datasets. We use to exclude the results for Llama-3.2-33 3-3n-...-- on AIME25, since its performance is approximately I point.
MATH Avg@1
AMC23
AIME24
AIME25
Avg
Model
Avg@ 16
Pass@ 16
1vg@16 16
Puss 16
Avg@ 16
Pass
GRPO based methods
Qwen3-4B-BB Base-GRPP
87.3
63.6
f 0.9
89.1
20.8 t
41.9
21.0
0.8
39.2
48.2
Entropy bonus
86.8
64.3
0.6
89.7
21.8 1.0
39.4
21.2
41.1
48.5
G. 1-MENTOR-GRE
87.6
63.2
: 0.7
89.1
39.3
23.0 = 0.4
40.4
49.1
L, w/ PPL bonus
87.7
67.8 + 1.3
89.5
23.3 0.9
48.5
23.5
1.0
42.5
50.6
53.8
30,0 # 0.3
59.4
+
30.7
32.3
4. w/ PPL bonus
55.8
32.8 + 0.5
65.0
14,6 1: 0.3
35.1
34.4
PPO based methods
Bren3--B-Base-PPP
86.6
64.1 + 0.6
87.2
17.8 :: 0.5
36.0
17.5 = 0.8
33.7
46.5
L. w/ Entropy bonus
83.9
66.3 + 1.2
87.5
17.9
30.6
19.5
0.4
46.9
1-MENTOR-PPO
85.8
60.9 0.9
85.9
1
39.4
19.6
0.7
36.5
46.3
L w/ PPL bonus
87.9
66.1 = 1.0
88.5
0.5
37.6
18.3
1.1
33.5
47.7
L. w/2 2 Heads
83.2
63.6 0,9
89.9
19.6 0.7
34.8
19.6
= 1.0
36.1
46.6
Heads
35.5
48.5
w/ 8 Heads
85.1
66.7 + 0.8
86.9
21.7
46.4
19.0
0.8
37.1
48.1
L. w/ 16 Heads
88.3
65.0 : 0.9
88.7
20.5 1: 0.8
41.9
20.0
1.3
38.8
48.6
Llama-3. Llama-3.2-3B- 1.2-3B-Instruct-PPO
51.9
+ 0.2
59.1
12.5
29.2
30.5
L, w/ 4 Heads
56.1
32.2 + 0.4
63.7
14.3 + 0.1
35.5
34.2
0.25
0.25
0.20
0.20
Agg22 Accuney
0.35
0.25
0.00
9.00
100
150
200
250
Taanig Steps
Ta
Figure 6: Comparison of Avg@ accuracy on AIME25 over training of vanilla GRPO and PPO (Baseline methods) and GRPO with PPL bonus and 16 head multi-head PPO (Our methods).
Across benchmarks, multi-head PPO consistently outperforms vanilla PPO. Using K 4 and K 16 heads yields average gains roughly +2 points, and we observe an increase of around 10 points Pass@ 16 on AIME datasets with Qwen3-4B-Baa model in many cases.
The performance of multi-head PPO generally increases with the number of heads K: with K delivers negligible gains over the baseline, and performance begins to plateau once K > 4, which suggests that modest number of heads can capture most of the curiosity signals needed.
The consistent outperformance of multi-head PPO (K > 4) over PPO with a PPL bonus suggests that the critic's disagreement reflects higher-level uncertainty about the long-term value of reasoning path, which provides a more strategic exploration signal.
Figure illustrates CDE's effective exploration strategy: while initially lagging the PPO/GRPO baselines in test accuracy, it achieves superior final performance. This initial trade-off prevents the model from prematurely exploiting spurious high-reward paths. As exploration expands state state-action coverage, the policy refines, enabling accuracy ceiling.
effective shift to exploitation that yields higher
We further provide an analysis of the multi-hean critic's computational cost, including memory usage and runtime, in Appendix C.
7

---

Published as a conference paper at ICLR 2026
4.3 UNDERSTANDING THE EFFECT OF THE PPL BONUS
Bonus weight decay is crucial. We compare four schedules for the bonus weight w-No decay, Linear, Cosine, and Stair- case-as illustrated in Figure 7, with the performance of models trained under each schedule summarized in Table Briefly, the No decay schedule maintains strong exploration through- out training, while the Staircase schedule reduces w abruptly, enabling strong exploration in the early phase and then remov- ing the bonus for final convergence. The Linear and Cosine schedules provide intermediate behaviors.
03
dROw No doces
250
2001
Figure 7: An illustration of differ- ent weight anneal schedules.
The results in Table underscore two insights: First, decay the bonus weight necessary, as all decay schedules outperform the no-decay baseline. Second, strong exploration in the early phase is crucial, with the staircase scheme proving most effective by sustaining high exploration initially to broaden statc-actionn coverage and then removing the bonus abruptly to allow stable convergence. We also track entropy dynamics, providing extra support for the importance weight decay (see Figure 11, Appendix E).
Zero-shot accuracy of GRPO models under different PPL bonus weight decay schedules. The schedules follow those illustrated in Figure
Table 2:
MATH Avg@1
AMC23
AIME24
AIME25
Avg
Model
Avg@ 16
Pass 16
Avg@ 16
Pass@16 16
Avg@ 16
Pass 16
Bonus Weight Decay Schedules
O Base-GRPO
87.3
63.6
91.1
21.0
41.9
20.8
39.2
48.2
L. No decay
85.1
64.5
84.6
20.8
39.0
22.3
36.2
48.2
L Linear decay
85.4 45.4
66.1
91.9
23.3
40.4
20.0
40.4
48.7
L Cosine decay
86.7
68.1
90.0
22.5
44.9
21.5
40.7
49.7
L, w Staircase decay
87.7
67.8
89.2
23.5
48.5
23.3
40.3
50.6
A Analysis of Calibration. As shown in Figure 8(a), models trained with standard GRPO suffer from phenomenon term calibration collapse: early in standard GRPO training, correct responses have lower PPL (higher confidence) than incorrect ones, but as training progresses this gap shrinks. By contrast, with a PPL bonus (subfigure (b)), this separation is sustained throughout training. This pattern can be partially explained by Theorem 1:1: while both standard GRPO and GRPO with PPL bonus tend to increase confidence on correct answers, the PPL bonus suppresses confident errors. thereby improving calibration.
This finding original and practically important. Ideally, trained model should be faith- ful ful-confident when its answer is correct and cautious when it is not. Better calibration enhances interpretability and supports inference-time selection strategies such as self-certainty BoN (Wang al., 2022) and DeepConf (Fu et al., 2025). It also connects to the growing literature calibrating LLMs, both during training (e.g., (Shen et al., 2024)) and at test time (e.g., (Ulmer et al., 2024)).
07
03
02
03
(a) GRPO without PPL bonus
(b) GRPO with PPL bonus
Figure 8: Average response PPL per training step, stratified by correctness.
8

---

Published as a conference paper at ICLR 2026
4.4 FURTHER ANALYSIS OF THE MULTI-HEAD CRITIC
Analysis of sub-sample fraction Ç. Additionally, we examine the sensitivity of the critic update to the hyperparameter ( (sub-sample fraction). We vary ( under two configurations-crittee with 16 heads and with 4 heads compare Ç E {0.5, 1j. As shown in Table 3, while smaller number of heads benefits from larger sub-sample fraction. the overall performance is stable across settings. The model demonstrates robustness to the masking fraction Ç, achieving similar results for both values tested (0.5 and 1.0).
Table Ablation study on sub-sample fraction Ç.
MATH
AMC23
AIME24
AIME25
Avg
Model
Avg@1
Avg@
Pass@ 16
Avg@
Pass@
Avg@
Pass
Mask fraction
16 Heads; Ç = 0.5
88.3
65.0
88.7
20.5
41.9
20.0
38.8
48.6
16 Heads; ;
85.4
65.3
85.3
21.0
39.2
21.7
43.2
48.4
4 Heads; Ç 0.5
86.1
66.4
85.8
18.1
36.7
23.1
39.1
48.4
4 Headds {
87.3
63.9
87.9
21.5
35.5
21.5
45.5
48.5
Analysis dynamics Beritic across different datasets and over the course of training. First, find that Bentie is sensitive to data novelty. As shown in Figure 9, we compared the average critic disagreement across three datasets. The disagreement is lowest the familiar training data (DAPO-17K) and significantly higher on unseen data from both an in-domain validation (AMC23) and an out-of-domain set (GPQA). This result supports the intuition that the multi-head crities tend show stronger disagreement on data that is less frequently encountered during training.
0.32
0.10
Sid Values
0.00
0.00
0.04
Second, the exploration bonus naturally anneals over the course of training. As the model repeatedly encounters similar reasoning paths, the critics' value estimates converge, thus reducing the bonus. This dynamic provides empirical support for interpreting the multi-head bonus as an implicit, count-based exploration mechanism (see Figure 8, Appendix E).
Ta
AMC23
GPOA
Figure 9: Distribution of the std of value heads across dif- ferent datasets.
4.5 SENSITIVITY ANALYSIS
In this subsection, we conduct more comprehensive ablation study on the hyperparameters k and in Equation 2.
Table 4: Sensitivity analysis of Qwwen-4B-Basse Bas----G-POO under different (x. a) settings.
Model
MATH Avg@1
AMC23
AIME24
AIME25
Avg
Avg@ 16
Pass 16
Avg@11 16
Pass@
Avg@1 16
Pass
Qwen3-4B Base-GRPO
87.3
63.6
89.1
20.8
41.9
21.0
39.2
48.2
3,0 (Original)
67.8
89.5
23.3
48.5
23.5
42.5
20.9
11.1
$4.6
1.5
12.7
1.3
11.4
8.67
2,
87.6
66.5
86.9
22.3
42.5
21.6
40.1
49.5
4,
88.6
68.2
90.1
23.6
46.6
22.7
40.8
50.8
Table 5: Sensitivity analysis of Qwen3-4B-Base-PPP under different (Ks or) settings.
MATH
AMC23
AIME24
AIME25
Avg
Model
Avg@1
Avg@16
Puss@16
Avg@
Pass@
Avg@
Dwee3-4B-BBssePPP
86.6
64.1
87.2
17.8
36.0
17.5
33.7
46.5
3,a 0.5 (Original)
87.3
63.9
87.9
21.5
35.5
21.5
45.5
48.5
1,a=
86.5
65.0
89,5
20.8
42.5
20.5
39.8
48.2
As illustrated Table 5, the critic-wise bonus is generally robust to hyperparameter choices; setting both x and produces nearly identical performance. The actor-wise bonus is relatively more

---

Published as a conference paper at ICLR 2026
sensitive. Generally, to get relative good performance, the clipping fraction k should not be set to small, i.c. the bonus should be constrained to be not exceeding at least 50% of the labeled reward.
5 RELATED WORK
5.1 REINFORCEMEN LEARNING (RI.) FOR L.M REASONING
Reinforcement Learning is central technique for advancing the reasoning capabilities of LLMs. Initial approaches relied on reward models that provided either outcome-based supervision, focusing on the final answer (Cobbe et al., 2021), or process-based supervision (Uesato et al., 2022). More recently, RLVR (Lambert et al., 2024) has emerged as powerful alternative, demonstrating significant performance on complex reasoning tasks in mathematics and coding (Guo et al., 2025). Parallel efforts aim to improve upon the standard RLVR paradigm with techniques such as self-evolving (Huang et al., 2025; He et al., 2025), parallel thinking (Zheng et al., 2025; 2026). Despite these advances. persistent concerns remain regarding robustness (Cai et al., 2025; Zhao et al., 2025; Huang et al., 2026) and a lack of exploration evidenced by entropy collapse (Cui et al., 2025; Shen, 2025), highlighting the need for more principled training approaches.
5.2 EXPLORATION REINFORCEMENT LEARNING
Efficient exploration remains central challenge in Reinforcement Learning, which aim to balance between exploration and exploitation (Sutton & Barto, 2018; Weng, 2020; Amin et al., 2021). Many foundational approaches random-based methods, such as adding Gaussian noise (Lillicrap et al., 2015), e-greedy policy (Sutton& & Barto, 2018), or applying entropy (Haarnoja et al., 2018) or KL regularization (Vieillard et al., 2020) to encourage policy stochasticity. While simple, these methods promote randomness rather than targeted discovery, which can be suboptimal (Dann et al., 2022) and yields inconsistent performance gains in complex settings like Deep RL (Andrychowicz et al., 2021) or LLM training (Cui et al., 2025; Shen, 2025). In contrast, a more powerful paradigm directs exploration by adding an intrinsic bonus to guide agents toward novel or uncertain parts of the environment. This includes count-based approaches like UCB (Lai, 1987), LinUCB (Li et al., 2010), and LSVI-UCB (Jin al., 2020), which use pseudo-counts of state-action visitations to explore rarely visited areas and have achieved near-optimal guarantees in bandits and linear MDPs. A parallel branch involves prediction-base
methods, such as ICM (Pathak et al., 2017) and RND (Burda et al., 2018b), which use predictive model's error as bonus to reward the agent for reaching hard-to-predict states.
Applying these guided exploration principles is growing field in LLM tuning. For instance, Bai et al. (2025) brings count-based principles to the RLHF process by introducing a coin flipping module. Others have focused on prediction-basel bonuses. Drawing inspiration from RND, Gao et al. (2025) use an auxiliary network to predict the output of a fixed target network, with the prediction error ICM-style approach, where the intrinsic
serving as a novelty signal. Sun et al. (2025) implement reward is derived from the prediction error of a learned forward dynamics model. However, these methods require training of additional modules and rely on effective representation of responses, which is challenging (Fu et al., 2024; Barbero et al., 2024). In contrast, CDE offers a simple yet effective approach. It derives intrinsic curiosity signals directly from the actor and crities, requiring only minimal modifications to the training framework. This design yields efficient exploration that is validated both theoretically and empirically on challenging math datasets.
CONCLUSION AND FUTURE WORK
We have presented Euriosity-Drivenr Exploration, an efficient technique that enhances agent learn- ing by incorporating curiosity signals from both the actor and the critic. Our approach is notably lightweight, demanding only minor modifications to the original training architecture. Its effec- tiveness is demonstrated by consistent accuracy improvements over strong baselines on a suite of challenging mathematical reasoning benchmarks, with these empirical results strongly corroborating underlying theoretical framework and intuition. The calibration collapse revealed in analysis aligns with recent findings on the root causes of LLM hallucination (Kalai et al., 2025), pointing to a promising avenue for future work.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
This work does not involve any ethical risks related to data privacy or misuse.
REPRODUCIBILITY STATEMENT
To ensure reproducibility, we provide detailed descriptions of training setups and prompt used in Appendix B. The datasets used are publicly available, and we will release the code upon publication.
REFERENCES
Susan Amin, Maziar Gomrokchi, Harsh Satija, Herke Van Hoof, and Doina Precup. A survey of exploration methods in reinforcement learning. arXiv preprint (xXiv:2109.0017,,, 2021.
Marcin Andrychowicz, Anton Raichuk, Piotr Stanczyk, Manu Orsini, Sertan Girgin, Raphaël Marinier, Leonard Hussenot, Matthieu Geist, Olivier Pietquin, Marcin Michalski, al. What matters for on-policy deep actor-critic methods? large-scale study. In International conference on learning representations, 2021.
Chenjia Bai, Lingxiao Wang, Lei Han, Jianye Hao, Animesh Garg, Peng Liu, and Zhaoran Wang- Principled exploration via optimistic bootstrapping and backward induction. In International Conference Machine Learning, pp. 577-587. PMLR, 2021.
Chenjia Bai, Yang Zhang, Shuang Qiu, Qiaosheng Zhang, Kang Xu, and Xuelong Li. Online prefer- ence alignment for language models via count-based exploration. arXiv preprint 1231 2025.
Federico Barbero, Andrea Banino, Steven Kapturowski, Dharshan Kumaran, João Madeira Araújo, Oleksandr Vitvitskyi, Razvan Pascanu, and Petar Veličković. Transformers need glasses! informa- tion over-squashing in language tasks. Advances in Neural Information Processing Systems, 37: 98111-98142 2024.
Yuri Burda, Harri Edwards, Deepak Pathak, Amos Storkey, Trevor Darrell, and Alexei A Efros. Large-scale study of curiosity- driven learning. arXiv preprint 1808 04335, 2018a.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprint arXiv: 12894, 2018b.
Hanyu Cai, Binqi Shen, Lier Jin, Lan Hu, and Xiaojing Fan. Does tone change the answer? evaluating prompt politeness effects on modern 1lms: Gpt, gemini, llama. arXiv preprint arXiv:2512.12812 2025.
Asaf Cassel and Aviv Rosenberg. Warm-up free policy optimization: Improved regret in linear markov decision processes. Advances Neural Information Processing Systems, 37:3275-3303, 2024.
Junyi Chu and Laura E Schulz. Play, curiosity, and cognition. Annual Review of Developmental Psychology, 2(1):317-343, 2020.
Kamil Ciosek, Quan Vuong, Robert Loftin, and Katja Hofmann. Better exploration with optimistic actor critic. Advances in Neural Information Processing Systems, 32, 2019.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110. 10. 14168, 2021.
Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang, Yuxin Zuo. Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, et al. The entropy mechanism of reinforcement learning for reasoning language models. arXiv preprint 2wXiv:2505.226/7. 2025.

---

Published as a conference paper at ICLR 2026
Chris Dann, Yishay Mansour, Mehryar Mohri, Ayush Sekhari, and Karthik Sridharan. Guarantees for pesilon-greedy reinforcement learning with function approximation. In International conference PMLR, 2022.
machine learning. pp.
Anthony Christopher Davison and David Victor Hinkley. Bootstrap methods and their application. Number 1. Cambridge university press, 1997.
Yichao Fu, Xuewei Wang, Yuandong Tian, and Jiawei Zhao. Deep think with confidence. arXiv preprint arXiv:2508. 15260, 2025.
Yuchen Fu, Zifeng Cheng, Zhiwei Jiang, Zhonghui Wang, Yafeng Yin, Zhengliang Li, and Qing Gu. Token prepending: A training-free free approach for eliciting better sentence embeddings from llms. arXiv preprint arXiv:2412.11556 2024.
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pp. 1050-1059.
PMLR, 2016.
Jingtong Gao, Ling Pan, Yejing Wang, Rui Zhong, Chi Lu, Qingpeng Cai, Peng Jiang, and Xiangyu Zhao. Navigate the unknown: Enhancing I1m reasoning with intrinsic motivation guided exploration. arXiv preprint arXiv:2505. 17621, 2025.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The Ilama herd of models. arXiv preprint arXiv:2407.2178.. 2024.
Daya Guo, Qihao Zhu, Dejian Yang. Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Yu Wu, YK Li, et al. Deepseek-coder When the large language model meets programming-the rise of code intelligence. arXiv preprint arXiv:2401. 2024.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in lims via reinforcement learning. arXiv preprins arXiv:2501. 2025.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy
maximum entropy deep reinforcement learning with a stochastic actor. In International conference machine learning, pp. 1861-1870. Pmlr, 2018.
Yicheng He, Chengsong Huang, Zongxia Li, Jiaxin Huang, and Yonghui Yang. Visplay: Self-evolving vision-language models from images. arXiv preprint arXiv:2511. 156611 2025.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Mcasuring mathematical problem solving with the math dataset. arXiv preprint www.::2200.008//00 2021.
Chengsong Huang. Wenhao Yu, Xiaoyang Wang. Hongming Zhang, Zongxia Li, Ruosen Li, Jiaxin Huang, Haitao Mi, and Dong Yu. R-zero: Self-evolving reasoning Ilm from zero data. arXiv preprint ar0iv:2508. 05004, 2025.
Yue Huang, Zhengzhe Jiang, Yuchen Ma, Yu Jiang, Xiangqi Wang, Yujun Zhou, Yuexing Hao, Kehan Guo, Pin-Yu Yu Chen, Stefan Feuerriegel, al. Probellm: Automating principled diagnosis of Ilm failures. arXiv preprint arXiv:2602. 12966, 2026.
Chi Jin, Zhuoran Yang, Zhaoran Wang, and Michael I Jordan. Provably efficient reinforcement learning with linear function approximation. In Conference on learning theory, pp. 2137-2143. PMLR, 2020.
Adam Tauman Kalai, Ofir Nachum, Santosh S. Vempala, and Edwin Zhang. Why language models hallucinate, 2025. why-language-me
URL https:/ /ttp///pp.aa./////////e//.. con/ index/
s-ha lue nate//
Tze Leung Lai. Adaptive treatment allocation and the aumti-armed bandit problem. The annals of statistics, pp. 1091-1114, 1987.
12

---

Published as a conference paper at ICLR 2026
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017.
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. Tulu 3: Pushing frontiers in open language model post-training. arXiv preprint (rXf::241111512 2024.
Lihong Li, Wei Chu, John Langford, and Robert E Schapire. A contextual-bandit approach to personalized news article recommendation. In Proceedings of the 19th international conference on World wide web, pp. 661-670, 2010.
Timothy P Lillicrapp Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint FrXXvv:509.00277 2015.
MAA. American invitational mathematics examination (AIME). Mathematics Competition Series, n.d.a. URL. htt.hhtts://mmme httpaa/ttommmaoommgeeaeee i ons/ /
MAA. American mathematics competitions (AMC 10/12). Mathematics Competition Series, n.d.b.
URL httpp::/mmaatt orgp::///maaaaa///e//.oo/
www.orrgggoo taargg/math-aae cosechhcommeiiitiioooo amm..
Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via
bootstrapped dqn. Advances in neural information processing systems, 29, 2016.
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International conference machine learning, pp. 2778-2787.
PMLR, 2017.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang. Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a benchmark. arXiv preprint 1:Xiv:2311.12000 2023.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.0666 063077 2017.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint wxXiv22422.03000 2024.
Han Shen. On entropy control in lim-rl algorithms. arXiv preprint arXiv:2509.0344 2025.
Maohao Shen, Subhro Das, Kristjan Greenewald, Prasanna Sattigeri, Gregory Wornell, and Soumya Ghosh. Thermometer: Towards universal calibration for large language models. arXiv preprins 1xXiv:2403.08819 2024.
Haoran Sun, Yekun Chai, Shuohuan Wang, Yu Sun, Hua Wu, and Haifeng Wang. Curiosity- reinforcement learning from human feedback. arXiv preprint arXiv:2501. 1 2025.
Richard S Sutton and Andrew G Barto. Reinforcement Learning: An Introduction. MIT press, 2018.
Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, OpenAl Xi Chen, Yan Duan, John Schulman, Filip DeTurck, and Pieter Abbeel. # exploration: A study of count-based exploration for deep reinforcement learning. Advances in neural information processing systems, 30, 2017.
Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Sicgel, Lisa Wang. Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process-and outcome-based feedback. arXiv preprint arXiv:2211.142755 2022.
Dennis Ulmer. Martin Gubri, Hwaran Lee, Sangdoo Yun, and Scong Joon Oh. Calibrating large language models using their generations only. arXiv preprint arXiv:2403.0549 2024.
Nino Vieillard, Tadashi Kozuno, Bruno Scherrer, Olivier Pietquin. Rémi Munos, and Matthieu Geist. Leverage the average: an analysis of kl regularization reinforcement learning. Advances in Neural Information Processing Systems, 33:111633-111722 2020.
13

---

Published as a conference paper at ICLR 2026
Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang, Xionghui Chen, Jianxin Yang, Zhenru Zhang, et al. Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for Ilm reasoning. arXiv preprint arXiv:2506.01933: 2025.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdh- ery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint 2rXiv:2203.1177,, 2022.
Lilian Weng. Exploration strategies in deep reinforcement learning. lilian- Jun 2020.
URL https://lilianner github..hh..... github.io/pon io/posts/ 2020-06- 07 07-explorationndt1...
geeng.github.io
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint 0093885005.0988, 2025.
Fei Yu, Yingru Li, and Benyou Wang. Uncertainty-aware search and value models: Mitigating search scaling flaws in llms. arXiv preprint 15Xiv:2502.//155, 2025a.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source IIm reinforcement learning system at scale. arXiv preprint arXiv:2503.
2025b.
Yulai Zhao, Haolin Liu, Dian Yu, SY Kung, Haitao Mi, and Dong Yu. One token to fool llm-as-a- judge. arXiv preprint arXiv:2507.08794. 2025.
Tong Zheng, Hongming Zhang, Wenhao Yu, Xiaoyang Wang. Runpeng Dai, Rui Liu, Huiwen Bao. Chengsong Huang, Heng Huang, and Dong Yu. Parallel-rl: Towards parallel thinking via reinforcement learning. arXiv preprint arXiv:2509 07980, 2025.
Tong Zheng, Chengsong Huang, Runpeng Dai, Yun He, Rui Liu, Xin Ni, Huiwen Bao, Kaishen Wang, Hongtu Zhu, Jiaxin Huang, al. Parallel-probe: Towards efficient parallel thinking via 2d probing. arXiv preprint arXiv:2602.03844 2026.
14

---

Published as a conference paper at ICLR 2026
A THE USE OF LLMS
We utilized Large Language Model (LLM) to proofread the manuscript for grammatical errors and to improve clarity and readability.
B TRAINING DETAILS
We conduct experiments using the verl training framework? All models are trained with the prompt from (Yu et al., 2025b), which is illustrated Figure 10. Detailed training configurations for CDE and the baseline models correct answer and r = -1 for an incorrect one. We also address a bias introduced by the training data 10KK (Yu et al., 2025b)), which consists solely of numeric answers. To prevent this from creating evaluation artifacts, we filter out problems from MATH-500 (Hendrycks et al., 2021) that require non-numeric answers.
provided in Table 6. We use binary reward signal:
for
Table 6: (a) Baseline training configurations. The GRPO setup is shared across all GRPO-based methods (e.g., -Qwen3-4B-Base- GRPO" and "w/PPL bonus" in Table 1); likewise, the PPO setup shared across all PPO-based methods. (b) CDE-specific configurations. The PPL settings are identical for both the GRPO "w/PPL bonus" and PPO "w/PPL bonus" variants.
Config
GRPO
PPO
actor-Ir
1e-6
1e-6
critic-Ir
le-5
critic-warmup
10
kl_coef
0.0
0.0
Config
PPL
2,4 Heads
8,16 Heads
max prompt.lenge length
2K
2K
mox.respponeelee lenghh
3K
3K
3
3
train_batch size
256
512
a
0.5
0.5 No decay
ppo.mini. batch
256
256
Staircase
No decay
ratio
0.20
1
0.5
sample temperature
1.0
1.0
(b)
rollout.n
8
4
total_training steps
300
300
discount factor 7 temperature
(a)
Solve the following math problem step by step. The last line of your response should be of the form Answer: SAnswer (without quotes) where $Answer is answer to the problem. (Problem)
Remember to put your answer on its own line after "Answer:".
Figure 10: The prompt for RLVR training.
C COMPUTATION COST OF MULTI-HEAD CRITIC
The additional computational overhead introduced by the multi-head critic is minimal. As illustrated in Figure 2, all heads share the same LLM backbone as the single-head critic; the only architectural difference lies in the final classification layer, whose output dimen- sion changes from to the desired number of heads. In VERL and related Transformer based codebases, this modification corresponds simply to setting the num numelabelll argument in AutoMode ForToker fi cat conion. ftt.ca......ooooeee....oo.... rrompreetra ind ned to the number of heads (e.g.. 4 or 16).
httpps:/ http:://ggtttuu..ut www..cto.bbooo...oooooooooooooot cengi ne/ver1
15

---

Published as a conference paper at ICLR 2026
To empirically validate that this architectural change incurs negligible overhead, compare the memory consumption and runtime of the multi-head critic against the same Qwen3-4B- Base backbone described in the main text. Results confirm that the additional cost marginal relative to the overall backbone computation.
the original single-head critic using
Table 7: Memory Usage for Different Critic Heads
Critic Heads
Total Parameters
Model Size (Ip16)
Model Size (fp32)
Head
4,022,470,637 4,022,478,340
7672.25 MB
15344.51 MB
4 Heads
7672.27 MB
15344.54 MB
16 Heads
4,022.509,072
7672.33 MB
15344.65 MB
To obtain a reliable measurement of computation time, we generated inputs with bat chessieeeee.eeee 8 and seq-len-22 leg-1en,56, 256, and ran both forward and backward passes of the critic for 500 iterations. We then report the averaged time consumption over these runs.
Table 8: Average Computation Time per Iteration (ms)
Heads
Forward (ms)
Backward (ms)
122.595 + 9.410
439.958 + 2.787
122.611 10.312
440.713 + 3.036
16
122.410 + 9.631
442.260 + 2.425
The results confirm that both memory overhead and computation time remain nearly unchanged.
D DETAILS ON HASH-BASED PSEUDO COUNT
The core idea is to map full prompt-response response trajectory compact hash code, which serves as a pseudo-state for count-based exploration. For a given trajectory, the LLM produces a sequence of last-layer hidden states, h (hi..... hD+T), where h; Rd Following prior work (Bai et al., 2025; Gao et al., 2025), we use the final hidden state of the last token as the trajectory's 's embedding, hiraj hD4T.
We then use SimHash (Tang et al., 2017) to generate a k-bit hash code. First, a fixed random projection matrix A R is generated with entries drawn i.i.d. from a standard normal distribution, (( I). The hash code o(g, o) is computed as Ølg. o) = sign( (Abtra)) E 11].. This k-bit vector is then mapped to an integer bucket index b E {O..... 2* 1}. Let r2(b) be the visitation count for bucket b. The exploration bonus is based on this count, shaping the original reward r(1, o) as follows:
F(g,o) = r(g,0)
(n(b)
where hyperparameter controlling the exploration bonus. This method offers lightweight, count-based guidance compared to prior work that requires matrix inversions (Li et al., 2010) or the training of additional modules (Bai al., 2025). Our SimHash-based exploration method is detailed in Algorithm 1, with results presented in Table 9, The experiment shows that this count-basee approach does not yield significant performance increase. We hypothesize that this due the reasons discussed in Section 3.1, namely the difficulty of representing complex trajectories with single embedding.
E ADDITIONAL EXPERIMENTAL RESULTS
We also experimented with mean- mrcan-pooling over all hidden states but found it produced less discriminative emheddings, leading more hash collisions.
16

---

Published as a conference paper at ICLR 2026
Table 9: Zero-shot accuracy of GRPO models with count based exploration bonus.
MATH Avg@!
AMC23
AIME24
AIME25
Avg
Model
@
Pass@
1v
Pass@ 16
Avg@
@ass@
Bonus Weight Decay Schedules
Swen3-4B-Base--ae Hush-bonus
87.3
63.6
91.1
21.0
41.9
20.8
39.2
48.2
91.0
20.8
39.3
21.2
40.3
49.0
Algorithm 1: Count-based exploration for RLVR through SimHash
Inputs: Policy To; aggregator g(-); )(:); random projection matrix A E IR*dd hash counts n(-] 0;
exploration coefficients B.
for each training iteration do Sample prompts q and generate responses o ~ TT(- 1)
for each (g. in the batch do
Obtain last-layer token states {h.),-
1//
hiraj g((h,)
// e.g., select the last state |
sign( Aheras) b - bucket(o)
// Compute k-bit hash code
1/ Convert hash code to integer index
m[bb] n mb +1
o) - r(q, o)
// Shape the reward with the bonus
Update Th using the shaped rewards F(g,
Analysis of Entropy Dynamics As highlighted in prior work, entropy provides an important lens for understand- ing exploration ability (Cui et al., 2025), where sharp decline in entropy often signals premature convergence and insufficient exploration. Figure 11 illustrates the en- tropy dynamics of bascline GRPO compared to our pro- posed methods. First, relative to the baseline, the PPL. bonus alleviates entropy collapse, demonstrating its role promoting exploration. Second, when comparing decay schemes, PPL with No Decay shows persistent fluctua- tions and fails to converge, whereas Staircase decay yields more stable entropy trajectories. This observation is sistent with our carlier findings that decaying the bonus weight is essential for ensuring stable convergence while still supporting effective exploration.
ARE
Figure 11: Dynamics of policy entropy over the training process. The bonus weight decay mechanism follows Fig- ure 7.
Analysis of Beriic Dynamics In Figure 12, present a cross-dataset analysis by calculating the average standard deviation of the value estimates across different questions. Specifi- cally, we evaluate three datasets: the training set (DAPO-17K), the in-domain validation set (AMC23), and the oft-of-domain validation set GPQA (Rein et al., 2023). We observe that the training set exhibits a smaller standard devia- tion compared to both the in-domain and out- of-domain validation sets. This pattern aligns with the intuition that multi-head critics tend to show stronger disagreement on data that is less frequently encountered during training.
0.14
0.12
Avergt
0.08
0.94
1.0
202
250
Training Steps
Figure 12: The average Bestic over training steps.
17

---

Published as a conference paper at ICLR 2026
F PROOF FOR CALIBRATION THEOREM
Define Fr, 0) = r(g. o) + bn o) where bb(g, o) = w min min(fx/(a o)(, - log (olg)) is bonus function where |o] is the length of response o. Note that w is redundant variable in theory because we can write (g, o) min(x"Ir(( 0)1, -oo log mT with x' wk and a' wa. Given that r(z, y) E {1, 11,. it suffices to consider (g, o) = minf", of log (olg)) Thus, as long as we use 1, we have sign(fn (g. o)) sign(r(q, o)). The introduce of bonus does not change the sign of the original correctness reward.
Consider single step policy optimization
Σ =(ol)))a(a -K (m((la))za(10))
Th+1 (-1g) arg max
"
which has closed-form form solution
Th (olgg exp exp
Th+1(olg) =
Σn ( exp (q,
For any question and response o. Define Z(g) = E. Th (O'lg) exp ( (g, o')), we have
log ( log (olg) (q, log (Z()))
Define (olgg) log < log (olg) the change of likelihood of response under
question at update step t. For two correct response of and of with length lot| and |oš I. and log (o (q) >
log T. (o5 [q) (i.e. of has larger perplexity), we have
lo
1o:1
An(ot 1g) An lq)
= o - (g
bn(g, bn bs( oz)
min(k,
log ( min |o
a [0g
log nA (o
Similarly, for two incorrect response of and 0g with log Th (0
log T. (05 (i.e.
1o
0j has larger perplexity), we have ( lq) (o,
lq)
Specifically, given a question q, for any response (01, 02) that has the same correctness label and a log (o1lg) [oo] log Th (02lq). we have
[
If 01) > log (Z))) and o2) > log (Z(4) then An(o1l)) >> and An(o2]q) 2 0 but 0ı has more likelihood increase.
If 01) > log (Z(4) and 02) < 1 log (Z()) then (x(o)gg) > and An(oz]g) < 0 where 01 's likelihood increase but 02 's likelihood decrease.
If fn(q. 01) < logg (Z()) and oz) < 1 log (Z(4)) then (o)l)) (n(o2lg) < 0 but 01 has less likelihood decrease. It is impossible that Pn 01) < 1 log (Z()) and (g. 02)
and
> ") log (Z(g) given that
(01, 02) (2) has the same correctness label and p log ()
lol log (o2lg).
G PROOF FOR CONSISTENCY OF MULTI-HEAD CRITIC BONUS
Linear MDP and Assumptions
Assumption G.1 (Linear MDP). We consider finite horizon M = (S,A, R, P. H) with horizon T, state space action space A, reward function R: A IR, and transition P: S x A S such that there exists a known feature & R&d and unknown unknown,eeutures 0, E IRd to ensure
R(s,a) a(
P(518,, a) = d(s,a) Tv(o'.
Without loss of generality, assume [()(s,a)] & 1 for all (s) and (b)))) & vd, [0| 2 vd.
18

---

Published as a conference paper at ICLR 2026
Lemma G.2 (Proposition 2.3 in Jin al. (2020)). For linear MDPs that satisfy Assumption G.I, there exists w E R2 such that
(7) =
Σ h-t
nh sh = s,00 = a = 0(s a) aaaa)"uu w
The linearity of Q-functions enables using regression technique to solve it. Consider a dataset with n observations D {$i,1, and denote the regression noise E Gt,² wt. We impose the following assumptions.
di,t ( Gr, where Gi.t is the Monte-Carle return. Let Φi, (( as.))
(A1) E El (h.e) 0 and [(e.))]== are i.i.d. o² zero mean-Gaussian for each fixed (; (A2) Ei= 0i, P. E₁ >
Jin et al. (2020) shows that doing value iteration optimistically estimated Q function achieve near-optimal regret for linear MDP, where the optimistic Q function is the combination of linear re- gression estimation and exploration bonus ,,,, = Bv AT, where An, = A Ex= $5,0%, and B is some constant. Below will formally connect our bootstrapped bonus with this term.
Formulation of the bootstrap multi-head critic
We accommodate the bootstrap multi-head into the linear-MDP setting. For any time step sample K mini-batches {S, [n]]]_] of size m = Çn uniformly without replacement from and construct the ridge estimator follows
(k) Vat
arg min
EGr, (G, <\]lw]
rESH
For any feature á Rd, we define the bootstrap multi-head bonus as
bl (#) std (0)
(k)
& k < k)).
Elliptical ("count-basedd) bonus in (Jin et al., 2020). The ridge estimator is constructed using all data across n trajectories as follows
Hn, arg min
Σ (Gi (G,)-
2 (\llw||?.
For any query feature o Rd, the bonus term is (p) VTTAA
Formal Version and Proof of Theorem 3.2
Theorem G.3. Under Assumption G.I and assumptions (A1)-(22), for any fixed time-step t and query ó E Rd,
Hoot (b)
K- n-400 BV oT 1
where B is some constans.
Proof. For any time-step èt and S4 = [n]] we have the explicit solution of the ridge regression
wn.t = A
Gi.
Conditioning on X, [T. TT.]. the conditional variance of the estimator is
Var( wast Xx)) = 0²8 (Anit - AAm33)
19

---

Published as a conference paper at ICLR 2026
From Assumption (A2), we have
Amina An,)) = Amin (V+ +
(1) = A + nemiin
Σ
2ia = Op(n).
(4)
==1
By definition of matrix operator norm, have |A (mir(Am,e))
Op(1/n). Therefore
Φ T A & |06 A.tllap = Op(1/n) and Φ Am,00 & Op((1/1²),
and
nøl (Amt -- x- no A -dd ~ 2 0.
Therefore, we have
t - -A A,7o
(5)
Before moving to 6i (d), we define the following quantities
AE Σ rES.
Σ C b
b, b,= b
Σ Gr.s rES#
Ab bg b.
Since Mi > O, matrix Bernstein for sampling without replacement yields | op Oplyn). From equation 4, we have Amin(An.e) X, similarly we have Amin( (An, + A A. Therefore, both matrices are invertible. Therefore, we have
(An, + (A) + AΣ)-1=1
Rearranging terms yield the following form
(An, + AΣ) = A,T A AE + -E)-1
Substituting this expression for (An, + AE) back into the right-hand side once more yields
(An, + AL) A A, + Rr.
where Rs AT! + AE)-1 and Rszlopp = Op 0p(1AA,,110 = Op(1/n2).
The k-th bootstrap ridge solution is
(k) wnt (An,t AE)
Subtracting @n,t A,, and inserting the expansion,
(K) n.t Wn,t = A, A M Wn.
AAAAAA Rybs))
first order
Since Gt., T.,Wer + ci,e, for any & we have
T Wn.t = Am (
Σ or,trr. (enee))) + TO A wn,)) TT., (6)
reSt
Standard results for ridge regression imply that lmwt @n,el12 Op(nn1122) Consequently. the second term is of order (n -1). Similarly, for the last term we have
T r. < ||*|| [ AE Ab) + || lop) = Op(n
We now analyze the conditional variance of the estimator o xt pp.-- Let D. ((or. (,,) be the fixed dataset. Conditional on D.,, the subsampled estimator (k) from the index set St, while the full-sample estimator Wn,t and the sum Σn &i, i constants.
n,t contains randomness solely
20

---

Published as a conference paper at ICLR 2026
Neglecting the remainder terms of order Op(n "), the conditional variance is dominated by the leading term
Var ④ " Ds) = Var Var(o' (0 - (n,e) (r))
a
T An Σ
Orienn Σ Disees Amio
reSk
52
A
Var
Σ
Pr, (D))) Am1
rESa
Let 1 = E Sx) be the inclusion indicator. We can rewrite the conditional variance
Var
Σ Or.tnn Dn)
Var
Σ
Lidr,es,s Dm))
From finite saiite-population sampling theory (simple random sampling without replacement), we have the moments
5(1 ()
E Dn] C,
Var(T, (n.) <(1 () Cov(Is, 13 Dn)
(i + f).
71
Therefore, we have
n
Var
Σ
1, Dm)
1
Σ
Var(I D + "'Cono Cov(I;, D Dn)or
11 1-1
<(C 5(5-1) n n-1 n
<(1 5)
(7)
For the first term in equation 7, we condition on the features { 1 Since the noise terms (cr, _ are i.i.d. zero-mean Gaussian, and the features are bounded, applying the Law of Large Numbers for weighted sums yields
P, 1 Σ 0u.ol.00
n
As for the second term in equation 7, we make further decomposition as follows
Σ
For the first term, have
( (),)) 1lop = > 0
Since Ei si (1:5) 0² Σ 1ー 1 we have B[I En dc 0(m). With Markov
inequality, we have -m 1 (n).. As for the second term, have > TT P, T = (p()). Therefore, for the second term in equation 7,
E 21,0, thus Σ
we have S(5-1) 1 Eis n-1 itjj J E,, E. and
Var Σ draert D.) P. <(1- (1--) 9
PESt
Therefore, conditioning on [ we have
(k) Var ④ 1 Dn)) P, 5-1 2 A
Σ
A
1- -
200 (Am)
\smt)
C
21

---

Published as a conference paper at ICLR 2026
Then, with equation 5, we have
Var ④ (k) Da) &. 020T An,
Finally, by the conditional strong law of large numbers, we have
(x) = std (fo T mi
& k = k)) 2 V
Var @TTun,
W n.t Drs
ov / TTA
22


Reviews:
[{"review_id": "jrom4Q2FfB", "reviewer": "Reviewer_XcFv", "summary": "+ Summary & Contributions\n\t- The authors focus their attention on the exploration challenge in RL fine-tuning of LLMs.\n\t- Following suit with curiosity-driven exploration strategies in deep RL, the authors propose their own curiosity-driven exploration methods for RL fine-tuning. \n\t- Following an actor-critic setup for RL fine-tuning, the authors propose using the perplexity of the actor's response distribution as an intrinsic reward for guiding exploration. Meanwhile, the authors employ a critic ensemble with bootstrap sampling to approximate the true Bayesian posterior over the value function induced by the current actor policy and use the corresponding ensemble variance as an intrinsic reward to further guide exploration.\n\t- A theoretical result is provided to show that the use of the ensemble critic standard deviation to guide exploration is an asymptotically consistent estimator of the elliptical potential bonus used by LSVI-UCB (a statistically-efficient exploration algorithm) under a linear MDP assumption.\n\t- Empirical results show that the proposed curiosity-based reward for exploration yield improved performance relative to standard GRPO and PPO.", "questions": "Please see above.", "limitations": "", "rating": null, "confidence": 4, "soundness": 1, "presentation": 2, "contribution": 1, "strengths": "+ Quality\n\t- Strengths\n\t\t- The authors display a nice insight in trying to leverage ideas for exploration strategies from deep RL to improve LLM exploration during RL fine-tuning.\n\t- Weaknesses\n\t\t* Major\n\t\t\t- There is absolutely no scientific evidence offered in this paper to support the assertion that LLM reasoning aligns with models of childhood exploration in developmental psychology (L69-72, L151-155). This kind of hyperbole is also rather pointless as the authors could just introduce their method without unsubtantiated nonsense to give their method the veneer of being grounded in cognitive science. As the authors already point out, there is much deep RL work leaning on ideas of curiosity which already serves as a sufficient foundation to warrant their study in the context of LLMs and RL-based fine-tuning.\n\t\t\t- In light of the authors' proposed curiosity signals as intrinsic rewards, they proceed to augment the computation of advantages (L273). This runs counter to how intrinsic motivation is normally applied in (deep) RL, where the underlying policy optimization algorithm (PPO, GRPO, etc.) remains entirely unchanged and is simply run on the augmented MDP whose reward function has been modified to include the original environment reward plus a (potentially) weighted intrinsic reward term. Why did the authors feel compelled to deviate from this recipe? Why is it necessary or even sensible to modify the definition of the advantage function based on curiosity?\n\t\t\t- In motivating their proposed actor curiosity signal based on surprisal (not perplexity -- see minor comment below), the authors claim that a response with low probability under the current policy \"resides in an underexplored region of its learned distribution.\" An alternative but equally (if not more) plausible possibility is that this response, either during the initial pre-training stage or in earlier rounds of fine-tuning, was determined to not yield correct responses, lead to negative advantages, and has accordingly had its policy of being generated reduce through successive policy-gradient updates. How are the authors able to take this surprisal curiosity signal and disambiguate between inexperience with a particular response from the implausibility of that response being optimal/correct? The fact that they can't seem to distinguish between those two critical scenarios is perhaps what warrants the subsequent clipping scheme (Equation 2).\n\t\t\t- The proof of Theorem 3.2 isn't actually a complete proof. There are numerous points where the authors simply claim a particular equation or inequality without any kind of explicit, step-by-step justification. Where are the steps to establish how Assumption A2 leads to $||\\Lambda^{-1}_{n,t}||_{\\text{op}} = O_p(\\frac{1}{n})$? Where did the expansion in L895 come from? It seems like a consequence of the Sherman-Morrison-Woodbury matrix identity, but the whole point is to give a complete proof detailing those steps. Equation 4 is just stated as a matter of fact without any concrete steps as is the subsequent inequality. The authors vaguely gesture towards \"finite-population sampling theory\" without so much as a single citation of an existing result being utilized to then state an equation for \"Var*\" (which denotes some entirely unspecified variance -- why *?). Overall, the proof is entirely incomplete and, more worryingly, seems to be an amalgamation of equations and inequalities likely taken from another proper RL theory paper (the LSVI-UCB paper perhaps) just to inject math into the paper for the sake of justifying the authors heuristic approach (see \"Mathiness\" in [2]).\n\t\t* Minor\n\t\t\t- What the authors continuously refer to as the perplexity of the actor is, in reality, just the surprisal [1]. The authors even acknowledge that what they refer to isn't actually the perplexity in a footnote (L215). What is the point of erroneously labeling the surprisal as the perplexity?\n\t\t\t- While being able to give names to each of the injected hyperparameters in Equation 2 seems useful, it results in an unnecessarily convoluted equation. Either there is an $\\frac{\\omega}{\\kappa}$ term which can be consolidated into a single hyperparameter or a $\\omega \\cdot \\alpha$ term which can also be consolidated. Why can't Equation 2 follow the standard form of $r(q,o) + \\lambda \\cdot B_{\\text{actor}}(q,o)$?\n\t\t\t- The authors claim that incorrect responses recieve a \"larger penalty\" by virtue of a smaller PPL bonus. Notice that a larger penalty is not the same thing as smaller reward bonus; relative to other actions, this might still end up inducing a positive advantage an increasing probability mass for incorrect responses. Also, it seems like the convolute structure of Equation 2 means this could be undone anyways through the $\\kappa$ parameter?\n\t\t\t- Calling Theorem 3.1 a theorem is rather generous. The authors have stepped through the possible cases for the signs of log-probability differences.\n\t\t\t- The authors mention a \"posterior distribution\" when discussing actor-critic methods. What is this posterior distribution over? Where did it come from? Actor-critic methods do not engage with any kind of Bayesian posterior distribution by default.\n\t\t\t- There is some exposition concluding Section 3.2 where the authors distinguish their incorporation of a surprisal exploration bonus as distinct from traditional entropy regularization. I agree with this, however what they do end up convincing me of is that their proposed surprisal bonus may be recovering (either entirely or partially) the objective tackled by maximum entropy (maxent) RL approaches. In this case, a more appropriate baseline to compare to would be Soft Actor Critic (SAC) [3], rather than PPO.\n\t\t\t- It's not clear why averaging is the correct thing to do for deriving an alternative GAE for the ensemble critic. If each ensemble member represents a distinct hypothesis about the underlying $V^\\pi$ then its not clear why averaging would be sensible. If anything, a posterior distribution over $V^\\pi$ implies a posterior over $Q^\\pi$ and, consequently, the advantage function itself.\n\n+ Clarity\n\t- Strengths\n\t\t- Overall, the paper is reasonably written and reasonably structured.\n\t- Weaknesses\n\t\t* Major\n\t\t\t- N/A\n\t\t* Minor\n\t\t\t- The presented results for a hasing based technique seem rather out of place for this work. At the very least, it seems like the space could be better utilized for some other purpose more central to the main contributions of this work and the presented result could be relegated to the appendix with little impact on the paper.\n\t\t\n\n+ Originality\n\t- Strengths\n\t\t- The incorporation of the specific proposed curiosity signals to enhance exploration in RL fine-tuning of LLMs is, to the best of my knowledge, novel.\n\t- Weaknesses\n\t\t* Major\n\t\t\t- It would be surprising to me if the authors are in fact the first to propose intrinsic motivation as a remedy for the well-known entropy loss/response distribution collapse issues known to plague RL fine-tuning of LLMs. Have the authors done a proper literature review for other intrinsic rewards aimed at remedying the same problem. Some of these seem to have been identified in Section 5 and yet not compared against as baselines in the reported experiments.\n\t\t\t- While curiosity-based exploration and intrinsic motivation are two remedies for facilitating better exploration in RL, they are by no means the only mechanisms for doing so. Indeed, existing work has already explored other avenues for addressing the same exploration challenge using different techniques [4,5]. The authors have done the bare minimum in their experiments, offering standard PPO/GRPO as baseline comparisons. A more thorough empirical evaluation would properly compare against other competing, alternative exploration strategies that go outside the avenue of curiosity/intrinsic motivation advocated for in this work.\n\t\t* Minor\n\t\t\t- N/A\n\n+ Significance\n\t- Strengths\n\t\t- The reported results would seem to confirm that, up to suitable hyperparameter tuning, the proposed curiosity signals do enhance exploration for RL fine-tuning to result in improved performance beyond GRPO/PPO.\n\t- Weaknesses\n\t\t* Major\n\t\t\t- Across the tasks, the gains in performance seem quite small numerically. Moreover, there is no reporting of how many trials/random seeds are used to obtain these results leaving ambiguous the questions of whether the proposed curiosity metrics consistently enchance exploration in a way that yields statistically-significant performance improvements. Also, given the small size of improvements, whether the proposed curiosity signals are worthwhile given the potentially laborious grid search needed to find suitable hyperparameter values.\n\t\t* Minor\n\t\t\t- N/A\n\t\t\t\n\t\t\n+ Final Remarks\n\t- Overall, I have identified critical issues with this paper on the axes of quality, significance, and originality. While the initial idea has some potential, there is much more work needed on the technical details and presentation as well as the empirical support for the method before publication.", "weaknesses": "Please see above.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AnALOf53KI", "reviewer": "Reviewer_fSmQ", "summary": "The paper first conducts preliminary experiments showing that counting-based exploration methods using response embeddings are ineffective for LLM RL and then proposes Curiosity-Driven Exploration (CDE) to guide exploration during LLM RLVR.\nThe framework contains two parts:\n   - exploration guided by actor curiosity: uses the negative seq logprob for reward shaping, with clipping and scaling coefficient to avoid reward hacking.\n   - exploration guided by critic curiosity: uses a critic model with multiple heads, each trained on a subset of data, and uses the ensemble value for GAE while taking the standard deviation across heads as an exploration bonus.\n\nExperiments mainly on Qwen3-4B-Base and math demonstrate the effectiveness of CDE, and additional experiments show that actor curiosity (negative logprob) can improve calibration.", "questions": "Please refer to the weakness section for main questions.\n - I am a bit confused about the training setup in this paper. Are you doing on-policy or off-policy training? The entropy curve of GRPO w/o PPL in Figure 11 looks somewhat abnormal to me, I would expect a higher entropy level if using clip-higher in off-policy setting. And clip-higher should be a stronger baseline. So I checked the hyperparameters table, and based on the bs and mini_bs, it appears to be on-policy, but clip_ratio is also listed, which typicality has no effect in on-policy training. Could you clarify the exact training setting used in these experiments? If it is indeed on-policy, have you also validated your methods under an off-policy setting?\n- The footnote explaining PPL bonus should probably be moved to the first page, since you start using this notation in the introduction, but the clarification appears a little late.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is easy to follow, with clear motivation and structure. The SimHash attempt is also informative.\n- Experiments show nontrivial improvements over vanilla GRPO and PPO baselines, supported by ablations on specific hyperparameters.\n- The calibration analysis is also a bonus.", "weaknesses": "- Using the negative log-probability for reward shaping is not entirely new. The same formulation has been explored in related work [1], and it is also closely related to entropy shaping, which is also widely explored in related works [2, 3]. The clipping mechanism involving $\\lvert A \\rvert / \\kappa$ is also conceptually similar to that in [3]. While direct comparisons are not strictly required by policy, discussing these related approaches would help position the contribution of this paper more clearly.\n- The proposed framework introduces several new hyperparameters, yet only the ablations for coefficient scheduling for GRPO and sub-sampling fraction for critic update are provided. Given the number of hyperparameters, it would be worthwhile to conduct a more comprehensive sensitivity analysis.\n- All experiments are conducted on Qwen3-4B-Base and math datasets. While I understand that it might be computationally expensive to scale up, it would be worthwhile to at least conduct experiments on other model families. To be honest, I won't be fully convinced by \"Qwen + math\" combination. Also considering the current field with various GRPO variants, in my opinion, vanilla GRPO is a weak baseline.\n\n[1] Know When to Explore: Difficulty-Aware Certainty as a Guide for LLM Reinforcement Learning, arxiv 2025.\n\n[2] GTPO and GRPO-S: Token and Sequence-Level Reward Shaping with Policy Entropy, arxiv, 2025. \n\n[3] Reasoning with Exploration: An Entropy Perspective on Reinforcement Learning for LLMs, arxiv 2025.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZfmhRDHBQt", "reviewer": "Reviewer_XET9", "summary": "This paper introduces 'Curiosity-Driven Exploration' (CDE) to improve RLVR exploration. It proposes two bonuses: perplexity (PPL) for GRPO and multi-head critic variance for PPO. The method is shown to fix a \"calibration collapse\" where baselines become overconfident in their errors.", "questions": "Please refer to the weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "* The proposed PPL bonus is a simple, intuitive, and well-motivated method for exploration. The paper provides compelling evidence that this simple bonus directly fixes the identified calibration collapse.\n* The paper is well-written and easy to follow.", "weaknesses": "* The experiment setting is a little vague. It is not clear in Table 1 whether the experiments on PPO with description `w/ x Heads` are combined with PPL. \n    * If yes, why `w/ 2 Heads` is worse than `w/ PPL`?\n    * If not, had the authors tried to combine PPL with a multi-head critic, and what are the results? \n* The discovery that a \"Staircase\" decay (i.e., a hard stop) for the PPL bonus is optimal (Table 2) is counterintuitive and feels ad hoc. The paper would be stronger if it investigated why a hard cutoff is so much better than a smooth anneal.\n* The paper's argument for dismissing count-based exploration seems contradictory. Section 3.1 argues that hash-based counting fails due to the \"poor expressiveness\" of LLM embeddings. However, the proposed multi-head critic method (Section 3.3) fundamentally relies on these same LLM hidden-state representations to feed its value heads. The paper does not resolve why these representations are considered too poor for hashing but sufficient for learning complex value functions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MpYUFfDsgj", "reviewer": "Reviewer_KJQc", "summary": "The paper proposes Curiosity-Driven Exploration (CDE) to improve exploration and calibration in RL-based fine-tuning of large language models. The method adds two intrinsic signals: a per-response perplexity bonus for the actor and a variance-based uncertainty bonus across bootstrapped value heads for the critic. Both are clipped and gradually reduced. Experiments on math-reasoning datasets with a Qwen-3-4B model show better accuracy and more stable confidence behavior. The contribution is mainly algorithmic and empirical, with a small theoretical part that connects the critic variance to UCB-style exploration.", "questions": "- Are the value targets trained with extrinsic-only rewards? \n- How are the bootstrap heads decorrelated (data resampling unit, optimizer states, seeds, dropout)?\n- Did you try actor-only and critic-only? Are the gains additive?\n- Could you add multiple seed averages / confidence intervals on the main tables?\n- Could you compare the approach with existing prior work in curiosity-driven RL?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- Idea that could have a large impact on the LLM community, which is easy to implement on top of existing PPO/GRPO pipelines.\n- Sample-specific actor bonus helps control over-confidence and keeps diversity among correct answers.\n- Critic variance is simple but works well; the UCB connection gives intuition.\n- Results are consistent across multiple datasets; training curves and calibration plots are convincing.\n- The paper is clear and well structured; contains ablation studies.", "weaknesses": "- Limited scope: all tasks are math; no larger model or other domains such as code or factual QA. How useful is it on other tasks?\n- No compute/cost analysis: multi-head critics increase memory and time, but this is not measured.\n- No failure cases or examples showing what happens if exploration weight is too high.\n- Reproducibility: no public code/configs mentioned.\n- Lacks references to traditional curious-driven approaches employed in RL community. See references for examples\n\n@InProceedings{pmlr-v260-bougie25a,\n  title     = {Exploring Beyond Curiosity Rewards: Language-Driven Exploration in RL},\n  author    = {Bougie, Nicolas and Watanabe, Narimasa},\n  booktitle = {Proceedings of the 16th Asian Conference on Machine Learning},\n  pages     = {127--142},\n  year      = {2025},\n  volume    = {260},\n  series    = {Proceedings of Machine Learning Research},\n  month     = {December},\n  publisher = {PMLR},\n  url       = {https://proceedings.mlr.press/v260/bougie25a.html}\n}\n\n@inproceedings{burda2019rnd,\n  title     = {Exploration by Random Network Distillation},\n  author    = {Burda, Yuri and Edwards, Harrison and Pathak, Deepak and Storkey, Amos and Darrell, Trevor and Efros, Alexei A.},\n  booktitle = {International Conference on Learning Representations (ICLR)},\n  year      = {2019},\n  url       = {https://openreview.net/forum?id=H1lJJnR5Ym}\n}\n\n@inproceedings{osband2016bootstrapped,\n  title     = {Deep Exploration via Bootstrapped {DQN}},\n  author    = {Osband, Ian and Blundell, Charles and Pritzel, Alexander and Van Roy, Benjamin},\n  booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},\n  year      = {2016},\n  url       = {https://proceedings.neurips.cc/paper_files/paper/2016/file/8d8818c8e140c4643106b7c59d6a9b2c-Paper.pdf}\n}", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)