Count Counts: Motivating Exploration in LLM Reasoning with Count-based Intrinsic Rewards
Authors: Xuan Zhang
Ruixiao Li
Zhijian Zhou
Long Li
Yulei Qin
Ke Li
Xing Sun
Xiaoyu Tan
Chao Qu
Yuan Qi
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Models
Reinforcement Learning
Natural Language Reasoning
Primary area: foundation or frontier models, including LLMs

Abstract: Reinforcement Learning (RL) has become a compelling way to strengthen the multi step reasoning ability of Large Language Models (LLMs). However, prevalent RL paradigms still lean on sparse outcome-based rewards and limited exploration, which often drives LLMs toward repetitive and suboptimal reasoning patterns. In this paper, we study the central question of how to design exploration for LLM reasoning and introduce MERCI (Motivating Exploration in LLM Reasoning with Count-based Intrinsic Rewards), a novel RL algorithm that augments policy optimization with a principled intrinsic reward. Building on the idea of count-based exploration, MERCI leverages a lightweight Coin Flipping Network (CFN) to estimate the pseudo count and further epistemic uncertainty over reasoning trajectories, and converts them into an intrinsic reward that values novelty while preserving the learning signal from task rewards. We integrate MERCI into some advanced RL frameworks like Group Relative Policy Optimization (GRPO). Experiments on complex reasoning benchmarks demonstrate that MERCI encourages richer and more varied chains of thought, significantly improves performance over strong baselines, and helps the policy escape local routines to discover better solutions. It indicates that our targeted intrinsic motivation can make exploration reliable for language model reasoning.

Published as a conference paper at ICLR 2026
MOTIVATING EXPLORATION IN LLM REASONING WITH COUNT-BASED INTRINSIC REWARDS
COUNT COUNTS:
Xuan Zhang Ruixiao Li Zhijian 12.3 Long Li', Yulei Qin", Ke Li", Xing Sun Xiaoyu Tan Chao Qu Yuan Qi Fudan University, "Shanghai Innovation Institute, Teeneent Youtu Lab
i CCorresponding authors
xuanzhang24 48m. fudan. edu. en, quchaol fudan. edu. en
ABSTRACT
Reinforcement Learning (RL) has become compelling way to strengthen the multi step reasoning ability of Large Language Models (LLMs). However, preva- lent RL paradigms still lean sparse outcome-based rewards and limited ex- ploration, which often drives LLMs toward repetitive and suboptimal reasoning patterns. In this paper, we study the central question of how design explo- ration for LLM reasoning and introduce MERCI (Motivating Exploration in LLM Reasoning with Count-based Intrinsic Rewards), novel RL algorithm that aug- ments policy optimization with principled intrinsic reward. Building on the idea of count-based exploration, MERCI leverages a lightweight Coin Flipping Network (CFN) to estimate the pseudo count and further epistemic uncertainty over reasoning trajectories, and converts them into an intrinsic reward that val- ues novelty while preserving the learning signal from task rewards. We integrate MERCI into some advanced RL frameworks like Group Relative Policy Opti- mization (GRPO). Experiments on complex reasoning benchmarks demonstrate that MERCI encourages richer and more varied chains of thought, significantly improves performance routines to discover better solutions. It indicates that our targeted intrinsic moti- vation can make exploration reliable for language model reasoning.
strong baselines, and helps the policy escape local
INTRODUCTION
Reinforcement learning (RL) (Sutton & Barto 2018) has become cornerstone of advancing the multi-step reasoning capabilities Large Language Models (LLMs), enabling them to tackle com- plex domains like competitive mathematics and code generation (Jacch et al. 2024 Guo et al. 2025 MAA 2025). However, these tasks feature sparse rewards, with feedback available only after com- pleting lengthy reasoning chain, making exploration a critical challenge. Recent breakthroughs, such as Group Relative Policy Optimization (GRPO) (Shao et al.] 2024) and Dynamic s sAmpling Policy Optimization (DAPO) (Yu al. 2025), have streamlined the training process by eliminating the need for an explicit value function. This yields local variability at the token level, but it does not produce exploration that coherent across the length of reasoning trajectory. To guide exploration in such frameworks, many prevalent techniques rely on entropy regularization to encourage local policy diversity. While effective, this approach limited for complex, long-horizon tasks. We see an opportunity to design complementary strategies that provide more directed, lemporally-consister exploration signals particularly for those tasks, motivating our investigation into principled explo- ration strategies compatible with modern value-free RL.
The exploration-exploittiii trade-off is a classic challenge in RL (Jin et al. 2018 Azar et al. 2017). Simple approaches such as e-greedy (Mnih et al. 2015) or Boltzmann exploration with entropy- based regularization (Mnih et al. 2016). inject undirected noise to encourage stochasticity (Osband et al. 2016a). While these "shallow" exploration methods visit all states theoretically, they can be exponentially inefficient in simple yet illustrative examples (Osband et 2016b) Kakade (003). In notoriously difficult exploration tasks like the video game Moniezuma's 's Revenge, these methods fail

---

Published as a conference paper at ICLR 2026
because the chance of discovering long, precise action sequences needed for reward is vanishingly small. In contrast, "deep exploration" " strategies are both theoretically and empirically superior in such scenarios. These methods follow the principle of "optimism in the face of uncertainty" (Kearns) & Singh 2002; Brafman & Tennenholtz] 2002 O' D'Doooguu et al. 2017). encouraging the agent to explore regions of the state-action space where its knowledge is limited. This is often implemented by generating an intrinsic reward to densify the sparse signal from the environment. Canonical examples include pseudo-counts (Bellemare et al. [016 Ostrovski et al., 20177) Bootstrapped DQN (Osband et al. 2016a), Random Network Distillation (RND) (Burda et al., 2019), the intrinsic cu- riosity module (ICM) (Pathak et al. 2017). and methods based on the Uncertainty Bellman Equation (UBE) (O Donoghue et al. 2017).
Although desirable, existing methods for estimating epistemic uncertainty (Mannor et al. 2007 do not scale to modern L.MSs. Deep Ensembles (Osband et al. [016aa] Lakshminarayaaan et al. 2017
which train multiple independent models, are prohibitively expensive. Monte Carlo dropout (Gal & Ghahramani 2016), though cheaper, still adds significant inference overhead. Other methods face architectural or theoretical hurdles: pseudo-count techniques (Bellemare et al.) 2016 Ostro- vski et al., 2017) depend on normalized probability densities and preclude efficient batching, while prriosity-driveer methods (Burda et al., | et al.) (017) lack theoretical guarantees on how the exploration bonus should decay. The UBE framework (O OO'oonghe et al. 2017), while princi- pled, relies on estimating local uncertainty, a notoriously difficult task often relegated to heuristics. This fundamental mismatch between classic uncertainty quantification and the scale of LLMs ne- cessitates a novel approach.
Our work is founded on a critical insight applicable to broad class of LLM reasoning lasks-spec those that are self-contained, such as mathematical problem-solving, where the model operates without an external, stochastic world. In this context of autoregressive genera- tion, the underlying Markov Decision Process (MDP) has known and deterministic transitions. When an L.LM a state s (the token sequence generated so far) selects an action a (the next token), the subsequent state s' = (s, a) is determined without ambiguity. This property dramatically simpli- fies the Uncertainty Bellman Equation, which propagates uncertainty from two sources: the reward function estimate (f) and the transition function estimate (P). With known transitions, the epistemic uncertainty of P is zero. The UBE thus reduces to simple accumulation of local reward uncertainty along trajectory. This reframes the intractable problem of estimating Q-value uncertainty into the more manageable one of estimating local reward uncertainty. To make this tangible, we propose to proxy this uncertainty using a measure of state novelty-a practical and effective approach in sparse-reward settings. To this end, we employ the "Flipping Coins" method (Lobel et al.) 2023). a computationally lightweight and theoretically grounded pseudo-coue countinn technique that provides a scalable estimator for this purpose. We formalize this entire approach in our proposed algorithm, MERCI (Motivating Exploration in LLM Reasoning with Count based Intrinsic Rewards). Our code is publicly available at https::///// //g: thub. com od 000888887/MRRCC /MERCT]
To our knowledge, this is the first work to derive and apply a deep exploration algorithm for LLM reasoning directly from a principled simplification of the UBE. By recognizing that the LLM serves as its own perfectly known world model, we bridge the gap between model-aware RL theory and the typically model-free application of RL to LLMs. Our method integrates this simplified UBE framework with the "Flipping Coins" pseudo-count module to generate an intrinsic reward. This reward, expressed
exploration bonus, guides policy optimization algorithms like GRPO to explore novel reasoning trajectories based on a coherent, uncertainty. Experiments on complex reasoning benchmarks demonstrate that this approach signifi- cantly improves performance, effectively mitigating the tendency of standard algorithms to converge on repetitive and suboptimal solutions. Our main contributions are summarized follows:
signal of epistemic
1. A Novel Theoretical Framework for LLM Exploration. We establish a new framework based on a key insight: the LLM's known and deterministic transition dynamics simplify the Uncertainty Bellman Equation. This renders principled, uncertainty-drinn exploration tractable at scale by reducing the intractable problem of Q-value uncertainty to a manage- able estimation of local reward uncertainty.
2. A Practical and Scalable Exploration Algorithm. We propose MERCI, a novel algo- rithm that operationalizes our theoretical framework. MERCI employs a highly scalable
2

---

Published as a conference paper at ICLR 2026
counting method to translate state novelty into a potent intrinsic reward signal, designed for seamless integration with modern, value-free policy optimization methods like GRPO.
3. State-of-the-ANe Performance on Complex Reasoning. Our extensive empirical evidence challenging reasoning benchmarks, including MATH and SQL generation, demonstrate that MERCI beats strong baselines. Its directed exploration mechanism mitigates prema- ture convergence and leads to the discovery of more robust and accurate solutions.
2 PRELIMINARIES
Coin Flip Network (CFN) CFN is a computationally efficient method of count-based exploration, which estimates a state's 's visitation count by solving a simple regression problem. The core idea is that state's visitation count be estimated by leveraging the statistical properties of the Rademacher distribution (i.e., random coin flips) (Lobel et al. 2023). The method works by set- ting up a supervised learning problem where neural network p. Le., the CFN, is trained to predict the average of random coin-flip vectors associated with each state it encounters.
For every visit state 8i; new random vector yi (i.e., the coin flips) is sampled from {-1, 1)d. The CFN fo is learned by solving arg min (a..s))DDDD (8(, y:)], where L is the mean-square error
& loss function and Defn is a dataset of state-label pairs. Considering the fair coin-flip distribution C over outcomes {-1,
flip this coin n times and average the results into zn Specifically, the second moment of the sample mean 2n is related to the inverse count: M2(2,, = B| =
E, i) 1² 1. E[ the variance of the sample mean of the coin-flip -lip distribution.
x Furthermore, by flipping coins each time, the variance of z can be reduced by a factor of 3. which implies reliable way for estimating the inverse count. To this end, we generate d-dimensional random vector C ~ (-1,1) 1 as label y, for state s;. The learning objective is described as:
(+) = arg min (=..)))D. ((s; v:)] = arg min
Σ
||c. l.
(1)
In the dataset Dofn. each occurrence of the same state will be paired with a different random vector. f cannot learn perfect mapping from states to labels since there are more than one (i.e., m) instances of the same state s;. Thus, it instead minimizes L by outputting the mean random vector for all instances of a given state: (x()) = Ei= C.. The pseudo-count can be estimated by:
d
[1f((s)) 1
Σ E
E
d j=1
3=1
By training fo on the objective described in Equation] T by: (s) = Nia) where (s) denote the counts of state s.
we can map states to approximate the count
Group Relative Policy Optimization (GRPO) GRPO (Shao et al., 2024) discards the value net- work in PPO (Schulman al. 2017) by calculating the advantage of each reasoning step against the value of the entire completed sequence. For each question q and its ground-truth answer GRPO samples a group of outputs { (R [ and then computes the normalized reward in each group as the estimated advantage:
C from the old policy Th.. with corresponding outcome rewards
T: mcan ((R))
110 if is equivalent oz),
A
where R,
(3)
std (()))
0.0 otherwise.
Adding KL penalty term the clipped objective in PPO, the objective of GRPO is expressed as:
losl
JaRPO(() E Ey 10:1 maw Go (111)
min (ri(0)A) -
7
1=1
(4)
-9 Dr.. [NO mnd)) )
To oc.ce)
where r)(
T (ox, (q, Os,c1)
3

---

Published as a conference paper at ICLR 2026
State Encoding
contrine lovel
A B
the final hidden layer
A B F
C
Query q
Trajectory y
D
Hidden expression Snidden
ABCDEF
Epistemic Uncertainty (MERCT)
Coin Flipping
a Head Tg (60
(a()
Uncertainty U (Fridden)
Shidden
1n G0of
Trajectory & Reward
Uncertainty. Guided Intrinsic Rewards to motivate exploration
Query q
Trajectory y
Reward
Update Policy
Figure I: Overview of the MERCI framework. Two separate networks are used: policy network trained with RL, and CFN network that provides an intrinsic reward. The CFN network. initialized from the same SFT checkpoint To. estimates state novelty to guide the exploration of ma.
Decouple Clip and Dynamic sAmpling Policy Optimization (DAPO) Building on GRPO, DAPO (Yu et al. 2025) removes the KL penalty, introduces clip-higher strategy and dynamic sampling, applies token-level policy gradient loss, and adopts overlong reward shaping.
METHODOLOGGY
In this section, we first establish the theoretical foundation for our approach by simplifying the Uncertainty Bellman Equation for the specific case of LLMs, and then introduce the full details of our novel algorithm, MERCI.
3.1 THE UNCERTAINTY BELLMAN EQUATION WITH KNOWN TRANSITIONS
The Uncertainty Bellman Equation (UBE) provides principled mechanism for propagating epis- temic uncertainty quantified the variance of the posterior distribution over Q-values-througe through time (O'Donoghue al.) 2017). For clarity, we will use the terms "uncertainty" and "variance" interchangeably throughout this section. Our core theoretical contribution stems from a key insight: the Markoy Decision Process (MDP) underlying LLM reasoning has a known and determin- istic transition function, P. This property dramatically simplifies the general form of the UBE, leading to a more direct and tractable equation for uncertainty propagation.
Formally, consider finite horizon, finite state and action space MDP, with horizon length H e N, state space S, action space A and rewards at time period denoted by p E R. A policy (mr ..... =²) is sequence of functions where each n²h S A + R4 is mapping from state-action pair to the probability taking action state time-step h and Σ. mh for all E S. At each time-step h the agent receives state s and reward r and selects action a² from the policy Th and the agent moves to the next state A+1 which is sampled with probability ph The Q-value, at time step h of particular state under policy π is the expected total return from taking that action at that state and following thereafter, i.c., (=) E
taking that action at that state, i.e., Th is the probability of
!1 D" | s, =
We adopt a Bayesian perspective as that (O' DDnogguu et al. 2017). We assume a prior over the mean reward function, r(s), and collect history of interactions Fi (states, actions, and rewards up to episode f) generated by a policy т. This history is used to form a posterior distribution over the mean rewards, which we denote dr| If we draw reward function estimate & 6+|,, the corresponding Q-function estimate, Q",, must satisfy the posterior Bellman equation for that sample:
( (*((s) Σ m/
a Ph pla 1 1Q (6' aa))),
(5)
4

---

Published as a conference paper at ICLR 2026
for all timesteps h = ,,..., H, with QT.
Since the transition function P for an LLM is a known delta function (i.c., for a given state a and action a, the next state s' (s, a) ). we have P rather than its posterior P in equation 5 This leads to recursive equation for the variance of the Q-value posterior, stated in the following proposition. In the following discussions, we may use the word uncertainty and variance (w.r.t. the posterios distribution) interchangeably. we denote et as the variance of random variable æ conditioned on the history Fr, which is E ( (x=
Proposition 1 (Uncertainty Bellman Equation for Known Transitions) Let (8,
A
V. a)] be the posterior variance of the Q-value at step h, conditioned on the history Ft. Given known and deterministic transition function, this uncertainty propagates according to the following Bellman equation:
("(s,a) S (.[+"(s)]+
+
Σ Tyy U st.a!
U C
where s' is the unique next state reached from (s, a), and U (+++(()) 0.
The proof follows from the analysis in O O'Donoghue et al. (017 by applying the law total vari- ance equation [ This result provides powerful recursive formula: the uncertainty of a state- action pair bounded by the immediate reward uncertainty plus the expected uncertainty of the unique subsequent state, where the expectation is over the policy's 's next actions. This reframes the complex problem of estimating Q-value variance into the more manageable task of estimating the local reward uncertainty, Ve (s)). The resulting Q-value variance, (s(s,, a), be used to define an exploration bonus inspired by Upper Confidence Bound (UCB) algorithms (Lattimore & Szepesvári 2020). Specifically, the policy can be encouraged to explore by modifying the optimiza- tion objective to - (s, a) + ay ( a), where a hyperparameter balancing exploitation and exploration. This approach is backed by strong theoretical guarantees for achieving low regret (Auer al. 2008 Jin et al. 2018).
From standard concentration inequalities, we know that the uncertainty over a mean reward estimate inversely proportional to the number of times that state has been visited, i.e., V.(r* (s) x (s) However, in the high-dimensional state space of language, exact state visitations are exceedingly rare. This necessitates a method to generalize counting to unseen but similar states. In the fol- lowing section, we describe how uncertainty.
scalable pseudo-count mechanism to estimate this local
3.2 ESTIMATE VARIANCE OF REWARD VIA CFN
Standard policy optimization driven by sparse, outcome-based rewards (e.g., GRPO) can lead to premature convergence on suboptimal solutions. MERCI addresses it via dedicated mechanism for principled exploration. The framework is illustrated in Figure
Our framework employs two distinct Large Language Models operating in parallel:
1. The Policy Network (mo): This is the agent that generates reasoning trajectories. It is initialized from a supervised fine-tuned (SFT) checkpoint, To. and its parameters are exclusively updated by the policy optimization algorithm (c.g.. GRPO).
2. The CFN Network: This network's s sole purpose is to estimate epistemic uncertainty. It is a separate instance of the LLM, also initialized from the same checkpoint To. A lightweight MLP, which call the CFN head (fo). is attached to its final hidden layer. CFN network is updated together via a supervised regression objective (detailed Scction 2).
The training process integrates these two networks as follows. During training step, reasoning trajectory 7 is first generated by the current policy network Tg. In the sequential decision-making process, define the state each step the contextual hidden representation Stadden output by the LLM backbone at that token position, which inherently captures the entire prefix of the generated sequence. The state Shidden then processed by the CFN head (s() to estimate the variance of the reward, computed by V[f(s) [
5

---

Published as a conference paper at ICLR 2026
Percentile Filtering
Spatial Coherence Filtering
Noise - suppression Filtering
rettins only fleed fraction of strongest signls comple
Nesze teken that bebng cort suaur "egien
PeTeNEs Iocetiiee atteched to costent that seivin pratlen
Mo Fittr Appled
Altar Perceedl Ribriig
Aftir Spatia Cohfrance Fhiring
Mer Saita sappessiioonn Fibeing
2 tobes
Tlacry
Creik Thirry
The 4.g
(how
Figure 2: The entire pipeline of bonus filtering. Step 1: We rank all tokens within a response by their associated bonus values and retain only those falling within a predefined top percentile (e.g., the top 50% this figure). Step 2: We only preserve clusters of adjacent tokens that consistently exhibit elevated bonuses (e.g., 3 consecutive tokens in this figure). Step 3: For example, in math reasoning task without external tools, any Python code potentially generated during LLM rollouts is semantically irrelevant and noisy, so we exclude them from the overall bonus calculation.
3.3 ADVANTAGE ESTIMATION
Calculating the Intrinsic Reward from Cumulative Uncertainty A critical detail of our method, derived directly from Proposition [ the precise calculation of the exploration bonus. The cor- rect approach to determine the uncertainty of a trajectory's 's value is to first sum the local reward variances at each step (we use the monte carlo estimation here), and only then take the square root of the total sum. This resulting value represents the standard deviation of the cumulative Q-value posterior and serves as our intrinsic reward.
This stands in stark contrast to a common but theoretically flawed heuristic in many RL explo- ration algorithms. Those methods often compute a per-step bonus proportional to the local stan- dard deviation and apply standard RL algorithm to the modified, "bonused" rewards. As demon- strated by O D'nogguee et al. (2017). this latter upproach-whiee is equivalent to summing standard deviations leads to a significant overestimation of uncertainty over long horizons. This miscalcula- tion can cause the agent to become overly optimistic, leading to prolonged and inefficient exploration of paths that are long but not necessarily promising. To illustrate the difference, consider trajectory of horizon H where the local reward variance at each step is o² 1. Correct Bonus (MERCI): The cumulative variance is Σ 1 H. The bonus, or standard deviation, is correctly calculated as VH. Heuristic Bonus: The per-step bonus is vi = 1. Summing these bonuses results in an overes- timated total bonus of Σı= 1 = H. MERCI adheres strictly to the former, pooretically-groundde calculation, ensuring the exploration signal accurately reflects the true cumulative epistemic uncer- tainty. Indeed, we compare this two calculation in our ablation study G.2.4
Budget-Aware Exploration Bonus Control The non-sparse exploration bonus introduces its own considcrable instabilities when becoming indiscriminately dense, which would invite LLMs seeking through aimless exploration. So, we enforce budgeted exploration, which reduces gradient variance and in turn stabilizes optimization and lowers noise in final answers. Concretely, three filtering stages are applied, shaping where and how the bonus can act. (1) Percentile filtering retains only a fixed fraction of the strongest signals within each sample, which tracks the gradual decline in bonus magnitude over training without manual retuning. (2) Spatial coherence filtering keeps tokens that belong to contiguous regions of elevated bonus and discards isolated spikes when numerically large, thereby yielding steadier updates.
(3) Noise-suppression filtering removes incentives at- tached to content that is unrelated to solving the problem, such as meaningless repetition, gratuitous

---

Published as a conference paper at ICLR 2026
code blocks, or rare characters generated solely chase the bonus. Together these stages allocate a controlled exploration budget that preserves useful exploration while safeguarding the primary reward signal. The overall pipeline of bonus control is illustrated in Figure 2
Advantage Normalization and Bonus Integration After bonus filtering, the normalized bonus B computed by first averaging the squared CFN outputs across all retained tokens and then applying square-root compression:
Σ EEI
t
B
(6)
where is the length of a trajectory, d is the dimension of CFN's outputs, and I is the set of retained tokens' indices.
To ensure comparability across trajectories sampled under the same prompt, we standardize sanjectory-aggregatee bonuses within each group of size and truncate negative values, preserv- ing only positive exploratory incentives:
A exxploration = max (0,
B, 1
G Σ Bj, j=l
G (B, 2)2. (7)
where
o=
Σ j=1
G
G
To prevent the bonus from overpowering outcome-basse -basod rewards, scale the standardized intrinsic bonus term by an exploration coefficient 7, and add it to the base advantage A For trajectories
whose base advantage is negative, we cap the augmented advantage with clipping factor a E (0, to prevent the intrinsic term from overwhelming the outcome signal:
min
A + 7AAxxporrtioonn + (1 + a)) A)
if A' 2 0;
A
(8)
min old + 7 exploration (1 - at) A old
if A < O.
We give an algorithmic description in Algorithm D in Appendix C
4
RELATED WORK
4.1 REINFORCEMENT LEARNING FOR LLM REASONING
Reinforcement learning (RL) (Sutton & Barto 2018 particularly Reinforcement Learning with Verifiable Rewards (RLVR), has been widely used to improve the reasoning abilities of large lan- guage models (LLMs). PPO a foundational policy gradient method, which ensures stable policy updates via clipped objectives, proving effective in reasoning tasks (Schulman et al. (017). It treats token positions in reasoning trajectories of LLM as distinct states for advantage estimation, but this approach comes at the cost of computational overhead from its joint policy-value optimization. Starting from PPO, recent efforts have developed some efficient and advanced frameworks such as GRPO (Shao al. 2024). By evaluating and normalizing rewards across group of entire gener- ated sequences, GRPO provides a more robust and efficient method for advantage estimation. This method of relative, sequence-level comparison sidesteps the complexities of token-level advantage estimation, proving far more effective for multi-step reasoning. The success of this holistic approach highlighted by its adoption and extension in subsequent research, such as DAPO (Yu al. 2025),
VAPO (Yue et al.] (2025) and Dr. GRPO (Liu et al.] (025). However, even advanced RL methods for LLMs face critical bottleneck: their dependence on external static and sparse reward struc- tures limits effective exploration. To overcome this, we integrate count-based intrinsic motivation into GRPO-like frameworks, incentivizing the model to explore more novel and diverse reasoning trajectories guided by epistemic uncertainty during training.
4.2 EXPL.ORATION IN REINFORCEMENT LEARNING
Effective exploration in RL is critical for navigating the fundamental dilemma between exploit- ing known rewards and exploring uncertain options to discover better policies. Some traditional
7

---

Published as a conference paper at ICLR 2026
exploration methods like RND (Burda et al. 2019). ICM Pathak et al. 2017), and Count-Based Exploration (Ostrovski et al. 2017 Fang et al. (2077)), encourage agents to explore novel or under- visited states via intrinsic rewards. However, their application LLMs faces significant challenges: the dynamic response length and vast action space. Most approaches for LLMs rely on undirected exploration, such simply encouraging exploration from an entropy perspective (Wen al. 2024 Wang et al. 2025 Cheng et al., 2025). These heuristic approaches often lack solid theoretical foun- dation guide policy models identify which states warrant greater exploration, leading subop- timal policies. To address these limitations, recent work has developed active exploration strategies
to estimate uncertainty from historical data and plan optimistically (Zhang et al.
2024,
Bai et al.)
2025 Cen et al. Chen et al. 2025 Gao et al., [025 Zhang et al.
2025 Dai et al.
2025
However,
curiosity-driven methods (Bougie & Watanabe| 2025 Gao et al.
2025 Dai et al. 2025)
lack theo-
retical guarantees on how the exploration bonus should decay, and the classical density-based meth-
ods for calculating pseudo-counts (Ostrovski et al. 2017] Bai et al. 2021) are resource-intensse time-consuming, and hard to fulfiil. Some methods (Tang et al. 201 7b Rashid et al. 2019 Lobel et al. 2023) instead explored alternatives to eliminate the usage of density models. In our work, we formally show that the deterministic nature of LLM transitions simplifies the general Uncertainty Bellman Equation to tractable form, providing the principled justification for how to aggregate the local pseudo-counts into sum-of-variance trajectory bonus, thereby distinguishing our method from purely heuristic exploration techniques. We take CFN (Lobel et al., (023) as our theoretical foundation for estimating the pseudo-count, introducing simple supervised learning objective to estimate a visitation count and further integrates intrinsic motivation.
5 EXPERIMENTS
To validate our hypothesis that encouraging novelty via MERCI promotes the policy's ability to discover more optimal solutions, we conduct comprehensive set of experiments two types of benchmarks: mathematical reasoning and SQL generation, and further provide in-depth analyses.
5.1 EXPERIMENTAL SETUP
Mathematical Reasoning Our backbone model is Qwen2.5-Math-77 (Yang et al. 2024). Our
training dataset is sourced from DAPO-17K (Yu et al. 2025), and we evaluate models on diverse set of challenging mathematical reasoning benchmarks, including AIME2024/2025 (MAA 2025) MATH500 (Hendrycks et al. (022)) OlympiadBench (He et al. 2024), College Math (Tang et al. and Minerva (Lewkowycz et al. 2022).
2024..
SQL Generation Our experiments are conducted on Llama-3.1-88 1-88-Innttruc (Grattafiori et al. 2024 We trained the Bird training set (Li al. 2023) and evaluated performance the Bird and Spider test sets (Yu et al. 2019).
Baselines and Configurations We conduct RL training experiments on both vanilla GRPO and DAPO using the veRL framework (Sheng et al. 2025). We additionally introduce two algorithms designed to encourage exploration as baselines: one uses entropy- advantage shaping et al. 2025), and the other incorporates intrinsic rewards via RND training (Gao al., 2025 ). In our experimental results, we refer to them as Entropy Adv. and iMentor, respectively. For the implementation of CFN, set the dimensionality d, which be intuitively interpreted how many times we have flipped coin, to 20. Detailed hyperparameters are presented Appendix E
5.2
MAIN RESULTS
5.2.
COIN FLIP NETWORK
To evaluate the effectiveness of the CFN, i.e. our exploration model, and enhance exploration ef- ficiency during RL training, we first generate responses from the backbone model on the training dataset and use these responses to perform a preliminary training of the CFN. This process enables it to develop basic understanding of which states are likely to occur rarely.
8

---

Published as a conference paper at ICLR 2026
Table 1: Performance mathematical reasoning benchmarks with pass@k and mean@k. The highlighted color represents the best within RL models, while underlined represents the second best.
(a) pass k results
AIME25
ATME24
Minerva
MATH500
OlympiadRench pass@16 /6
College pass
Avg.
pass 256
Pass
pass
pass
Owen2.5.Math
53.3
70.0
50.4
88.6
56.7
44.2
60.5
GRPO
53.3
76.7
64.U
91.8
59.7
49.2
65.8
GRPO w/ Emtropy Adv.
56.7
76.7
62.5
91.2
59.4
48,9
65,9
iMentor
GRPP MERCI (ours)
f
DAPO
56.7
76.7
66.9
92.0
60.9
48.3
66.9
DAPO w/ Entropy Adv,
60.0
48.5
67.9
iMentor
56.7
50.
-AAOO MERCI (ours)
60.0
83.3
66.5
62.1
50.2
69.0
(b) mean @ k results
AIME25
AIME24
Minerva meon 16
MATH500
OlympiadBench menon @ 16
College
Avg.
meron @ 2,56
meon @ 256
meGn @ 16
meam
@wen2.5-Math
64.6
GRRO
11.2
28.7
41.8
79.0
40.3
42.0
40.5
GRPO w/ Eatropy Adv.
12.1
28.9
42.0
81.0
40.6
42.6
41.2
GRPO w/ iMentor
11.9
29.0
42.2
78.9
40.7
42.4
40,9
GRPO w/ MERCI (ours)
13.4
29.6
44.1 7
80.7
42.6
42,9
42.2 1
DAPO
16.5
31.9
41.0
81.5
41.4
41.0
42.2
DAPO w/ Entropy Adv.
17.2
33.3
44.5
80.9
41.4
41.6
43.2
DAPO w/ iMentor
17.4
32.0
46.7
82.3
42.8
43.3
44.1
DAPO w/ MERCI (ours)
18.4 1
35.2 1
44.8 7
82.4
44.3
44.2
44.9 1
For this pretrained CFN, conduct two evaluations: (1) Within a single response, visualize the estimated uncertainty assigned by the CFN to each token position; (2) For all collected responses, apply the method described in Section 3.3 to select the top 30% of tokens with the highest bonus in each response, filter them accordingly, and then perform statistical analysis on the retained token sequences. The results of Experiment (1) and (2) are presented in Figure 3 5 and Figure in Appendix G respectively. We can observe that token sequences assigned higher uncertainty by the CFN predominantly correspond to novel reasoning paths, Python code along with its outputs, or specialized mathematical terminologies. This observation aligns with our hypothesis that more novel token positions tend induce higher epistemic uncertainty and are therefore assigned higher values by our CFN.
In addition, the CFN exhibits three further important findings: (1) when directly applied esti- mate the uncertainty of responses in the SQL Generation task, the CFN trained on mathematical reasoning tasks produces estimates that align well with our intuition and analysis, which indicates the generalization ability of CFN; (2) for reasoning trajectories that are linguistically close but not identical, the CFN successfully captures their semantic similarity, yielding correspondingly similar uncertainty estimates; (3) our CFN bonus provides a non-redundant signal and effectively measures the policy's epistemic uncertainty (i.e., lack of knowledge). The detailed results are shown in Figure
10 and Figure [ in Appendix G.1
5.2.2 RL TRAINING
The CFN in the RL phase is initialized using the pretrained CFN and is then co-trained with the policy model during RL training.
Our primary results for RL training are summarized Table and Table 2 As shown Table [ MERCI delivers consistent gains over both vanilla GRPO and DAPO across mathematical reason- ing benchmarks when measured by pass@ @k and mean@k Gains are most pronounced on the AIME suites, which is the most challenging, and remain robust on the other datasets. Consistently higher mean @ k suggest better overall sample quality with uniform and stable gains. addition, MERCI also yields improvements in pass @k, pointing to enhanced exploration and calibration rather than

---

Published as a conference paper at ICLR 2026
Table 2: Performance on SQL generation benchmarks with greedy sampling and pass
Bird (in domain)
Spider (out of domain)
Model
Greedy
Pasc 16
Greedy
Pas
Pa 16
Llama-3 3.1-BB Instruct
42.4
68.5
75.1
69.0
91.0
94.6
+ GRPO
60.7
72.2
74.6
74.7
81.0
82.9
GRPO w/ Entropy Adv.
60.8
72.1
73.9
74.7
83.2
84.5
+
GRPO w/ iMentor
62.8
72.3
74.2
75.0
84.1
85.2
GRPO w/ MERCI (ours)
63.0
72.8
74.9
78.0
85.61
+ DAPO
63.2
73.9
75.9
76.8
86.1
87.2
DAPO w/ Entropy Adv.
62.3
73.2
75.9
77.5
86.
87.6
+ DAPO w/ iMentor
62.7
73.9
76.1
77.2
86.4
88.2
DAPO MERCI (ours)
64.1
77.3
narrow best-case gains. As shown in Table SQL generation results on Bird and Spider also mirror the earlier findings. Especially, MERCI yields larger out-of-domain gains, i.e., the Spider test set. It indicates that MERCI effectively pushes LLMs to use general SQL patterns that transfer better to different schemas. Additionally, the cross-domain experiments in Appendix [.2. indicate that MERCI play an important role in improving out-of-domain robustness, even when the under- evidenced by the training dynamics in Figure 13 in Appendix and our case study in Appendix (.2.6] we further observe that MERCI enhances exploration and calibration by densifying multiple valid reasoning trajectories while discouraging gratuitous chain elongation. It concentrates probability mass more diverse yet more reliable good solutions that are expressed in shorter, more focused traces, raising the floor of candidate quality. Besides, our case study also indicates an increased proportion of steps devoted to higher-level rea- soning abilities. This shift from length-based search to concise, well-calibrated reasoning improves sample efficiency and reduces eттоr correlation. It learns prune mask-irrelevant branches and con- centrate computation on promising hypotheses, yielding more intelligent and efficient exploration.
lying training data is highly domain-spedii somaai--peecilized...
5.3 ABLATION STUDIES AND SCALING EXPERIMENTS
We conducted these experiments on the mathematical reasoning task and vanilla GRPO. The detailed experimental results are presented in Appendix G.2.2 and Appendix G.2.4 From these results, we first confirm that crucial components, including bonus filtering and our normalized trajectory- aggregated uncertainty estimation, are fundamental to the method's success. Furthermore, the results reveal MERCI's superior exploratory efficiency: efficiently and yields strong pass performance, but also demonstrates remarkable stability over the long term in scaling experiments. Finally, sensitivity analysis on the key hyperparameter choices, e.g., the ry cosine schedule and Top-p% % in bonus filtering, are included in Appendix G.2.5
algorithm not only identifies good solutions
6 CONCLUSION
In this study we introduced MERCI, a principled exploration strategy for LLM reasoning that har- nesses the deterministic transitions of language trajectories. By reframing the Uncertainty Bellman Equation under known transitions we replaced expensive Q variance estimation with tractable count based proxy for reward uncertainty. The result is an intrinsic signal that guides Group Rela- tive Policy Optimization and its variants toward diverse and coherent reasoning paths. Experiments on challenging mathematics and SQL benchmarks reveal consistent gains in pass rates and in mean score, verifying that our method steers policies away from shallow entropy driven randomness and toward productive inquiry. The Coin Flip Network delivers this benefit with minimal compute over- head and can be trained in parallel with the policy model, which makes the approach attractive for large scale systems. Experiments on mathematical reasoning and SQL generation show stable training dynamics, diverse reasoning paths, accurate solutions, and robust outcomes at scale.
10

---

Published as a conference paper at ICLR 2026
7 ACKNOWLEDGEMENT
This work was supported by the National Natural Science Foundation of China (Grant No. 82394432, and 92249302), and the Shanghai Municipal Science and Technology Major Project (Grant No. 2023SHZDZX022.
ETHICS STATEMENT
This work adheres to the ICLR Code of Ethics. this study, no human subjects or animal ex- perimentation was involved. All datasets used were sourced in compliance with relevant usage guidelines, ensuring no violation of privacy. We have taken care to avoid any biases or discrimi- natory outcomes in research process. No personally identifiable information experiments were conducted that could raise privacy security concemn. We are committed to maintaining transparency and integrity throughout the research process.
used, and no
REPRODUCIBILITY STATEMENT
We have made every effort to ensure that the results presented in this paper reproducible. All models and datasets used in our work are publicly available, and the code is openly available at: / com/ add88s87) /MERCI The experimental setup, including training steps, model configurationss and hardware details, is described in detail in the appendix. We have also provided full description assist others in reproducing our experiments.
REFERENCES
Peter Auer, Thomas Jaksch, and Ronald Ortner. Near-optimal regret bounds for reinforcement learning. Advances neural information processing systems, 21, 2008.
Mohammad Gheshlaghi Azar, Ian Osband, and Rémi Munos. Minimax regret bounds for reinforce- ment learning. In International conference on machine learning, pp. 363-272. PMLR, 2017.
Chenjia Bai, Lingxiao Wang, Lei Han, Jianye Hao, Animesh Garg, Peng Liu, and Zhaoran Wang- Principled exploration via optimistic bootstrapping and backward induction. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 577-587.PMLR, PMLR, 18-24 Jul 2021. URL /ttttps://rroceeeeee
ml wtw.lngg....s......e.mmm v 313 wtt.aaaiille html
Chenjia Bai, Yang Zhan, Shuang Qiu, Qiaosheng Zhang, Kang Xu, and Xuelong Li. Online prefer- ence alignment for language models via count-based exploration. In International Conference on Learning Representations, 2025.
Marc Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. Advances neural information pro- cessing systems, 29, 2016.
Nicolas Bougie and Narimasa Watanabe. Exploring Beyond Curiosity Rewards: Language-driven exploration in RI. In Vu Nguyen and Hsuan-Tien Lin (eds.), Proceedings of the 16th Asian Conference on Machine Learning, volume 260 of Proceedings of Machine Learning Research, pp. 127-1422 PMLR, 05-08 Dec 2025. URL wtwps:ttp///ccc httpt/ss///poccceiiiiieo mlr .wt.mprsss//26// bougie25a.htm html
Ronen Brafman and Moshe Tennenholtz. R-max-a general polynomial time algorithm for near- optimal reinforcement learning. Journal of Machine Learning Research, 3(0ct):213-231, 2002.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. In International Conference on Learning Representations, 2019. URL hetps:
//openrevient www.::/.wwwwww///eett.o d= 13

---

Published as a conference paper at ICLR 2026
Shicong Cen, Jincheng Mei, Katayoon Goshvadi, Hanjun Dai, Tong Yang, Sherry Yang. Dale Schu- urmans, Yuejie Chi, and Bo Dai. Value- incentiiie preference optimization: A unified approach to online and offline rlhf. In The Thirteenth International Conference Learning Representa-
fions.
Mingyu Chen, Yiding Chen, Wen Sun, and Xuezhou Zhang. Avoiding exp (mmax) scaling in rlhf through preference-based exploration, 2025. URL [tttpss//arriivoorg/aa/////////p
00666
Daixuan Cheng, Shaohan Huang, Xuekai Zhu, Bo Dai, Wayne Xin Zhao, Zhenliang Zhang, and Furu Wei. Reasoning with exploration: An entropy perspective, 2025. BRLhhtts:///rxxt
org/ /agg/bb//aa 2506. 14758
Rumpeng Dai, Linfeng Song. Haolin Liu, Zhenwen Liang, Dian Yu, Haitao Mi, Zhaopeng Tu, Rui Liu, Tong Zheng, Hongtu Zhu, and Dong Yu. Cde: Curiosity-driven exploration for efficient reinforcement learning large language models, 2025. URL
2509. 09675
Yarin Gal and Zoubin Ghahramani. Dropout as bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pp. 1050-1059.
PMLR, 2016.
Jingtong Gao, Ling Pan, Yejing Wang, Rui Zhong, Chi Lu, Qingpeng Cai, Peng Jiang, and Xi- angyu Zhao. Navigate the unknown: Enhancing Ilm reasoning with intrinsic motivation guided exploration. arXiv preprint arXiv:2505.176211 2025.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and et al. The Ilama herd of models. 2024. https://///ass orq. www.aos//////ooooooo//..../a..aa abs 204.bb//22777333
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang. Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseek-rl incentivizes reasoning in llms through reinforce- ment learning. Nature, 645(8081) ::633 1:663638.. 2025.
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pp- 3828-3850, 2024.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardr. Measuring mathematical problem solving with the MATH dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. URL httttttss//ppereee//// wtwp:e/ewwwe..eeeeuooeeeeeaeeeee ntt www.:eeeo/oooiiteee Byy
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint arXiv:2412.16720, 2024.
Chi Jin, Zeyuan Allen-Zhu, Sebastien Bubeck, and Michael Jordan. Is q-learning provably effi- cient? Advances in neural information processing systems, 31, 2018.
Sham Machandranath Kakade. On the sample complexity of reinforcement learning. University of London, University College London (United Kingdom), 2003.
Michael Kearns and Satinder Singh. Near-optimal reinforcement learning in polynomial time. Ma- chine learning, 49(2):209-232, 2002.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell, Simple and scalable predictive uncertainty estimation using deep ensembles. Advances neural information processing systems, 30, 2017.
Tor Lattimore and Csaba Szepesvári. Bandit algorithms. Cambridge University Press, 2020.
12

---

Published as a conference paper at ICLR 2026
Aitor Lewkowyez, Anders Johan Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Venkatesh Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Son Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. Solving quantitative rea- soning problems with language models. Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL https: battps://openrevies opw.reviewwwwweeeee net/ fforumiid IFXTZERXcM?
Jinyang Li, Binyuan Hui, Ge Qu, Binhua Li, Jiaxi Yang, Bowen Li, Bailin Wang, Bowen Qin, Ruiying Geng, Nan Huo, Xuanhe Zhou, Chenhao Ma, Guoliang Li, Kevin C. C. Chang, Fei Huang, Reynold Cheng, and Yongbin Li. Can I1m already serve as a database interface? big bench for large-scale database grounded text-to-sqls, 2023.
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang. Chao Du, Wee Sun Lee, and Min Lin. Understanding r 11-zero-like training: A critical perspective, 2025. URL URLhhttss://axxvv
org/abs/ abaaaba25555 20783
Sam Lobel, Akhil Bagaria, and George Konidaris. Flipping coins estimate pseudocounts for exploration in reinforcement learning. In International Conference on Machine Learning, 2023.
MAA. American invitational mathematics examination (aime), 2025. https://maa.ors
Shie Mannor, Duncan Simester, Peng Sun, and John N Tsitsiklis, Bias and variance approximation in value function estimates. Management Science, 53(2):308-322, 2007.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, et Human-level control through deep reinforcement learning, nature. 5 518(7540):529-533 2015.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference machine learning, pp. 1928-1937. PmLR, 2016.
Brendan O' O'Donoghue, Ian Osband, Rémi Munos, and Volodymyr Mnih. The uncertainty bell- man equation and exploration. International Conference on Machine Learning. 2017. URL
https://ap1 www.eeeeniiiooom inteanattccccchooooooo ar. wttp/oorgsu/aooooo 6201691111
Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. Advances neural information processing systems, 29, 2016a.
Ian Osband, Benjamin Van Roy, and Zheng Wen. Generalization and exploration via random- ized value functions. In International Conference on Machine Learning, pp. 2377-2386. PMLR, 2016b.
Georg Ostrovski, Marc G Bellemare, Aäron Oord, and Rémi Munos. Count-based exploration with neural density models. In International conference on machine learning, pp. 2721-27727 PMLR, 2017.
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International conference on machine learning, pp. 2778-2787. PMLR. 2017.
Tabish Rashid, Bei Peng, Wendelin Bochmer, and Shimon Whiteson. Optimistic exploration even with a pessimistic initialisation. In International Conference on Learning Representations, 2019.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017. /
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, Y.K. Li, Y. Wu. and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024. wtl.httts:///xxivcoom http://///a.....r///////0...... 03300
13

---

Published as a conference paper at ICLR 2026
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. In Proceedings of the Twentieth European Conference on Computer Systems, EuroSys 22, pp. URL http::/dx doi. 1444/
ACM,
March 2025. doi: 10.1 145/3689031
363 3696075
Richard S. Sutton and Andrew G. Barto, Reinforccmmet Learning: An Introduction. The MIT Press, second edition, 2018. URL
html
Haoran Tang. Rein Houthooft, Davis Foote, Adam Stooke, OpenAI Chen, Yan Duan, John Schul- man, Filipp DeTurck, and Picter Abbeel. #cxploration: A study of count-based exploration for deep reinforcement learning. In I. Guyon, U. Von Luxburg. S. Bengio, H. Wallach, R. Fergus, S. Vish- wanathan, and R. Garnett (eds.), Advances in Neural Information
Curran Associates, Inc., 2017a. URL In Processing Systems, volume 30. http:////prrccceecnnnnnnnrr....p ftlpe///pccooo177112222222222202
Papereddaapp... pdf
Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, OpenAI Xi Chen, Yan Duan, John Schul- man, Filip DeTurek, and Pieter Abbeel. exploration: study of count-based exploration for deep reinforcement learning. Advances in neural information processing systems, 30, 2017b.
Zhengyang Tang, Xingxing Zhang, Benyou Wang, and Furu Wei. Mathscale: Scaling instruction tuning for mathematical reasoning. In International Conference on Machine Learning, pp. 47885- 47900. PMLR, 2024.
Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang. Xionghui Chen, Jianxin Yang, Zhenru Zhang, Yuqiong Liu, An Yang. Andrew Zhao, Yang Yue, Shiji Song, Bowen Yu, Gao Huang, and Junyang Lin. Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for Ilm reasoning, 2025. URL https:://acxii.orr//aa//
1506601933
Muning Wen, Junwei Liao, Cheng Deng, Jun Wang. Weinan Zhang, and Ying Wen. Entropy- regularized token-level policy optimization for language agent reinforcement, 2024. URL.
https://arxiv: org/ abs 2200.///222020000
An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xue, Runji Lin. Tianyu Liu, Xingzhang Ren, and Zhenru Zhang. Qwen2.5- math technical report: Toward mathematical ex- pert model via self-improvemennt, 2024. URL /tttps://arxivi .rg/ abs 24009. 12122
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source IIm reinforcement learning system at scale. arXiv preprint arXiv:2503. 14476, 2025.
Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. Spider: A large-scale human- labeled dataset for complex and cross domain semantic parsing and text-to-sql task, 2019. URL
pktps://aare
org/ abs 1809.8800 08887
Yu Yue, Yufeng Yuan, Qiying Yu, Xiaochen Zuo, Ruofei Zhu, Wenyuan Xu, Jiaze Chen, Chengyi Wang, Tian Fan, Zhengyin Du, Xiangpeng Wei, Xiangyu Yu, Gaohong Liu, Juncai Liu, Lingjun Liu, Haibin Lin, Zhiqi Lin, Bole Ma. Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Ru Zhang, Xin Liu, Mingxuan Wang, Yonghui Wu, and Lin Yan. Vapo: Efficient and reliable reinforcement learning for advanced reasoning tasks, 2025. URL https://arxxx org/abs/2504 2504 05118
Kongcheng Zhang, QI YAO, Shunyu Liu, Yingjie Wang, Baisheng Lai, Jieping Ye, Mingli Song, and Dacheng Tao. Consistent paths lead to truth: Self-rewarding reinforcement learein for LLM reasoning. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. URL https:: /tttp:://operreiee. http://www..............o../o/..
Shenao Zhang, Donghan Yu, Hiteshi Sharma, Ziyi Yang, Shuohang Wang, Hany Hassan, and Zhao- ran Wang. Self-exploring language models: Active preference elicitation for online alignment. arXiv preprint arXiv:2405.193.. 2024.
14

---

Published as a conference paper at ICLR 2026
A THE USE OF LARGE LANGUAGE MODELS
We utilize an LLM to assist with paper editing and correcting grammatical errors.
B ENTROPY REGULARIZATION AS GENERALIZED e- E-GEEDY EXPLORATION
We provide a mathematical derivation showing that entropy regularization corresponds to softmax exploration strategy. This be interpreted a generalized form of e-greedy exploration that intelligently allocates exploration probability based on the relative quality suboptimal actions.
regroay-re policy improvement. Given state and advantage estimates A(s, a) for actions a E A, consider the entropy-regularized optimization:
n = arg max (11s)
Σ
(als) ((s,)) (H
where H(T) = -E T(ass) log ((ass) and > 0 is the entropy coefficient. The well-known solution is the Boltzmann/softmax distribution:
T = scp(A(s,a)/3)
EreA exp(A(s.b)/3)
Connection to e-greedy. Let a" E arg max, A(s, a) and denote the advantage gaps A, A(s,a*) A(s, a) 2 O. The probability of selecting the optimal action is
P+ ==<(a" (8) 1+ E aaza exp( (Aa13)
We can define a state- and value-dependent exploration probability (g(s) = 1 - p.. This allows us to decompose the policy
T (0)))) da - =x(())=((11))
where (s(ass) & exp(-A./8) 3) is probability distribution over the set of suboptimal actions.
This formulation reveals that softmax exploration is generalized form of e-greedy. However, unlike the standard e-greedy rule, its exploration is not uniform. The distribution q, intelligently assigns higher probability to suboptimal actions that are closer optimal (i.e., having smaller advantage gap A.). Only under the strong and often unrealistic condition that all suboptimal actions are equally bad (A, const. for f a²) does qa approach a uniform distribution, making the strategy resem- ble standard e-greedy. Thus, entropy regularization typically leads to more efficient exploration strategy than its uniform counterpart.
C ADDITION DESCRIPTIONS FOR OUR METHOD
Our algorithmic description for MERCI is as follows:
In the pipeline of bonus filtering, the step 3, i.e., stree-suupression Filtering, can vary across tasks and can be optionally applied configured depending on specific task requirements.
15

---

Published as a conference paper at ICLR 2026
Algorithm Motivating Exploration in LLM Reasoning with Count-based Intrinsic Rewards
Imput: policy model coin flipping network fe. dataset D, iterations N, outcome-based reward function R, exploration coefficient 7, clipping factor a.
for i 1 to N do
Generate y ~ <T(-12) for each prompt x in D, and use R to compute Aola via Equation 3
Extract hidden expression hhidden of each token in y described in Section 3.2
Compute y's bonus with the process introduced in Section 3.3 and Figure 2 then incorporate
into the original advantage by applying 7 and via Equation 8
Generate random vectors and update the parameter Φ via Equation
Update the LLM policy using Anow in Equation end for Output: Fine-tuned To and fø-
D PROOF OF PROPOSITION
The proof follows the methodology presented in (O' D'Donoghue al. 2017). According the defi- nition of the conditional variance, we have:
V Vel2" (s, a)] E, 1 (s. - E. EE, a))?
27
Eg E, (()
(*)s) -E + Σ P -
27
E T ( (s+t((',aa)) - E, a')) (DD)
+ E,
The second equality holds by expanding the square and assuming that the reward estimate (*() and the next-step Q-value estimate Q++1 (s',al d') are conditionally independent, which makes their cross-product term zero.
Now, we focus on the second term. Noting that Ev Th Ph = 1, this term represents a weighted sum. Since the function f(x) = x² is convex, we can apply Jensen's inequality.
For convex function S. weights w; that sum to 1, and random variables Zc, Jensen's inequality states:
E ʃ Σ wi
EwrZi
< Σ w.E [S(2.)
Applying this to expression gives:
E,
Σ
T s/ (Q)tt(6',a')) -
8'.0'
Σ The pha Eg (Qh+t((',a') - E,(@h+
a'
ph
dead (5,,a)) a')]
Combining the results, arrive the final inequality:
V.(Â" (s, a)) v, (s)] + Σ nh a Pha V, 2
(8', a'))
e'.a'
This shows that the variance of the Q-value at step is bounded by the variance of the immediate reward plus the expected variance of the Q-value at the next step, h + 1.
16

---

Published as a conference paper at ICLR 2026
Table 3: Our CFN training configurations on mathematical reasoning tasks.
Hyperparameter
Value
Optimizer
AdamW
Learning rate in the pretraining process
1e-3
Learning rate the RL training process
le-4
Training batch size Mini-batch size
512 8
E DETAILED TRAINING CONFIGURATIONS
E.1 TRAINING DATA AND REWARD FUNCTION
Mathematical Reasoning
For both our train dataset and test dataset, we use the following system
prompt:
System Prompt
Please reason step by step, and put your final answer within \boxed
We use an outcome-based reward function that assigns +1 for correct final answers and -1 otherwise.
SQL Generation For both our train dataset and test dataset, we do not explicitly use any system prompt. We add the following contents at the beginning of the user prompt:
Prompt
Task Overview:
You are a data science expert. Below, you are provided with a database schema and a natural language question. Your task understand the schema and generate valid SQL query to answer the question.
The outcome-based reward function is dense: scnae answer_score format_score, where:
1.0,
if Result(S) = Result(G)
answer-score
min
[gold_dict resultddict) 1.0) x 0.8 if Result(S) * Result(G)
count
Above, S is the generated solution string (predicted SQL query), and G is the ground truth query. Result(() the set of execution results returned by the database when executing the SQL query Q
E.2 CEN TRAINING CONFIGURATION
For CFN training, we first generate rollouts from the backbone model on the training dataset and use these responses to perform preliminary training of the CFN. This process enables it to develop basic understanding of which states are likely to occur more rarely. During the RL training phase, we initialize the exploration model with the parameters of the pretrained CFN prevent the information provides at the outset from misleading the policy model.
Mathematical Reasoning ical reasoning tasks.
We use the hyperparameters in Table for CFN training on mathemat-
SQL Generation We use the hyperparameters in Table for CFN training on SQL generation tasks.
17

---

Published as a conference paper at ICLR 2026
Table 4: Our CFN training configurations on SQL generation tasks.
Hyperparameter
Value
Optimizer
AdamW
Learning rate in the pretraining process
3e-4
Learning rate the RL training process
le-4
Training batch size Mini-batch size
128 8
Table 5: Our RL training configurations mathematical reasoning tasks.
Hyperparameter
Value
General training hyperparameters
Optimizer
AdamW
Policy learning rate
le-6
Training batch size
512
Samples per prompt
8
Mini-batch size
32
Max prompt length
1024
Max response length
3072
Rollout temperature
1.0
Method speciffic hyperparameters Top p% in step of bonus filtering
30%
Initial y in Equation g
0.4
Equation 8
0.5
E.3 RL. TRAINING CONFIGURATION
conducted on 32 NVIDIA H20-96GB GPUs. For the reproduced Entropy Adv. and iMentor methods, we adopt the same general training hyperparameters as listed in Table S and T while their method-spesd -specff hyperparameters follow the configurations reported in the original papers.
Our experiments
Mathematical Reasoning We use the hyperparameters in Table RL training mathematical reasoning tasks. Notably, during RL training, we applied a cosine decay schedule to the discount factor 7, configured that by step 200 it reached 10% of its initial value. The same applies in the SQL generation task.
In addition focusing on mean @ k. also place considerable emphasis on pass@k. However, observe that as vanilla GRPO training progresses, increases in mean @ are generally accompanied by sharp decreases in pass kk which is also presented Appendix G.2.2 Therefore, ensure comparability across both types of metrics, we train each experiment for 120 steps vanilla GRPO. For DAPO, we train each experiment for 160 steps (including data sampling and filtering).
SQL Generation
We use the hyperparameters in Table for RL training on SQL generation tasks.
We train each experiment for 160 steps vanilla GRPO, and 240 steps DAPO (including data sampling and filtering).
F INFERENCE CONFIGURATIONS
Mathematical Reasoning We use rollout temperature of 0.6, top-p sampling with = 0.95, and maximum response length of 4096 tokens. We adopt 256 for the small but challenging AIME2024/2025 datasets (30 problems each), = 16 for Minerva, MATH500, and Olympiad- Bench, and for College Math, balancing computational cost and difficulty.
18

---

Published as a conference paper at ICLR 2026
Table 6: Our RL training configurations on SQL generation tasks.
Hyperparameter
Value
General training hyperparameters
Optimizer
A
Policy learning rate
le-6
Training batch size
128
Samples per prompt
8
Mini-batch size
64
Max prompt length
8192
Max response length
4096
Rollout temperature
1.0
feethod-spectpee hyperparameters Top 9% in step of bonus filtering
20%
Initial y in Equation g
0.1
a in Equation g
0.5
PICAE
Fantuan
otter
Ferectin
eince
Sona
Servece
Entermnc
Turssten
Siree
Figure 3: An example of token-level estimated epistemic uncertainty within response. Red regions indicate relatively higher uncertainty estimates assigned by the CFN to the corresponding token positions, while blue regions indicate relatively lower estimates. The same applies hereafter.
SQL Generation We use a top-p sampling with p = 0.95, and a maximum response length of 4096 tokens. We use rollout temperature of 0.0 for greedy sampling, and rollout temperature of 1.0 to evaluate @.
G ADDITIONAL EXPERIMENTAL RESULTS
G.1 COIN FLIPPING NETWORK
Uncertainty Estimation We conduct two experiments evaluate CFN as described in Section 5.2.1 and the detailed results are presented as follows in Figure 3 and Figure]
Generalization Ability of CFN We directly apply the CFN trained on mathematical reasoning tasks to estimate the uncertainty of responses in the SQL Generation task, and some examples are shown in Figure g Figure 9 and Figure 10 Since SQL code is indeed infrequently encountered, exhibits higher uncertainty, and the CFN correspondingly produces noticeably elevated estimates. In contrast, the uncertainty values assigned to other natural language reasoning sequences are largely consistent with intuition and expectations. This directly demonstrates that the CFN is capable of
19

---

Published as a conference paper at ICLR 2026
List
Henter
cenntty
Jun
everage
SLacE
prnt
Clym
Syteriry
Dooghht
purliii
sutbera
Figure An example of token-level estimated epistemic uncertainty within a response.
seatthy
Lacke
mm
10
From
19m1
mr
The
thaniture
Figure An example of token-level estimated epistemic uncertainty within a response.
20

---

Published as a conference paper at ICLR 2026
Farmed
foredd
Durt
CEGT Histion
T/L
rigater
CFC
CTURS
Fight
10.0 2o
MCT
Woyal
HGL
prlace
waltest
CITY
Exprried
Kuteer
Sinee
ctrt
eggh
Inla
rendeiy
witword
L:tagge
preattilyy
priteillyy
Free
Viglt
Clork
Fight
Claagl
Sppected
eigins
otlaciad
T4L
sireapy
sEgoost
Firdt
Th11
Inange
selected
Figure An example of token-level estimated epistemic uncertainty within a response.
(or
166
using quadratic
3. "Find
144
(and)
need to fend the
132
"Delaamie
2. "Find the
104
By Mitaas formulas
99
Angle Bisector
atroady smolest
8R
85
need find
54
"Delaamnne
Phrnsa
4. "FFind the
78
es trinngie
"Undorstanding the Problem:
tapezoid We need to
Rearranging
use the fact
86
Substituting
"Understtndd the Problem
we can equata the which simpllies tn
58
67
The
3. "Finding the
55
using Heron's formutia
100
125
150
175
Frequency
Figure 7: A statistical analysis of the occurrence frequency of contiguous token segments within each response that fall within the top 30% of bonus values (after filtering out code-related segments).
21

---

Published as a conference paper at ICLR 2026
Tree
CAST
REATT
Lasst
2014
Gal
MODE
Tack
200L
Fallans
LIMeT
Citlorta
sligile
rate
Lodts
carry
Teat
Pabllc
INCER
Figure 8: An example of lekel-level estimated epistemic uncertainty within response for the SQL generation task. The CFN is trained on mathematical reasoning tasks.
claciaa
actectt
stocie
Larry
Figure 9: An example of token-level estimated epistemic uncertainty within a response for the SQI. generation task. The CFN is trained on mathematical reasoning tasks.
leveraging the LLM's general features and translating them into an effective novelty estimate across domains.
Semantic Capturing To assess the CFN's ability to capture semantic similarity and to determine whether it can provide reasonable uncertainty estimates for sequences that are linguistically close but not identical, we present in Figure IT several example statements along with their corresponding aggregated trajectory uncertainties. The results are obtained from the CFN at training step 80.
The underlying rationale and semantics of (I) and (2) are linguistically close but not identical, and the resulting overall uncertainty estimates for the trajectories are also close. In contrast, (1) and
pridine
Cimeiity
typr acteride
Petrac
Eirst
Scredd
Closs
tayk
Isteccal
CONNCTTd
ELEET
conecter
Figure 10: An example of token-level estimated epistemic uncertainty within a response for the SQL generation task. The CFN is trained on mathematical reasoning tasks.
22

---

Published as a conference paper at ICLR 2026
(3)
1. Counting Subsets:
Total Number of Subsets:
1. Elementwise Perspective:
Since has elements, its power set contains 2"
The set has (since S has 6 elements).
subsets
Instead of counting subsets directly, the process per element: each element of independently lands A. both, or neither.
2. Choosing Subsets A and B: Both A and B can any subset of Therefore, there 64 64 = 4096 possible pairs of (A.
Selecting and
Both subsets ane chosen unif uniformly from all subsets, giving total of 64 ordered pairs B).
4096
2. Characterizing the Event:
No element can simultaneausly
lie in and also in both and A. Thus, every element must choose from restricted of valid membership patterns.
Condition Contained in S-4:
3. Condition for to be Contained
For given subset can be any subset of any subset of 5 A. The number subsets of is
Fix particular A, B acceptable precisely when either subset of A or or subset the complement There 214 subsets of and S - A 26-1A subsets of S A. Thus, number possible 2 1..
3. Configuration Counting: For each element, determine how many assignments of membership in A and B satisfy constraint: then raise that number to the 6th power
number of subsets of 5 is 152**M
Therefore, for each subset A, are 2 26-141 valid subsets B.
0.2547
0,3452
0.2323
Figure 11: An example of reasoning trajectories and corresponding aggregated uncertainties.
(3) differ more substantially, leading larger discrepancies in the trajectory-aggregalee uncertainty estimates.
Relation with Entropy, Score and Rollouts Variance We record the following information dur- ing the process policy model rollout: (a) the CFN uncertainty estimates and policy entropy each token position; (b) the average CFN uncertainty estimate for each trajectory and the corresponding within-group rollout variance; and (c) the average CFN uncertainty estimate for each trajectory and the final score (i.e., empirical success rate). The scatter of these records are shown in Figure 12
These results indicate that: (a) Positions at which CFN predicts higher uncertainty do not necessar- ily correspond to higher entropy. Policy entropy measures the policy's action randomness, whereas the CFN is designed to measure the model's epistemic uncertainty regarding its own state knowl- edge. Thus, results confirm that the CFN provides unique, non-redundant signal that cannot be simply replaced by the policy's action probabilities. (b) For trajectories that ultimately fail, the average uncertainty estimated by the CFN tends to be higher. This aligns with understanding, since highly novel states typically correspond to regions that the model has insufficiently explored. In such under-explored regions, the probability of ultimately solving the problem should naturally be lower. (c) In terms of trajectory uncertainty and the variance of the corresponding within within-group group roll- results, no particularly pronounced correlation observed. Some trajectories with zero rollout variance exhibit relatively high uncertainty. which may be due to the fact that we generate rollouts using the base model; in datasets of relatively higher difficulty, this can lead to a larger number of entirely incorrect trajectories.
G.2 RL TRAINING
G.2.1 CROSS-DOMAIN EXPERIMENTS
To further evaluate the cross-domain effectiveness of our approach, we transfer the RL models trained on the Mathematical Reasoning dataset to downstream tasks such as MMLU-Pro and GPQA for testing. We convert each problem into multiple-choies question (MCQ) format, and the system prompt is as follows. For GPQA, we sample up to 16 times, whereas for MMLU-Pro, we sample only once due its large scale. The results shown in Table
23

---

Published as a conference paper at ICLR 2026
Suaten Uhoostrrry Hig Score
Sutten Unsertaddy Varaato
22
-17
-.2
Paay DENe:
(3)
(c)
Figure 12: Relations between the CFN Uncertainty Esitimation and Entropy, Score and Rollouts Variance.
Table 7: Results of cross-domain experiments MMLU-Pro and GPQA.
GPQA
MMLU-Pro
mean @
pass @
pass @
pass@
Qwen2.5-Math
8.98
47.5
53.6
5.8
GRPO
24.3
59.8
61.6
28.3
+ GRPO w MERCI
26.4
65.0
69.6
29.1
DAPO
26.2
61.2
70.5
37.4
DAPO w MERCI
27.4
64.3
73.7
39.5
System Prompt
What of the following is the right choice? Please reason step by step, and put your final answer within \boxed()) The final answer must be capital letter like A. B, C, or D.
The results show that incorporating MERCI consistently provides gains on top of both GRPO and DAPO, particularly GPQA pass @8 and pass @ 16, as well MMLU-Pro pass@1 These findings suggest that our MERCI play an important role in improving out-of-domain robustness, even when the underlying training data is highly domain-specializzdd
G.2.2 SCALING EXPERIMENTS
As a comparison, we scaled the vanilla GRPO baseline experiment, continuing to train the GRPO baseline to the 260th training step. We observe that, although extended training increases mean @ k it substantially degrades pass@k, consistent with the limitations of GRPO discussed in Appendix By contrast, our MERCI algorithm realizes its exploratory potential earlier, rapidly identifying good solutions and exhibiting improvements in pass k as well.
E
G.2.3 TRAINING DYNAMICS
We report the training dynamics of both validation accuracy and response length on mathematical reasoning tasks in Figure 13
G.2.4 ABLATION STUDIES
To verify the effectiveness of the modules in our method, we conducted ablation studies on the mathematical reasoning task and vanilla GRPO, and present the results in Table
As evidenced by the preceding experimental results, both noise filtering and normalized trajectory- aggregated uncertainty estimation critical our method; without them, training can become
24

---

Published as a conference paper at ICLR 2026
Table 8: Results of scaling experiments on vanilla GRPO and mathematical reasoning benchmarks.
(a) pass k results
AIME25
AIME24
Minerva pas 16
MATH500
OlympiacBench
College pas
Avg.
pars 256
pass 256
pass 16
pass /6
76.7
GRPO-scaling180
50.0
66.7
61.4
90.2
57.3
48.2
62.3
GRPO scalinggn
46.7
70.0
59.6
88.8
56.4
48.1
61.6
GRPO MERCI
80.0
63.2
91.4
60.9
48.9
67.4
MERCI-scaling100
61.4
90.4
48.5
GRPO MERCI-scalin2200
50.0
70.0
61.2
89.4
58.5
47.5
62.8
(b) mcan @ k results
AIME25 mear ( 256
AIME24 mear 256
Minerva mear 16
MATH500
OlympladBench mean (8 16
College mean
Avg.
mean 16
GRPO
GRPO 180ling100
13.1
27.1
42.0
78.9
41.2
42,9
40.9
GRPO-scainng260
12.7
28.3
42,8
78.7
40.8
42.7
41.0
MERCI
44.1
80.7
GRPO MERCI-scaling 80
14.1
43.0
80.7
41.9
GRPO MERCI-scalin2200
12.9
30.9
43.6
80.6
42.5
43.0
42.3
Accurayy
Length
Veldattion
Responas
0.00
0.34
230
150
0.0
Tea
Tratere R0yy
Figure 13: Validation (i.e., MATH500) accuracy and response length during training.
unstable and may underperform the baseline algorithm. Furthermore, percentile and spatial coherence filtering direct attention to uncertainty at key positions, while the method yields more accurate estimates, thereby further improving our algorithm's 's performance.
G.2.5 HYPERPARAMETER CHOICES
Dimension of CFN As comparison, we the dimension of CFN 40, and the experimen- tal results are shown in Figure 14] We posit that if the CFN dimension were significantly larger, the high-dimensional LLM hidden states, when projected through the network head, might experi- ence reduced distinguishability. A large dimension could potentially oversmooth the feature space, causing subtle important differences between novel and seen states become less pronounced.
Decay of the exploration coefficient To discuss how the slower/faster cosine schedule decay impacts the final performance, we conducted experiments with the decay steps set to 100, 200, and 300, respectively, and the results are presented in Table TO.
The experimental results demonstrate that the cosine decay of to 10% by step 200 provides better overall performance. Schedules with a faster decay rate (e.g., decaying to 10% by step 100) led to insufficient exploration time. The policy quickly stabilized into suboptimal reasoning paths, result- ing in a lower ceiling for the final performance. Conversely, schedules with a slower decay (e.g., decaying by step 300) hindered convergence late in training. The persistent, strong intrinsic reward introduced excessive noise or bias, preventing the policy from focusing on maximizing the external task reward, thus degrading the final performance and stability.
25

---

Published as a conference paper at ICLR 2026
Table Ablation studies on vanilla GRPO and mathematical reasoning benchmarks. *p & s filter- ing is reduced-form representation of percentile and spatial coherence filtering. **The difference between cumulative std and cunulative variance (i.e, our main results) has been introduced Para- graph 1 of Section 3.3 i The setting of token integration expresses that, rather than computing uncertainty at the trajectory level for entire trajectories, we directly add an uncertainty estimate to each token-level advantage. MERCI w/o normalization is the results of strictly following the sum- then-sqrt computation and divide the result by constant factor (set to 1000), without applying any additional normalization.
(a) pass k results
AIME25
AIME24
Minerva
MATH500
OlympiadBeench
College pasr 8
Avg.
pass 256
pas 16
pass 16
16
GRPO
53.3
76.7
64.0
91.8
59.7
49.2
65.8
GRPO MERCI w/o p & filtering*
56.7
73.3
90.0
60.0
47.9
65.6
GRPO MERCI w/o noise filtering
50.0
89.2
48.9
1
GRPO MERC CI w/ camulative sid*
56.7
76.7
65.1
90.6
60.2
48.1
66.2
GRPO MERCI w/ token integration
46.7
70.0
62.5
91.6
59.3
49.1
63.2
GRPO MERCI
60.0
80.0
63.2
91.4
60.9
48.9
67.4
(b) mean @ k results
AIME25
ALME24
Minerva mcan 16
MATH500
OlympiadBench meaa 16
College meon@8
Avg.
mean 256
mean 256
meon
GRPO
11.2
28.7
41.8
79.0
40.3
42.0
40.5
GRPO MERCI w/o p filtering filtering
11.7
28.1
44.9
79.9
39.9
42.6
41.2
77.3
GRPO MERCI w/o cormalization
80.0
GRPO + MERCI cummlaative sid
14.2
29.1
43.8
79.8
41.2
43.0
41.9
GRPO MERCI token integration
23.7
40.2
77.5
39.9
42.4
39.3
GRPO MERCI
13.4
29.6
44.1
80.7
42.6
42.9
42.2
Validation
0.78
0.2
GAPO
0.68
GPPO-MERCK--20)
GAPO- MBNC//d)6)
Training Step
Validation (i.e., MATH500) accuracy during training across different choices on the dimension of CFN.
Figure 14:
Top-p% % Used in Filtering For the percentile values used in the filtering step (i.e., top p%), likewise conducted experiments with settings of 20%, 30%, and 50%, respectively, and the results shown Figure 15
G.2.6 CASE STUDY
We analyzed two cases ALME202//AIME2002 to examine the effect of incorporating method, and the results are as follows.
In Case Study 1 AIME2024, compared with the DAPO solution, DAPO+MERCI method provides a significantly clearer and more rigorous derivation. By organizing the substitutions into structured sequence and isolating each variable through simple exponential equations, it avoids
26

---

Published as a conference paper at ICLR 2026
Table 10: Results of the speed of cosine schedule decay.
(a) pass K results
ALME22
AIME24
Minerva poss 16
MATH500
OlympiadBench
College pass @8
Arg.
possi 256
Pass 256
pass 16
pass 16
63.2
GRPO MERCT-decnystep 100
56.7
80.0
62.1
90.6
61.3 60.3
48.6 48.7
66.6
GRPO MERCI decuyslep3300
$6.7
76.7
62.9
90.8
66.0
(b) mean @k results
AIME25
A1ME22 meam @ 256
Minerva
MATH500
OlympiadBench meon&
College mean
Avg.
mean 256
meam 16
mean 16
GRPO MERCT-deeaysttp200
13.4
29.6
80.7
42.6
42.9
42.2
GRPO MERCI-dccasstpp10
12.9
28.9
43.8
80.0
43.1
43.4
42.0
GRPO + MMERCI-deaysttpp100
14.4
27.0
41.3
80.6
41.5
42.4
41.2
Validation Accuracy
Validation Accuracy
0.74
GRPO
GRPO
0.70
& MERCI(20%)
0.70
CB0-MMERCIddeccyyssep10000
GRPO (30C)(30%)
GRPO-MMERCKsdccayyyseeepppp2
0.68
0.68
GAPO+MERC([50%)
GRPO-MMERCHoeccaysseep330
0
20
40
60
100
120
20
40
60
80
100
120
Training Step
Training Step
Figure 15: Validation (i.e., MATH500) accuracy during training across different choices top p% and cosine decay step.
the excessive algebraic accumulation seen in the DAPO approach. The resulting argument is more transparent, mathematically systematic, and pedagogically effective.
In Case Study on AIME2025, the DAPO baseline's 's solution relies on extensive algebraic expan- sion followed by heuristic symmetry arguments that are not rigorously justified, causing disconnect between computation and conclusion. In contrast, our DAPO+MERCI method's solution adopts a structurally motivated approach based logarithmic differentiation, which directly relates the loca- tions of the roots to the behavior of the critical points. Although more concise, it provides a coherent analytic framework and logically valid path the correct values.
In summary, these studies consistently validate the observation that the incorporation of the uncertainty-driven intrinsic reward fundamentally alters the policy model's approach to multi-step reasoning: 1. Enhanced Strategy Coherence and Conciseness: CFN's novelty mechanism cuts "cognitive noise" and long, self-correcting detours, leading significantly improved strategy coherence and sample efficiency; 2. Cognitive Migration to Higher-Level Reasoning: The in- creased proportion steps devoted to higher-level reasoning the most significant indicator of improved cognitive efficiency; 3. Discovery of Novel and Elegant Solutions: The provided case study example also demonstrates that the MERCI-guide -uidded policy finds more elegant, efficient, or fundamentally different mathemaatical/strroturr approaches to solving problems.
Case Study on AIME2024
Question: Let x, and be positive real numbers that satisfy the following system of equations:
27

---

Published as a conference paper at ICLR 2026
logz 1 = yz Then the value of 1oog((")) 2 is where and n are relatively prime positive integers. Find + n.
2
logz
M.
=
3
log2
my
2
4
Response (DAPO): Let's solve the given system of equations step by step. The system of equations provided is:
I
1
log2
12
2
!/
log2
12
2
1
loga
zz
4
First, can rewrite each logarithmic equation in exponential form:1. log2 yz
implies y= 2 = / so = / yz.2. loga (+) 3 implies _ 23, so = 2 .2.33
log2 ay
implies _ 23, so = 2 xy.
my Now, we can substitute the expressions for y, and into each other to find the values of 2. y, and z:
From the first equation: From the second equation: y 23 az
//2 1z
From the third equation: : = 24 ry
By multiplying these three equations together:
x == (v2 yz) (2 xz) (2t my)
v.y.2= v2 2 2 (yz x2 x2)
z.y.z = v2. 2 2 (2² x.y 2 2 2 (2'
2.9.2-22thh (19)2²
8.9.:-2 (+
z. 2 (22)222)
z.y.z= 2 22) 1= (ryz)
1 2 (zyz)
xyyz 2 1
Now can /2 yz. y 23 x, y, and 2:From x /2 yz:
Ez, and 2 ay find the individual values of
V2 122
2 #
Let's find the exponents x, y. and z in the expression x. = V2 yz:
v2. yz
I = 2 1/z
x² 2 y 22
28

---

Published as a conference paper at ICLR 2026
We know zy² 2- 18
S0:
2 2 t
a x² 27 2 1
x² 2
x² 2 12
x² = 2
T 2
Similarly:
23 22
y 2 2 1
y/
8² = 23 2
y² - #
8² = 2
2 3
y² 2
y 2
Similarly:
2 xy
= 2 2 ス 22 2 = 2 2 -
24.2-
: 2 .2 21
2 2-
2
2 = 2
Now we need to find
log, (x (x²y²²?)|
x Ayy2 22
(2 2/4
(2 (2-
3
2
12
2
1 y2² 22 2
2- 4
1 2 - 2-
56 54 40
32 2 = 2
Now we find the logarithm:
llog (x 222 = logg (2- 25
25
8
8
Thus, m 25 and n = 8, so m + n = 33.
Let's confirm with Python:
29

---

Published as a conference paper at ICLR 2026
Python Code
Cal calculating the valueimport math
Values found
x_val 2** (-7/24)y_val 2** (-3/8)
2_val 2** (-5/12)
calculate the expression x"4 yr3 * 222
expression_value x_val**4 y_val+43 z_val*42 Calculate log2 of the expression value vog_value = math.loge (expression_vallee)
Absolute value
abs log value abs (log_ value)
print (abs lob__oo__
Output
3.125
The calculation confirms that |log2<xny* 2x²))=²))
3.125, which matches derived
value. Thus, = 25 and n 8, so m + = 33. Therefore, the answer is:
33
Response (ERCO+MERCT) To solve the given system of equations, start by rewriting the logarithmic equations in exponential form:
loga
2. loga
1
3
3. loga
=
4
Using the definition of logarithms, we can convert these into the following equations:
23 V2
2 2/2 2. x2
3. = 24 +/2
From these equations, we get:
v2 yz
y 2/2 2z
2 = 2 ry
Now we substitute equation (1) into equation (2):
= V/2 (V2 yz)= = 2/2 v2 yz² = 2 yy² = 2 y2²
Dividing both sides by we get:
2 22
So,
2² = 2
30

---

Published as a conference paper at ICLR 2026
= +/2
Now substitute equation (1) into equation (3):
2 V/2 (V2 yz)y - 1/2 v2 2 yy²= y²²
Dividing both sides by 2, we get:
2 yp
So,
y² $
2 !
(5)
Now we substitute equation (2) into equation (3):
22 (V2 rz)a
42
-2- 1 =
x²z 2 122
Dividing both sides by get:
1 = 2 1
So.
x² == 2 t2
= 2 &
(6)
Now that we have the values:
x = 2 214
= 2
== 2
We need find the value of |loga First, we caleulate x² 3²
(=²)²=2)
x²
2
7
= 2
3
2
2
2
12
2 O0
So,
x²y²²²²
2
2
To add the exponents, we need common denominator:
9
5
56
54
40
56
54
40
150
25
8
6
48
48
48
48
48
8
So,
x 33
2
25
Then,
log (x 2x2)) loog (2-#)
25
8
Thus,
log, (2 y 2222
25
25
8
The fraction 25 is already in its simplest form, 25 + = 33.
25 and 8. Therefore, +
The final answer is:
33
31

---

Published as a conference paper at ICLR 2026
Case Study on AIME 2025
Question: There are exactly three positive real numbers such that the function
f(z) (x 18) (x- 72) 98) k)
defined over the positive real numbers achieves its minimum value at exactly two positive real numbers r. Find the sum of these three values of k.
Response (DAPO): To solve this problem, we need to analyze the function ((x) = (r-18)(r-72166 (12))(=--8]22) 98)(==k) and find the values of l such that the function achieves its min- imum value at exactly two positive real numbers æ. F.First, let's rewrite f(xx) in more man- ageable form:
f(x) (o - 18)(x 72) - 98)(x - k)
To find the critical points of (1x), we need to find the derivative (x) and set it to zero. Let's define:
g(x) (z 18)(x 72) 98)(x k).
Then,
f(x) g(x)
Using the quotient rule to differentiate f(z):
f((x) g (xx)) x - g(x) .1
zg'(x) - ((x)
x²
x²2
We need to find g(x) and its derivative g'(x):
g(z)
18)(2 72) 98)(2 - k).
To find g'xx), we can use the product rule:
g'(x) = (x (2)))x 98) - k) + (a - 18)( 98)( - k)+
(x- 18) 72) -2) + (s 18) 72)(z 98).
Now we substitute g(z) and gxx into f'(x):
fx))) z ((lx-7))6 98) k) (x 18) 98) K:)
x²
x [xx 18) (2) + (x 18) 72) 98)]
x²
(x - 18)(x 72) - 98) (x= k)
z²
For f(x) to have minimum at two positive real numbers x, f'(x) = 0. This gives us the equation:
[x (z) (x - k) + (a 18) (2) - k) + 18) 72) - k)]
+ - 1x - 72) - 98) - (x - 18) - (2 - 98) - k) = O.
We can factor out (xr 18)(x - 72)(x 98)(2 k) from the numerator:
[1 (2) 98) + (x (x) 98) + (x (=) 72) (x 18) 72)
- 18)(x (2 - 98)
0.
We need to find the values of & such that the function achieves its minimum value at exactly two positive real numbers The critical points are given the roots of the derivative, and the behavior of the polynomial (() suggests the critical points occur when the polynomial's derivative has exactly two distinct positive roots.By By Vieta's formulas and symmetry, can deduce the
This equation will have critical points when the numerator is
32

---

Published as a conference paper at ICLR 2026
sum of the roots. The roots of the polynomial are related the values 18, 72, 98, and k. We need to determine the values of k such that the critical points occur exactly twice. This means we need to analyze how the polynomial behaves around its critical points, and the critical points will be symmetric around the average of the roots.By By symmetry, the critical points will be symmetric around the average of the roots. This means:
18 + 72 + 98 + k
critical point.
For there to be exactly two positive critical points, the polynomial's derivative must have exactly two distinct positive roots, meaning the polynomial must have a double root at two distinct positive values. This can only happen if the polynomial has double root two distinct points, and the critical points will be related to the values 18, 72, 98, and k. The sum of the roots taken behavior suggests the critical points occur at the midpoints of the roots:
a time will give the possible values of The polynomial's
18 + 72 45,
+2 98
98 + 18 58, and
k + midpoint.
85,
2
2
2
The sum of the distinct positive values of k that make this possible can found by solving:
=5 85
18 98.
This gives:
hn = 145 188 = -43,
k2 = 145 145 = 0,
1: 145 0 = 145.
These values must be positive and real, so we need the correct ls values such that there exactly critical points:
k 45 85
- 18 72 - 98 433 not valid,
h = 18 72 98 18 - 72 - 98 0, not valid,
k3 = 18 7 98 + 18 72 98
From symmetry and polynomial properties, the sum of these valid values is:
k + h2 + kB 140.
We need the sum:
h = 140 - 18
72 98 18 + 72 98 140.
The sum of these three valid values:
kj + 2/ + k3 = 140.
The correct values for are 38, 62, and 140, and the sum is:
38 + 62 + 140 240.
Finally, we confirm:
240
Response MEACO+MERCI; To solve the problem, we need to analyze the function f(z) (a- (8) (z) 98) k) and determine the values of k such that the function achieves its minimum value at exactly two positive real numbers æ. Let's break down the problem step-
by-step.
1. Rewrite the Function:
(x (2) - 72)(x (x) k)
fxx)
We can expand the numerator:
(x 18) (2) - 98) kk)
33

---

Published as a conference paper at ICLR 2026
To simplify the analysis, we define a new polynomial P(x) = (ar (8 72) 98)
k). So the function becomes:
P(x)
[(x)
2. Find the Derivative: We need to find the critical points by taking the derivative of fx)).
Using the quotient rule:
(((x) (()x) x- (()))
x²
Setting ('(x) = O, we get:
('(x)
P'(x) ( P(o)=0 P'(x) P(x)
P(x)
T
This is equivalent finding the roots of:
1
dr (In P(x)) =
Polynomial P(x): The polynomial P(x) (x -x 18) (2) 9x k) has roots at æ 18, 72, 98, k. The derivative (xxx will have roots that are related to these values. For f(x) to have exactly two positive values of where achieves its minimum, the critical points must be such that the second derivative test indicates a minimum at exactly two points. Analyze the Critical Points: The critical points occur when:
d
P'
dx ((x 18) 72) 98)(2 k))
Setting P'(x) x P(z). this equation satisfied when equals some value that balances the roots of P(z). For f(z) to achieve its minimum at exactly two positive values of x, the critical points must specific values related to the roots of P(x))
5. Symmetry and Values of k: The function f(r) will achieve its minimum at exactly two points if the critical points are symmetrically placed around the midpoint of the roots of the polynomial. This symmetry suggests that k must be chosen such that the critical points occur at specific values.
The critical values for can be determined by considering the balance of the roots. Specifi- cally, if we solve:
P'(x) P(x) æ
1
find the values of k. By symmetry and considering the midpoint, the values of k are likely to related to the average of pairs of roots. The roots are 18, 72, 98, and kt By solving the above equation, find the values:
h = 8, 32, 200
These values ensure that the function achieves its minimum at exactly two points. Summing these values:
8 + 32 + 200 = 240
Thus, the sum of these three values of & is:
240
34


Reviews:
[{"review_id": "hOUuy3JJ7q", "reviewer": "Reviewer_ZHHS", "summary": "This paper introduces MERCI, a framework for motivating exploration in LLM reasoning using count-based intrinsic rewards. The method is grounded on a theoretical simplification of the Uncertainty Bellman Equation (UBE), which allows for estimation of trajectory uncertainty using a Coin Flip Network (CFN). The authors then convert the uncertainty into an intrinsic reward that can be integrated into RL algorithms like GRPO. Empirical evaluations on mathematical reasoning and SQL generation validate the effectiveness of MERCI.", "questions": "1. Can MERCI generalize to non-mathematical tasks such as MMLU-Pro, GPQA?\n\n2. CFN may underestimate uncertainty for states that are linguistically close but not identical. Did the author observe or analyze any instances of failure?\n\n3. Figure 8 shows the response length shrinks when MERCI is used. Is the policy truly “exploring”, or is the bonus merely penalising verbosity? I think an entropy-bonus scatter plot would help.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Principled Theoretical Simplification: The paper provides a thoughtful application of the UBE to the LLM setting by removing the need to estimate environmental uncertainty. The reduction of Q-value uncertainty estimation to local reward uncertainty is precise and well-motivated.\n\n2. The design of MERCI to use CFN for state visitation estimation is both conceptually clear and interesting.\n\n3. The comparative results against GRPO, DAPO and the visualization of token-level estimated uncertainty support the claim proposed by authors.", "weaknesses": "- The bonus is scaled by a cosine schedule ($ \\gamma $ decays to 10 % by step 200). There is no ablation on slower/fast decay, and no discussion of how this impacts the final performance.\n\n- It seems that different reasoning tasks need to train the corresponding CFN (*e.g.*, MATH and SQL), which may limit its generalization ability. What is the computational cost of training a CFN? Can a single CFN be trained for tasks across different domains?\n\n- I suggest citing relevant works [1][2][3] that also introduce intrinsic/curiosity reward bonuses to promote exploration.\n\n---\n\n[1] CDE: Curiosity-Driven Exploration for Efficient Reinforcement Learning in Large Language Models. arXiv preprint:2509.09675\n\n[2] Consistent Paths Lead to Truth: Self-Rewarding Reinforcement Learning for LLM Reasoning. NeurIPS 2025\n\n[3] Exploring Beyond Curiosity Rewards: Language-Driven Exploration in RL. PMLR 2025", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2kWaGOKSOT", "reviewer": "Reviewer_8z7R", "summary": "This paper proposes MERCI (Motivating Exploration in LLM Reasoning with Count-based Intrinsic Rewards) to add intrinsic rewards to RL for training LLM reasoning.", "questions": "See weakness.", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "1. The paper is well motivated and the writing is clear.\n2. Exploration in RL, especially in the LLM domains, is important, given that the language domain is large and exploration is hard.\n3. The experiment seems to support the effectiveness of the method.", "weaknesses": "1. The Coin Flipping Network Head is never trained. Can it capture the vast space of languages and truly distinguish the novelty of the exploration steps? E.g., for two semantically similar trajectories, say two word-level-distinct responses that have the same meaning, why can we expect a random MLP head to capture this without training?\n2. I appreciate the examples in Figure 4-6. But I still encourage the authors to provide a quantitative analysis of the improved exploration efficiency beyond the examples and eval performance, as this is the main claim of the paper.\n3. The proposed method is similar to some previous work, such as [1]. The authors cited this paper along with some other similar works, but failed to discuss the novelty and distinctions.\n\n[1] Gao et al., Navigate the Unknown: Enhancing LLM Reasoning with Intrinsic Motivation Guided Exploration.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aOcUyirRN8", "reviewer": "Reviewer_XirT", "summary": "The paper proposes MERCI (Motivating Exploration in LLM Reasoning with Count‑based Intrinsic rewards), an exploration module for value‑free RL fine‑tuning of LLMs. The key theoretical premise is that, in auto‑regressive reasoning, state transitions are known and deterministic, so the Uncertainty Bellman Equation (UBE) reduces to a cumulative sum of local reward‑estimate variances along a trajectory. This motivates an intrinsic bonus proportional to the standard deviation of the trajectory value, obtained by summing per‑step uncertainties and then taking a square root. To estimate per‑step uncertainty at scale, the authors attach a lightweight Coin Flipping Network (CFN) head to a separate copy of the model: the head learns coin‑flip regression to produce a pseudo‑count ($\\propto 1/n(s)$) from hidden states, which is used as a proxy for reward‑estimate variance. The bonus is filtered (percentile, spatial coherence, noise suppression), standardized within a GRPO group, and then added to the advantage used by GRPO/DAPO. Experiments on mathematical reasoning (AIME24/25, MATH500, OlympiadBench, College Math, Minerva) and text‑to‑SQL (Bird, Spider) show consistent gains over GRPO and DAPO; ablations and training‑dynamics plots support design choices. Figures 1–2 illustrate the architecture and the three‑stage bonus filtering pipeline; Proposition 1 formalizes the UBE with known transitions; Tables 1–2 report the main results.", "questions": "1. For Equation (6), why average variances before sqrt (length‑normalized) if Proposition 1 suggests a sum‑then‑sqrt? Can you report results with strict sum‑then‑sqrt (without group standardization) and then with a separate length‑normalization factor?\n\n2. What exactly is the \"state\" counted by CFN? Is it the hidden state of the CFN network at each token (Figure 1)? Do you use any normalization to ensure that counts are comparable across prompts and lengths? How do you handle near‑duplicates vs. paraphrases?\n\n3. Do high CFN bonuses correlate with (a) lower empirical success rate at those positions, (b) higher variance across rollouts, or (c) higher entropy from the policy?\n\n4. Could you clarify more on the relation between your work and the work by Lobel et al.?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The \"known transitions\" observation simplifies UBE to a tractable recursion with immediate reward‑variance terms only (Proposition 1), clarifying why trajectory‑level uncertainty should sum per‑step variances before taking the square root. This is a helpful bridge between theory and practice in value‑free RL for LLMs.\n\n2. The paper uses CFN to approximate pseudo‑counts from hidden states is elegant and inexpensive, avoiding ensembles/dropout/RND; it runs in parallel to policy learning (Figure 1). The paper also pretrains the CFN on rollouts to avoid cold‑start behavior.\n\n3. The paper presents consistent empirical improvements across settings. The gains are steady rather than one‑off, and are larger OOD (Spider), supporting the exploration claim.", "weaknesses": "1. Theory–implementation mismatch on the trajectory bonus.\n\n   Section 3.3 argues (correctly under the UBE derivation) that one should sum local variances along the trajectory and then take a square root. However, Equation (6) computes the bonus as the square root of the mean variance over retained tokens (division by $\\ell$ ), and then normalizes within a group (Equation (7)). This normalization helps length‑invariance, but it is not what the UBE prescribes. The ablation labeled \"cumulative std\" vs. \"cumulative variance\" (Table 8) touches a related point but does not fully clarify the discrepancy between sum vs. mean in Eq. (6). Please reconcile the stated principle (\"sum‑then‑sqrt\") with the actual estimator and report results using the strict sum‑then‑sqrt computation.\n\n2. CFN is pretrained on backbone rollouts. What is the wall‑clock/compute overhead and how does MERCI compare to similarly \"primed\" baselines (e.g., RND pretraining)?\n\n3. The \"GRPO‑scaling\" baseline (step 260) shows degraded pass@k; it would be fair to also show MERCI‑scaling beyond 120 steps to check stability under longer training.\n\n4. CFN dimension d=20, group size G=8, cosine schedule for $\\gamma$ (to 10% by step 200), and the top‑p% used in filtering (20–30%). Sensitivity curves for these choices would help practitioners.\n\n5. There are several typos (e.g., section 3.2 title \"REWARE\", \"Mathmeatical\", \"iterarions\", \"discribed\") and a few places where the exposition could be tightened (e.g., clearly defining \"state\": token prefix vs. hidden state embedding; whether repeats are exact or approximate).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iAbeP2B16w", "reviewer": "Reviewer_T5WN", "summary": "This paper addresses the challenge of exploration RL for LLM reasoning. Prevalent RL paradigms rely on sparse, outcome-based rewards and limited exploration. The authors introduce MERCI to augment policy optimization with a principled intrinsic reward. Building on the idea of count-based exploration, MERCI leverages a lightweight Coin Flipping Network (CFN) to estimate pseudo-counts and epistemic uncertainty over reasoning trajectories, converting these into an intrinsic reward that values novelty while preserving the learning signal from task rewards.", "questions": "See weakness.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper revisits the classic exploration-exploitation delima in the field of of LLM reasoning and RLVR, which is a significant topic\n2. The proposed method is rather novel compared to recently widespread entropy-based methods\n3. The proposed Coin Flipping Network Head is lighweight and the authors have verified it on both GRPO and DAPO algorithms.\n4. The authors have conducted experiments in both math and sql domains.", "weaknesses": "1. Missing baselines: There have been many works trying to solve the exploration problem such as i-MENTOR [1] and entropy-based advantage shaping [2], I would suggest the authors to compare with those baselines for a more up-to-date comparison\n\n2. Lacking analysis: It is rather informal that the authors present the ablations in the Appendix. Moreover, readers would like more analysis on the impact of the algorithms on the RL-trained behaviors, such as whether this algorithms introduces some novel strategies, solutions, or higher level of reasoning abilities, such as cogintive behaviors. \n\n[1] Navigate the Unknown: Enhancing LLM Reasoning with Intrinsic Motivation Guided Exploration  \n[2] Reasoning with exploration: An entropy perspective.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)