Random Policy Valuation is Enough for LLM Reasoning with Verifiable Rewards
Authors: Haoran He
Yuxiao Ye
Qingpeng Cai
Chen Hu
Binxing Jiao
Daxin Jiang
Ling Pan
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Models (LLMs)
Reinforcement Learning
RLVR
Math Reasoning
Diversity
Primary area: foundation or frontier models, including LLMs
TL;DR: A minimalist yet highly effective RL method that maintains both quality and diversity for LLM reasoning

Abstract: RL with Verifiable Rewards (RLVR) has emerged as a promising paradigm for improving the reasoning abilities of large language models (LLMs). Current methods rely primarily on policy optimization frameworks like PPO and GRPO, which follow generalized policy iteration that alternates between evaluating the current policy's value and improving the policy based on evaluation. While effective, they often suffer from training instability and diversity collapse, requiring complex heuristic tricks and careful tuning. We observe that standard RLVR in math reasoning can be formalized as a specialized finite-horizon Markov Decision Process with deterministic state transitions, tree-structured dynamics, and binary terminal rewards. Though large in scale, the underlying structure is simpler than general-purpose control settings for which popular RL algorithms (e.g., PPO) were developed, suggesting that several sophisticated techniques in existing methods may be reduced or even omitted. Based on this insight, we prove a surprising result: the optimal action can be recovered from the Q-function of a fixed uniformly random policy, thereby bypassing the generalized policy iteration loop and its associated heuristics. We introduce \underline{\textbf{R}}andom P\underline{\textbf{o}}licy \underline{\textbf{V}}aluation for Div\underline{\textbf{e}}rse \underline{\textbf{R}}easoning (ROVER) to translate this principle into a practical and scalable algorithm for LLM math reasoning, a minimalist yet highly effective RL method that samples actions from a softmax over these uniform-policy Q-values. ROVER preserves diversity throughout training, allowing sustained exploration of multiple valid pathways. Across multiple base models and standard math reasoning benchmarks, ROVER demonstrates superior performance in both \textbf{quality} (\textbf{+8.2} on pass@1, \textbf{+16.8} on pass@256) and \textbf{diversity} (\textbf{+20.5\%}), despite its radical simplification compared to strong, complicated existing methods.

Published as a conference paper at ICLR 2026
RANDOM POLICY VALUATION IS ENOUGH FOR LLM REASONING WITH VERIFIABLE REWARDS
Haoran He
Yuxiao Ye Qingpeng Cai Chen Hu Binxing Jiao
Daxin Jiang Ling Pan
'onng Kong University of Science and Technology KKuaishou Technology SttpFuun haoran, hel connect.. ust. hk, 1ingpan?usst ust.
ABSTRACT
RL with Verifiable Rewards (RLVR) has emerged as promising paradigm for improving the reasoning abilities of large language models (LLMs). Current methods rely primarily on policy optimization frameworks like PPO and GRPO, which follow generalized policy iteration that alternates between evaluating the current policy's ss value and improving the policy based on evaluation. While ef- fective, they often suffer from training instability and diversity collapse, requiring complex heuristic tricks and careful tuning. We observe that standard RLVR in math reasoning can be formalized as a specialized finite-horizon Markov Deci- sion Process with deterministic state transitions, tree-structured dynamics, and binary terminal rewards. Though large in scale, the underlying structure is sim- pler than general-puroose control settings for which popular RL algorithms (e.g.. PPO) were developed, suggesting that several sophisticated techniques in exist- ing methods may be reduced even omitted. Based this insight, prove a surprising result: the optimal action can be recovered from the Q-function of fixed uniformly random policy, thereby bypassing the generalized policy itera- tion loop and its associated heuristics. We introduce Random Policy Valuation for Diverse Reasoning (ROVER) to translate this principle into a practical and scalable algorithm for L.M math reasoning, a minimalist yet highly effective RI. method that samples actions from a softmax over these uniform pornny Q-values. ROVER preserves diversity throughout training, allowing sustained exploration multiple valid pathways. Across multiple base models and standard math rea- soning benchmarks, ROVER demonstrates superior performance in both quality (+8.2 on pass@) 1, +16.8 on pass @ 256) and diversity (+20.5 %), despite its radical simplification compared strong, complicated existing methods.
"Simplicity the ultimate sophistication." Leonardo da Vinci
Strategies
330
55.2117
60
Fasses (N)
18.3
7.0
:20
:10
Rovar (our))
kataro Zorroaee-
avo
Dero
Rise
Cosine
Utiley
ROVER
(b)
(c)
Figure 1: (a) Pass@1 & Pass@255 results on Qwen3-8B-Band Base averaged over AIME24, AIME25, and HMMT25 tasks. (b) Illustrative example demonstrating that ROVER achieves high-quality solutions with a lightweight procedure while maintaining diversity. (c) Comparison of different methods on multiple diversity metrics. Higher value denotes better diversity.
"Equal contribution.

---

Published as a conference paper at ICLR 2026
INTRODUCTION
RLVR has emerged as a promising paradigm for post-training LLMs and enhancing reasoning ca- pabilities (Jacch et al.) 22224, Guo et al., 2025). The field has primarily relied on Proximal Policy Optimization (PPO) (Schulman et al. 7). a powerful algorithm originally designed for standard deep RL benchmarks such as computer games and robotic control. This general-purpose algorithm and its specialized derivatives like Group- Delup-ee Policy Optimization (GRPO) (Shao et al. (2024) have achieved notable successes in improving LLM reasoning performance. Fundamentally, cur- rent methods follow the generalized policy iteration (GPI) (Sutton et al.) 1998) paradigm, which iteratively alternates between evaluating the current policy and improving it based on the evaluation.
Despite its success, they suffer from unstable learning dynamics (Yang et al. 2025a) and entropy collapse (Huang et al., 2024] Yang & Holtzman| 2025) induced by the reward-maximin -maimizzing nature within the iterative policy cralaation-mpprovenetn
cycle. As the policy continuously evolves, the evaluation target becomes non-stationary, leading training instability and narrowed exploration spaces. Recent variants mitigate this through an intricate ballet of heuristic techniques such as clip- ping (Yu et al. 2025), KL regularization (Liu et al., (025a), and data selection (Liang et al., (025). While incorporating these tricks offers partial improvements, they add layers of implementation complexity and typically require careful, case-specific tuning (Liu et al., 2025d).
We take a fundamentally different approach by examining the underlying structure of LLM math reasoning tasks with verifiable rewards. Unlike standard RL environments that sophisticated RL algorithms like PPO were originally designed for and evaluated (e.g.. discrete computer games with cyclic state transitions that forms a graph instead of a tree (Bengio et al.] (021), robotics with con- tinuous spaces, possibly with stochastic transitions and intermediate rewards), standard RLVR for math reasoning corresponds to a specialized finite-horizon Markov Decision Process (MDP) with deterministic, trec-structured transitions, and binary terminal reward. In this structurally simplified MDP, each action induces deterministic and new branch, and each partial sequence has exactly one parent state. This critical observation leads us to central question: whether we are applying unnecessarily complex tools to a structurally simpler (albeit larger) problem, and is there a mini- malist yet highly effective RLVR algorithm that maintains both quality and diversity under this specialized MDP structure? Our theoretical amalysis reveals a surprising result under this scenario: the optimal actions can be derived by simply evaluating a fixed uniformly random policy and then selecting actions greedily based on its Q-values. This surprising finding means that we can bypass the standard GPI cycle to identify optimal policies, which requires only policy evaluation of the simplest possible policy (uniformly random), without iterative evaluation of the updated policy and without the many heuristic tricks that plague current methods. Although it was widely believed that this kind of uniform policy is trivial that cannot provide meaningful guidance for control (Asadi & Littman 2017). the value of uniform policies (He et al. 2025b) has been observed empirically in specific discrete environments (Laidlaw et al. 2023) recently, and we provide a first theoretical analysis account for LLM math reasoning and leverage it as the foundation of our approach.
However, as in standard reward-maximin maximizing RL, while naive greedy selection guarantees optimal- ity, it sacrifices diversity critical for reasoning tasks eti ct al. (024). To balance quality and diversity, leverage key insight based on our analysis: uniform-policy Q-values capture the probability of successful continuations that lead to positive rewards. As this creates a natural value map of the reasoning landscape, we sample actions via softmax over the uniform-policy Q-values, which main- tains performance guarantees while aligning with modern LLM practices (Sheng et al. 2024 Kwon al. 2023). To translate our theoretical insights into practical and scalable algorithm for LLM reasoning which involves vast state and action spaces as well as long horizons (a wide and deep tree), we present Random Policy Valuation for Diverse Reasoning (ROVER). ROVER efficiently parameterizes the Q-function intrinsically based on the LLM's parameters, which eliminates the need for separate value network and also leverages the L.M'' strong priors for efficient navigation in the vast token space and stabilizing training through relative improvements. To mitigate the high variance caused by the reward signals, we leverage group reward centering inspired by Naik et al. 2024), and broadcast the reward to improve training efficiency.
Our contributions are as follows: (i) We prove a surprising result: in the deterministic tree-structured MDPs with binary terminal rewards that characterize math reasoning, the optimal action can be derived directly from Q-values evaluated under a uniformly random policy, a finding that funda- mentally simplifies RI. for this domain. (ii) We introduce ROVER, a practical and minimalist RI.
2

---

Published as a conference paper at ICLR 2026
algorithm that is scalable to LLM reasoning tasks through simplified framework compared to the current complicated methods. (iii) Despite ROVER's ' radical simplification, extensive experiments across diverse tasks and various model scales demonstrate that it consistently achieves superior performance, yielding +8.2 improvement on pass @ and +16.8 improvement on pass mass@2256 on the competition-level AIME24, AIME25, and HMMT25 tasks. Interestingly, we observe ROVER can find novel reasoning strategies absent from the base model and models trained through standard RL approaches (GRPO), thereby evidencing its potential to push the reasoning boundary. Our codes are available at https:://///... / github........ com/ wooaa/innnerei tinnerhrheeeeeeeee ROVER
2 PRELIMINARIES
RL with Verifiable Rewards in LLMs. We investigate reinforcement learning (RL) for post- training LLMs with verifiable rewards, such as mathematical reasoning tasks. We formulate the problem Markov Decision Process (MDP), defined by tuple (S, V. R. P.%. 7,)).. Here. the state space S denotes all finite-length strings formed by the concatenation of elements in V. The action space V is the vocabulary set. We set the discount factor y = 1 in practice. R: S V TR. is the binary reward function, and P S x V + S is a deterministic transition function. At the beginning of each episode, a prompt x is sampled from the initial state distribution X. At each step t, the LLM selects an action d₂ E V according to To (-)-(se), and then transits to the next state 0e) by concatenation. This autoregressive generation continues until forming an entire response y = {au, Q1 011l }. and finally receives a verifiable reward r(z, y) E {0, 1}. The goal is to Iearn policy x¹ arg max, y-x(() [r y)] by maximizing the expected cu- mulative reward r. The prevailing works leverage policy gradient (Williams) 1992) and a surrogate objective introduced by PPO (Schulman et al.] (017) to optimize No:
S1+1 {a. ag.
((Ø) = ax-To, [b] E. (min (1S, clip(I,,, 1 - clow, 1 (.ig)) - 3Dxx (Ta) (1)
(arlst) (sc) is the importance sampling ratio, Taaa is the behavior pol- icy to sample data, 8t = {ær, ] is current state, flow and thigh is the clipping range of impor- tance sampling ratios, DK denotes the KL regularization term, and A, is the advantage of current action. A, is implemented differently across RL algorithms, such as REINFORCE++ (Hu et al.) 2025a) and GRPO (Guo et al., 2025). For example, GRPO (Guo et al. 2025) samples G >
where IS, To (as|s))/
responses for each prompt and estimates the advantage A r(a.3)) ) -ean[[ (r(x.))) within std((((r.
each group to reduce variance. Notably, while existing policy optimization methods rely on a KI-divergence penalty (Dx1) to prevent catastrophic forgetting and maintain exploration during continual learning (Liu et al., (225a)). our approach achieves these desiderata without such an ex- plicit regularization term. For comprehensive discussion of related work, please see Appendix D
Generalized Policy Iteration (GPI). GPI (Sutton et al., 1998) is a unifying view that describes many RL algorithms (c.g.. PPO)
of
Evaluate
illustrated in Fig. _ GPI consists of two interacting processes, which are policy evaluation that estimates how good policy is, (e.g., via Q* az) r(st, +1 [Q" ((++)1, value func- tion, or advantage function), and policy improvement that updates the policy to prefer actions scored better by the current estimates arg max, Q* a) or other methods). Littman & Szepesvári (1996) introduced generalized Bellman update which update the Q-function by
Value of
anaate
Value
Exelunte
Value
of
(e.g.. (s))
Figure 2: Illustration of GPI.
Q(5c: ac) +- r(Se: 0x) + Ex,ies PPP(s, (z 8141) X Bati (18411:0441 with any arbitrary op- erator X that replaces the max operator typically used in Q-learning (Sutton al. 1998 HHarnoja et al., 2017; Pan et al. 2020), while the mean operator was traditionally considered unsuitable for optimization in general control tasks (Asadi & Littman (017). GPI-based methods require an al- ternative learning over these two processes until finding the fix point, where the learning target remains non-stationary throughout training (Mnih et al., 2015). In contrast, our proposed method relies solely on policy evaluation to derive the Q-values of a fixed, uniform random policy, which is much simpler for training and implementation (with a high-level illustration in Fig.
ROVER: RANDOM POLICY VALUATION FOR DIVERSE REASONING
RLVR for math reasoning can be cast as a decision-making problem in specialized finite- horizon MDP M with deterministic transitions and binary terminal rewards (correct or incorrect) in a tree-structured space (each state has a unique parent and actions lead to disjoint subtrees).
3

---

Published as a conference paper at ICLR 2026
This contrasts with general- RL settings that often fea- ture general control problems with stochastic dynamics, com- plex reward structures, and discrete (or continuous) graph- based state spaces where states can have multiple parents or even cycles. Although the PPO family achieves promising re- sults in LLM reasoning, it was designed for general control and can encounter entropy and diversity collapse in RLVR, which also introduces unnecessary com- putational overhead and complexity (Guo et al. (025).
Evaluate xy
Uniform policy
T
Q Ne
Greedy action selection
Optimal policy
Figure 3: Illustration of ROVER
(greedy).
Motivated by this structural mismatch, we consider an important question overlooked in the liter- ature: can there exist a minimalist and simple RL approach that exploits these properties of RLVR MDP to achieve both high quality and diversity? In contrast to adding various implementation- level tricks to PPO/GRPO, we present ROVER, which is built upon surprising discovery: simply evaluating uniformly random policy and selecting actions greedily based on its Q-values is suf- ficient for optimal behavior in this context (Fig. avoiding the complexities of modern deep RL algorithms (Schulman et al. 2017) and can bypass the traditional GPI loop in Fig. 2
We first establish the theoretical basis of this unexpectedly simple yet optimal approach in $ extend it to achieve diversity while maintaining performance guarantees in $ $3.1..1 and present a practical algorithm that scales to large spaces and long horizons for math reasoning in 3.2
3.1 THE RANDOM POLICY VALUATION FRAMEWORK
We start from the simplest possible policy, the uniform random policy [,[[|ss) = TAT. where A denotes the set of available actions. The corresponding Q-value for Tu can be estimated using the generalized Bellman update (Littman & Szepesvári, 1996, Sutton et al., 1998) with the mean oper- ator (Asadi& & Littman 2017). The mean operator corresponds to evaluating uniform policy, and the update is simplified to Q*x (s, a) + r(s,a) + o Ex'EA QTM (s', aa) for deterministic transi- tions and 7 (Hu al. (0255) that we consider as discussed in $ 2 The literature of classical RL suggests that this mean operator is insufficient for optimal control in general MDPs (Asadi & Littman 2017), it averages across all actions without preference for optimal ones, providing little guidance. While few recent studies have empirically noted the potential utility of uniform-poll policy values in certain discrete games (Laidlaw et al.) 2023 He et al., (2025b), these observations have remained primarily empirical, with limited theoretical justification.
1
In our context, LLM math reasoning induces finite-horizon, deterministic, tree-structured MDPs with binary terminal rewards (correct/incorreet) For a root state = x (i.e., prompt), the reachable transition graph a rooted tree, where each state has a unique path from sg and distinct actions from a state lead to disjoint subtrees. Under this context, we prove that simply evaluating the fixed uniform policy and acting greedily with respect to its Q-values already achieves optimality in Theorem [ The proof can be found in Appendix[A.I]
Theorem 1. Consider a finite-horizon episodic MDP with deterministic transitions, tree-structured state space, and binary terminal rewards R(s) f0, R} where R > 0 (R for a correct solution, 0 otherwise). Let Tu be the uniform policy, and QTa its corresponding Q-function. Define the greedy policy with respect QT by (s) = arg max, QT= (s,a), a), then grroddy is optimal.
From Theorem [ we discover that for the specific MDP structure of LLM math reasoning, the optimal control problem reduces to a much simpler form than previously recognized. This suggests two significant implications: First, despite the perceived complexity of LLM math reasoning tasks, their underlying decision structure ex- hibits a more tractable structure than commonly assumed. Second, the mean operator, although generally insufficient for optimal con- trol, proves to be surprisingly powerful when paired with greedy action selection strategy in this context.
Coreec (A 11
Surprisingly, although the uniformly random policy itself is far Figure 4: Intuition of ROVER from optimal behavior, its Q-values have a meaningful interpreta- (greedy) with greedy. tion here, which equals the probability that, after taking a at and then acting uniformly random until termination, we obtain a correct outcome. As illustrated in Fig. 4 when QT (s, a) = 0, it indicates that possible continuation from (s,a) can lead to a correct solution. Conversely,
4

---

Published as a conference paper at ICLR 2026
higher values indicate more promising directions. By acting greedily with respect to these values, we effectively eliminate branches that cannot lead to valid solutions while prioritizing the most promising paths. This property enables optimality through a remarkably computationally simple mechanism: we need only estimate Q* (s, a) by policy evaluation for a fixed uniform policy Tu- without off-policy corrections the implementation complexity of popular methods like PPO and GRPO. Additionally, since our approach evaluates fixed uniform policy rather than iteratively improving learned policy, it mitigates the non-stationarity issues that plague many modern deep RL methods ( Van Hasselt et al.) (2016), which can also be advantageous for the highdimensional, complex LLM math reasoning tasks.
A Didactic Example. To empirically validate the optimality of the greedy policy derived from the Q-function of a uniformly random policy, we design a tabular environment as illustrated in Fig. ((a) The environment is deterministic, tree-structured MDP capturing the essential properties of LLM math reasoning tasks while remaining transparent for analysis (and we will introduce how to scale up the method in 3.2). Starting from an initial null state, a policy executes an action a A = [A, C, D) by appending it to the current state sequence. We consider an episodic setup with binary terminal rewards, with 4 specific terminal states (ACD, BDC, CAB, DBA) yielding reward of 1 and all others yielding O. From Fig. we observe that the simple mechanism of acting greedily with respect random policy's Q-function also learns generate the sequence with the highest reward, achieving the same optimal behavior as Q-learning (with e-greedy exploration).
R-D
DRA
R-b
Nid
0.00 3000
0.000.00
23 000
-44
1V11
0.00
10
5:12 0.11
8:12
Not 0.05 000 0: 0:06
0 0.00
0:25 0.00
-14
Captured CapturedModss Modes
0.000.00.000
0.00
0:25 0.000000000000 0.00
-34
0.00
00 0.00
0.12
0.00
B22
0.00 0.00
BDC
ACT
0.00
0.05
150
0U00
OF
ACT
0
2:000.0000
1.0
no00
001
0.00
8
0.00
0:00
00 0.00
ACD
BDC
ACD
0.000.000
00T
1:00
0:00
020
Arto
ROVER ROVER Q Qlooring
R
(gpoady)
(a) Toy MDP
(b) Q-learning
(c) ROVER (greedy)
(d) ROVER
(c) Mode Coverage
Figure 5: (a) Illustration of the tabular MDP. (b)-(d) Comparison of learned Q-value maps. Accord- ing to the Q-values, standard Q-learning with é-greddy exploration converges to the mode ACD. ROVER (greedy) assigns the highest Q-values to optimal actions, but still converges to a single mode BDC due to its greedy behavior. ROVER is able to assign equally high Q-values to all optimal actions. (e) Q-learning and ROVER (greedy) converge to single mode despite both being optimal, whereas ROVER successfully covers all 4 optimal modes.
3.1.1 BEYOND GREEDY SELECTION: BALANCING QUALITY AND DIVERSITY
While our theoretical analysis shows that the simple scheme of greedy selection over the Q-values of uniform policy is already enough for achieving optimality, this deterministic approach often leads to mode collapse and sacrifices diversity (Fig. 5(c)). For LLM math reasoning tasks, as a given prompt can elicit multiple viable responses that yield correct solutions, diversity is critical for robust problem-solving (Li et al., 2025a), which is also important for improving pass@k k performance and generalization to novel problems.
Our analysis reveals a key insight: the Q*=(s,a) characterizes the probability of successful con- tinuations following the action a, where higher Q-values indicate action branches with denser suc- cessful pathways. To improve the diversity of policy generation, hased on this insight, we transi- tion from deterministic to stochastic action selection by converting QT into a soft sampler, i.c., TT (als) = E.' expl("u (s.a')/p) where p is a temperature parameter. This strategy selects actions proportional to their estimated success probability, which is able to explore multiple reasoning path- ways for improving diversity, rather than committing to a single path. Additionally, it aligns with contemporary LLM decoding strategies (Kwon et al., 2023), making it readily integrable into exist- ing training frameworks (Sheng et al. 2024). The following result shows that our softmaxing QT. approach maintains a guaranteed level of performance relative to the optimal policy, with the bound tightening as temperature decreases. The proof can be found in Appendix (A.2
exp(Q""(s.a)/00
Theorem 2. Consider the same MDP M, and let QTa (s, a) denote the Q-function under the uniform random policy nu from state-action pair (8,a). N(s) = Ma: Q==(s,a)) a) 0}} be the number of
5

---

Published as a conference paper at ICLR 2026
zero-valued actions at state A(s) be the number of available actions state s. and P denotes the set of key states where both optimal and suboptimal actions exist, i.e., P {5 1 S N(s) S
"sp(Q"=(s.a)/pp) A(s) - 1}. Given the softmax policy T (als) ex. with temperature p > 0, and Pr (s|so) the probability of reaching s from so with the policy Ta the value function of the N(e)
induced policy n, satisfies: (sa(s)) R E.E Pr* +exptmxpn Q==(s..)/o)
Theorem 2 characterizes that the temperature trades off between diversity and quality. As p in- creases, the policy samples more diverse actions while still favoring higher-value paths. When p approaches zero, the performance gap between the softmax policy and the optimal policy vanishes, showing that our
approach maintains performance guarantees.
empirically demonstrate that it achieves an effective tradeoff. While both greedy approaches (Q-learning and ROVER (greedy)) achieve op- timal reward but collapse to a single solution mode, ROVER (with p 1) successfully identifies all four optimal modes while maintaining 100% success rate. Our Sivverstt--seekinn RL approach stands in contrast typical RL diversity methods that often rely on complex and task-related re-
Justification. In our didactic example (Fig.
ward engineering (He et al. 2025a Cheng et al. 2025 Li et al. 2025a) or post-hoc sampling tech-
niques (Shur-Ofry et al. 2024 Chen al. 2025b) without guarantees, while remaining simple.
3.2 PRACTICAL IMPLEMENTATION
We now adapt our method to LLMs, where the induced MDP still remains deterministic and tree- structured, but presents computational challenges due to long horizons (deep trees) and large vo- cabularies (wide branching). To address these challenges, we introduce practical techniques to ap- proximate, stabilize the training process, and improve sample efficiency as summarized in Alg. [ while preserving the core idea of random policy evaluation. We also provide gradient analysis and connections to policy gradient methods in Appendix B
Algorithm 1: Random Policy Valuation for Diverse Reasoning (ROVER)
Input: pre-trained LLM iT. epochs M. prompt dataset D, group size n, Ir 17, temperature p for epoch m = {1, .M) do Set T0 +- To; Sample batch of prompts B 'D via TTaa for each prompt x E B do
L Rollout responses and compute rewards: {ur) ~ Teata (+1z); ((z);;==;; =r; E.= T
for each prompt- response pair {a, y:) in batch do for each state 8, E {x, }i do
1|51+1) = p( log To (at+1 - log Taai (a++ [st+ 1)) Obtain ((a)(s)) <- F + M Exwrev Q1 // V: the vocabulary set.
Compute Q-value
LROVER E. 18:1 EL= Elul-1 [IQ(at):))
sg: stop gradient.
0 9-6 0 VoLOVER by AdamW optimizer
Q Parameterization. While we begin with a reference LLM, we lack a pre-trained Q-function. Training Q-model from scratch presents substantial costs due the large scale of action and state spaces. A compelling approach is to represent the Q-function directly through the LLM's intrinsic parameters (Li et al. 2025b), thereby eliminating the need for separate value net- work. Fortunately, as indicated in Theorem 2 and following the mean operator for evaluating the value of the uniform policy in $ 3.1] the policy To and Q-values can be intrinsically linked through p log (a,a, with p the temperature, which captures the relative preference over ac- tions within each state (though a state-dependent constant term is omitted for efficient practical implementation). However, this direct formulation is unstable in practice since the learning target drifts as the policy changes and the Q-value updates are prone to divergence. To mitigate this in- stability, we introduce a relative Q-function that measures the improvement over fixed baseline: Q(5t. ax) = o( log (o(a)(s)) log Te..a (a, (x)), where To.ja is the behavior policy used to sample data in each epoch, serving as stable anchor that reduces fluctuations. This parameterization ters the initial Q-values around zero and ensures the model learns the change relative to the previous
policy instead of absolute values.

---

Published as a conference paper at ICLR 2026
vorriiiience Reward. To create stable and dense reward signal for learning unffrrmpoolig Q-values, we sample n responses for each prompt to reduce estimation variance and enrich our ap- proximation of the value landscape. Inspired by Naik et al./ 2024), subtract the empirical average reward of the n responses from the raw rewards to obtain mean-centered rewards. Specifically, the centered reward is given by f(ar, yt) r(x: yi) 1 Σ.= r(æ, yt), where r(2: yk) reflects the cor- rectness of the corresponding response y/ given the prompt This is also related to GRPO's style of estimating the advantage function, but without the standard deviation normalization term (Liu al.) 2025c). Additionally, to ensure efficient credit assignment, especially for long reasoning chains, we broadcast this centered reward Fzz y4) to every token in the generation following Hu et al. (2025b)
n
EXPERIMENTS
Although simple, our method substantially enhances both the quality and diversity of LLM gen- erations, leading to improved reasoning capabilities on complex tasks. We evaluate our approach on two verifiable tasks that require sophisticated reasoning: countdown tasks, which have multiple valid answers, and math competitions, which possess single, unambiguous answers.
4.1 COUNTDOWN TASKS
We begin evaluating our method on the countdown task. Given an array of numbers and a target, the LLM must find the correct sequence using the four basic arithmetic operations (+, -, x, +-) to reach the target number. We selected Countdown since it offers a restricted search space and multiple valid answers for a question that enables tractable analysis of both the reasoning behavior and diversity.
Setup. We evaluate on the ZinyZero (Pan et al. 2025) dataset with 1,024 test problems. We em- play Qwen2.5-3B [Team] (024) as our base model, which demonstrates near-zero accuracy this specific task that establishes a clear baseline for improvement. We benchmark our method against the well-recognized GRPO (Shao al. 2024) and two GRPO variants designed for policy en- tropy preservation: one with varying KL coefficients and another incorporating the clip- higher tech- nique ( Yu et al. (025). Detailed task descriptions and the training details are in Appendix B
T. 2
Quaetion: Using the numbers (21, 3.. 1,14)
Auguat That Sceree
rrvat equation that equak
(ROVER' selatiens
GAPO's
Enempy
()-335 1622
14 + 2
44 1n Kith (35 (21 /T)
E.
135
Siace
Stops
qual (haEEScom)
Figure 7: ROVER successfully finds 17 diverse solution equa- Figure 6: Performance of our method and baselines over training tions, while only 3 different on countdown tasks. The y-axis of (c) denotes the number of found equations are given by GRPO. distinct correct solution equations, averaged over 1024 questions.
(a) Test Score
(b) Entropy
(c) Diversity&Quality
Results Analysis. From the results shown Fig. 6 we have the following observations: (i) In terms of test scores shown in Fig. 6(a) our method surpasses all baselines after 400 training steps, ultimately reaching the highest ceiling performance. Conversely, the GRPO with a KI. coefficient of 0.01 performs distinctly worse, indicating that its performance is hampered by excessive regulariza- tion. We attribute the efficacy of our method to the preservation of high policy entropy throughout training. As shown in Fig. 6(b) our method's entropy decays gracefully while remaining signifi- cantly higher than that of the baselines, which either collapse (GRPO w/o KL) or fluctuate errati- A stable high entropy encourages sustained exploration, which is the
cally (GRPO w/ Clip_highens primary driver of our model's performance, enabling it to achieve the highest scores on both qual- ity and diversity metrics, as validated in Fig. 6[C) where our method finds more diverse solutions to address a question. Fig. 7 further provides a visualization example to demonstrate the solution diversity of ROVER.
Ablation on temperature p. Consistent with standard LLM sampling practices (Sheng et al., 2024]. we set temperature p = 1 for softmax sampling for all experiments without any task-specific tuning. This parameter balances the exploration-exploittionn trade-off: + encourages greedy, deterministic behavior, while higher values promote diverse sampling.
7

---

Published as a conference paper at ICLR 2026
Our ablation study p in Fig. confirms that p 1 achieves a robust and desirable performance. A higher temperature causes under-exploitation and slower con- vergence, while a lower value triggers premature ex- ploitation, causing an accelerated collapse in policy entropy and constrained exploration space. In the ex- treme case where o 0.001, the near-deterministic policy sampling leads to severe training instability (ev- idenced in test score), highlighting the importance of
ROVER noweR or fork poWER of tnd
Prggos THe Stare
480
700
(a) Entropy
(b) Test Score
Figure Performance under different p.
a balanced temperature for effective exploration. We
further investigate the effect of p on math reasoning tasks, where similar conclusions are validated. Results are provided in Appendix G.3.
4.2 REASONING ON MATH TASKS
Training Setup. We employ models of various sizes for validating the efficacy of our proposed method, including Qwen3-8B-Base, Qwen3-4B-Base, and DepsSek-RR--isiil--Wee--- the results of DeepSeek-1 1,5B are provided and analyzed in Appendix [F] due to space limitations.
where
All models are trained on the open-source DeepScaler dataset (Luo al. 2025). A binary re- ward is assigned by the open-source verification tool math baath.vee ve r Ey (Kydličck & Face (0255 upon the completion of LLM generation. We employ standard RLVR methods as baselines, including
GRPO (Shao et al. 2024), REINFORCE++ (Hu et al. 2025a). and DAPO (Yu et al.) 2025).
Evaluation. We select various widely-ackkoolleded math reasoning benchmarks: AIME24 (MAA
2024), AIME25 (MAA) 2025), HMMT25 (Balunović et al.] 2025). OlympiadBench (He et al.
2024 , AMC23 (O- 2024), and MATH500 (Hendrycks et al.] 2021), along with the O.O.D (Rein et al. 2024). We report pass @ 1 and @ for comprehensive analysis, where pass @ measures diversity and the reasoning boundary (uue et al. 2025). With in- creased diversity, the model has a higher probability of discovering a correct reasoning path within attempts. More details about the experimental setup can be found in the Appendix G.I
benchmark
Table I:
@ass@1 results across different methods on mathematical and O.O.D benchmarks. The highest and the second-best scores are shown in bold and underlined, respectively.
Mathematical
O.O.D
P
AIME 2024
AIME 2025
HMMT 2025
Olympiad Beach
AMC 2023
MATH 500
GPQA diamond
Avg.
Qwen3-4B-Ber 4B-Base
Base Model
4.9
0.8
27.3
55.6
9.7
20.3
GRPO
16.4
9.4
2.4
43.6
$7.0
79.9
38.7
35.3
DAPO
0.7
78.4
REINFORCE+
14,8
7.8
2.8
42,3
57.9
76.8
31.8
33.5
ROVER (Ours)
17.6
12.6
3.1
45,4 +88.1
$7.1
1-19
80.5
+:244
39,5
1:+29.8
36.5
1111.22
Qwen Biven3-8B-Bose
Base Model
11.5
8.8
0.8
34.7
48.1
8
29.1
28.8
GRPO
16.8
15.1
4.8
48.6
66.9
81.9
43.8
39.7
DAPO
20.8
15.2
3.6
49.0
67.9
84.3
46.6
41.1
REINFORCE++
19.4
16.7
7.1
47.6
63.5
83.6
46.3
40.6
ROVER (Ours)
30.6
4:9.1
22.7
13.9
14.6
1338
56.4
421.7
74,8
26.7
89.6
+-20
50.2 1 4:1.1
48.4
419.6
4.2.
PERFORMANCE ANALYSIS
ROVER consistently outperforms all RL baselines in terms of average pass@1. As detailed in Table I ROVER consistently outperforms standard RL methods across all model sizes. For the Qwen3- Base model, ROVER achieves pass@) improvements of +7.3 and +8.2 over the strongest baseline, averaged on all benchmarks and on the subset of AIME24, AIME25 and HMMT25, re- spectively. The superiority of our method over baseline methods becomes more pronounced on increasingly challenging tasks. Notably, for Qwen3-8B- Dwen3-8B-Base, ROVER delivers substantial relative improvements of +47.1 % on AIME24 and +35.9% on AIME25 over the pers-rerfonnnn baseline. On HMMT25, ROVER nearly doubles the performance of the strongest baseline, REINFORCE++.
ROVER significantly improves p. The average pass@k & over a dataset reflects the proportion of problems model can potentially solve within & trials, serving the model's reasoning breadth and diversity. To demonstrate the effectiveness of our method in incentivizing reasoning diversity, we compare ROVER with baselines by scaling pass@k from
robust evaluation metric of
8

---

Published as a conference paper at ICLR 2026
(AA)
Divertyy 1n erokerr Groonggee
ROVER
UVI des
unty
535
(a) AIME 2024 Figure 9: p k of ROVER and baselines on Qwen3-8B-Bas
(b) AIME 2025
(c) HMMT 2025
Figure 10: Quality- Diversity tradeoff.
to 256. Consistent with previous observations (Yue et al. 2025 Li et al. 2025a), the results in Fig. 9 reveal that while standard RL baseline methods enhance pass @ I, their performance quickly saturates and plateaus, ultimately underperforming the base model at large values. For example, trend that is also observed on HMMT25 for k > 32. In contrast, our method demonstrates sustained and sigmificant performance gains & increases, consistently surpassing all the baselines and the base model (+16.8 over the best baseline on pass 2266 averaged on AIME24, AIME25 and HMMT25). This advantage is particularly pronounced on the most challenging HMMT25 task, where our method's 's pass @ k score continues to accelerate while all baselines have saturated. We attribute the improved pass mass@A k to ROVER's ability to maintain a relatively higher entropy during training (see Fig. 22). which ensures sustained exploration of different reasoning strategies and enhances reasoning diversity.
DAPO even shows worse performance on AIME25 after &
ROVER shows remarkable generalization on O.O.D tasks. To further evaluate the generalization capability of ROVER, we incorporate the GPQA-diamond benchmark, a challenging unrhh-uneelated task containing 198 graduate-level questions in biology, physics, and chemistry. The results in Table [ demonstrate ROVER's 's stronger generalization beyond the training distribution, achieving the best performance on the unseen GPQA-diamond benchmark.
4.2.2 DIVERSITY ANALYSIS
ROVER possesses the highest diversity across differrntt metries. To quantify reasoning diver- sity, we employ the "number of distinct strategics" metric from NoveltyBench (Zhang et al. 2025c). ).). Specifically, we sample up 32 correct responses for each problem from the AIME24 datasets, and leverage Claude-3.5-Somel sponse pairs (template in Fig. 27). A higher number of distinct strategies (classes) indicates greater reasoning diversity. We report the results in Fig. 10 (with a 0.6 decoding temperature) and the results across different decoding temperatures in Fig. 24 From Fig. TO, observe that ROVER demon- strates relative diversity improvements of +6.8 % and +20.5 % when compared with GRPO and the average of all three basclines, respectively. Conventional RL approaches struggle improve diver- sity merely through increasing sampling temperature during inference, while ROVER consistently improves the Pareto front between quality and diversity. For a more comprehensive quantitative analysis of generation diversity, we refer to Appendix G.4] which includes results for additional metries such as utility (Zhang et al. 2025c) and cosine distance (Fig. 25
as the LLM judger to determine strategic equivalence between these re-
4.2.3 BEHAVIORAL ANALYSIS
ROVER scales best at test-time due to maintained diversity. Test-time scaling has received signifi- cant attention due to its potential to enhance rea- soning performance, where majority voting is a fun- damental baseline for evaluating LLM scalability test-time (Liu et al., 20255). Fig. I1 confirms that ROVER's maj @ performance scales robustly, con- sistently improving upon the base model across all k values, even on the most challenging HMMT25 task. This superior scalability stems from ROVER's ability to maintain diverse distribution over valid reasoning paths, while baseline methods suffer from mode collapse, causing them to confidently converge on similar incorrect solutions and preventing performance gains from additional samples.
0110
0200
(a) AIME 2024 Figure 11: Maj@k k performance of ROVER and baselines Qwen3-8B-Baal
(b) HMMT 2025

---

Published as a conference paper at ICLR 2026
Enhanced reflection behaviors. To analyze the reasoning pat- terns learned via ROVER, we adopt the forking tokens defined in Wang et al. (2255) (see Table { and quantify the normal- ized frequency of these tokens in the generated outputs (256 rollouts per prompt on AIME24, AIME25, and HMMT25).
1.0
SOATE
1.00
GRRO
0.00
0.40
Fig. [12 shows models trained with ROVER generate sig- nificantly higher proportion of these forking tokens, particu- larly those associated with rethinking and self-correction (e.g., 'wait' and 'however'). As detailed in Fig. [8] ROVER en- courages the model to actively reflect upon, verify, and pivot between different reasoning strategies, rather than committing to a single reasoning path. Fig. 17 examines an AIME24 problem where ROVER discovers two additional novel strategies compared to the base and GRPO-trained models, showing ROVER's potential to push the reasoning boundary.
0.90
100 www.eaa.. Or SEiM
Figure 12: Comparison of reflec- tion frequency. ROVER outputs more reasoning-related tokens.
4.3 REASONING ON TOOL-USE TASKS
We further validate that ROVER performs well in tool-use scenarios. Specifically, following the Search-R1 Jin et al. 2025) setup, we fine-tune the Swwe2..5-3B-Bl. model on the combined train- ing sets of Natural Questions (NQ) and HotpotQA. We evaluate ROVER after 300 training steps. Table 11 presents the performance across multiple benchmarks, spanning both General QA and Multi-hop QA tasks. The results show that ROVER not only works effectively with tools but also outperforms the original GRPO in terms of average performance by 1.8%.
Table 2: Performance comparison on General QA and Multi-hop QA benchmarks.
General QA
Multi-hop QA
Avg.
NQ
TriviaQA
PopQA
HotpotQA
2wiki
Musique
Search-RI-Bas (GRPO) (Jin al. 2025
0.421
0.583
0.413 0.443
0.297
0.274
0.066
0.342
ROVER
0.477
0.600
0.290
0.286
0.069
0.360
Diversity comparison. To evaluate diversity, we generate 32 responses for each prompt from the Bamboogle benchmark with temperature set 1.0, using the cosine distance between agent sponses as the diversity metric. As shown in Table I2] ROVER demonstrates substantially better di- versity. An agent capable of generating diverse queries can retrieve more varied and comprehensive information from the corpus, therehy increasing its chances of discovering the key evidence required to answer prompt correctly. Conversely, model lacking diversity may fall into monotonous query patterns, leading to a failure loop where it repeatedly fetches the same unhelpful results from the search engine and fails to solve the problem.
Table 3: Diversity comparison evaluated on Bamboogle benchmark using cosine distance between agent responses.
Method
Cosine Distance
Search-R1- R1-Base (GRPO) (Jin et al. 2025 ROVER
0.042
0.065
CONCLUSION AND LIMITATION
We present ROVER, minimalist approach to RLVR that achieves high-quality and diverse reason- ing policies from uniformly random valuation, which eliminates the need for complex evaluation- improvement loops with superior performance and diversity compared to existing SOTA methods.
Our experiments are limited math reasoning tasks with models up to 8B parameters due re- stricted computational resources. The practical implementation of ROVER for scaling up to large action spaces and long horizons also introduces approximation. Although the empirical success suggests robustness in the underlying principles despite these approximations, an interesting future direction is to further bridge this gap. We consider these as opportunities to reconsider RLVR from the first principles, develop more robust simplified approaches, and extend ROVER to other tasks (from capable pre-trained models). We believe that our approach establishes a valuable founda- tion for future research by demonstrating the power of surprising simplification in this domain, and hope that it inspires future research to adapt and extend these insights to other structures while maintaining the core benefits of simplicity for high-quality performance and diversity preservation.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work of Haoran He, Yuxiao Ye, and Ling Pan is supported by National Natural Science Foun- dation of China 62406266. This work was supported by computing resources and infrastructure provided by StepFun. We are grateful to researchers from StepFun for their valuable feedback and contributions.
REPRODUCIBILITY STATEMENT
To ensure reproducibility, we provide detailed proofs our theoretical results in Appendix A De- tailed experimental setup and hyperparameters used during training and evaluation can be found in Appendix [.T] Moreover, we provide the anonymous codebase at pattps://anonymous. 4cpen. scieen///ieene///RRRR
REFERENCES
Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadace, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting sinforce--tylee optimization for learn ing from human feedback in llms. ACL (1), pp.
AI-MO.
Ame
2023.
http:///ugiinffee...//ttttttttt
aimo Limo-valle www.iaaliiaaiiaooo on- amc 2024.
Kavosh Asadi and Michael L Littman. An alternative softmax operator for reinforcement learning. In International Conference on Machine Learning, pp. 243-252. PMLR, 2017.
Mislav Balunović, Jasper Dekoninck, Ivo Petrov, Nikola Jovanović, and Martin Vechev. Matharena: Evaluating iims on uncontaminated math competitions. arXiv preprint 1xXiv:2505.2328. 2025.
Emmanuel Bengio, Moksh Jain, Maksym Korablyov, Doina Precup, and Yoshua Bengio. Flow network based generative models for non-iterative diverse candidate generation. Advances in neural information processing systems, 34:27381-27388 2021.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03377. 2021.
Minghan Chen, Guikun Chen, Wenguan Wang, and Yi Yang. Seed-grpo: Semantic entropy enhanced grpo for uncertainty-aware policy optimization. arXiv preprint arXiv:2505.12346. 2025a.
Yang Chen, Zhuolin Yang. Zihan Liu, Chankyu Lee, Peng Xu, Mohammad Shoeybi, Bryan Catan- zaro, and Wei Ping. Acereason-nemottrnn! Advancing math and code reasoning through rein- forcement learning. arXiv preprint arXiv:2505. 16400, 2025b.
Zhipeng Chen, Xiaobo Qin, Youbin Wu, Yue Ling, Qinghao Ye, Wayne Xin Zhao, and Guang Shi. Pass k training for adaptively balancing exploration and exploitation of large reasoning models. arXiv preprini arXiv:2508. 20t::25508....1.. 2025c.
Daixuan Cheng, Shaohan Huang, Xuekai Zhu, Bo Dai, Wayne Xin Zhao, Zhenliang Zhang, and Furu Wei. Reasoning with exploration: An entropy perspective. arXiv preprint arXiv:2506.14777
2025.
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capa- bilities. arXiv preprint arXiv:2507.06261 2025.
Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang, Yuxin Zuo. Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, et al. The entropy mechanism of reinforcement learning for reasoning language models. arXiv preprint 2rXiv:2505.226111 2025.

---

Published as a conference paper at ICLR 2026
Kanishk Gandhi, Ayush Chakravarthy, Anikait Singh, Nathan Lile, and Noah D Goodman. Cogni- tive behaviors that enable self improving reasoners, or, four habits of highly effective stars. arXiv preprint ar arXiv:2503.01307. 2025.
Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint N.Kiv:2501.12944 2025.
Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In International conference on machine learning, pp. 1352-1361.
PMLR, 2017.
Andre He, Daniel Fried, and Sean Welleck. Rewarding the unlikely: Lifting grpo beyond distribution sharpening, arXiv preprint arXiv:2506.02335. 2025a.
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al. Olympiadbench: challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. arXiv preprint arXiv:2402.140000 2024.
Haoran He, Emmanuel Bengio, Qingpeng Cai, and Ling Pan. Random policy evaluation uncov- policies of generative flow networks. In Forty-second International Conference Machine Learning, 2025b. BtR.sttss:/ hwtp///eeeevvveeeeeeeeeeeeeeeeee ew net. / forum? d=pbkwh70iihi
Jujie He, Jiacai Liu, Chris Yuhao Liu, Rui Yan, Chaojie Wang, Peng Cheng, Xiaoyu Zhang, Fuxiang Zhang, Jiacheng Xu, Wei Shen, et al. Skywork open reasoner technical report. arXiv preprint arXiv:2505.223 12, 2025c.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.0387. 2021.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. arXiv preprint A9Xi: 09751, 2019.
Jian Hu, Jason Klein Liu, Haotian Xu, and Wei Shen. Reinforce++: An efficient rIhf algorithm with robustness to both prompt and reward models. arXiv preprint arXiv:2501.03326. 2025a.
Jingeheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung-Yeung Shum. Open-reasoner-zert zero: An open source approach to scaling up reinforcement learning on the base model, 2025b. URLhttps: /wwxiviivvv./////a//ea/////aav.a org/ aas///////// 22002/53..00 24290
Audrey Huang, Adam Block, Dylan J Foster, Dhruv Rohatgi, Cyril Zhang, Max Simchowitz, Jor- dan T Ash, and Akshay Krishnamurthy. Self-improvement in language models: The sharpening mechanism. arXiv preprint arXiv:2412.01951, 2024.
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint arXiv:2412.1677 2024.
Yuxian Jiang, Yafu Li, Guanxu Chen, Dongrui Liu, Yu Cheng, and Jing Shao. Rethinking entropy regularization in large reasoning models. arXiv preprint arXiv:2509.251333 2025.
Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. Search-rl: Training lims to reason and leverage search engines with reinforcement learning, arXiv preprint trXiv:2503.09516 2025.
Woosuk Wwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium Operating Systems Principles, 2023.
Hynek Kydlíček and Hugging Face. Math-verify. MethhVerrs fy] 2025.
hetps: ccm htan/hhggingfface/
12

---

Published as a conference paper at ICLR 2026
Cassidy Laidlaw, Stuart J Russell, and Anca Dragan. Bridging r1 theory and practice with the effective horizon. Advances in Neural Information Processing Systems, 36:58953 59007,, 2023.
Tianjian Li, Yiming Zhang, Ping Yu, Swarnadeep Saha, Daniel Khashabi, Jason Weston, Jack Lan- chantin, and Tianlu Wang. Jointly reinforcing diversity and quality in language model generations. arXiv preprint ar XXxv:2500002544, 2025a. URL https: /ttp://aaivv../.vv/.aie org
02534
Yi-Chen Li, Tian Xu, Yang Yu, Xuqin Zhang, Xiong-Hur Chen, Zhongxiang Ling, Ningjing Chao, Lei Yuan, and Zhi-Hua Zhou. Generalist reward models: Found inside large language models. arXiv preprint 2506:25506.23335 2025b.
Ziniu Li, Tian Xu, Yushun Zhang. Zhihang Lin, Yang Yu, Ruoyu Sun, and Zhi- Luo. Remax: A simple, effective, and efficient reinforcement learning method for aligning large language models. In ICMI, 2024.
Xiao Liang, Zhongzhi Li, Yeyun Gong, Yelong Shen, Ying Nian Wu, Zhijiang Guo, and Weizhu Chen. Beyond pass @ 1: Self-play with variational problem synthesis sustains rlvr. arXiv preprint arXiv:2508. 14029, 2025.
Michael L Littman and Csaba Szepesvári. A generalized reinforcement-lerming model: Conver- gence and applications. In ICML, volume 96, pp. 310-318, 1996.
Mingjic Liu, Shizhe Diao, Ximing Lu, Jian Hu, Xin Dong, Yejin Choi, Jan Kautz, and Yi Dong. Prorl: Prolonged reinforcement learning expands reasoning boundaries in large language models. arXiv preprint (tli::2555.24666 2025a.
Runze Liu, Junqi Gao, Jian Zhao, Kaiyan Zhang, Xiu Li, Biqing Qi, Wanli Ouyang, and Bowen Zhou. Can lb LLM surpass 405b LLM? rethinking compute-optimal test-time scaling. In Work- shop on Reasoning and Planning for Large Language Models, 2025b.
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding rl-zero-like training: A critical perspective. arXiv preprint 1xXiv:5503.20883 2025c.
Zihe Liu, Jiashun Liu, Yancheng He, Weixun Wang, Jiaheng Liu, Ling Pan, Xinyu Hu, Shaopan Xiong, Ju Huang, Jian Hu, et al. Part Tricks or traps? deep dive into rl for Ilm reasoning. arXiv preprint 2:Xi::250.008211 2025d.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 1rXiv:1711.05101, 2017.
Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y. Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Li Erran Li, Raluca Ada Popa, and Ion Stoica. Deepscaler: Surpassing ol- preview with a 1.5b model by scaling rl. htt
site/
-RL-19681 00l---66110221160000006666033111
2025. Notion Blog.
MAA. American invitational mathematics examination aime, 2024.
MAA. American invitational mathematics examination aime, 2025.
Marlos C Machado, Marc G Bellemare, Erik Talvitie, Joel Veness, Matthew Hausknecht, and Michael Bowling. Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. Journal of Artificial Intelligence Research, 61:523-5660 2018.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 51 (18(754))55295555 533-533,95333. 2015.
Abhishek Naik, Yi Wan, Manan Tomar, and Richard S. Sutton. Reward centering. In Rein- forcement Learning Conference, 2024. URL htttp::/ net/ het/forum?id AVAcUmAhxT
13

---

Published as a conference paper at ICLR 2026
Jiayi Pan, Junjie Zhang, Xingyao Wang, Lifan Yuan, Hao Peng, and Alane Suhr. Tinyzero. TinyZero, 2025. Accessed: 2025-01-24.
Ling Pan, Qingpeng Cai, and Longbo Huang. Softmax deep double deterministic policy gradients. Advances in neural information processing systems, 33: 1767-7777,,77, 2020.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Di- rani, Julian Michael, and Samuel Bowman. Gpqa: A graduate-level google-prodt q&a bench- mark. In First Conference on Language Modeling, 2024.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 2017.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song. Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemati- cal reasoning in open language models. arXiv preprint urXiv:2402.03300, 2024.
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv: 2409.19255 2024.
Michal Shur-Ofry Ofry, Bar Horowitz-Amsalen Adir Rahamim, and Yonatan Belinkov. Growing a tail: Increasing output diversity in large language models. arXiv preprint arXiv:2411.02986 2024.
Chenglei Si, Diyi Yang, and Tatsunori Hashimoto. Can llms generate novel research ideas? large- scale human study with 100+ nlp researchers. arXiv preprint arXiv:2409.04 2024.
Richard S Sutton, Andrew G Barto, et al. Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998.
Qwen Team. Qwen2.5: A party of foundation models, September 2024, URL httphhtppp///vvttte wetthub.icbbloom http:////lloggmee....333 qwen2.
Qwen Team. Qwq-32b: Embracing the power of reinforcement learning, March 2025. URL htttt://wwwnl..omm
io wtt.olooo//...ooo...... bwtgb/oo////// qwq- 32b/
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q- learning. In Proceedings of the AAAI conference artificial intelligence, volume 30, 2016.
Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang, Xionghui Chen, Jianxin Yang, Zhenru Zhang, et al. Beyond the 80/20 rule: High-entropy minority tokens drive effeective reinforcement learning for Ilm reasoning. arXiv preprint arXiv:2506.01939, 2025.
Ronald J Williams.
Simple statistical learning. Machine learning, 8(3):229-255) 1992.
algorithms for connectionist reinforcement
Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao. Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025a.
Chenghao Yang and Ari Holtzman. How alignment shrinks the generative horizon. arXiv preprint arXiv:2506. 17871, 2025.
Zhicheng Yang. Zhijiang Guo, Yinya Huang, Yongxin Wang, Dongehun Xie, Yiwei Wang, Xiaodan Liang, and Jing Tang. brethhbreadth synergy in rlvr: Unlocking Ilm reasoning gains with adaptive exploration. arXiv preprint arXiv:2508. 13755, 2025b.
Jian Yao, Ran Cheng, Xingyu Wu, Jibin Wu, and Kay Chen Tan. Diversity-aware policy optimization for large language model reasoning, arXiv preprint urXiv:2505.23433 2025.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu. Lingjun Liu, et al. Dapo: An open-source Ilm reinforcement learning system at scale. arXiv preprint arXiv:2503. 14476, 2025.
14

---

Published as a conference paper at ICLR 2026
Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Shiji Song, and Gao Huang. Does inforcement learning really incentivize reasoning capacity in llms beyond the base model? arXiv preprint arXiv:2504.13837 2025.
Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl- 20o: Investigating and taming zero reinforcement learning for open base models in the wild. In Second Conference on Language Modeling. 2025.
Qingyang Zhang, Haitao Wu, Changqing Zhang, Peilin Zhao, and Yatao Bian. Right question is already half the answer: Fully unsupervised IIm reasoning incentivization. arXiv preprint arXiv:2504.05812, 2025a.
Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, Fei Huang, and Jingren Zhou. Qwen3 embedding: Advanc- ing text embedding and reranking through foundation models. arXiv preprini 17X7 2025b.
Yiming Zhang, Harshita Diddee, Susan Holm, Hanchen Liu, Xinyue Liu, Vinay Samuel, Barry Wang, and Daphne Ippolito. Noveltybench: Evaluating language models for humanlike diversity. arXiv preprint arXiv:2504.000 50..0020........ 2025c.
Yuzhong Zhao, Yue Liu, Junpeng Liu, Jingye Chen, Xun Wu, Yaru Hao, Tengchao Lv, Shao- han Huang, Lei Cui, Qixiang Ye, al. Geometric-mean policy optimization. arXiv preprint arXiv:2507.20673. 2025.
Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao. Kai Dang, Yuqiong Liu, Rui Men, An Yang, et al. Group sequence policy optimization. arXiv preprint arXiv:2507.18071, 2025.
15

---

Published as a conference paper at ICLR 2026
A PROOFS IN $ 3.1
A.1 PROOF OF THEOREM
Theorem Consider an episodic finite-horizon episodic MDP with binary terminal rewards R(s) E {0, RJ where R > 0 (R. for a correct solution, O otherwise). Let Tu be a uniform policy, and let Q*a denote its Q-function. Define the greedy policy with respect to QT by Tgroody (8) arg max QT (8, a). Then groody the optimal policy.
Proof. As the underlying graph is tree, starting from so under policy Tgreedy gives unique chain So - + ... + Sn- By definition, for any state-action pair (s. a). if the subtree below (s, a) does not contain correct terminal state, then QT=(s; a) = conversely, if its subtree contains a correct terminal state, then QTH (s, a) > 0. Therefore, so we choose do arg max, QT (s, a), the next state 81 will necessarily lie on a path that reaches a correct terminal state. We keep proceeding until 8n-1, and Tgreedy (a) -n1) arg max, QT ($x-1, a) also selects the optimal action (as ($n-1,0, = H(sn-1 an- = R).
A.2 PROOF OF THEOREM 2
Theorem 2 Consider the same MDP M. and let QT* () denote the Q-function under the uniform random policy Ta from state-action pair (s,a), N(s) = Ka Q*=(s, ().a) =0}1 be the number of zero-valued actions at state A(s) be the number of available actions at state s, and P denotes the set of key states where both optimal and suboptimal actions exist, i.e. "x.(Q""(s,a)/0" P = fs 1 (()) I A(s) 1}. Given the softmax policy Tz(a|s) = exp((rr (x.a")"uss with temperature p > O. and (s) is the probability of reaching s from su with the policy the value function of the induced policy satisfies the following lower bound: (**(su) 2
R (1- E.EE P (s|80) Mes)eexpmmxx, Q=a(s,a]/p)
M(s)
Proof. Let sample trajectories from the initial state so using policy л.. For any incorrect trajec- tory r that achieves a reward value (one that fails to reach a correct terminal state with positive reward R), there must exist at least one key state along For each let denote the last key state along T.
The probability of trajectory can be factored as:
Pr(T) = "r* 180) II
T.. (ap(s)))
(2)
12t-
where t, denotes the index of state s, in the trajectory sequence.
Let Ta, denote the set of all incorrect trajectories, then we have that
Pr(Tw) Σ Pr(r).
TET
For any key state s E P. let (ss) denote the set of incorrect trajectories for which s the last key state. Since the underlying MDP M has tree structure, the sets {T(s)) form a partition of T Therefore, we have that
Pr(Tw)) Σ (s(oo))
Σ
II (a (2),
(4)
sEP
16

---

Published as a conference paper at ICLR 2026
As the state is the last key state any trajectory in T(s), we have that
(aaal)))) = Pr(Q"* (s, (s = 0|s, ma), TET(s) f2t
Σ
II
(5)
where Pr(Q""(s,a) = 0|s, nx) is the probability that policy Va selects an action with zero Q-value state
By the definition of the softmax policy, we have that
Pr(Q (s, a) 0(5,
E
Ty (a|s)
Tz (s,4)=0
"sp(Q"(ssa)//
Σ
Σn exp((xTm (s,
anQTM (s,a)=0
((s)
(8)
N(s) + Em (x. exp(Qru((,())
N(s)
(9)
((s) + exp(max,/ Q (s, a
Combing Eq. ④4 Eq. 5). and Eq. 9 we have that
N(s) N(s) exp(max,'' QT (s, a')p)
Pr(Tw) S Σ Pr (8|s0)
(10)
PEP
By definition, the value function of n, is related to the probability of correct trajectories:
(so) = (1 - Pr(Tw))
(11)
Substituting our upper bound on Pr(Tw), we have that
(*0) 2 R (1 1- Σ Pr Pr N(8) exp(maxx (sn (s)aa)))))))))))))))))){))))((
N(s)
(12)
For any key state E P. we have that max. QT d') > 0 by definition. As + O, the right-hand side in Eq. (12 converges to R, which is the optimal value.
A.3 ADDITIONAL THEORETICAL ANALYSIS
Theorem 3. Consider a finite-horizon MDP with deterministic transitions and tree-structured state space. Let n. be uniform policy, QT= denote its Q-function, and the discount factor 1. For any non-terminal state s, define C to be the set of terminal states with correct rewards, T(s. the next state reached by taking action a from state and T(s) the of all possible trajectories from state to terminal states. Define two disjoint subsets of actions at state s, where A+(s) = {a e does not ends in C}. The intermediate rewards are given by reasonable Process Reward Model (PRM) r. Terminal rewards R indicate a correct solution (R > 0), and otherwise. PRM is considered reasonable the following conditions hold for all non-terminal states S: Va A+ Ya' (s): T(s, a r(s, a+), and (ïi) Va,, (s), Va_ E A (s), VT E TT )), VT+ T(T a ))) Eier- Ti < Eier+ Tú. If these reasonable PRM conditions hold, then the greedy policy with respect to QT defined by Tgreody (s) = arg max, Q (s, a) always selects actions leading to correct terminal state.
A| E T(T(s, a)) such that ends C). and A (s) {a AlVT T(T(s,
17

---

Published as a conference paper at ICLR 2026
Proof. For any state s, the state-action Q-function under the uniform random policy mu be expressed as
a=) = r() V (T(,sa)))
(13)
where VTv (s)
P(T) CiEr r..
For any at E (+ and a (s)s,, the second PRM condition implies that VT E a))(s, VT+ E T(T(s,tt)): Eier To Eier! Y4. [This condition ensures that the cu- mulative reward of any sub-trajectory starting from T(s,a (1s,a) (a child state leading to incorrect solutions only) is less than any sub-trajectory starting from ass,at) (a child state leading at least a correct solution).)]
Therefore, have that (T=(((ssa < (TT
Based on the first PRM condition, we have that ((s,a-) < r(s, r(s,at), (This condition ensures that actions maintaining at least a path to correct solutions receive higher intermediate rewards than those leading only to incorrect solutions.)
Therefore, obtain that
QT as r(s, + (T=(((s, r(s,a()++ (V"s,at)+VV<((C)) - Q**(saat). (14)
Thus, at any state 8, Tgreedy (s) = arg max, (s. a) will always select actions from (s) over actions from A A (s) (which means that consistently choose actions that lead to states from which at least one sub-trajectory to correct terminal state exists).
Due to a reasonable PRM, by induction, following the greedy policy Tgreedy with respect to Q* always navigates a correct solution, which guarantees its solution optimality.
B GRADIENT ANALYSIS
In this section, analyze the relationship between method and existing policy optimization methods from the gradient perspective.
Proposition 1 Assume only log na has parameters (i.e., LLM policy depends on 0). Define im- portance sampling ratio IS
Ta(a ne (als)' where Teaa is the behavior policy. Denote as our mean- centered reward. Then the gradient of ROVER's objective takes the following form, which is similar to policy-gred policy-gradient:
VoLROVER Es,,,e'>P [((F Q') - loog IS) Vo log [o(a]s)], where Q' = -
Σ Q(a'ss))
a'Ev
(15)
Proof. Recall the details of our method provided
3.2
ROVER has the following loss function:
LROVER F.
(i+
Σ Q(a'ls') Q(s,
(16)
V
18

---

Published as a conference paper at ICLR 2026
Let
u == +
Σ Q(a'(s') - Q(s,aa)
M
a'EV
+
Σ
Q(a'(s') - (log Te - log 18
M
d'EV
(17)
= +
Σ Q(a'(s') - -log
T
M
d'EV
f + Q(d'(s) (() log IS.
d'ev
Then the gradient is
VoCROVER s' Vouj
(18)
Given that Q' = 1b En'Ev Q(a'ls)) (5), where the gradient of Q' is stopped (see Alg. and a0. does not involve gradient backpropagation, by combining Eq. 17 and Eq. 18 we have:
V CRoVER BR
( +
Σ
Qa''ss') - V
V
A'EV
(19)
=
(i+ Σ Q(a'ls') l0
Note the gradient of typical policy optimization method. i.c., GRPO Shao al. 2024). is
VeLGRPO -A.,alA111 IS V = [A IS V, log (
(20)
Therefore, we have the following key observation:
Both gradients share the term Ve log ne (core of policy gradient).
When importance sampling ratio IS - 1 (small policy update), i.e., log IS -> 0, so:
CROVER R [( Q') Vo log To]. Vo E Vo log mp].
These two objectives can be approximately equal if we remove the term Q' in ROVER and the advantage in GRPO is normalized without the standard deviation term
B. EMPIRICAL JUSTIFICATION
Ablation on the term Q'_ The Bellman target used for Q-value updates is composed of two com- ponents: centered reward, F. and the expected Q-value of the successor state under a uniform policy,
Q' = M Eatt Ev Q(a++1 (84+1). )). We ablate the contribution of Q' in the Bellman target by scaling with coefficient 3 = [0.0, 0.2, 1.0, 5.0]. The results show that this term is essential: removing it (8 0) causes collapse in entropy and response length (see Fig. [3Cc) and [3(d). leading to
sharp drop in pass@A & performance. Conversely, an overly dominant Q-term (5 5.0) diminishes the reward signal, which also degrades performance. Crucially, as shown in Fig. 13(a) and [3(b) our method not sensitive to the precise scaling of this term, with performance remaining stable across a wide range (8 from 0.2 to 1.0). By default, we set 1.0 in other experiments. Detailed pass @ & performances under different / values are shown in Fig.
14
19

---

Published as a conference paper at ICLR 2026
111 1.19
0.175
0.51
0.150
0.52
2
0.30
0.48
0.100
20302 28 RAALAS
0.75
0.43
20 2012--00------2222
3.12--m-a
0.
02
100
5.0
0.2
1.0
2.0
Coofficiont
Confficion: F
(a)
(b)
(C)
(d)
Figure 13: (a)&bb) Impact of coefficient 3 in ROVER on pass @ass@1 & pass@64, average performance on AIME24, AIME25, HMMT25 reported. The X-axis is on a log scale. (c)&(d); Entropy and response length curves throughout training. All experiments are conducted on Qwen3-4B-Ban with
L.M decoding temperature 1.0, and trained for 300 steps.
0.0
0.7
0.5
0.7
0.4
HOYER (0-3
Anysa (1-42)
0.5
Ceer3-4t-Sew
0.4
0.3
04
0.5
HoMell (Bo1
Hotee (jei it
toven (2-0
roven (p-0 c
sOVSa (0:0
(1
0.1
0.0
00
(a) AIME 2024
(b) AIME 2025
(c) HMMT 2025
Figure 14: pass@A k performances under different value of coefficient /3 in ROVER All experiments are conducted on Qwen3-4B-Base and trained for 300 steps.
C GENERALIZABILITT OF ROVER
Null 0.06 0.06 0.06 0.06
Null 0.06 0.06 0.06 0.07
A 0.00 0.00 0.25 0.00
0.02 0.01 0.20 0.01
0.7
-.8
B 0.00 0.00 0.00 0.25
B
0.03 0.02 0.01 0.21
0.6
States
C 0.25 0.00 0.00 0.00
0.8
Q-Valu
States
0.21 0.01 0.02 0.01
Valuue
D
0.00 0.25 0.00 0.00
0.03 0.21 0.02 0.01
0.4
AC
0.00 0.00 0.00 1.00
0.4
AC
0.03 0.08 0.05 0.80
BD
0.00
0.00 1.00 0.00
BD
0.09
0.07 0.81 0.05
0.2
CA
0.00
1.00
0.00
0.00
0.2
CA
0.05
0.80 0.09 0.07
0.1
DB
1.00
0.00
0.00 0.00
DB
0.84
0.06 0.05 0.08 0.08
A
C Actions
B
D
A
B
D
Actions
(a) Intermediate Rewards
(b) Stochastic Transitions
Figure 15: Q value maps ROVER across different environmental settings within a tree-structured MDP. (a) ROVER generalizes well to environments with intermediate rewards, successfully learning optimal values. (b) ROVER also handles stochastic environments well, where the Q values of optimal actions are much larger than sub-optimal actions.
20

---

Published as a conference paper at ICLR 2026
C.I GENERALIZING TO ENVIRONMENTS WITH INTERMEDIATE REWARDS
Since widely-used RLVR benchmarks basically follow the deterministic dynamic transitions with terminal reward that indicates the correctness, we modify the didactic MDP in Fig. [(a] in our paper to validate the practical effectiveness of ROVER for intermediate rewards and stochas- tic transitions. For intermediate rewards, we assign reward of 1 the intermediate states {A. B, C, D, AC, BD, CA, DB) that can lead to the correct terminal state. In this environment, ROVER still captures all optimal solutions with 100% success rate. We find that the Q value map
learned by ROVER is optimal, as shown in Fig. 15
C.2 GENERALIZING TO ENVIRONMENTS WITH STOCHASTIC TRANSITIONS
Following the stochastic dynamics defined in environments like Atari (Machado et al. 2018). we modify the environment dynamics be stochastic by setting P(s a, s, a) 0.75, P(s s s a) 0.25, where a' denotes the action different from the chosen action a. This means that there is 0.25 probability for the agent to travel a random path, thereby ensuring the dynamic stochastic. Our empirical results demonstrate that ROVER can effectively handle this kind of stochasticity. If the dynamic becomes deterministic at test time, ROVER successfully captures all modes with a 100% success rate. The learned Q value map is still near-optimal, shown in Fig. [5] If the dynamic remains stochastic at test time, ROVER achieves 77% success rate, which is still high considering the 25% randomness in the environment.
C.3 GENERALIZING TO ENVIRONMENTS WITH GRAPH-STRUCTURED MDP
To evaluate ROVER's generalizability to graph-structured environments, we insert graph connec- tion between (A, B, C, D}, which means a state in {A, B, C, D} can have multiple parent nodes, leading to graph-structured MDP. We provide illustration of this graph structure in Fig 16 Our empirical results show that ROVER is still optimal under graph-structured MDPs, achieving 100% success rate with all four modes covered.
Nall
0.06
0.06
0.06
0.06
0.00
0.00
0.00
0.00
A
0.00
0.00
0.25
0.00
0.00
0.06
0.06
0.06
088
B
0.00
0.00
0.00
0.25
0.06
0.00
0.06
0.06
Skales
0.25
0.00
0.00
0.00
0.06
0.06
0.00
0.06
Graph Connection
0.00
0.25
0.00
0.00
0.06
0.06
0.06
0.00
AC
0.00
0.00
0.00
1.00
0.00
0.00
0.00
0.00
04
BD
0.00
0.00
1.00
0.00
0.00
0.00
0.00
0.00
CA
0.00
1.00
0.00
0.00
0.00
0.00
0.00
0.00
DB
1.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
-00
B
B
4A Actions
(a) Graph Connection
(b) Q- Value Map
Figure 16: Q-value maps of ROVER on a graph-structured MDP. (a) States {A, B, C, D} connect with each other, thereby making a graph- structured MDP. (b) '#A', 'BB', '#C', '#D' denote the graph connection actions. We observe that ROVER still learns optimal Q-values under this graph structure.
D RELATED WORK
2025a Comanici et al. 2025) has found great success in post-training LLMs on verifialle tasks. To bypass the need for the value model of PPO (Schulman et al.) 2017). many actor-only variants have been proposed, such GRPO (Shao et al. 2024), RLOO (Amaadian et al. 2024), ReMax (Li et al. 2024). and RE- INFORCE+ Hu et al. 2025a). Nevertheless, leading algorithms like GRPO still exhibit unstable learning dynamics and prone model collapse. Recent works propose add various heuristics on advantage normalization (Liu et al.) 2025c] Zheng et al., (2025] Zhao et al. 2025). clipping ra- [Lu et 2025a), entropy loss (He et al.
RL with verifiable rewards (RLVR) (Guo et al.
2025 Team 2025 Yang et al.
tio Yu et al. 2025), KI. regulariizttion
2025e Zhang al.)
21

---

Published as a conference paper at ICLR 2026
2025a).
reward bonous (He al. 2025a Li al. 2025a Yao et al. 2025 Cheng et al. 2025 Chen 2025a c). data augmentation Yang et al. 2025b Liang et al. 2025 and others (Cui et al. Wang al. 2025). Crucially, these existing works are still constrained by the same surro- PPO objective, and often necessitate complex, case-specific tuning (Liu 2025d). Our work departs from this paradigm, proposing method grounded in random policy al.
et al.
2025
gate.
holicy-gradient-basee
valuation that offers a minimalist yet theoretically guaranteed approach to fine-tuning LLMs.
Table 4: Summarization of MDP structures between different tasks, considering the discrete Atari task from traditional RL and the countdown task from RLVR. While traditional RL. tasks have smaller spaces and shorter horizons, the underlying MDP structure can be much more complex than LLM RLVR tasks that feature deterministic, episodic, tree-structured MDPs (which have larger spaces and longer horizons and leverage powerful pre-trained model that can navigate in the large space).
Feature
Traditional RL Tasks (e.g,, Atari)
LLM Reasoning Tasks (e.g., Countdown)
Transition dynamics
Stochastie/Detrrmnnstie
Deterministic
Reward function
Stochastic/Determinsssee
Deterministic
Intermediate/Fpp pinisii
Episodic
State space structure Observability
Graph-like (often be cyclic)
Tree-like (no eycles)
Can be partial observable
Fully observable
Action space
Smaller
Larger
Horizon
Shorter
Longer
E THE COUNTDOWN TASK
Task Details. Countdown Gandhi al. 2025) a math reasoning task capable of evaluating the
arithmetic capabilities of LLMs. Below illustrates the toy example of the countdown task:
nums: [19, 36, 55, 7].
target:
65,
answer:
55 + 36 - 7 19,
where the LLM should find the correct solution using the given numbers and basic arithmetic op- erations (+, x, +). The simplicity of the Countdown's 's reasoning path, yet challenging for small LLMs to solve effectively, makes it an accessible test bed for math reasoning.
Training Details. We use the training and testing dataset provided by TinyZero (aan al. 2025 The training dataset contains 327680 problems, and the testing dataset contains 1024 unseen prob- lems. A reward of given if the LLM finds the correct equation; Otherwise, it receives a zero reward. We set the batch size to 128 and the mini batch size to 64 during training. Optimization is conducted by an Adam W (Loshchilov & Hutter (0117) where learning rate is 10 The response length is set to Ik for both training and evaluation. We rollout 5 responses per prompt to calculate the mean-centered reward. Other configurations follow the default setting of TinyZero (Pan et al., 2025). For the baseline of GRPO with higher technique,
clip ratio Cow 0.2 and
Chigh = 0.4. Note that all the experimental settings across different methods remain the same for fair comparison.
F RESULTS OF MATH TASKS ON DEEPSBEK-1.55 1.5B
We provide the details of the training setup on BeepSeek-R1-Diiee -i--isilll-Qeen--.. model (Guo al., 2025) as follows.
We employ the datasets provided by DeepScaler (Luo al. 2025 which contains 40k verifiable math questions.
Built upon the veRI. infra (Sheng et al. 2024 we set batch size 128 and mini-batch size 64.
22

---

Published as a conference paper at ICLR 2026
Table 5: Results of SeepSee--R--istill--eeeees
on typical math competition tasks. The high
and the second-bes best scores are shown in bold and underlined, respectively.
Pass@1
Pass@64
Models
AIME24
AIME25
AMC23
MATH
AIME24
AIME25
AMC23
MATH
DeepSeek- All-DistlllQuee Qwen-1.weee 1.5B (Guo et 2025
29.3
24.3
62.5
82.9
79.8
58.3
92.9
97.3
DeepSealeR-1., 1.5B (Lao 2025
41.6
30.8
73.4
87.7
78.5
62.9
95.0
96.8
ProRLy2-Qaen- Llu 2025a
52.6
81.5
79.2
59.7
94.3
96.
ROVER (Ours)
42.2
31.2
74.3
88.3
80.6
64.4
95.2
97.1
use the AdamW (Loshchilov & Hutter 2017) 10 for gradient backpropagation.
optimizer with constant learning rate of 1 x
We rollout responses per prompt calculate the mean-centered reward F.
Following DeepScaler (Luo et al., 2025), we first train DeepSeek-R1 -iii------illl--e------ for 1k steps with a 8k response length. Then scale the response to 16k for an additional Ik training steps. Experiments are conducted on 8 H200 GPUs for around 5 days.
Following the evaluation scripts provided by veRL, we sampling temperature of 0.6, nucleus sampling (Holtzman et al.] 2019) with top-p = 0.95, and a maximum response length of 24k for evaluation. We evaluate our ROVER-trained model and previous SOTA models such DeepScaler- 1.5B (Luo et al. 2025) and ProRLv2-1.5B (Liu et al. 2025a) on AIME24, AIME25, AMC23, and MATH tasks. We rollout 128 responses per prompt for each task, and report both the pass@ (avg@) 128) and pass @ 64 (calculated by an unbiased estimator (Chen et al. 2021h) for comprehen- sive comparison.
From the results summarized in Table we observe that ROVER achieves the best performance in terms of both pass@) and pass@64 scores compared with DeepScaler, which trained on the same dataset as ours. Note that the comparison with ProRLv2 is not fair since ROVER uses more than 3x smaller datasets (40k (ours) vs. 136k (ProRLv2)). Moreover, the training ROVER only lasts for around 960 GPU hours, while ProRLv2 is trained for 16k GPU hours. However, thanks to the better reasoning diversity brought by our method, ROVER can achieve higher scores than ProRLv2 on pass @ 64.
G RESULTS OF MATH TASKS ON QWEN MODELS
G.1 TRAINING AND EVALUATION DETAILS
Training Details. To ensure a fair comparison, both ROVER and baselines trained using the same learning rate, batch size, and training steps (see T). We fix 600 training steps for ROVER and baselines. During each training step, 128 x 8 samples are involved to calculate gradients. The computational requirements are approximately 1,280 GPU hours for experiments initialized with Qwen3-SB-Base and 832 GPU hours for those with Qwen3-4B-Base.
Evaluation Details. Default hyperparameters evaluation are summarized in Table To compute average pass@1 1, we sample 256 independent runs for AIME24, AIME25, HMMT25, and AMC23 for comprehensive evaluation to reduce the variance introduced by the relatively small sizes of these benchmarks, while 10 runs are sufficient for the larger OlympiadBench, MATH500, and GPQA- diamond benchmark.
Baselines. We compare ROVER with the following baselines:
GRPO (Shao et al. 2024): It employs standard implementation framework with token- level mean aggregation loss, serving as fundamental baseline for LLM reinforcement learning.
DAPO Yuu et al. 2025): It extends GRPO by introducing several techniques enhance LLM training efficiency. These include clip-higher, dynamic sampling, and overlong re- ward shaping. We set <iow = 0.2, Ehigh 0.28.
23

---

Published as a conference paper at ICLR 2026
Table 6:
Default hyperparameters for RL train-
ing.
Hyper-parameter
Value
Table 7:
Default hyperparameters for evaluation.
Temperature
0.6
Hyper-parameter Value
Response length
8 x 1024
Temperature
0.6
Responses per prompt
8
Response length
24 x 1024
Train batch size
128
top-p
0.95
Mini batch size
32
PPO_epock
Learning rate
le - 6
Prompt Find the number of ways place a digit in each cell of 2x3 grid so that the sum of the two numbers formed by reading lefl to right is $9996 and the sum of the ihree numbers formed by reading to bottom is SBas, The grid below is exammpe of such anaingement because $8-381-8895
50-9-81-985 in@ubagginoraay00HH Whiine D
Whhlin Winl Winline Vand(arayy)] Lat's think stop step output the final
answer within libuved).
Qwen3-88-Bell B8-nnas-BBBBeeee-e and GRPO's Strategy 1
Qwwan-88-Baase 88--ase and GRPO's Strategy
Conditio 1: The of of the two numberr lornaed by nerding lall to right 500.
expuelaon 1.1((100x (100s o)
()- 4 (100e
(100a 10b c) (1000 10p 890.
oj- 11
simpplly gai
by reading top
110- 100
For each ic from 1o Find wald's V. (b, o) para that sataty V 100 10 lkq 10b 11c
(10a d) (1Cb (10c-10 This leade
Veu 10d th...
Aczonding stars and bars theorem the number nonoregaation integer soluttem is odeere wach owracle i digi ae leavorejh 1-3--)
Pettern foundi Sd5 from 9, vafue $d5
corresponds to $d5 valid combinations.
\binom(100(2) 1boxed(45).
Theewlon,
+
lboxed(45).
ROVER's Strategy
ROVER's Strategy 2
ROVER'S Strategy 3
ROVER's Strategy
Let' donote
Sibmracing equation from equariin
For Pack grid, we houn
thme key For wich
-5. 8.....2 b 3
and dividing by
cquations: inoiizonaa (100a werceal Jum (10l (til (100 i - BB.
T_1 a b d of
-
100
from
Since Vo) a dig (D
fis determince ned by: gF= 9 o
Now, we noad find the number of notenegation intagee aodutional it * 19 where V leg b 100 B-1, Using the inelusion-excluuion principle
-a ls must eaisdy: (a S-ce
Anthyaing
Number of vold ba poirs equats (8-r1)
(al being digit D-0), discover
New we need to ind the mamber of solution to a--ba b 8 whore a b. digit. This a classic "stars and bars" problem
inteaer
sxtadyngg inwpueity
exist only when a =
Pattern found: for $c$ from to each value of Sc$ corresponds to (9-Sc8) valid combinations.
9. each
Total
n'lenom WG2) 2101)
exactly corresponding
with the corndnent a solrtions Bi) non-negative integers
2. Solutions where one variable a 10.43
combinations of (d.e). Therefore:
mime d5 - res
limon + 2112
Tharsfor, the uua number is. + +1 1boxed(45).
210 - 165 1boxed(45).
Tota valid solutors
1boxed(45).
1binom(10)2)) (45).
Figure 17: Illustration of strategies discovered by Qwen3-8B--ase,, GRPO and ROVER Responses sharing the same color represent strategically identical approaches. While Qwen3-81B-Bee -Bsse and GRPO find two distinct strategies 1&2, ROVER not only discovers the same two strategies but also uncovers two additional strategies 3&4. For example, beyond discovering the Stars and Bars theorem (strategy 1). ROVER also discovered a solution based on the aeclusion-exclusiin principle (strategy 3), which demonstrates ROVER 's capability in pushing reasoning boundaries.
REINFORCE++ eHu al. 2025a):
Different from GRPO, incorporates global advantage normalization (across responses correspond to different prompts within a batch), resulting in an unbiased approach that significantly improves training stability. We implement the REINFORCE:++baselin version in this paper.
All baselines are rigorously implemented following the official veRL. recipes (Sheng et al. 2024).
G.2
CASE STUDIES
Discovered strategies comparison. To intuitively show the enhanced diversity of ROVER we present a representative prompt from AIME24 that holds multiple potentially feasible strategies. For each model, 32 samples are generated and subsequently clustered based on strategic equiva- lence using an LLM judge (the prompt of the LLM judge is given by Fig. 27). Representative CoT examples for each cluster are illustrated in Fig. 17
24

---

Published as a conference paper at ICLR 2026
Prompt
Prohability
Alice and tib play the following game stot of 5r5 tokens has before them The player take times wet Alice going first On each tun, the player remsvees citter $15 sakes $45 tokens from the stack. Whoever cenoves the last soken wiss. Find the number poséiive
Sacl lone then regardiess Alice's $20245 which there stmiagy Bob that guamico that by
will
final within (boxed
ROY
GRYO'S hightr
Response
aack
Aenual
Bood
Figure 18: Token probability differences between ROVER and GRPO (visualized by the heatmap). ROVER exhibits a significantly higher probabilities to tokens associated with reasoning contrasts or shifts, exemplified by" Wait
Token probability comparison. We visualize a case study to show the token probability differences between ROVER and GRPO in Fig. [8| representative prompt from AIME24 is selected), ROVER demonstrates higher probabilities for tokens indicating contrasts or shifts, particularly "wait", which facilitates the exploration of alternative reasoning paths, thereby contributing increased strategic diversity. The specific forking tokens mentioned in S are shown in Table g
Table 8: Forking token categories and their corresponding tokens.
Category
Tokens
mathematical_setup
suppose, assume, given, define wait, however, unless thus, also
coutrasts_shift progression addition
G.3 ADDITIONAL. EXPERIMENT RESULTS
Pass @ results Qwen3-4B-Bar Basee.. As shown in Fig. 19 similar results Qwen3-8B-Bal
ROVER demonstrates consistently superior pass@k performance on Qwen3-4B- -B--ase across all values, while other RL baselines drop when becomes higher.
novet
0.7
0.5
GEPO
Ower) 15- 1i-8soo
0.4
passak
pessggk
8.4
23
0.3
RONRR
0.2
ROVER
DAPO
DAPO
0.1
0.1
0.0
0.0
0.0
125
(a) AIME 2024
(b) AIME 2025
(c) HMMT 2025
Figure 19; pass@k performances of ROVER and baselines (Qwen3-4B-Base)
25

---

Published as a conference paper at ICLR 2026
Maj@k results. The supplemental results of maj @ performance AIME25 for Qwen3-8B- Base is shown in Fig. 20) To mitigate random variations in evaluation results, we adopt a repeated sampling approach for computing maj@ k: k responses are ran- domly sampled from the response collection, and this sampling procedure is repeated 1000 times with the average value re- ported.
0.32
0.33
DePO
SEPO
0.25
G.3.1 ABLATION OF TEMPERATURE P
Consistent with the findings on the countdown task in Fig. the training temperature serves as an exploration-explotaann trade-off. A large p (p = results in more stochastic behavior and constant entropy throughout training, which affects the per- formance (see Fig. 21p. Conversely, smaller p (p 0.01) leads to greedy and deterministic policy, which compromises diver- sity (e.g., reduced pass @ k)) for improved pass@) @1 performance. By default, we set = in other experiments.
Figure 20: Maj @k performance ROVER and baselines on AIME25 for Qwen3-8B-Base.
ROVER 00-0
0.00
0.35
Entrogy
0.225
0.00
09555
6.20
0.05
KOvER (.0)
ROVER (p-20)
6.05
ROVER Lo 4.00
0.03
100
150
200
250
300
Stop
(a)
(b)
Figure 21: Impact of temperature p. All experiments are conducted Qwen3-4B-Ban and trained for 300 steps. (a): pass@1 k results (average performance on AIME24, AIME25, HMMT25 are reported). (b): entropy curves throughout training.
G.3.2 TRAINING DYNAMICS
We present the training curves of entropy Fig. within training batch visualized in Fig. 23
22
The min, mean, and max values and Q'
0.32
ROVER
EOvE1 GRAD
0.06
Entropy
0.0c
0.02
0.22
300
400
202
100
(b)
Figure 22: Training curves of entropy for ROVER baselines. (a) (b) are results on Qwen3- 8B-Base and Qwen3-4B-Ban respectively. The entropy of ROVER is maintained at relatively higher level, and can even increase stably at later training stages, indicating expanded exploration space. In contrast, the entropy of baselines inevitably decreases to low level.
26

---

Published as a conference paper at ICLR 2026
0.32
Filmn bech!
0:04
FiMax in
4..3
0.34
0.00
Step
Step
Step
(a) F (Min in batch)
(b) F (Mean in batch)
(c) F (Max in batch)
2.000
(Pfeen batchh
2,000
0/ Mau in batch)
120
12
2,002
2,002
2.000
200
400
202
400
(d) Q' (Min in batch)
(c) Q' (Mean in batch)
(f) Q' (Max in batch)
Figure 23: Absolute scales of and Q' throughout training for ROVER (trained on Qwen-8B-Base
G.3.3 COMPARISON WITH ENTROPY REGULARIZATIN METHODS
To investigate the effectiveness of ROVER's diversity-enhancing capability, we implement an entropy-aware variant of GRPO comparison. Specifically, the GRPO objective is augmented with entropy term: r'(8, a) r(s, a) (=((s=)))), where H denotes the policy entropy and controls the strength of diversity encouragement. Following (Jiang et al. 2025), we set a = 0.005 and train the Qwen3-4B-Ban model to compare performance across methods.
As shown in Table while the entropy bonus successfully maintains higher entropy levels, it leads to performance degradation in GRPO. In contrast, ROVER achieves a superior balance, simultancously improving performance while preserving significantly higher entropy.
Table 9: Comparison with entropy regularization methods. Pass@ I is averaged over AIME24, AIME25, and HMMT25.
Pass@1 1
Entropy (step 300)
Owen3-4B-Bass + GRPO
9.40
0.008
Qwen3-4B-Ban + GRPO (a 0.005)
9.31
0.020
Qwen3-4B-Base + ROVER
11.10
0.055
G.4 MEASURING THE DIVERSITY OF LLM RESPONSES
In addition to the number of distinct strategies mentioned in 4 we additionally incorporate two diversity metrics for a comprehensive evaluation. These diversity metrics are introduced as follows.
No. of Distinct strategies (Zhang al. 2025c). It categorizes all generated responses into equiva- lent strategy classes and counts the total number of distinct classes.
Utility (Zhang al. 2025c). It combines diversity and quality using a user patience model where users have a probability of requesting additional generations. It rewards novel responses while ap- plying geometric decay account for diminishing user attention over multiple generations. Models capable generating multiple correct responses with distinct strategies will receive a higher utility score.
27

---

Published as a conference paper at ICLR 2026
Cosine Distance. We embed all responses using Swven-8B-Emmeedee Zhang et al. 2025b and compute the average pairwise cosine distance between response vectors. Higher distances indicate greater semantic diversity among generated responses. Specifically, given set of generated sponses [y1, V2...... yo }. let E(y)) E Rd denote the 1.2-normalized emhedding vector of response yi obtained from @wen3-8B-Embeedee The pairwise cosine similarity between responses yt and y is:
Sy, Uj) Bly)) Byy))
The average pairwise cosine similarity is:
n
$
S Uj). jzh
n(n 1)
Finally, the cosine distance is defined - S.
As supplement to Fig. 10 results
trade-off across t [.93,0.9, 1.2] are shown
in Fig. 24
Furthermore, we demonstrate the comparison on all three diversity metrics under different decoding temperatures in Fig. 25 ROVER consistently exhibits greater diversity across all decoding temper- atures.
Stratopess
BOVER
ROVER
Diversity Le of Distinet Strategies)
ROVER
Destrit Strateg
2.99
RERPORCE 1
RERFORCE-+
2:30
Dieerratyy
0.2 0.20
018
0.22
224
E21 022 020 024
Quality (Poss@1)
Quality (PosS(31)
Quality (Passg0)
(a) f 0.3
(b)
0.9
(c) 1.2
Figure 24: Quieity-diversity trade-off with different decoding temperature (AIME24).
Distinct Strategles 530
Distinct Strategles
Distinct Strategies
120
Datance
SoVER BANo
Utiity
Distance
Uuilty
Distance
0470
Quilty
0120
0900
GEP0
(a) 0.3
(b) 0.9
(c) 1.2
Figure 25: Comparison of multiple diversity metrics with different decoding temperatures (AIME24).
G.5 PROMPTS
We present the prompt template for RL training and evaluation in Fig. 26 and the prompt for LLM judger in Fig. 27
28

---

Published as a conference paper at ICLR 2026
1lim_staae >suser
qqusstion]
Please reason step by step, and put your final answer within Aboxed
im__nddl
itastart >assistant
Figure 26: Prompt template for RL training and evaluation. The base model uses the same prompt template as the trained model during evaluation.
H RESULTS OF MATH TASKS ON LLAMA MODELS
To demonstrate the broad applicability of ROVER across different model architectures, we extend the hard training
experiments beyond the Qwen family by training dataset from simpleRL-zoo Zeng et al. (2025), which consists of level 3-5 problems from the MATH training As shown in Table [O. ROVER achieves an avg@5 accuracy of 67.8% on MATH500, outperforming GRPO by 8.8%. These results further validate the generalizability of ROVER to diverse base models.
Table 10: Performance comparison on MATH500 when using Llama3. 1-8B-Innstucc
MATH500 (5vg@5)
Llama3. Baamma3.1-BB-.. I-8B-Instruct ROVER
67.8
Llama3. 1-88---sstruue + GRPO
59.0
Llama3. Instrutt
51.9
I RESULTS ON TOOL--USE TASKS
Discussions about the MDP of the Tool-Use Domain. Regarding tool use, it is crucial to distin- guish between the stochasticity inherent in external tools (e.g., a search engine R) and the deter- minism of the state update mechanism P. As illustrated in Fig. 28 the output generated by tool acts as an observation from the environment. Once this observation is realized and received by the agent, it becomes fixed component of the context history. Consequently, the transition function P(s (9t 0e), defined as the concatenation of the current state and the new token sequence, remains strictly deterministic even in scenarios involving stochastic tool execution.
1). This structural property ensures that our theoretical framework holds
In addition reasoning tasks, we further validate ROVER in tool-use scenarios based the Search- RI Jin et al. (2025) setup. Specifically, we fine-tune the Qwen2.5-3B-Ball model the combined training sets of Natural Questions (NQ) and HotpotQA. Due to resource and time constraints, we evaluated the ROVER checkpoint at step 300. Although this falls short of the 500 steps used for GRPO in the original Search-R1, it allows for a meaningful comparison with the reported GRPO results.
Performance comparison. Table II presents the performance across multiple benchmarks, span- ning both General QA and Multi-hop QA tasks. The results show that ROVER not only works effectively with tools but also outperforms the original GRPO in terms of average performance by 1.8%.
Diversity comparison. To evaluate diversity, we generate 32 responses for each prompt from the Bamboogle benchmark with temperature set 1.0, using the cosine distance between agent sponses as the diversity metric. As shown in Table 12 ROVER demonstrates substantially better diversity.
We hypothesize that diversity is particularly crucial for tool-use tasks and represents promising future direction for our ROVER algorithm. An agent capable of generating diverse queries
29

---

Published as a conference paper at ICLR 2026
You are given the original prompt and two model -generated responses. Determine whether the two responses use different strategies to solve the problem.
Use the following guidelines to identify different strategies:
Mathematical Tools and Concepts:
Using di different mathematical tools (e.g., differentiation ws. integration, series expansion vs. direct computation) Applying different theorems or properties (e.g., mean value theorem vs. fundamental theorem calculus)
Di fffrent mathematical domains (e.g.r algebraic vs. geometric, analytical vs. combinatorial)
2. Solution Structure Differences:
-Different variable substitutions transformations Different equation setups for the same problem
Different ways of breaking down the problem into subproblems
3. Specific Examples of Different Appreaches:
Direct computation ion vs. recursive method Forward solving vs. backward solving (working from the answer) Algebraic manipulation va. numerical approximation
Using contradiction vs. direct proof
Using induction va. direct formula
Coordinate-bare vs. coordinate- free methods
Even if two solutions arrive at the same answer, they should be consi dered different if they:
Use different. key mathematical tools or theorems
Follow different logical sequences in eritical steps
Represent the problem using different mathematical frameworks
Break down the problem in substantially different ways
Original prompt: (prompt]
Generation on 0: (generationO) Generation [generation1]
Question: Do Generation 0 and Generation 1 use different strategies? First analyze the key mathematical tools and solution structure used in each solution, then respond with "[[yes]] if the generations strategies or [[no]] ] if they do not.
different
Figure 27: Prompt for LLM judger to determine whether two responses use different strategies. We refined the prompt proposed in (Li et al. 2025a) enhance the LLM judge's capability for more nuanced strategy classification.
retrieve more varied and comprehensive information from the corpus, thereby increasing its chances of discovering the key evidence required answer prompt correctly. Conversely, model lacking diversity may fall into monotonous query patterns, leading to failure loop where it repeatedly fetches the same unhelpful results from the search engine and fails to solve the problem.
30

---

Published as a conference paper at ICLR 2026
esearthe cleardito detected it then
Phecoooaade from StarchRi
Rerreve sranch reaiisud - RR()
Extract search Pusse -itarlo <heacte)
millouty <inforatiooo c'aferatioo
Current state: 4
Action: token
Current state
(token toloe..
find sauzE
9:
alogac. by vbon VoDKo's tengrance "cersous"
(Deterministic Append)
5p+1
Inserctio Carinen fesarones ieformatiin Steearcho
$ Next state: = 1 token,
Erample from SearchRR
Search Engine (Stochantic ourput)
Probability=1
Deterministic Transition Dynamics P
Figure 28: Illustration of the transition dynamics in the tool-use domain (c.g., Search-RI (Jin et al, 2023)). The search result is modeled probabilistic component of the agent's s transition function. Thus, while the content of search results may vary due the stochastic nature of the search engine, the state transition logic (concatenation) remains deterministic.
stochastic observation from the environment rather than
Table 11: Performance comparison General QA and Multi-hop QA benchmarks.
General QA
Multi-hop QA
Avg.
NQ
TriviaQA
PopQA
HotpotQA
2wiki
Musique
Search-R1 -base (GRPO)
0.421
0.583
0.413
0.297
0.274
0.066
0.342
ROVER
0.477
0.600
0.443
0.290
0.286
0.069
0.360
LLM USAGE DETAILS
In compliance with ICLR 2026 policies on large language model usage, we disclose that L. are mainly used for two purposes this work:
LLM Judge for Strategic Equivalence Assessment: We employed LLMs as judges to deter- mine whether model responses strategically identical (see Fig. 27p. This constitutes a core component of our research methodology. We have carefully validated the safety and reliability of the LLM judge outputs through systematic verification procedures.
Writing Enhancement: We utilized LLMs to polish the paper's writing at the syntactic and grammatical levels. All LLLLLLMMennaatt content has undergone thorough human review and verification to ensure accuracy, appropriateness, and compliance with academic standards.
All LLM outputs were subject to careful human oversight and validation. We take full responsibility for the accuracy and integrity of all content in this paper, including any sections enhanced with LLM assistance.
Table Diversity comparison evaluated on Bamboogle benchmark using cosine distance between agent responses.
Method
Cosine Distance
Search-R -base (GRPO)
0.042
ROVER
0.065
31


Reviews:
[{"review_id": "a4Sa5uiAFS", "reviewer": "Reviewer_Qj2i", "summary": "This paper proposes ROVER (Random Policy Valuation for Diverse Reasoning), the key idea is the optimal policy can be derived simply by greedily selecting actions based on the Q-values of a fixed, uniformly random policy. This bypasses the need for complex Generalized Policy Iteration (GPI) loops used in standard algorithms like PPO and GRPO. To maintain diversity (which pure greedy selection sacrifices), ROVER samples actions via a softmax over these uniform-policy Q-values. Empirically, ROVER outperforms strong baselines (GRPO, REINFORCE++, DAPO) on math benchmarks (AIME, MATH500, etc.) in both solution quality.", "questions": "1. Theorem 1 proves that acting greedily according to the random policy's Q-values is optimal. Why, then, does the practical ROVER algorithm use a softmax sampler instead of a purely greedy approach?\n2. How would this be helpful if extend to other more complicated domain than math?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The core finding—that a uniform random policy's Q-values are sufficient for optimal control in this specific MDP structure—is a significant and surprising theoretical contribution that challenges prevailing RL paradigms for this domain.\n2. ROVER drastically simplifies the RL pipeline by removing actor-critic loops, reference models (explicitly), and complex hyperparameter tuning associated with PPO/GRPO. It is computationally efficient and easy to implement.\n3. The method consistently outperforms state-of-the-art baselines across multiple models (Qwen-4B, 8B, DeepSeek-1.5B) and challenging benchmarks (AIME, HMMT), showing gains in both pass@1 and pass@k (diversity).", "weaknesses": "1. The theoretical guarantee relies on a strict MDP structure (deterministic, tree-structured, binary terminal rewards). While standard for current math benchmarks, this may not hold for more complex reasoning tasks involving tool use, intermediate feedback, or stochastic environments, potentially limiting broader applicability.\n2. While the paper claims robustness, the ablation shows performance is sensitive to the temperature parameter $\\rho$. An extremely low $\\rho$ causes collapse, while a high $\\rho$ reduces to random sampling. The default $\\rho=1$ works well here but might need tuning for other domains.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "y97trRGpeZ", "reviewer": "Reviewer_8GQr", "summary": "This paper simplifies reinforcement learning with verifiable rewards (RLVR) for LLM reasoning. Exploiting the deterministic, tree-structured MDP with binary rewards, the authors prove that evaluating a uniform random policy's Q-function suffices for optimal reasoning—bypassing iterative policy improvement (PPO/GRPO). The proposed ROVER algorithm samples via softmax over these Q-values, balancing diversity and performance. Experiments show significant gains in reasoning quality and diversity versus complex RL baselines.", "questions": "1. Scaling to Longer Horizons and Larger Action Spaces. How does ROVER’s computational and memory footprint compare to PPO/GRPO at the largest vocabularies and problem sizes considered? Can the authors clarify if action batching, logit pruning, or any approximations are required to keep training feasible?\n\n2. Reward Shaping/Process-level Signals. Have the authors tested ROVER with partial or intermediate rewards (e.g., for partially correct math steps or process rewards)? If so, what is observed? If not, are there theoretical or practical obstacles?\n\n3. Extension to Non-binary, Non-deterministic Environments. How does ROVER behave when rewards are non-binary or transitions are stochastic? Is there a degradation in quality/diversity, and are there principled ways to extend the theorem to these cases?\n\n4. Failure and Negative Cases. Can the authors discuss cases where ROVER’s performance flatlines or drops compared to PPO/GRPO (e.g., pathological tasks, extremely long horizons, or ambiguous reward settings)?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Algorithmic Minimalism: ROVER dispenses with conventional policy improvement loops and ad hoc stabilizers (e.g., KL penalties or clipping), yielding a strikingly simple and easy-to-tune algorithm, as clearly depicted in Figure 3 (p. 3).\n\n2. Strong Empirical Performance and Strategic Diversity: Across standard math reasoning and countdown tasks, ROVER consistently outperforms strong baselines (e.g., GRPO, DAPO, REINFORCE++) in both solution accuracy (Table 1, p. 8; Figure 9) and the generation of diverse, correct reasoning paths (Figures 7, 10, 17).\n\n3. Reproducibility: Public release of source code enhances transparency and facilitates replication.", "weaknesses": "1. Narrow Theoretical Scope: The optimality guarantee for ROVER strictly requires deterministic, tree-structured MDPs with binary terminal rewards. The paper offers no empirical or theoretical exploration of how—or whether—the method degrades gracefully under violations of these assumptions (e.g., stochasticity, intermediate rewards, or graph-structured environments), limiting its applicability to more realistic RLVR settings.\n\n2. Oversimplified Reward Modeling: ROVER assumes only terminal binary feedback, ignoring process-based signals, partial credit, or intermediate rewards increasingly used in LLM alignment. The paper provides no discussion of how the framework might extend to richer reward structures, hindering practical deployment.\n\n3. Limited Model Scale in Experiments: All evaluations use models up to 8B parameters, falling short of current state-of-the-art scales (e.g., 70B+). This constraint tempers claims about ROVER’s readiness for deployment in cutting-edge LLM systems.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iVNylBmgwE", "reviewer": "Reviewer_okcT", "summary": "This paper finds that in LLM math reasoning, the optimal action can be recovered from the Q-function of a fixed uniformly random policy, thereby bypassing the generalized policy iteration loop and its associated heuristics. Based on this observation, the authors introduce an algorithm called Random Policy Valuation for Diverse Reasoning (ROVER).", "questions": "See weakness.", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper is clearly written and well motivated.\n2. The experiments validate the effectiveness of the method.\n3. The method is simple and seems to be scalable.", "weaknesses": "1. The motivation and the method itself are very similar to [1]. Although the authors cited [1] in the paper, more detailed discussions are needed, given the similarities. Specifically, [1] provided theoretical results, while the authors stated that \"these observations have\nremained primarily empirical, with limited theoretical justification.\" Besides, the authors stated that the main difference lies in that their paper provides the first theoretical analysis that accounts for LLM math reasoning. What are the exact differences that prevent their method and analysis from being useful in math reasoning?\n2. Regarding the experiment results, the performance of the baselines is too low in my opinion. For example, in [2], they report 33.33 and 25.42 for AIME 24 and 25, fine-tuned from Qwen3-8b-base using DAPO. The results are 10+ points higher than the same baseline reported in this paper, even outperforming the proposed method by a considerable margin. I understand that the training dataset and hyperparameters are different. But given that (1) their training dataset is barely MATH, instead of DeepScaler which is more in-distribution of AIME, and (2) the performance gap is huge, I believe more careful hyperparameter tuning is needed for both the baselines and the proposed method for a fair comparison.\n3. The proposed method does not use random policies. It rather uses the old policy for the rollouts. Thus, the naming is not that appropriate in my opinion. Besides, the main experiments are conducted on Qwen models, which fail to fully demonstrate the effectiveness of the method. From the appendix, fine-tuning on the DeepSeek-R1-Distill-Qwen models seems to give small improvements.\n\n[1] Laidlaw et al., Bridging Reinforcement Learning Theory and Practice with the Effective Horizon.\\\n[2] Wang et al., Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YmO7gBB8US", "reviewer": "Reviewer_PNvP", "summary": "This paper presents ROVER, a simplified reinforcement learning algorithm for improving the reasoning abilities of large language models. The authors argue that standard methods like PPO are overly complex for the specific structure of math reasoning tasks, which they model as deterministic, tree-structured Markov Decision Processes. The core insight is that an optimal policy can be derived directly by evaluating a fixed, uniform random policy, thus avoiding the unstable iterative loops of generalized policy iteration. ROVER implements this by learning Q values based on a random policy and using a softmax sampler to promote diversity. Experiments show that ROVER significantly outperforms strong baselines in both solution quality and diversity.", "questions": "1. Could the authors discuss how you expect ROVER's performance to be affected in environments that violate the assumptions, such as tasks involving tool use with a chance of failure or those with partial credit rewards?\n\n2. Algorithm 1 presents the loss function for updating the model. Could the authors please provide the explicit mathematical formula for the stop gradient target used in the L_ROVER calculation? The description in lines 295 and 296 is slightly ambiguous about how the target for the Q value regression is constructed from the next state.\n\n3. The paper shows superior diversity compared to PPO-style methods. Did the authors consider comparing ROVER to other approaches that explicitly encourage diversity, for example, by adding a learned or fixed entropy bonus to the model's reward?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper proposes an elegant simplification of the reinforcement learning process for this specific problem domain. By identifying the unique properties of the math reasoning MDP, the authors justify moving away from complex generalized policy iteration. \n\n2. The empirical results are consistently strong and well presented. The proposed method, ROVER, not only achieves higher scores on pass@1 metrics but also shows substantial gains in diversity and pass@k at larger k values. \n\n3. The work offers a valuable conceptual contribution by reframing the problem. The key insight is the mismatch between general-purpose RL algorithms and the specialized structure of LLM reasoning tasks. This perspective is insightful.", "weaknesses": "1. There is a gap between the theoretical foundation and the practical implementation of the Q function. The theory relies on the true Q values of a uniform random policy. However, the algorithm approximates this with a relative Q function based on the log probabilities of the current and previous policies. The paper does not provide a clear justification for why this specific practical choice is a good proxy for the theoretical quantity.\n\n2. While the baselines are relevant, the comparison could be strengthened by including methods that explicitly aim to improve diversity through mechanisms other than softmax sampling. For instance, algorithms that incorporate an entropy bonus in the reward function or use other diversity-seeking objectives.\n\n3. The novelty of the algorithmic components in the practical implementation is somewhat limited. The use of a relative Q function, mean-centered rewards, and reward broadcasting are techniques inspired by or used in prior works like GRPO and REINFORCE++. The main novelty lies in the theoretical motivation for combining them in this simplified framework, not in the components themselves.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)