Diversity-Incentivized Exploration for Versatile Reasoning
Authors: Zican Hu
Shilin Zhang
Yafu Li
Jianhao Yan
Xuyang Hu
Leyang Cui
Xiaoye Qu
Chunlin Chen
Yu Cheng
Zhi Wang
Venue: ICLR 2026 Poster (2026)
Keywords: LLM reasoning
Reinforcement learning with verifiable rewards
efficient exploration
diversity
Primary area: foundation or frontier models, including LLMs
TL;DR: We promote global sentence-level diversity to incentivize deep exploration for versatile LLM reasoning.

Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a crucial paradigm for incentivizing reasoning capabilities in Large Language Models (LLMs). Due to vast state-action spaces and reward sparsity in reasoning tasks, existing methods often struggle with deficient exploration and poor sample efficiency. In the paper, we propose **DIVER** (**D**iversity-**I**ncentivized Exploration for **V**ersatil**E** **R**easoning), an innovative framework that highlights the pivotal role of global sequence-level diversity to incentivize deep exploration for versatile reasoning. We first conduct a primary empirical study to reveal a strong positive correlation between global diversity and reasoning capacity. Building on this insight, we introduce global diversity incentives as an intrinsic reward to promote deep exploration in a semantically structured space. Incorporating the intrinsic reward, we develop a potential-based reward shaping mechanism to preserve optimal policy invariance and design simple heuristics to mitigate possible reward hacking. Experimental results show that DIVER outperforms competitive RLVR baselines with various exploration strategies on both in-domain and out-of-domain tasks, excelling in both Pass@1 and Pass@k evaluations. Our code is available at https://github.com/NJU-RL/DIVER.

Published as a conference paper at ICLR 2026
DIVERSITT-INNENTIIIIEDD EXPLORATION FOR VER- SATILE REASONING
Zican Hull-
Shilin Zhang Yafu Li
Jianhao Yan
Xuyang Hu²
Leyang Cui
Xiaoye Qu Chunlin Chen'
Yu Cheng
Zhi Wang!
Nanjing University Shanghai AI Laboratory
Westlake University 'Tee Chinese University of Hong Kong
{2 f canhu, shi shwsishiiieeee whiiinzanggga smmail. ju. edu en yaful yelgyllmml...om chengyuêcse. cuhk. edu. hk ju edu.cn cn
ABSTRACT
Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a cru- cial paradigm for incentivizing reasoning capabilities in Large Language Models (LLMs). Due to vast state-action spaces and reward sparsity in reasoning tasks, existing methods often struggle with deficient exploration and poor sample effi- ciency. In the paper, we propose DIVER VersatilE Reasoning), an innovative framework that highlights the pivotal role global sequence-level diversity to incentivize deep exploration for versatile rea- soning. We first conduct primary empirical study to reveal a strong positive correlation between global diversity and reasoning capacity. Building on this in- sight, we introduce global diversity incentives as an intrinsic reward to promote deep exploration in semantically structured space. Incorporating the intrinsic reward, we develop a potential-based reward shaping mechanism to preserve op- timal policy invariance and design simple heuristics mitigate possible reward hacking. Experimental results show that DIVER outperforms competitive RLVR baselines with various exploration strategies on both in-domain and off-oi-domain tasks, excelling in both @ass@ 1 and Pass @k evaluations. Our code is available at https//ggghub.com//UUU///////
Incentivized Exploration for
INTRODUCTION
Reinforcement Learning with Verifiable Re- wards (RLVR) has advanced reasoning capa-
bilities in Large Language Models (LLMs) through rule-based verification on model's re-
sponses (Guo al. 2025 Hu al. 2025a
Zeng et al.j 2025).
central challenge the fundamental exploration -epitiionee tradcoff highlighted in classic RL litera- ture (Lillicrap al.) 2015 Haarnoja al. 2018). Unlike traditional RI. environments with relatively small, well-defined state- action spaces (Sutton & Barto 2011 Wang al. 2024b; Zhang al. 2025). LLM poli- cles operate in vast, high-dimensional textual spaces with complex semantics, where the number of possible state-action pairs grows exponentially with sequence length (Gupta et al. 2024 Ahn et al. 2024 Hu et al. 2025b). This combinatorial explosion greatly increases the difficulty of effective exploration in textual reason- ing, especially under limited computing resources. Furthermore, the inherent reward sparsity in
(a) Local exploration
(b) Global exploration (Ours)
Local token-level vs. Global sequence- level exploration. We incentivize deep exploration to broaden diverse pathways for versatile reasoning.
Figure 1:
"Equal contributions. Zican Hu and Shilin Zhang are listed alphabetically by last name. This work was conducted during internship Shanghai AT Laboratory. "Corresponding authors. Project lead.

---

Published as a conference paper at ICLR 2026
challenging reasoning tasks creates massive "reward deserts" where the model receives little to no
meaningful feedback most of the time (Haarnoja et al., 2017; Wu et al.) 2025), hindering the dis- covery of improved pathways due to the lack of valid feedback signals (Zheng et al.) 2024 Zhan
et al. 2026). Therefore, existing methods often struggle with deficient exploration and poor sample
efficiency (Deng et al. 2023 Chen al. 2025b).
Thus, introducing efficient exploration mechanisms becomes essential for enabling LLMs to contin- ually self-improve their reasoning abilities through the iterative trial-and-error cycle (Ladosz al, 2022 [Song et al. 2025). Current LLM literature has explored several attempts based on increasing the uncertainty in the policy's 's action selection process (ao et al. 2025). such as alleviating policy entropy collapse (Cui et al.] 2025b), branching only high-entropy tokens (Liu et al., (202a)). and retaining policy gradient updates for high-entropy tokens only (Wang et al., 2025). These dithering strategies inject randomness into the policy's i action distribution, facilitating exploration by promot- ing the selection of uncertain actions. Typically, they incorporate diversity at the local action level (i.e., token level) to help the policy escape local optima and accelerate training.
While action-level uncertainty estimates allow the agent to direct its exploration toward potentially informative states, they are insufficient on their own to ensure efficient exploration (Osband et al., 2016). Provably efficient schemes require far-sighted, deep exploration that is directed over multi- ple timesteps, inducing temporally-extende local exploration, deep exploration is essential to significantly broaden the global sequence-level diversity of reasoning pathways, stimulating the model to discover novel and effective solution pat- terns. This principle of optimizing global diversity is essential for advancing the deep exploration capabilities of frontier RL algorithms (Eysenbach et al., 2019 Grillotti al.) (024), showcasing remarkable efficiency for solving intricate tasks in more human-like manner (Celik et al.) 2024).
diversity at higher level (Osband et al., 2019). Unlike
However, efficient deep exploration mechanisms remain largely underexplored in LLM reasoning.
Built on these insights, we propose DIVER (Diversity- -ncentiiy-- Exploration for VersatilE Reasoning) that emphasizes the pivotal role of global sequence-level diversity to incentivize deep exploration for versatile LLM reasoning. We first conduct a primary empirical study where evi- dence reveals a strong positive correlation between global diversity and reasoning capacity. This finding motivates us explicitly optimize sequence-level diversity during RL training. To this end, we formulate the global diversity across group responses as an intrinsic reward, incentivizing deep exploration in a semantically structured space. When incorporating this intrinsic reward, we design a potential- based reward shaping mechanism to preserve optimal policy invariance and develop sim- ple heuristics to mitigate possible reward hacking. Specifically, we employ two -os-lpe-mmple metrics to quantify the diversity inherent across group responses, Textual Diversity and Equational Diversity, while in principle any other metrics are compatible with framework. We hope this study inspires further investigation into global diversity and incentivizes efficient deep exploration mechanisms for broadening LLM's versatile reasoning capacities.
Experimental results demonstrate that DIVER consistently outperforms competitive RLVR baselines across six math reasoning benchmarks (AIME24/25, AMC. OlympiadBenchh, Minerva, MATH500). Notably, DIVER shows stronger generalization capabilities with a +3.2 points improvement over the GRPO baseline on out-of-domain benchmarks (ARC-c, GPQA* MML.U-rro) To evaluate exploration effectiveness through multi-attempts, we employ the Pass@k k metric, where DIVER consistently surpasses all baselines. The most substantial gain appears on AIME25, where DIVER achieves +6.7 points improvement in Pass@32 32 performance. Our in-depth analysis reveals that DIVER's s advantage stems from its global sequence-level diversity and deep exploration capability.
2 RELATED WORK
Exploration in RL. Exploration techniques are key to solving high-dimensional, sparse-reward
RL problems (Ladosz et al. (022). They can be roughly categorized into three kinds: 1) injecting
stochastic noise into behavior policies (Lillicrap et al.) 2015 Fujimoto et al. 2018); 2) incorporat-
ing policy entropy into the optimization objective (Haarnoja et al. 2017 2018): and 3) introducing
intrinsic rewards independent of environmental feedback, such as count- based bonuses (Bellemare
et al. 2016 Wang et al. 2026). information gains (Houthooft al., 2016)) the novelty of experi- ence Pathak et al. [2017] Burda et al., 2019). Our work extends the third category from classical RL to L.M reasoning tasks. Frontier RL algorithms adopt the principle of promoting global diversity
2

---

Published as a conference paper at ICLR 2026
Response
Diversity Measure
Reward
Policy Optimization
01
02
OG
A1
T
q
Policy
O2
01
O2
d2
d
d2
dj
...
dz
d
2
int
X+
P2
A2
&
22
Prompt
:
OG
OG
dg
G
TG
Ac
masked
Adv.
Prob.
Verifier
Figure 2: Overview of DIVER where we formulate the global sequence-level diversity of response of within group of rollouts intrinsic reward r!th to incentivize deep exploration. Diversity incentives are applied to correct solutions only to align shaping rewards with the true objective.
improve their deep exploration capabilities (Liu et al.) 2026), such learning diverse skills in an unsupervised manner (Eysenbach et al., (019), inducing diverse behaviors with constrained op- timization (Grillotti et al. 2024), and acquiring diverse skills using mixture-of-experts (Celik al.) 2024). As efficient deep exploration mechanisms remain largely underexplored in LLM reasoning, our method aims to address this gap.
Diversity for LRMs. Current literature has attempted to promote diversity in LLM reasoning by increasing the uncertainty in the action selection process. Some studies directly manage individual tokens at a micro level. (Cui et al.) (025b) applies clip and KL penalty constraints to tokens that tend to cause entropy collapse. (Liu et al. 2025a) selectively branches high-entropy tokens only to enhance exploration in test-time RL. (ang al. 2025) leverages high-entropy minority tokens to steer the model toward diverse reasoning pathways. Other approaches achieve similar results by reformulating optimization objectives. (Yo al. (2025) injects token-level diversity measure into policy optimization. (Cheng et al., 2025) augments the advantage function with an entropy-based term. Yuu et al., 2025) increases the upper bound for clipping the importance sampling ratio to emphasize low- lw-probabilityy tokens. In summary, these methods typically incorporate diversity at the local action level, facilitating exploration by promoting the selection of uncertain actions.
Recently, (Chen et al. 2025b) uses the @ass@k k metric as the training reward, leveraging multiple candidate solutions one trial enhance the exploration abilities of LLMs. However, does not explicitly account for global diversity, since it does not seek to optimize diversity across candidate solutions. concurrent work is (Li et al. 2025) that trains a partitioning classifier to measure diversity and amplifies the advantage function by the diversity assessment. Beyond algorithmic-level exploration strategies, data-level approaches have also been proposed to improve training efficiency. Prompt selection methods (Chen et al. 2025a Bae et al. 2025 Qu et al. (025) filter training samples based on difficulty or informativeness to enhance exploration efficiency. The key distinction of our method lies in how diversity is measured and how it is embedded within policy optimization.
METHOD
In this section, we first give the problem statement where the reasoning task is formulated as an RL problem. Then, we present a primary empirical study to show the impact of global sequence- level diversity reasoning performance. Finally, we introduce DIVER in detail, with principled formulations to quantify diversity, guarantee optimal policy invariance, and mitigate reward hacking.
3.1 PROBLEM STATEMENT
RL is based on the Markov decision process (MDP) formulation with a tuple (S, A, R),,, where S/ is the state/action space, T(s')s a) S A S [0, oo] is the transition operator that defines the probability density function of transitioning to state s' E conditioned on taking action a e A in state E S, R(s,a) a) S x A r R is the reward function, and y (0, 1] is the discount factor.
3

---

Published as a conference paper at ICLR 2026
solve all
solve none
in-domain
out out-of--omaan
Training Rato(
32.2
A2.
Training Ratay (A6)
$0-
Average Scores
80-
-2.
Test Scores
01.2
high diversity ty diversity
166
300
20-
22
50
high diversity
228
low diversity
40
30
100
300 Steps
200
400
100
200 300
400
100
200 300
400
ARC-C
GPQA
MMLU
Average
Steps
Ste ps
Figure 3: Performance comparison between high-diversity (red) and low-diversity (blue) training. solve all: Number of samples with all rollouts correctly solved. solve none: Samples with no correct rollouts. in-domain: Average test scores across training steps for in-domain bench- marks. out ddffdomain: Final performance for out-of-domain benchmarks.
Group Relative Policy Optimization (GRPO). We build the DIVER framework upon GRPO (Shao al. 2024). a prominent RLVR baseline that incentivizes reasoning capability in LLMs using rule- based verifiable reward function. GRPO discards the critic model and instead computes advantages using rule-based rewards from group-level comparisons. For each query g. the policy Taay gener- ates G candidate responses (o1,..., OG) ). Each response is evaluated by a binary reward function r, E {0, }} that checks whether the extracted answer matches the golden answer, yielding rewards rii........ This verifiable reward design effectively mitigates reward hacking (Gao et al. 2023), enabling robust scaling of RL training. The policy my is then updated by maximizing:
G lod
JGRFO(0) = E (o ~ aacaaall
CLIP(,,, 3
(I)
G i=1 1=1
where Pt, = =0
g.0 is the importance sampling ratio for the token at timestep in response Og- The clipped objective CLIP(ps.:: A.) = min Ar, clippp,......11111_1 updates within the trust region (Schulman et al., 2017). The KL regularization term constrains the policy from deviating too far from a reference model Ter.. The advantage A, is computed as:
c)A,) ensures stable
A
Mr Or
where
H4 G
E
o,
1
(r--
3.2 THE IMPACT OF SEQUENCE-LEVES DIVERSITY ON RLVR
While a line of recent work (Liu et al. 2025a Wang al. 2025) focuses on promoting local token- level diversity, explore and analyze how global sequence- sequence-level level diversity within data affects RL fine-tuning for LRMs. We first conduct an illustrative experiment that compares training the model using rollouts with different levels of sequence-level diversity.
Figure B shows the performance of filtering GRPO rollouts into high-diversity (red) and low- diversity (blue) subsets train separate models, using the proposed diversity metrics (TD and ED in Sec. (.3). The experimental details are provided in the Appendix [B.]] During training, high- diversity training simultaneously yields lower "solve all" rate (1.1 points decrease) and lower "solve none" rate (3.1 points decrease). Intuitively, the high-diversity scheme trades off a small fraction of responses to explore wider range of reasoning patterns, enabling the search for novel breakthroughs when conventional solutions fail. This property is particularly appealing in practice, since achieving success on complex problems carries greater value than obtaining uniformly correct answers on simple questions. During testing, high-diversity training achieves higher test scores, with an increase +1.8 points for in-domain benchmarks and an increase of +2.6 points for out-of- domain benchmarks. This encouraging result verifies that promoting global diversity can broaden LLM's reasoning capacity. A noteworthy point is that the superiority of high-diversity training is more pronounced for out-of-domain tasks compared in-domain ones. This observation is con- sistent with our motivation, where emphasizing diversity enables exploring a broader spectrum of reasoning patterns and expands generalization capacity.
4

---

Published as a conference paper at ICLR 2026
3.3 METRICS FOR QUANTIFYING SEQUENCE-LEVEL DIVERSITY
The above insight highlights the necessity of promoting global diversity to enable more effective exploration for RLVR, thereby incentivizing versatile LLM reasoning. Naturally, is crucial to develop efficient metrics capable of quantifying sequence-level diversity in reasoning tasks. To this end, we design two easy-to-implemen metries, Textual Diversity and Equational Diversity. As a general framework, our method is also compatible with any other diversity metrics.
Textual Diversity (TD). It refers to the text-level mutual dissimilarity across group responses from given query. We adopt the BLEU score (Papineni et al., (002). popular automated and inexpensive metric that correlates highly with human evaluation, to quantify the text similarity, and invert to measure dissimilarity. essence, BLEU measures similarity by calculating the overlap of n-grams (word sequences) between candidate text and the reference text, with higher scores indicating greater similarity. Appendix C.2 presents the detailed definition.
For each candidate response o; in the group (0i,...,cc) we calculate its BLEU similarity to every other response and derive its TD within the group as
TD(o:) G G --1E, jEIGIMA (1
1
(1-BBAC (I:BLEU(O::;))
Equational Diversity (ED). It refers to the differences in the formulas contained across group responses to given query, aiming to characterize underlying reasoning patterns for mathematical tasks. A math problem often admits multiple solutions with distinct formulaic patterns, and explor- ing diverse problem-solving approaches can increase the likelihood of finding the correct answer.
Let F(o;) denote the set of formulas extracted from response on, and F-s denote the set of formulas collected from other responses in the same group
F-; U setaro F(o;), [G] = (1, (1.....G). G}.
(4)
Then, the per-response equational diversity is defined as the ratio of unique formulas response o to the total number of formulas it contains:
((oo)
ED(a))
if [F(o))) > 0; or 0, otherwise.
(5)
F((o))
3.4 PROMOTING GLOBAL DIVERSITY FOR DEEP EXPLORATION
The findings in Sec. motivate us explicitly optimize global diversity during RL training. As shown in Figure 2 we formulate the global diversity as an intrinsic reward to incentivize deep exploration in a semantically structured space. For a group responses {01, ..., OG), ). we calculate the pairwise dissimilarity using the proposed diversity metrics in Sec. 3.3, yielding G G matrix D where each element de denotes the diversity between responses o, and oj. Then, the diversity of response o; within the group is calculated by simply averaging its dissimilarities to all others as d(o:)= - d(ox==-- Eje, d. i.e., averaging across the corresponding row in the diversity matrix D.
To promote global diversity RLVR, a natural option is to directly supply the quantified diversity additional reward to guide the learning process. However, this naive shaping can change the optimal policy and mislead the agent into learning suboptimal policies (Ng et al., 1999). Hence, we adopt a potential-base rating the intrinsic reward (Wang et al. 2023 Müller & Kudenko 2025). We formulate the intrinsic reward Rint the difference between the sequence-level diversities d(.)) of adjacent states as
reward shaping scheme to preserve optimal policy invariance when incorpo-
Rim de, 1e41) 7d($ d(s:),
where d(-) is the exactly the potential function over states E S. In the LLM setting of Eq. ① the states and action within a given query-response pair are defined as 8, == oi.] at == 01 and 8141 = [g, s.]1 Then, the specific intrinsic reward becomes
Rma(lg, st 01 (g. 0r))++1)) >d((g. se)) d(lg, 01,s++1),, i i= 1, 1,.., G.
5

---

Published as a conference paper at ICLR 2026
Since GRPO inherits the PPO principle (Schulman al. 2017) that derives policy gradients at the sequence level, the intrinsic reward for a complete response pair is calculated as
Rim (1 oil)
Σ 1=
T-1
7 Rim (1 o1 014+1, o...+1l)
Σ == [1yd(l1.00.gt d(lg, 0.gıl)]
(8)
7 d(aa o.,rr) d(2)
= =7 - d((g, o:]),
where T denotes the terminal step, and d( (g o;]) is the global diversity of response o,, which can be calculated by metrics in Sec. 3.3 The diversity of a constant query q zero, i.e., d(q) O.
This diversity-incentivized intrinsic reward complements the traditional rule-based assessment re- ward R(.)- By incorporating the diversity measure, we design versatile evaluation system that values both correctness and solution diversity, yielding the new reward function R-(-) as
R' ((g. o.]) = R[[g, a:l) A (inr((9,o.))
where is the shaping ratio that balances between accuracy and diversity. Maximizing the intrinsic reward incentivizes the model to explore diverse reasoning pathways at the sequence level, facil- itating the discovery of novel and effective solutions complex problems. Finally, we substitute the augmented reward function r R' R'([q. o;]) for the original reward r; = R((g, o;]) in Eq. 2 to calculate the advantage function under the GRPO framework.
When including the intrinsic reward, we will transform the original MDP M = (S, A, r) to a new one M' (S. A, 7), where R' = + XRm.. Since we are learning a policy for the transformed MDP M in the hope of using it in the original one M, it is essential to ensure that this transformation does not mislead the agent into learning suboptimal policies. Theorem | guarantees the optimal policy invariance when incorporating global diversity an intrinsic reward, validating the effectiveness of our reward shaping mechanism. Appendix CerondixAApprrr prereents the detailed proof.
Theorem 1 (Optimal Policy Invariance). Let M (S, A, T, R, denote the MDP for the LLM reasoning task. d(-): S + IR is real-valued function that computes the sequence-level diversity d(s) of the state within a group of rollouts. We formulate (-m(-): S AxS + I as an intrinsic reward function that is the difference between sentence diversities of two adjacent states, such that for all S. s' S, Rim a, s') >d(ss) - d(s). Then, with any constant balancing A.T,
ratio X, every optimal policy in the transformed MDP M'= = A, R. optimal policy in M, and vice versa.
XRmu 7) will also be an
The ingemuity of our reward-shaping design lies in that by setting the shaping reward as the dif- ference between the diversities of adjacent states as in Eq. (6), the intrinsic reward for a complete query-response pair is derived as the diversity of the final response as in Eq. (8). This elegant for-
mulation avoids the need to calculate the diversity of any intermediate sentences, which saves a significant amount of computation while ensuring optimal policy invariance.
3.5 MITIGATING REWARD HACKING
Including an additional shaping reward could increase the risk of reward hacking, a phenomenon where an RL agent exploits flaws or ambiguities the reward function to achieve high rewards without genuinely solving the intended task (Pan et al.) 2022). This is particularly concerning in language models, where the complex nature of reasoning tasks makes reward functions susceptible to biased exploitation (Liu et al. 20256). Although DIVER preserves optimal policy invariance after reward shaping, the model may still ever-exploit intrinsic rewards and neglect the primary objective during training. Since the primary reward for reasoning correctness is sparse and difficult to attain, obtaining the reward for diversity is considerably easier, especially when addressing hard problems. To this end, we design simple heuristics to mitigate the potential risk of reward hacking as follows. Ablation study in Sec. 4.4 verifies the successful mitigation of possible reward hacking.
Balanced Shaping. We clip the diversity reward to be int clip(r."t: [[g o,]) and a is predetermined upper bound that prevents the model from excessively ex- ploiting the shaping reward. Moreover, gradually reduce the balancing ratio À during training.
; 0, o),, where riint

---

Published as a conference paper at ICLR 2026
Akin to the classic celloration-exploitiii tradeoff in RL philosophy (Sutton & Barto 2018), prefer exploring diverse solutions early and tend to exploit accumulated knowledge later.
Conditional Shaping. We only include the shaping reward to correct responses within the group as r = r; + À rim I(r)), where ((r:) is indicator function that equals if the response is correct and 0 otherwise. This conditional shaping ensures that the diversity incentive only rewards genuinely correct solutions, preventing the model from trading off correctness for diversity. The de- sign principle effectively aligns shaping rewards with the true objective, addressing potential reward hacking concerns while promoting valuable, diversified exploration across the solution space.
EXPERIMENTS
We comprehensively evaluate and analyze our method to answer the following research questions: Can DIVER improve performance while maintaining effective global exploration and reliably extending to other models? ii) Can DIVER achieve an effective and broader exploration scope that unlocks enhanced reasoning capacity? iii) What is the appropriate configuration of DIVER for balancing diversity, reward stability, and exploration horizons?
4.1 EXPERIMENTAL SETTINGS
Datasets and Evaluation. Our training data is subset of OpenR 1-Math -20kk (Face 2025 with
prompts collected from NuminaMath 1.5 (LI ct al. 2024) following the LUFFY (Ynn et al. 2025). We evaluate six mathematical reasoning benchmarks: AIME24/25, AMC (Li et al. 2022), Min- erva (Lewkowycz et al., 2022), OlympiadBench (He et al., (024). and MATH-500 (Hendrycks et al.) 2021 Main results report Avg@ 32 for the smaller test sets (AIME24/25, AMC), and Pass I for others. For cross-domain, we test on ARC-c (Clark et al. 2018), GPQA-diamond (GPQA*) (Rein)
al. 2024). and MMLU-Pro (Wang al. 2024a).
Baselines and Training. We compare DIVER against two categories of baselines: 1) Estab- lished RLVR methods: Simp
2025a), and PRIME- SimpleRL MepleRl--onnn -oo Zeng et al. 2025), OpenReasoner Zeroouu et al.) (Cui al. 2025a). 2) Our reproduction of exploration RL methods: GRPO w/ clip siip-hight weaggigerrrgi Yu et al. 2025), which modifies clip ratio to encourage exploration; Eot et al. 20256), which addresses policy entropy collapse through covariance- based techniques; and P Passēk Training (Chen et al., 2025b), which uses Pass @ k as the reward to adaptively balance exploration and exploitation. We set use 0.28 for higher clip following GPPO w/ Clip-higher. Sample batch size is 128, update batch size 32, with rollouts per prompt. All experimental details are documented in Appendix B.2
0 to remove the KI. loss term and
4.2 MAIN RESULTS
Reasoning Performance on Qwen2.5-Math-7B. Table illlustrates DIVER's s evaluation results compared established RLVR methods (SimpleRL-Zoo, OpenReasoner--Zero and PRIME-Zero). All implementations based on Qwen2.5-Math-7B DIVER with Textual Diversity (TD) and Equa- tional Diversity (ED) achieves average scores of 42.3 and 43,0 on six mathematical benchmarks, outperforming Zern by +2.0 points. On challenging out-of-domain tasks, DIVER reaches 58.4 average score, surpassing OpenReasoner-Zeero by +6.8 points, with notable gains on ARC-c (+10.1) and GPQA (+12.5). These results confirm that encouraging diverse reasoning paths the sequence level enhances model generalization significantly.
Comparison with Exploration RL Methods. We evaluate DIVER against representative explo- ration RLVR methods: GPRO w/ Clip-higher (undirected exploration), Entropy-RL (action-level exploration), and Pass@k Training (within-growy bootstrap sampling). Experimental results demon- strate that DIVER outperforms the best exploration method, Entropy-RL, by +1.2 points on average across in-domain tasks. This improvement is particularly pronounced on challenging benchmarks, with +4.6 point advantage OlympiadBench. Notably, on out-of-domain tasks, both DIVER and the global exploration approach @ Training exhibit superior generalization compared to local exploration techniques. DIVER surpasses the best local exploration method, Entropy-RI. by +2.4
7

---

Published as a conference paper at ICLR 2026
Table 1: Performance comparison across in-domain and out-of-domain tasks based 5 5-Math- MB. Best results in bold and second best underlined. DIVER- TD and DIVER-ED represent our approach implemented with Textual Diversity and Equational Diversity, respectively. DIVER-MIX combines both metrics during training to achieve optimal diversity.
In-Domain Performance
Out-of- Performance
Model
AIME 24/25
AMC MATH-500 Minerva Olympiad Avg.
ARC-e GPQA
MML.--Pro Avg.
Owen2.55Mathh77 5-Math-7B
11.8/6.3
56.8
16.9
25.4
38.1
31.5
27.3
Previous RLVR methods
SimplcRL-Zoo
25.2/12.0
57.6
76.2
27.2
41.0
39.9
22.0
20.4
32.5
25.0
OpenReasoner-7em
16.5/15.0
52.1
82.4
33.1
47.1
41.0
66.2
29.8
58.7
$1.6
PRIME-Zero
17.0/12.8
54.0
81.4
39.0
40.3
40.7
73.3
18.2
32.7
41.4
Exploration RL Methods
GRPO Clip-higher
18.9/16.4
81.2
28.7
40.7
82.1
55.2
Entropy-RL
23.6/12.8
58.4
82.8
31.6
41.5
41.8
80.7
38.8
48.4
56.0
@a Training
20.9/15.7
52.3
83.8
32.7
43.8
41.5
79.3
37.8
49.0
55.3
Our Methods
DIVER-TD
22.5/16.9
59.4
82.2
27.9
44.7
42.3
83.4
49.5
58.4
DIVER-ED
20.9/15.7
59.7
84.0
31.6
46.1
43.0
83.4
36.2
49.9
56.5
DIVER-MIX
23.8/16.4
60.9
84.4
29.4
44.0
43.1
84.1
41.3
51.0
58.8
DiVERtOun))
Enkrogy-LL
GPRG m/ tli--igher
Passigi Training solve all
ED (1)
TD (*)
Entropy
solve none
0.5
Trining Lose
30
0.7
0.4
0.8
0.5
100
200
200
400
100
200
300
400
100
200
300
400
300
200
300
400
1.00
200 Steps
300
400
Steps
Steps
Steps
Steps
Figure 4: Training dynamics comparison with other exploration method across different metrics. indicates metrics where higher values are more diverse for and TD.
points average. This advantage is especially evident benchmarks that assess conversational capabilities, such as GPQA, where DIVER outperforms Entropy-RL by +3.5 points. The relatively weaker performance of GPRO w/ Clip-higher indicates that merely increasing constraints without strategic direction fails to fundamentally enhance model generalization capabilities.
Training Dynamics of Exploration Methods. Figure. iillustrates the various metrics DIVER compared other exploration methods. We aim to maintain sufficient exploration (high ED and TD) while preventing excessive entropy growth that may lead to model collapse. With this logic in mind, observe that Pass@) Training and Entropy-RL. diversity metries decline over time, indi- cating reduced exploration, while GPRO w/ Clip-higher maintains diversity but experiences prob- lematic entropy increases later. In contrast, DIVER achieves optimal balance with high diversity and consistently reasonable entropy levels, enabling controlled exploration without excess randomness. Notably, DIVER's 's "solve all" rate grows more gradually, but lower "solve none" rate in training demonstrates effective exploration without compromising solution quality.
Extending DIVER to Diffferntt Models. We further explore DIVER's adaptability across var- ious language models, including small, weak different architecture models. As shown in Fig- ure and Table DIVER maintains effectiveness across different model backbones including Qwen2.5-Math- 1.5B, Bree2.5--B-Bartt and LLaMA-3.1 3.11--8..BBBB Insturet. DIVER consistently outper- forms baselines, improving over GRPO w/ ligheh by +1.7, +1.7, and +1.5 points on in-domain tasks respectively, with even larger gains of +1.5, +1.3, and +1.9 points out-of-domain tasks. Ad-
8

---

Published as a conference paper at ICLR 2026
DIVER(OUrs]
Entropy-RL
Clip-higher
Pass@k Training
AJME24
AIME25
AMC
MATH500
Perfermance
0.6
0.9
0.5
0.4
0.4
0.3
0.8
0.900
0.3
0.7
0.875
0.2
0.850
2
8
16
32
16
32
2
8
16
32
16
32
Olymplad
ARC-c
GPOA
MMLU-PPr
0.70
0.975
0.9
0.8
0.7-
0.7
0.50
0.900
0.6
0.6
32
16
32
16
Pass@ikk
Figure 6: Comparison of different Pass@k performance across in-domain and out-of-domain bench- marks based on Qwen2. 5 Mathhahhh 7B. Complete results are available in Table 4
model generating longer responses (2500-3500 tokens), show DIVER outperforming GPRO w/ Clip-higher by +1.6 points in-domain and +3.4 points out-of-domain, demonstrating its effectiveness with longer reasoning horizons.
ditionally, experiments with Di-tcck-R--Disiil--Me-
4.3 EXPLORATION SCOPE AND REASONING CAPACITY
We evaluate Pass@k, a metric for model reason- ing upper-bounds (Cheng al. 2025 for k E {2,4, 8, 16, 32} across all tasks (Figure 6 Table 4
BIVRROouR
GF cih gher
DIVER consistently outperforms baselines across nearly all benchmarks. Moreover, Pass@32 most closely reveals reasoning exploration scope, where DIVER achieves superior performance on challeng-
60-
60-
Perfomance
4L4
ing benchmarks with 50.0 on AIME25 (+6.7 over Entropy-RL) and 68.9 points on OlympiadBench (+2.1
Indeamain
oui-o-sammin
in-damain
Dui-el menn
over GRPO w/ Clip-higher). Overall, compared to ex- isting exploration methods, DIVER uniquely enhances @ass@k capability without compromising Pass performance. To visually demonstrate our conclu- sion, we examine case studies of multi-attempt roll- outs (Appendix F DIVER generates diverse yet co- herent reasoning paths leading to correct solutions, while Entropy-RL explores specific decision points, and GRPO w/ Clip-higher and @ass@k Training ex- hibit wide but unproductive exploration, all leading to incorrect answers. These results confirm DIVER's 's su- perior exploration scope unlocks higher reasoning capacity.
LL:MA-31-88B Instrcc
18.5
Performance
20
In domain
OUZ of oomaann
In moma
DUC of dominn
Figure 5: Average scores across in-domain and out-of-domain tasks with different models. Complete results in Table 9
To address concerns about the scale of Pass we extend our experiments to k=128, 256. 512, and 1024 across AIME24, AIME25, and AMC benchmarks, including the base model Qwen2.5.
Math-7B and all baselines. The DIVER demonstrates consistent superiority of DIVER (table 55 across all settings: 1) DIVER achieves the highest Pass @K scores across nearly all benchmarks and k values, reaching 86.7% AIME24 and 100% on AMC at Pass @ 1024. 2) The performance gap increases with larger k, highlighting DIVER'S 's superior ability to generate diverse correct solutions. On AIME24, DIVER outperforms the second-best baseline by 6.7 points at Pass @ 1024. A)All RL methods substantially improve over the base model. but DIVER consistently achieves the best results, confirming that diversity-driven exploration provides significant advantages.

---

Published as a conference paper at ICLR 2026
4.4 ANALYSIS
Mitigating Reward Hacking. We compare four shaping strategies: applying diversity rewards to (1) correct responses, (2) all re- sponses. (3) errors only, and (4) all responses with a length penalty. As shown in Figure 7 rewarding diversity on incorrect or all re- sponses severely degrades test performance, with response lengths exploding as the model exploits long rollouts to gain higher diversity bonuses. Introducing a length penalty miti- gates length explosion but still results in poor accuracy. In contrast, the conditional shaping strategy, which applies diversity rewards only to cor- rect responses (red line), effectively constrains exploration to valid solutions and mitigates reward hacking.
Correct
Errer
A1
All length
Test Score
Training Accuracy
Response Length
0.35
Lewes hacking
0.5
2000-
Jaggh oxplesio
0.30
1000
0.2
100
200
300
400
100
200
300
400
100
200
300
400
Steas
Figure 7: DIVER with rewarding diversity in cor- rect, all, error, and all (w/ length penalty) responses.
Longer Horizons Improve Performance. To investigate suitable exploration horizons (i.e., range of text for diversity calculation) for reasoning, we evaluate diversity metrics across different token horizons (i.e., the first 200, 500, 1000 tokens of the trajectory) ver- sus complete responses. Figure 8 shows full responses maximize performance, while shorter horizons significantly reduce both en- tropy and performance. The consistent im- provement with increasing horizon length confirms the reasoning benefits from global sequence-level diversity.
All Iobers
1000 tokens
500 tokens
200 tolens
Test Score
Entorpy
Response Lenth
1800
0.40
1600-
0.35
04-
1400
0.2-
1200
0.0
100
200
300
100
200
300
300
200
300
Steps
Steps
Steps
Figure 8: DIVER with varying horizon constraints. A longer horizon allows for higher global diversity.
Diversity Enhances Reasoning Quality To validate whether our approach achieves higher quality and more meaningful reasoning sults, we evaluate all test responses using Seepsek-ai/DepSeee- 2-Exx judge model, ranking responses generated by the base model, baselines, and DIVER for each prompt. The evaluation comprehensively assesses four key dimensions: Completeness, Logical Consistency, Relevance, and Efficienen Fig. demonstratss DIVER solutions rank best (Rank 1) in 76.4% of cases, more than double the next best method. This confirms our diversity measures capture meaningful improvements in reasoning quality. not merely surface-level variations.
Reasoning Quality Distribation
DIVER(ours)
7.5
-60
GPRD w/ Clip-tlihher
21.1
23.9
24.9
17.8
13.2
-50
Entropy-RLE
22.9
19.7
22.0
22.9
12.5
-40
-30
Passgi Training
31.7
22.0
16.8
17.7 11.8
-20
23.6
14.8
2
Rank
Figure 9: Average Ranking reasoning qual- ity across six mathematical benchmarks as judged by DeepSeek- V3.2-Exp. Complete results are available in Table R
CONCLUSIONS, LIMITATIONS, AND FUTURE WORK
We presented DIVER, an innovative approach that enhances LLM reasoning from new perspective of diversity. In contrast to existing methods that focus primarily local token-level diversity, we examine the role of global sequence-level diversity in incentivizing deep exploration, revealing positive correlation with reasoning capacity. Evaluations showed DIVER achieves consistently higher reasoning capabilities on in-domain tasks and stronger generalization on out-of-domain tasks. DIVER considers single-turn RLVR, while multi-curn settings hold greater promise for unlocking agent RL's potential in real-world applications. We leave as future work. Another direction is to employ more powerful diversity metrics, such using LLM-as-a-judge (Gu al. 2024).
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENTS
We extend our gratitude to all the reviewers for their valuable feedback and suggestions. This work completed during internship at the Shanghai Artificial Intelligence Laboratory. It was supported in part by the National Natural Science Foundation of China (No. 62376122), and in part by the National Key Research and Development Program of China (No. 2025YFA1016904).
ETHICS STATEMENT
We aware of any major ethical concerns arising from our work. Our study is conducted en- tirely within the mathematics domain, using only publicly available models and datasets for training and evaluation. No human subjects were involved, and research does not introduce sensitive or potentially harmful insights.
REPRODUCIBILITY STATEMENT
We provide the experimental setups in S.1 with with further details in Appendix [B.2] The code available at supplementary material, and will be released on GitHub the future. Additionally, we will make the weights of the DIVER models publicly available via platforms such as Hugging Face Community upon acceptance.
REFERENCES
Janice Ahn, Rishu Verma, Renze Lou, Di Liu, Rui Zhang, and Wenpeng Yin. Large language models for mathematical reasoning: Progresses and challenges. arXiv preprint arXiv:2402.00157. 2024.
Sanghwan Bac, Jiwoo Hong, Min Young Lee, Hanbyul Kim, JeongYeon Nam, and Donghyun Kwak. Online difficulty filtering for reasoning oriented reinforcement learning. arXiv preprint arXiv:2504.033300 2025.
Marc Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. Advances neural information pro- cessing systems, 29, 2016.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. In Proceedings of International Conference on Learning Representations, 2019.
Onur Celik, Aleksandar Taranovic, and Gerhard Neumann. Acquiring diverse skills using curricu- lum reinforcement learning with mixture of experts. In Proceedings of International Conference on Machine Learning, pp. 5907-5933, 2024.
Xiaoyin Chen, Jiarui Lu, Minsu Kim, Dinghuai Zhang, Jian Tang, Alexandre Piché, Nicolas Gontier, Yoshua Bengio, and Ehsan Kamalloo. Self-evolving curriculum for 11m reasoning. arXiv preprint arXiv:2505. 14970, 2025a.
Zhipeng Chen, Xiaobo Qin, Youbin Wu, Yue Ling, Qinghao Ye, Wayne Xin Zhao, and Guang Shi. Pass@ k training for adaptively balaneing exploration and exploitation of large reasoning models. arXiv preprint 250iv:2508. 10751, 2025b.
Daixuan Cheng, Shaohan Huang, Xuekai Zhu, Bo Dai, Wayne Xin Zhao, Zhenliang Zhang, and Furu Wei. Reasoning with exploration: An entropy perspective on reinforcement learning for Ilms. arXiv preprint arXiv:2506. 14758, 2025.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. MtXiv:1803.05457v11 2018
Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang. Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, et al. Process reinforcement through implicit rewards. arXiv preprint arXiv:2502.01456, 2025a.
11

---

Published as a conference paper at ICLR 2026
Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang. Yuxin Zuo, Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, et al. The entropy mechanism of reinforcement learning for reasoning language models. arXiv preprint arXiv:2505.22617, 2025b.
Jia Deng, Jie Chen, Zhipeng Chen, Daixuan Cheng. Fei Bai, Beichen Zhang, Yingian Min, Yanzipeng Gao. Wayne Xin Zhao, and Ji- Ji-Rong Wen. From trial-and-eroor to improvement: A systematic analysis of Ilm exploration mechanisms in rlvr. arXiv preprint urXiv:2508.07534 2025.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need:
Learning skills without a reward function. In Proceedings of International Conference on Learn ing Representations, 2019.
Hugging Face. Open rl: A fully open reproduction of deepseek-rl, January 2025. URL https: httchub.com/hggee ag open- r1
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-
critic methods. In International conference on machine learning, pp. 1587-1596. PMLR, 2018.
Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pp. 10835-10866. PMLR, 2023.
Luca Grillotti, Maxence Faldor, Borja León, and Antoine Cully. Quveity-diversity actor-critic:
Learning pergrrerforiing and diverse behaviors via value and successor features critics. In Pro- ceedings of International Conference on Machine Learning, pp. 16416-16459, 2024.
Jiawei Gu, Xuhui Jiang. Zhichao Shi, Hexiang Tan, Xuehao Zhai Chengjin Xu, Wei Li, Ying- han Shen, Shengjie Ma, Honghao Liu, et al. A survey on LLLM-as-ajuddge arXiv preprint 1.. 2024.
Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501. 2025.
Neha Gupta, Harikrishna Narasimhan, Wittawat Jitkrittum, Ankit Singh Rawat, Aditya Krishna Menon, and Sanjiv Kumar. Language model cascades: Token-level uncertainty and beyond. arXiv preprint 2004 10136, 2024.
Tuomas Haamoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In International conference machine learning, pp. 1352-1361. PMLR, 2017.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International confer- ence on machine learming, pp. 1861-1870. Pmlr, 2018.
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang. Yuxiang Zhang, er al. Olympiadbench: challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3828 8828-3850, 2024.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2 003.03874, 2021.
Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. Vime: Variational information maximizing exploration. Advances neural information processing sys- tems, 29, 2016.
Jingcheng Hu. Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung- Yeung Shum. Open-reasoner-zeru An open source approach to scaling up reinforcement learning on the base model, 2025a. htL /ttp::////xiii/ wttp:xxwww.og/////////000/ 24290
12

---

Published as a conference paper at ICLR 2026
Zican Hu, Wei Liu, Xiaoye Qu, Xiangyu Yue, Chunlin Chen, Zhi Wang. and Yu Cheng. Divide and conquer: Grounding LLMs as efficient decision-makinn agents via offline hierarchical reinforce- ment learning. In Proceedings of International Conference on Machine Learning, 2025b.
Pawel Ladosz, Lilian Weng, Minwoo Kim, and Hyondong Oh. Exploration in deep reinforcement learning: A survey. Information Fusion, 85:1-22, 2022.
Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ra- masesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Systems, 35:3843-3857, 2022.
Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Huang, Kashif Rasul, Longhui Yu, Albert Q. Jiang. Ziju Shen, et al. Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions. https: //huggingface.. http://gga.c..o/tatteeeeeeeeeeee cat.ttssessssseeeeeoooooss.s.sse Nwm.inmmmatttaaaana namath 2024. Hugging Face repository, 13:9.
Jia LI, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Costa Huang, Kashif Rasul, Longhui Yu, Albert Jiang, Ziju Shen. Zihan Qin, Bin Dong, Li Zhou, Yann Fleureau, Guillaume Lample, and Stanislas Polu. Numinamath. (attps://githus http://www.ccccoo/ooeeootoo-oaee aatog-ppppressspppppaa/////aaaap http:///ttttomooooaatttttttttttt 2024.
co Co/AT-M0//KminaMattaa
Tianjian Li, Yiming Zhang, Ping Yu, Swarnadeep Saha, Daniel Khashabi, Jason Weston, Jack Lan-
chantin, and Tianlu Wang. Jointly reinforcing diversity and quality in language model generations. arXiv preprint arXiv:2509.02534 2025.
Timothy P Lillicrap. Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv: arXiv: /509 2015.
Jia Liu, ChangYi He, YingQiao Lin, MingMin Yang, Fei Shen, ShaoGuo Liu, and TingTing Gao. Ettrl: Balancing exploration and exploitation 11m test-time reinforcement learning via entropy mechanism. arXiv preprint arXiv:2508.1 1356, 2025a.
Jinmei Liu, Haoru Li, Zhenhong Sun, Chaofeng Chen, Yatao Bian, Bo Wang, Daoyi Dong, Chunlin Chen, and Zhi Wang. Beyond the dirac delta: Mitigating diversity collapse in reinforcement fine-tuning for versatile image generation. arXiv preprint (tfi::2001.12400) 2026.
Tianqi Liu, Wei Xiong, Jie Ren, Lichang Chen, Junru Wu, Rishabh Joshi, Yang Gao, Jiaming Shen, Zhen Qin, Tianhe Yu, et al. RRM: Robust reward model training mitigates reward hacking. In International Conference Learning Representations, 2025b.
Henrik Müller and Daniel Kudenko. Improving the effectiveness of potential-based reward shaping in reinforeement learning. In Proceedings of International Conference on Autonomous Agents and Multiagent Systems, pp. 2684-2686, 2025.
Andrew Y Ng. Daishi Harada, and Stuart J Russell. Policy invariance under reward transforma- tions: Theory and application to reward shaping. In Proceedings of International Conference on Machine Learning, pp. 287, 1999.
Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. In Advances in neural information processing systems, volume 29, 2016.
Ian Osband, Benjamin Van Roy, Daniel Russo, and Zheng Wen. Deep exploration via randomized value functions. Journal of Machine Learning Research, 20(124):1 12,, 2019.
Alexander Pan, Kush Bhatia, and Jacob Steinhardt. The effects of reward misspecification: Mapping and mitigating misaligned models. In International Conference on Learning Representations, 2022.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311-318, 2002.
13

---

Published as a conference paper at ICLR 2026
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity- exploration by self-supervised prediction. In International conference on machine learning, pp. 2778-278777 2787. PMLR. 2017.
Yun Qu, Qi Wang, Yixiu Mao, Vincent Tao Hu, Björ Ommer, and Xiangyang Ji. Can prompt difficulty be online predicted for accelerating rl finetuning of reasoning models? arXiv preprint MrXiv:2507.04632 2025.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA: A graduate-Ievel google-proof q&a benchmark. In First Conference on Language Modeling, 2024. URL.https::/pppeeeeeee net//ooum? 1d=T167584b98
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.063477 2017.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathe- matical reasoning in open language models, 2024. URL/ttts://aaxii wwtpwvv/.rr//b//2222222
03300
Yuda Song, Julia Kempe, and Remi Munos. Outcome-based exploration for Ilm reasoning, arXiv preprint arXiv:2509.0694/ 2025.
Richard Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. The MIT Press, second edition, 2018.
Futing Wang, Jianhao Yan, Yun Luo, Ganqu Cui, Zhi Wang, Xiaoye Qu, Yue Zhang, Yu Cheng, and Tao Lin. Think longer to explore deeper: Learn explore in-context via length-incentivized reinforcement learning. arXiv preprint arXiv:2602. 11748, 2026.
Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang, Xionghui Chen, Jianxin Yang, Zhenru Zhang, et al. Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for I1m reasoning. arXiv preprint arXiv:2506.01933 2025.
Yiming Wang, Ming Yang. Renzhi Dong, Binbin Sun, Furui Liu, et al. Efficient potential-based exploration in reinforcement learning using inverse dynamic bisimulation metric. In Advances in Neural Information Processing Systems, volume 36, pp. 38786-38797 2023.
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et Mmlu-pro: more robust and challenging multi-task language understanding benchmark. arXiv preprint 2765::240 2024a.
Zhi Wang, Li Zhang, Wenhao Wu, Yuanheng Zhu, Donghin Zhao, and Chunlin Chen. Meta-DTT: Of- fline meta-RI. conditional sequence modeling with world model disentanglement. In Advances in Neural Information Processing Systems, volume 37, pp. 44845-44870, 2024b.
Wenhao Wu, Fuhong Liu, Haoru Li, Zican Hu, Daoyi Dong, Chunlin Chen, and Zhi Wang. Mixture- of-experts meets in-context reinforcement learning. In Advances in Neural Information Process- ing Systems, 2025.
Jianhao Yan, Yafu Li, Zican Hu, Zhi Wang, Ganqu Cui, Xiaoye Qu, Yu Cheng, and Yue Zhang. Learning to reason under off-policy guidance. In Advances in Neural Information Processing Systems, 2025.
Jian Yao, Ran Cheng, Xingyu Wu, Jibin Wu, and Kay Chen Tan. Diversity- policy optimization for large language model reasoning. arXiv preprint arXiv:2505.23443 233333,... 2025.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu. Lingjun Liu, et al. Dapo: An open-source Ilm reinforcement learning system at scale. arXiv preprint arXiv:2503. 14476, 2025.
14

---

Published as a conference paper at ICLR 2026
Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl- z0o: Investigating and taming zero reinforcement learning for open base models the wild. In Second Conference on Language Modeling, 2025.
Runzhe Zhan, Yafu Li, Zhi Wang, Xiaoye Qu, Dongrui Liu, Jing Shao, Derek F Wong, and Yu Cheng. ExGRPO: Learning to reason from experience. In Proceedings of International Con- ference on Learning Representations, 2026.
Shilin Zhang, Zican Hu, Wenhao Wu, Xinyi Xie, Jianxiang Tang, Chunlin Chen, Daoyi Dong, Yu Cheng, Zhenhong Sun, and Zhi Wang. Text-to-decision agent: Offline meta-reinforcement learning from natural language supervision. In Advances in Neural Information Prucessing Sys- tems, 2025.
Qinqing Zheng, Mikael Henaff, Amy Zhang, Aditya Grover, and Brandon Amos. Online intrin- sic rewards for decision making agents from large language model feedback. arXiv preprint arXiv:2410.23022, 2024.
15

---

Published as a conference paper at ICLR 2026
Appendix
A Optimal Policy Invariance in DIVER
17
Experimental Details
18
B. Diversity based Filtering
18
B.2 Detailed settings
18
B.3 System Prompt
18
Detailed Metrics Definition
20
Pass@k Performancel
20
BLEU Score
20
Equation Extraction
21
More Analyze
22
D.I
Analysis of Diversity Metries Validity
22
D.2 Analysis of Reasoning Quality
22
D.3 Analysis of Model Scale and Reasoning Horizon
22
D.4 More Comprehensive Ablations
24
Case Study
25
The Use of Large Language Models (LLMs)
31
16

---

Published as a conference paper at ICLR 2026
A OPTIMAL POLICY INVARIANCE IN DIVER
Following the classical reward shaping study (Ng et al. 1999), we give the proof of Theorem ! which guarantees the optimal policy invariance when incorporating global diversity an intrinsic reward.
Theorem 1 (Optimal Policy Invariance). Ler M A,, R, denote the MDP for the LLM reasoning task. d(.): - R is real-valued function that computes the sentence-level diversity d(s) of the state s within a group of rollouts. We formulate (-m(-): S Sx A X + RR as an intrinsic reward function that is the difference between sentence diversities of two adjacent states, such that for all E S, a & A,s' E S, R, = =d('') - d(s). Then, with any constant balancing ratio X, every optimal policy the transformed MDP M' (S, A,T, XBma 7) will also be an optimal policy in M, and vice versa.
Proof. For the original MDP M, we know that its optimal Q-function Qir satisfies the Bellman optimality equation (Sutton & Barto 2018):
(ir(s,a) E, R(s,a,8') 7 (sr(s',d))
(10)
With some simple algehraic manipulation, we can get:
as) - Ad(s) Ez [(ss,a,s)) + (7dd(5)) - d(()) + max
a'EA
(11)
If now define QM (s, a) - (s, Ad((s) and substitute that and (s a, =dd=)) d(s) into the previous equation, we can get:
(s, Esr
R(s, R(s,a,s') + as + max Qxr
a'En
(12)
R'(s,,,')) + max =E,
max Qu (s
a'd
which is exactly the Bellman optimality equation for the transformed MDP M', where R' = R +
ARim is the reward function for M'. Thus, (s, Qm Qi (s, a) Ad(s), and the optimal policy for M' therefore satisfies:
The (s) arg max (Mr a) aEA
arg max
(s, Ad(s))
(13)
aEA
arg max (s, aEA
and is therefore also optimal in M. To show every optimal policy in M is also optimal in M'. simply apply the same proof with the roles of M and M interchanged (and using Riat as the intrinsic reward). This completes the proof.
17

---

Published as a conference paper at ICLR 2026
B EXPERIMENTAL DETAILS
B. DIVERSITY BASED FILTERING
To compare the model training process using rollouts with different diversity levels, we conduct an illustrative experiment to filter GRPO rollouts into high-diversity and low-diversity subsets to train separate models. Specifically, the policy generates 2 responses for each query. For high-diversity scheme, G responses of highest diversity metrics (TD and ED) are filtered the candidate responses .1, OG for GRPO training. For low-diversity scheme, responses of lowest diversity metrics are filtered. Both approaches maintain identical computational costs, they the same number of generated responses and differ only in the selection criteria applied during filtering.
B.2 DETAILED SETTINGS
Datasets Our training data is subset of OpenR --Math-2200 (Face 2025), with prompts collected from NuminaMath 1.5 (LI et al., 2024). We follow the LUFFY (? dataset construction methodol- ogy but differ in that we do not incorporate off-policy reasoning traces, ours is purely on-policy approach.
Evaluation We evaluate our method on six mathematical reasoning benchmarks: AIME 2024 AIME 202.5] AMC (Li et al. 2024), Minerva ([ewkowycz et al. 2022). Olympiadßench (He et al. 2024]. and MATH-500 (Hendrycks et al., (021). Our main results report avg for the smaller test sets (AIME 2024, AIME 2025, AMC) and pass @ for the larger benchmarks. For Pass evaluation, we generate completions and select the one with the highest reward score. For cross- domain generalization, test ARC-c (Clark al. 2018), GPOA-diamond (GPQA*) (Rein et al., 2024). and MMLU-Pro.
RL Practice We set GPPO w/ Clip-higher. Detailed implementation parameters are provided in Table 3 All training experiments are conducted using A100 GPUs. We train 350 steps Qwen2.5-Math-77h and 200 steps for others. Our implementation is based on ver["] which uses vLLM as the rollout generators. We are thankful for these open-source repositories.
remove the KL loss term and use 0.28 for higher clip following
Table 2: Computation overhead analysis.
DIVER(ours) GRPO w/ Clip-Higher Entropy-RI Pass@k Training
Time Cost (350 Steps)
29.75h
28.46h
28.13h
30.73h
The computation overhead of calculating textual diversity and equational diversity is negligible, since both metrics only involve rule-based calculations (e.g., n-gram matching for BLEU score or string recognition for equation extraction) without any feedforward or backpropagation of large- scale models. As shown in Table DIVER only incurs 5% increase in training time compared to GRPO and Entropy-RL baselines, while requiring less training time than Pass@ @ Training bascline.
B.3 SYSTEM PROMPT
We use the same system prompt for training and inference in all our models except LLaMA-3.1-8B. Instruct:
8192
stttte///agggnnseer//////s/s//sr 2024
htte:e//hogghnnaeeec//rrsssssssr hhups//ggthhb.com/oooccgggggeee
18

---

Published as a conference paper at ICLR 2026
Table 3: Hyperparameter settings
Hyperparameter
Value
Hyperparameter
Value
max prompt length
1024
KL coefficient
0.0
response length
8192
train temperature
1.0
num generations G
eval temperature
0.6
gpu memory utilization
0.85
entropy coefficient
0.0
learning rate train batch size mini batch size
le-6
high clip ratio en
0.28
128
low clip ratio ej
0.20
32
shaping ratio X
0.1
use dynamic batch size
True
diversity upper bound
0.65
validate batch size
512
Your task is to follow a systematic, thorough reasoning process before providing the fi- nal solution. This involves analyzing, summarizing, exploring, reassessing, and refining your thought process through multiple iterations. Structure your response into two sec- tions: Thought and Solution. the Thought section, present your reasoning using the for- mat: "<think>>y in thoughts </think>\n" Each thought should include detailed analysis, brainstorming, verification, and refinement of ideas. After "</think>> (n" the Solution section, provide the final, logical, and accurate answer, clearly derived from the exploration the Thought section. If applicable, include the answer in [boxed[] for closed-form re- sults like multiple choices mathematical solutions. This is the problem: {QUESTION} Assistant: <think>
User:
For LLaMA-3.1-88 1.-MM---3.--BB-nsttuct we use a simplified prompt which only includes the CoT prompt:
User: [QUESTION]
Answer: Let's think step by step.
19

---

Published as a conference paper at ICLR 2026
C DETAILED METRICS DEFINITION
C.I. P PERFORMANCE
Given a question we employ the model to generate k independent and identically distributed (i.i.d.) responses. Each response evaluated by binary reward function, yielding r; E 0, where 1 indicates a correct response. The pass@k metric quantifies the probability of obtaining at least one correct response among the samples:
p P
V (n 1) =E
r.)
(14)
While pass @ evaluates evaluates the model's accuracy first attempts. pass @ metric emphasizes the model's 's ability to generate diverse solutions and improve success rates through sampling. To rigorously demonstrate DIVER's consistent advantage, we repeat all experiments across Pass to Pass@32, and report the means and standard deviations for a total of runs. As shown in figure 6 and table our method consistently outperforms all baselines approaches (GRPO w/ Clip-higher, Entropy-RL, and Pass @k Training) across the spectrum of pass @ k metrics (k=2 to k=32) on both in- distribution benchmarks and out-of-distribution benchmarks. The performance gap particularly significant at higher k values, where DIVER demonstrates superior exploration capabilities and achieves the highest average scores.
Table 4: pass performance.
In-Distribation Performance
Out-of-Distributiin Performsacce
Pass
Method
AIME 24/25
AMC
MATH--50
Minerva
Olympiad
Avg.
ARC-c GPOA" MMLU-PPr Arg.
Qwen2.5-Maill-7B
23.3 3.3/122
55.4
74.7 0.8
27.2
1.3
35.5
08
38.1
89.2
$2.0
57.4
Clip-higher
19
85.7
Entropy-RL
Training
2333::34/22
DIVER(ours)
67.9
1.4 87.7 0.1 42.9
52.1
512
90.3
$7.7
60.2
69.4
Qwen2.5 Maill
30011 5.8 67.1 3.
815
0.6 35.5
438
45.8
92.3
65.0
66.4
74.6
GPRO w/ Clip-higher
122--11.772.55
3.9 72.3
88.
1.1 47.3
533 133-00
92.3
65.0
66.4
74.6
Entropy RL
38.9 3.8/28. 3.8
=
89.5
0.1
40.6
1.2
54.7
$4.5
92.2
64.3
66.6
74.4
Pass 8% Training
367 8-4 71.3
3.0
88.5
= 0.2
42.2
1.1
55.7
0.7
53.2
92.2
66.5
64.8
74.5
DIVER(burs)
37.8 4: 1.9/24.4
75.9
4.3
90.5
0.5
47.3 1
1,9 57.7
0.4
55.6
1.6
93.3
70.4
69.1
77.6
Qwen2.5-Maile 5-Math-BB
12.2 1: 5.1/233 3:1
77.1
2.1
86.7
1.2
11.3
53.7
95.3
75.5
74.3
81.7
Clip-higher
356 5.1/30.
77.9
1.8
90.7
75.5
Entropy-RL
Training
DIVER(ours)
4566 :: 1,9/30.0 0.0
83.9
92.9
0.1 50.6
61.8
0.7 60.8
95.9
$1.1
76.3
Matn2.5-Math-7B
43.3 12.0/23.3 1
81.1 3.0
89.9
0.3
19.1 1
56.5
56.0 2.2
96.8
84.7
80.2
87.2
GPRO w/ Clip-higher
45. 5,1/37.
84.3
143-244 24
92.3
59.3 + 2.2
63.
2.3
96.8
84.7
81.2
87.2
k
Entropy RR
$8.0 2.1
93.7
0.2
50.7 + 1.6 62.8
0.3
63.2
95.7
78.6
79.7
84.7
Pass 0% Training
50.0 1 6,7/3441
8.1
2.
92.3
1 0.1
50.6
1.8 61.0
0.5
62.3
0.2
95.9
82.7
77.7
85.4
DIVER(
47.8 =: 5.1/38.
3.8
88.0
0.0
94.4
0.4
56.5 +: 2.9 66.0
0.7 65.3
1.0
97.3
89.3
81.7
$9.4
Qwen2.5 5-t.---MhhPP
52.2 1: 5.1/3.00
88.8
92.1
0.5
54.4
1.0
62.2
0.8
63.3
0.9
97.9
89.8
85.8
91.2
Clip-ligher
56.7 10.0042.221
92.0
94.4
54.9
65.8
0.8
97.9
89.8
85.8
91.2
Entropy-Ee
2.042.2.222211111
Phesétkk Training
56.7
5.8/3333
0..4007
$3.8
(IuER()urs)
58.9
3.8/8.9899
1.9
92.4
95.4
64.4
69.0
70.5
98.0
89.8
87.0
C.2 BLEU SCORE
BLUE measures the similarity between a candidate string e and a reference string by calculating the n-gram (short phrases of length n)) precision, while also penalizing overly short outputs through
20

---

Published as a conference paper at ICLR 2026
Table 5: Performance comparison with larger pass @ values.
Benchmark
Method
pass@ 128
pass@256
pass 512
pass 1024
Owen2.5-Math-
63.3
70.0
76.7
80.0
GPRO w/ Clip-Higher
70.0
73.3
76.7
76.7
AIME24
Entropy-RI.
70.0
73.3
76.7
76.7
Pass@k Training
70.0
76.7
80.0
80.0
(VIER(
80.0
83.3
86.7
Qwen2.5-Man -5tth.5-
40.0
56.7
63.3
66.7
GPRO w/ Clip-Higher
63.3
66.7
70.0
AIME25
Entropy-RL. RL Pass Training
63.3
66.7
73.3
63.3
63.3
70.0
73.3
(VIER(
66.7
70.0
76.7
Owen2.5-Math--
92.8
97.6
97.6
97.6
GPRO w/ Clip-Higher
94.0
96.4
97.6
98.8
AMC
Entropy-RR RL
96.4
97.6
98.8
Pass@k Training
95.2
96.4
97.6
100.0
(oIE)(
96.4
97.6
98.8
100.0
brevity penalty as
N Σ
Pn EgEG, min(C.(o).
BLEU = BP exp
Wn Pn Pn
(15)
Egea, Ce(g)
where N is the maximum n-gram length considered, Pn the modified precision for n-grams of size Wn is the weight for each n-gram level (usually uniform, e.g., Wn 1/N), G, is the set of rn-grams, and Co(g)/C, (g) counts how often the r-grrm g appears string c/r. The brevity penalty BP is defined as
11 [exp(1 - |r//|c),
ld 12/r
BP
(16)
otherwise.
C.3 EQUATION EXTRACTION
We extract mathematical formulas using regular expressions that identify three common LaTeX notation patterns: \ [ AAJ.V AV, and $ $. Our implementation employs re. findall ()
capture these patterns and stores them set structure eliminate duplicates.
To evaluate the extraction accuracy, we compare the automatically extracted equation counts against cuman-annotated ground-truth values. For demonstration, randomly sample responses from different query inputs. Table 0 presents the equation counts obtained through our extraction method alongside the corresponding ground-truth values. The observed failure rate is approximately (8/%))))).
We note that since Equational Diversity (ED) serves as an intrinsic reward signal to encourage ex- ploration during training, perfect extraction accuracy is not required. The metric only needs to approximate response diversity with sufficient fidelity to guide effective exploration. The observed accuracy level is adequate for this purpose, evidenced by the consistent performance improve- ments shown in our main results.
Table 6: Equation extraction accuracy evaluation on randomly sampled responses.
Response
Response 2
Response 3 Response 4 Response 5
No. of equations via extraction
14
25
13
17
18
Ground-truth value
17
25
18
17
18
21

---

Published as a conference paper at ICLR 2026
D MORE ANALYZE
D.1 ANALYSIS OF DIVERSITY METRICS VALIDITY
To verify whether TD and ED trily capture semantic-level reasoning diversity, we conduct cor- relation analysis comparing them with two semantic similarity metrics: 1) LLM-based scores: We compute the cosine similarity of the model's 's hidden states from the final layer outputs using Qwen2. 5- Ins ruct ruct for all baselines and our method; 2) Embedding- based scores: We use google beapbedddngggeenmn m000 to encode responses into embeddings and compute their co- sine similarity.
For each method (DIVER and baselines), we generate multiple responses per prompt and compute all diversity metrics within each response group. Evaluated on 1,560 rollouts total, the average met- rics are shown in the table (t indicates higher values represent greater diversity; I indicates the opposite). The results demonstrate strong consistency between TD/ED and LLM-/Embeddingg -based alternatives. Methods achieving higher TD/ED scores consistently exhibit lower LLM-/Embeddinng based semantic similarity scores, confirming that our choice of diversity metries effectively captures meaningful semantic diversity rather than merely surface-level variation. This validates the general- ity of TD and ED as reliable proxies for reasoning diversity.
Table 7: Correlation between diversity metrics across different methods. Higher TD/ED (1) and
lower LLM/Embeddingg similarity scores (L) indicate greater diversity.
Method
(t)
ED (1)
LLM-based ())
Embedding-based ()) pass@
pass @8
DIVER(ours)
0,702
0.477
0.936
0.881
43.1
60.8
GRPO w/ Clip-Higher
0,805
0.575
0.916
0.841
40.7
57.4
Entropy-RI.
0.549
0.294
0.,477
0.954
41.8
58.0
Pass@k Training
0.441
0.200
0.959
0.967
41.5
56.3
Moreover, our diversity metrics both computationally lightweight and easy to implement. The computation overhead of calculating textual diversity and equational diversity is negligible, since both metrics only involve rule-based calculations (e.g., n-gram matching for BLEU score string recognition for equation extraction) without any feedforward or backpropagation of large-scale mod- els (e.g., the above LLM-/Embedding based metries).
D.2 ANALYSIS REASONING QUALITY
To validate whether our approach achieves higher quality and more meaningful reasoning results, we evaluate all test responses using deapseek-ai /Deepeeee--eeeeeeeeeeeeeeeeeeeee 2-Exp as a judge model, rank- ing responses generated by the base model, baselines, and DIVER for each prompt. The evaluation comprehensively assesses four key dimensions: Completeness, Logical Cons Consistency, Relevance, and Efficiency. The table g presents the final result over all six benchmarks: ALME24/25, AMC, MATH-500, Minerva, and Olympiad. It demonstrates DIVER's consistent su- periority, achieving 1.52 average ranking compared to baseline methods ranging from 2.57 to 3.1. DIVER solutions rank best (Rank 1) in 76.4% of cases-more than double the next best method. This confirms our diversity measures capture meaningful improvements in reasoning quality, not merely surface-level variations.
D.3 ANALYSIS OF MODEL SCALE AND REASONING HORIZON
We further explore DIVER's adaptability across various language models, including small, weak or different architecture models. As shown in Figure DIVER consistently outperforms baselines across all model variants, demonstrating its strong generalization capabilities.
To further validate DIVER's effectiveness models with longer reasoning horizons, conducted 7B as the base model, which typically
additional experiments using SeopSeek-RI-Dstill--- generates much longer responses (2500-3500 tokens). The results in Table show that DIVER maintains its consistent superiority even with these extended reasoning processes, achieving 60.1 average in-domain performance and 50.9% out-of-domain performance, compared to GPRO w/
22

---

Published as a conference paper at ICLR 2026
Table 8: Ranking reasoning quality across six mathematical benchmarks as judged by DeepSeek-
V3.2-Exp. Overall Avg. shows aggregate performance and lower Avg. Rank(J) is better.
Benchmark
Model
Avg. Rank(()
Rank 1(best)
Rank2
Rank 3
Rank4
Rank5 (worst)
wwen2.5-Math-7B
3.19
15.8%
23.6%
16.7%
14.8%
29.1%
GPRO w/ Clip-Higher
2.77
21.1%
23.9%
24.9%
17.8%
13.2%
Overall Avg.
Entropy-RI.
2.86
22.9%
19.7%
22.0%
22.9%
12.5%
Pass@k Training
31.7%
22.0%
16.8%
17.7%
11.8%
DIVER(ours)
1.52
76.4%
7.5%
3.8%
5.2%
3.53
6,7%
23.3%
16.7%
16.7%
36.7%
GPRO w/ Clip-Iligher
6.7%
30.0%
23.3%
16.7%
AIME24
Entropy-RL
3.47
6.7%
13.3%
30.0%
26.7%
23.3%
Pass@k Training
2.73
26.7%
16.7%
20.0%
30.0%
6.7%
DIVER(ours)
1.77
66.7%
6.7%
16.7%
3.3%
6.7%
Owen2.5-Math-1.
3.73
0.0%
30.0%
10.0%
16.7%
43.3%
GPRO w/ Clip-Hligher
2.83
16.7%
26.7%
23.3%
23.3%
10.0%
AIME2S
Entropy-RI. RL.
3.00
20.0%
13.3%
233.3%
33.3%
10.0%
Pass@k Training
2.70
26.7% 766
20.0%
26.7%
10.0%
16.7%
DIVER(ours)
1,83
60.0%
13.3%
16.7%
3.3%
6.7%
3.17
20.5%
18. 18%1%
15.7%
15.7%
30.1%
GPRO w/ Clip-Higher
21.7%
20.5%
31.3%
15.7%
10.8%
AMC
Entropy-RI.
2.82
26.5%
18.
18.
21.7%
15.7%
Pass@k Training
2.49
31.3%
25.3%
16.9%
15.7%
10.8%
DIVER(ours)
1.27
85.5%
8,4%%
1.2%
3.6%
1.2%
Qwen2.5. Maath-B
39,8%
21.6%
9.0%
16.6%
GPRO w/ Clip-Higher
2.27
43.8%
18.2%
16.2%
11.0%
10.8%
MATH-500
Entropy-RL
2.24
43.4%
20.0%
13.6%
14.8%
8.2%
Pass@k Training
2.19
48.2%
16.2%
13.4%
12.6%
9.6%
DIVER (ours)
1.40
82.6 %
5.0%
4.8%
4.8%
2.8%
Qwen2,5-Math-1 5-Math-7B
3.14
12.1%
26.8%
21.3%
14.3%
25,4%
GPRO w/ Clip-Higher
2.93
17.6%
22.8%
24.3%
19.5%
15.8%
Minerva
Entropy-RI.
2.79
18.0%
25.4%
26. 1%
20.6%
9.9%
Pass@k Training
2.70
25.4%
25.0%
21.3%
12.1%
DIVER(ours)
1.41
82.4%
5.9%
4.4%
2.6%
4.8%
5-Math-7B
3.18
15.7%
22.
18.5%
16.2%
27.6%
GPRO w/ Clip-Higher
2.72
20.4%
25.4%
28.0%
14.0%
12.1%
Olympiad
Entropy-RI.
2.81
21.1%
22.8%
20.9%
24.2%
10.9%
Pass@k Training
2.60
25.9%
28.5%
17.3%
16.4%
11.9%
DIVER(ours)
1.45
81.2%
5.9%
4.0%
4.8%
4.0%
Table Overall performance on six competition-level benchmark performance on Qwen2.5-Math- 1.5B, Qwen2.5-7B-Baal LLaMA-3.1-801 100-3.1-8B-IA Instruutt and DeepSeek-RI -1-Disttll---ee---7
In-Distribution Performance
Out-of Out-of-Distribation Performance
Model
AIME 24/25
AMC MATH-500
Minerva
Olympiad
Avg.
ARC-
GPQA
MML.U.Pr
Avg.
Qwen2.5-3ke 5-Math-1.3B
GRPO Clip-higher
9.9/7.7
42.8
68.6
20.6
30.5
59.5
30.2
39.9
DIVER
11.088.3
44.3
71.8
23.9
33.6
32.2
33.2
59.1
31.8
41.4
Q.een.5.78-Bort
GRPO w/ Clip-higher DIVER
15.8/12.1
49.2
78.2
27.6
39.1
37.0
89.8
28.6
55.4
57.9
20.9/12.9
76.0
38.7
59.2
LLaMA-3. 1-8-8--IIttrr
GRPO w/ Clip-higher DIVER
6.9/1.4
23.3
$3.0
23.5
22.0
21.8
89.2
14.8
50.8
$1.6
7.7/1.5
28.8
56.4
22.8
21.7
23.1
88.2
20.4
52.0
53.5
DeepSeek- RI-Diilll Qwen-7B
GPRO w/ Clip-Higher
49.3/35
81.7
90.2
35.7
59.0
58.5
85.2
2.6
$4.9
47.5
DIVER(our)
51.1/36.9
82.0
92.6
36.8
61.2
60.1
85.6
10.7
56.6
50.9
Clip- Higher'ss''s ' 58.5% and 47.5% respectively. This confirms that our approach scales effectively to longer-horizon reasoning scenarios.
23

---

Published as a conference paper at ICLR 2026
D.4 MORE COMPREHENSIVE ABLATIONS
Test Scores
Analysis of diversity reward shaping coefficient. We further analyze the sensitivity of DIVER to the diversity reward shaping coefficient Fig. TO demonstrates performance remains stable across different À values, with test score curves following sim- ilar trajectories regardless of coefficient magnitude. Even when becomes relatively large, we observe only minor performance degradation without triggering reward hacking. This robustness to hyperparameter selection makes DIVER practical for real-world applications, it doesn't 't require precise tuning of diversity re- ward weights.
0.40
0.35
A=0.00
A=0.01
0.30
A-0.1
A-1.0
100
200
300
400
Steps
Figure 10: Average test scores with varying coefficients (A).
Longer Horizons Improve Performance. It important to clarify that the "horizon" in anal- ysis refers to the window size used for calculating diversity metrics, rather than the full response length, Fig [1 For instance, when use horizon 200, we compute Textual Diversity (TD) and Equational Diversity (ED) metrics only on the first 200 tokens of a generated response, even though the complete response may be much longer (1600-1806 tokens in our main experiments).
all takens
1000 tokens
500 tokens
200 tokknss
AIME
AIME25
AMC
MATH 500
0.60
0.22
0.16
0.20
0.80
0.16
0.50
0.14
0.10
0.70
0.08
0.06
0.40
100
200
300
100
200
300
100
200
300
100
200 Steps
300
Stopps
Steps
Stops
Minerva
Olympiad
Average
Enterpy
0.40
0.40
0.5
0.2
0.30
0.25
0.0-
100
200
300
100
200
300
100
200 Stops
300
100
200
300
Stops
Steps
Stops
Figure 11: DIVER performance under different token horizon constraints.
Hyperparameter Robustness. To analyze hyperparameter sensitivity, we evaluate DIVER and baseline across different group sizes and training temperatures. As shown in Table I0 DIVER demonstrates superior robustness to hyperparameter variations. When reducing group size from G 8 G 4, DIVER shows smaller relative performance decline than GPRO w/ Hlip-Higher, while maintaining 2.5 point advantage. Similarly, when decreasing temperature from t = 1.0 to =06,, DIVER exhibits minimal performance fluctuation (0.3 point drop) compared to the baseline's more significant 1.2 point reduction. This stability across different configuration settings highlights DIVER's reliability in practical scenarios where optimal hyperparameter tuning may be constrained.
Table 10: Performance comparison across different hyperparameters.
Hyperpaarameter
Model
AIME AIME 25
AMC
MATH-500
Minerva
Olympiad
Avg.
Group Size G
Clip-Higher
164
40.7
DIVER
23.8
1644
60.9
84.4
29.4
44.0
43.1
GPRO w/ Clip-Higher
16.3
12.1
55.8
78.6
30.9
39.1
38.8
DIVER
20.5
14.
29.4
42.2
41.3
Train Temperature
GPRO w/ Clip-Higher
18.9
16.4
$7.3
81.2
28.7
41.5
40.7
DIVER
23.8
16.4
60.9
84.4
29.4
44.0
43.1
0.5
GPRO w/ Clip-Higher
DIVER
20.6
18.0
14.1
14.4
54.9
59.8
80.0
83.2
29.8
33.1
40.3
42.6
42.8
39.5
24

---

Published as a conference paper at ICLR 2026
E
CASE STUDY
Question and Answer
<system prompt This is the problem: For k 3, we define an ordered k-tuple of real numbers (I1, U2, [k) to be special if, for every i such that 1 < i S k. the product 21 22 .... Fk 2?. Compute the smallest value of it such that there are least 2009 distinct special k-tuples.
Prompt:
Answer: "$12$"
DIVER Case
Response 1:
<think> 1
31 T2
Fh 1.
Then for each i:
Tj or 24
Each z; can be either 1 or -1, and the product of all æ; must be We need count the number of such k-tuples. The number of k-tuples of I's and -I's (i.e., each æ; can be 1 or -) whose product 1 can be counted as follows: the number of -1's must be even (since an even number of - - will result in a product of 1). The total number of k-tuples where each z; is either changing the sign of any one element changes the parity of the number of -1 ss), hence there are 2* such tuples. 2. If P -1:
1 is 2 Half these tuples will have an even number of -1's (since
T2
Tk 11
Then for each i:
-1 = x²
This is impossible for real numbers since x² 2 for all real 2i. Thus, there are no solutions when / = -1 with real numbers. Therefore, the valid solutions come only from the case P 1, and there are 2 such k-tuples. We need 2 2009:
2 > 2009.
To find the smallest k:
k: > log, 2009.
We calculate log, 2009:
log, 2009 Rs 10.97
Since 1 must be an integer, take the ceiling of 10.97, thus:
k-1211
k 2 12.
</think>
From the analysis above, the smallest & such that there are at least 2009 distinct special k- tuples is & 12, because 2 2048 > 2009, and & 11 would yield only 220 1024 tuples, which is less than 2009.
Hence, the smallest l for which there are at least 2009 distinct special k-tuples is = 12. The final answer is:
12
Response 2:
25

---

b (2009). Published as a conference paper at ICLR 2026
<think>
For P # O, we can divide both sides by P:
1 1
For this equation to hold, two conditions must be satisfied: 1. p = 1, implying - 0 or P (because P is a real number). 2. (-1) 1, implying that n must be even.
From 2/4 1 = O, we get:
k: 2
k = 2. =
However, the problem states k: 2 3. so k: is not valid. Therefore, P 1. So, P 1 and thus (vP)² = 1, meaning z? for all i. Therefore, each H; can only be or - Now, we need (-1) = 1. meaning k- - n is even, implying n and & have the same parity (both are either even both are odd). Next, let's determine the number of distinct k-tuples. If each 2; can be either 1 or 1,, there are 2 possible k-tuples in total. However, not all of them will satisfy the condition, because they must maintain P
The product 21 ...
and only if an even number of z; 's are 1. The number of ways to choose an even number of 1''s out of positions given by the sum binomial coefficients of even indices:
Σ OSISk I even
is known that the sum of binomial coefficients of even indices equals 2
Σ OSj even
2k
=
Thus, there are 2 distinct special k-tuples. We need 2 > 2009:
2 2 2009.
To find the smallest k:
Since logg((20g)) = 10.97, round up to the next integer because k must be integer:
k 1 12 11.
This gives 2 12. </think> The smallest that satisfies 2*
2009 is k 12. To verify, when k 12:
12-1 221 2048,
which is indeed greater than or equal to 2009, Thus, the smallest & such that there are at least 2009 distinct special k-tuples is 12. Therefore, the smallest l is:
12
Entropy-RL Case
Response 1:
<think>
the given problem states that k > so we need to reconsider another approach since vP and then need
isn't applicable directly for k > 3. So let's say P 12 where
26

---

Published as a conference paper at ICLR 2026
t = If lett vP, get
2 = 1 = 1-- = = e( - 1
This implies that 12 Or th 1. First, if = 0. then O, and thus all the coordinates should zero yielding once again the trivial solution, leading to the previous case. For t = 1. we have two solutions:
t. = and 2. = -1
then P 1, and each x; +1, and when 11 P 1, and again each I; = +1.
When
Hence, each coordinate can be either -.. giving 2 total combinations of k-tuples. Since we already have the trivial all-zero solution giving just one, we'll focus on the remaining ones, the total number of solutions 2* 1. We need 2 2009, Let's calculate smallest k
We need find the smallest integer i such that
2 12 2009;
or
2¹ 2010.
Now let's calculate powers of 2:
2 = 1024,
2 2048.
follows that the smallest value for which 2 2010 is when 11. </think>>
Now that we've reasoned through the problem. we conclude:
That would mean & 11 would yield 2 2048, and then there'd 'd be 2048 1 2047
nonzero solutions plus trivial solution leading to 2048 total tuples; thus = 11 suffices
because even = 10 would not yield enough (as 2 1 1023 2009). So the final answer must be:
Response 2:
<think> least 2009 distinct special k-tuples, must have
2 2009.
We need to find the smallest integer & such that 2* 2009. Let's calculate the powers of
20 1024,
2 2048.
We can see that 2 = 1024 < 2009 and 2 2048 > 2009. Thus, the smallest k such that 2* 2009 is 11.
</think>
The smallest value of & such that there are at least 2009 distinct special k-tuples is
27

---

Published as a conference paper at ICLR 2026
GRPO w/ Clip-higher Case
Response 1:
<think>
I 0, then p= 0, and all x; must be zero, which gives us more solution.
So, for k > 3, we have k - distinct solutions (the h roots of unity plus the solution
r r= We need least 2009 distinct special k-tuples. Therefore, we need k 1 2 2009, which gives us & 2 2010. The smallest value of l: that satisfies this inequality is k = 2010. Let's verify this. For k 2010, we have 2010
2009 distinct solutions from the roots O, which gives exactly 2009 distinct special k-tuples.
of unity plus the solution Therefore, the smallest value such that there are at least 2009 distinct special k-tuples is k 2010. </think>
The smallest value of l such that there are at least 2009 distinct special k-tuppes is
2010
Response 2:
<think> To count the number of distinct special k-tuples, need to consider the number of solutions to these equations. If k - -2 is even, we have at least 2 distinct tuples (1.1...., 1). If A: is odd, To have at least 2009 distinet special k-tuples, we need to find the smallest & such that the number of solutions at least 2009. If k
and (-1,-1,.....
have complex tuples involving -1.
even, we have at least solutions. If k - 2 is odd, we can have more solutions, but we need to count them carefully. For k odd, we can have tuples with even number of -I's and the rest 1's, and permutations of (I.- -r, 11 -1 -l..... -1, 1).
The number of distinct tuples grows quickly with k., and we need to find the smallest & such that the number of solutions is at least 2009. Let's calculate this step by step. For k 2 odd, the number of distinet tuples is related to the number of ways choose positions for and the values of and Let's check the number of solutions for different A:
-For & 3, we have (1, 1) and (-1, -1, 1), solutions. For 4, have 1, (-1, -1 1,--1,1111 and permutations of (x. -x, -1, 1.,-11-111 -1), so more than 2 so- lutions. For &o 5, have 1,,,,, (-1, -1,-1,1 -1 1), and more solutions involving 1.
We can that l: increases, the number of solutions increases. We need find the smallest & such that the number of solutions is at least 2009.
Let's calculate the number of solutions for k 2m + 3 (since l must be odd): For h = 2m + 3, we have at least the tuples with all 1's and all 1 plus permutations of (x.. xr, 1, -l..... -)). The number of such tuples related the number of ways to choose positions for -1 and the values of and =.
The number
distinct tuples for number of ways to choose positions for 1 and x, This grows quickly with k.
2m + 3 at least +
By trial and error, we can see that for = 2011, we have 2 = 2009, which is odd, and can have least 2009 distinct tuples. </think>
The smallest value of & such that there are at least 2009 distinct special k-tuples is k = 2011. The final answer is:
2011
28

---

Published as a conference paper at ICLR 2026
Pass @k Training Case
Response 1:
<think>
Since P pk/2, we have cases consider: 1. If 0: In this case, all x; must be zero because otherwise, if any 2; is non-zero, the product would not be zero. This gives us the k-tuple (0, .....)...) 0), which is indeed solution. For P 0, it forms one special k-tuple. If P + 0: We can divide both sides of the equation by pk/2 (since 0):
= (-1)^-a
This implies that (-1)f-a 1, meaning A: must be even. Consequently, and & must have the same parity (both both odd). Moreover, since P (vP)2 (VP)*. (-1)6-8, we get:
1 1 (-1)
This implies & - a even, so and are of the same parity (both even or both odd). Next, need to count the number of distinct k-tuples. For non-zero P. the number of distinct k-tuples is the number ways to choose (with a S k) such that a and & share the same parity. The number of choices for (with 0 S I k)) having the same parity k is k+1 if is odd and + if even. Including the zero product case, we need the total number of distinct k-tuples be least 2009:
(number of k-tuples with P + 0).
is odd, there are
k+1 choices for a, making the total number of distinct k-tuples:
2
k
2
If k is even, there are + choices for making the total number of distinct k-tuples:
k
+
We need this total to be at least 2009. We will check both cases separately. 1. For & odd:
k+L1 2 2009 1 2
A +
+ 2009,
k + 2 2008,
k + Z 4016,
4015.
2. For ls even: 5/2 + 2009
k + 2 2 2009,
2
A > 2007,
2
4014.
The smallest that satisfies either condition is = 4015 (since 4015 odd and satisfies the first condition). Hence, the smallest & is 4015.
</think>
To find the smallest i such that there are least 2009 distinct special k-tuples, we need to satisfy the condition based on the derived inequalities. We found that for k 2 4015 (when k odd) or 4014 (when even):
1. If k is odd, the smallest & satisfying k 4015. 2. If k is even, the smallest & satisfying
2 4014.
29

---

Published as a conference paper at ICLR 2026
Since 4015 is the smallest value from these cases, it gives the final answer:
4015
Response 2:
<think
Now, let's consider the possible values of ri. Since P = =². and P is either -1 (if k- is even), we have x² or x = -1 The equation x. = has two solutions: x; = or z; -1 The equation x -1 has no real solutions. Therefore, each 2; must be either or -I.
If P I, then the number of 1 in the tuple must be even. P -1, then the number of -1's in the tuple must be odd. We need to count the number of such tuples.
I's Let's denote the number of - the tuple by m. If
1, then m must be even, and if P 11, then must be odd. The number of ways choose positions out of k for the -I's is (()). So, the total number of special k-tuples is the sum of (*) for all even m plus the sum of (+)) for all odd m. is known that the sum of the binomial coefficients for even m and odd m are equal and each sum to 2 Therefore, the total number of special k-tuples is 2 + 2 2
We need this to be at least 2009. So, we need to find the smallest such that 2 2 2009.
We can calculate the powers of 2:
20 1024,
2 2048.
Since 2048 2 2009, the smallest is 11. We can verify that 2 2048, which is indeed
greater than or equal to 2009. Therefore, the smallest value such that there are at least 2009 distinct special k-tuples is 11.
</think>
The smallest value of l such that there are at least 2009 distinct special k-tuples is
11
30

---

Published as a conference paper at ICLR 2026
F THE USE OF LARGE LANGUAGE MODELS (LLMS)
We utilize Large Language Models (LLMs) to assist with polishing the writing and improving text readability. Specifically, LLMs are employed for proofreading, enhancing grammar, and refining sentence structure. The LLM was used solely for editorial purposes to improve clarity and did not contribute research ideation, experimental design, implementation, analysis, or scientific conclu- sions. All core research contributions, experiments, and analyses were conducted independently by the authors without LLM assistance.
31


Reviews:
[{"review_id": "PBXbPmH7ab", "reviewer": "Reviewer_wris", "summary": "This paper introduces DIVER (Diversity-Incentivized Exploration for Versatile Reasoning), a framework that uses global sequence-level diversity as an intrinsic reward to enhance exploration in reinforcement learning for Large Language Models (LLMs). The framework incorporates diversity incentives as an intrinsic reward and utilizes a potential-based reward shaping mechanism to ensure policy invariance and mitigate reward hacking. Empirical results demonstrate that DIVER outperforms existing RLVR baselines, improving performance on both in-domain and out-of-domain tasks, with notable improvements in the Pass@1 and Pass@k metrics.", "questions": "For rollout data that already exhibits high entropy, is it still necessary to introduce additional diversity incentives? Could this approach unintentionally encourage false positives, where the model generates diverse but incorrect responses? How does DIVER ensure that diversity incentives do not lead to an over-exploration of irrelevant or incorrect reasoning paths?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper studies an important problem on how to boost the exploration \n\n2. The paper provides comprehensive experimental results, evaluating DIVER on multiple benchmarks (in-domain and out-of-domain). It also assesses performance from different perspectives, including final benchmark performance and training dynamics.\n\n3. DIVER achieves performance improvement across different benchmarks, but the improvement is not consistent compared to baselines methods.", "weaknesses": "1. While Textual Diversity (TD) and Equational Diversity (ED) are designed to incentivize exploration, it is unclear whether they truly measure meaningful diversity in reasoning. For example, ED could be easily hacked by generating irrelavent formulas. As shown in figure 4, DIVER leads to a decrease in the solve_all rate, which raises concerns about whether the model is exploring paths of lower quality.\n\n\n2. The paper claims that \"Longer Horizons Improve Performance\", but as shown in Figure 7 and 8, all experiments use response length at 1-2k tokens. It remains unclear whether this finding holds for thinking models with longer horizon reasonings. Verification on such models is needed.\n\n3. More comprehensive ablations are necessary to demonstrate the effectiveness of the proposed method. As show in Figure 9, a larger $\\lambda$ could bring almost no performance improvement.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mWxBvNncKf", "reviewer": "Reviewer_mNMY", "summary": "The paper proposes DIVER, a reinforcement learning framework that enhances reasoning in LLMs by introducing global sequence-level diversity as an intrinsic reward.Built upon GRPO, DIVER uses a potential-based reward shaping to preserve optimal policy invariance and mitigate reward hacking.Experiments on multiple math reasoning benchmarks show consistent but modest gains in Pass@1 and Pass@k (up to 32) compared to Entropy-RL and Pass@k Training.", "questions": "1. One confusing point is that in Table 3, GRPO is actually the best baseline in most cases (compared to Entropy-RL and Pass@k Training), which seems inconsistent with the expected performance of these methods.\n\n2. Would DIVER still outperform GRPO and the base model at larger values of k?\n\n[1] Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Shiji Song, and Gao Huang. Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? arXiv preprint arXiv:2504.13837, 2025.\n\n[2] Zhipeng Chen, Xiaobo Qin, Youbin Wu, Yue Ling, Qinghao Ye, Wayne Xin Zhao, and Guang Shi. Pass@K Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Models. arXiv preprint arXiv:2508.10751, 2025.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper presents a clear motivation and a well-structured method for enhancing exploration in LLM reasoning.\n\n2. Experiments are comprehensive and demonstrate consistent improvements across multiple benchmarks and model scales.", "weaknesses": "1. The improvement on Pass@k is relatively small and only evaluated up to k = 32, while [1] tests up to k = 1024, including the base model.\n\n2. The comparison on Pass@1 in Table 1 seems not entirely fair, since methods like Pass@k Training [2] mainly aim to improve Pass@k. Their Pass@1 performance is usually enhanced through a two-stage training process (first optimizing Pass@k, then fine-tuning for Pass@1).\n\n3. In Table 3, GRPO mostly remains the best baseline, and the base model results are not included, making DIVER’s advantage less consistent.\n\n4. The diversity metrics (BLEU, formula overlap) may not effectively capture semantic-level reasoning diversity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MTVCvbf6nE", "reviewer": "Reviewer_RcXm", "summary": "The paper identifies global sequence-level diversity as a driver for better exploration and improved reasoning capability in RL finetuning of LLMs, and proposes DIVER: an intrinsic-reward framework that (1) measures sequence diversity (Textual Diversity via inverted BLEU, and Equational Diversity via formula uniqueness), (2) injects diversity as a potential-based shaping reward to preserve optimal policy invariance, and (3) uses simple heuristics to mitigate reward-hacking. Experiments on several math benchmarks and general domains show consistent gains over GRPO and other exploration baselines; ablations and diagnostics support the claims.", "questions": "see weaknesses", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The motivation to enhance exploration through global sequence-level diversity is clear and compelling.\n2. The method is simple and pricipled.\n3. The experimental evaluation is thorough.", "weaknesses": "1. Limited generality of diversity metrics: The choice of diversity metrics lacks clear guidance. The two proposed ones have different strengths and weaknesses and may not generalize to all scenarios. It would be natural to consider combining them or testing alternatives such as LLM-based or embedding-based measures.\n2. No analysis of computational cost: The paper does not discuss whether computing diversity metrics adds significant overhead or affects training efficiency.\n3. Theorem 1 offers limited novelty: It follows directly from prior reward-shaping theory[1], and the reward-hacking solution breaks its assumptions, reducing its relevance.\n4. Insufficient ablations and sensitivity analysis: Key components and hyperparameters (e.g., λ, reward-hacking solution) are not thoroughly tested for robustness.\n5. Missing discussion of potentially related prompt selection methods: Prior work promotes uncertainty by selecting prompts with success rates near 0.5 [2,3,4]. Since both pursue response uncertainty, their connection deserves discussion.\n\n[1] Policy invariance under reward transformations: Theory and application to reward shaping.\\\n[2] Self-Evolving Curriculum for LLM Reasoning\\\n[3] Online Difficulty Filtering for Reasoning Oriented Reinforcement Learning\\\n[4] Can prompt difficulty be online predicted for accelerating rl finetuning of reasoning models?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FN5UBPhjUa", "reviewer": "Reviewer_BRzv", "summary": "This paper studies RL from verifiable rewards for LLM reasoning. The authors argue that token-level exploration is not enough, and we should explore entire reasoning paths. They propose a small change to GRPO: for each prompt, sample G rollouts, measure sequence-level diversity across that group, and add this as an intrinsic reward. The bonus is only given when a rollout is verified correct and is clipped. On math benchmarks (AIME/AMC/Minerva/OlympiadBench/MATH-500) the method improves Pass@1/Pass@k. Overall, the contribution is mixed: conceptual (global/trajectory exploration) and algorithmic (a training-time diversity bonus) with solid empirical support in the math setting.", "questions": "- How exactly do you extract equations for ED (parser, regex, normalization)? What is the failure rate?\n- What is the training overhead (tokens/sec, wall-clock) vs GRPO?\n- How sensitive is the method to λ, group size G, and temperature?\n- Could you make the code available for reproducibility?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- Very simple, plug-and-play idea. Easy to add to GRPO and existing RL-LLM training frameworks.\n- Clear empirical gains on hard math tasks and strong improvements.\n- Training diagnostics are helpful (diversity/entropy/“solve-none”).\n- The correctness mask and clipping are practical and effective to avoid length explosion.\n- Works across several backbones and seems to be applicable to various LLM architectures.", "weaknesses": "- Theory vs practice mismatch. The shaping argument (policy invariance) assumes a state-only potential. Here the bonus is group-dependent and masked by correctness. The paper does not provide a formal justification for invariance in this setting.\n- Equational diversity extraction is under-specified. The paper defines the set metric but does not describe the extraction pipeline (parser/regex, failures, error rate). Hard to reproduce precisely.\n- The paper lacks ablation studies to understand the impact of the hyperparameters or the performance of the method. \n- Scope limited to single-turn math. The “versatile reasoning” claim is not tested on code, multi-turn, or tool-use tasks.\n- Code is not available which hinders reproducibility.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)