Diversity-Aware Policy Optimization for Large Language Model Reasoning
Authors: Jian Yao
Ran Cheng
Xingyu Wu
Jibin Wu
KC Tan
Venue: NeurIPS 2025 spotlight ()
Keywords: LLM
Policy Optimization
Diversity
Primary area: deep_learning
TL;DR: We propose a diversity-aware policy optimization method for LLM reasoning that introduces token-level diversity focusing on positive samples, achieving higher performance improvement on mathematical benchmarks while generating more diverse solutions.

Abstract: The reasoning capabilities of large language models (LLMs) have advanced rapidly, particularly following the release of DeepSeek-R1, which has inspired a surge of research into data quality and reinforcement learning (RL) algorithms. Despite the pivotal role diversity plays in RL, its influence on LLM reasoning remains largely underexplored. To bridge this gap, this work presents a systematic investigation into the impact of diversity in RL-based training for LLM reasoning, and proposes a novel diversity-aware policy optimization method. Across evaluations on 12 LLMs, we observe a strong positive correlation between the solution diversity and potential@k (a novel metric quantifying an LLM’s reasoning potential) in high-performing models. This finding motivates our method to explicitly promote diversity during RL training. Specifically,  we design a token-level diversity and reformulate it into a practical objective, then we selectively apply it to positive samples. Integrated into the R1-zero training framework, our method achieves a 3.5\% average improvement across four mathematical reasoning benchmarks, while generating more diverse and robust solutions.

Diversity-Aware Policy Optimization for Large Language Model Reasoning
Jian Yao', Ran Cheng Xingyu Wu', Jibin Wul Kay Chen Tan'
Department of Data Science and Artificial Intelligence, The Hong Kong Polytechnic University
2 Department of Computing, The Hong Kong Polytechnic University
The Hong Kong Polytechnic University Shenzhen Research Institute, Shenzhen, China
(ran-peter.cheng. xingy.wi wu, jihin.wu, kctan) @polyu.edu.hk
Abstract
The reasoning capabilities of large language models (LLMs) have advanced rapidly, particularly following the release of DeepSeck- -1,, which has inspired surge of research into data quality and reinforcement learning (RL) algorithms. Despite the pivotal role diversity plays in RL, its influence on LLM reasoning remains largely underexplored. To bridge this gap, this work presents a systematic investigation into the impact of diversity in RL-based training for LLM reasoning, and proposes novel diversity- aware policy optimization method. Across evaluations on 12 LLMs, we observe a strong positive correlation between the solution diversity and Potential@1 @k (a novel metric quantifying an LLM's reasoning potential) high-performing models. This finding motivates our method to explicitly promote diversity during RI. training. Specifically, we design a token-level diversity and reformulate it into practical objective, then selectively apply it to positive samples. Integrated into the RI-zero training framework, our method achieves 3.5% average improvement across four mathematical reasoning benchmarks, while generating more diverse and robust solutions. The code is available at zero_Div
1 Introduction
Recently, the reasoning capabilities of large language models (LLMs) have made remarkable progress, with significant improvements showcased by OpenAl-ol (35), DeepSeek-RI [13), and Kimi-k1.5 43 Among these advancements, two key innovations have contributed significantly: First, the adoption of a rulc-based reward system significantly streamlines the training process by focusing exclusively rewarding correct final answers and proper output formats, thereby eliminating the complexity associated with process-basee brased reward models 30 47 Second, the introduction of a lightweight reinforcement learning (RL) algorithm [33
removes the need for separate critic model, substantially reducing computational overhead and accelerating the training process. The success of DeepSeek-R1 has attracted numerous follow-up studies [62], which broadly fall into two categories. The first category focuses on improving the quality of training data 33 [15 21,
43
2] emphasizing rigorous data set curation through filtering, deduplication, and verification. The second category refines RI. algorithms, including detailed optimizations for PPO-based methods such as VCPPO [56) and VAPO (55): enhancements GRPO for stability and speed, such as DAPO (54).
Dr.GRPO [31] and SRPO 63] : as well as alternative approaches such as REINFORCE+ (20].
While RL has been extensively applied to LLM reasoning, the role of diversity remains largely unexplored this context, even though it plays a crucial role in RL research 19 36
37
"Correspondence author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

12 58 66, 6 52 In traditional RL tasks, incorporating diversity is widely recognized to facilitate exploration by promoting the selection of more stochastic policies. which helps the policy escape local optima and accelerate the convergence of training. This hypothesis has been experimentally validated previous work [19] [1
[6]. Beyond empirical evidence, theoretical analyses suggest that policies with higher entropy (a measure of diversity) can smooth the optimization landscape []]
These findings naturally lead us to ask the following question: Is promoting diversity essential
during RL training for LLM reasoning?
Intuitively, an LLM capable of generating diverse responses could broaden the exploration of reasoning paths, enabling the model to avoid overfitting to narrow solution patterns in mathematical or logical tasks. To formally address this question, we conduct an evaluation of diversity in LLM reasoning, with specific focus on mathematical problem-solving. We introduce a novel metric, Potential @ k, to quantify an LLM'' reasoning potential (the possible performance gain after RI. training). We empirically analyze 12 representative LLMs,, examining both their solution diversity and Potential @ scores. Notably, our results reveal strong positive correlation between solution diversity and Potential @ scores among nigh-performing models, which suggests that diversity directly contributes to improved final performance after RL training.
The empirical findings motivate us to promote diversity during RL training for LLM reasoning. A commonly used approach for this goal is entropy regularization. However, directly increasing the average entropy of LLM outputs can introduce length bias, as longer responses inherently exhibit higher entropy. To address this, we introduce a token-level diversity metric and reformulate the diversity objective into a practical form. Moreover, promoting diversity often entails a quality- diversity trade-off. To mitigate this, we strategically apply diversity enhancement only to positive samples, thereby enriching solution diversity while preserving training stability. This design is akin to fostering diversity in high-quality policies in population-bas -based RL training, ensuring that exploration is guided by task-relevant performance criteria [48). Finally, we integrate
diversity objective into the RI RI-zero training method and evaluate the enhanced approach across 4 mathematical reasoning benchmarks. Experimental results demonstrate 3.5% average performance gain over standard RI-zero training, while our method can generate more diverse solutions.
To summarize, our key contributions are:
We present the first formal investigation into the role of diversity in LLM reasoning. Through experiments on mathematical benchmarks, identify positive correlation between solu- tion diversity and an LLM's reasoning potential, as measured by our proposed Potential@k metric. This finding provides empirical motivation for incorporating diversity into policy optimization.
We propose a novel token-level diversity objective, which is reformulated into practical metric and selectively applied to positive samples. This design is further supported through gradient behavior analysis, offering an insight for balancing quality and diversity during optimization.
We evaluate our method on four mathematical reasoning benchmarks, each comprising at least 500 problems with stable evaluation metrics. Our method achieves a 3.5% aver- age improvement over standard 1zeerro training and consistently produces more diverse solutions.
2 Preliminary
2.1 RL for LLMs
In the context of RL for LLMs, we frame the LLM generation process as an RL problem. Here, the LLM is modeled as policy that produces outputs (actions) conditioned input prompts (states) and receives evaluative feedback (rewards) for its generated responses. This formulation aligns the sequential decision- decision-making nature of language generation with RL's nnte-actiio-rewarr framework, enabling systematic optimization of the model's behavior through reward signals.
Formally, in the context of LLM generation for mathmatical problem-solving, where each prompt is a question, we define the prompt as g E Q, where Q represents the set of all possible questions. The set of all potential text outputs forms an action space O. Each output consists of tokens, denoted

---

as o ( To generate output, policy no parameterized by 0 is employed, which generates the output according the distribution:
To (olg) = II To(o"1q,05%)
(I)
where oc = (0"" o² ,...o'--1).
2.2
Reinforcement Learning algorithm
The RI-zero training method proposed by DeepSeek-RI [13 has attracted significant research attention due to its computational efficiency and effectiveness. In our work, we adopt this training method as backbone. RI RI-zero incorporates two key innovations: the GRPO algorithm [40] and rule-based reward function. In this section, we introduce both components.
Group Relative Policy Optimization (GRPO) GRPO streamlines the process by eliminating the need for separate critic model, which is usually as large as the policy model, and instead estimates baselines using group scores. Specifically, for each question GRPO samples a group of outputs {or, 02 .g....... from the old policy wold and optimizes the policy Te by maximizing the following objective:
JGRPO(TT) (TB) = (.g.
Ta(ai19) Ai, clip( To(o,,9) (ota(o:)g) 1- - e,,
min
Tota(oila)
(2)
where and B are hyperparameters, the KL term is defined as
Tre (oi1g) - log (o(9))
Tref (0ilg) -, (o(0))g)
D [.I.
(3)
and the advantage A, is computed using group of rewards fr,,72, ra):
A; r; mean( (r1,,2:---r
(4)
Reward functions In line with DeepSeek-RI accuracy rewards and format rewards. The accuracy reward model assesses whether the response is correct by comparing the predicted answer the golden reference answer, while the format reward model ensures that the final answer is presented in a \boxed{) format for reliable verification.
implement types of rule-based rewards:
3 Correlation between LLMs' reasoning potential and solution diversity
The role of diversity has long been established critical in traditional RL tasks. Numerous studies [19
36 g 37 have shown that promoting diversity can enhance the final quality of the policy. However, impact in the realm of RL for LLM reasoning still remains under-explored. In this section, we investigate the relationship between solution diversity and the reasoning abilities of LLMs on mathematical benchmarks. We adopt the equation diversity prior work (49) quantify the variety of solutions generated for mathematical problem-solving. For reasoning ability, we introduce a novel metric evaluate an LLM's training potential (related to the performance gain achieved after RL training).
For each question, calculate: (1) Pass@1 accuracy using greedy decoding, and (2) Diversity with (3) Potential@k k both evaluated from 16 sampled responses (1emperature=0.9).
Experimental setup
We evaluate 12 L.MMs the MATH benchmark [16]
For diversity, we adopt the metric (denoted as Div-Equ) from prior work 49 which measures the ratio of distinet equations among the responses:
N
U4
Div-Equ
(5)
A

---

Lrear grosics
Promane divenuty
Afte: optimizzion innomaliatiin
1/6
poodice sample
prouubiitt diccciona
(a)
Figure 1: (a) Evaluation of Pass@1 accuracy, Div-Equ diversity, and Potential @ 16 across 12 LLMs on the MATH benchmark. Model naming conventions: Prefixes denote base architectures (Q: Qwen2.5-Math, DS: DeepSeekMath, M: Mistral, L: Llama, DRQ: DeepSeck-RI-Distill- -wenn NM: NuninaMath); suffix "-I' indicates I-Intruuct'. (b) Illustration of probability movement during diversity optimization on positive samples.
where U4 and A; are the sets of unique equations and all equations extracted from the k sampled responses (with & = 16 our experiments) question i, respectively, And 500 is the amount of the data.
For Potential, we define metric termed Potential @k quantify the model's capability correct answers within l trials (with & = 16 in our experiments) on its Pass@ failure samples. Formally:
N Ei=1 Pass@k(()) (1 Pass@1(q.)
Potential@k
ΣN
Pass@ (q:))
where qi denotes the i-th question.
la The results show a bifurcated pattern: For LLMs with limited reasoning ability (Pass 1 0.4). observe significant relationship between solution diversity and model potential. For stronger performers (Pass@ 1 > 0.4), a clear positive correlation emerges between these metrics. Linear regression this high-performing subset yields R² = 0.81, confirming a strong predictive relationship where increased diversity corresponds to higher model potential.
Empirical findings The results are shown in Figure
Through an investigation of the Objective the GRPO algorithm, we observe that for each question in the training set, if all samples within group are either entirely positive or entirely negative, the advantage score becomes O, resulting in no gradient update. Crucially, the training signal originates from the reward discrepancy between positive and negative samples within the group, which is inherently linked to our definition of potential (to some extent, the algorithm's 's improvement can be characterized by the dynamics of this potential metric, as discussed in Appendix BB). This indicates that promoting diversity for LLM may result in higher performance after RI. training.
Takeaways A positive correlation between the LLM''s reasoning potential and solution diversity observed in our experiment. As illustrated in Section the optimization direction is guided by correct answers in multiple sampled responses. This directly links our Potential @k metric to RL training improvements. Hence, the observation strongly motivates us to enhance diversity during the RL training process.
4 Diversity-aware policy optimization
Building on the insights from Section 3 in this section, we introduce entropy-based diversity and propose its targeted application to positive samples during policy optimization for L.LM reasoning. We incorporate this diversity objective into the R1-zero training method [13], which employs the

---

GRPO algorithm with the reward function defined in Section 2.2 We refer to this enhanced approach as Rl-zero-Div.
4.1 Entropy-based diversity
A straightforward approach is to define diversity as the average entropy of the LLM's outputs per question i.e., (rrto@_(((=(-ll))) However, this formulation introduces length bias: longer responses inherently exhibit higher entropy (due to more token-level uncertainties), causing the metric to artificially favor longer outputs regardless of actual solution diversity. To address this issue, we introduce token-level entropy, which calculates the entropy for each token sampled from the old policy Vold- Formally, we define:
(io( = 2
H(
T
where T is the length of the output.
During training, the gradient of diversity with respect to the policy To in the M(TT(-1l. oft) is intractable. We therefore reformulate the diversity objective enable effective backpropagation:
1
(Do) Eg aow
Eøg ~ mo [log To(ō" 44, 19,0
T
1
To 19, x-1)
log To (0' (0'1
Tota q, o2e)
A proof for the last equation can be found in Appendix A. In practice, building on the R1-zero training method, we can use samples within the group calculate Objective g
4.2 Promoting diversity on positive samples
Empirical evidence indicates that the direct application of Objective innavverentlly increases diversity in incorrect solutions. Intuitively, negative samples offer more room for diversity enhancement, which can skew the model's optimization process. To address this issue, we concentrate on promoting diversity exclusively within positive samples:
T Tota(o"19, ost) log To (o lg, oz0)]
1 1o(0'1g,0<$1
T
(Te) = Som
- I(r = 1)
Σ
(9)
where II(-) denotes the indicator function and r is the accuracy reward for output
This is akin to fostering diversity in high-quality policies in population based RL training 488. while we focus on positive samples rather than policies here. Beyond intuitive justification, we further justify this design by analyzing the gradient each token.
According to Equation
g
we have:
Ve (TT((o' qq o log Ta(o'la lq, o<t]] Totalo' (o,ool,,, o=t)
mg Dio (TO) = Qowaaaaaaaal
(10)
Thus, the gradient be decomposed into per-token contributions (each term in the summation contributes a component). Up to a constant scaling factor, the gradient from each token is:
-ooTo q xt log mo(o ozt) -(1 log To o x=t)]= Vono (o lg, o²t).
(11)
Hence, for tokens with small probabilities (in that case (o lg, <t) e1, and this holds for most of tokens since the sum of probability is equal to 1), the gradient aligns with Vo g, o =² This suggests that the diversity component's gradient actively promotes increasing the probability of
5

---

low-proban wwwprobabilltty tokens, which inherently offer substantial growth potential. However, this tendency undesirable for negative samples. Thus, excluding diversity enhancement for negative samples mitigates conflicts between solution quality and diversity. A visual illustration is provided in Figure
1b Moreover, the experimental results in Section 5.3 and Appendix further support our design.
Finally, we incorporate the diversity optmization into the standard R RI-zero training, and use the samples in the group to calculate the diversity, yielding the final training objective:
J (Te) =JRRPO(TT) + J.i (Te)
=E 2(a) (Taall G
1
G
min To(O)lg)
A,, clip To(o.lg)
1011-,,10 Tola(a;l)) 1 +, (4)
Tola(oi
1=1
3 -l((r; 1)
1 T.
To To mo(oola,0 ,5 Totd 1,
log (o(ool, (12)
where is the diversity weight and i denotes the i-th sample in the group. In practice, choose 0.01. Other implementation details are provided in Section [.1 Appendix D
Experiments
In this experimental section, aim to address the following questions:
QI. Can our method effectively enhance reasoning abilities and provide diverse solutions?
Q2. Does the design of the diversity coefficient À influence the results?
Q3. Does our method demonstrate consistent performance across different model sizes?
5.1 Experimental setup
Base models We choose Qwen2.5-Math (Qwen BB) [5] our base model, which is commonly used for mathematical reasoning benchmarks [59 67 24). Additionally, we conduct an ablation study using --en2.5-MMth-1 (Qwen1.5B)B to assess the effectiveness of our approach in smaller LLMs.
Benchmarks We selected 4 mathematical benchmarks to evaluate the models' reasoning abilities: GSM8K (7], MATH500 [6], Olympiad Bench [4, and College Math 42 Each contains at least 500 data points for testing. We excluded some commonly used mathematical benchmarks that provide limited data, c.g, AIME24 with 30 items, as they can lead to unstable and biased evaluation outcomes. We train the base model on the GSM8K training set and then evaluate on the 4 benchmarks.
Baselines The most pertinent baselines for comparison are the base model itself and the base model trained via RI-zero. Additionally, we incorporate the latest prominent "RI "RI-zero-Like models with similar backhones for reference: SimpleRI-Zoo [59], PRIME-Zero-78 7B (9. It is important to note that these methods are trained with different computational resources and datasets, making direct comparisons challenging. Our approach is designed to enhance diversity rather than compete directly with these methods. In fact, our method is compatible with and can be integrated into these existing approaches.
Implementation details For Ri-zrr-DD-i,, we train the base model on the GSM8K training set using the loss function in Equation [2] with learning rate of 3 x 10-6 and the AdamW optimizer. During rollout, we sample responses with temperature of 0.9 and train for epochs. Our implementation is built TRL [46] and runs on 8xA6000 GPUs. For R1-zero,
maintain identical settings to RI-zero-De Div but exclude the diversity objective. For other baselines, we evaluate open-sourced models downloaded from Hugging Facd" following the settings recommended in their original papers.
Additional implementation details are provided in Appendix D
https://huggingface.co

---

Table 1: Pass@I accuracy on mathematical benchmarks.
Olympiad Bench
Method
GSM8K
MATH500
College Math
Avg
Qwen2.5-Math7B
57.6
51.8
16.1
21.4
36.7
RI-zero
88.7
74.6
37.3
43.3
61.0
R1-zero-Div (Ours)
91.7
78.2
40.1
47.6
64.4
SimpleRL-Zoo
90.2
80.0
39.0
47.2
64.1
Eurus-2-7B-PRIAE
88.0
74.4
39.4
46.6
62.1
Table 2: Avg@8 accuracy on mathematical benchmarks.
Olympiad
Method
GSM8K
MATH500
College Math
Avg
Bench
Qwen2.5-Math-7B
53.37 (0.56)
48.10 (0.82)
15.80 (0.22)
19.36 (0.14)
34. 16
R1-zero
87.77 (0.86)
72.97 (1.20)
37.26 (0.52)
42.22 (0.31)
60.06
Ri-zero-Div (Ours)
90.64 (0.89)
76.92 (1.24)
39.19 (0.55)
47.49 (0.32)
63.56
SimpleRL-Zoo
89.46 (0.87)
77.15 (1.23)
39.43 (0.57)
47.19 (0.34)
63.31
Gurus-2-7B-PRIA
88.31 (0.86)
73.92 (1.18)
36.56 (0.50)
45.27 (0.30)
61.02
5.2 Main results
R1-zero-Div enhances reasoning abilities We evaluate the reasoning performance using Pass@) accuracy, shown in Table ! In experiment, R R1-zero-Div demonstrates superior performance compared to RI-zero, achieving an average improvement of 3.5%. Despite being trained with limited computational resources (discussed in Appendx B)) RI --zero-Divv achieves comparable results to state- of-the-art methods (SimpleRL-Zoo and Eurus-2-713 2-10--2-7BPPRR- PRIME.. These results suggest that promoting diversity on positive samples in training can effectively enhance the model's reasoning capabilities. Also, following the recommendations in prior work [4] [7]. we evaluated 8 samples per question with temperature of 0.5. We report Avg @ and its standard error in the Table The conclusion regarding the effectiveness of our approach remains consistent with the pass @ metric results.
R1-zero-Div generates díverse solutions We empirically demonstrate that RI-zero-Div produces more diverse solutions than other RL-finetuning baselines. Our evaluation on GSM8K test set generates 5 responses for each of 1.319 questions, measuring diversity through three metrics: Div-Equ, and two additional metrics in prior work (66]: (1) N-gram diversity (proportion of distinct n-grams per response, capturing intra-diversity) and (2) Self-Bla BLEU diversity (100 minus Self BLEU score, capturing inter-diversity). All metrics range from 0 to 100, with higher values indicating greater diversity. As shown in Table while RL fine-tuning methods significantly reduce diversity (compared to the base model), R1-zero-DD Div effectively preserves diversity. We further provide concrete examples in Appendix E showing that RI -l-zero- generates distinct solutions for the same question.
5.3 Ablation study
We conduct an ablation study to analyze (1) the impact of different diversity weights and (2) our method's generalization capability smaller base models.
Analysis on the choice of diversity weights À 4 Pass@1 accuracy when applying
different À values to promote diversity on positive samples (denoted as "pos"). The results demonstrate that small values (A 0.02) effectively enhance reasoning performance, with À 0.01 emerging as the optimal choice in our experimental setup. We further compare diversity promotion strategies: positive samples only ("pos")) versus all samples ("pos+neg"). The marginal improvement observed when applying diversity to all samples supports methodological design choice in Section 4.2

---

Table Diversity of different methods on GSM8K test set.
Method
Div-Equ
Div-N-gram Div-Self-BLEU
Qwen2.5-Math-7B
92.26
29.29
85.98
Eurus-2-7B-PRIMEE
60.86
24.08
48.20
SimpleRL-Zo0
74.89
25.41
49.32
RI RI-zero
75.02
27.75
56.00
zero-Div (Ours)
79.29
29.60
58.89
Experiment on 1.5B base model We perform both Rl-zero-Div and Ri-zero on the Qwen2.5-Man 1.5B base model (51), with results shown in Table The experiments demonstrate that, compared to RI-zero, R R1-zero-Div Div enhances reasoning performance on out of benchmarks, achieving an average improvement of 2.3%, validating the scalability of our approach to smaller model.
Table Ablation Study on different diversity weights on mathematical benchmarks
Method
GSM8K
MATH500
Olympiad Bench
College Math
Avg
X = 0
88.7
74.6
37.3
43.3
61.0
A 0.05, pos
88.1
74.8
38.2
45.8
61.7
À = 0.02, pos
90.7
76.0
38.4
45.9
62.8
x= 0.01, pos
91.7
78.2
40.1
47.6
64.4
À 0.01, pos+neg
89.8
76.6
39.6
46.9
63.2
Table Ablation Study on Qwen2.5-Math-1.. base model
Olympiad Bench
Method
GSM8K
MATH500
College Math
Avg
Qwen2.5-Math-1.5.
39.4
36.4
23.0
6.6
26.3
RI-zero
82.9
66.4
32.1
43.1
56.1
RI-zero-Div (Ours)
83.2
70.4
32.0
43.9
57.4
Related work
RL for LLM reasoning The reasoning capabilities of LLMs have seen remarkable progress recently, with notable improvements demonstrated by OpenAl-ol [35. DeepSeek-Rl [13], and Kimi-k1.5 [... Our work builds upon the RI-zero training method proposed by DeepSeek-RI [3] which significantly improves LLM reasoning through two innovations that simplify the training pipeline and accelerate training: the GRPO algorithm 44), which replaces critic models with group score baselines, and rule-based reward system that focuses solely on final answer correctness and output format.
Subsequent research has advanced this approach in two directions: (1) improving training data quality 228
21 [8] and (2) refining RL algorithms. Regarding RL algorithm refinement, one category focuses on PPO-like methods. SimpleRL-zero (60) demonstrates that PPO with replacing the reward model by rule-based reward function can significantly improve the LLM's reasoning ability. VinePPO (25 leverages the flexibility of language environments to compute unbiased Monte Carlo-
33
15
54
based estimates, eliminating the need for large value networks. VCPPO (56) employs pretrained
value model to address value initialization bias and decouples Generalized Advantage Estimation
(GAE) computation between the actor and critic to mitigate reward signal decay. VAPO (55] further
shows that value-based RL frameworks outperform value-free methods in long Chain-of-Thoure

---

reasoning. The second category focuses on GRPO enhancements for stability and speed. DAPO [4 identifies the critical shortcomings (entropy collapse, training instability, and biased loss) in the original GRPO algorithm and addresses them via decoupled clipping and dynamic sampling.
Dr.GRPO reveals two biases GRPO: response-level length bias and question-level difficulty bias. SRPO [63] introduces a two-stage history- resamligg method to improve training efficiency.
Alternative approaches explore algorithms like REINFORCE: Kimi-k1.5 [43 demonstrates stable
training with REINFORCE-like policy gradients, while REINFORCE++ [0 and GPG [5] aim to
enhance REINFORCE's s stability and scalability, respectively.
Diversity in RL Research on policy diversity in deep reinforcement learning can be categorized into three groups based on how diversity is utilized 48] The first category uses diversity primarily to improve exploration efficiency, where diversity emerges as a byproduct of maximizing final task The second category treats diversity either as constraint (opti- mizing quality subject to diversity constraints) or as an objective (optimizing diversity under quality constraints) [32
performance [19 I1
36
8
37 [3).
66). The third category optimizes quality and diversity simultancously, known as Quality-Diversiiy
64 12 58
RL methods [6 38 45 3 Our work extends the first paradigm to
RL for LLM reasoning. While existing research in this category has proposed various diversity metrics, such as distance regularization between the current policy and a previous policy [19] reward randomization [4], we develop our approach based on a simple yet effective entropy-based diversity metric.
Diversity in LLMs Prior work has explored diversity in LLMs across several domains. GEM 29 proposes methods to preserve diversity during supervised fine-tuning, while Bstar [61] investigates the exploration-exploiaaee diversity in reinforcement learning from human feedback 34 [6] and LLM ensembles 44) However, diversity remains understudied in RL for LLM reasoning scenarios. To our knowledge, we are the first formally analyze diversity and propose principled diversity-awars training method for this setting.
tradeoff in self-improvement settings. Additional studies have examined
Conclusion, limitations, and discussion
In this research, we investigate the role of diversity in RL for LLM reasoning. Through comprehensive evaluations across 12 LLMs, we empirically establish strong positive correlation between a model's reasoning potential and the diversity of its generated solutions, underscoring the necessity of fostering diversity during RL training. To this end, we introduce a novel diversity-awar policy optimization method that optimizes the teken-level diversity in positive samples. Experimentally, we demonstrate that our method not only enhances LLMs reasoning ability but also generates more diverse solutions. By bridging the gap between diversity promotion and policy optimization, we aim to provide new insights for advancing the robustness and creativity of LLMs in complex reasoning scenarios.
Due to computational constraints, our experiments were conducted on 8XNVIDIA A6000 GPUs, which restricted our analysis to 1.5B and 7B parameter-scale models. This naturally introduces a policy optimization method to larger-scale LLMs remains to be explored. While our method demonstrates significant improvements on mid- sized models, extrapolating these findings to larger architectures may require adjustments to the entropy regularization scheme or training dynamics, given the known differences in optimization landscapes across model scales. We urge future research to investigate these scalability challenges and hope our work will inspire the community to explore diversity-enhancee RI. strategies for both small and large LLMs, fostering more robust reasoning capabilities across the spectrum of model architectures. Beyond scaling, the aiversity-aware optimization mechanism could also be extended to suggesting its broader applicability beyond reasoning tasks.
limitation: the generalizability of our diversity-
other LLM tasks [65]
Another promising future direction lies in the semantic definition of diversity. In this work, we employ entropy- based regularization to implicitly promote diverse behaviors during LLM generation, which captures statistical variance in output distributions. However, many real-world applications demand user-intended diversity (e.g., requiring both algebraic and arithmetic solutions to a math problem, or generating code with distinct algorithmic approaches). Such scenario-specific diversity requires explicit modeling of user-defined diversity, a challenge well-studied RI. [0 48). By bridging LLM reasoning with explicit diversity optimization from RL, future work could unlock

---

more controllable and context-aware generative capabilities, addressing the gap between statistical diversity and human-inte intaanintentional variety in complex tasks.
Acknowledgments and Disclosure of Funding
This work was supported in part by the National Natural Science Foundation of China under Grant U21A20512 and the Research Grants Council of the Hong Kong SAR under Grant No. C5052- 23G, Grant PolyU 15229824, Grant PolyU 15218622, and Grant PolyU 15215623. This work was also partially supported by Guangdong Basic and Applied Basic Research Foundation (No. 15441155000019) This work was also partially supported by the Research Grants Council of the Hong Kong SAR (Grant No. PolyU15217424 PolyU25216423) and the Hong Kong Polytechnic University (Project IDs: PO043563).
References
[1] Zafarali Ahmed, NicolasLe Roux, Mohammad Norouzi, and Dale Schuurmans. Understanding the impact of entropy on policy optimization. International Conference on Machine Learning, 2019.
[2] Alon Albalak, Duy Phung, Nathan Lile, Rafacl Rafailov, Kanishk Gandhi, Louis Castricato, Anikait Singh, Chase Blagden, Violet Xiang, Dakota Mahan, et al. Big-math: A large-scale, high-quality math dataset for reinforcement learning in language models. arXiv preprint arXiv:2502. 17387, 2025.
[3] Sumeet Batra, Bryon Tjanaka, Matthew C Fontaine, Aleksei Petrenko, Stefanos Nikolaidis, and Gaurav Sukhatme. Proximal policy gradient arborescence for quality diversity reinforcement learning. arXiv preprint arXiv:2305. 13795, 2023.
[4] Nikhil Chandak, Shashwat Goel, and Ameya Prabhu. Incorrect baseline evaluations call into question recent 11m-rl claims, 2025. Notion Blog, 2025.
[5] Xiangxiang Chu, Hailang Huang, Xiao Zhang, Fei Wei, and Yong Wang. Gpg: A simple and strong reinforcement learning baseline for model reasoning. arXiv preprint 25X4v:2504.02546 2025.
[6] Geoffrey Cideron, Thomas Pierrot, Nicolas Perrin, Karim Beguir, and Olivier Sigaud. Qd-rl: Efficient mixing of quality and diversity in reinforcement learning. corr abs/2006.08505 (2020). arXiv preprint arXiv:2006.08500 08505, 2020.
[7] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110. 10. 14168. 2021.
[8] Edoardo Conti, Vashisht Madhavan, Felipe Petroski Such, Joel Lehman, Kenneth Stanley, and Jeff Clune. Improving exploration in evolution strategies for deep reinforcement learning via population novelty-seeking agents. Advances in neural information processing systems, 31, 2018.
[9] Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, al. Process reinforcement through implicit rewards. arXiv preprint arXiv:2502.01455
2025.
[10] Li Ding. Jenny Zhang. Jeff Clune, Lee Spector, and Joel Lehman. Quality diversity through human feedback: Towards open-ended diversity-driven optimization. arXiv preprint 10. 1210 2023.
[11] Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity all you need: Learning skills without reward function. arXiv preprint arXiv:1802.06077 2018.
10

---

[12] Mahsa Ghasemi, Evan Scope Crafts, Bo Zhao, and Ufuk Topcu. Multiple plans are better than one: Diverse stochastic planning. In Proceedings of the International Conference on Automated Planning and Scheduling, volume 31, pages 140-148, 2021.
[13] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501.12445 2025.
[14] Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang. Yuxiang Zhang, et al. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. arXiv preprint arXiv:2402. 14008, 2024.
[15] Zhiwei He, Tian Liang, Jiahao Xu, Qiuzhi Liu, Xingyu Chen, Yue Wang, Linfeng Song, Dian Yu, Zhenwen Liang, Wenxuan Wang, et al. Deepmath- 103k: A large-scale, challenging. decontaminated, and verifiable mathematical dataset for advancing reasoning. arXiv preprini MMXiv:2504.11456 2025.
[16] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXhv:2103.03874, 2021.
[17] Andreas Hochlehnert, Hardik Bhatnagar, Vishaal Udandarao, Samuel Albanie, Ameya Prabhu, and Matthias Bethge. A sober look at progress in language model reasoning: Pitfalls and paths to reproducibility. arXiv preprint arXiv:2504.07055. 2025.
[18] Andreas Hochlehnert, Hardik Bhatnagar, Vishaal Udandarao, Ameya Prabhu, and Matthias Bethge. Curatedthoughts: Data curation for rl training datasets, 2025.
[19] Zhang-Wei Hong, Tzu- Yun Shann, Shih-Yang Su, Yi-Hsiang Chang, Tsu-Jui Fu, and Chun-Y Lee. iiverstt-drriven exploration strategy for deep reinforcement learning. Advances in neural information processing systems, 31, 2018.
[20] Jian Hu. Reinforce++: A simple and efficient approach for aligning large language models. arXiv preprint arXiv:2501.03262, 2025.
[21] Jingeheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung- Yeung Shum. rpen-rreasner--zero: An open source approach to scaling up reinforcement learning on the base model, 2025.
Heung-Yeung
[22] Beichen Huang, Xingyu Wu, Yu Zhou, Jibin Wu, Liang Feng, Ran Cheng, and Kay Chen Tan. Evaluation of large language models solution generators in complex optimization. IEEE Computational Intelligence Magazine, 20(4):56-70, 2025.
[23] Yuxiao Huang, Wenjie Zhang, Liang Feng, Xingyu Wu, and Kay Chen Tan. How multimodal integration boost the performance of Ilm for optimization: Case study on capacitated vehicle routing problems. In 2025 IEEE Symposium for Multidisciplinary Computational Intelligence Incubators (MCI), pages 1-7. IEEE, 2025.
[24] Hugging Face. Open rl: A fully open reproduction of deepseek-rl, Januurr 2025.
[25] Amirhossein Kazemnejad, Milad Aghajohari, Eva Portelance, Alessandro Sordoni, Siva Reddy, Aaron Courville, and Nicolas Le Roux. Vineppo: Unlocking rl potential for Il reasoning through refined credit assignment. arXiv preprint arXiv:2410.016: 2024.
[26] Robert Kirk, Ishita Mediratta, Christoforos Nalmpantis, Jelena Luketina, Eric Hambro, Edward Grefenstette, and Roberta Raileanu. Understanding the effects of rlhf on Ilm generalisation and diversity. arXiv preprint arXiv:2310.06452 2023.
[27] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph F. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large lan- guage model serving with pagedattention. Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.

---

[28] Xuefeng Li, Haoyang Zou, and Pengfei Liu. Limr: Less is more for rl scaling, arXiv preprint arXiv:2502.11886. 2025.
[29] Ziniu Li, Congliang Chen, Tian Xu, Zeyu Qin, Jiancong Xiao, Zhi-Qui thi-Quan Luo, and Ruoyu Sun. Preserving diversity in supervised finettuning of large language models. In The Thirteenth International Conference on Learning Representations, 2025.
[30] Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfih International Conference on Learning Representations, 2023.
[31] Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding rl -l-zero-like training: A critical perspective. arXiv preprint arXiv:2503. 20783, 2025.
[32] Muhammad Masood and Finale Doshi- Velez. Diversity-inducing policy gradient: Using maximum mean discrepancy to find set of diverse policies. arXiv preprint MXXiv:996.00000 2019.
[33] Chunyang Meng, Shijie Song, Haogang Tong, Maolin Pan, and Yang Yu. Deepscaler: Holistic autoscaling for microservices based on spatiotemporal gnn with adaptive graph learning. In 2023 38th IEEF/ACM International Conference on Automated Software Engineering (ASE), pages 65. IEEE, 2023.
[34] Sonia K Murthy, Tomer Ullman, and Jennifer Hu. One fish, two fish but not the whole sea: Alignment reduces language models' conceptual diversity. arXiv preprint 1.X44:2411.04427, 2024.
[35] OpenAL. Learning to reason with 1lms.
httpss//openai. catpaaai..omiiiiieeeee
reeason----------------- with 11ms, 2024.
[36] Jack Parker-Holder, Aldo Pacchiano, Krzysztof M Choromanski, and Stephen J Roberts. Ef- fective diversity in population based reinforcement learning. Advances in Neural Information Processing Systems, 33:18050- 10500-8808, 2020.
[37] Zhenghao Peng, Hao Sun, and Bolei Zhou. Non-local policy optimization via diversity- regularized collaborative exploration. arXiv preprint (XXi:200660078/ 2020.
[38] Thomas Pierrot, Valentin Macé, Felix Chalumeau, Arthur Flajolet, Geoffrey Cideron, Karim Beguir, Antoine Cully, Olivier Sigaud, and Nicolas Gelber. Diversity policy gradient for sample efficient quality-diversity optimization. In Proceedings of the Genetic and Evolutionary Computation Conference, pages 1075-1083, 2022.
[39] Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pages 3505-3506, 2020.
[40] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300 2024.
[41] Zhenggang Tang, Chao Yu, Boyuan Chen, Huazhe Xu, Xiaolong Wang, Fei Fang. Simon Du, Yu Wang, and Yi Wu. Discovering diverse multi-agent strategic behavior via reward randomization. arXiv preprint arXiv:2103.0456 2021.
[42] Zhengyang Tang, Xingxing Zhang, Benyou Wang, and Furu Wei. Mathscale: Scaling instruction tuning for mathematical reasoning. arXiv preprint urXiv:2403.02888. 2024.
[43] Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1.5: Scaling reinforcement learning with llms. arXiv preprint arXiv:2501.12599 2025.
12

---

[44] Selim Furkan Tekin, Fatih Ilhan, Tiansheng Huang. Sihao Hu, and Ling Liu. LIm-topla: Efficient Ilm ensemble by maximising diversity. arXiv preprint 1XXx:22100033533, 2024.
[45] Bryon Tjanaka, Matthew C Fontaine, Julian Togelius, and Stefanos Nikolaidis. Approximating gradients for differentiable quality diversity in reinforcement learning. In Proceedings of the Genetic and Evolutionary Computation Conference, pages 102-1111, 2022.
[46] Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, Shengyi Huang, Kashif Rasul, and Quentin Gallouédec. Trl: Transformer reinforce- ment learning. https:: thub........ com//huggingface/tt) 2020.
[47] Peiyi Wang, Lei Li, Zhihong Shao, RX Xu, Damai Dai, Yifei Li. Deli Chen, Yu Wu, and Zhifang Sui. :htherhepherd: Verify and reinforce llms step-by-step without human annotations. arXiv preprint arXiv:23 www.:33.2.00000 08935, 2023.
[48] Shuang Wu, Jian Yao, Haobo Fu, Ye Tian, Chao Qian, Yaodong Yang. Qiang Fu, and Yang Wei. Quality-similar diversity via population based reinforcement learning. In The eleventh international conference on learning representations, 2023.
[49] Ting Wu, Xuefeng Li, and Pengfei Liu. Progress or regress? self-improvement reversal in post-training. arXiv preprint urXiv:2407.05013. 2024.
[50] Xingyu Wu, Sheng-hao Wu, Jibin Wu, Liang Feng, and Kay Chen Tan. Evolutionary com- putation the era of large language model: Survey and roadmap. IEEE Transactions on Evolutionary Computation, 2024.
[51] Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xue, Runji Lin, Tianyu Liu, Xingzhang Ren, and Zhenru Zhang. Qwen2.5- technical report: Toward mathematical expert model via self- aaffimmprovemet. arXiv preprint arXiv:2409. 120.2440.11122, 12/22,
2024.
[52] Hanlin Yang, Jian Yao, Weiming Liu, Qing Wang, Hanmin Qin, Kirk Tang, Jiechao Xiong, Chao Yu, Kai Li, Junliang Xing. et al. Diverse policies recovering via pointwise mutual infor- mation weighted imitation learning. The Thirteenth International Conference on Learning Representations, 2025.
[53] Jian Yao, Weiming Liu, Haobo Fu, Yaodong Yang, Stephen McAleer, Qiang Fu, and Wei Yang. Policy space diversity for non-transitive games, Advances in Neural Information Processing Systems, 36:67771 2023.
[54] Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. Dapo: An open-source 11m reinforcement learning system at scale, 2025. URL https://arxiv. org/abs/25033 14476, 2025.
[55] Yufeng Yuan, Qiying Yu, Xiaochen Zuo, Ruofei Zhu, Wenyuan Xu, Jiaze Chen, Chengyi Wang, TianTian Fan, Zhengyin Du, Xiangpeng Wei, et al. Vapo: Efficient and reliable reinforcement learning for advanced reasoning tasks. arXiv preprint TrXiv:2504.05//1 2025.
(56) Yufeng Yuan, Yu Yue, Ruofei Zhu, Tiantian Fan, and Lin Yan. What's behind ppo's collapse in long-cot? value optimization holds the secret. arXiv preprint urXiv:2503.0149) 2025.
[57] Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model? arXiv preprin! arXiv:2504. 13837, 2025.
[58] Tom Zahavy, Brendan O'Dongguee, Andre Barreto, Volodymyr Mnih, Sebastian Flennerhag, and Satinder Singh. Discovering diverse nearly optimal policies with successor features. arXiv preprint arXiv:2 2ex:::2206600066 106.00669, 2021.
[59] Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models the wild, 2025.
13

---

[60] Weihao Zeng, Yuzhen Huang, Wei Liu, Keqing He, Qian Liu, Zejun Ma, and Junxian He. 7b model and 8k examples: Emerging reasoning with reinforcement learning both effective and efficient. https: hatp::///uuttttlnn notion site/ site/simplerl reason, 2025. Notion Blog.
[61] Weihao Zeng, Yuzhen Huang, Lulu Zhao, Yijun Wang, Zifei Shan, and Junxian He. B-star: Monitoring and balancing exploration and exploitation self-taught reasoners. arXiv preprint arXiv:2412.17256. 2024.
[62] Chong Zhang, Yue Deng, Xiang Lin, Bin Wang. Dianwen Ng, Hai Ye, Xingxuan Li, Yao Xiao, Zhanfeng Mo, Qi Zhang, et al. 100 days after deepseek-rl: A survey on replication studies and more directions for reasoning language models. arXiv preprini arXiv:2505.0055: 00551, 2025.
[63] Xiaojiang Zhang, Jinghui Wang, Zifei Cheng, Wenhao Zhuang, Zheng Lin, Minglei Zhang, Shaojie Wang, Yinghan Cui, Chao Wang, Junyi Peng, et al. Srpo: A cross-domain imple- mentation of large-scale scale reinforcement learning on Ilm. arXiv preprint arXiv:2504.14286. 2025.
[64] Yunbo Zhang, Wenhao Yu, and Greg Turk. Learning novel policies for tasks. In International Conference on Machine Learning, pages 7483-7492. PMLR, 2019.
[65] Yu Zhou, Xingyu Wu, Jibin Wu, Liang Feng, and Kay Chen Tan. HM3: Hierarchical multi- objective model merging for pretrained models. arXiv preprint arXiv:2409 2024.
[66] Zihan Zhou, Wei Fu, Bingliang Zhang, and Yi Wu. Continuously discovering novel strategies via reward-switching policy optimization. arXiv preprint arXiv:2204.02246, 2022.
[67] Yuxin Zuo, Kaiyan Zhang, Shang Qu, Li Sheng, Xuekai Zhu, Biqing Qi. Youbang Sun, Ganqu Cui, Ning Ding. and Bowen Zhou. Ttrl: Test-time reinforcement learning. arXiiv preprint 2504 /6084, 2025.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes, the main claims made in the abstract and introduction accurately reflect the paper's 's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Please refer the Limitation Section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
15

---

Justification:
The paper has some theoretical analysis. We provide complete and correct analysis.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Please refer Experiment Section and Appendix.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
16

---

Answer: [Yes]
Justification: Our code and instructions are included in the supplementary material. The data we use for the experiments are all from open-access datasets.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm ec/
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips. //nips.cc/pil wwwppbblccgguiicuceccccceoeooopp Detggudddse//eSubbiiiinn htt.SSbbbiiiiooooiiiii.oooi for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Please refer Experiment Section and Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Please refer to Experiment Section.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
17

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please refer to the implementation details in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Question:
Answer:
[Yes]
Justification:
The research conducted in the paper conform with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We list potential positive societal impacts in the Appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
18

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We properly credit data, paper, and ideas that used in this paper.
Guidelines:
existing assets. The authors should cite the original paper that produced the code package or dataset.
The answer NA means that the paper does
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
19

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We document well about the asset.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
20

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification:
We decrible the pipeline to fine-tune the LLM in the Experiment Section.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy for what should or should not be described.
21

---

A Theoretical analysis
A.1 Proof for Equation 8
The equation we want to prove is:
By Coomatatl))))
BB (1 (log Tolo" 0,00111
T
1
To qg o ( q, oze) log to 19, (0
(13)
T
Since T' is random variable that depends on Vold. the proof is not straightforward. We prove it in two stages.
(1). When is fixed, the proof proceeds straightforwardly by examining each term in the summation.
Note that o is sampled from Vold while o' is sampled from na, hence:
T
E rog
T
t=1
E PynQ,orsatalll
(Bow Dog To'' 4 []]]]]]]
T
Σ
By 2,0 (
[Bo
[']]]]
T
To(o 10(0"19,05t ost
B o
E Eornoue((a
log To(oo1
T
Tald (0" (1, o51)
1
To(0'lg,0<t)
log T0(0"10,000
2 os (
T
Toialo" 14, 05r)
(=1
To(o'lg, oct) Bald(o'lg,00t
Byg (oomaaaall))
log T000'10.00
T
To(o'1l, lg, oct) Tata(o"lg o<T) log To (o
(14)
The second and fifth equations hold because we may add or remove any random variables that are not in the target expectation. The third equation results from applying importance sampling to reweight probabilities.
(2). For the case that is a random variable, roughly, the idea is to apply the law total probability:
T
E
Ex - /0 /logo
Te To 0
1
-Egre
P(T Bo To)
Eg m log mologo St]]]
(15)
To
And apply case (1) to finish the proof.
B More discussion
More discussion about Potential@ @k
22

---

The definition of Potential @k aims to quantify the performance improvement achievable through RL training for LLMs. By examining its formulation, we derive:
Potential@k Ei Pass@k(()) . -1-11 10-ssa0 (4))
Pass@k((:) Pass@1(q:)) (16)
En
Pass@1(a:)
This metric essentially captures the discrepancy between Pass@k k and Pass While Pass is often treated the performance boundary for RL training on LLM (57), our Potential @k specifically measures the performance gain from RL training, approximated by subtracting Pass @ (a measure for initial performance) from @ass@k.
For each question 4 before training begins, if Pass@1(g.) = 1. the question is already mastered with no improvement potential. When Pass@1(q:) 0 but Pass@k(g) 1, GRPO training uses positive samples from & trials to teach the correct response. If both P = and Pass@k(q;) 0, the question provides no training signal it remains unsolved. Hence, definition of Potential @k k focuses training on questions with partial capability, excluding both mastered and unsolvable questions, thereby capturing the true learning potential through the Pass@k to Pass performance gap.
Why the performance of our reproduction of R1-zero is worse than the state-of-the-an methods reproductions (e.g. SimpleRI-Zoo)?
We believe the performance gap between our Rl-zer reproduction and SimpleRL-Zoo's implemen- tation stems primarily from resource constraints. Our experiments were conducted on modest AxA6000 GPUs setup, necessitating several efficiency optimizations: we employed the simpler GSM8K dataset, constrained generation lengths (appropriate for GSM8K's s short responses). In contrast, SimpleRL-Zoo utilized significantly more powerful 2x8 H100-80G GPUs, trained more complex datasets with longer response lengths. Importantly, our study's primary objective was not to surpass SimpleRL-Zoo's results, but rather to demonstrate that our diversity diversity-enhanced method outper- forms standard Ri-zer. Our method of independence can be applied to enhance the SimpleRL-Zoo and other state-of-the-arr methods.
Why are the 4 mathematical datasets chosen?
We require the number of data points in the test dataset to be at least 500. Since we find that the results are unstable when we test small dataset. In some recent work, they report by sampling many times and calculate the average. However, we think the distribution shift issue still exists (i.e., the small test data may biasly represent the hard/mediummeess- benchmark)
C Broader impacts
Our diversity-aware RL approach for LLM reasoning offers valuable benefits for Al applications. First, in education, generating multiple valid reasoning paths could enhance Al tutoring systems by providing alternative solution strategies to students. Second, for scientific research, the improved ability to explore diverse reasoning approaches may aid in hypothesis generation and problem-solving where multiple perspectives are valuable.
D Implementation details
We provide more details for experiments in Section 5
D.1 Experiment environment
For training R1-zero and RI RI-zero-Div, the codebase runs on Python 3.11, utilizing TRL 0.16.0 46 with Py PyTorch 2.5.1. We employ DeepSpeed (39] for distributed training and incorporate vLLM 0.7.2 277 for efficient rollout, all deployed on 8x NVIDIA A6000 GPUs. Each experiment runs for 3 days. For other baselines, we evaluate open-sourced models downloaded from Hugging Face")
https://huggiingaace.co
23

---

A conversation between User and Assistant. The user asks question, and the Assistant solves it. The assistant first thinks about the reasoning process in the mind and then provides the user with the answer. The assistant should 1) Identify solutions into logical, numbered steps. 3) Verify results using alternative methods or substitutions. Put your final answer within Wboxed
concepts and required formulas. Break down
Figure 2: System prompt
For evaluation, we utilize the code from Qwen2.5 2.5-Math. To caleulate Pass@ I, we use greedy decoding for our models and baselines, except for SimpleRI-Zoo [59] which we evaluate using temperature=1 and top-p=0.95 as suggested in their paper.
D.2 Hyperparameter settings
We provide the system prompt in Figure 2 and other detailed hyperparameter settings in Table 6 Due to computational resource constraints, train the simpler dataset (GSM8K), which allows for a shorter maximum response length, and use a well-designed prompt to obtain a stronger initial checkpoint. The experiment settings for R R1-zero and R1-zero-DD Div the same except for À 0 in R1-zero and À = 0.01 in R1-zero-Div.
Table 6: Hyperparameter settings
Hy Hyperparameter
Value
General settings
dataset
GSM8K
max prompt length
256
max completion length
756
num generations
6
use vllm
true
vllm gpu memory utilization
0.5
torch dtype
6
learning rate
3.0e-06
Ir scheduler type
cosine
beta
0,0001
zero stage
2
offload optimizer device
CPU
offload param device
distributed type
DEEPSPEED
Base model: Qwven/Qven2.5-Matth2 num train epochs
2
device train batch size
1
gradient accumulation steps
64
Base model: Qwenn(Qwnn.5-MMalhee train epochs
3
per device train batch size
6
gradient accumulation steps
16
httt::///ggthbbbb.c////ee/////// Katth

---

Entropy over training steps
0.19
0.18
0.17
Entropy
0.16
0.14
0.13
0.001 pos.neg
0.01 (R1-2ero-Div)
A 0 (R1-zero)
0.12
50
100
150
200
250
Training step
Figure Entropy during the RL training
Table 7: Avg @8 GPQA Diamond dataset
Method
GPQA Diamond
Qwen2.5-Math-7B
28.5 (0.64)
RI-zero
32.7 (0.70)
R1-zero-Div (Ours)
34.3 (0.79)
SimpleRL-Zoo
33.9 (0.56)
Funus-2-7B-PRIME
34.0 (0.74)
E More experiment results
E.1
Entropy during the RL training
We analyze the entropy dynamics during training. As shown the Figure when À = 0 (i.e., the baseline R1-zero method), entropy collapses rapidly, indicating loss of exploration. When applying the diversity objective with À = 0.01 to all samples (both positive and negative). entropy exhibits late-stage exploration during training; however, this tends degrade model quality, as reflected by the final performance in Table In contrast, applying the diversity objective with = 0.01 exclusively to positive samples strikes better balance between quality and diversity, yielding the optimal final performance.
E.2 Evaluation on GPQA
We present the avg@8 results on the GPQA Diamond dataset in Table where we directly evaluated the models listed in Table [
These results demonstrate the generalizability of our method.
E.3 Solution diversity in R1-zero-Div
We show the case where our method generates diverse solutions in Figure The text outputs generated by our model (R1 (R1-zero-Div training on Qwen2.5-22 (wwen2.5-Mah-77B) and we render LaTeX expressions for better visualization. illustrated in Figure 4 the model generates two distinet yet mathematically valid solutions to the problem. The first solution demonstrates elegant mathematical
25

---

Question: There exist real numbers and y, both greater than 1, such that log, (v/*)
Jog, (a**)
10. Find
Solution
Given the equations:
log, (yr)
10 and log, (I²²) = 10
First, simplify the equations using logarithmic properties:
log,(y") = 10 = x log, (1) = 10 = log,(v) 2
10
log, (a,,(zz)) = = 4y log, (z) 10 log, (a)
By
2. Using the change base formula, we know that log,, (y)
Therefore:
los, (x]
log, (1) and
logy (z) by
10 50 2y
50
2zy
my
25
Thus, final answer is: 25
Solution 2:
Given the equations:
log, (v/) 10 and log, (xr")
we the properties logarithms to simplify these equations.
Simplifying equation:
log, (vr') 10
logg (v) 10
log, (3)
Using the change base formula log (v)
loctal losll)
get:
Jos(y)
kug(F)
This implies: (xg
10logie]
Simplifying the second equation:
(og,((z"))
4y log, (z)
log y
3y
Using the change base formula log,(2)
los(
we get: los(s)
logt
Togta)
This implies: log(2) (z)
kog(s)
Zy
Now we have two equations:
log(y) lolgef))
log(æ)) = 5 singfu)
2y
Substituting the second equation into the first:
log(y)
10 Shai2)
25 logfe)
zy
Dividing both sides log(y) (assuming log(y) o 0):
1
25
zy
25
Thus, value my is:
Figure 4: Solution generated by R1-zero-Div
insight by utilizing the reciprocal property of logarithmic functions. The second solution adopts more systematic approach, first establishing two equations for log(y) and log(z) (more precisely. Ig y and z). And then solve these two equations. Both methods ultimately arrive at the correct solution, showcasing the model's ability to discover multiple valid reasoning paths.
26


Reviews:
[{"review_id": "wqukEcirIA", "reviewer": "Reviewer_zoHB", "summary": "The paper begins by presenting experiments that demonstrate a strong positive correlation between solution diversity and the potential performance improvements of LLMs trained using RL. Building on these findings, the authors propose a diversity-aware policy optimization approach, which augments the original RL training objective for correct responses with an entropy maximization term to encourage solution diversity. Experiments on Qwen2.5-Math-7B and Qwen2.5-Math-1.5B are conducted to validate the effectiveness of the proposed method.", "questions": "Please see the Weaknesses within Strengths and Weaknesses.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "> **Strengths**\n\n1. The paper grounds its proposed diversity-aware policy optimization approach in preliminary experimental results and mathematical analysis, supporting the idea that enhancing solution diversity can improve the performance of LLMs trained with RL. This makes the approach both promising and intuitively reasonable.\n\n2. The paper evaluates the trained LLMs on four benchmarks and conducts ablation studies to validate the effectiveness of the proposed diversity-aware policy optimization approach.\n\n> **Weaknesses**\n\n1. Entropy control is a widely used technique in reinforcement learning, and the proposed diversity-aware policy optimization method introduces only minor modifications to the standard entropy maximization objective. However, the paper does not include a comparison with the baseline approach that uses vanilla entropy maximization.\n\n2. The performance of the trained LLMs appears to be evaluated based on a single run, which may undermine the reliability of the experimental results [1]. To provide a more robust evaluation, especially when greedy decoding is not used, we recommend reporting the Avg@K metric, which averages performance scores over K independent runs.\n\n**References**\n\n[1] Incorrect Baseline Evaluations Call into Question Recent LLM-RL Claims", "was_revised": true, "final_justification": "4: Borderline accept", "rebuttal": "We thank the reviewer for their insightful comments and valuable time spent on evaluating our work. We are encouraged by the reviewer’s recognition that our approach, grounded in preliminary experiments and mathematical analysis, is both promising and intuitively reasonable. To address the raised concerns, we have added additional experiments to further validate the robustness and effectiveness of our approach, and we provide detailed explanations point by point below.​\n\n**W1: Entropy control is a widely used technique in reinforcement learning, and the proposed diversity-aware policy optimization method introduces only minor modifications to the standard entropy maximization objective. However, the paper does not include a comparison with the baseline approach that uses vanilla entropy maximization.**\n\n**A1**:\nThanks for your feedback. We would like to kindly point out that:\n- While entropy control is widely used in RL, its adaptation to RL for LLMs is non-trivial. Our key contributions to the algorithm are: We identify that vanilla entropy control exhibits a length bias. We address this by introducing token-level entropy, which we reformulate into a tractable form. This modified entropy term is then strategically applied to positive samples to further mitigate the conflict between quality and diversity and enable more stable and effective training.\n- We have conducted experiments using direct entropy regularization (i.e., encouraging $E_{q\\sim Q} [H(\\pi_{\\theta}(·|q))]$) and report the results in the following table (labeled Qwen2.5-Math-7B-R1-zero-Div-vanilla). The result shows that vanilla entropy regularization yields only marginal improvements over Qwen2.5-Math-7B-R1-zero. Analysis of failure cases reveals that the model tends to generate longer, less meaningful responses, with higher output token entropy than other methods, which indicates its inability to balance entropy and response quality under vanilla entropy regularization.\n\n**W2: The performance of the trained LLMs appears to be evaluated based on a single run, which may undermine the reliability of the experimental results. To provide a more robust evaluation, especially when greedy decoding is not used, we recommend reporting the Avg@K metric, which averages performance scores over K independent runs.**\n\n**A2**: Thanks for the insightful suggestion to enhance evaluation stability. Following the recommendations in [1,2], we evaluated 8 samples per question with a temperature of 0.5. We report Avg@8 and its standard error in the table below. The conclusion regarding our approach’s effectiveness remains consistent with the pass@1 metric results. We acknowledge the potential for evaluation variance and note that our chosen benchmark includes over 500 test samples to further mitigate such variance.​ We will report both Avg@K and Pass@1 in the revision version.\n\n**Table: Avg@8 on mathematical Benchmarks.**\n\n|                Method               |      GSM8K      |     MATH500    |  Olympiad Bench | College Math |  Avg  |\n|:-----------------------------------:|:---------------:|:--------------:|:---------------:|:------------:|:-----:|\n|           Qwen2.5-Math-7B           |   53.37 (0.56)  |  48.10 (0.82)  |   15.80 (0.22)  | 19.36 (0.14) | 34.16 |\n|       Qwen2.5-Math-7B-R1-zero       |  87.77  (0.86)  | 72.97 (1.20)   |   37.26 (0.52)  | 42.22 (0.31) | 60.06 |\n| Qwen2.5-Math-7B-R1-zero-Div-vanilla |   88.58 (0.86)  |  73.32 (1.17)  |   35.91 (0.50)  | 44.79 (0.29) | 60.65 |\n|  Qwen2.5-Math-7B-R1-zero-Div (Ours) |   **90.64 (0.89)**  |  **76.92 (1.24)**  |   **39.19 (0.55)**  | **47.49 (0.32)** | **63.56** |\n| |\n|     Qwen2.5-Math-7B-SimpleRL-Zoo    |   89.46 (0.87)  |  77.15 (1.23)  |   39.43 (0.57)  | 47.19 (0.34) | 63.31 |\n|  Qwen2.5-Math-7B--Eurus-2-7B-PRIME  | 88.31 (0.86)    |  73.92 (1.18)  |   36.56 (0.50)  | 45.25 (0.30) | 61.01 |\n\n[1] Incorrect Baseline Evaluations Call into Question Recent LLM-RL Claims\n\n[2] A Sober Look at Progress in Language Model Reasoning:\nPitfalls and Paths to Reproducibility\n\n\n\nWe hope these revisions adequately address your concerns, and we are happy to provide any further clarifications to facilitate the review process!"}, {"review_id": "jNcXT95zGC", "reviewer": "Reviewer_FPC2", "summary": "This paper investigates the role of diversity in RL-based training for LLM reasoning and proposes a diversity-aware policy optimization method. The authors introduce Potential@k, a metric quantifying reasoning potential, and demonstrate a positive correlation between solution diversity and this metric across 12 LLMs. They propose a token-level diversity objective applied selectively to positive samples, integrated into R1-zero training. Experiments show 3.5% average improvement on mathematical reasoning benchmarks while generating more diverse solutions.", "questions": "- How does the method scale to larger models (>7B parameters) and longer reasoning sequences?\n- Can the approach be extended to other reasoning domains beyond mathematics?\n- What is the computational overhead of the diversity objective during training?", "limitations": "The authors acknowledge key limitations: computational constraints limiting model scale, focus on mathematical reasoning only, and unknown generalizability to larger architectures. The semantic definition of diversity (statistical vs. user-intended) is identified as future work.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "* Strengths *\n- Novel empirical insight: First systematic investigation of diversity's role in LLM reasoning with strong correlation evidence (R²=0.81)\n- Well-motivated approach: Potential@k metric provides clear theoretical justification for diversity promotion\n- Practical method: Token-level diversity objective avoids length bias and selective application to positive samples balances quality-diversity trade-off\n\n* Weaknesses * \n- Limited scale and scope: Only 1.5B-7B models tested; mathematical reasoning exclusively; computational constraints acknowledged\n- Modest improvements: 3.5% average gain is meaningful but not dramatic; some individual benchmark improvements are marginal\n- Theoretical gaps: Gradient analysis provides intuition but lacks formal guarantees; entropy regularization is standard approach", "was_revised": true, "final_justification": "Rating 4.\n\nthe absolute gains appear modest;\nthe experiments are not verified on larger scale models", "rebuttal": "We thank the reviewer for the perceptive comments and the time dedicated to evaluating our work. We are greatly encouraged by the positive assessment, particularly regarding the novel empirical insights, well-motivated approach, and practical utility of our work. To address the raised concerns, we have supplemented additional experiments and provide detailed responses point by point below.\n\n**W1: Limited scale and scope: Only 1.5B-7B models tested; mathematical reasoning exclusively; computational constraints acknowledged**\n\n**A1**: Thanks for the feedback. Regarding the model scale, while our current experiments focus on 1.5B-7B models due to computational constraints, we have extended our evaluations to include the Llama-3.2-3B-Instruct model (please refer to **Q2**), which further validates the approach's generalizability across model architectures. Scaling to larger models (>7B parameters) remains an important direction for future work, pending increased computational resources. As for the scope beyond mathematical reasoning, we address this in detail in our response to **Q2** below.\n\n**W2: Modest improvements: 3.5% average gain is meaningful but not dramatic; some individual benchmark improvements are marginal**\n\n**A2**: Following the Reviewer zoHB's suggestion, we have included the Avg@8 metric along with standard deviations in our results (Please refer to A2 in rebuttal to Reviewer zoHB). While the absolute gains appear modest, the improvements are statistically significant across key benchmarks. Notably, our method offers strong practical value: it requires few lines of code modifications to implement, making it highly efficient to integrate into existing RL pipelines.\n\n**W3: Theoretical gaps: Gradient analysis provides intuition but lacks formal guarantees; entropy regularization is standard approach**\n\n**A3**: \n- We sincerely appreciate the reviewer’s insightful comments on the theoretical aspects of our work. We acknowledge that formal theoretical guarantees are currently absent, particularly given the limited existing theoretical frameworks addressing the trade-off between diversity and quality optimization  (especially in the context of LLM). E.g.  \"these models lack a natural means to control the inherent trade-off between quality and diversity\" [1], \"As such, the reward-diversity trade off typically relies on heuristics.\" [2], \"without giving a mathematical theoretical analysis\" [3]. \nWhile such theoretical grounding remains sparse in the literature, we have endeavored to provide empirical rigor through comprehensive gradient sign analysis. Specifically, our analysis reveals that for negative samples, optimizing diversity gradients conflicts with quality objectives for the majority of tokens (at most 2 tokens are exceptions due to the summation of probability being 1).\n- While entropy regularization is a standard approach, our key contribution lies in identifying that vanilla entropy control exhibits a length bias (We have added an experiment in **A1** in the rebuttal to Reviewer zoHB). To address this, we introduce token-level entropy, reformulated into a tractable form, and strategically apply this modified entropy term exclusively to positive samples to enable more stable and effective training.\n\n**Q1: How does the method scale to larger models (>7B parameters) and longer reasoning sequences?**\n\n**A4**: Due to current computational constraints, our experiments have focused on models up to 7B parameters, as training and evaluating larger models (e.g., 13B or 70B) would require substantially more computational resources beyond our current capacity.​\nTo validate the method’s generalizability across architectures, we extended our evaluations to include the Llama-3.2-3B-Instruct model (Please refer to Q2). Results from this additional experiment demonstrate consistent performance gains, confirming that our approach can effectively scale to alternative model architectures.​\nWe recognize the value of testing larger models and longer sequences, which we plan to explore as future work once additional computational resources become available.​\n\n**Q2: Can the approach be extended to other reasoning domains beyond mathematics?**\n\n**A5**: Thanks for this important question. We have conducted additional experiments across other reasoning domains:\n- (1) We evaluated our trained (on math dataset) models on the GPQA Diamond benchmark [4], with results shown in Table A. \n- (2) We applied our approach to the Llama-3.2-3B-Instruct model on the K&K dataset [5], a logic reasoning task previously studied in LLM-RL research [6]. We trained on 3-person scenarios and evaluated on both 3-person (-3ppl) and 4-person (-4ppl) tasks (Table B).\n\nThe results demonstrate that: (1) Improvements from enhanced diversity in mathematical domains generalize to graduate-level Q&A tasks (GPQA Diamond); (2) The approach is validated across both model architectures and reasoning domains.\n\n**Table A:  Evaluation on GPQA Diamond benchmark​ (Avg@8)**\n\n|               Method               | GPQA Diamond |\n|:----------------------------------:|:------------:|\n|           Qwen2.5-MATH-7B          |  28.5 (0.64) |\n|       Qwen2.5-Math-7B-R1-zero      |  32.7 (0.70) |\n| Qwen2.5-Math-7B-R1-zero-Div (Ours) |  **34.3 (0.79)** |\n| |\n|    Qwen2.5-Math-7B-SimpleRL-Zoo    |  33.9 (0.56) |\n|  Qwen2.5-Math-7B-Eurus-2-7B-PRIME |  34.0 (0.74) |\n​\n**Table B:  Performance of Llama models on K&K benchmark​ (Avg@8)**\n\n|               Method              |   K&K-3ppl  |   K&K-4ppl  |\n|:---------------------------------:|:-----------:|:-----------:|\n|       Llama-3.2-3B-Instruct       |  8.8 (1.17) |  2.8 (0.39) |\n|   Llama-3.2-3B-Instruct-R1-zero   | 24.5 (1.47) | 17.3 (1.06) |\n| Llama-3.2-3B-Instruct-R1-zero-Div | **27.8 (0.95)**|  **20.0 (0.75)** |\n\n\n\n**Q3: What is the computational overhead of the diversity objective during training?**\n\n**A6**: As stated in Equation 12 in the main paper, our method leverages the sampled data from GRPO, eliminating additional sampling costs. The only overhead comes from gradient calculations for the diversity term, which in our environment adds no more than 2% to the training time compared to the R1-zero method.\n\n\n[1] Understanding the Quality-Diversity Trade-off in Diffusion Language Models\n\n[2] Effective Diversity in Population Based Reinforcement Learning\n\n[3] Quality-Diversity with Limited Resources\n\n[4] GPQA: A Graduate-Level Google-Proof Q&A Benchmark\n\n[5] On Memorization of Large Language Models in Logical Reasoning\n\n[6] Logic-RL: Unleashing LLM Reasoning with Rule-Based Reinforcement Learning​\n\n\nWe hope these revisions adequately address your concerns, and we are happy to provide any further clarifications to facilitate the review process!"}, {"review_id": "mNMlWj92XF", "reviewer": "Reviewer_CURd", "summary": "The paper investigates how encouraging solution diversity can boost RL of LLMs on reasoning tasks. Analyzing on mathematical benchmarks, the authors show a strong positive link between the potential@k metric and the diversity of generated answers, suggesting that more varied reasoning paths signal greater room to train. Building on this, they propose a token-level entropy regularizer that is applied only to reward-positive samples during GRPO training, so that it prevents length bias and limits quality-diversity trade-offs. Empirically the method lifts Pass@1 accuracy across math benchmarks with richer solution variety. Key contributions are the study of diversity in LLM reasoning and empirical evidence that diversity promotion reliably enhances several models’ reasoning performance.\n.", "questions": "N/A", "limitations": "Yes, the authors discussed extensively in the conclusion section.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths\n- The authors provided clear empirical motivation to their method by showing that diverse reasoning paths signal larger RL head-room.\n- Applying token-level entropy only to reward-positive samples is a reasonable approach.\n- The empirical results demonstrated effectiveness of the proposed method on math benchmarks.\n\nWeaknesses\n- All experiments target math reasoning and Qwen2.5 models. It is unclear if the method/findings generalizes to other domains and models.\n- In figure 1, potential@k correlates with diversity only when pass@1 > 0.4, and the metric itself is heuristic. There lacks a solid scientific explanation on the models with high diversity yet low performance and remains inconclusive to make a statement on diversity-performance relation.\n- Relying on token-level entropy may still reward verbose yet redundant answers.\n- (minor) Diversity-based exploration and token-level entropy are being heavily studied in recent literatures. The proposed method is reasonable but not entirely new.", "was_revised": true, "final_justification": "I have read all other reviews and authors' rebuttal. They have properly addressed my raised questions in weaknesses, and thus I've increased my rating from 3 to 4.", "rebuttal": "We thank the reviewer for their insightful comments and the valuable time dedicated to evaluating our work. We are encouraged by the reviewer’s recognition that our approach—applying token-level entropy exclusively to reward-positive samples—is reasonable and grounded in clear empirical motivation. To address the concerns raised, we have conducted additional experiments to further validate the generalizability of our approach, and we offer detailed point-by-point explanations below.\n\n**W1: All experiments target math reasoning and Qwen2.5 models. It is unclear if the method/findings generalizes to other domains and models.**\n\n**A1**:Thanks for this important feedback. We have conducted additional experiments across other reasoning domains and on other model:\n\n- (1) We evaluated our trained (on the math dataset) models on the GPQA Diamond benchmark [1], with results shown in Table A. \n- (2) We applied our approach to the Llama-3.2-3B-Instruct model on the K&K dataset [2], a logic reasoning task previously studied in LLM Reasoning research [3]. We trained on 3-person scenarios and evaluated on both 3-person (-3ppl) and 4-person (-4ppl) tasks (Table B).\n\nThe results demonstrate that: (1) Improvements from enhanced diversity in mathematical domains generalize to graduate-level Q&A tasks (GPQA Diamond); (2) The approach is validated across both model architectures and reasoning domains.\n\n**Table A:  Evaluation on GPQA Diamond benchmark​ (Avg@8)**\n\n|               Method               | GPQA Diamond |\n|:----------------------------------:|:------------:|\n|           Qwen2.5-MATH-7B          |  28.5 (0.64) |\n|       Qwen2.5-Math-7B-R1-zero      |  32.7 (0.70) |\n| Qwen2.5-Math-7B-R1-zero-Div (Ours) |  **34.3 (0.79)** |\n| |\n|    Qwen2.5-Math-7B-SimpleRL-Zoo    |  33.9 (0.56) |\n|  Qwen2.5-Math-7B-Eurus-2-7B-PRIME |  34.0 (0.74) |\n​\n**Table B:  Performance of Llama models on K&K benchmark​ (Avg@8)**\n\n\n|               Method              |   K&K-3ppl  |   K&K-4ppl  |\n|:---------------------------------:|:-----------:|:-----------:|\n|       Llama-3.2-3B-Instruct       |  8.8 (1.17) |  2.8 (0.39) |\n|   Llama-3.2-3B-Instruct-R1-zero   | 24.5 (1.47) | 17.3 (1.06) |\n| Llama-3.2-3B-Instruct-R1-zero-Div | **27.8 (0.95)** | **20.0 (0.75)** |\n\n\n**W2: In figure 1, potential@k correlates with diversity only when pass@1 > 0.4, and the metric itself is heuristic. There lacks a solid scientific explanation on the models with high diversity yet low performance and remains inconclusive to make a statement on diversity-performance relation.**\n\n**A2**: Thanks for the insightful feedback. \n- For \"the metric itself is heuristic\", we would like to kindly point out that, as we stated in Appendix B in the main paper, the metric potential@k essentially captures the discrepancy between Pass@k and Pass@1. While Pass@k is often treated as the performance boundary for RL training on LLM [4], our Potential@k specifically measures the performance gain from RL training, approximated by subtracting Pass@1 (a measure for initial performance) from Pass@k.\n For each question $q_i$ before training begins, if Pass@1($q_i$) = 1, the question is already mastered with no improvement potential. When Pass@1($q_i$) = 0 but Pass@k($q_i$) = 1, GRPO training uses positive samples from k trials to teach the correct response. If both Pass@1($q_i$) = 0 and Pass@k($q_i$) = 0, the question provides no training signal as it remains unsolved. Hence, our definition of Potential@k focuses training on questions with partial capability, excluding both mastered and unsolvable questions, thereby capturing the true learning potential through the Pass@k to Pass@1 performance gap.\n- Explanation for models with high diversity but low performance: When quality is relatively low, exploration offers limited benefits to the model, this is supported by several studies in reinforcement learning (purely novelty-driven exploration struggles when the agent is essentially random [5]; when the policy is essentially random, the curiosity does not help learning of the extrinsic task[6]). Focusing on LLM reasoning, one extreme case is when a model’s samples contain no correct answers at all; no matter how high the diversity of its responses, it cannot access the correct answer.\n\n**W3: Relying on token-level entropy may still reward verbose yet redundant answers.**\n\n**A3**:\n- We deeply appreciate the reviewer’s astute observation regarding the potential for token-level entropy to incentivize verbose or redundant responses. However, this represents a challenging trade-off between quality and diversity that is inherent in such frameworks, as noted in prior RL literature [7,8,9]. Our approach mitigates this conflict by focusing token-level entropy exclusively on reward-positive samples, thereby aligning diversity with quality. This refinement is validated by our experimental results.\n- Moving forward, we recognize the value of exploring hybrid strategies that **further** balance diversity and quality, and one promising avenue could involve integrating efficient reasoning mechanisms [10,11] to optimize the generation of diverse yet concise responses.\n\n\n**W4: (minor) Diversity-based exploration and token-level entropy are being heavily studied in recent literatures. The proposed method is reasonable but not entirely new.**\n\n**A4**: Thanks for noting that. We would like to gently note that entropy-based methods have indeed gained attention in **very recent** studies. We will discuss these work in the final version.\n\n[1] GPQA: A Graduate-Level Google-Proof Q&A Benchmark\n\n[2] On Memorization of Large Language Models in Logical Reasoning\n\n[3] Logic-RL: Unleashing LLM Reasoning with Rule-Based Reinforcement Learning​\n\n[4] Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?\n\n[5] Learning to Discover Skills through Guidance\n\n[6] Redeeming Intrinsic Rewards via Constrained Optimization\n\n[7] Curiosity-Driven Reinforcement Learning from Human Feedback\n\n[8] Exploring the Performance-Reproducibility Trade-off in Quality-Diversity\n\n[9] Quality-Diversity Actor-Critic: Learning High-Performing and Diverse Behaviors via Value and Successor Features Critics\n\n[10] Stop Overthinking: A Survey on Efficient Reasoning for Large Language Models.\n\n[11] Think When You Need: Self-Adaptive Chain-of-Thought Learning\n\n\nWe hope these revisions adequately address your concerns, and we are happy to provide any further clarifications to facilitate the review process!"}, {"review_id": "JIql0iimXo", "reviewer": "Reviewer_kVwK", "summary": "This paper demonstrate a empirical finding that response diversity is correlated with the RL training performance. The authors then design an auxiliary loss to optimize the diversity of responses, which enhance the performance of GRPO algorithm. The evaluation is solid and comprehensive, and the paper writing is clear and fluent. Good job!", "questions": "Major Concerns:\n1. The empirical analysis demonstrate there is positive correlation between equation diversity and potential@k. However, when optimize the diversity of rollout responses, the author used entropy. Can you justify the gap between these two diversity metrics? What about assign higher weights to questions with high equation diversity during training? Or what's the relations between response entropy and potential@k?\n\n2. Table 3, it seems that in proper lambda, the pos + neg is better than pos with lambda = 0.05 or 0.02. This indicates that the influence of lambda is more important than excluding negative samples. Could you provide the ablations between pos+neg and pos in all lambda settings? \n\nMinor Concerns:\n1. Please directly write out the model names in Table 1. The \"base\" and \"base+\" is not that straightforward.\n2. Could you open-source your code?", "limitations": "yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\nThe paper insightfully discovers the correlation between the diversity of responses under GRPO and the algorithm's effectiveness, and leverages this correlation to optimize the GRPO algorithm. It also achieves excellent results in the evaluation.\n\nWeakness:\nThere are some minor flaws in writing, and the rest can be seen in the Questions.", "was_revised": true, "final_justification": "5 accept. I have read all rebuttals and  the authors have answered my questions properly.", "rebuttal": "We are grateful to the reviewer for their perceptive comments and the time dedicated to assessing our work. We are greatly encouraged by the positive assessment (the paper is insightful, the evaluation is solid, and the writing is clear and fluent). To address the concerns raised, we conducted additional experiments and we provide detailed explanations point by point below.\n\n**Q1: The empirical analysis demonstrate there is positive correlation between equation diversity and potential@k. However, when optimize the diversity of rollout responses, the author used entropy. Can you justify the gap between these two diversity metrics? What about assign higher weights to questions with high equation diversity during training? Or what's the relations between response entropy and potential@k?**\n\n**A1**: Thank you for this insightful question. The two diversity metrics are strongly correlated：(1) We conduct a re-evaluation in empirical analysis using entropy as the diversity metric, and the result shows that potential@k exhibits a positive correlation with entropy (with $R^2 > 0.73$) in high-quality models, which is consistent with the correlation observed for equation diversity.\n(2) Optimizing entropy can also enhance equation diversity, as evidenced by results in Table 2 in the main paper. This confirms that entropy serves as a valid proxy for diversity in our setup. While equation diversity represents an **explicit** measure, direct optimization of such explicit metrics in RL is often challenging and unstable due to their discrete and task-specific nature. Hence, we choose entropy as a more general **implicit** metric, which aligns with diversity objectives and enables stable training.\n\n**Q2: Table 3, it seems that in proper lambda, the pos + neg is better than pos with lambda = 0.05 or 0.02. This indicates that the influence of lambda is more important than excluding negative samples. Could you provide the ablations between pos+neg and pos in all lambda settings?**\n\n**A2**: We provide the full results for $\\lambda=0, 0.01, 0.02, 0.05$ under both pos and pos+neg settings in the table below. The results indicate that both the choice of $\\lambda$ and the application of a diversity objective on positive samples are important. Specifically, the selection of $\\lambda$ balances quality (the accuracy) and diversity, while applying a diversity objective on positive samples further stabilizes training and tends to converge to better quality.\n\n| $\\lambda$ | GSM8K | MATH500 | Olympiad Bench | College Math | Avg |\n|---------------|-------|---------|-----------------|--------------|------|\n| 0 | 88.7 | 75.6 | 38.2 | 46.2 | 62.2 |\n| 0.01, pos | **91.7** | **78.2** | **40.1** | **47.6** | **64.4** |\n| 0.01, pos+neg | 89.8 | 76.6 | 39.6 | 46.9 | 63.2 |\n| 0.02, pos | 90.7 | 76.0 | 38.4 | 45.9 | 62.8 |\n| 0.02, pos+neg | 87.9 | 75.4 | 37.2 | 45.0 | 61.4 |\n| 0.05, pos | 88.1 | 74.8 | 38.2 | 45.8 | 61.7 |\n| 0.05, pos+neg | 86.7 | 75.8 | 36.7 | 44.5 | 60.9 |\n\n\n**Q3: Please directly write out the model names in Table 1. The \"base\" and \"base+\" is not that straightforward.**\n\n**A3**: Thank you for the suggestion. We will replace the abbreviations \"base\" and \"base+\" with the full model names in the revised version to ensure clarity.\n\n**Q4: Could you open-source your code?**\n\n**A4**: Yes, we will after the paper is accepted. We had uploaded the code for your review.\n\nWe hope these revisions adequately address your concerns, and we are happy to provide any further clarifications to facilitate the review process!"}]

Author rebuttal:

Meta-review: The paper establishes positive correlation between diversity in RL training and the LLM reasoning, and proposed a simple yet effective novel diversity-aware policy optimization method to encourage diversity among positive solutions. Experiments on Qwen2.5-Math-7B and Qwen2.5-Math-1.5B validates the effectiveness of the proposed method. Overall reviewers are happy with the motivation, writing and execution of the paper, with some minor ask to extend the experiments beyond math reasoning and qwen model family.
Decision: Accept (spotlight)