Post-training Large Language Models for Diverse High-Quality Responses
Authors: Yilei Chen
Souradip Chakraborty
Lorenz Wolf
Ioannis Paschalidis
Aldo Pacchiano
Venue: ICLR 2026 Poster (2026)
Keywords: Large language models
Diversity
Reinforcement learning
Post-training
Primary area: foundation or frontier models, including LLMs

Abstract: Reinforcement learning has emerged as a popular method for post-training large language models (LLMs). While improving the model's performance on downstream tasks, it often reduces the model's output diversity, leading to narrow, canonical responses. Existing methods to enhance diversity are limited, either by operating at inference time or by focusing on lexical differences. We propose a novel training method named DQO (Diversity Quality Optimization) based on determinantal point processes (DPPs) to jointly optimize LLMs for quality and semantic diversity. Our approach samples and embeds a group of responses for each prompt, then uses the determinant of a kernel-based similarity matrix to measure diversity as the volume spanned by the embeddings of these responses. Experiments across instruction-following, summarization, story generation, and reasoning tasks demonstrate that our method substantially improves semantic diversity without sacrificing model quality.

Published as a conference paper at ICLR 2026
POST-TRAINING LARGE LANGUAGE MODELS FOR DI- VERSE HIGH-QUALITY RESPONSES
Yilei Chen" Boston University
Souradip Chakraborty
Lorenz Wolf
University of Maryland, College Park
University College London
Yannis Paschalidis
Aldo Pacchiano"
Boston University
Boston University
Broad Institute of MIT and Harvard
ABSTRACT
Reinforcement learning has emerged as popular method for post-training large language models (LLMs). While improving the model's performance on down- stream tasks, it often reduces the model's output diversity, leading narrow, canonical responses. Existing methods to enhance diversity are limited, either by operating inference time by focusing lexical differences. We propose a novel training method named DQO (Diversity Quality Optimization) based on determinantal point processes (DPPs) jointly optimize LLMs for quality and semantic diversity. Our approach samples and embeds group of responses for each prompt, then uses the determinant of kernel-based similarity matrix mea- sure diversity as the volume spanned by the embeddings of these responses. Ex- periments across follooiinn.....oiingg summarization, story generation, and rea- soning tasks demonstrate that our method substantially improves semantic diver- sity without sacrificing model quality. We open-sourse the code at: hhtps::/ github. com wtw.uu/.cooaaiirttallle..e. divrssityiiiii quality opt ieptimiien miniaaaion cewalityyppiimii
+27.2%
Normalized scores
0.8
+19.7%
+21.4%
0.4
+61.7%
0.2
Reasoning
Summarization
Instructi Cntruucton-following
Creatiie-writing
Figure 1: The quality and diversity scores of responses generated by the model trained with DQO compared the model trained solely with reward. The reported quality score the reward one generation. The diversity score is the average across multiple diversity metrics, each computed over 10 responses. All responses are generated with temperature 1.0.
"Correspondence ylchen9 hu.edu and pacchian @ bu.edu

---

Published as a conference paper at ICLR 2026
1 INTRODUCTION
Large language models (LLMs) are typically post-trained to better align with human intentions and perform effectively on downstream tasks (Ouyang et al. 2022 Bai et al. 2022). Reinforcement learning (RL) is commonly used to either maximize an existing reward function, or a reward model trained from human preference data (Ziegler al. 2020; Stiennon et al. 2020 Ouyang et al.. [022]
Bai et al. 2022 DeepSeek-Al et al. 2025). These methods substantially improve the output quality for targeted tasks. However, a widely observed drawback is that post-training often leads to a sharp reduction in output diversity, with models converging on narrow set of canonical responses (Kirk) (ct al. 2023 Murthy et al. 2024] Anderson et al. 2024 Xu et al. 2025 Casper et al., 2023). This loss of diversity is problematic across multiple dimensions: it limits reasoning and personalization by restricting alternative solution paths or user eser-preferred styles; it undermines test-time performance by reducing test-time search capabilities, robustness to distribution shift, and coverage of reward modes; and it weakens training dynamics by limiting exploration and the discovery of novel strategies.
Current efforts to promote diversity in LLM outputs are mostly limited to inference- time interven- tions such as temperature scaling (Ackley et al.) 1985), top-k sampling (Holtzman et al. 2020), and related strategies (Nguyen et al., 2024] Franceschelli & Musolesi (025). While these approaches help in improving the spread across the response distribution, they often only introduce shallow, token-level variation and fail produce truly diverse or meaningful responses. In many cases, they also hurt output quality, making the model less reliable or coherent. More critically, they cannot recover modes absent from the base model's learned distribution. A more principled approach is to optimize for diversity during training, but this poses two central challenges: defining diversity in computationally efficient and theoretically sound way, and balancing diversity with response quality.
Recent attempts to improve diversity during training largely remain at the lexical level. Yao et al.) (2025) encourage variation with a token-level entropy regularizer, but such measures fail to capture semantic diversity, which is often more meaningful to humans. [anchantin et al. (2025) extend direct preference optimization (DPO) (Raaillv et al., 2023) by selecting the most diverse candidate among high-reward responses, yet their notion of diversity is still based on surface features such as generation probability or word counts. Likewise, Li et al. (2025) aim to preserve diversity in supervised fine-tuning by carefully constraining probability transfer between tokens during updates, again focusing on token-level variation. More broadly, these approaches prioritize local lexical differences rather than encouraging models to generate responses that span distinct semantic modes. Most related to our work, [Cung et al. (2025) introduce a DPO variant that weights loss by average embedding distance, but remains PPO-specific and considers only pairwise distances, which can yield degenerate solutions.
In this work, we propose a principled training method based on determinantal point processes (DPPs) (Kulesza et al. 2012) to directly optimize LLMs for both quality and diversity in generated re- sponses. Unlike token-level entropy or lexical perturbations, our approach operates at the semantic level. Specifically, for each prompt we sample set of responses, map them into an embedding space using a pretrained encoder, and compute a similarity matrix via a kernel function. The diver- sity score is then defined as the determinant of this matrix, which corresponds to the volume spanned by the response embeddings. Optimizing this objective encourages the model to generate responses that span subspace in the answers' embedding space with the largest volume. The reward of each response can be regarded as a scaling factor of the corresponding embedding vector, providing an interpretable mechanism to balance quality against diversity. We refer to our algorithm as DQO (Diversity Quality Optimization). DQO is highly flexible and can be layered on top of existing state-of-the-art methods such as GRPO, making broadly applicable in practice. We evaluate DQO across sorruction-folowiing summarization, story generation, and reasoning tasks, and demonstrate that it significantly enhances semantic diversity while maintaining high response quality.
We summarize our contributions as the following,
Principled framework for Diversity Quality Optimization : We propose a principled method, DQO, for post-training LLMs to generate diverse, high-quality responses. DQO is a flexible approach that can be applied on top of existing reinforcement learning algorithms, such as PPO and GRPO.
2

---

Published as a conference paper at ICLR 2026
Semantic diversity beyond lexical variation We demonstrate that the DPP-based for- mulation provides a theoretically grounded framework for defining diversity, ensuring that responses span the semantic space both broadly and meaningfully.
Qiveity-diversity trade-off': We experimentally show that DQO improves semantic di- versity while preserving response utility, coherence, and task accuracy across wide range of tasks. In addition, we conduct extensive ablations to illustrate the trade-off between quality and diversity.
PRELIMINARIES
2.1 NOTATIONS
For case of readability, summarize some frequently used notations here. We use x and y to represent a prompt and a response, respectively. We represent a group & responses ......... V/ by Vi-k and denote {1/..... V/-1, Ve+1...... y by y-r- We I E B to represent the identity matrix with size k. And dot(-) represents the determinant a matrix.
2.2 REINFORCEMENT LEARNING
Reinforcement learning has become a widely adopted approach for post-training LLMs with ei- ther an existing reward function or the one inferred from a preference dataset (c.g.. RLHF). With the reward function, the model is typically optimized by maximizing the following KL.regularized objective,
arg (a( [
where ((To) y (r,,, y)] is the expected return and 33 is a hyperparameter that balances the KL divergence penalty and the rewards. Among existing algorithms, PPO (Schulman et al.) 2017) and GRPO (Shao al., (024) have demonstrated strong empirical performance, by intro- ducing some practical techniques including the clipping mechanism and group-based advantage estimation, respectively.
2.3 DETERMINANTAL POINT PROCESSES (DPPs)
In this work, we quantify the diversity of LLM generated outputs based on ideas derived from the Determinantal Point Process (DPP) literature (for a comprehensive introduction to DPPs, please refer to Kulesza et al. (2012)). Below introduce the definition of an L-ensemble, which is subclass of DPPs
Definition (L-ensemble) Let y= (1,2...,NN) be a ground set, and y be random subset. Suppose L E NNN is a real symmetric positive semi-definite matrix. We say L defines an L-ensemble, if for every A ACD y,
Pr(Y = AA x det(La),
where LA is the submatrix of indexed by A.
If we think of the entries of L as measurements of similarity between pairs of elements, such as the dot product of the feature vectors of items, the determinant (e.(). corresponds the squared volume spanned by the feature vectors of items in A, which increases when the vectors are diverse and decreases when they are redundant or highly correlated. Thus, DPPs are well-suited to represent distributions over subsets of items where sets consisting of diverse items more likely.
For instance, consider two-element set A
{i.j). By definition.
Li: Lij Lis
Pr(A)
Luby
Lig Lags
Lij
items and j are highly similar, then Lij is large, resulting in a low probability. Conversely, diagonal (i.e., Tag 0), there are correlations and the elements occur independently.
3

---

Published as a conference paper at ICLR 2026
Policy Update
V
Reward
Y/2
Policy
DPP-Diversity
RL-Algo
Model
:
Semantic Diversity
Yk
Robust to Degenerate Solutions
Figure 2: The DPP-based diversity metric of DQO promotes meaningful semantic diversity while ensuring robustness agains degenerate solutions.
3 DIVERSITY QUALITY OPTIMIZATION
We now formally define our notion semantic diversity inspired by DPPs and how incorporate it in reinforcement learning algorithms to jointly optimize quality and diversity.
3.1 DIVERSITY VIA DPPS
Based on the above definition DPPs, given a group of responses Vicks we can formulate their diversity score as,
Div(y det(Lo(V1.))
(2)
where Lo (yra)) model which map response into dimensiinnsii semantic space. Although in most of this
f(((y:) d(y;)), f is a kernel functiio and o(-) is a selected embedding
work we set the kernel function as the dot product, (((())) o(y)) = (o(y)), Φ(y;) )) Appendix D also explore Gaussian kernel function and provide additional results. For simplicity, when it clear from the context, we will omit the subscript in Lag.
Our definition of diversity offers main advantages. First, operates in the embedding space of the responses, allowing to capture semantic diversity which is typically what humans intend. Second, its determinant-based definition induces notion of group diversity that overcomes the limitations of simple pairwise distance metrics. A key limitation of pairwise distance measures, such as the average distance across responses, their well-known vulnerability "clustering" effect, as noted My/Paarer--ollee Holder et al. (2020). An algorithm optimizing for this metric might produce responses that form few distinct, widely separated clusters, creating misleading sense of diversity. In contrast, our determinant-baan -baeed metric, which encourages the formation of a parallelepiped with a large volume the embedding space, directly addresses this issue. The determinant is highly sensitive the linear independence of the response vectors. If responses form tight clusters, the vectors within a cluster become nearly linearly dependent, causing the determinant of the similarity matrix to approach zero, regardless of the large distances between clusters. This correctly identifies a lack true diversity. Furthermore, the determinant is more robust measure because it recognizes when responses, despite having large pairwise distances, are confined lower-dimensional subspace. This forces the system to explore the full high-dimensional embedding space, ensuring genuine diversity that simple pairwise distances fail capture.
degenerate
3.2 QUALITY-DIVERSITY OBJECTIVE
Using the DPP based diversity metric we now present the objective optimized by our DQO algorithm. For each prompt ær we sample i responses Vı (=(=) from the policy, similarly to the sampling performed as part of GRPO. Instead of optimizing only the reward, we incorporate a diversity term based on the logarithm of our diversity metric into the objective. This yields the objective given by
(3) where (V(1)) is defined in 2 The hyperparameter controls the trade-off between quality and diversity. Maximizing Inio(TT)) directly optimizes the policy for both quality and semantic diversity in generated responses.
(Di)(TT) Sre
Σ r(a, vi) + a log det(Ls (KL LKL [Te]l
4

---

Published as a conference paper at ICLR 2026
In fact, it can be shown that by optimizing (3),
the optimal policy satisfies,
Tdio (Ur:klz))))) x Tref (Ur:k / (r) oxp (3
Σ r(z, yi) + a log dot (ct((L
(4)
For simplicity of exposition, suppose B = a. We can define a reward-augmentted embedding vector for the prompt-response pair (z, y) as (z, v) exp
r(a.y)
(ref(yl2) o(y). Here, the reward
acts as a scaling factor of the original semantic embedding. With the formulation of the reward- augmented embeddings, we can show that our optimal policy satisfies,
Adie (V1.slz) det det(Lş y1:k))
For the complete derivation, refer the reader Appendix The expression above shows that our optimal policy (5) assigns probabilities to groups of responses in proportion to the determinant of the Gram matrix constructed from their embedding vectors. Geometrically, this means the policy selects groups of vectors in the response embedding space according to the squared volume of the parallelepiped spanned by those vectors.
The balance between quality and diversity also admits a clear geometric interpretation. The em- bedding vector y consists of two components: semantic embedding vector, which determines its direction, and reward, which determines its norm. To maximize the volume of the spanned space, one should select vectors that are well separated from each other (i.e., diverse responses) while also having large norms (i.c., high-quality responses). The overall trade-off is governed by the hyperpa- rameter a.
Connections with D-Optimal Design In the classical theory of experimental design, the goal is to select design points that most informative for estimating unknown parameter vector. Consider the linear model represent the experimental conditions under which data is observed, and their selection determines how precisely the parameter f can be estimated. The associated information matrix is defined as
0+C, where one collects design points ž1,.... 2n. E Rd These design points
M = Σ 42 = TZ
where = [2. 2nl is the d design matrix. The criterion of D-optimal design selects these design points to maximize (M), since this minimizes the volume of the confidence ellipsoid for unknown parameter (Kiefer 1959 Pukelsheim 2006). By Sylvester's determinant identity,
det(M) = det(ZZ det(Z Z),
(012). so D-optimality equivalently expressed maximizing the determinant of the Gram matrix (the kernel matrix) of the design vectors under the dot-product kernel. Our DQO objective can be seen as direct analogue this construction. The role of the design vectors is instead played by the both the semantic content of response and quality signal. Maximizing the determinant of the Gram matrix built from these embeddings is therefore analogous to maximizing information gain in D-optimal design: encourages the selected responses to be as linearly independent as possible in the dispace, ensuring that they collectively span a high-volume region that balances semantic diversity and reward. In this way, DQO can be viewed as extending the principle of D-optimal design from parameter estimation in bandits the joint optimization of quality and diversity in language model responses. See a more thorough discussion in Appendix
(see, e.g., Horn & Johnson
embeddings (ar, y), which incorporate
3.3 ALGORITIM
We noticed that directly optimizing 3 presents challenges, including high variance in stochastic gradient estimates and risks of numerical instability. To address these issues, we now present a practical algorithmic formulation that stabilizes training and makes the QD objective feasible in
5

---

Published as a conference paper at ICLR 2026
practice. To identify the source of these challenges, we begin by computing the gradient of (Do) which is given by (for simplicity, we omit the KL-regularization term here),
Vlog (log)) w)) + log det (ott((((xx))))
(6)
The first issue is that the determinant of L(y1a) can be close to zero, which results in very large negative value of
This unbounded diversity term destabilizes training and com- plicates the trade-off between quality and diversity, the point that only carefully chosen is effective. To mitigate this issue, we propose to consider the determinant the matrix L(y1:)) + Ig instead. It can be shown that by adding identity matrix, have k log(det (T [x.))
which is well-bounded. Briefly, adding an identity matrix to our objective can be regarded as a reg- ularization term; we further discuss its effect on the objective in Appendix and provide ablation results in Appendix D
The second issue is that the gradient consists of the sum of the gradients of k responses yi-k. which causes it to have high variance, especially for large k. To mitigate the issue of inflating variance, we propose use leave-one-out (loo) gradient estimators by subtracting the log- de-eeerrmmnnnt of the gram matrix which leaves one response out,
k
V log (u(l)) (n(a, ye) + a log det(L(y).) + 1x) det Lyy(L(y-s) + 1x-1)
vloo (T)) = -m
Importantly, it can be shown that the loo estimator is unbiased and has nice property the bound- edness of its value shown in Lemma (for the proof, please refer to Appendix (B). Lemma shows that the diversity term is non-negative, with upper bound of order log(k)) which increases slowly as l: becomes large. This property stabilizes training and makes DQO robust to large values of k.
Lemma Let write the eigenvalues of L(y1=k) d ... 2 \1, then have 1 A 2 den(( 2 + \₁. And the eigenvalue of L(g1:)) is always in [0, k] since the embedding dou(L(y-(++1)
vectors are normalized, we have 1+k 2 det(L(grx)+1) 2 1 and log(1+k) 2 log det(L(g)=)+1)
det(((v-))+7) 2 0.
det(L(y-+++]
EXPERIMENTS
In this section, we conduct series of experiments to evaluate the performance of DQO in generating diverse and high-quality responses. Specifically, we aim to answer the following questions:
Does DQO improve diversity in responses and how does it compare with the reward-only baseline and other existing quality-diversity algorithms?
Does DQO achieve a favorable balance between quality and diversity? Can the model preserve or improve task performance while enhancing diversity?
Is the performance of DQO consistent across different tasks and settings?
How does DQO manage the trade-off between quality and diversity, and how do its hyper- parameters influence performance?
4.1 CITY RECOMMENDATION
We begin with simple synthetic experiment on city recommendation clearly illustrate the di- versity achieved by DQO. In this task, the model was prompted to recommend a city for traveling along with a concise reason. The exact prompt we used is provided in Appendix [H] We compared DQO with GRPO and also implemented a variant of DQO using the average pairwise distance as the diversity score, which we refer to as DQO-pairwise distance; the original algorithm is denoted as -gte-eetrmnnnat. The results are summarized in Figure 3 and the full details, including exact numbers and city names, are reported in Appendix

---

Published as a conference paper at ICLR 2026
DQO clearly encourages the model to generate more diverse recommendations. When trained solely with the reward, the model tends to con- verge on recommending the same city repeatedly. For using the pairwise distance score, we observed that the model's recommenda- tions were dominated by two major cities, which aligns with our previous analysis. This occurs because high pairwise distance can be achieved with two widely separated clusters. In contrast, the determinant-basse approach penalizes linear dependence, encouraging responses to span the space as broadly as possible. As shown in Fig- ure 3 DQO-determinant produces the most di- verse set of recommendations.
the diversity
From left to right: GRPO, DQO- pairwise distance, DQO-determinant. Each cir- cle represents a different city, with the size pro- portional to the number of times it was recom- mended. For each model, we sampled 100 times with temperature of 1.0.
Figure 3:
4.2 GENERAL TASKS
In the above section, we show the effectiveness DQO in promoting diversity on the controlled city recommendation task, we now turn to a broader evaluation general language model tasks. We implemented extensive experiments on four different kinds of tasks including reasoning (GSM8K (Cobbe et al. 2021 summarization (CNN-dailymail (See et al.] (017)). story- story-writing (Common- Gen (Lin et al. 2020)) and cestructiio-folloiing (Dolly (olovee et al. (2023)).
We compare DQO to the baseline algorithm which trains the model solely with reward. For rea- soning tasks, we use GRPO, while for non-reasoning tasks, we adopt PPO. We also compare DQO algorithms: with other two popular For the detailed experimental setup, please see Appendix
(He et al. 2025) and GRPO-
entropy (ao et al. 2025).
In this work, employ reward model to provide quality scores. Notice that, for reasoning tasks, we also rely on the reward model rather than outcome-basse rewards, due to the observed phe- nomenon of reward hacking with outcome reward: with only outcome reward provided, the model trained with the quality-diversity objective learns to first generate a correct answer to the question thus securing the outcome reward, then generate some random irrelevant contents to artificially increase diversity (for details, please refer to Appendix G). During training, rewards are normalized by dividing by an empirical maximum value to ensure comparable scale with the diversity score, whereas during evaluation, we report the unnormalized rewards.
We report passên metric (i.e., the highest score among n responses) as measures of the quality in the responses with n varies from 1 to 10. Without special clarifications, responses are sampled with a temperature 1.0. And we use multiple metrics to measure the diversity in the responses which we summarize below,
Distinct-n: Count the ratio of unique n n-grams among the responses.
Self-BLEU (Papineni et al., 2002) and Self-ROUGE (Lin 2004) score: ries to measure the similarity of languages. Note these scores measure the similarity, to be consistent with other metrics, we report Score.
Two popular met-
LLM-as-a-Judge: We prompt an advanced model GPT-4o-mini to judge the model's out- put in terms of the diversity, serving as a surrogate for human judgment. In Appendix F. we show the results of a small-scale human study demonstrating that LLM-as-a-Judge produces evaluations consistent with human assessments.
We first compare the performance of DQO against all baseline algorithms. The results are sum- marized in Table []] To evaluate the model, we select a representative non-reasoning task and a reasoning task. Among all algorithms, DQO is the only one that achieves both high diversity and high quality scores across both tasks. GRPO-likelihood shows performance comparable to DQO on GSM8K, whereas GRPO-entropy performs well on GSM8K
the Dolly task but underperforms but poorly on Dolly. These results demonstrate that DQO consistently delivers strong performance in post-training LLMSs to produce diverse and high-quality generations.
7

---

Published as a conference paper at ICLR 2026
Table I: The quality and diversity scores achieved by DQO compared other baseline algorithms on the instruction-following task: Dolly, and the reasoning task: GSM8K. Diversity metrics are calculated across 10 generated responses per prompt.
Diversity t
Quality f
Method Task
distinct-1 1
distinct-4
self-bleu
self-rouge
pass@1
pass 10
Dolly
PPO
0.24
0.64
0.41
0.49
5.65
8,39
GRPO like-like
0.26
0.70
0,46
0.54
5.86
8.50
GRPO-entropy
0.36
0.75
0.56
0.57
4.71
7.70
DQO
0.28
0.69
0.46
0.54
5.92
8.74
GSM8K
GRPO
0.09
0.32
0.09
0.21
76.8
87.9
GRPO-likelihood
0.26
0.86
0.53
0.59
50.9
80.4
GRPO GRPO-entropy
0.10
0.38
0.12
0.25
77.0
92.6
DQO
0.10
0.42
0.14
0.31
76.3
91.2
4.3 QUALITY-DIVERSITY BALANCE
Here, we present more fine-grained results on the diversity and quality of responses generated by the model trained with DQO. For comparison, we also include the results of the model trained solely with the reward.
Quality. In Figure we show the pass@n performance across four tasks with n varying from to 10. DQO exhibits better performance than the baseline model especially when n is large. Besides, in the case of n = 1, our model has similar or better performance to the baseline. Together, the results show that our method does not hurt pass@1 performance while providing better passEn performance with > indicating that our model can generate both high-quality and diverse re- sponses.
Doo
DCO
1LD
380
surpld naprom
GSMSK
(b) CaN-dailymail
(c) Dolly
(d) Gen
Figure 4: The performance of the trained model on pass@n metrics. For DQO, we set hyperparam- 1.0 and & 4.
Diversity. The superior performance passán already suggests that method enhances sponse diversity. To further validate this, we present six diversity metrics Figure 4 For each metric, higher values indicate greater diversity. As shown in the figure, DQO consistently outper- forms the baseline model, demonstrating a clear advantage in diversity. In particular, for the LLM- as-a-judge metric, the advanced model GPT-4o-mini strongly recognizes the diversity of responses generated by our approach (See Appendix Ib. highlighting improvements at the semantic level.
Pareto frontier. To illustrate how DQO achieves favorable balance between quality and diver- sity, we plot the Pareto frontiers DQO and the baseline model by varying either the training steps the sampling temperature in Figure 6 Across different sampling temperatures (the right in Fig- 6), our model consistently occupies the upper-right region relative to the baseline, demonstrating robust advantage balancing quality and diversity at the inference stage. Similarly, when varying
8

---

Published as a conference paper at ICLR 2026
dedecce
cef rugge
dere
oet taag
dirtect
difive
cat raue
dadincs4 Lo
detnc-
dbiinds L24
dadhcc
(a) GSM8K
(b) CNN-dailymail
(c) Dolly
(d) Gen
Figure 5: The performance of the trained model on six diversity metrics. For each metric, the higher value means the higher diversity. For DQO, set hyperparameters 1.0 and A = 4. And the
diversity metries are calculated across 10 generated responses per prompt.
the training steps (the left 6. our model remains Pareto-optimal throughout the entire train- ing process, indicating that it consistently achieves a better quality-diversin balance throughout the entire training process.
PPG
0.85
GRPD
1.0
DQ0
DQO
0.20
0.0
0.75
0.70
0.-
0.85
0.7
0.55
0.6
030
0.5
045
0.4
0.5
0.6
0.7
0.8
0.9
1.0
0.9
0.1
0.2
0.2
0.4
Diversty
Diversity
(a) Gen
(b) GSM8K
Figure 6: Pareto frontiers on quality and diversity of our model and the baseline. On the left, each point is a model trained with different training steps and the sampling temperature set as 1.0. On the right, we take the final trained model but vary the sampling temperature.
4.4 ABLATION STUDY
DQO introduces key hyperparametens: and k. The hyperparameter a controls the weight of the diversity term in the training objective, while k is the number of responses sampled per prompt in the algorithm. Both parameters jointly influence the trade-off between output quality and diversity. To analyze their impact, we conduct experiments across different values of a and k, and the results on Dolly task are summarized Table 2 (for the ablations on GSM8K task, please refer to Appendix D).
From Table 2 we can observe the trade-off between quality and diversity when changing the value of a k. Both increasing a and & can enhance the diversity of generated responses, however, in- creasing & incurs additional computational costs since more responses must be generated. Overall, compared with the baseline model, DQO exhibits robust improvements in both quality and diversity across a wide range of a and values, suggesting that does not require highly sensitive hyperpa- rameter tuning.
In addition to the ablations on different values of a and k. we also conduct ablations on different embedding models, kernel functions and on the weight of the identity matrix in the determinant calculation, controlled by the parameter y in dLe(L((g ) tlg). Due to space constraints, we refer readers to Appendix D for detailed results.

---

Published as a conference paper at ICLR 2026
Table The quality and diversity scores of the model trained with different values of hyperparam- eters k and on the Dolly task. Diversity metrics are calculated across 10 generated responses per prompt.
Diversity 1
Quality
Method
distinct-1
distinct-4
self-bleu
self-rouge
pass
pass 10
PPO
0.24
0.64
0.41
0.49
5.65
8.39
0.5,
0.28
0.69
0.44
0.53
5.84
8.79
1.0, k 4
0.28
0.69
0.47
0.54
5.92
8.73
1.5,
0.33
0.79
0.57
0.61
5.47
8.56
2.0, k 4
0.35
0.82
0.54
0.64
5,42
8.69
k 2, 1.0
0.24
0.62
0.40
0.50
5.71
8.13
k = 4, = 1.0
0.28
0.69
0.47
0.54
5.92
8.73
k 6, 1.0
0.31
0.76
0.49
0.58
5.71
8.83
8, a = 1.0
0.32
0.79
0.52
0.61
5.64
8.64
CONCLUSIONS
In this work, we propose an algorithm DQO to post-train LLMs for diverse high-quality responses. Based on determinantal point processes, DQO defines the diversity group of responses as determinant of a kernel-based similarity matrix of the embeddings of those responses. This defini- tion of diversity has straightforward interpretation as the squared volume of the space spanned by the embeddings of the response. We conduct extensive experiments across different kinds of tasks, and show that DQO can optimize the model to generate significantly more diverse responses while maintaining high quality in the generated responses.
Although DQO achieves superior performance, there are some limitations in this work. First, the quality-diversiiy objective is vulnerable to reward hacking when using the outcome reward. A re- ward model is needed which limits the applicability in many reasoning tasks where the outcome reward commonly used. Second, DQO relies the embedding models to map responses into a semantic space. The performance of DQO depends on the quality of these embeddings. A more principled and adaptive method for measuring diversity could better capture the underlying semantic variation, potentially adjusting automatically to the specific requirements of different tasks, which may emphasize different aspects of diversity, thus achieves better performance.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
David H. Ackley, Geoffrey E. Hinton, and Terrence J. Sejnowski. A learning algorithm for bolzmann machines. Cognitive Science, 9():147-166, 1985, ISSN 0364-0213. doi: htpps:
URL https: /wwww www.siiiindeiieetttttttt com/
Eltayeb Ahmed, Uljad Berdica, Martha Elliott, Danijela Horak, and Jakob N Foerster. Intent factored generation: Unleashing the diversity in your language model. arXiv preprint arXiv:2506.09659) 2025.
Barrett R Anderson, Jash Hemant Shah, and Max Kreminski. Homogenization effects of large language models on human creative ideation. In Proceedings of the 16th Conference on Creativity & Cognition, 2024.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, 7ac Haffield-Dddss, Danny Hernan- dez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Ka- plan. Training helpful and harmless assistant with reinforcement learning from human feedback, 2022. URL BtLpstts:: / arxiv wrg. abs/ / 2200222242... 05862
Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomek Korbak, David Lindner, Pedro Freire, Tony Tong Wang. Samuel Marks, Charbel-Raphael Segerie, Micah Carroll, Andi Peng, Phillip J.K. Christoffersen, Mehul Damani, Stewart Slocum, Usman Anwar, Anand Siththaranjan, Max Nadeau, Eric Michaud, Jacob Pfau, Dmitrii Krasheninnikov, Xin Chen, Lauro Langosco, Peter Hase, Erdem Biyik, Anca Dragan, David Krueger, Dorsa Sadigh, and Dylan Hadfield-Menell. Open problems and funda- mental limitations of reinforeement learning from human feedback. Transactions Machine Learning Research, 2023. ISSN 2835-8856. URL https: htttps://penreeiew... wwwp/eveee.eeeee/////e/ net/ forum? id=bx24kpJJEBb Survey Certification, Featured Certification.
John Joon Young Chung, Vishakh Padmakumar, Melissa Roemmele, Yuqian Sun, and Max Kreminski. Modifying large language model post-training for diverse creative writing, 2025.
URL htttps/s://axxi .rg/ abs 17126
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14101 14.1668 2021.
Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. Free dolly: Introducing the world's first truly open instruction-tuned 1lm. 2023, URL htt.:///w.................aaaaa. opaee--pee--eeee
DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang. Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang. Bochao Wu, Bei Feng. Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang. Han Bao, Hanwei Xu, Haocheng Wang. Honghui Ding, Huajian Xin, Huazuo Gao. Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang. Lecong Zhang, Liang Zhao, Litong Wang. Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, panpan Huang, Peng Zhang, Qiancheng Wang. Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou Shanhuang Chen, Shengfeng Ye, Shiyu Wang. Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye, Tao Yun, Tian Pei, Tianyu Sun, T. Wang, Wangding Zeng, Wanjia Zhao Wen
11

---

Published as a conference paper at ICLR 2026
Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang. W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xi- aosha Chen, Xiaowen Sun, Xiaoxiang Wang. Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang. Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. Deepscck-rl: Incentivizing reasoning capability in Ilms via reinforce- ment learning. 2025. TRLLhtts:: //axiiv../iiivaaaaaaaaii wwt./bggg//////o// abw./5/11.-- 12948
Steve Fisk. A very short proof of cauchy's interlace theorem for eigenvalues of hermitian matrices. arXiv preprini math/5502408, 2005.
Giorgio Franceschelli and Mirco Musolesi. Diffsampling: Enhancing diversity and accuracy in neural text generation, 2025. URL htttppp:/aaxxx...rrr/aa/////2/. 14037
Yanzhu Guo, Guokan Shang, and Chloé Clavel. Benchmarking dingsaiii of large language models. CoRR, abs/2412.100711, 2024.
URL https:///5...crr//00.45555////
2412. 10271
Andre He, Daniel Fried, and Sean Welleck. Rewarding the unlikely: Lifting grpo beyond distribution sharpening. arXiv preprint arXiv:2506.0220 0255055, 2025.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In International Conference on Learning Representations, 2020. URL https:
//openreviee ew. net/ for.moorrmrmr/////m dey.gg//dddyyGrrrrrr
Roger A Horn and Charles R Johnson. Matrix analysis. Cambridge university press, 2012.
Jack Kiefer. Optimum experimental designs. Journal of the Royal Statistical Society: Series (Methodological), 21(2):222-304, 1959.
Robert Kirk, Ishita Mediratta, Christoforos Nalmpantis, Jelena Luketina, Eric Hambro, Edward Grefenstette, and Roberta Raileanu. Understanding the effects of rlhf on Ilm generalisation and diversity. CORR, abs/2310.06452, 2023. URL
1310.06452
Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. arXiv preprint arXiv:2302.09664 2023.
Alex Kulesza, Ben Taskar, et al. Determinantal point processes for machine learning. Foundations and Trends in Machine Learning, 5(2-3)) 2012.
Aditya Kusupati, Gantavya Bhatt, Aniket Rege, Matthew Wallingford, Aditya Sinha, Vivek Ra- manujan, William Howard-Snyder, Kaifeng Chen, Sham Kakade, Prateek Jain, et al. Matryoshka representation learning. Advances in Neural Information Processing Systems, 35:30233-30249, 2022.
Jack Lanchantin, Angelica Chen, Shehzaad Dhuliawala, Ping Yu, Jason Weston, Sainbayar Sukhbaatar, and Ilia Kulikov. Diverse preference optimization. arXiv preprint arXiv:2501.18101, 2025.
Ziniu Li. Congliang Chen, Tian Xu, Zeyu Qin, Jiancong Xiao, Zhi-Quan Luo, and Ruoyu Sun. Preserving diversity in supervised fine-tuning of large language models. ICLR, 2025.
Bill Yuchen Lin, Wangchunshu Zhou, Ming Shen, Pei Zhou, Chandra Bhagavatula, Yejin Choi, and Xiang Ren. CommonGen: A constrained text generation challenge for generative commonsense reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 1823-1840 Online, November 2020. Association for Computational Linguistics. URL https: // 7/www. ac web. www. /eee.oorg//mthhooog//22222 findi ngs emn p. 165. 65.
12

---

Published as a conference paper at ICLR 2026
Chin- Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pp. 74-81, 2004.
Chris Yuhao Liu, Liang Zeng, Yuzhen Xiao, Jujie He, Jiacai Liu, Chaojie Wang, Rui Yan, Wei Shen, Fuxiang Zhang, Jiacheng Xu, Yang Liu, and Yahui Zhou. Skywork-reward-v2. Scaling preference data curation via human-ai synergy. arXiv preprint ar arXiv:2507.013522 2025.
Sonia K Murthy, Tomer Ullman, and Jennifer Hu. One fish, two fish, but not the whole sea: Align- ment reduces language models' conceptual diversity. arXiv preprint arXiv:2411.04427, 2024.
Minh Nhat Nguyen. Andrew Baker, Clement Neo, Allen Roush, Andreas Kirsch, and Ravid Shwartz-Ziv. Turning up the heat: Min-p sampling for creative and coherent Ilm outputs. arXiv preprint arXiv:2407.01002. 2024.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Gray, John Schulman, Jacob Hilton, Fraser Kel- ton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neu- ral Information Processing Systems, 2022. URL https: http::::///peeeeeeewee net ffrrumrrdiiii TGSKACXEON
Vishakh Padmakumar and He He. Does writing with language models reduce content diversity? In The Twelfth International Conference Learning Representations, 2024. URL https:
/ //opereeview. wttprriwwwwee.oorr..e fott//oronnidd-eeeetttt
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311-318, 2002.
Jack Parker- Parker-Holder, Aldo Pacchiano, Krzysztof Choromanski, and Stephen Roberts. Effective Di- versity in Population Based Reinforcement Learning. In Advances in Neural Information Pro- cessing Systems 34. 2020.
Jack Parker-Holder, Aldo Pacchiano, Krzysztof M Choromanski, and Stephen J Roberts. Effective diversity in population based reinforcement learning. Advances Neural Information Processing Systems, 33:1 13:18050-188 18062, 2020.
Friedrich Pukelsheim. Optimal design of experiments. SIAM, 2006.
Rafael Rafailov, Sharmm Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https:
//openzeview.o www.teertmmiiieeemoeeoeeeemmoeee http://oorron..-csPSSIXeeeeee
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy 2017.
optimization algorithms. arXiv preprint
Abigail See, Peter J. Liu, and Christopher D. Manning. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1073-1083, Vancouver, Canada, July 2017. Association for Computational Linguisties. doi: 10. 18155//11/111--0999 URL https:
//www. ac www.aaclweb... http://www...aaaatoooo/////-----
Chantal Shaib, Joe Barrow, Jiuding Sun, Alexa Siu, Byron C Wallace, and Ani Nenkova. Stan- dardizing the measurement text diversity: A tool and comparative analysis, 2024. URL
https:://openeeveww /
wewperrwwwweenee/oooomii...m. http:/ooummmmoooooo jyRCi r300q
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi. Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathe- matical reasoning in open language models, 2024. URL RRL/httpa:/// ://pxiiv///x..v../// otgg/bs//4/00
03300
13

---

Published as a conference paper at ICLR 2026
Alexander Shypula, Shuo Li, Botong Zhang, Vishakh Padmakumar, Kayo Yin, and Osbert Bas- tani. Evaluating the diversity and quality of LLM generated content. In Second Conference on Language Modeling, 2025. URL/ttts: http:///peeeeeeeeeeeeeeee ott.://wee.eeermmmoooo000o0ea66.
Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback. In Proceed- ings of the 34th International Conference on Neural Information Processing Systems, NeurIPS '20, 2020.
Jens Tuyls, Dylan J Foster, Akshay Krishnamurthy, and Jordan T Ash. Representation-basee explo- ration for language models: From test-time to post-training. arXiv preprint arXiv:2510. 1668, 2025.
Weijia Xu, Nebojsa Jojic, Sudha Rao, Chris Brockett, and Bill Dolan. Echoes in ai: Quantifying lack of plot diversity in Ilm outputs. Proceedings of the National Academy of Sciences, 122(35), 2025.
Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xue, Runji Lin, Tianyu Liu, Xingzhang Ren, and Zhenru Zhang. Qwen2.5-math technical report: Toward mathematical ex- pert model via seff-improvement. arXiv preprint urXiv:2409.122
2024.
Jian Yao, Ran Cheng, Xingyu Wu, Jibin Wu, and Kay Chen Tan. Diversity awarr policy optimization for large language model reasoning. arXiv preprint 2rXiv:2505.23433 2025.
Daniel M. Ziegler. Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences, 2020. URL httph:tt///att //axxiv org/ abs/1909, 08593
14

---

Published as a conference paper at ICLR 2026
A
RELATED WORKS
Evaluating Diversity of LLMs. Several works have focused on evaluating the diversity of LLM generated content (Guo al., 2024; [Shiib al. 2024), also on investigating the impact of post- training on diversity metrics (Kirk et al. [023 Shypula et al. 2025). The lack of diversity in LLM generated content also affects text written by humans using LLMs (Padmakumar
He 2024
Improving Diversity of LLMs. There are mainly two lines of works on promoting diversity in LLMs. One focuses on inference strategies. [guyyen et al. (2024) proposed a decoding method to reallocate the next-token probabilities which they show can increase the entropy of the correct solutions. The DifiSampling strategy, proposed by |Franceschelli largest difference between consecutive probabilities of tokens in sorted distribution to promote di- versity while maintaining correctness. [Ahmed et al. (2025) proposed a two-stage inference strategy which consists of high-temperature key words sampling process and low-temperature expansion procedure.
Musolesi (2025), considers the
Another line of works focus on the training strategy best clicit diversity from LLMs. Lanchantin iet al.] (2025) proposed diverse preference optimization. They selected the most diverse response from the high-reward group and the least diverse response from the low-reward group to form the preference pair. The selection based on some diversity criteria. Yao et al.. (2025) shows that by adding entropy term of correct answers the reward-based objective, LLMs can improve the di- versity while maintaining the quality. Different from those using reinforcement learning algorithms, Li (2025) instead study the supervised finetuning approach. They proposed carefully-designe update strategy to mitigate the distribution collapse in SFT, thus encourages diversity. Most related our work, Chung et al. (2025) propose variant of DPO that weights the loss by the average pairwise distance in cosine similarity after embedding responses, this however, is limited to DPO, considers only pairwise distances, and requires sampling k 2 responses per prompt in the train- ing dataset. Concurrently, [Tuyls et al. (2025) proposed a similar ippresentation-basee objective to encourage the exploration of large language models.
Determinantal Point Processes. Determinantal point processes (DPPs) (Kulesza et al. 2012), are a class of probabilistic models that arise in quantum physics and random matrix theory for modeling repulsion. DPPs are well-suited for modeling diversity. Holder-Holder al. (2020) proposed DaPsd algorithm to train a population diverse polices in reinforcement learning for better exploration.
Semantic embeddings. Our work relies on embedding models to map natural language responses to feature vectors in the semantic space which is related to works on representation learning and semantic embeddings (Kuhn et al. 2023: Kusupati et al. 2022).
15

---

Published as a conference paper at ICLR 2026
B THEORETICAL RESULTS AND PROOFS
Lemma. Suppose 1b(2, v) V exp( r(ay))
((aw))) Trae (u/z) Φ(v), then the optimal policy defined in (4)
satisfies Thi (V1sk())))) det (L (11:k)) when a 3.
Proof.
Let E Raxk have columns
((b)). The Gram matrix
L L=
Now suppose scale each column ((vs) by factor ai, and denote
A diag(ai. ..... ak), ='= BA.
Then the new Gram matrix is
=' ='ABB''' A(B B)A ALA.
Taking determinants,
det(L') (eL(A)A) = (A) det(L) det(A) (det(A))) det
Since (et(A) I ai obtain
det(
II
ai
det
1=1
Recall that Tii (2) is defined as when = B,
1
Tei (21:)00)) x Tref (11:)< exp
Σ rxx, vi)
+ log det(L
a
Tref (B1skl))))) exp
Σ r(x, y.)
det(Lo(v1:k)
r(x, Wi)
)) del(Lo(yiis))
II Cml
(ner(Wilz)) exp
The second equality holds because Vi sampled independently. Combined with the result above, we have d (y1.k (r) x det(L (B11k))
Analysis det (()::k)) and (L) + Ix). Maximizing det(T.) is equivalent to max- imizing the volume of the parallelepiped spanned by the selected feature vectors, which enforces strict linear independence: any subset that induces a singular receives zero score. In contrast, maximizing de + I) introduces a lidge-like regularization. Indeed, if L BB" for feature matrix B E Rixd
we have
det(T [L + 1) = det (BB + 7) = dT + BB).
This is precisely the determinant of regularized seatter matrix, analogous to the role of TBB++DI in ridge regression. From this viewpoint, adding stabilizes the objective by preventing collapse along directions of near-linear dependence and avoiding the degeneracy of zero determinants.
A complementary interpretation arises from Bayesian linear models and Gaussian processes. In Bayesian linear regression with Gaussian prior w ~ AN(0, /) and unit-variance observation noise,
the marginal likelihood normalization involves det + B B) 2. Similarly, in Gaussian process regression, the log marginal likelihood includes log de + I²I), with o² corresponding to the noise variance. Setting ²² 1 recovers the det(L objective. Hence, det(L I) can be viewed as the determinant under a model with prior noise floor, which softens the diversity requirement and balances between variance explained by the selected items and baseline level of uncertainty.
16

---

Published as a conference paper at ICLR 2026
Eigenvalue Interlacing Theorem (Fisk, 2005).
Suppose A Rex* is symmetric. Let B R xm with m < n2 principal submatrix (obtained by deleting both i-th row and i-th column for some values of t). Suppose A has eigenvalues d .... S An and B has eigenvalues 31 S .... S 8m. Then,
A S Bx I Aktn-m, for l = ,,.,,,
And if m = n -. one has,
X S 31 S A2 S 32 S Bn-1 S An
Proof. We use the Courant-Fischer min-max theorem. For symmetric matrix A E IR with eigenvalues d ... I An the k-th eigenvalue can be characterized as
Ar
A
min SCR dim (S)=
max #ES
x
Similarly, for the principal submatrix B mx with eigenvalues BI ...
B.. we have
y By
Bx min TCR" dim(T)=k
max NET
yyy
Now observe that B is obtained by restricting A to a coordinate subspace (corresponding to remov- ing rows and columns). Hence any y I can be embedded into R² by padding with zeros. Under this embedding, the Rayleigh quotient is preserved:
By
x Ax x Txx
where is padded with zeros.
1 V
Therefore, the feasible subspaces for B are restrictions of those for A. This leads to the inequalities
d S 8x S Aktn-m,
k = 1,..., m.
In the special case m = = 1, the inequalities expand into the chain
Aι S 31 S A2 S 32 S ... ...S... Bn S Ans
which is exactly the interlacing property.
Lemma. Let's write the eigenvalues of L(yra)) d 2 '1 then have 1 A 2 (ee(L(vne)++)) 2 + d1. And the eigenvalue L(y1:)) is always in [0, k] since the em- bedding vectors are normalized, we have 1 + h 2 det(L(vnaa+(e)) 174 2 1 and log(1 k) 2
det(L(y-1)-ta-)
(at(L(rr)+))) 2 0.
log det(L(y (+1a-1)
Let's write the eigenvalues of L(y-s) as Bk-1 2 ... 2 31. Based on Eigenvalue Interlacing Theorem, have,
Proof.
det(L(N1kk) Ix) = (1+A))
At11 2
11+
det(L(y-i) + 1n-1)
1 3
and,
det(L(yyis) + Ix) L(y(L( + 4+-1)
k-
1 A
(1 + Ax)
<1 Xx
1 1 3
Since L(g is positive semidefinite, it holds A; 2 0, Ví. And we have
== X = tr(((g)1))) =
due to the normalization of the feature vectors. Hence, have k > X \ι 20
17

---

Published as a conference paper at ICLR 2026
Connection to D-Optimal Design We construct controlled toy setup to empirically compare our setting to D-optimal selection strategy against the baseline of uniform sampling. highlighting scenarios where diversity plays critical role in achieving robust performance across varied reward functions
Setup and Notation: Let d = denote the dimensionality of the embedding space, and let {Φ1: .2....,,99 ΦNN C Rd a set of normalized candidate embeddings. We synthetically construct the pool to be imbalanced along coordinate directions: nhe 40: vectors near the r-axis, By = 40: vectors near the y-axis and n2 10: vectors near the z-axis (rare) with 90. Small Gaussian noise x (0, 10 10++1) added prevent rank-deficiency.
To illustrate our hypothesis, we compare two sampling strategies: Uniform: ours by solving the following optimization problem
x for all and
N
max WEAN
log det
Σ
whord + 81
N
S.t.
Σ
= 1, 0 5 wi S
where 5 10-9 ensures numerical stability. The DPP based approach promotes spectral coverage
across all directions. Next, we define four linear reward directions: lig = [1, 0. 0 (high mass),
ky 1,00 (high mass), k- [0, 1] (low mass), Riaa I 11 1, 1] (balanced). The
v3
expected reward under policy w and reward is
N Σ
Exww [ik, [1,,
wy (ki Φi)
We compute the information matrices: E(w) = Σin w. and visualize their eigenvalues. Uniform sampling has low spectral mass in the z-direction. In contrast, our DPP based approach balances mass across all directions. This confirms that log det promotes coverage.
Thus this simulation reveals a failure mode of uniform sampling in imbalanced datasets. D-optimal design reallocates weights toward rare directions, yielding significantly improved performance in worst-case reward scenarios. The log-determinant acts diversity-promoooie surrogate, superior to naive metrics like pairwise distance or cosine dissimilarity.
18

---

Published as a conference paper at ICLR 2026
C EXPERIMENTAL SETUP
Data preparation For GSM8K dataset, we directly use the training and test split. For CNN- dailymail dataset, we take the test split, and select 8, 000 data points as the training set and 024 data points as the test set. For Dolly dataset, there is only one training split of 15, 000 data points. We divided it into two subsets with the ratio of 0.2. For Gen, we use the training split, remove data with repetitive key words, and divided the set into two subsets, each containing 8, 000 and 1, 024 data points respectively.
Training configuration We use GRPO algorithm for GSM8K task and use PPO algorithm for the other tasks. We use Qwen2.5-MATH-II.5 Yangg et al., 2024) as the base model for GSM8K task, and use Llama3.2-1B for the other tasks. For all tasks, use the reward model Syywork/SkwwrkkRRewarddd Reward--V2LLama-322211 (Liu et al., 2025) and embedding model sentence- For baseline algorithms,
directly the hyperparameters ported in their papers. For GRPO-likelihodd we use a = 0.25 and for GRPO-entropy, we use 0.01.
Maasfforeer/alll-MiiiMMM--L
Table 3: Training configurations. For max prompt and response length, we use different values for different datasets. From left right, it corresponds GSM8K, CNN-dailymail, Dolly, Gen.
training batch size
128
training epoches
actor learning rate
le-6
critic learning rate
le-5
rollout temperature
1.0
max prompt length
256, 1536, 1024, 64
max response length
256, 256, 512, 128
19

---

Published as a conference paper at ICLR 2026
D ADDITIONAL EXPERIMENTS
D.1 CITY RECOMMENDATION
The following table shows the numerical results that were used to plot Figure For each model, we sampled 100 times and countered the times of each city being recommended.
Table 4: The frequencies of each city being recommended for models trained with different methods.
Method
City (Frequency)
GRPO
Tokyo (97); New York (3)
DQO-pairwise distance
New Orleans (48); Asheville (37); Budapest (8); Barcelona (7)
Budapest (45); Chiang Mai (22); New Orleans (19); Hanoi (7);
DQO-determinant
Krakow (1); Kanazawa (1); Ottawa (I); Nashville (1);
Tokyo (1); Bangkok (1); Singapore (1)
D.2 ADDITIONAL ABLATION STUDIES
Embedding models. We did ablations across different embedding models. The results are shown in T For the hyperparameters, we use = 1.0 and k 4. From the ablations results, it is evident that our proposed method DQO robust to different embedding models.
Table 5: Comparison of different embedding models across multiple tasks. Diversity metrics are calculated across 10 generated responses per prompt.
Diversity f
Quality 1
Embedding Task
distinct-1
distinct-4
self-bleu
self-rouge
pass 1
Q10
Dolly
all-MiniLMM-L6-v2
0.28
0.69
0.46
0.54
5.92
8.74
all-mpnet-base-v2
0.29
0.72
0.47
0.55
5.75
8.57
gle-small
0.30
0.68
0.47
0.53
5.89
8.50
GSM8K
all-MiniLM-L6-v2
0.10
0.42
0.14
0.31
76.3
91.2
all-mpnet-base-v2
0.10
0.39
0.13
0.28
76.5
90.9
bge-base-en-vl.55
0.09
0.39
0.14
0.28
77.4
90.4
CNN
all-MiniLM--6-v22
0.25
0.75
0.41
0.61
11.08
13.02
all-mppettbass-v2
0.26
0.77
0.43
0.62
10.39
12.70
gte-small
0.25
0.74
0.41
0.60
10.55
12.89
Gen
all-MiniLM-L6--22
0.45
0.96
0.77
0.74
10.44
12.36
gte-small
0.45
0.96
0.78
0.73
10.48
12.31
c5-base-v2
0.43
0.97
0.80
0.76
10.32
12.26
Value and k: T in the main page, we show the ablation results of a and on Dolly task. Here, T the results on GSM8K task.
Kernel function. DQO formulates the diversity as the determinant of the kernel matrix. By default, we use the dot product kernel function, i.c., Φ(y1, V2) (41, y2). We study the effect of different kernel functions DOO. We implemented same experiments using Gaussian kemel
20

---

Published as a conference paper at ICLR 2026
Table The quality and diversity scores of the model trained with different values of hyperparam- eters k and a the GSM8K task. Diversity metrics are calculated across 10 generated responses per prompt.
Diversity 1
Quality
Method
distinct-1
distinct-4
self-bleu
self-rouge
pass
pass 10
GRPO
0.09
0.32
0.09
0.21
76.8
87.9
0.5,
0.09
0.33
0.09
0.22
74.6
89,2
1.0. k 4
0.10
0.42
0.14
0.31
76.3
91.2
1.5,
0.11
0.48
0.19
0.34
76.1
92.6
2.0, k 4
0.13
0.54
0.21
0.40
76.7
92.7
5.0, k
0.16
0.62
0.28
0.44
77.7
93.3
= 2, 1.0
0.11
0.40
0.13
0.26
73.9
90.3
k 4, 1.0
0.10
0.42
0.14
0.31
76.3
91.2
6, a = 1.0
0.10
0.44
0.17
0.33
76.5
92.1
8, a 1.0
0.11
0.47
0.16
0.32
74.9
90.8
function, i.e., Φ(y1, 122 kernel functions on Dolly and GSM8K tasks. DQO demonstrates robust performance under different kernel functions.
exp(
Iive v
Table 7
shows the performance of DQO with different
Regularization by introducing identity matrix. To solve the numerical explosion issue, in- troduce an identity matrix when calculating the determinant: det ((yh: + 744)). ). It can shown adding identity matrix plays role as regularization. By default, simply y We test DQO with different values of y. The results are summarized in Table g and Table
Table 7: The quality and diversity scores the model trained with different kernel functions on Dolly and GSM8K tasks, Diversity metries are calculated across 10 generated responses per prompt.
Diversity f
Quality 1
Method
distinct-1
distinct-4
self-bleu
self-rouge
pass@1
pass@10 10
Dolly
1.0, gaussian
0.29
0.72
0.48
0.56
6.45
8.61
a 1.0, dot product
0.28
0.69
0.46
0.54
6.56
8.74
a 2.0, gaussian
0.34
0.79
0.54
0.61
6.12
8.64
2.0, dot product
0.35
0.82
0.54
0.64
6.41
8.69
GSM8K
a 1.0, gaussian
0.10
0.43
0.16
0.31
77.1
90.9
a = 1.0, dot product
0.10
0.42
0.14
0.31
76.3
91.2
2.0, gaussian
0.11
0.48
0.19
0.36
75.2
91.2
a 2.0, dot product
0.13
0.54
0.21
0.40
76.7
92.7
We from Table when 0.1, the diversity in responses surges while the quality collapses. This is consistent with our analysis. The identity matrix can be regarded as a regularization term. The objective will prioritize diversity more if is low. In addition, when a 0.5, can decreasing 7 does not affect the quality much as in the case where a 1.0. This is because also controls the balance between quality and diversity. When a is low, the diversity is less important in the objective, hence, the effect of decreasing is diluted. The phenomenon is consistent on GSM8K task.
21

---

Published as a conference paper at ICLR 2026
Table 8:
The performance of DQO with different identity matrix weights on Dolly task. Diversity metrics are calculated across 10 generated responses per prompt.
Diversity
Quality
Method
distinct-1
distinct-4
self-bleu
self-rouge
pass
pass@10
a = 1.0, +0.17
0.57
0.96
0.79
0.86
3.44
6.38
1.0, +0.57
0.37
0.83
0.56
0.66
6.72
8.90
a 1.0, +/
0.28
0.69
0.46
0.54
6.56
8.74
a 0.5, +0.17
0.34
0.84
0.58
0.68
6.04
8.75
a 0.5, +0.57
0.28
0.71
0,44
0.54
6.31
8.72
0.5, +1
0.28
0.69
0.43
0.53
6.47
8.77
Table 9: The performance of DQO with different identity matrix weights GSM8K task. Diversity metrics are calculated across 10 generated responses per prompt.
Diversity
Quality
Method
distinct-1
distinct-4
self-bleu
self-rouge
pass 1
pass@ 10
a 1.0, +0.17
0.23
0.79
0.41
0.52
73.3
93.9
1.0, +0.5/
0.11
0.49
0.19
0.36
77.5
92.0
a = 1.0, +I
0.10
0.42
0.14
0.31
76.3
91.2
a 2.0, +0.17
0.38
0.88
0.58
0.74
63.5
91.3
2.0, +0.57
0.15
0.59
0.24
0.42
78.0
93.2
a 2.0, +1
0.13
0.54
0.21
0.40
76.7
92.7
E COMPUTATIONAL COST ANALYSIS
Compared to quality-only post-training, DQO introduces additional diversity term, which re- quires computing diversity scores during training. In this section, we show that the resulting com- putational overhead is negligible in practice.
DQO computes the determinant of the similarity matrix, whose computational complexity is O1.3) where k is the number of answers generated per prompt. However, since k is generally small (< 10), the total number of floating- point operations is on the order of few thousand which corresponds to only microseconds of runtime. And our ablations further support the claim. The effect of increasing enough to choose a small value k,
the value of a is similar to increasing k; so practically which makes the calculation of determinant negligible.
The primary additional cost in DQO arises from using the embedding model to map the response to feature vectors. But since embedding models are non-autoregressive and relatively with small sizes, its computational overhead is modest.
We report the conerete results of the average time taken per training step of DQO compared with the baseline in Table [0ble|00bele below. As shown, the additional computational cost introduced by DQO is negligible relative GRPO.
Table 10: Training time per optimization step (seconds per step) for GRPO and DQO across different tasks.
Task
GRPO (s/step)
DQO (s/step)
Reasoning
36.844
36.216
Summarization
34.826
34.463
Instruction folloooiinn
40.962
41.485
Creative-
20.963
22.462
22

---

Published as a conference paper at ICLR 2026
F LLM-AS-A-JUDGH FOR DIVERSITY EVALUATION
Evaluating diversity in generated responses is inherently challenging. While human evaluation pro- vides the most reliable assessment, it is expensive and difficult scale. As practical alternative, we adopt LLM-as-a-judge as a surrogate for human judgment.
To validate that LLM-as-a-judge can perform reliable evaluation, we conducted small-scale human study comparing the diversity of responses generated by DQO and the baseline model. We randomly selected 18 prompts from the test of four benchmarks (five prompts per task, except for sum- marization, where we selected three to keep the evaluation time manageable). For each prompt, we collected four responses from each model.
Participants were asked to assign dense preference score between and indicating which model produced more diverse responses. The order of the two models was randomized prevent any potential bias or information leakage.
The results are presented below: each column corresponds to a prompt, and we report the preference score assigned to DQO. Both the LLM-aas-a-jude and human evaluators consistently rated DQO as producing more diverse outputs. Morcover, the human study results are highly correlated with the LLM-as-a-judge evaluation and agree on the majority of prompts, supporting the reliability of using LLM as a judge for diversity evaluation.
Table 11: Comparison between LLM-as-at LaM-as-a-judge and human evaluation. The score shows the pref-
erence the responses generated by DQO model in terms of diversity. Each column prompt sampled from the test dataset.
2
11
13
14
15
16
17
18
LLM-assa-judge 0.70 0.80 0.70 0.80 0.55
0.40
0.60
0.60
0.50
0.80
0.90
0.70
0.80
0.60
0.60
0.75
0.70
0.60
Human evaluation 0.65 0.74 0.79
0.67 0.62
0.51
0.65
0.61
0.88
0.69
0.59
0.62
23

---

Published as a conference paper at ICLR 2026
G REWARD HACKING WITH OUTCOME REWARD
We observed an interesting phenomenon when doing experiments on GSM8K task. With only out- come reward provided, the model trained with the quality- -iversyddiversity objective learns first generate a correct answer to the question thus securing the outcome reward, then generate some random or irrelevant contents artificially increase diversity. See the concrete example below.
The ssward-hackin phenomenon does not exist when we use reward models, since the reward model evaluates the quality of the entire response rather than just the final answer.
We explored a potential solution by computing the cosine similarity between the embeddings of the prompt and the response and multiplying this score with the diversity term. The intuition is that only responses semantically aligned with the prompt contribute to the diversity objective. However, we observed that the model can still exploit this mechanism by generating correct answers followed by random contents but superficially related to the prompt. See the concrete example below.
We believe nutcome-based reward alone is inherently susceptible to reward hacking. The outcome reward is widely used in reasoning tasks due to its simplicity and effectiveness, however, when combined with other objectives, it is vulnerable to reward hacking. This phenomenon presents an intriguing challenge for future study.
An example of reward hacking with outcome reward
Prompt: Gretchen has some coins. There are 30 more gold coins than silver coins. If she had 70 gold coins, how many coins did Gretchen have total? Let's think step by step and output the final answer after "####".
Response: If Gretchen had 70 gold coins and there are 30 more gold coins than silver coins, then the number of silver coins she had is
70 30 40 silver coins.
The total number of coins Gretchen has is the sum of gold and silver coins, so is
+ 40 110 coins.
#HIH 110
The correct answer is: 110
Which digit is first to appear in the units place of
5 + 52 + 5 + 2004 + 2005 ?
(A)
(B)
(C) 2
(D) 5
To determine which digit is first to appear in the units place of the sum
2004 5 + 52 53 52004 + 2005
we need to analyze the units digits of the powers of 5. First, let's observe the units digits of the first few powers of 5:
24

---

Published as a conference paper at ICLR 2026
example of reward hacking with prompt-response similarity
Bobby takes 30 min lunch and two 15 minutes break per day the office. After 5 days, how many hours do his lunches and breaks add up to? Let's think step by step and output the final answer after "####".
Prompt:
Response:
Bobby takes 30 minute lunch and 15 minute break twice day. So in total, Bobby takes 30 + 15 60 minutes lunch and breaks per day.
After 5 days, Bobby takes total of
60 300 minutes of lunch and breaks.
Since there are 60 minutes in an hour, Bobby's s lunches and breaks add up
300
5 hours.
60
####
Bobby's lunches and breaks add up to 5 hours.
Bobby's dog is diet. For every meal he eats, he needs to walk him for 2 minutes. When he eats twice, he walks him times that amount. How many minutes did Bobby walk his dog if he let him eat 7 meals? Let's think step by step and output the final answer after "#HIH""
Bobby's dog needs to walk 2 minutes for every meal he eats.
2 14 minutes.
####
25

---

Published as a conference paper at ICLR 2026
H PROMPTS
We listed the prompts we used in this work.
Prompt for city recommendation
Prompt= "Recommend city visit along with reason. Respond ONLY with valid JSON with the following schema. Do not include explanations or extra text.
"city": "<city name>"" "reason": "<reason>"
Prompt for GSM8K
Prompt = "(QUESTION) Please reason step by step, and put your final answer within []xoed
Prompt for CNN-dailymail
Prompt
f'Summarize the following article: (ARTICLE)
Summary:"
Prompt for Dolly
prompt f'### Instruction:
(INSTRUCTION)
### Input:
(INPUT)
### Response:"
Prompt for Common-gen
Prompt f'Write short story of 3-4 sentences that includes the following keywords: (KEY WORD]. The story should be coherent, creative, and naturally incorporate all the keywords. Story:"
Prompt of LLM as a diversity judge
Instructions=""hhe following are two files, each records responses generated by model. The "prompt' field stores prompts, and the 'response' field stores the corresponding groups of generated responses for the prompts. Please judge two models in terms of the diversity of their generated responses. You should give each model score between [0.1]." Input= F'Files: model 1: (FILE); model 2: (FILE)."
26

---

Published as a conference paper at ICLR 2026
I LLM AS A DIVERSITY JUDGE
We use advanced LLMs (GPT-40-mini) to score the diversity of the responses generated by the models. The following are the evaluations given by GPT-44-mini. For fair comparison, we did two comparisons by changing the order of two models and taking the average scores.
Diversity judged by gpt-4o-ming mini GSM8K; Model (GRPO), Model (DQO)
To evaluate the diversity of responses generated by the two models, can analyze the patterns seen in the "responses" field of each model. Model 1:
Characteristics:
The majority of responses are quite repetitive, consistently using phrases like "To solve this problem, we'll break down" or variations thereof.
There appears to be a lack varied strategies or unique approaches in the generated answers, with many instances simply restating the problem-solving process in a similar structure.
Diversity Score: Given the high level of similarity and repeated phrasing, I would rate the diversity of responses for Model as 0.2/1.
Model 2:
Characteristics:
This model exhibits a somewhat greater variety in response structure, with different steps highlighted in the problem-solving process.
Phrases like "Step X: Determine the..." " and "Let's denote the total number..." introduce variation, and there seems to be a more thoughtful approach to sev- eral responses.
While there are still some repeated forms, the presence distinct steps and strategies indicates broader array of response possibilities.
Diversity Score: The variety and different methodologies present in the responses
lead me to rate Model 2's diversity as 0.5/1.
Summary:
Model Score: 0.2/1
Model Score: 0.5/1
Model demonstrates better response diversity compared to Model 1, which relies heavily on similar language and structure.
27

---

Published as a conference paper at ICLR 2026
Diversity judged by gpt-4o-mini on CNN-dailymail; Model (PPO), Model 2 (DQO)
To assess the diversity of the responses generated by the two models, we evaluate the variety and uniqueness of the content provided in the responses to identical prompts. Model
The responses from Model I vary in terms of length and detail, but many responses are quite structured and similar in approach.
There some variation in naming conventions and specific details, but overall, a noticeable pattern exists in how the information is expressed.
Many responses use similar phrasing, leading to less diversity overall.
Model 2:
Responses from Model exhibit greater variation phrasing, structure, and detail.
The content shows a wide range of descriptions and perspectives, which enhances the uniqueness of each response.
There is noticeable effort to vary not just the wording but also the structure of each response, leading to a more diverse set of outputs.
Scores: Based on this analysis, we assign the following scores for diversity in generated responses:
Model 1: 0.4
(Limited diversity, responses similar in structure and phrasing)
Model 2: 0.8
(High diversity, significant variation in responses)
These scores reflect the extent variety present the responses generated by each model.
28

---

Published as a conference paper at ICLR 2026
Diversity judged by gpt-4o-mini on Dolly; Model 1 (PPO), Model (DQO)
To assess the diversity of responses generated by both models, we analyze the responses across various prompts, looking for variances in creativity, structure, and content diversity. Model Analysis:
The responses from Model often maintain similar structure, primarily focusing on straightforward factual information, lists, or direct answers to prompts.
The responses appear to have consistent formatting, providing succinet and clear answers, but can feel repetitive and lack creative exploration.
While some responses do reflect variety of information (e.g., multiple examples, details), the overall tone and presentation remain quite uniform.
Model 2 Analysis:
Model demonstrates greater variety of response styles, incorporating markdown-like formatting, lists, and varying degrees of detail and structure across outputs. This format enhances readability and introduces an engaging style.
The responses in Model 2 often include additional editorial commentary creativ- ity (e.g.. humorous tones in astrology interpretations, narrative elements product recommendations).
The use of diverse sentence structures and clear formatting contributes to sense dynamism in responses.
Scoring:
Model 1: 0.4 (due to consistent structure and less creative diversity in responses)
Model 2: 0.8 (due varied response formats, creative styles, and more engaging outputs)
Final Scores:
Model 1: 0.4
Model 2: 0.8
29

---

Published as a conference paper at ICLR 2026
Diversity judged by gpt-4o-mini on Common-gen; Model (DQO), Model (PPO)
To evaluate the diversity of the generated responses from both models, we analyze the con- tent and variety of the responses provided for the same prompts. Model
The responses feature varied character names and professions, indicating range of different story premises.
While the structure is somewhat consistent (retired individuals reflecting on their past), each response introduces new character with unique backstories or contexts.
The diversity is primarily in the character names and their respective professions, leading to different narrative angles.
Model
The responses are more repetitive in structure; most of them start similarly with
"An older woman pauses..."
There a notable lack of variation terms of settings and character actions across the responses. Although the overall premise and sentence structure remain largely the same.
details differ (such as the specific verb object),
This limits the diversity of storytelling comparison to Model
Scores:
Model 1: 0.8 Scores high for its unique character introductions and storytelling approaches, exhibiting good diversity.
Model 2: 0.4 Scores lower due to the repetitive structure and similarity in re- sponses, which significantly reduces the diversity in storytelling.
Summary: Model 1 demonstrates a greater range of ideas and creativity in responses, while Model lacks variety, leading to a more uniform storytelling style.
30


Reviews:
[{"review_id": "d5gLdrUW56", "reviewer": "Reviewer_J8Sx", "summary": "This paper introduces a post-training method called DQO (Diversity Quality Optimization) that is designed to jointly optimize LLMs for quality and semantic diversity. Using Determinantal Point Processes, the authors gather the embeddings of multiple responses from each prompt and measure diversity as the volume spanned by these embeddings by taking the determinant of their pairwise similarity matrix. This diversity measure is added to the objective function for RL post-training. DQO maintains a higher quality-diversity Pareto frontier, matches the baseline for pass@1 and scales better than the baseline for pass@n >1.", "questions": "Q1. Can you explain how equation 4 and 5 follow from 3?\n\nQ2. Can you get more results with different embedding or reward models? (embeddings models most importantly).\n\nQ3. Can you provide a description and analysis of the computational cost of the DPPs", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "- Well written \n\n- Clear presentation \n\n- Very good motivation and an effective solution to sustaining diversity \n\n- The determinantal point process is a explained clearly and can have a lot of potential in other pipelines that use frozen embeddings", "weaknesses": "## Minor \n\n- The intro reads more like related works. Could be shaped to flow better. \n\n- Incredibly small fonts in the axis labels\n\n- Missing citation semantic entropy [1]\n\n## Major \n\n- **Matrix via similarity Kernel Bottleneck**: There is a k by k matrix to find the determinant of. Is it cubic complexity? Is there a way to mitigate that? How much more is the overhead? A computational cost comparison is pertinent.\n\n- **Reward Model reliance**: The paper is introduced as a general post-training solution but it still requires a good frozen embedding model and a good reward model. Can this method even be reasonably applied to the vast space of situations where those are not available? \n\n- **Embedding Model**: It is necessary to see that the results hold for different embeddings models and embedding approaches. For example something like the Matryoshka representations [2] would be really helpful in supporting the generalizability of this method.\n\n\n[1] Kuhn, Lorenz, Yarin Gal, and Sebastian Farquhar. \"Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation.\" arXiv preprint arXiv:2302.09664 (2023).\n\n[2] Kusupati, Aditya, et al. \"Matryoshka representation learning.\" Advances in Neural Information Processing Systems 35 (2022): 30233-30249.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cNQT9D1DLV", "reviewer": "Reviewer_Fy8p", "summary": "In this paper, the authors have proposed proposes DQO (diversity quality optimization), a training method that uses DPPs to encourage llms to generate semantically diverse responses while maintaining quality. They do this by defining diversity as the determinant of a kernel matrix constructed from response embeddings. This diversity measure can be incorporated into standard RL objectives like PPO or GRPO as a simple regularizer term in the loss function.", "questions": "1. What happens with 7B or 13B models? Does the diversity-quality tradeoff change? \n2. How expensive is computing k embeddings and determinants per update?\n3. Can you run a human study (even small-scale) where annotators judge semantic diversity? This is essential for validating the core claim of this paper.\n4. How sensitive is DQO to the underlying embedding model?\n5. Does improved diversity during training lead to better out-of-distribution generalization or just different in-distribution behavior? Maybe choose an OOD summary task for this.", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written with good intuitive explanations.\n2. This paper addresses an important problem in RL-finetuning research, i.e. diversity collapse. If the limitations (see below) were addressed, this could be useful for the community.", "weaknesses": "1. I strongly feel that the reward hacking failure with outcome rewards (in appendix E) should be on the main submission. This isn't a minor edge case. Such outcome rewards are standard for mathematical reasoning or coding tasks. Given that the attempted fix using prompt-response similarity also failed, this issue severely limits the method's applicability and should be acknowledged in the main experiments.\n2. Claims about \"semantic diversity\" are never validated with human judges, and even for LLM-as-judge there's only one model used. I feel that the authors should take a subset of generations and atleast show some correlation between human responses and their LLM-as-judge.\n3. Figure 4 shows pass@n curves but the differences are often small and lack error bars. Also (minor), please increase the text size for fig 4 and 5. \n4. What happens when you don't set β=α? The derivation in Eq. 5 requires this, but it's a strong constraint. This constraint means you can't independently tune:\na. How much you penalize deviation from the reference policy (β)\nb. How much you reward diversity (α)\nIf β controls exploration vs. exploitation in the standard RL sense, and α controls quality vs. diversity tradeoff, being forced to set them equal is quite restrictive. In practice, you might want β=0.01 for conservative updates but α=1.0 for strong diversity. So why was such a constraint added?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Xq2F6lEny5", "reviewer": "Reviewer_BeWa", "summary": "This paper addresses output diversity degradation in large language models (LLMs) following post-training with reinforcement learning methods like PPO and GRPO. The authors propose Diversity Quality Optimization (DQO), a novel training framework that jointly optimizes for response quality and semantic diversity. DQO introduces a Determinantal Point Process (DPP)-based diversity objective in the RL training loss. For each prompt, the method samples a set of responses, maps them into a semantic embedding space using a pretrained encoder, and computes a kernel-based similarity matrix. The diversity score corresponds to the determinant of this matrix. To address computational challenges, the authors introduce two key modifications: (i) adding an identity matrix for regularization, and (ii) employing leave-one-out (LOO) gradient estimators to reduce variance. The authors evaluate DQO across diverse tasks, including instruction-following, summarization, story writing and reasoning.", "questions": "- How is the quality score evaluated for generation tasks ssuch as Dolly, summarization and story-writing?\n- What is the k value in the main experiment (Table 1)?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper addresses an important and timely problem in LLM post-training, where training with reinforcement learning often reduces the diversity of generated output.\n- The proposed method can be easily integrated with existing RL algorithms, showing potential for broad applicability in post-training methods.\n- The experimental evaluation covers multiple task categories and shows consistent improvements in diversity metrics while maintaining comparable quality.", "weaknesses": "While the paper is clearly written and the method is well-justified, it could be further strengthened through additional experimental analyses.\n\n- As acknowledged by the authors, the proposed approach relies heavily on the quality of the pretrained embedding model used to compute semantic similarity. Evaluating DQO with different embedding encoders would strengthen the empirical analysis and clarify its generalizability. How does performance vary with different embedding models (e.g., larger models, task-specific embeddings)? Are certain embedding models more suitable for particular task types?\n- While the LOO estimator is introduced to mitigate variance, the paper lacks ablation study to isolate its effect on performance or stability.\n- Experiments focus only 2 models (Qwen2.5-MATH-1.5B for reasoning, Llama3.2-1B for other tasks). No exploration of different model families. It's unclear whether findings generalize to larger models. How does model capacity affect the quality-diversity trade-off?\n- The paper fixes the reward model (Skywork-Reward-V2) across all tasks. How sensitive is DQO to reward model quality or choice?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5w7v1AW5dL", "reviewer": "Reviewer_RQPB", "summary": "This paper introduces DQO, a principled framework for post-training LLMs to optimize both diversity and quality of responses. Key contributions include: a flexible method compatible with existing RL algorithms like PPO and GRPO; a DPP-based formulation that ensures semantic diversity beyond lexical variation; and experimental demonstration of an effective quality-diversity trade-off across multiple tasks while maintaining utility and coherence.", "questions": "See questions in weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper proposes a post-training reinforcement learning method from the perspective of diversity in content generated by LLMs, which simultaneously ensures generation quality and diverse semantic capabilities. Its research perspective and solution approach are innovative.\n\n2. The paper presents a clear dual objective function for optimizing LLMs to ensure both quality and diversity, while also addressing two challenging issues in optimizing this function and providing solutions. The overall methodological approach is well-structured.\n\n3. The paper covers a comprehensive range of experimental perspectives, including performance in two dimensions of generation, the impact of increasing diversity on performance, and performance variations under different task settings, conducting thorough experiments.", "weaknesses": "1. The paper explains in the Introduction how existing reinforcement learning methods improve the diversity of generation at the lexical level but lacks an analysis comparing them with methods like instruction tuning. Instruction tuning uses the same training data template to enable LLMs to learn different tasks, naturally enhancing the model's ability to generate diverse downstream tasks or stylistic language. Since this paper primarily discusses the issue of generation diversity, although it focuses mainly on reinforcement learning, I believe it is still necessary to explore the necessity or fundamental differences compared to methods like instruction tuning.\n\n2. The paper proposes a training method based on determinantal point processes (DPPs) to enhance the generation quality and diversity of LLMs but lacks motivation for adopting DPPs. DPPs have been used in recommendation systems to simultaneously ensure the relevance and diversity of recommended results. Although the authors provide a definition of DPPs in Section 2.3, I still believe the authors need to more clearly explain the principle of how DPPs can improve both aspects simultaneously and the motivation for using them to train LLMs in the Introduction.\n\n3. Some of the figures in the paper lack necessary data labels, dimension identifiers, and summary titles (e.g., Figure 3), requiring readers to read extensive text to understand the content of the figures. Some tables (e.g., Table 1 and Table 2) also lack necessary highlights to allow readers to quickly draw performance conclusions.\n\n4. The paper draws conclusions by comparing the DQO method with baseline methods on general language model tasks, showing that DQO can simultaneously ensure performance in both quality and diversity. However, it lacks a detailed analysis of the differences between the methods. For instance, why do GRPO-likelihood (on the GSM8K task) and GRPO-entropy (on the Dolly task) still perform better in diversity than DQO? Since DPO incorporates considerations for diversity compared to the baseline, why is the improvement in diversity performance not significant for DPO, while there is an improvement in generation quality?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)