Evaluating the Diversity and Quality of LLM Generated Content
Authors: Alexander Shypula
Shuo Li
Botong Zhang
Vishakh Padmakumar
Kayo Yin
Osbert Bastani
Venue: COLM 2025 (2025)
Keywords: Diversity;Alignment;LLMs;Evaluation;Program Synthesis;Code Generation;Creative Writing
TL;DR: We introduce a methodology/dataset for evaluating the diversity and quality of open-ended LLM generated content. We find RLHF and more broadly preference-tuning meaningfully increase diversity of generations.

Abstract: Recent work suggests that preference-tuning techniques—such as Reinforcement Learning from Human Feedback (RLHF) methods like PPO and GRPO, as well as alternatives like DPO—reduce diversity, creating a dilemma given that these models are widely deployed in applications requiring varied outputs. We argue that diversity without consideration of quality has limited practical value. To address this issue, we introduce a framework for measuring effective semantic diversity—diversity among outputs that meet quality thresholds—which better reflects the practical utility of large language models (LLMs). Using open-ended tasks that require no human intervention, we find counterintuitive results: when using diversity metrics that do not explicitly consider quality, preference-tuned models—particularly those trained via RL—often produce outputs with lower diversity; however, these same preference-tuned models generate greater effective semantic diversity than supervised fine-tuned (SFT) or base models. Our analysis further shows another trend: while larger models may exhibit greater effective semantic diversity than smaller models, the smaller models are consistently more parameter-efficient at producing unique content within a fixed sampling budget. These findings have practical implications for applications that require diverse yet high-quality outputs, from creative assistance to synthetic data generation.

Published as a conference paper at COLM 2025
Evaluating the Diversity and Quality of LLM Generated Content
Alexander Shypula' Shuo Li"
Botong Zhang'
Vishakh Padmakumar
Kayo Yin Osbert Bastani shypula@seas.upen..edu
University of Pennsylvania, New York University, UC Berkeley
Abstract
Recent work suggests that preference-tuning techniques- as Rein- forcement Learning from Human Preferences (RLHF) methods like PPO and GRPO, as well as alternatives like DPO-reduce diversity, creating dilemma given that these models are widely deployed in applications requiring varied outputs. We argue that diversity without consideration of quality has limited practical value. To address this issue, we introduce a framework for measuring effective semantic divverityy-ivversity among outputs that meet quality thresholds- better reflects the practical utility of large language models (LLMss). Using open-ended tasks that re- quire no human intervention, we find counterintuitive results: when using diversity metrics that do not explicitly consider quality, preference-ture eunded moddes--patticultti those trained via RL-often produce outputs with lower diversity; however, these same preference-tuned models generate greater effective semantic diversity than SFT or base models. Our analy- sis further shows another trend: while larger models may exhibit greater effective semantic diversity than smaller models, the smaller models are consistently more parameter-efficient at producing unique content within a fixed sampling budget. These findings have practical implications for applications that require diverse yet high-quality outputs, from creative assistance to synthetic data generation.
Introduction
As large language models (LLMs) increasingly serve tools for ideation, synthetic data generation, and creative assistance, their ability to produce diverse yet high-qualiiy outputs has become critically important. While recent advances have dramatically improved model performance, relatively little attention has been paid to systematically measuring and optimizing for the dual objective of quality and diversity.
Consider user asking an LLM to generate story premises or researcher using an LLM to create synthetic training data. In these scenarios, the utility of the model depends not only on producing coherent set of responses but also on generating outputs that span a meaningful range of ideas or examples. This contrasts with traditional LLM evaluation paradigms, which often optimize for a single correct answer.
The challenge lies in defining what constitutes meaningful diversity. Diversity without tokens are maximally diverse but utterly useless. What
quality is trivial to q instead diversity among outputs that meet threshold quality or acceptability. We refer to this as effective semantic diversity and argue that it is more accurate measure of an LLM's practical utility in open-ended generation tasks.
This distinction is particularly relevant in today's landscape of post-training LLMs, where preference-tuned
sodelssttraing via methods such as Direct Preference Optimization (DPO), Proximal Policy Optimization (PPO), and Group Relative Policy Optimization (GRPO) increasingly the standard. These methods have been highly successful at aligning models with human preferences, yet their impact on output diversity remains

---

Published as a conference paper at COLM 2025
Comparison of Key Metrics Across Models
Effective Semantic Diversity
(Ours)
Validity (Quality)
Neural (Cosine) Diversity
50
Effectiv Sertaralc Deveraiy
60
3.-
(Quailey)
30
(Dur))
(Cosine)
20
30-
20
20
10
100
8.00
0.25
0.50
0.75
1.00
1.25
8.00
0.25
0.50
0.75
1.00
1.25
8.50
0.25
0.50
0.75
1.00
1.25
Temperature
Temperature
Temperature
www............................. www.ee..........................
Llama 3.1 70B Instruct
Liama 8B Instruct
Liama 3.1 Base
Figure 1: Diversity and Quality metrics our open-ended programming dataset when modulating the Temperature parameter across different models. We use the CODE- BERTSCORE (Zhou et al., 2023) model for neural cosine diversity.
disputed. The dominant research narrative is that preference tuning harms diversity, and many open questions persist, including the effect of model size and the ability to generate unique content.
We introduce a principled framework for measuring high-quality diversity that:
Requires no human evaluation at inference time
Accounts for the fundamental quality-diversity interplay
Enables meaningful comparison across model families and training techniques
In Figure we illustrate our implementation of effective semantic diversity for three LLAMA- 3.1 family models under varying temperature settings. Using neural cosine diversity alone, the base LLM appears most diverse, despite producing an extremely low proportion of valid (high-quality) generations. Our effective semantic diversity metric penalizes both excessively low and excessively high temperatures, as well as models that struggle to generate coherent content.
With this framework, our experiments reveal insights into preference-tuning and the diversity-quality trade-off. Using diversity metrics that do not explicitly consider quality, preference tuning may indeed reduce diversity relative to supervised fine-tuning (SFT). However, in both code generation and creative writing, we find that preference-tunee models exhibit greater effective semantic diversity. A deeper analysis in code generation shows that when restricting to high-quality generations only, preference-tuned models pro- duce less semantically diverse content within that subset. Nonetheless, preference tuning increases the proportion of high-quality outputs to such an extent that this gain outweighs the reduction in diversity per high-quality sample.
In code generation, preference semiinn-sppecilllly reinforcement learning (RL)-is associated with reduced lexical and syntactic diversity but no loss in semantic diversity. For open- ended creative writing, preference tuning is linked to greater diversity in lexical patterns. Finally, when evaluating parameter efficiency for generating unique programs within fixed sampling budget-for example, when creating unique synthetic data-we find that smaller models, down to around 500 million parameters, are often the most efficient choice.
Background and Related Work
LLM Alignment with Preference Tuning. As LMs (Bengio et al., 2000; Radford et al., 2019) have become more powerful, work has been done to improve their instruction- following ability and to mitigate the likelihood of generating undesirable content. RLHF with PPO (Schulman et al., 2017; Ziegler et al., 2019; Ouyang et al., 2022) has emerged as highly effective technique for aligning L.LSs with human preferences. Subsequently,
2

---

Published as a conference paper at COLM 2025
numerous alternative methods to PPO have been proposed, such as DPO (Rafailov et al., 2024), Rejection Sampling (Touvron et al., 2023), and GRPO (Shao et al., 2024). In numerous works, it has been reported that over-optimizing for the reward model eventually leads to incoherent or undesirable outputs (Ziegler et al., 2019; Stiennon et al., 2020; Korbak et al., 2022). It is suggested that optimization strategy that places all probability on the highest- reward outcome is optimal for this loss function but will inevitably lead to distribution collapse (Lanchantin et al., 2025). In preference-tuning has been instrumental in mitigating this effect, as it allows the preference-tuned model to retain some of the distributional properties of the base LLM (Korbak et al., 2022). Therefore, in theory, KL-regularized RL should preserve some attributes of diversity present in base models.
LLMs, KL-divergence regularization
Approaches to Measuring Diversity. Due to the high cost of human evaluation, popular methods for automatically measuring diversity typically fall into either lexical or neural approaches. In the earlier days, when LMs often struggled to generate rich content, lexical metrics were commonly used to assess diversity. These metrics generally involve calculating summary statistics over nt-rrmms, such as Distinct-N (Li et al., 2016; Du & Black, 2019) and Self-BLEU (a modified BLEU metric) (Zhu et al., 2018), and remain widely used today (Guo et al., 2024; Shaib et al., 2024). As early foundation models for language representation, such as BERT (Devlin et al., 2019), gained popularity and were adopted for modeling sentence similarity (Zhang et al., 2019; Reimers & Gurevych, 2019) and code similarity (Feng al., 2020; Zhou et al., 2023), they were eventually proposed for measuring diversity in natural language (Lai et al., 2020; Tevet& & Berant, 2021; Stasaski & Hearst, 2022). However, they have not yet been applied to measuring diversity in programming languages.
While lexical and neural diversity metrics have been used to evaluate whether one LLM is more diverse than another (Kirk et al., 2023; Guo et al., 2024), to our knowledge no prior work has assessed whether neural diversity metrics truly capture the diversity of effective semantic content in L.M.-generated outputs. Given that LMs range from small models that often produce incoherent content to large, powerful models with sophisticated capabilities, varying safety attributes, and distinctive "styles," it remains unclear whether existing diversity metrics can robustly reflect meaningful semantic content across such wide range of distributions. Models used for evaluating diversity, such Sentence- BERT (Reimers& & Gurevych, 2019), are typically trained and evaluated on human-authored text; for diversity assessment, however, they are expected to generalize to highly diverse, potentially off-distribution sets of LLM LLM--enerrted text. The closest prior attempt to evaluate lexical and neural diversity metrics is Tevet & Berant (2021), which examined whether neural diversity metrics can capture variation induced by the sampling temperature parameter and diversity in murtnewritten content; in the latter case, neural models were found to underperform human judgment.
Diversity of LLM Content. Given the expense of human evaluation, most insights into the novelty and diversity of LLM outputs are based on linguistic and neural measures of diversity. Zhang et al. (2020) introduce the notion of quality-diversity trade-off in natural language generation when evaluating sampling algorithms such as nucleus sampling (Holtz- man et al., 2019) and temperature sampling. McCoy et al. (2023) investigate whether smaller LMs exhibit linguistic novelty by evaluating combinations of n-grams absent from the training corpus. With the advent of more chat-based LLMs like CHATGPT (Ouyang et al., 2022; OpenÅl et al., 2023), expectations for these models to serve creative assistants and produce diverse responses have risen, but their capabilities have been questioned. For example, the first release of CHATGPT was reported to be incapable of generating diverse jokes (Jentzsch & Kersting, 2023). In more recent study, Kirk et al. (2023) fine-tune 7B ALPACAFARM (Dubois al., 2024) models with SFT and PPO for neural text summarization and find that PPO reduces both lexical and neural diversity in the summaries. Padmakumar & He (2023) show that human-writter essays assisted by an RLHF-tuned model are less diverse than those assisted by a base model when assessed using neural diversity measures. Guo et al. (2024) benchmark the lexical, syntactic, and neural diversity of 7B parameter lan- guage models but do not probe whether preference-tuned models are more or less diverse than SFT or base alternatives. A separate line of work has examined LLMs from a social cognition perspective, investigating whether LLMs reflect the opinions and conceptual
3

---

Published as a conference paper at COLM 2025
associations of macro-level human populations (Santurkar et al., 2023; Murthy et al., 2024). Recent work has also explored the role of diversity metrics in evaluating and improving synthetic data quality for LLMs: Yu et al. (2023) investigate LLMs as attributed training data generators; Divekar & Durrett (2024) propose SYNTHESIZRR, a etrrival-augmmened framework for generating synthetic datasets with greater lexical and semantic diversity; Chen et al. (2024) study the downstream impact of synthetic data diversity on training LLMs; and Miranda et al. (2025) introduce quantitative metric, the diversity coefficient, to measure variability in natural language datasets. While these works find that LLMs may not reflect the diversity of entire human populations, remains unclear how this impacts their ability to generate effectively diverse tiveet--prtticulrlly when faced with competing constraints of quality and helpfulness.
3 Measuring Effective Semantic Diversity
3.1 Problem Formulation
Let D {x denote dataset of prompts, where each prompt x; is designed to elicit a range of possible outputs from a language model. For each prompt X;, we generate K g
outputs P; 8g ..... 8 ~ f(- where/ f denotes the generative distribution of the language model under evaluation.
To evaluate the overall diversity of model on the dataset D, we compute the average diversity score:
N E D (Pi),
AvgDiv m
(1)
where Div,, is diversity metric and denotes the specific type of metric being used. In our formulation below, Div, refers to our proposed effective semantic diversity metric. However, Div,, could represent any diversity metric, including lexical metrics like Distinct n-grams and average cosine distance.
Validity function. To determine diversity at the individual prompt level, we define validity function g + 1), where G is the space of generations.
Semantic funtion. Crucially, we determine the semantic uniqueness of generations based on their meaning rather than superficial textual differences. Two outputs may differ signifi- cantly in their lexical form yet convey identical semantic content a distinction that purely lexical or syntactic metrics fail to capture. Our goal is to define diversity metric over set of generations that captures both this semantic distinctiveness and the validity of outputs.
We define a semantic function S G + S, where S is the semantic space. Two generations g'g' 9 are said to be semantically equivalent and only if 5(g) 5(g'). The detailed definition of S on each domain is described in Section 3.2.
We then formalize effective semantic diversity in two complementary ways:
Semantic diversity. We define the effective semantic diversity of a generation set P; as the number of semantically unique valid generations, normalized by the total number of generations in the set:
Divixed (P.) (Set((((gg) 1 of E P,,
(2)
K
where Set(.) denotes the set of unique elements and | denotes cardinality. This measures the proportion of generations that are both valid/high-quas and semantically unique within the group of generations.
However, we found that Equation (2) can be confounded by the number of samples under consideration if analyzing variable-sized subset of valid-only programs; Appendix A.2 for more detail. To address these concerns, we adopt the pairwise diversity metric suggested in Tevet & Berant (2021):
4

---

Published as a conference paper at COLM 2025
Input Description:
Example Input:
Multiple datasets.
35.68 68 139.77 51. 15 359.82
Each dataset consists of four real num-
01.37 163.92 41.78 272.25
bers: a, b, c, d.
51.15 359.82 82 -34.58 58 301.52
There are no more than 30 datasets.
Function Signature:
Write a function f f(inputs) that processes the list of tuples where each tuple contains four real numbers.
from typing import List, Tuple
def f(inputs: List[Tuple[float, float, float, float]])): .1
inputs: a list of tuples, where each tuple contains four real numbers
Figure 2: An example of open-ended problem description from our dataset.
1
Dav (P:)
Σ
dsem (8)1
(3)
EP 1sicKs
where the effective semantic distance function dsem 9 x + {0,1} is defined as:
0 if ((g) = v(g) = 0,
dgem (g)
O if V(g = ((g) = and S(g) = S(g)),
otherwise.
This pairwise approach normalizes by the total number possible pairs, making robust to variations in the number of valid generations across different prompts. However, in domains such as natural language where strict notions of semantics do not apply, we define soft approach to measuring dsem
dsem (g), vgg) (gg)) x (1 (1- (imimm))).)))
(5)
Here Sim : 9 x G - [0, 1] is a similarity function that does not explicitly consider quality, for example, assigned by human AT judge.
3.2 Dataset, Validity, and Semantic Equivalence Checking for Programs
We chose to instantiate methods of validity and semantic equivalence checking using both programs and natural language. We found it advantageous use programs avoid the scenario of using LLMs to evaluate LLMs for diversity, especially if we seek to evaluate increasingly powerful LLMs with weaker LLMs. Furthermore, there has been relatively little empirical work that studies the robustness of using LLMs to judge the diversity of LLM-generated content. Lastly, our implementation of effective semantic diversity for pro- grams avoided the costs of utilizing commercial LLMs judge. Research in programming language semantics has a rich history in the formalization of program correctness, termina- tion, and semantic equivalence (Hoare, 1969; Plotkin, 1981; Floyd, 1993; Pierce, 2002). Using program execution on test cases, we can confirm two programs are semantically distinct they compute different values over the same suite of input test cases!. Additionally, we can
N.B. Test cases may fail to test edge cases where two programs may differ. However, instead of over-reporting correctness, this phenomenon would only penalize diversity when two programs equivalent most test cases and hypothetically differ some edge-case, which find tolerable.

---

Published as a conference paper at COLM 2025
impose constraints of validity quality that are easy to check: such as the program always returning a value without syntax or other runtime errors.
The key question, however, is to define open-ended programming tasks. Generating pro- grams as an open-ended task is not necessarily radical: in chat applications, users may desire to brainstorm programs code projects. Moreover, generating synthetic programming tasks is already an important part of improving foundation LLMs for code (Dubey et al., 2024; Shypula et al., 2024).
We constructed our dataset by adapting competitive programming-style problems into open-ended abstracted programming tasks. In Figure 2, we show an example problem description from our dataset. For each problem description in our dataset, we provide an "Input Description" in natural language specifying the input format, an "Example Input" demonstrating potential inputs the function would handle, and a "Function Signature" providing a concrete specification of the function name and typing hints for the inputs. We manually reviewed, and if necessary, edited all final descriptions, removing any potential references to the original programming task, standardizing the function name to (...), and using highly generic argument names. We used competitive programming problems from CODENET (Puri et al., 2021) and accompanying test cases from ALPHACODE (Li et al., 2022) as a starting point for our dataset. In total we have 108 unique programming tasks/ liiksspprr prompts that were adapted from CEDENET into this open-ended format. Using each of these, we can sample arbitrary number of generations to calculate Div (P;). Further details about the test set, its creation, and our execution environment are provided in Appendix A.3.
3.3 Dataset for Natural Language
For natural language, we created our dataset by taking a subsample of 100 creative writing prompts from the WRITINGPROMPE dataset (Fan et al., 2018). We took random subset of the test set, prioritized [WP] tagged prompts, and avoided prompts that could generate sensitive inappropriate stories.
Experimental Setup
Diversity Metrics. We now formalize the diversity metric (P; with particular focus on our proposed notion of effective semantic diversity. As introduced in Section 3.1, this metric relies on the choice of valid function V and semantic function S.
Let T = { } be fixed set of test inputs. For any generated program, let g(ti) de- notes its execution on test case t E T, and ol denotes the corresponding output. Specifically, for code we define the validity function V G + {0, 1 as follows:
V(g) {
if g(4) executes without error and produces non-null output VI E (1,...,L). L), otherwise.
In other words, program g is considered valid if it runs without raising any errors (e.g., SyntaxError, ValueError) and produces non-null outputs for all test cases in T.
In the domain of natural language, prompt LLM judge with set of criteria to judge the quality of the creative generation and then normalize the score to fall between [0, 1].
Next, we define the semantic function S 9 + OL, where O denotes the output space. The function maps program g to its output trace the test set:
S(g) =
Two programs g and g' are considered semantically equivalent if and only if their outputs are identical across all test cases:
S(g) S(g')
g(t)) = g(())) 4 L}.
For natural language, for Sim, the semantic similarity function that does not explicitly consider quality, we prompt LLM judge to evaluate the conceptual and thematic overlap

---

Published as a conference paper at COLM 2025
Model Family
Base
SFT
DPO
RL
Llama-2-7b-chatal
LLaMA 2 7B
Llama-2-7b-hf
tulu-2-7b
tulu-2-dpo-7b
(PPO)
Llama-2-70b-chatir
LLaMA 2 70B
Llama-2-70b-hf
tulu-2-70b
tulu-2-dpo-70b
(PPO)
Tulu-3-8B-DPO
Tulu-3-SB (PPO)
LLaMA 3.1 8B
Llama-3.1-8B
TulU-3-8B-SFT
Llama-3.1-8B-Instuucl
Tulu-3.1-8B (GRPO)
"lulu-3-7BB-DPO 1lama-3.1-708-Imsrrrct
LLaMA 3.1 70B
Llama-3.1-70B
Tuul-3370B-SFT
Tulu-3-70B (PPO)
Table 1: Model Post-Training Categorization. We organize all models under consideration by their base model and post-training method. All models to the right of "Base" are post- trained with the specified method from the base model.
between two generations and normalize the score. For all LLM judge tasks we utilize ppt-4.1-minis -205--0-22222-1111 Due to the scale of our experiments, we subsampled 32 pairs with replacemet from all possible pairs for each set of generations.
For the remaining diversity metrics, we adopt Equation (3) and utilize the following dis- tance functions in place of dsem ( For Lexical diversity, we use Expectation-Adjuste Distinct rt-rams (EAD) with n-grams of length #4 For syntactic diversity in code, we adapt the Distinct-N metric to the Abstract Syntax Tree (AST) of each generated program: to isolate the syntactic structure of program (e.g, for-loop instead of recursion) from superficial choices (e.g., variable names), we canonicalize all program identifiers. For two programs, we calculate the ratio the number unique subtrees of height H across both programs to the total number of subtrees of height H in both programs, where H=4. In order to measure neural diversity for code, we adapt existing methods of neural diver- sity metrics (Tevet & Berant, 2021) to our domain using CODEBERTSCORE (Zhou et al., 2023). We use CODEBERTSCORE because it closely resembles the models used in the NLP literature to evaluate diversity (Tevet & Berant, 2021) and has been evaluated as effective in the evaluation of neural codes. We also attempted to use CODELLAMA-7--NSTTER embeddings and ICESCORE (Zhuo, 2024) to evaluate diversity and found CODELLAMA similar to CODEBERTSCORF and ICESCORE to correlate strongly with temperature, such as the lexical diversity metric. See the the additional results in Appendix A.1. For natural language, we report the average score from the LLM judge D before normalizing it by the generation's 's quality.
Models and Sampling. The main empirical questions we seek to answer in our work surround the effects of different post-training algorithms and model size on diversity (SFT, DPO, PPO, GRPO, etc). In order to isolate the effects of different post-training strategies, utilize the TULU-2 (Ivison et al., 2023) and LLAMA-2 (Touvron et al., 2023) as well as the TULU-3 (Lambert et al., 2024) and LLAMA3.1 (Dubey et al., 2024) families of models. TULU-2 consists of post-trained LLAMA-2 models, and TUL.U33 consists of post-trained LLAMA3.1 models. In Table 1, we provide a categorization of the post-training methods. Using these we can compare SFT-tuned model to DPO-tuned model from the same family. For each problem description x; in our dataset, we generate K 32 programs, yielding 3,456 total programs sampled for each experiment. In cases where models are post-traines with numerous algorithms, we often categorize the model with the most aggressive algorithm used, e.g. we would categorize a model adapted with DPO and PPO as PPO.
In addition to our questions regarding the effects of post-training and model size, we also investigate which sizes and classes of models are most efficient on a per-parameter basis in generating unique examples. This is an attempt to understand which types of models could be optimal for large-scale synthetic data generation. We relax the strict necessity to pair models by the same base model, and use models from QWEN2.5-CODER (Hui et al., 2024), QWEN2.5 (Yang et al., 2024), and as well DEEPSEEK-RE -E-SSEEE-R--DSTILL (Guo et al., 2025) family. Our goal was to test a broader set of sizes as well as models tuned specifically for reasoning tasks (i.e. DEEPSEEK-RI
7

---

Published as a conference paper at COLM 2025
Validity
Semantic
Syntactic
Lexical
Neural
Comparison
W (p)
ES (d)
W (p)
ES (d)
W (p)
ES (d)
W (p)
ES (d)
W (p)
ES (d)
BASE vs. INST (ALL)
<0.001
1.33
<0.001
1.34
<0.001
-1.10
0.028
-0.47
<0.001
-1.53
BASE VS. INST-SFT
<0.001
1.37
<0.001
1.36
0.266
-0.34
0.339
0.21
0.339
-0.42
BASE VS. INST-DPO
0.006
1.11
0.005
1.10
0.014
-0.84
<0.001
-1.68
<0.001
-2.37
BASE VS. INST-RL
<0.001
1.67
<0.001
1.54
0.012
-1.20
0.035
-0.66
<0.001
-2.12
BASE VS. INST-PREE
<0.001
1.34
<0.001
1.29
<0.001
-1.45
0.001
-0.77
<0.001
-2.29
SFT VS. DPO
0.151
0.51
0.266
0.39
<0.001
-0.90
0.003
-0.63
<0.001
-1.52
SFT VS. RL
0.004
3.19
0.004
2.49
0.004
-2.43
0.004
-1.20
0.004
-2.29
SET vs. PREF
<0.001
0.97
0.002
0.77
<0.001
-1.30
<0.001
-0.83
<0.001
-1.86
DPO vs. RL
1.0
0.10
0.301
-0.14
0.039
-0.60
0.004
-0.78
1.0
-0.01
SM. VS. LG.
0.030
0.20
0.017
0.26
0.170
0.15
0.485
0.09
0.269
0.12
SM. VS. LG. BASE
0.063
0.57
0.063
0.56
0.063
0.74
0.063
0.69
0.031
0.80
SM. Vs. LG. INST
0.111
0.070
0.27
0.683
0.06
0.539
-0.11
0.759
-0.05
SM. VS. LG. PREF
0.330
0.13
0.277
0.20
0.454
0.10
0.330
-0.19
0.389
0.14
Table 2: Model Comparison Results. Results from Wilcoxon's Bignedd-Rank Test p-values: W (p), and Effect Size measured by Cohen's D: ES (d). Bold p-values are below 0.05, and bold d-values have an absolute value greater than 0.8 (large effect size). For effect sizes, positive values indicate the second model type in the comparison has higher values. "Inst" Instruction-tunee
Trainedd, "Pref" Preference-tuned and combines RL and DPO models. The columns correspond to metrics Section 4 (e.g. "Semantic" corresponds to effective semantic diversity).
Prompt selection. The choice of prompt can affect the nature of generations. Because of this, we created three separate prompt templates: a zero-shot prompt, a two-shot prompt, and two-shot prompt with chain-of-thought reasoning. This design allows to probe generation behavior across a variety of settings. The few-shot examples included in the prompts were simple, manually written examples shared across all problems in the dataset. We provide the examples in Appendix A.6.
Statistical tests. We aim to determine if factor such post-training algorithm or model size increases or decreases diversity compared to another. To rigorously summarize these results, we report the two-tailed Wilcoxon Signed-Rank Test (to measure significance) and Cohen's i D (to measure effect size) for AvgDiv over the entire dataset for each statistic. benefit of the non-paraeetric Wilcoxon statistical test is that we can make rigorous conclusions even if only limited samples are available. We always pair models from the same family and vary whether the model larger fine-tuned with a different strategy while fixing all other factors unless otherwise noted. For example, when isolating model size, we would compare LLAMA3.1 1-B...188-IINTRUUE
with Zero-Shot prompting to LLAMA3.1 INSTRUCT with Zero-Shot prompting, and so Because of the differences in post-training pipelines and to avoid confounding factors, we avoid pairing TULU post-trained models with LLAMA preference-tuned models.
5
Experimental Results
5.1 Effect of Post-Training and Preference-Tuning on Diversity
Higher effecitve semantic diversity introduced by preference tuning. We summarize results across all diversity metrics when comparing base models and their instruction- tuned counterparts, as well as when comparing different post-training techniques, for programming tasks in Table and natural language tasks in Table 3. Each row compares two fine-tuning approaches; for each metric, the columns report the statistical significance W (p) and effect size ES (d). A statistically significant positive effect indicates that the second post-training strategy outperforms the first for that metric.
8

---

Published as a conference paper at COLM 2025
Validity
Semantic
Lexical
Neural
Comparison
W (p)
ES (d)
W (p)
ES (d)
W (p)
ES (d)
W (p)
ES (d)
BASE Vs. INST (ALL)
<0.001
2.02
<0.001
1.50
<0.001
1.65
<0.001
-3.60
BASE VS. INST-SFT
<0.001
2.19
0.001
1.66
<0.001
2.53
<0.001
-2.61
BASE VS. INST-DPO
0.001
1.51
0.014
0.89
<0.001
2.27
<0.001
-4.27
BASE VS. INST-RL
<0.001
4.88
<0.001
3.91
0.022
0.73
<0.001
-5.68
BASE VS. INST-PREF
<0.001
2.18
<0.001
1.53
<0.001
1.34
<0.001
-4.58
SET VS. DPO
<0.001
1.52
0.001
0.96
<0.001
1.12
<0.001
-1.61
SFT vs. RL
0.004
5.79
0.008
2.76
0.004
3.60
0.004
-4.67
SET vs. PREF
<0.001
2.36
<0.001
1.54
<0.001
1.48
<0.001
-2.39
DPO vs. RL
0.055
0.51
0.025
0.59
0.82
0.28
0.02
-0.40
SM. VS. LG.
<0.001
0.41
<0.001
0.53
0.04
-0.21
0.009
-0.20
SM. vs. LG. BASE
0.031
3.04
0.031
3.35
0.844
0.05
0.062
-1.21
SM. VS. LG. INST
0.001
0.40
0.006
0.42
0.003
-0.32
0.055
-0,26
Table 3: Creative Writing (Natural Language) Model Comparison Results. Bold p-values are below 0.05, and bold d-values have an absolute value greater than 0.8 (large effect size)
Semantic
Syntactic
Lexical
Neural
Comparison
W (p)
ES (d)
W (p)
ES (d)
W (p)
ES (d)
W (p)
ES (d)
SFT vs. DPO
0.052
-0.71
0.791
-0.35
1.000
-0.12
0.016
-0.89
SFT vs. RL
0.004
-3.12
0.426
-0.28
0.820
0.08
0.004
-2.25
SET vs. PREF
<0.001
-1.31
0.473
-0.32
0.946
-0.05
<0.001
-1.39
DPO vs. RI.
0.301
-0.43
0.012
-1.16
0.008
-1.24
0.570
-0.05
Table 4: Validity-Controlled Model Comparison Results for Code. Bold p-values are below 0.05, and bold d-values have an absolute value greater than 0.8 (large effect size)
We observe a clear trend: all post-training techniques increase both effective semantic di- versity and validity relative to base models. RL methods, in particular, yield substantial improvements over SFT in effective semantic diversity. We also find an interesting pattern: preference tuning tends substantially reduce syntactic and lexical diversity in program- ming tasks, yet increases these metrics in natural language creative writing. These results suggest that in domains requiring high-quality and diverse outputs, preference-tuned mod- els can outperform both SFT and base models. Furthermore, in creative writing writing-where whhree diverse word choice and stylistic variety are often prwpeabeeeepeeeeeeeeeeeeeeeee tuned models may hold an advantage in stylistic capabilities. Additional experiments are provided in Appendix A.7 and Appendix A.8.
Semantic diversity driven by higher quality. In Table 4, we conduct statistical tests similar to those in Table 2 for code, but restrict the analysis to only valid generations. We find that within this subset, preference-tuned models generally have more semantic duplicates than their SFT-tuned counterparts. This suggests that while preference tuning, particularly RL, tends to increase semantic duplication within high-quality generations, the overall increase in the number of high-quality generations more than compensates for this effect.
5.2 Effects of Model Size
Larger models increase semantic diversity. In Table 2 and Table 3, we compare small and large models within the same family. both code and natural language, larger models generally exhibit higher semantic diversity. In code, this increase does not come at the expense of lexical or syntactic diversity; however, in natural language, we observe a small but significant decrease in lexical diversity. We attribute the improvements in quality and effective semantic diversity to the well-documented strengths of larger models in helpfulness and quality.

---

Published as a conference paper at COLM 2025
Model Diversity Efficiency vs. Model Size
Owen 0.50 Owen Codar 1 oven
Diversity Efficiency Scale
Owen 2.53
denn Coder
Ownn 3,50
Tilk S. 88 (CRPO)
Talk 5.80 SP1
Despseok 2.35
Owen 308
10r
Gwen Coder
Owin 703
Deepdouc
(P90) Owin 1a0
Usee 3.1
Alignment Method
Deep 320 You
30
SFT
DPO
Liuma
REASONING
Dieeplinnc R3 730
3,58
1.38
36
an
14h
101
7ôB
Model Size (Billions of Parameters) LLo Scale
Figure 3: Model efficiency: We plot the parameter efficiency of model in generating unique examples vs. model size (log-scale) on our programming dataset.
Smaller models may be more compute-efficient for generating unique examples. While larger models tend to produce more diverse outputs per generation, we also examine which models are most efficient given fixed computational budget. Consider a researcher aiming to generate large number of unique synthetic programs: is it better to take fewer samples from very large LLM, many samples from smaller LM, or find a "sweet spot" in between? In Figure 3, we explore this question by plotting the parameter efficiency for diversity against model size, where efficiency is measured as the effective semantic diversity (per Equation (2)) normalized by the number of model parameters. For a fixed sampling budget of 32 generations per prompt in the programming dataset, smaller models consistently more efficient on a per-parameter basis. We believe that further work should be done to investigate whether this pattern continues to hold the number of samples increases, to ensure semantically unique programs do not saturate faster for certain model classes than others.
Discussion and Conclusion
We propose a novel strategy for studying effective semantic diversity by leveraging code execution to jointly measure and balance quality and diversity. In the natural language domain, we use LLM judges as a proxy for human evaluation. Using these methodologies, conduct an extensive empirical analysis of LLM diversity and find counterintuitive insights into how factors such as post-training and model size meaningfully influence diversity. We encourage further work to further broaden the range of empirical questions explored in this area.
Acknowledgements
We would like to thank Christopher Watson, Cassidy Laidlaw, and other members of Berkely Al Research for feedback our draft. This material is partly based on research sponsored in part by the Air Force Research Laboratory (agreement number FA8750-19-2-0200 and award W911NF-20-1-0080), an Amazon Research Award Fall 2023, and an Amazon/ ASSET Gift for Research in Trustworthy AI. The U.S. Govt. is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon. The views and conclusions contained herein are those of the authors and should not be inter- preted as necessarily representing the official policies or endorsements, either expressed or implied, of the Air Force Research Laboratory or the U.S. Government.
10

---

Published as a conference paper at COLM 2025
References
Yoshua Bengio, Réjean Ducharme, and Pascal Vincent. A neural probabilistic language model. Advances neural information processing systems, 13, 2000.
Hao Chen, Abdul Waheed, Xiang Li, Yidong Wang, Jindong Wang, Bhiksha Raj, and Marah I. Abdin. On the diversity of synthetic data and its impact training large language models, 2024.
Jacob Devlin, Ming-Wor Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pp. 4171-4186, 2019.
Abhishek Divekar and Greg Durrett. Synthesizrr: Generating diverse datasets with retrieval augmentation, 2024.
Wenchao Du and Alan W Black. Boosting dialog response generation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama herd of models. ar preprint or Xiv:2407 21783,, 2024.
Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. Advances in Neural Information Processing Systems, 36, 2024.
Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical neural story generation. arv preprint XiXx:v80050000 04833, 2018.
Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Lin- jun Shou, Bing Qin, Ting Liu, Daxin Jiang, et al. Codebert: A pre-trained model for programming and natural languages. arXiv preprint arXiv:2002.08155, 2020.
Robert W Floyd. Assigning meanings to programs. In Program Verification: Fundamental Issues in Computer Science, pp. 65-81. 65-81. Springer, 1993.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song. Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in lims via reinforcement learning. arXiv preprini arXiv:2501. 1 12948, 2025.
Yanzhu Guo, Guokan Shang, and Chloé Clavel. Benchmarking linguistic diversity of large language models. Xiu preprint ar 212::2241..1027,, 2024.
Charles Antony Richard Hoare. An axiomatic basis for computer programming. Communii cations of the ACM, 12(10):576-580 1969.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. are preprint ar 1rXi::1904.09511, 2019.
Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, et al. Qwen2. 5-coder technical report. ar Xiv preprint arXio:2409.12186 2024.
Hamish Ivison, Yizhong Wang, Valentina Pyatkin, Nathan Lambert, Matthew Peters, Pradeep Dasigi, Joel Jang, David Wadden, Noah A Smith, Iz Beltagy, et al. Camels in changing climate: Enhancing Im adaptation with tulu 2. ar preprint arXiv:2311.10702, 2023.
Sophie Jentzsch and Kristian Kersting. Chatgpt is fun but it is not funny! humor is still challenging large language models. In The 61st Anmual Meeting Of The Association For Computational Linguistics, 2023.
11

---

Published as a conference paper at COLM 2025
Robert Kirk, Ishita Mediratta, Christoforos Nalmpantis, Jelena Luketina, Eric Hambro, Edward Grefenstette, and Roberta Raileanu. Understanding the effects of rlhf on Ilm generalisation and diversity. arXiv preprint ar xrXio:2310.06452, 2023.
Tomasz Korbak, Ethan Perez, and Christopher L Buckley. RI with kl penalties is better viewed as bayesian inference. arviv preprint ar xrXit:2205.11275, 2022.
Yi-An Lai, Xuan Zhu, Yi Zhang, and Mona Diab. Diversity, density, and homogeneity: Quantitative characteristic metrics for text collections. arXiv preprint arXiv:2003.0852)
2020.
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. T ulu 3: Pushing frontiers in open language model post-training. arv preprint ar Xfx::22411.15124 2024.
Jack Lanchantin, Angelica Chen, Shehzaad Dhuliawala, Ping Yu, Jason Weston, Sain- bayar Sukhbaatar, and Ilia Kulikov. Diverse preference optimization. arXiv preprint ar axXi::2501.1810, 2025.
Mina Lee, Percy Liang, and Qian Yang. Coauthor: Designing human-ai collaborative writing dataset for exploring language model capabilities. In CHI Conference on Human Factors in Computing Systems, CHI '2, pp. 1-19. ACM, April 2022. doi: 10.1145/349111 3502030. URL h www.doo.ooo..oo.oooo...o.oo org/org////o / (1145/349110..502222
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. A diversity-promoting objective function for neural conversation models. In Kevin Knight, Ani Nenkova, and Owen Rambow (eds.), Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 110- 119, San Diego, California, June 2016. Association for Computational Linguistics. doi: 10.18653 /0066553v///// 111116-011. URL https: http//aacannoooooooorrooooo org/N16 1014.
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, al. Competition-level code generation with alphacode. Science, 378(6624;:1092-1097 2022.
R Thomas McCoy, Paul Smolensky, Tal Linzen, Jianfeng Gao, and Asli Celikyilmaz. How much do language models copy from their training data? evaluating linguistic novelty in text generation using raven. Transactions of the Association for Computational Linguistics, 11: 652-670, 2023.
Brando Miranda, Alycia Lee, Sudharsan Sundar, Allison Casasola, Rylan Schaeffer, Elyas Obbad, and Sanmi Koyejo. Beyond scale: The diversity coefficient data quality metric for variability in natural language data, 2025. URL https::/arxii...tt httrrg/aab///...../ 13840.
Sonia K Murthy, Tomer Ullman, and Jennifer Hu. One fish, two fish, but not the whole sea: Alignment reduces language models' conceptual diversity. ar preprint arXiv:2411.04427. 2024.
OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. ar Xiv preprint arXiv:2303.08774 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 25:27730-27744, 2022.
Vishakh Padmakumar and He He. Does writing with language models reduce content diversity? Xiv preprint arXio:2309.05199 2023.
Vishakh Padmakumar and He He. Does writing with language models reduce content diversity?, 2024. URL https://arxiv. wwwll.oorg/aaaae httgg//b//209../
12

---

Published as a conference paper at COLM 2025
Benjamin C Pierce. Types and programmingg languages. MIT press, 2002.
Gordon D Plotkin. A structural approach to operational semantics. 1981.
Ruchir Puri, David S Kung, Geert Janssen, Wei Zhang, Giacomo Domeniconi, Vladimir Zolotov, Julian Dolby, Jie Chen, Mihir Choudhury, Lindsey Decker, et al. Codenet: A large-scale ai for code dataset for learning diversity of coding tasks. arXiv preprint ar arXiv:2105.12633 2021.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al.
Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in Neural Information Processing Systems, 36, 2024.
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. Xrv preprint ar XXix:1908.10084 2019.
Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Ttai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, et al. Code llama: Open foundation models for code. arviv preprint ar Xrio:2308.11255) 2023.
Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. Whose opinions do language models reflect? In International Conference on PMLR, 2023.
Machine Learning, pp.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arviv preprint ar 1tt::1707.06347, 2017.
Chantal Shaib, Barrow, Jiuding Sun, Alexa F Siu, Byron C Wallace, and Ani Nenkova. Standardizing the measurement of text diversity: A tool and comparative analysis of scores. Xiv preprint ar Xii::2403.00553, 2024.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Wu, al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint ar arXio:2402.03300 2024.
Alexander G Shypula, Aman Madaan, Yimeng Zeng, Uri Alon, Jacob R. Gardner, Yiming Yang, Milad Hashemi, Graham Neubig, Parthasarathy Ranganathan, Osbert Bastani, and Amir Yazdanbakhsh. Learning petformance-mpprovine code edits. In The Twelfth International Conference on Learning Representations, 2024. URL https: /ttpr///////eeee..... net/ forum?id=ix7rLVHKyy.
Katherine Stasaski and Marti A Hearst. Semantic diversity in dialogue with natural language inference. Xrv preprint ar XrXi:2205500199, 2022.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-3021, 2020.
Guy Tevet and Jonathan Berant. Evaluating the evaluation of diversity in natural language generation. In Paola Merlo, Jorg Tiedemann, and Reut Tsarfaty (eds.), Proceedings the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume,, pp. 326-346, Online, April 2021. Association for Computational Linguistics. doi: 10.18653 ww/2022.eacc.main.
URL https: https::/acc.atthollgg////// org///////22// http://e221.ee.1eeeeeeeeeaeeeeea eacl-main 25.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint +xXix:2307.09288 2023.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. technical report. aic preprint arXit:2412.1511 2024.
13

---

Published as a conference paper at COLM 2025
Yue Yu, Yuchen Zhuang, Jieyu Zhang, Yu Meng, Alexander Ratner, Ranjay Krishna, Jiaming Shen, and Chao Zhang. Large language model as attributed training data generator: A tale of diversity and bias. In Whirty-Seventh Conference Neural Information Processing Systems Datasets and Benchmarks Track, 2023.
Hugh Zhang, Daniel Duckworth, Daphne Ippolito, and Arvind Neelakantan. Trading off diversity and quality in natural language generation. arviv preprint arXiv:2004110450, 2020.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. ar Xiv preprint ar Stt:190440966755 2019.
Shuyan Zhou, Uri Alon, Sumit Agarwal, and Graham Neubig. Codebertscore: Evaluating code generation with pretrained models of code. arXiv preprint ar arXiv:2302.05557, 2023.
Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. Texygen: A benchmarking platform for text generation models. In The 41st international ACM SIGIR conference on research & development in information refrieval, pp. 1097-1100, 2018.
Terry Yue Zhuo. Ice-score: Instructing large language models to evaluate code. In Findings of the Association for Computational Linguistics: EACL 2024, pp. 2232-2242, 2024.
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human prefer- ences, ar Xio preprint ar Xie:11999.08533, 2019.
14

---

Published as a conference paper at COLM 2025
A
Appendix
A.1 Additional Neural Diversity Metrics
In addition to using CODEBERTSCORE, we also demonstrate neural diversity with respect to temperature sweeps similar to those in Figure 1 using CODELLAMA-7B-INSTRUU et al., 2023) embeddings. We also use an LLM-as-a-jugee to calculate pairwise similarity using ICESCORE (Zhuo, 2024) using got-4o-mini as the LLM-judge. We find that ICESCORE correlates heavily with temperature and does not seem to reflect quality. Similarly with CODELLAMA embeddings, the LLAMASB-BASS model's generations consistently consid- ered more diverse than the INSTRUCT variants despite the base model generally producing far more low-quality and invalid outputs.
(Roziere
Comparison of Neural Metrics Across Selected Models
ICE Score
Embedding (Cosine) Diversity
1.0
0.8
0.8
ICE volue
0.0
0.6
0.4
0.4
Embredeing
0.2
0.2
0.6
0.7
0.0
0.9
1.0
1.5
322
1.3
1.4
0.6
0.7
0.8
0.9
1.0
1.1
3.2
1.3
1.4
Temperature
Temperature
Llama
Figure 4: Neural diversity metrics when modulating the Temperature parameter across different models. We report neural diversity metrics using the ICESCORE (Zhuo, 2024) and cosine diversity of CODELLAM--7-INSTTRUU embeddings.
A.2 Sample Size Confounding Diversity and Analysis of Pairwise Diversity Metric
In practice, we found that varying the size of the subset under consideration could dramati- cally affect the calculation of Equation (2): for the same model, as samples tended to get larger, this metric would decrease. When we use this metric for most our experiments, sufficient, because we keep the number of samples constant. However, when analyze the subset valid-only programs, the size of this population may vary from model to model depending the average quality of generations. Thus it is necessary to use Equation (3).
Analysis of Pairwise Diversity Metric. We provide an analysis Equation (3). For given large language model f, we assume that only finite number K of distinct semantic meanings can be generated by f. We first establish that the original semantic diversity metric converges to zero the number of sampled responses tends to infinity. Specifically, the original semantic diversity metric is defined as
N
n
where N is the number of distinct semantic clusters, and is the number of sampled responses. Since only a finite number of semantic meanings can be generated by f. the number semantic clusters is bounded above, implying the existence of a constant C > 0 such that N 2 C1. Therefore, we have
N
C1 for all sufficiently large n.
15

---

Published as a conference paper at COLM 2025
Now, observe that
lim C1 1-300 17
0,
so applying the squeeze theorem, we conclude that
N 100 77
lim
Next, we show that the new metric defined in Equation (3), converges to a constant as >> co. As before, assume that there K distinct semantic meanings in total, and let nt denote the proportion of responses corresponding to the k-th semantic meaning, This implies that the number of times each semantic meaning is sampled is HtZ, where ΣK 1 T = 1. Thus, we have
E PsPjEPis)
(Pi Pj)
(TTg) (TTR) kihh
Σ (ATTT)))) kith
where the summation is taken over distinct semantic meanings k and h, and (Pist Pj) measures the semantic distance between generations. Moreover, the number of possible response pairs is
n(n - 1)
2
2
Thus, the new metric becomes
1 Σ
Exfp FT
2 Et (Tx
Mdist
(pist(Pi,/) (pi, p,)
2 Σ
(T<TTh))
+
(2)
P.PjEPisi
n n(n
1)
n1
Finally, we have
2 /I*/h (T(
lim
2
Σ ( +
Σ TT.
kis
That is, as + co, the value of the new metric converges to the constant value:
Σ k
TT.
A.3 Dataset Creation and Additional Details
We created our dataset through multi-step process starting from the CODENET dataset and testcases from ALPHACODE. The process model assistance for scaling, and manual validation.
problem standardization, language
Initial Processing. We began by randomly selecting a single problem description from IBM CodeNet. We used this to seed to construct examples for 1-shot prompt for Language Model to assist us. Specifically we manually wrote one example of the following outputs that should be generated for each individual problem description from CODENET
1. A canonicalized problem description
2. A wrapper function that would take any generation for that function, parse inputs for the function, and instrument the generation with the entire suite of test cases
3. A property-based testing function for generating additional test cases when needed
Dataset Expansion. We then wrote prompt templates for both gpt-3.5-turbo- 5-turbo-0125 and gpt-4o-2024- 2024-11-22222 11-00 (depending on the iteration). These templates were designed to take our 1-shot prompt, and then prompt the LLM to repeat the same for the new example we select from CODENET. We randomly sampled over 300 programs from CodeNet and attempped to generate the three components for each original problem description from CODENET We
16

---

Published as a conference paper at COLM 2025
then saved these into individual files, and also wrote them into an HTML document for manual review.
Manual Review and Selection. For the over 300 problem ids that were processed, we then manually inspected the output components checking for the following criteria:
1. The language model correctly formatted inputs into our desired format
2. The problem appeared novel relative to the existing problems had already added to our dataset.
We tracked the original CodeNet problem IDs and validation results in a spreadsheet. These were the 108 examples then used for our dataset.
Manual Editing of Problem Descriptions. After selecting our problems, we then manually went through each of the three components, and manually edited them to be correct: large amount required revisions, as the LLM assistant made mistakes. We saved our manually edited components to be further processed.
Test Case Integration. For each of the individual problem descriptions, we then merged test cases from CODENET and ALPHACODE We required at least 10 test cases per problem. For the three problems that lacked sufficient test cases, we used our property-based testing scripts to generate 100 additional cases, such that we would have sufficient coverage. We then saved the canonicalized problem descriptions, the function to extract and parse input test cases, and the input test cases into dataset.
Final Checks. During experimental validation, we found and fixed multiple faulty problem description argument parsing functions. We then saved these corrected version as our final dataset.
Additional Dataset Details. In our experiments, we instruct LLMs to return their outputs. We maintain wrapper script that executes the function f(...) for each input, obtains the returned result, and then serializes the object as well as its type (recursively) into a string. We then utilize these strings as the basis for our semantic comparisons. For each generation, we execute all test cases and capture the resulting outputs.
Because LLMs often generate natural language to accompany generated programs, extract- ing programs from the generations is a non-trivial task, especially for pre-trained models. We developed an extraction utility that extracts not only the target function f(..), but also any helper functions and imports that may be relevant. To safely execute programs at scale, we perform all execution inside an isolated DOCKER container to prevent adverse consequences of blindly executing LLM outputs.
A.4 Raw Experimental Results for All Experiments
In Table and Table 6, we include all our raw results used in our analysis.
A.5 Additional Information on the Syntactic Diversity Metric
We extract and process all Abstract Syntax Trees (ASTs) using Python's AST library with Python version 3.12.0, and report metrics for subtrees of height 4. Because syntactically incorrect programs do not parse, we can only calculate this metric over the subset of syntactically correct generations.
A.6 Prompts Used Experiments
In Figure 7, Figure 8, and Figure 9, show the prompting templates we across all code experiments.
17

---

Published as a conference paper at COLM 2025
Model
Prompt
Validity Semantic Lexical Neural
3lama-3.1-7055
Zero-Shot
69.42
11.32
97.72
29.12
1lamm-3.17708-nnstree
Twu-Shot
36.28
6.02
98.56
49.98
1aama-31-70B-nnstruct
Two-Shot CoT
L1am--31-Tull-3-7088
Zero-Shot
91.95
15,95
98.92
21.20
11amm-311-7olu-3-7088
Two-Shot
90.31
18.63
98,90
26.17
Uaam-3.11-Tul--3-77
Two-Shot CoT
89.71
19.63
99.04
30.73
1lama-3.-Tulu-3-200--D--
Zeno-Shot
90.18
17.87
99.05
24.92
1lama33--Tata---70--PPP0
Two-Shot
90.10
99.02
Llmma-3j-lutt:--700--DPP
Two-Shot CoT
89.39
20.11
98.95
32.27
1lmma-31-Tula:32700-5F7
Zero-Shot
68.33
22.83
98.42
56.75
-lama-3--Tutu-3-000-5FF
Two-Shot
58.09
18.22
98.71
57.09
1lama33117btuu--008-5577
Swo-Shot CoT
Llama-3.1-70B
Zero-Shot
56.12
17.97
96.51
59.86
11ama-3.1-700
Two-Shot
40.72
10.07
98.13
67.81
Llama-3.1-700
Two-Shot CoT
52.35
17.76
98.07
69:17
1ama-3
Zeno-Shot
37.28
5.99
98.13
47.89
Llama-31-90-Insrruet
Two-Shot
24.11
99.24
Llam--3---B-Instrucct
Two-Shot Col
24.31
3.05
99.14
53.22
L-ma-3.1-T
Zero-Shot
90.55
17.83
98.89
24.17
Llama-3.1-Tulu-3111.
Two-Shot
87.77
21.40
98.71
34.59
Llama-31-Tbluu331-881
Two-Shot CoT
92.37
1lama-31-Toll-3-88B
Zero-Shot
90.72
18.96
99.05
24.83
Ulma-3.117
Two-Shot
85.47
19.08
98.87
33.14
Llama-3.-TTul:3--8B
Two-Shot CoT
91.11
18.49
98.97
26.63
Zero-Shot
88.84
18.13
98.84
30.67
Lamma33--Tulu:338--DDP0
Two-Shot
86.59
19.84
98.82
34.71
Laama331-lulu:388B-D00
Two-Shot Col
89.83
19.51
98.94
29.30
Llama-3--Tullu--888-5FFF
Zero-Shot
58.60
19.76
98.47
60.15
1lama-31--Tulu--8888877
Two-Shot
55.41
18.75
98.70
58:21
1lam--33--fulaa-888-551
Two-Shot CoT
1-mm--311-8
Zero-Shot
32.20
9.85
96.39
69.74
Llama-3.1-8B
Two-Shot
28 69
6,31
71.12
Llama-3.1-8B
Two-Shot CoT]
37.89
11.80
98.57
74.33
1lama-2-70b-cha-htt
Zero-Shot
30.37
8.05
97.76
64.03
Llama-2-00b-hhattt
Two-Shot
23.02
3,86
99.20
63.88
Two-Shot CoT
24.64
4.34
99.24
65.06
tulu-2-706
Zero-Shot
75.60
17.52
51.31
tulu-2-706
Twu-Shot
71.17
14.54
98.74
52.00
talu-2-706
Swo-
tulu-2-dpo-706
Zero-Shot
78.64
15.52
98.73
44.07
tulu-2-dpo-706
Two-Shot
75.10
16.63
47.46
tulu-2-dpo-706
Two-Shot Col
78.60
17.68
98.84
49.00
Llama-2-70b-hf
Zero-Shot
22.87
7.11
96.85
75.34
Ll7mb-2-70b-tf
Two-Shot
21.30
4.93
98.71
75.63
1lamm-2-70b-ff
Two-Shot CoT
29.24
8.90
98.59
77.89
Llama-2-7b-chll
Zeno-Shot
67.20
1amma-2-7-ccht--i
Twa-Shot
16.54
2.44
99.48
67.81
Two-Shot CoT
17.28
tulu-2-7b
Zero-Shot
53.99
12.95
98.07
66.84
tulu-2-7b
Two-Shot
67.74
tulu-2-7b
Twu-Shot Coll
51.51
12.86
98.47
69.61
tulu-2-dpo-7b
Zero-Shot
64.67
13.41
98.49
58.75
tulu-2-dpo-7bb
Two-Shot
59.52
12.74
98.74
60.59
tolu-2-dpo-7b
Twa-Shot CoT
62.36
14.19
98.67
62.58
Llamu-2-7b-hf
Zeno-Shot
1lama-2-7b-hf
Twa-Shot
17.70
3.44
99.08
77.63
Llama-2-7b-hf
Two-Shot CoT
23.52
Table 5: Raw Results for All Natural Language Experiments
18

---

Published as a conference paper at COLM 2025
Model
Prompt
|Validity
Semantic Lexical
Syntactic Neural
1lama-3.1-700
3.1-700-Insttruu
Zero-Shot
37.36
32.96
67.19
71.98
5.60
(amm--31-7UB-Instruu
Two-Shot
11.89
11.40
66.47
65.19
4.51
1lama-3.1-70- Instructt
Two-Shot CoT
1l-ma-31-Tuule33-008
Zenu-Shut
39.55
32.41
73.91
78.09
8.88
Two-Shot
41.49
33.22
72.76
75.33
4.43
Llama-311-Tutu-3-700
CoT
40.88
32.85
73.26
76.69
8.31
1lama-3.-TTul----006
Zern-Shot
38.57
32.93
75.37
79.81
9.54
1lmma-317Tut:::32000DDPPP
Two-Shot
38.27
32.85
74.45
78:27
6.78
1lama-331-Tulu:33700-DPP
CoT
39.18
33.81
74.89
78.93
9.07
Llama-3--Tut::-2008-555
Zero-Shot
37.36
32.96
67.19
71.98
5.60
JJama-31-TTbl--3-00B-
Two-Shot
35.48
29.37
68.76
71.38
5.23
Jamma-317Tuh::3-00-55777
Two-Shot CoT
36.42
31.12
Llama-3.1-70B
Zenv-Shot
20.18
24.02
54.05
55.47
3.51
3lmm--3.1-708
Two-Shot
13.28
14.70
57.87
57.00
3.46
Llama-3.1-700
CoT
15.17
18.00
57:25
58.00
3.64
IJama-3.
1amma-3.-88-Innsteee
Zem-SShot
15.47
18.98
56.40
60.12
3.41
Two-Shot
8.54
10.37
56.09
57.54
3.22
1lama-3.188-Insttrue
Two-Shot
9.94
12.70
55.63
58.68
3.35
Llama-.1-TTuu-311155.
Zero-Shot
37.67
32.41
73.32
77.66
8.91
Llama-3.--Tulu-311-88
Two-Shot
37.06
32.67
72.45
76.10
6.83
Llama-3.1-Tutu-311-88
Two-Shot CoT
38.57
33.70
73.19
77.53
Zern-Shot
38.27
32.93
74.52
78.64
9.22
Two-Shot
36.73
31.85
73.19
76.69
6.20
Llama-3.1TTbb--3-88
CoT
37.97
32.85
73.91
77.83
8.66
-amma-31-Tulu---888--11
Zero-Shot
37.06
32.41
74.06
78.28
8.91
Jlama-31-TTll:3-00-DD00
Two-Shot
36.42
31.70
73.19
76.69
6.83
Llama-3.-Tullu:-88--DDe
Two-Shot
37.67
32.56
73.75
77.53
8.44
Jlama-31-Tuluu3-60-555
Zero-Shot
35.18
29.37
68.17
71.38
5.23
Llama-3.--Tulu-3-8---5-
Two-Shot
34.85
29.67
68.01
71.18
5.40
1lama-331-Tub:33688-511
Two-Shot
1ama-3311-B
Zero-Shot
11.89
15.99
47.23
48.41
2.78
Llama-3.1-8B
Two-Shot
12.41
51.05
51.11
2.73
Llama-3.1-8B
Col
11.28
14.70
50.43
51.51
2.91
Llama-2-77 2-70h---0h-
Zero-Shot
11.28
15.84
52,30
55.87
3.30
Llama-2- 2-7b-chtth-
Two-Shot
6.43
8.52
51.82
53.57
3.11
Llama 2-7b--hh0u
Two-Shot CuT
7.82
10.67
52.14
54.36
3.25
tulu-2-70b
Zero-Shot
29.52
64.74
68.35
5.06
tulu-2-70b
Two-Shot
29.27
26.22
64.58
66.62
4.57
tulu-2-70b
Two-Shot CoT
28.52
tulu-2-dpu-70b
Zenv-Shut
33.26
30.41
67.03
70.28
5.40
tulu-2-dpo-20b
Two-Shot
31.25
27.70
66.71
68.35
tulu-2-dpu-70b
Two-Shot
32.56
29.37
66.87
69.52
5.23
Llama-2-70b-hf
Zero-Shot
8.24
12.11
44.15
45.33
2.59
1-7bb-2-70b-hf
Two-Shot
6.43
9.96
46.58
47.02
2.54
hf70b-h
Two-Shot
8.24
11.26
46.27
47.42
2.68
1lama-2-7b-chan
Zero-Shot
12.56
52.31
3.03
1lama-2-7b-chathle
Two-Shot
5.82
7.78
48.58
50.18
2.84
Two-Shot CoT
tulu-2-7b
Zenv-Shot
24.39
24.46
58.25
61.66
4.13
tulu-2-7b
Two-Shot
22.56
59.89
tulu-2-7b
Two-Shot CuT
23.78
23.41
58.41
60.89
3.98
tulu-2-dpo-7b
Zero-Shot
26.83
26.07
61.08
64.36
4.46
tulu-2-dpu-7b
Two-Shot
24.39
23.41
60.61
62.43
3.98
tulu-2-dpo-7b
Two-Shot
25.91
25.19
60.92
63.56
4.30
Llama-2-7b-hf
Zeno-Shot
10.22
Llama-2-7b-hf
Two-Shot
5.52
8.37
43.84
44.33
2.32
1lama-2-7b-hf
Two-Shot CoT
6.73
9.52
43.68
44.53
2.46
Table 6: Raw Results for General Code Experiments
A.7 Constrained Generation
We implement a validity oracle that only accepts lists of integers with a maximum length of 1000. The constraints are illustrated in the following code snippet, where output denotes the LLM generation:
19

---

Published as a conference paper at COLM 2025
Effective Semantic Diversity (Max over all prompts)
11.41
40.62
Traniing Category
37.56
17.27
Base
Effective Semantic Diversity
SFT
32 57
24.07
20-
18.75
13.69
15
10
7,25
0.2
LLaMA 78
LLaMA 70B
LLaMA 3.1 8B
LLaMA 3.1 708
Usna U.n05-77-16
2-76
talo app-
Uniaa 2- Tb-thar--t of
Ua 2 79B-1
tule 78b
Bill tpon 78b
Liema 7ith hatt-
Uemaa mama--a.. Rilu 3-80 sPy
Ulamta3, 3.13a-311-30 1-300
Elame. 4g fes-ferrrttt
Ulena Mute 45 DPO
Liema Ptato-3 8-80
Laama Tallo 4:
Lierne 3-Ball-- 718 S7T
Lamm 1-709
Liema 708 Intract
Limna 1ilk DPO
Laare 2 T-la 708
Figure 5: Effective semantic diveresity scores for all 19 models evaluated in our experi- ments, grouped by model family. Each bar is color-coded according to the post-training method, as categorized in Table 1
Example AST with Variable
Example AST with Variable Y'
Canonicalized AST for Either
Expr
Expr
Expr
Lambda
Lambda
Lambda
args:
UnaryOp
args:
UnaryOp
args:
UnaryOp
Pp
operand
C
operand
ap
operand
USub
Name: X
USub
Name: Y
USub
Name:
Figure 6: An Example of Canonicalizing an Abstract Syntax subtree used in the Distinct- CAST metric. The expression under consideration is for simple lambda expression that negates a given variable. The first two ASTs are not equal because of the usage of the variables X and Y, respectively, even though they are alpha-equivalent expressions. The AST on the far right canonicalizes identifier names such as arguments and variables so that both expressions would be equivalent.
assert "sinstance list)
assert all (isinstannce (x, int) for x in output) assert lon(out) 1000
Furthermore, we modify all prompts to instruct LLM to comply with these constraints. We show results in Table 7. In general, our findings are consistent with our previous results. The most notable difference is an increase in both statistical significance and effect size for semantic diversity when comparing DPO-tuned models RL-tuned models for code generation. While this difference is of interest, it does not affect our main conclusions.
A.8 Natural Language Experiments
In this section, extend our experiments to natural language tasks, including creative writing, argumentative writing, and brainstorming. Specifically, we obtained 10 prompts each for argumentative writing and creative writing from the CoAuthor dataset (Lee et al.,
20

---

Published as a conference paper at COLM 2025
jproblen_deecrription)
Now please implement the function f; do not return anything, the function should print the result of the operation.
should terminate within seconds.
Figure 7: Zero-Shot Prompt: Our template for our zero-shot prompt, where the problem description would be input inside the curly braces.
Comparison
Validity (Quality)
Semantic Diversity
W (p)
ES (d)
Winner
(p)
ES (d)
Winner
Base vs. Inst.
0.001
1.23
Inst.
0.001
1.27
Inst.
Small vs. Lg.
0.001
0.35
Lg.
0.001
0.35
Lg.
SFT vs. Pref.
0.001
1.12
Pref.
0.001
1.04
Pref.
SFT DPO
0.064
0.69
DPO
0.064
0.64
DPO
vs. RL 0.004
3.28
RL
0.004
3.14
DPO vs. RL
0.237
-0.19
DPO
0.039
-0.44
DPO
Comparison
Syntactic Diversity
Lexical Diversity
Raw Neural Diversity
W (p)
ES (d)
Winner
W (p)
ES (d)
Winner
W (p)
ES (d)
Winner
Base vs. Inst.
0,001
-0.88
Base
0.701
-0.16
Base
0.010
-1.61
Base
Small vs. Lg.
0.248
0.14
Lg.
0.645
0.02
Lg.
0.036
0.16
Lg.
SET Pref.
0.001
-0.95
SFT
< 0.001
-0.60
SFT
0.001
-2.24
SFT
SFT vs. DPO 0.001
-0.64
SFT
0.007
-0.46
SFT
0.000
-1.77
SFT
SET vs. RL 0.004
-2.18
SET
0.008
-1.04
SFT
0.004
-2.98
SFT
DPO vs. RL
0.426
-0.15
DPO
0.301
-0.37
DPO
1.000
0.05
RL
Table 7: Constrained Generation Model Comparison Results.
2022), which has been previously used in the literature (Padmakumar & He, 2024). We also manually curated a dataset of 10 brainstorming prompts designed to reasonably reflect creative assistance tasks for an LLM-based assistant.
We employed GPT-4.1-mini as a judge for both the quality and diversity of generations, selecting it to balance robustness with our API budget. Separate evaluation prompts were constructed for each of the three tasks to specify the criteria for quality and diversity assessment. For example, the criteria for creative writing tasks were as follows:
Creative Quality Criteria Consider the following rubric when evaluating:
1. Overall, holistic, and cohesive readability of the story (not merely compilation of elements).
2. Relevance of the story to the provided prompt.
3. Use of key narrative elements-vocabuulryy choice, imagery, setting, themes, dia- logue, characterisation, and point of view.
4. Structural elements and presentation demonstrating control over spelling, grammar, punctuation, paragraphing, and formatting.
5. Overall plot logic, including hook, conflict, initial crisis, rising and falling action, and sinteemmnn/resoluttiinn
6. Creativity, innovation, and and avoidance of clichés and derivative tropes.
introduction of new knowledge,
Creative Diversity Criteria Consider the following criteria when evaluating similarity:
21

---

Published as a conference paper at COLM 2025
#H# Input Description: 1. An integer N( N) (1
N( N(NN )) 18000), representing some quantity or size.
### Example Input:
1000
#*# Function Signature:
Write function "f(N) that takes in the input.
ppyhhon
def f(N: int):
...
N: an integer
Now please implement the function f; do not return anything, the function should print result the operation.
It should terminate within 30 seconds.
def f(N: int):
print(n**2)
### Input Description:
1. floating point number N(N () (1 AC V) 10080), representing some quantity or size.
**# Example Input:
13
#*# Function Signature:
Write function "f(N)" that takes in the input.
>> ppython
def f(N: float):
...
N: float float
Now please implement the function f; do not return anything, the function should print the result of the operation.
It should terminate within 30 seconds. def f(N: float):
=
while N 1:
N N i ++ print(i)
Now please implement the function f; do not return anything, the function should print the result of the operation.
It should terminate within 30 seconds.
Figure 8: Two-Shot Prompt: Our template for our two-shot prompt, where the problem description would be input near the end inside the curly braces.
1. Semantic Overlap:
Do the responses share similar underlying themes, ideas, narra- tive elements, or emotional content?
2. Thematic Consistency: Do both responses explore similar themes or motifs?
Our prompts instructed the model to assign score from 1-10 for each element. Thus, the first prompt had a maximum possible score of 60, and the second had a maximum possible score of 20. We used OpenAl's grammar-constrained decoding to ensure integer outputs, combined with a chain-of-thoo of-thought reasoning component to improve robustness, All scores were normalized by the maximum possible score.
22

---

Published as a conference paper at COLM 2025
For diversity evaluation, we sub-sampled 32 pairs from all possible pairs with replacement and asked the LLM judge to score the similarity between the two generations. The diversity score was computed as:
Diversity Score 1 1 Sim 5 gf)
Effective Semantic Diversity Because we could only calculate pairwise diversity metric for natural language, we applied Equation (3) from the paper to compute effective semantic diversity across sub-sampled pairs. For completeness, we used two methods to measure pairwise effective semantic diversity that we considered reasonable and consistent with our intended intuition.
The pairwise diversity metric from Equation (3) is defined as:
ESD pair (Pi)
Σ jec
dsem (g/r8))
where dsem 19 G + {0,1} givenby:
0 if either generation is invalid, if both are valid and semantically identical, if both are valid and semantically distinct.
dsem (g:
For natural language, we adapt this definition using Hard Thresholding:
dsem (g/
{
if LLM
(g) g/) > 0.5 and LLMqual
(uMMuuui(8) (g)) > 0.5,
otherwise.
We also adapt it using Soft Weighting:
dsem (g/ = LLMaiv (g) x LLMqual LLMqual
We present results for the creative writing task in Table 8, the argumentative writing task in Table 9, and the brainstorming task in Table 10. Overall, our findings for natural language tasks largely mirror those observed for code. Across all experiments, we consistently observe that post-training is associated with higher effective semantic diversity relative to base models. Furthermore, RL-tuned models generally exhibit higher effective semantic diversity than SFT-tuned models, often more markedly than DPO-tuned models. In argumentative and creative writing, larger models tend to achieve higher effective semantic diversity than smaller models, although this trend is less consistent in the creative brainstorming task. Importantly, in these natural language settings, we find little to no evidence that post- training strategies induce lexical mode-collapse. Moreover, when quality not taken into account, Raw Neural Diversity (as assessed by the LLM-judge) tends to be higher for less aggressive post-training regimes (e.g., base models outperform instruction-tuned models, and SFT models outperform RL-tuned models). However, once quality is incorporated into the evaluation, this effect can be fully reversed, consistent with our findings for code. Consequently, more aggressive post-training regimes, such as PPO, are ultimately associated with higher effective semantic diversity.
These experiments also underscore potential advantages of using code to evaluate effective semantic diversity. For code, our validity criteria (quality threshold) required both syntactic correctness and successful execution of all test cases without runtime errors; in contrast, for natural language, LLM-judge scores are less interpretable and may be susceptible to reward hacking. Additionally, code execution for diversity and quality assessment completes relatively quickly and incurs minimal cost, whereas LLM-judge evaluation can be computationally and financially expensive, necessitating careful selection of sample sizes
23

---

Published as a conference paper at COLM 2025
to control API usage. Finally, in environment where increasingly powerful models are being developed, using code to evaluate the qiversity-quality trade-off may offer distinct advantages over relying weaker models to evaluate potentially stronger models.
Comparison
Validity (Quality)
Soft Effective Semantic Diversity
Hard Effective Semantic Diversity
W (p)
ES (d)
Winner
(p)
ES (d)
Winner
W (p)
ES (d)
Winner
Base vs. Inst.
< 0.001
1.69
Inst.
0.001
0.80
Inst.
0.001
1.67
Inst.
Small vs. Lg.
0.001
0.37
Lg.
0.47
Lg.
0.020
0.41
Lg.
SFT vs. Pref.
< 0.001
2.13
Pref.
0.010
0.62
Pref.
0.050
0.58
Pref.
SFT vs. DPO
< 0.001
1.33
DPO
0.042
0.42
DPO
0.176
0.39
DPO
SFT vs. RL
0.004
5.80
RL
0.074
0.99
0.203
0.83
RL
DPO vs. RL
0.008
0.40
RL
0.129
-0.45
DPO
0.203
-0.29
DPO
Comparison
Lexical Diversity
Raw Neural Diversity
W (p)
RS (d)
Winner
W (p)
ES (d)
Winner
Base vs. Inst.
< 0.001
1.30
Inst.
0.001
-3.21
Base
Small vs. Lg.
0.202
-0.13
Lg.
0.010
-0.20
Small
SFT Pref.
< 0.001
0.95
Pref.
0.001
-2.38
SFT
SFT vs. DPO
< 0.001
0.90
DPO
0.001
-1.51
SFT
SFT vs. RL
0.004
1.70
RL
0.004
-5.57
SFT
DPO vs. RL
0.570
-0.15
DPO
0.098
-0.37
DPO
Table 8: Creative Writing Model Comparison Results.
Comparison
Validity (Quaiity)
Soft Effective Semantic Diversity
Hard Effective Semantic Diversity
W (p)
F$ (d)
Winner
(p)
FS (d)
Winner
W (p)
F$
Winner
Base vs. Inst.
< 0.001
1.93
Inst.
0.001
0.86
Inst.
0.001
1.69
Inst.
Small vs. Lg.
0,248
0.12
Lg.
0.026
0.54
Lg.
0.044
0.37
Lg.
SFT vs. Fret.
< 0.001
1.55
Pref.
0.49
0.032
0.41
Pref.
SFT vs. DPO
< 0.001
1.07
DPO
0.213
0.26
DPO
0.266
0,22
DPO
SFT vs. RL.
0.088
2,40
RL
0.129
1.07
RL
0.129
1.14
RL
DPO RL
0.020
0.22
RL
0.10
0.12
RL
Comparison
Lexical Diversity
Raw Neural Diversity
W (p)
ES (d)
Winner
(p)
ES (d)
Winner
Base vs. Inst.
0.033
0.16
Inst.
0.001
-3.07
Base
Small vs. Lg.
0.010
-0.33
Small
0.594
-0.00
Small
SFT vs. Pref.
0.919
-0.21
SFT
< 0.001
-1.90
SFT
SFT vs. DPO
0.970
-0.25
SFT
< 0.001
-1.43
SFT
SFT vs. RL
0.910
-0.14
SFT
0.004
-3.11
SFT
DPO vs. RL
0.426
0.21
RL
0.359
-0.20
DPO
Table 9: Argumentative Writing Model Comparison Results.
24

---

Published as a conference paper at COLM 2025
Comparison
Validity (Quality)
Soft Effective Semantic Diversity
Hard Effective Semantic Diversity
W (p)
ES (d)
Winner
(p)
ES (d)
Winner
W (p)
ES
Winner
Base Inst.
0.001
1.27
Inst.
0.001
1.05
Inst.
0.002
0.86
Inst.
Small vs. Lg.
0.859
-0.32
Small
0.645
-0.34
Small
0.546
-0.23
Small
SFT vs. 0.339
Pref.
-0.22
SFI
SFT vs. DPO
0.14
DPO
0.970
0.01
DPO
0.151
-0.10
SFT
SFT vs. RL
0.020
0.72
RL
0.039
0.34
RL
0.426
0.09
RL
DPO RL
0.055
0.24
RL
0.164
0.13
0.203
0.12
RL
Comparison
Lexical Diversity
Raw Neural Diversity
(p)
ES (d)
Winner
W (p)
(d)
Winner
Base Inst.
0.001
1.20
Inst.
0.001
-3.16
Base
Small vs. Lg. 0.036
-0.58
Small
0.001
-0.48
Small
SFT vs. Pref.
0.759
-0.13
SFT
0.001
-1.64
SFT
SFT vs. DPO
0.176
-0.31
SFT
0.001
-1.32
SFT
SFT vs. RL
0.301
0.16
RL
0.008
-2.19
SFT
DPO vs. RL
0.027
0.45
RL
0.734
0.36
RL
Table 10: Brainstorming/ Creative Assistance Model Comparison Results.
25

---

Published as a conference paper at COLM 2025
#H# Input Description:
1. integer NO V) (1 N(N V) 18000), representing some quantity or size.
##* Example Input:
1000
#*# Function Signature:
Write a function "f(N) that takes in the input.
"ppyhhnn
def f(N: int):
N: an integer
Now please implement the function f; do not return anything, the function should print the result of the operation.
should terminate within seconds. First describe the function you would write, then implement it. The following function will print out the square of the input number. We will take the square using the ** operator in Python within the print statement.
def f(N: int):
print(n**2)
### Input Description:
1. floating point number NCN (1 AC \) 10000), representing quantity or size.
### Example Input:
143.23 23
### Function Signature:
Write a function *f(N)* that takes in the input.
"-python def f(N: float):
N: float
Now please implement the function f; do not return anything, the function should print the result of the operation.
should terminate within 30 seconds. First describe the function you would write, then implement it.
The following function will calculate the number of times the input number can be divided 2 before it becomes less than 1. We will increment counter variable i each time divide the number by inside while loop.
def f(N: float):
i = 0 while N 1:
N = 2 ++ print(i) derolennddscription
Now please implement the function f; do not return anything, the function should print the result of the operation.
should terminate within 38 seconds. First describe the function you would write, then implement it.
Figure 9; Two-Shot Chain-of-Thougge Prompt: Our template for our two-shot Chain-of- Thought prompt, where the problem description would be input near the end inside the curly braces.
26


Reviews:
[{"review_id": "jRTVKRx4LC", "reviewer": "Reviewer_p6Pc", "summary": "The authors introduce Effective Semantic Diversity, a metric where an LLM generation counts toward diversity only if it (a) passes a simple functional-correctness check and (b) is semantically distinct from repeated samples (K). \nThey build a 108-prompt benchmark derived from CodeNet and AlphaCode, and sample several programs per prompt from dozens of Llama-family models  trained with SFT, DPO, PPO and GRPO, and test many temperature and nucleus-sampling variations. The authors promise to release a Docker runner, AST validator, and test-case generator, which is a nice touch but a bit future-looking.\n\nI read this paper as an attempt to move the \"diversity\" discussion in LLM generation beyond the usual surface‐level token diversity measures. Some of the results are genuinely interesting, and the authors have put significant effort into the empirical work. A few citations for metrics/applications of diversity metrics in synthetic data are missing from the Introduction and Related work: [1-5]\n\n[1] Yu et al. (NeurIPS 2023): \"Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Bias\"\n\n[2] Divekar et al. (EMNLP 2024): \"SynthesizRR: Generating Diverse Datasets with Retrieval Augmentation\"\n\n[3] Chen et al. (http://arxiv.org/abs/2410.15226): \"On the Diversity of Synthetic Data and its Impact on Training Large Language Models\"\n\n[4] Miranda et al. (https://arxiv.org/abs/2306.13840): \"The diversity coefficient as a data quality metric for variability in natural language \ndata\"\n\n[5] Barnes et al. (ACL WASSA 2023): \"ChatGPT is fun, but it is not funny! Humor is still challenging Large Language Models\"", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) There's a strong clarity in the problem motivation, and ESD-like metrics are much-needed in long-form synthetic data. The authors nail why token-level diversity is unsatisfying, and offers a metric that rewards useful variety. I like that the \"validity × diversity\" framing because it is a simple tradeoff and easy to compute in domains where a runnable oracle exists. An additional metric like AUC would be useful for comparing models. \n\n2) with 19 LLMs x four training regimes x three sampling schemes x non-parametric statistics, I think the authors have given the conclusions good grounding in experiments. The ablations identify where diversity gains origninate from.\n\n3) While the core contribution is analysis-based, the appendix provides a stability property for the metric, which which is reassuring as it means rankings probably will not flip erratically with more samples. It's a good attempt to put the ESD metric on a theoretical footing.\n   \n\n4) The finding that small models are parameter-efficient at producing unique valid programs is a good fit for the \"Technological Impact\" dimension. It can be expanded a bit more.", "reasons_to_reject": "1) My biggest issue with this paper is the framing of it. All evidence comes from code generation. It is too restricted in domain to make such general claims about the goodness of certain RL approaches. The paper repeatedly hints that the insights will transfer to other open-ended tasks, but there are no tests of these claim. Without at least a pilot natural-language study, it's hard to be convinced. One option is to reframe the paper to focus on code generation...this is a more limited but still interesting contribution and I think would stand on its own.\n\n2) It would be good to see a small experiment on a different domain to back the broader claims. The authors could run a paraphrase generation task with a grammar checker as the validity oracle, for example. \n\n3) Validity oracle is a bit weak for code. A program passes if it runs without error and prints anything... there's many situation where degenerate code can be \"valid\". Several prompts in the appendix look vulnerable to this loophole. This will artificially inflating ESD.\n\n4) Needs more transparency on computing costs of method for reproducibility, as CoLM guidelines explicitly ask reviewers to weigh accessibility of compute (hardware specs, GPU-hours, etc).\n\n5) Minor typos / wording issues:\nL82: \"undersirable\" → undesirable                               \nL160: Section heading \"Semantic funtion\" → function              \nL240–241: \"the the additional results\"\nL137–138: \"more / less diverse diverse than …\" \nFig. 5 caption (L559) \"semantic diveresity\" → diversity                          \nL559–563: Figure 5 caption begins with a lowercase clause and no period\nL 71–73:   Sentence beginning \"Additionally, in an attempt to understand [...]\" ends abruptly.\nL195–197: \"1 N.B.\" inside main text interrupts flow, can be a footnote\n(Multiple places): Phrases like \"very clear trend\" or \"critical moment\" feels super journalistic. More careful language and quantitative results (i.e. numbers from expts section) would help the results speak for themselves.", "questions_to_authors": "1) Could you report even a tiny natural-language experiment? say, paraphrase generation with a grammar checker as the validity oracle to back the broader claims? This would give more confidence, and I am open to increasing my score during rebuttal if this was provided, as it directly counters my main issue with this work.\n\n2) Have you tried stricter validity checks (e.g., unit tests with expected outputs) to see whether your diversity ranking is robust? A short experiment on a subset would suffice.", "extra_scores": {}}, {"review_id": "v0M07g1VVh", "reviewer": "Reviewer_7jf8", "summary": "This paper focused on the influence of different preference-tuning methods on LMs' generation diversity. Through a new metric proposed, this paper obtained a viewpoint against consensus, that is, preference-tuning can bring beneficial influence on LMs' generative diversity. Several models and preference-tuning methods are compared in an open-ended programming task.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper brought two valuable insights on the evaluation of generation diversity:\n- Firstly, generation validity should be one of the necessary prerequisite conditions.\n- Secondly, semantic diversity might be more valuable than syntactic and lexical diversity.", "reasons_to_reject": "- Weak and Insufficient Empirical Setting\n  - Although ideation and creative assistance are core targets of generation diversity (lines 23-25), this paper only evaluated their metric on a programming task, in which creativity and diversity are not as important as other tasks (i.e. creative writing, scientific hypothesis proposal and so on). \n  - Considering the main contribution of this paper is the proposed metric, the sufficient empirical study on diverse tasks is particularly essential. Thus, I believe more tasks should be considered in the experiments.\n- (Perhaps) Overclaim on the effectiveness to eliminate correlation with temperature\n  - It is clear that the metric proposed does not demonstrate a “linear correlation\" agaist temperature.\n  - However, correlation represented by higher-order functions still exists in the proposed metric, that is, the score in the new metric is significantly higher in the middle (independent with the models and methods).\n\nI look forward further discussion with the authors and will reconsider my assessment after my concerns are addressed.", "questions_to_authors": "- What further insights can this metric bring to the design of reinforcement learning algorithms for LMs?", "extra_scores": {}}, {"review_id": "w1XBYdPy4b", "reviewer": "Reviewer_rZeC", "summary": "This paper introduces a framework for evaluating the effective semantic diversity of outputs generated by large language models (LLMs), focusing on diversity among responses that meet certain quality thresholds. Specifically, they use the program generation task as the open-ended text generation task to evaluate, in order to avoid using LLMs to evaluate LLMs for diversity. The study reveals that preference-tuning techniques, such as RLHF methods like PPO and GRPO, while reducing lexical and syntactic diversity, surprisingly enhance effective semantic diversity compared to supervised fine-tuning (SFT) or base models. This enhancement arises not from increasing diversity within high-quality outputs but from generating a higher proportion of high-quality outputs overall. The research also uncovers that smaller models are more parameter-efficient in producing unique content within a fixed sampling budget.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The diversity of LLM generated content is an interesting topic. \n\n* The idea of using the program generation task as the open-ended text generation task to evaluate diversity is an brilliant. Since we have many tools like program execution and abstract syntax tree to subjectively evaluate the diversity and effectiveness of generated content on this task.\n\n* They evaluate various LLMs with different settings (base/SFT/DPO/etc.), and report the statistical value like p-values and Cohen’s D to show the significance of some assumptions about the diversity of LLMs' generation. \n\n* The findings, like \"RLHF methods, while reducing lexical and syntactic diversity, enhance effective semantic diversity compared to SFT\", are potentially helpful for the researchers in the area of LLMs.", "reasons_to_reject": "* Using the program generation task is an brilliant idea. However, it still brings the limitation that models' performances on program generation are not necessarily consistent to the general performances.\n\n* The sec. 5, \"Erimental Results\" is a little bit short, lack more detailed explainations and discussions about the experimental results. Instead of the p-values and Cohen’s D, some tables/figures to show the absolute measurement of semantic diversity is also meaningful. (Figure-3 is not very clear, since it contains so many information.)\n\nMinor problem:\n* Figure-1 is a little bit ambiguous. It takes me a few minutes to figure out that the three model names (Llama 3.1 70B Instruct, etc.) are not aligned with the three sub-figures respectively.", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper evaluates the diversity of generated code, conditioned on generation quality (functional correctness), as well as some baseline diversity measures. They present several comparisons between different post-training approaches based on their proposed measure, and show that preference tuning might actually increase the “diversity” of generated code.

*Strengths*

- Brings into focus a critical aspect of generated code: its validity, even though the paper uses a weaker version of validity as pointed out by some reviewers. Regardless this emphasis on validity / diversity is valuable.
- The paper displays very strong empiricism, also evidenced in the author response.


*Weaknesses:*

- Given the emphasis on code generation, this paper will benefit from scoping its title and abstract better, are more narrowly. This is regardless of the additional empirical results in creative writing, where by the authors’ own admission: “domains like creative writing or essay-writing may be more open-ended and lend themselves better for semantic diversity measurement.”
- Some of the language in the paper seems somewhat non-standard for a scientific paper, this is something the authors promise to fix.

It would have been an easy decision if the paper did not make broader claims about all language, but with a narrower scoping will really benefit the paper.
Decision: Accept