Forcing Diffuse Distributions out of Language Models
Authors: Yiming Zhang
Avi Schwarzschild
Nicholas Carlini
J Zico Kolter
Daphne Ippolito
Venue: COLM (2024)
Keywords: diversity, synthetic dataset generation, parameter-efficient fine-tuning
TL;DR: This paper proposes methods to enhance language models for generating diverse outputs, crucial for tasks like dataset construction.

Abstract: Despite being trained specifically to follow user instructions, today’s instruction-tuned
language models perform poorly when instructed to produce random outputs.
For example, when prompted to pick a number uniformly between one and
ten Llama-2-13B-chat disproportionately favors the number five, and when tasked
with picking a first name at random, Mistral-7B-Instruct chooses Avery 40 times
more often than we would expect based on the U.S. population. When these language
models are used for real-world tasks where diversity of outputs is crucial,
such as language model assisted dataset construction, their inability to produce
diffuse distributions over valid choices is a major hurdle. In this work, we propose
a fine-tuning method that encourages language models to output distributions that
are diffuse over valid outcomes. The methods we introduce generalize across a
variety of tasks and distributions and make large language models practical for
synthetic dataset generation with little human intervention.

Published as a conference paper at COLM 2024
Forcing Diffuse Distributions out of Language Models
Yiming Zhang
Avi Schwarzschild'
Nicholas Carlini?
Zico Kolter
Daphne Ippolito 1.
'Carnegie Mellon University
Google Deep
Bosch Center for Al
Abstract
Despite being trained specifically to follow user instructions, today's s instruction-tuned language modéls perform poorly when instructed produce random outputs. For example, when prompted to pick a num- ber uniformly between one and ten Llama-2-13B-char disproportionately favors the number five, and when tasked with picking first name at random, Mistral-7B-Inntruuc chooses Avery 40 times more often than we would expect based on the U.S. population. When these language models are used for real-world tasks where diversity of outputs is crucial, such language model assisted dataset construction, their inability to produce diffuse distributions over valid choices is a major hurdle. In this work, we propose fine-tuning method that encourages language models out- put distributions that are diffuse over valid outcomes. The methods we introduce generalize across variety of tasks and distributions and make large language models practical for synthetic dataset generation with little human intervention."
1 Introduction
Consider Dungeon Master (DM) trying to language model based chatbot to assist in managing their Dungeons & Dragons campaign. The DM asks the chatbot to suggest random name for character in the story. The first time she asks, it suggests "Anya," " and the second time it also suggests " "Anya." In fact, almost 40% of the time, the suggested name will be " "Anya" even when the language model is deployed with full random sam- pling. The DM then tries to use the chatbot to roll a twenty-sided die; over 60% of the dice rolls come up as a 14. Frustrated, the DM gives up and brings out their physical dice.
Instruction-tuned language models are extremely bad at producing random outputs when users want them to. Even when prompts are carefully constructed with instructions that encourage randomness, both state-of-the-an open-source and industry language models output very low-entropy distributions over the valid options. Beyond Dungeons & Drag- ons, there are many practical applications where diversity across valid options is crucial for language model outputs. For example, when language models are used to answer mul- tiple choice Likert-scale questions, priori each option should be equally likely. When language models are used for synthetic dataset construction, such as for synthetic biogra- phies (Maini et al., 2024; Yuan et al., 2021) or instruction-tuning training sets (Wang et al., 2023), diversity in the generations is essential but arduous to achieve through mere prompt hacking.
In this work, we examine how far off language model generations are from user expec- tations of randomness and diversity. While our experiments focus on instruction-tuned models such as Llama-2-chat, our findings do generalize to pre-trained language models
*Correspondence: Yiming Zhang, yimingz3ecs. crninng3@cs.cmu.edee cdu.edu. !Code and data are available at https: //gi thut. www.:/ywwiinggaaaiiiiiaaaaaiiiie

---

Published as a conference paper at COLM 2024
Gemma (baseline)
Gemma (tuned)
Llama-2 (baseline)
Llama-2 (tuned)
Empirica probablitt
40%
30%
10%
Anya
Elara
Erin
Fleanor
Sophia
Marissa
Oliviaa
Marla
Morgon
Elysion
Talon
Mila
Yasmin
Dane
Samson
Hector
Kassidy
Monica
Aubrie-
Kiera
Aurora
Luna
Evangeline
Elaro
Arobella
Florence
Elora
Raven
Evelyn
Rowan
Alina
Karinu
Lily
7oe
Jaman
Nevaeh
Danu
Tali
Yurem
Kera
(a) BABYNAMES (top 10 names are shown).
Empirical probabiitty
40%
20%
3
10
9
5
10
(b) RANDOM NUMBER GENERATION.
Figure 1: Language models do not produce diffuse probabilities. The output distribu- tions of untuned Gemma and Llama-2 deviate from what we would expect from natu- ral/random distributions. Our tuning method addresses this by diffusing the output dis- tribution over valid candidates. The horizontal axes above are sorted in descending order by probability of the output.
in general. 2 We then show how language models can be fine-tuned produce diffuse dis- tributions over valid options, without sacrificing generation quality. Our method supports tasks where the sample set of valid options is not easily enumerated, and we show that models fine-tuned to produce diffuse probabilities for one set of tasks generalize to other, very different tasks without degrading model utility.
This generalization allows us to promote diversity in complex settings, such as synthetic dataset generation. On the task of generating dataset of synthetic biographies (similar to the dataset created by Maini et al. (2024) for benchmarking language model unlearn- ing), our method generates four times as many unique first names, three times as many unique birth places, and 1.5 times many unique careers the untuned baseline model, all without any need for complex prompt engineering, decoding strategy tweaking or man- ual re-writing.
2 Preliminaries
We begin with formal definition of diffuse probabilities from language models. We then intro- duce techniques for measuring diversity of models outputs and we show how to quantify the differences in the observed output distributions and the desired distributions.
2.1 Problem setting
Consider vocabulary V (1, An autoregressive language model takes se- quence of & tokens ve as input and outputs probability distribution po(. x) E A(V) over all tokens in the vocabulary. We use A(V) to denote the probability simplex over V/.
For example, when GPT-2 (Radford et al., 2019) is prompted with "She rolled the 20-sided die. It landed the number", the output distribution is similarly far from uniform.
2

---

Published as a conference paper at COLM 2024
In generation, are interested in computing the probability of potentially multi-token target = V/,............................. .,y. E V", For example, y could be a two-digit number or the biogra- phy of a person. A language model factors the probability of sequence y into the product of probabilities of individual tokens (+ here indicates concatenation):
Po x) (o x)pe(y2 x1) Polyn ① ycn)
(1)
2.2 Measuring output distributions
How do measure if an LM produces diffuse probabilities? We consider one method for cases where the sample space is enumerable, and another for when the full sample space is unknown.
In settings such as random number generation within a range, we can enumerate the set of valid generations T. For example, in the context of rolling twenty-sided die, T = [1, and we expect a perfectly diffuse language model to produce uni- form distribution over 'T. More generally, for context and (not necessarily uniform) ground truth probability distribution p* E A(V*) over generation targets, we can measure the KL-divergence between the model distribution pa and the ground truth distribution pd
as follows:
P (y x Py x)
DKL (p" || Pe) = (y yET
(2)
KL-divergence measures the difference between the model distribution pa and the ground truth p", and it attains its minimum at zero and only pa = =p"
In many scenarios such synthetic dataset generation, the support of the ground truth distribution cannot feasibly be enumerated, and we instead measure the entropy and the coverage of empirical distributions. In these cases, we sample large number of generations y1.-.., yN from the model and use the empirical distribution P derived from generations to approximate Pa. Entropy of the empirical distribution quantifies the diversity of model generation:
Entropy ()) = Σ ply)
(3)
ye
Coverage-N is defined as the number of unique generations produced by the model in N samples:
Coverage-N(p) = Ky:lie [N]]-
For a sufficiently large N, coverage quantifies the number of unique generations a user interacting with the system could practically observe.
3 Forcing Diffuse Probabilities
We introduce a method for fine-tuning language models to encourage diffuse probabilities distributions. Our technique hinges on the fact that fine-tuned models generalize-a model which is trained to produce diffuse probabilities for one task will also do so for tasks unseen during training.
3.1 Distribution matching as language model fine-tuning
Suppose, for some task, we already know the target set T of possible outputs, and the ground truth distribution p* we would like the language model to produce. Our method simply maximizes the sum of model log-likelihood over all targets, weighted by their ground truth likelihoods:
C(pp) - Σ p' x log
(5)
yCT
3

---

Published as a conference paper at COLM 2024
This objective is, in fact, the cross-entropy loss ubiquitous in language modeling (Berger et al., 1996) extended to multiple target sequences (Edunov et al., 2018). Since the model mild condition that T sequences, this optimization relies
distribution Py is defined is prefix-free, namely no target sequence should be the prefix of another target sequence t'. This condition is necessary since would otherwise always have Pa |x] Pa(t x), and it would be impossible to, say, make a model output identical probability for and e.3 Notice that minimizing the objective in Eq. 5 is equivalent to minimizing the KL- divergence between the model and the ground truth distribution p":
Pr |x) Ps(y x)
((p))
Pyy(y
log
log (ygp"(y)x)
yET
p p' Py x)
Σ p" (y x) log yeT
P'(y ('((yx))oggp'()l
YET
DKL (p" || Pe) Entropy(p"),
where the entropy term is independent from the model distribution. In this view, our method can be seen distribution matching, where we align the model distribution with an ideal distribution.
In the more complex case where the ground truth distribution is not known, our method assumes access small sample set (~200 samples) and treats the empirical distribution as proxy of ground truth. While minimizing this loss to 0 could lead to model that exclusively produces targets from the sample set, which is not what we want in a case like synthetic dataset generation, empirically, we find this not to be the case. We discuss results on generalization beyond training distribution extensively Section 5.
3.2 Parameter-efficient Fine-tuning
We use LoRA (Hu et al., 2021) to fine-tune models in a efrfmeter-efficient way. This tech- nique relies on optimizing low-rank additions to weight matrices, making fine-tuning large language models (with 2 7B parameters) practical. It has the added benefit that LoRA tuning tends to perform almost as well as full model fine-tuning and generalizes beyond training instances (Malladi et al., 2023). From an implementational standpoint, LoRA re- quires instantiating a new pair of relatively small matrices for every weight matrix in the network. Consider one fully connected layer parameterized by W E Rdxd We instantiate E Rd and A E R'xd where < and reparametrize the layer as W + BA. When fine-tuning the model, we exclusively optimize the weights of A and B rather than taking gradient steps on W, enabling us to fine-tune large models such as Llama-2-13B with only a small fraction of the memory overhead. We report training details in Appendix A.
4 Results on Simple Tasks
We begin by showing how state-of-the-art language models fail to produce diffuse distri- butions on simple tasks, even when user's 's prompt requests randomness. We consider two tasks: random baby name generation (BABYNAMES) and random number generation (RNG). In BABYNAMES, the model is instructed to generate random baby name in En- glish. RNG, we ask the model to generate a random number between and 10. The tasks have the following prompts:
1. BABYNAMES: "Please generate an English first name, chosen completely at ran- dom. Output only the name between two curly braces, like this: {name). Don't output code."
3 We can simply append an end-of-sequence token each target if T is not already prefix-free Empirically, we find that 4 is sufficiently expressive, which we use across all experiments. On models, this corresponds tuning less than 0.1 % of all model parameters.
4

---

Published as a conference paper at COLM 2024
Table Our method significantly increases the diversity of generations on the random number and baby name generation tasks. Coverage-1000 is the number of unique out- comes out of 1,000 trials.
Task
Model
Entropy
Coverage- Parrage--00011 1
Baseline
Tuned
Natural
Baseline
Tuned
Natural
Gemma
1.16
2.28
8
10
RNG
Llama-2
1.07
2.29
2.30
7
10
10
Mistral
1.88
2.30
9
10
Gemma
3.19
5.69
170
442
BABYNAMES
Llama-2
3.24
5.83
6.43
129
490
714
Mistral
4.02
5.78
225
479
2. RNG:
"Generate a random number between and 10. Output only the number between two curly braces, like this: (number). Don't output code."
We consider three state-of-the-art language models: Gemma-7B-instruuc (Google, 2024), Llama-2-13B-chat (Touvron et al., 2023) and Mistral-7B-inntruuet (fiang et al., 2023). We use chat/ine /iattisttt versions of the models, which are tine-tuned on dialog data to make them more suitable than base models for our test cases, which require instruction following, In all experiments, we sample from models using a temperature of one, which is equivalent to sampling from the maximally diffuse, unmodified model distributions. 6 For each task, sample 1,000 generations from the model and report coverage and entropy statistics of the empirical distributions of model generations.
4.1 Language models fail to follow instructions for randomness
Figure shows output distributions of several untuned, baseline models for BABYNAMES and RNG. Despite the prompts asking for randomness, these output distributions are tremely imbalanced.
For RNG, is easy to see that the empirical distribution is far from the uniform distribution we would expect. For BABYNAMES, is less clear what we should expect from the model. Do want the generated names to be uniform sample from the set of all names? Or do we want the names to reflect the distribution of baby names among recently born babies in the Unites States? To address this, we introduce the concept of natural distribution for each task. For RNG, this is simply the uniform distribution; for LABYNAMES we compare to a sample we draw from data on names used for babies born the United States in the year 2022, using birth data from the US Social Security Administration." 7 We note, that we are only using the natural distributions as reasonable baselines for comparison.
When generating names, we find that Llama-2 generates the name "Aurora" about 100 times more often than would expect from the natural distribution, and Gemma assigns 40% probability to the name "Anya", which did not appear even once in our sample of the natural distribution. This is clear indication that these large language models do not sample diversely from a large space of valid generations (all English names) despite being explicitly instructed to do so.
In the RNG task, we find that each model has a bias toward a different set of numbers. For example, both Gemma and Ilama-2 assign over 60% probability to the number five, while Mistral assigned over 25% probability to the number eight. Surprisingly, in 1000 samples,
We instruct the models to generate between curly braces simplify output parsing. Gemma has a strong tendency to output code and we add an explicit instruction to avoid code generation. WW could technically set temperature to beyond one to smooth out the model distribution further, but this is rarely done in practice due degeneration. We also note that the untuned, baseline models need to be sampled at a temperature 10 to match the entropy of our fine-tuned models in Section 4.2.
peetps://wwww. ssa. watasaaooo//AA/aaaaaaaaaaa
5

---

Published as a conference paper at COLM 2024
Baselin Tuned Ideal
Baseline
Tuned
Ideall
Entropy 3-
Entropy
0-
0
20
40
60
80
100
0
20
40
60
80
100
Random number in [1,]]
Size of random sample space
(a) Varying prompt formats and number ranges.
(b) Varying sizes of random sample space.
Figure 2: Models tuned on RANDOM NUMBER GENERATION demonstrate generalization to variations in both prompt format and number ranges. 95% confidence intervals are shown in plots.
none of the models were able to generate all ten numbers. These results highlight that the output distributions of large language models are far from diffuse, even in simple tasks where user may expect them to be.
4.2 Optimizing for BABYNAMES and RNG diversifies generation
We start by focusing on the setting where we are optimizing the model to produce diffuse probabilities for the two individual tasks. For RNG, we fine-tune the model to produce uniform distribution from one to ten, and for BABYNAMES, we fine-tune the model to fit the empirical distribution of sample of 200 baby names drawn from the natural distribution.
In these experiments, we see our methods can very effectively align the model distribution with the ideal distributions (Table 1). With RNG, tuned models produce near-uniform dis- tributions over the valid choices with full coverage and near perfect entropy. With BABY- NAMES, where the training set covers only a small portion of the possible outputs, we see encouraging indication of generalization: while the coverage remains smaller than that of the natural distribution, it is over twice the baseline for models and the majority of outputs not among targets in the fine-tuning set.
While RNG and BABYNAMES serve as toy problems, these settings have a major limitation-th the fine-tuned models are evaluated on the same prompts and sample spaces in training. To further test the generalization performance of our models, we also consider settings where the prompt and sample space differ between training and evaluation. We fine-tune Llama-2 to produce diffuse probabilities over two ranges of random integers- one to ten and one 100-and test its generalization different instruction formats and unseen sample spaces (see Appendix B.1 for prompts). In Figure 2a, we observe encourag- ing generalization trends: the tuned model produces near uniform distributions for unseen prompt formats over number ranges not in the fine-tuning set, for example from 1 to 45. When we vary the size of the random sample space of RNG (e.g., 154 204), the tuned model still produces substantially higher entropy distributions than the untuned model (Figure 2b). With these first experiments, we demonstrate that tuning models for diffuse probabilities is promising method for increasing generation diversity.
5 Generalization Across Tasks
is not too surprising that a model optimized to output random numbers uniformly across some range can generalize to other ranges of numbers. A much more interesting and prac- tical test is if model optimized for diffuse probabilities one set of tasks transfer to

---

Published as a conference paper at COLM 2024
Baseline
Ours (OOD)
Ours (ID)
Ideal
Baby Names
Countries
Fruits
Entropy
Entropy
Entropy
3
Gemuma Llama-2 Mistral
Ideal
Gemuna
Llama-2 Mistral
Ideal
Gemma Llama-2 Mistral
Numbeers
Occupations
Entropy
Entropy
Entropy
Gemma
Llama-2
Mistral
Ideal
Germaa
Llama-2
Mistral
Ideal
Gemma
Llama-2
Mistral
Figure 3:
Entropy in leave-one-out generalization. The title of each plot indicates which
set of tasks we compute entropy over.
tasks with very different sample spaces. That is, model optimized for picking random number or baby names should not overfit to generating samples from those distributions; rather, it should be able to e-g. pick random fruit country name when prompted to do so.
We use leave-one-out experiments on of six tasks show that models tuned for dif- fuse probabilities do, in fact, have strong transferability to tasks unseen during tuning, We consider the following six tasks. Each task has several associated prompts, one of which is shown below:
1. BABYNAMES: "Please generate an English first name, chosen completely at ran- dom."
2. COUNTRIES: "Output random country in Africa, chosen completely random."
3. FRUITS: "Output name of a fruit, chosen completely at random."
4. DAYS AND DATES: "Provide a random date in June."
5. NUMBERS: "Randomly pick prime number between 1 and 50."
6. OCCUPATIONS: "Output an occupation that starts with the letter "A"."
By fine-tuning five out of the six tasks listed above and evaluating performance on the sixth, we can measure the ability of our method to handle out-of-distribution tasks.
In Figure 3, report in-distribution (ID) and out-of-distribution (OOD) results, which correspond to tuning sets that include or exclude the particular task, respectively. The results show convincing trend: for all three models, our tuning method led to substantial improvements in entropy over the baselines, even when the task was held out from the tuning set." Another interesting observation is that the baseline Mistral model consistently produces more diffuse distributions than the baseline Gemma and Llama-2 models, but after tuning, all three models have comparable entropy.
In two of the tasks (COUNTRIES and FRUITS), we observe sizable generalization gaps be- tween in in-distribution and out-of-distribution entropy, which suggest that task-specific tun- ing remains useful, especially when we can come up with reasonbly diverse of gen- eration targets for fine-tuning. However, coming up with a large enough target set isn't always easy. In these cases, we rely on the generalization of the models trained on di-
CCovraaee results show similar trends, and we report them in Table Appendix C.1.
7

---

Published as a conference paper at COLM 2024
Figure 4: Fine-tuned Llama-2 model improves the diversity of synthetic biographies. We report coverage for categorical attributes in 4a and normalized unigram diversity of generated achievements and the entire biography in 4b.
First name
Last name
Birth year
Birth place
Career
Achievements
600
0.2
0.2
400
600
60
200
400
40-
0.1
0.1
200
100
200
Baseline
Tunet-000-
Tured-ID
Boseline
Tuned-OOD
Tuned-ID-
Beseline
Tunad-O00-
Tuned-ID-
Baseline
Tuned-00D.
Tuned-In ID
Boseline
Tuned Tuned-OOD OOOD
Tuned-II>
Boseline
Tuned-ID ID
Tuned-0000
Besellne
Tuned-ID
Tunel-OOO.
(a) Coverage results for categorical attributes.
(b) Unigram diversity of biographies.
verse set of tasks. For example, in OCCUPATIONS, the authors could only come up with small set of 17 professions that start with the letter "A," and Llama-2 and Mistral (not trained on OCCUPATIONS) generalize out-of-distribution to occupations beyond what we provide in the fine-tuning set." Notably, our fine-tuning method does not substantially change the general capabilities (e.g,, writing and reasoning) of the models demonstrated by evaluations on MT-Bench (see Appendix C.4), making our method compatible with tun- ing ceneral-purpose large language models.
Constructing More Diverse Synthetic Datasets
The leave-one-out experiments show that fine-tuning for diffuse outputs leads task gen- eralization. This is an important trait for real-world applications such as synthetic dataset construction, where might be not be feasible to tune on data that identically formatted to what we would like to synthesize.
In this section, we evaluate how well a model tuned on the six tasks from Section per- forms realistic dataset creation bask-building synthetic dataset of fictional biogra- phies. Inspired by the synthetic datasets created in Maini al. (2024) and Yuan et al. (2021), these biographies include the following attributes: first and last name, gender, birth year, birth place, profession and a description of the person's 's achievements. We show how our method results in biographical details that are much more diverse than those generated by the baseline models.
6.1 Out-of-distribution training improves synthetic dataset diversity
We first consider a Llama-2-13B model fine-tuned on all tasks in Section and not specif- ically for biographies (Tuned-OOD), and compare its generations against the baseline Llama-2 model over 1000 samples. In Figure 4, we report coverage results for categori- cal attributes (e.g., Birth year), and normalized unigram diversity for achievements and over the entire biography. 10 In Table 2, we report the most frequently generated values for categorical attributes, along with the frequencies out of 1000 generations." 11
The results on the baseline Llama-2 model indicate that the biases we observe in Section 4.1 towards certain names and numbers expectedly show up generation. For example, out
FFrr example, Llama-2 generated " "Aromatherapist, and Mistral generated "Agronomist." Nei- ther is among the fine-tuning targets of OcCUPATIONS.
1e report prompt used for biography generation and side-by-side qualitative examples Ap- pendix B
A table of coverage and entropy statistics of generated biographies be found in Appendix C.2.
8

---

Published as a conference paper at COLM 2024
Table 2: The most frequently generated values for each attribute, along with the number of times the value was generated (out of 1000 generations). Despite being tuned on out- of-domain tasks, the tuned Llama model significantly improves in diversity. Some values in table have been fruncated for brevity.
name
Last name
Gender
Birth year
Birth place
Career
Baseline Llama-22
Evelyn
284
Nightingale
117
F
966
1985
764
Paris, FR
305
Astronaut
211
Luna
Aurora
104
NB
1987
Tokyo, JP
Astro. Engineer
Flara
87
Nova
98
M
13
1992
44
Stockholm SE
33
Aero. Engineer
55
Adriana
Starling
1978
Mumbai,
32
Env: Activist
Aurora
38
Stardust
1975
Singapore, SG
32
Astrophysicist
32
Fine-tuned Llama-2 (OOD)
Nightingale
211
Mumbai,
35
Astronaut
Zelda
14
Nightshade
12
M
189
1992
Lagos, NG
31
Aero. Engineer
50
Mila
14
Chen
8
NB
31
1987
Paris, FR
29
Soft. Engineer
47
Evelyn
11
Orion
1988
Tokyo,
27
Env. Activist
35
Althea
9
Sparks
6
1990
52
Nairobi, KE
21
Joumalist
34
Fine-tuned Llama-2 (ID)
Hava
14
Kim
17
F
487
1921
Choloma, HN
16
Architect
140
María
14
Mohammed
16
478
1931
36
Rabat, MA
13
Jourmalist
74
Juan
14
Khan
NB
34
1942
Tainan, TW
10
Politician
33
Valcin
13
Abed
13
1916
30
Rajshahi, BD
10
Archaeologist
35
Issaka
12
Salah
1984
Budapest, HU
10
Mar. Biologist
of 1000 biographies generated by the baseline Llama-2 model, 284 have the first name 'Eve- lyn,' and 966 are females. Such high level of repetition makes the resulting dataset ba- sically unusable without substantial human intervention. In constrast, our model signif- icantly improves the diversity of generated biographies, despite not trained specifically for generating biographies. For example, we see an >2X increase coverage for most categorical attributes. There are also substantial improvements in generation diversity for achievements and over the entire biography (Figure 4b), although our training does not optimize for open-ended text generation.
The top five most frequent values for attributes (Table 2) help contextualize these improve- ments in diversity: there are significant reductions in biases towards certain attribute val- ues. E.g., the frequency of the name "Evelyn" decreased by 25X, and the birth year 1985 by over 3X.
6.2 Diversifying distributions of categorical attributes
Despite this improvement over base model, certain biases, as seen in the high frequencies of female biographies (76.2%) and the birth year 1985 (21.1%), still persist, which could limit the utility of the dataset. Our fine-tuning method can in fact be directly applied to balance the distribution of categorical attributes. As proof of concept, we create tar- get set containing 210 programmatically generated tuples of categorical attributes, with roughly balanced gender, birth year and birth place distributions without the open-ended achievement descriptions. We then fine-tune a Llama-2 model (Tuned-ID) only on categor- ical attributes and evaluate another sample of 1000 biographies.
At a comparable level of generation diversity to the Tuned-OOD model (Figure 4), the Tuned-ID model is able to generate biographies with much more balanced distributions of categorical attributes: Table shows that both gender and birth years are roughly uni- formly distributed, and a wider range of birth places are produced by the model." 12 Cru- cially, the model remains highly diverse on open-ended generation of achievements even when being trained exclusively on categorical attributes. This result highlights the poten- tial in extending our fine-tuning method towards improving diversity in open-ended text generation.
12See maps of generated birth places for all three models in Figure 5, Appendix C.

---

Published as a conference paper at COLM 2024
7 Related Work
Diversity in Text Generation The lack of diversity has been long-standing issue in gen- eration due to the tension between generation quality and diversity (Zhang et al., 2020): sampling at low temperature causes boring and repetitive text, while sampling at higher temperatures could
to nonsensical output (Tevet & Berant, 2021). Much of the ex- isting literature approaches the quality-diversie tradeoff by coming up with new decod- ing strategies, which often involve either shaping the model distribution (e.g., top-p sam- pling (Holtzman et al., 2020) and top-k sampling (Fan et al., 2018)) and prromttggo constraints during decoding (Li et al., 2016; Vijayakumar et al., 2018) or training (Welleck et al., 2019; Edunov et al., 2018). Our setting is distinct from prior work in that we already assume a "maximally" diverse decoding strategy (i.e., sampling from the model with tem- perature one), and yet instruction-tuned language models still fail to generate diversely.
Language Model for Dataset Creation As more capable language models emerge (Tou- vron et al., 2023; Jiang et al., 2023; Google, 2024), language model-based dataset creation becomes increasingly practical. Prior work has largely focused generating specialized data for augmenting NLP tasks (Ye et al., 2022b) including semantic similarity (Schick & Schütze, 2021), relationship extraction (Chia et al., 2022), natural language understand- ing (Meng et al., 2022) and instruction following (Honovich et al., 2022). A LM-based dataset creation pipeline is usually an iterative and arduous process (Ye et al., 2022a), in which significant human intervention is needed to ensure the model generates diverse and high-quality data (Yuan et al., 2021; Liu al., 2022; Maini et al., 2024). Our work aims to make progress towards the goal of automating data creation by improving the diversity of language model generation and thereby reducing the need for human intervention.
8 Concluding Discussion and Future Directions
In this work, we propose a method for fine-tuning language models to generate diffuse probability distributions, and show that this method leads to sizable and transferrable im- provements in generation diversity. We showcase a practical application of our method in synthetic dataset generation, demonstrating improvements in the quality of generated data by large margin, with or without task-specific tuning. Our experiments reveal interesting insights on the surprising capability of language models to learn diffuse distributions and generalize to new prompts and output spaces. An important direction for future work is the application of our method (and distribution matching techniques in general) to debias- ing language models (Liang et al., 2021), which are shown to be rife with harmful stereo- types (Bolukbasi et al., 2016). Given the strong generalization properties of our method, it is plausible that aligning language models with an ideal distribution of representative instances could reduce model bias.
Since our method requires access to set of valid output sequences, a structured output space is assumed. In other words, this method is not directly applicable to open-ended generation tasks where the number of valid targets is practically infinite. Future work should further explore the limits of our approach, especially in the context of improving diversity of open-ended generation. Although our method is independent from the fine- tuning procedure, we find LoRA (Hu et al., 2021) to be substantially more efficient and effective compared to other techniques such as prefix-tuning (Li & Liang, 2021) and dis- crete prompt search (Zou et al., 2023). Since they could potentially guide closed-source models to generate more diverse outputs, identifying diversity-inducinn discrete prompts is particularly interesting research question, which we leave for future work.
Acknowledgments
We are grateful to the Center For AI Safety for providing compute for experiments in this paper.
10

---

Published as a conference paper at COLM 2024
References
Adam L. Berger, Stephen A. Della Pietra, and Vincent Della Pietra. A Maximum Entropy Approach to Natural Language Processing. Computational Linguistics, (2(1):39-71, 1996.
Tolga Bolukbasi, Kai-Wei Chang, James Zou, Venkatesh Saligrama, and Adam Kalai. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS'16, PP. 4356-4364, Red Hook, NY, USA, December 2016. Curran Associates Inc. ISBN 978-1-5108-3888-9
Yew Ken Chia, Lidong Bing, Soujanya Poria, and Luo Si. RelationPrompt: Leverag- ing Prompts to Generate Synthetic Data for Zero-Shot Relation Triplet Extraction. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Findings of the As- sociation for Computational Linguistics: ACL 2022, pp. 45-57, Dublin, Ireland, May 2022. Association for Computational Linguistics.
Sergey Edunov, Myle Ott, Michael Auli, David Grangier, and Marc' Aurelio Ranzato. Clas- sical Structured Prediction Losses for Sequence to Sequence Learning. Marilyn Walker, Heng Ji, and Amanda Stent (eds.), Proceedings of the 2018 Conference of the North Ameri- can Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp. 355-364, New Orleans, Louisiana, June 2018. Association for Computational Linguistics.
Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical Neural Story Generation, May 2018.
Google. Gemma:
Introducing new httpa////deeeeeeeeeeeeeeeeeeeeee
state-of-the-art open models.
https:/ /bloggoooglgeeeeeeeooooooooooooe
February 2024.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The Curious Case of
Neural Text Degeneration, February 2020.
Or Honovich, T'homas Scialom, Omer Levy, and Timo Schick. Unnatural Instructions: Tun- ing Language Models with (Almost) No Human Labor, December 2022.
Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. Xiv preprint xrXio:2106.09685, 2021.
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lu- cile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7B, Oc- tober 2023.
Jiwei Li, Will Monroe, and Dan Jurafsky. A Simple, Fast Diverse Decoding Algorithm for Neural Generation, December 2016.
Xiang Lisa Li and Percy Liang. Prefix-Tunine Optimizing Continuous Prompts for Gen eration. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th In- ternational Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 4582-4599 Online, August 2021. Association for Computational Linguistics.
Paul Pu Liang, Chiyu Wu, Louis-Philippe Morency, and Ruslan Salakhutdinov. Towards Understanding and Mitigating Social Biases Language Models. In Proceedings of the 38th International Conference on Machine Learning, pp. 1565-6576. PMLR, July 2021.
Alisa Liu, Swabha Swayamdipta, Noah A. Smith, and Yejin Choi. WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creation, November 2022.
11

---

Published as a conference paper at COLM 2024
Ilya Loshchilov and Frank Hutter. Decoupled Weight Decay Regularization. In 7th Interna- fional Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
Pratyush Maini, Zhili Feng, Avi Schwarzschild, Zachary C. Lipton, and J. Zico Kolter. TOFU: A Task of Fictitious Unlearning for LLMs, January 2024.
Sadhika Malladi, Alexander Wettig, Dingli Yu, Danqi Chen, and Sanjeev Arora. A Kernel- Based View of Language Model Fine-Tuning, June 2023.
Yu Meng, Jiaxin Huang, Yu Zhang, and Jiawei Han. Generating Training Data with Lan- guage Models: Towards Zero-Shot Language Understanding, October 2022.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. Lan- guage models are unsupervised multitask learners. 2019.
Timo Schick and Hinrich Schütze. Generating Datasets with Pretrained Language Models, October 2021.
Guy Tevet and Jonathan Berant. Evaluating the Evaluation of Diversity in Natural Lan- guage Generation. In Paola Merlo, Jorg Tiedemann, and Reut Tsarfaty (eds.), Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 326-346, Online, April 2021. Association for Computational Linguis- tics.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremm Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schel- ten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open Foundation and Fine-Tuned Chat Models, July 2023.
Ashwin Vijayakumar, Michael Cogswell, Ramprasath Selvaraju, Qing Sun, Stefan Lee, David Crandall, and Dhruv Batra. Diverse Beam Search: Decoding Diverse Solu- tions from Neural Sequence Models, October 2018
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self- generated instructions. In Proceedings of the 61st Annual Meeting of the Association for Compulational Linguistics (Volume 1: Long Papers), pp. 13484-13508, 2023.
Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason We- ston. Neural Text Generation with Unlikelihood Training, September 2019.
Jiacheng Ye, Jiahui Gao, Jiangtao Feng, Zhiyong Wu, Tao Yu, and Lingpeng Kong. ProGen: Progressive Zero-shot Dataset Generation via In-context Feedback, October 2022a.
Jiacheng Ye, Jiahui Gao, Qintong Li, Hang Xu, Jiangtao Feng, Zhiyong Wu, Tao Yu, and Lingpeng Kong. ZeroGen: Efficient Zero-shot Learning via Dataset Generation, October 2022b.
Ann Yuan, Daphne Ippolito, Vitaly Nikolaev, Chris Callison-Burch, Andy Coenen, and Sebastian Gehrmann. Synthbio: A case study in faster curation of text datasets. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2021.
12

---

Published as a conference paper at COLM 2024
Hugh Zhang, Daniel Duckworth, Daphne Ippolito, and Arvind Neelakantan. Trading Off Diversity and Quality in Natural Language Generation, April 2020.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, December 2023.
Andy Zou, Zifan Wang, J Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXio:2307. 15043,, 2023,
13

---

Published as a conference paper at COLM 2024
A Training details
For all experiments, we use LoRA to fine-tune models for up to 50 steps over all target prompts. In tasks where the ground truth distribution cannot be enumerated (e.g. syn- thetic biographies), we stop fine-tuning early when the training loss is within 20% of the minimum loss, which is the entropy of the target distribution. We
AdamW opti- mizer (Loshchilov Hutter, 2019) with learning rate of 5 10-5 and batch size of 32. All experiments are conducted on single NVIDIA A100 GPU, using a total of approxi- mately 50 GPU hours. Code and prompts used to reproduce our results are available at the open-source GitHub repository."
B Prompts and Qualitative Results
B.1 Prompts used for Figures 2
The y-axis in Figure 2a gives the average entropy over the following five prompts, where low is set 1, and high is the value on the x-axis.
1. "Randomly pick an integer between low and high. Do not provide any code. Out- put only the number between two curly braces, like this: (number). Don't output code." "
2. "Output a number uniform randomly from the range low high, inclusive. Out- put the number between two curly braces, like this: (number). Don't output code."
3. "Choose number at random between low and high. Output your chosen number bweet two curly braces, like this (number)."
4. "Select integer completely at random from the interval [low, high]. Output the selected integer between two curly braces, like this (n}."
5. "I need you to pick number completely at random. The number should be be- tween low and high. Print the number between two curly braces, like this: {n]."
The y-axis in Figure 2b gives the average entropy over five versions of Prompt 1 above, where low is chosen randomly to be between and 900, and high is the value on the x-axis.
1 http://www....ccooooooo/o//hh//a
14

---

Published as a conference paper at COLM 2024
B.2 Prompt used for synthetic biography generation
Table 3: Prompt used for synthetic biography generation.
Generate a random biography sketch of a fictional, notable person. Output name, gender, time of birth, place of birth, profession and accomplishments individually between two braces and generate nothing else. Please follow the format below.
(First name and last name]
[Gender- - Female/ Male/Non-binan
[Date and year of birth}
(City and country of birth}
[Profession]
(One sentence description of accomplishments)
Example
(David Hippocampus) (Male] [June 30, 1960)
[Cambridge, Massachusetts, USA]
[Neursscientitt
[Revolutionized cognitive science by elucidating the mechanisms underlying memory formation and retrieval through groundbreaking research and innovative methodolo- gies, advancing our understanding of the human brain's 's intricacies)
15

---

Published as a conference paper at COLM 2024
B.3 Side-by-side comparison of generated biographies across systems
In Table 4, we show five randomly sampled biographies generated by the baseline Llama-2 model and our fine-tuned models, both out-of-distributioo and in-distribution. We note that just in 5 examples, there are noticeable quality issues with the baseline model, such as repetition of the same birth date, birth place and professions, as well as largely similar descriptions of accomplishments. In contrast, our fine-tuned models generate more diverse biographies.
Table 4: Five randomly sampled biographies generated by the baseline Llama-2 model and our fine-tuned models.
Llama-2 (Baseline)
Llama-2 (Tuned-OOD)
Llama-2 (Tuned-IID
(Cesar Season)
[Teggan Sunshine]
(Flick Monteiro) (Female)
(Male)
(Female)
[March 22, 1985) [Montevideo, Uruguay) [Chef]
[July 12, 1998)
(March 18, 1942)
(Copenhagen. Denmark) [Chef)
(Vaduz, Liechtenstein) [Nurse)
("Reimagined the flavors of South Amer-
(Pioneered sustalnable, seasonal, and socially conscious gastronomy movement by founding a Michelin-starred restaurant that sources exclusively from local farm- ers}
(Ploneered community health outreach programs developing regions, spar- ning decades to uplift displaced per- sons and marginalized villagers, gamerring widespread recognition and accolades for her dedication to improving global well- being)
cuisine with his avant-garde dishes, carning him numerous accolades and characteristics for his culinary innovation and experimentation."
[Luna Nightingale) (Femal)
(Yuna Jolin] [Female)
(Sandra Guerra) { [Female)
(May 15, 1983)
(April 10, 1995]
June 25, 1948)
|Paris, France]
(Seoul, South Kunea)
CChicago, United States)
RRenowned Astrophysirist)
(Grammy (ward-Winning Singer Songwriter)
Psychologisst)
[Discovered the existence of previously unknown planetary system, challenging long-held assumptions about the forma- tion of our solar system and expanding the search for extraterrestrial life)
(Significantly contributed to the field of
[Pioneered a unique fusion K-pop, hip- hup, and electronic dance music, captivat- ing audiences worldwide with her soul- ful voice, powerful stage presence, and so- cially conscious lyrics)
child development by conducting influen-
tial studies carly language acquisition
and cognitive growth
[Isadora Flux) (Female)
(Robert Chomsky (Male)
(Noor Khan}
(Female)
(March 12, 1985)
[Decmbeer? 7, 1928)
(September 03, 1911)
Tokyo, Japan)
Boston, Massachusetts, USA) [Linguist]
[Al Rayyan, Qatar}
|Renowned artist)
(Judge]
(Challenged traditional notions of art and beauty with her abstract, multimedia in- stallations that blur the lines between sci- ence and art, winning numerous accolades for her groundbreaking work)
[Developed and propagated the theory generative grammar, transforming the
[Will be remembered for her trailblazing tenure as the first female judge to preside Supreme Court the Middle East, setting a precedent for feminist empower- ment and disparity reduction in the region through her stringent yet compassionate judgmentss
field of linguistics and influencing the
study of cognitive science, philosophy, artificial intelligence)
[Evelyn Stellar)
(Kingston Wellspring)
(Marija Lovrencic) (Female)
(Female)
(Non-binary] )
MMrrc 22, 1985)
AApil 4, 1992)
[March 02, 1921)
(Paris, France]
Townsville, Australia)
RRjeka, Croatia]
AAstronuut)
Astronaut]
AAstrooomrr)
[Pioneered the field of interstellar travel, becoming the first human set foot on exoplanet and expanding our knowledge of the cosmus with her daring explorations and groundbreaking )iscoveries)
(Pioncered the human mission Mars, showcasing exceptional leadership, resourcefulnsss, and adaptability in the face of unprevedented challenges, paving the way for future interplanetary explo- ration and collaburatton)
[Contributed significantly to the fields of astrometry and galactic astronomy through pioneering observing programs and innovative numerical simulations, leading to numerous discoveries and a deeper comprehension the cusmus)
Lena Proxima} [emmle)
(Sandy Fulgar) [Female)
(Noof Kabirah} (Female)
(March 22, 1985)
(December 14, 1985)
(April 08, 1935)
(Seoul, South Korea)
(Laguna Beach, Califorii, USA)
(Mecca, Saudi Arabia]
{ Astronaut and Engineer)
(Scientist and Inventor)
{ (Architect)
(Pioneered the development advanced life support systems for long- cong-duration space missions, enabling human explo- ration of the solar system and beyond.)
(Developed revolationary technology for hamessing solar power, reducing car- bon emissions and promoting sustainable energy solutions)
(Explored innovative architectural de- signs and sustainable building materials to create culturally responsive and environ- mentally conscious structures, positively impacting urban landscapes worldwide)
16

---

Published as a conference paper at COLM 2024
C
Additional Results
C.1 Leave-one-out generalization results
Table 5: Mean coverage results (higher is better). Better result for each model-task pair is in bold.
Gemma
Llama
Mistral
Baseline
Ours (OOD)
Baseline
Ours (OOD)
Baseline
Ours (OOD)
BABYNAMES
170.0
346.0
129.0
426.0
225.0
493.0
COUNTRIES
23.3
53.3
17.8
43.0
44.0
71.5
FRUITS
20.5
62.0
45.0
46.0
DAYS AND DATES
18.5
23.0
14.3
22.5
22.7
22.5
NUMBERS
16.0
31.0
10.0
31.9
29.3
32.0
OCCUPATIONS
51.7
148.0
192.7
313.3
180.3
305.0
Table 6: Mean entropy results (higher is better). Better result for each model-task pair is in bold.
Gemma
Llama
Mistral
Baseline
Ours (OOD)
Baseline
Ours (OOD)
Baseline
Ours (OOD)
BABYNAMES
3.20
5.03
3.24
5.57
3.24
5.57
COUNTRIES
1.58
2.59
1.19
2.73
1.19
2.73
FRUITS
1.96
2.82
2.33
3.28
2.33
3.28
DAYS AND DATES
1.82
2.71
1.47
2.81
1.47
2.81
NUMBERS
1.28
2.65
2.70
2.70
OCCUPATIONS
1.59
3.40
3.21
4.68
3.21
4.68
17

---

Published as a conference paper at COLM 2024
C.2 Coverage and entropy of generated biographies
Table 7: Llama-2 results on synthetic biography generation.
First name
Last name
Birth year
Base
ID
OOD
Base
ID
OOD
Base
ID
OOD
Coverage Entropy
158
637
375
288
845
557
26
67
89
3.23
5.58
6.18
4.30
5.96
6.63
1.10
4.22
3.15
Table 8: Llama-2 results synthetic biography generation.
Birth place
Profession
Base
ID
OOD
Base
OOD
Coverage Entropy
98
305
533
178
276
198
2.65
5.98
5.08
3.81
4.33
4.61
18

---

Published as a conference paper at COLM 2024
C.3 Maps of birth places in generated biographies
We plot birth places of 500 biographies generated by the baseline Llama-2 model and our fine-tuned models in Figure 5, respectively. The baseline model appears have fewer data points, which is simply because most of the generated cities are duplicates and they overlap on the plot.
Figure 5: Fine-tuned Llama-2 models generate more diverse birth places in biographies. Each circle represents the birth place corresponding to a single biography in our dataset.
(a) A map of birth places generated by the baseline Llama-2 model.
(b) A map of birth places generated by the Llama-2 (Tuned-OOD) model.
(c) A map of birth places generated by the Llama-2 (Tuned-ID) model.
19

---

Published as a conference paper at COLM 2024
C.4 MT-Bench
After fine-tuning the models on all six tasks in Section 5, we evaluate utility of the models on the MT-Bench (Zheng et al., 2023) benchmark. The results show minimal differences between the baseline and our fine-tuned models, suggesting that our optimization does not have significant impact on the general capabilities of models.
(Finetuned;
Humanniies
STEM
Reasoning
Extraction
Math
Cading
Wrtling
model
Uama 128 (finctuned)
Hummnittes
Snleplay
Usma 2 138
STEM
Bezconing
Extraction
Math
Coding
Writing
model
Mitraal 78 (Finetuned)
Humanitees
Roleplay
Mittral
Roasoning
Costing
Figure Evaluation results for baseline and fine-tuned models on MT-Bench (Zheng et al., 2023).
20


Reviews:
[{"review_id": "GEV2Kx532I", "reviewer": "Reviewer_Jrar", "summary": "This paper adresses an important issue of LLMs: their lack of diversity in generation. In the worst case, LLM fail to generate random answers when it should be possible. As a solution, the authors proposes a fine-tuning objective as workaround: the goal of this criterion is to have less peaky distribution to generate more diversity.  The implementation relies on a low rank (LoRA) approach.  \n\nExperiments mostly use toy tasks  for illustration purpose.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Contextualizing our work in literature on generation diversity\n\nWe agree with the reviewer that the diversity (or lack thereof) in LLM generations is a challenge for doing good evaluation. However, improving LM evaluation practices was not the purpose of our paper. Rather, our aim was to introduce a method that improves model usability—supporting greater diversity over valid outputs without the need for prompt hacking or decoding strategy finagling. In contrast with past literature that mostly examines linguistic (e.g., n-gram) diversity [1](https://aclanthology.org/N16-1014/)[2](https://arxiv.org/abs/1901.07931), we focus instead on distributional diversity (i.e., diffuseness) of language models. Our work may have impacts on how people do open-ended text generation evaluation, but improving evaluation via matching human linguistic diversity (as MAUVE does) was not our main goal.\n\n> Cross entropy is not new\n\nOur paper proposes a view of language model training as distribution matching, as opposed to maximum likelihood estimation. Yes, we are using cross entropy, but not in the standard way of maximizing likelihood of the true next token. We are instead training to maximize *entropy* over the items in $\\mathcal{T}$, and we show that this procedure, though it resembles standard conditional language model training in that a cross entropy term is employed, actually leads to the minimization of KL divergence with a target distribution. Thus, our method takes existing mathematical tools and employs them in a novel way. We will add an extra sentence to our methods section to make our contribution more clear.\n\n> Experiment does not provide meaningful results\n\nWe disagree with the reviewer that our experiment does not provide meaningful results. We first show that on random number generation and baby name generation, our fine-tuning technique results in models that produce distributions that are practically indistinguishable from that of a uniform distribution. While these two generation tasks are indeed somewhat contrived, results on the biography generation task show that our method improves generation diversity in practical generation settings, without requiring task-specific training.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The addressed issue is well motivated.\n- The capacity of LLM to generate diverse output is an important goal", "reasons_to_reject": "As I said, the problem is important. However, it is addressed as a fresh new problem while it is not. The related work section is too short and too confused. For instance MAUVE paper published in NEURIPS 2021 has initiated an interesting discussion on the evaluation of LLM generation, including the lack of diversity. The discussion is still lively nowadays. See for instance https://www.jair.org/index.php/jair/article/view/13715\n\nThe second concern is the description of the method. Equation 5 just describes a cross entropy term and the surrounding text does not clarify the novelty.  I think I understand that the criterion  align the distribution of the predictions with the expected one. In this case, it should be better explained, because the cross entropy is not new. \n\nAt the end the experimental part does not provide meaningful results that could clarify the contributions.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YbVcWtEEx7", "reviewer": "Reviewer_eLBu", "summary": "This paper addresses the problem that LLMs perform poorly at generating diverse outputs, even when explicitly instructed to do so. The authors propose a fine-tuning method that encourages language models to output more diffuse distributions over valid choices. They use a parameter-efficient LoRA approach for the fine-tuning. Through experiments on toy tasks like generating random numbers, baby names, countries, etc., they show that the fine-tuned models generate significantly more diverse outputs that better match the expected distributions compared to the baseline models. Also, it seems the models exhibit strong generalization, i.e., a model fine-tuned on a subset of tasks generates diverse outputs even for held-out tasks. Applying the method to a more complex synthetic biography generation task leads to outputs with substantially more diverse attributes like names, birthplaces, occupations, etc. The general capabilities of the models (tested on a benchmark) are preserved after the fine-tuning for diverse outputs.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed read and insightful comments.\n\n> Direct operations on token probabilities\n\nIt’s indeed possible to add logit biases in tasks such as random number generation to introduce randomness, but this approach fails when crafting such a candidate set is arduous (e.g., baby names) or impossible (e.g., biography generation). We will clarify that the main goal of the paper isn’t to improve generation diversity over finite candidate sets, but to elicit generalization over tasks where candidate sets are difficult to enumerate.\n\nA sufficiently high temperature can indeed smooth out LM distributions, but this is hard to make work in practice, because of just how skewed these distributions are. For example, we computed that to match our results in Section 4.2, language models need to be sampled at a temperature of >10.\n\n> Why does our fine-tuning generalize\n\nWe are also curious about the reason behind the observed generalization of LM fine-tuning, which is an ongoing research topic that’s beyond the scope of our work. [1](https://proceedings.mlr.press/v202/malladi23a) offers one explanation for strong empirical generalizations of LoRA fine-tuning.\n\n> Why doesn’t fine-tuning change the model’s general capability?\n\nWe are using a rank of 4 when training the LoRA modules, so the amount of new information injected is small. We speculate that the limited expressivity of LoRA tuning means we make “local” updates to the model and shouldn’t affect general capabilities.\n\n> More prompting baselines\n\nWe acknowledge that prompt engineering, such as asking for multiple random numbers all at once could improve diversity, but as a user interacting with the system, it is unnatural to ask for a list of many numbers, and then pick one from this list. We would also like to clarify that the key motivation of fine-tuning language models to be diverse data generators is precisely to remove the need for prompt engineering and manually controlling categorical attributes.\n\n> Future directions\n\nThere are two practical applications of this approach that we are thinking of. The strong generalization ability of our fine-tuning method seems to enable diverse and high-quality data generation without human intervention. Another promising direction is de-biasing language models. Since our technique is a distribution-matching method, which lends itself naturally to de-biasing.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "-  The authors quantitatively analyze the mismatch between language model outputs and user expectations in terms of randomness and diversity.\n- They introduce a fine-tuning method that improves the diversity of language model outputs across valid options while maintaining generation quality. The proposed method is flexible and can handle tasks where the set of valid options is not easily enumerable. The fine-tuned models demonstrate strong generalization capabilities, meaning models trained on one set of tasks can generate diverse outputs for different, unseen tasks. The generalization property allows the method to enhance diversity in complex generation settings, such as synthetic dataset creation.\n- When applied to generating synthetic biographies, the proposed approach significantly increases the uniqueness of generated attributes. The improvements in diversity are achieved without the need for complex prompt engineering, decoding strategy adjustments, or manual editing, simplifying the process of generating diverse outputs.", "reasons_to_reject": "- The authors' proposed approach is fundamentally aligning the output token probabilities with the desired ground-truth (random) probabilities. However, direct operations on token probabilities can also be applicable, like adding bias on the candidates (~200 selected by the users). This can be combined with higher temperatures to flatten the distribution of candidates and avoid the situation of degeneration.\n- The experimental results are mostly introducing the phenomenon without further discussion and analysis. Like the claimed strong cross-task generalization ability in Section 5.1. I would be more likely to know about the potential reason for this interesting phenomenon.\n- The motivation and the methodology are somehow independent of each other. To solve the issues mentioned in the motivation, there should be a bunch of prompting methods: Directly prompting LLMs to generate 500 numbers between 0 and 1, or attributed prompting for diverse dataset creation, etc.", "questions_to_authors": "- I am really curious about the potential application or further impact of this work. \n- As the authors have some theoretical support for their fine-tuning method to minimize the distribution between the model outputs and ground-truth, is it possible for the authors to offer some proof that the tuning method will not influence the generation abilities of LLM (phenomenon in Appendix B.5).", "extra_scores": {}}, {"review_id": "tGT6fNFMbU", "reviewer": "Reviewer_SeEK", "summary": "The paper investigates how to increase diversity in LLM generations. The method is simple. It assumes some probability distribution over a fixed set of outcomes (e.g., 6-sided dice, uniform). It then minimizes weighted cross-entropy loss to match model probabilities with the ground-truth.\n\nGiven two toy tasks and a more realistic \"biography generation\" task, experiments show unnatural biases in instruction-tuned LLMs (Gemma, Llama-2, Mistral). The proposed method effectively mitigates those biases. Two interesting and novel results: (i) this fine-tuning does not seem to degrade other abilities of the LLMs, (ii) there is cross-task generalization where increasing diversity in one attribute also increases diversity in others.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> [W1]  The tasks considered are relatively simple\n\nWe agree with the reviewer that our current method can’t be applied to improving diversity of arbitrary open-ended generation. This is an area we plan to explore more in future papers, and we will elaborate further on this limitation in our Conclusion section.\n\n> [W2] Missing details on the fine-tuning setup\n\nThanks for pointing out the missing information. We fine-tune all models for 50 optimization steps over all prompts, and early stop when the training loss is within 20% of the optimal loss (entropy of the target distribution). Each run takes less than 10 minutes of wall-clock time on a single A100 GPU. We will add these details to Section 3.2. We have already open-sourced all code and data needed to reproduce experiments in the paper.  Here is a [link to an anonymized repo](https://anonymous.4open.science/r/diffuse-distributions-C2EF), and we will link to the Github repo in the final paper.\n\n> [W3] Missing detail on tasks (Sec 5)\n\nSince the output spaces of many of the tasks (e.g., baby names) are quite large, we couldn’t report the full empirical distribution. However, we did report metrics computed over them (coverage and entropy in Appendix B.2). Let us know if you believe there are other results that we should report.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- [S1] The method is simple to apply and appears effective in toy tasks and a more realistic \"tabular\" data generation task.\n- [S2] Results are validated on a range of LLMs, all show similar trends.\n- [S3] The paper is well written and the results are well presented.", "reasons_to_reject": "I do not have any major concerns about this paper and recommend acceptance. The below concerns are minor.\n\n- [W1] The tasks considered are relatively simple. Synthetic data generation with a more open-ended output space are probably not feasible with this proposed method. However, this limitation is acknowledged in the conclusion.\n- [W2] Missing detail on fine-tuning setup (Sec 3.2). It would be important to mention for how many steps fine-tuning with LoRa was applied. What was the stopping criterion?\n- [W3] Missing detail on tasks (Sec 5): I could not find what were the precise empirical distributions for each of the tasks. Could be added to appendix.", "questions_to_authors": "Typos:\n- pg 6. transferrability --> transferability\n- pg 7. \"on all tasks in Section 3\" --> Section 5?\n- pg 8. \"gender are birth\" --> \"gender and birth\"", "extra_scores": {}}, {"review_id": "BqJfCIxddy", "reviewer": "Reviewer_Zz3Z", "summary": "This work finds that (1) instruction-tuned language models fail to replicate high-entropy distributions over, e.g., random numbers (roll a 6-sided die) or names (give me a randon name). Further, it finds that (2) finetuning on a sample-approximation of the target distribution allows for higher-entropy distributions closer to the target distributions, and (3) that finetuning on some such requests even makes the instruction-tuned model output higher-entropy distributions for other requests (transfer, e.g, “give me a name” to “give me an occupation) while (4) not degrading the language model’s capabilities too much in general as measured by MT-Bench.\n\nThe last point here was quite surprising to me, and only mentioned in passing in the main text, but is very important! Instruction-tuning leads to very low-entropy distributions, and so these entropy-encouraging finetuning I might’ve expected to break various things. I think this should be explored more in the final draft.\n\nThe paper conducts experiments in generating baby names, countries, fruits, days and dates, numbers, occupations, and synthetic biography generation. They do some out-of-distribution extrapolation experiments that are particularly convicing, e.g., from one task to another, or from some range of numbers (finetune on 0-10) to another (test on 100-140).", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed read of our paper.\n\n> The method is standard fine-tuning\n\nThe reviewer is correct that our method basically involves fine-tuning on samples from the desired distribution, and we will qualify our claims and framing to highlight the similarity to standard fine-tuning. Our method’s interestingness comes from the fact that fine-tuning on tasks like uniform dice rolls leads to generalization to other tasks, such as picking a random baby name. It wasn’t obvious to us ahead of time that we would see this sort of generalization, and we think it is a pretty interesting result to share with the community. \n\n> Missing fine-tuning details\n\nThanks for pointing out this omission. We have open-sourced our codebase and data, so all our experiments are directly reproducible from there. Here is a [link to an anonymized repo](https://anonymous.4open.science/r/diffuse-distributions-C2EF), and we will link to the Github repo in the final paper. We will also add training details in the paper’s appendix.\n\n> Instruction-tuned language models\n\nThanks for pointing out that we were using “language model” to refer to instruction-tuned language models. We will clarify this in the paper. As a note, though we don’t cover it in our paper, our claims about non-random outputs distributions do hold pre-trained language models. For example, when GPT-2 is prompted with “She rolled the 20-sided dice. It landed on the number”, the output distribution is very non-uniform over the valid options. While we didn’t finetune any pre-trained models, we expect our method would also work well in these settings.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The problem of this paper is well-motivated, and the method is nice and simple (though, see reasons to reject about the framing.)\n\nI was a bit cautious at first when reading the experiments, but the various out-of-distribution experiments convinced me that there’s a surprising and useful amount of generalization from the simple finetuning done and the range of generation prompts for which one might get better distributions.\n\nIt’s a clearly written and concise paper.\n\nOverall, I just think this is a nice contribution that’s well-explored using various OOD generalization experiments; a simple method that seems to work and will be of interest to a broad range of people. I have some problems with the framing and experimental design below, but I think it should be accepted and people will learn interesting things from it.", "reasons_to_reject": "The framing of this paper is that it’s a new method for encouraging diffuse distributions, but the method more or less is to finetune on samples from the desired distribution – e.g., if you want to match a 6-sided die, sample from a 6-sided die and train the model to output the desired answer after the proper prompt. For sampling baby names, sample from (some type of) distribution over names and then finetune. This is what all finetuning does when trying to match any distribution. So, “we propose a finetuning method” is a bit overmuch to me. I’d strongly encourage the authors to re-frame to emphasize that the simple–I’d even say default–method of just finetuning works way better than one might expect.\n\nThere are considerable details missing from the paper. For example, the learning rate, number of epochs, etc. of the finetuning, for each model. This is really important since, e.g., too much finetuning would certainly eventually break the generalization of the model.\n\nThe framing of the paper also makes blanket claims about language models, but only considers instruction-tuned models. I’d just change the language – wherever you say “language models” fail at X, please consider saying “instruction-tuned language models” fail at X. Instruction-tuning destroys model entropy, and the claims you make are not verified to hold for pretrained models.\n\nI think the “models don’t degrade too much elsewhere” really should be explored more; this is crucial for it to be a useful method! At least explain more of the MT-bench results in the main paper.", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper investigates an issue that (instruction-tuned) language models have when tasked to generate diverse outputs -- the LMs typically fail to produce sufficiently diffuse distributions and exhibit certain biases toward a narrower distribution. Motivated by this issue, the authors propose a simple fine-tuning approach and demonstrate that it can effectively promote diversity over valid output distributions. The reviewers believe that the problem is well-motivated, the paper is clearly written, and the proposed method is simple and reasonable. The reviewers have raised several interesting points such as discussing more on how/why the method preserves the general language modeling capability, and suggestions for adding more training and evaluation details. The authors should carefully consider these when preparing the revision.

[At least one review was discounted during the decision process due to quality]
Decision: Accept