Modifying Large Language Model Post-Training for Diverse Creative Writing
Authors: John Joon Young Chung
Vishakh Padmakumar
Melissa Roemmele
Yuqian Sun
Max Kreminski
Venue: COLM 2025 (2025)
Keywords: creative writing generation
diversity
post-training
TL;DR: In creative writing generation, we facilitate diversity in LLM outputs by counting in how each training instance differs from other instances with the same prompt.

Abstract: As creative writing tasks do not have singular correct answers, large language models (LLMs) trained to perform these tasks should be able to generate diverse valid outputs. However, LLM post-training often focuses on improving generation quality but neglects to facilitate output diversity. Hence, in creative writing generation, we investigate post-training approaches to promote both output diversity and quality. Our core idea is to include deviation---the degree of difference between a training sample and all other samples with the same prompt---in the training objective to facilitate learning from rare high-quality instances. By adopting our approach to direct preference optimization (DPO) and odds ratio preference optimization (ORPO), we demonstrate that we can promote the output diversity of trained models while minimally decreasing quality. Our best model with 8B parameters could achieve on-par diversity as a human-created dataset while having output quality similar to the best instruction-tuned models we examined, GPT-4o and DeepSeek-R1. We further validate our approaches with a human evaluation, an ablation, and a comparison to an existing diversification approach, DivPO.

Published as a conference paper at COLM 2025
Modifying Large Language Model Post-Training for Diverse Creative Writing
John Joon Young Chung', Vishakh Padmakumar", Melissa Roemmele', Yuqian Sun' & Max Kreminski! Midjourrny New York University
Abstract
As creative writing tasks do not have singular correct answers, large lan- guage models (LLMs) trained to perform these tasks should be able to generate diverse valid writings. However, LLM post-training often focuses on improving generation quality but neglects to facilitate output diver- sity. Hence, in creative writing generation, we investigate post-training approaches promote both output diversity and quality. Our core idea is to include deviation- the degree of difference between training sample and all other samples with the same prompt-in the training objective fa- cilitate learning from rare high-quality instances. By adopting our approach to direct preference optimization (DPO) and odds ratio preference optimiza- tion (ORPO), we demonstrate that we can promote the output diversity of trained models while minimally decreasing quality. Our best model human-created
with 8B parameters could achieve on-par diversity dataset while having output quality similar to the best instruction-tuned models we examined, GPT-4o and DeepSeek-R1. We further validate our approaches with a human evaluation, an ablation, and comparison to an existing diversification approach, DivPO.
1 Introduction
In creative writing, there is no single "gold" answer but multiple valid ways that the writing can unfold (Flower & Hayes, 1981). For example, given a writing prompt "write story about dog on the moon," many different stories might be written in response, with the focus ranging from the dog's adventure to the dog's lonely moon life. This kind of divergent thinking ability has also been stressed as one aspect of creative intelligence (Runco & Acar, 2012; Guilford, 1957). Hence, when training large language models (LLMss) to generate creative writing, these models should learn to find and consider diverse paths and endings. While various post-training approaches- (PPO) (Ouyang et al., 2022) or direct preference optimization (DPO) (Rafailov et al., 2023) can strongly increase the quality of LLM output, this tuning also seems to decrease output diversity (Padmakumar& & He, 2024; Anderson et al., 2024; Kirk et al., 2024; Xu et al., 2024). Low output diversity can cause issues in creative task contexts it can lead users of LLM assistants to produce homogenous content (Anderson et al., 2024; Padmakumar & He, 2024) or show a limited set of biased outputs to the user despite the existence of various valid responses (Venkit al., 2024; Narayanan Venkit et al., 2023; Jakesch al., 2023).
as proximal policy optimization
Previous research investigated ways to increase LLM output diversity, but has largely fo- cused on how to make the best use of already post-trained models, e.g, by prompting (Wong et al., 2024; Hayati et al., 2024) or adjusting sampling temperature (Chung et al., 2023). While a few works studied tuning the model facilitate generation diversity, they targeted narrow set of simple tasks (e.g., baby name generation) (Zhang et al., 2024b).
"oorrespondenc:: schun@@iiiourrney.com
We provide our code in https: h/gg: wttp:/ccoo////eeeeeeeeeee http://sttttttlll/l//////sstttat

---

Published as a conference paper at COLM 2025
Prompt (x)
Responses per prompt (y)
Losing
Diversified DPO or ORPO
Diversified Model
Winning
Dariation
For each response the dattase, measure the deviation, the degree how much the response differs from others with the same prompt.
During post-training. esch pair's loss is weighted with winning response's deviation.
Trained model generates diversified and high figh-quality responses
Figure 1:
Our post-training approach to diversify creative writing generation while main- taining quality.
For creative writing generation, we explore post-training approaches to facilitate output diversity while maintaining quality. We propose consider diversity as a part of training objectives. Specifically, we factor in deviation, a measure of how much training instance differs from other candidate samples for the same prompt. We incorporate deviation into different LLM post-training methods, introducing diversified versions of DPO and odds ratio preference optimization (ORPO) (Hong et al., 2024).
Our results demonstrate that diversified DPO and ORPO could facilitate semantic and style diversity (defined in Section 5.1) while minimally decreasing the quality of writing, Our trained models had higher diversity than existing instruction-tuned models, such GPT- (Anthropic, 2024), or DeepSeek-R2 (DeepSeek-Al et al., 2025). The writing quality of our best model (a Llama-3. 1-88-based diversified DPO model) was on par with that of existing best-quality models while having a similar level of diversity to a dataset created by humans. Human evaluation that compares diversified DPO, original DPO, and GPT-4o further validates that our approach could generate diverse output while maintaining output quality. With an additional experiment that varies the maximum number of training instances per prompt, we show that, except when there are too few instances per prompt, our approach robustly increases output diversity while having on-par quality with the non-diversified approach and contemporary work, DivPO (Lanchantin et al., 2025). With too few instances per prompt, the quality drops with our approach. However, we demonstrate that this quality issue can be fixed by slightly tweaking the objective function or training with high-quality instances.
4o (OpenAI, 2024a), Llaude3..5-oomme
2 Related Work
While LLM post-training approaches, including PPO (Ouyang al., 2022), DPO (Rafailov et al., 2023), ORPO (Hong et al., 2024), have improved the model's instruction following capability and output quality, these often resulted in low output diversity (Kirk al., 2024; Go et al., 2023; Casper et al., 2023). Researchers observed such decreased diversity in creative writing generation, with similar narrative elements echoed repetitively in multiple generations (Xu et al., 2024). Low diversity could be detrimental in creative task settings. For example, researchers found that people's creative products can homogenize when they use post-trained LLMs (Anderson et al., 2024; Padmakumar & He, 2024; Chen et al., 2025).
way to facilitate output diversity is adjusting decoding approaches (Ippolito et al., 2019). For beam search, researchers proposed penalizing siblings (Li et al., 2016) or counting in diversity during search (Vijayakumar et al., 2017). For sampling, researchers explored adjusting temperatures (Tevet & Berant, 2021; Chung et al., 2023). However, these often have quality-diversity trade-offs (Chung et al., 2023; Tevet Berant, 2021; Zhang et al., 2021), limiting the extent of usage. To alleviate it, researchers introduced top-k (Fan et al., 2018), top-p (Holtzman et al., 2020), and min-p (Minh et al., 2025) sampling, cutting out the least probable tokens during decoding. Researchers also introduced adaptive temperature
2

---

Published as a conference paper at COLM 2025
to balance quality and diversity (Zhang et al., 2024a). With improving instruction-folloiing capability, researchers explored prompting-basee diversification, such as self-critiquing for diversity (Lahoti et al., 2023), iteratively prompting to avoid previous generations (Hayati et al., 2024), and guiding LLM generation with answer space dimensions (Wong et al., 2024; Suh et al., 2024), answer set programming (Wang & Kreminski, 2024), or evolutionary algorithms (Bradley et al., 2024). Only few, however, investigated tuning LLMs for diversity. Zhang et al. (2024b) one, but it deals with simple tasks of baby name or random number generation. DivPo (Lanchantin et al., 2025) is another, which filters the preference dataset to have highly diverse winning instances and lowly diverse losing instances. While facilitated diversity, it still showed trade-offs between the quality and diversity of outputs. To diversify creative writing generation while preserving quality, we leverage deviation- how training instance differs from other instances as part of the learning objective.
3 Preliminaries
We start with the problem setting of creative writing generation and the metrics. Then, we explain two post-training approaches we extended: DPO and ORPO.
3.1 Problem Setting and Metrics
For creative writing tasks, given prompt (x), LLMs (0) should generate diverse, high- quality outputs (yi) so that end-users can get various valid options. Here, we define "valid" as whether the generated output satisfies the prompt. For instance, if user inputs prompt to discover possible next events in their story, showing diverse branches would be desirable. In this setting, we formulate quality and diversity follows:
Quality(xx, 0) E. [r)x,y))
where yi po(y(x)
(1)
Diversity(1,, Diversityy (Y) where Y Y= (y
(2)
The above assumes sampling N responses from x. calculates a reward (i.e., validity and
quality of a response) over a prompt-response pair, and Diversity, evaluates diversity of all sampled responses. While many options exist for Diversity, (Cox et al., 2021), we use mean pairwise distance between set of outputs:
Diversity, y (Y) I |
[d(y where Y= (
Note that d is distance function and EN-1 different from all other samples.
j-1ff1 [d(y,)))) indicates deviation, how y; is
3.2 Post-training Approaches
As post-training could be noisy with limited samples, some post training methods like DPO require supervised fine-tuning (SFT) to guide the trained models to have desired generation behaviors. Starting from pre-trained models, the SFT model can trained by maximizing the likelihood of response y given prompt in dataset D with the below loss:
LSTT - -E D [log Po(y)x)]
(4)
We did not experiment with a seminal post-training approach, PPO. PPO is known to be technically difficult and complex to train (Casper et al., 2023). Having robust reward model for creative writing is difficult due to subjectivity in evaluation, which was also the case in our context. Unlike other approaches, our pilots on PPO did pan out in the examined creative writing dataset.
3

---

Published as a conference paper at COLM 2025
3.2.1 Direct Preference Optimization
DPO is approach that optimizes the policy model directly the dataset of (x, y" yy), increasing the likelihood of yd (a winning instance) over y' (a losing instance). With the ratio between the likelihood of the policy model and the reference SFT model as an implicit reward, the training objective is follows:
Pe (o(y51x)
pe po(y
DDO (xyy CCD
log aB log
B log
PSET PsFr(y'(x) )
)
(5)
PSFT (y"
3.2.2 Odds Ratio Preference Optimization
another approach that directly optimizes over (x,y") y²). y²), However, for signaling preferences, it does not use an SFT model as reference. Instead, it uses the odds ratio preference signal to model training:
ORPO is
oddso(y o where oddsp(y)x) 1 po(y(x)
po(y x)
Oy v/)
(6)
Specifically, the training objective combines log- lieliikelihood (over the winning responses) and log odds ratio as below. Note that ORPO starts training from a base model, not an SFT model.
(1y0xx)) oddsg (y (x)
CORPO
E (xyx") ED
log po()" + logo
4 Diversified DPO and ORPO
We introduce approach to promote both quality and diversity in LLM post-training. The main idea is to factor in deviation (0) into the objective function. We define deviation as "the degree of how much training instance differs from all other instances with the same prompt." By factoring this in, we aim to increase the likelihood of models generating output of high quality that deviates from "typical" outputs. Note that we assume there are enough prompts with more than three responses, as the concept of "deviation" holds only for this case (i.e., with two responses, deviations would be the same for both of them).
Diversified DPO (DDPO) To count deviation into DPO, for each pair of winning-losing responses, we weighted the pair's training objective with the deviation of the winning instance. As the pair's 's training objective is to learn the winning response's 's behavior, this weighting would emphasize rare winning instances more than common winning ones.
Po
Po [x) PSFT (y (x)
LDDOO = - (xy"") ED
5 log o log PSFT (y"
B log
(8)
Diversified ORPO (DORPO) We extended ORPO similarly, by weighting the pair loss with deviation of the winning instances. Here, note that we weighted both log-likelihood and log odds ratio terms, as both contribute to learning the behavior of winning response.
8 log 1x) + A Tth log olog oddso(y (1
LDORPO
E [xyx ED
(9)
oddsg ( (x(xx)
We explain how we quantify 5 in Section 5.1 and D.
4

---

Published as a conference paper at COLM 2025
5 Experiments
5.1 Experiment Settings and Metrics
Dataset We focus on creative writing by using the //wrissngPrromps dataset (Fan et al., 2018)'. The data originates from r/ //wriiingPrrmppl subreddit, where users post writing prompts (x) and other users share their creative writing to the prompts as comments (4). Hence, there can be multiple writings for single prompt, which we could leverage to train a model that generates diverse outputs to the same prompt. Moreover, each writing has a user upvote score, s, which we used signal for the writing quality. We use these scores to 1) train a reward model for evaluation and 2) craft a binary preference dataset with pairs of winning and losing responses to train generation models." While we split the data into train and test sets (421330 and 45868 prompt-response pairs, respectively), we describe data processing details in Appendix A.
To evaluate the quality and diversity, sampled four instances per each evaluation prompt. Then, we measured the quality of each sample and the diversity between those four instances. We used 1000 evaluation prompts from the test dataset, resulting in 4000 samples. We detail our sampling approach in Appendix B.
Evaluation Task and Metrics
We automatically evaluated the model output's quality and diversity. For the output quality (reddit t-reardd,, we trained a reward model out of (x,y,s) triplets in the dataset as regression model that predicts from (x, y). We describe the training details and performances of the reward model in Appendix C.
For diversity evaluation, we embedded sampled y with embedding models and measured mean pairwise cosine distances between all samples from the same prompt. We focused on 1) semantic diversity and 2) style diversity. For embedding models, we used (Sturua et al., 2024) and AnnaWegmann/ SSttne-Eyyylldddddde mann al., 2022), respectively for semantic and style diversity. Note that style embeddings capture whether the same or different people would have written a set of writings.
(Weg-
Deviation Measure For DDPO and DORPO, we calculate the deviation of each in- stance (d%i). Focusing on semantic and style deviations, we calculated deviation for yi as EN ii= (d(y; (where d is cosine distance after embedding y; and Y* = { Note that we used
htta///iiinneebeeddnn--ww and AmnakemmammSStyy---mbeeddd tively, for semantic and style embedding. Considering that weights (here, deviations) should be larger than zero and we would want the impact of the total weights to correspond to the size of the dataset per prompt, we transformed deviations to have 1) minimum of (min((8"),, [8") Jy,r=) = 0, unless deviations are all the same) and 2) a sum equal to their count (sum(180) Jy,r=) = |Y*).. We also experimented with mixing semantic and style deviations to consider both diversity types LLM training. We provide details on deviation calculation in Appendix D.
respec-
Conditions As baselines, we examined SFT, DPO, and ORPO as baselines. We also ex- amined post-trained versions of the base models (trained by base model providers, as Instruct) and four other instruction-tuned models (got-40-2024-11-20 as GPT-4o (Ope- as
nAI, 2024a), 01 01-2024-12-17 as 01 AO 2024b), Claude-3. 5-sonntt (Anthropic, 2024), and DeepSeek-R1 (DeepSeek-AIi al., 2025)). For GPT-4o, to see whether diverse generation is achievable only with prompting, we evalu- ated an iterative prompting approach that asks the LLM to generate results far different from previously generated ones (GPT-40-iter, prompt details are in Appendix B). We also computed metrics for the original test dataset (Gold). For DDPO and DORPO, we exam- ined versions that consider only semantic deviation (using jinaai/ rinaai/jinare aaina--meedding--v...
hhttps: hatiss://hgggnnggaaeee wttaaaa....cccaeeeeeeeeeeeeaaaae co/datasets/euclaa wwtpe/eaaaaaaeeiiaaeeeeeeeeeaeae tttp/riiingPropptss preferences
Upvote scores are somewhat noisy reward signals. For example, some decent writings might have gotten low attention and received similar upvotes compared to lower-quality creative writings.
5

---

Published as a conference paper at COLM 2025
DOPO-sem, DORPO-sem), only style deviation (using www.a//aaam//tttttee-teeeeeeeeee DDPO-sty, DORPO-sty), and both types of deviations (DDPO-both, DORPO-both).
Training Methods We started with the following pre-trained models: 1-8B (Grattafiori et al., 2024) and risttalla/Mistral--B--y et al., 2023). For SFT, DPO, ORPO, DDPO, and DORPO, we did parameter efficient tuning with LoRA (Hu et al., 2022), using a rank of 128 and an alpha of 256. This approach optimizes low rank additions to weight matrices and allows efficient tuning of large language models (2 7B parameters).
(Jiang
We first trained SFT models as the starting point for DPO training. For this, we used cosine scheduler with a warmup, which went through a half cycle per epoch with a maximum learning rate of 3e-5. We trained with Adam optimizer for single epoch with batch size of 2. Evaluation and checkpoint saving were done for every 5000 steps, and we used the models with the lowest evaluation loss. Note that all training in this work was done with Accelerate (Gugger et al., 2022) and DeepSpeed ZeRO-2 (Rajbhandari et al., 2020) in bfloat16, 6, in parallel over six NVIDIA H100 SXM GPUs.
For DPO, DDPO, ORPO, and DORPO, we trained models using the fixed dataset in offline settings. For all approaches, we used a linear learning rate with a maximum of 5e-6. Except for DPO and DDPO for and DDPO on m /Misstal--7--o..... accumulation of 2 due to the GPU memory. DPO and DDPO were trained with B of 0.1 for three epochs. We trained ORPO and DORPO with Ả 0.25 for four epochs, as starts from the base model, not the SFT model.
the batch size was 2. For DPO we used the batch size of with a gradient
5.2 Results
Figure 2 shows results on writing quality and diversity. Existing instruction-tuned mod- els GPT-4o, #GT-44-iter, 01, 3.laude33.5-oonee #DeepSeek-RI and Instruct)- formed one cluster, where they show high reward but with low diversity. This aligns with previous findings that some post-trained models show low output diversity (Kirk et al., 2024; Go et al., 2023; Casper et al., 2023). Human-crafted o Gold showed a lower reward score than these models but the diversity was far higher. SFT models showed the lowest re- ward scores. Their diversity results were higher than the existing instruction-tuned models, but lower than Gold. Both DPO and @RPO increased reddi t-rwward compared to SFT while either maintaining or decreasing diversity. DPO resulted in higher reddi t-reaadd but often in lower output diversity than ORPO. Note that our trained models had higher diver- sity than existing models (even higher than GPT-44-iter, which used prompts), which might be due to the diversity of responses for each prompt in our training dataset.
The results show that our approach promotes targeted diversity while minimally decreas- ing the quality. Compared to DPO, DDPO-sem and IDDPO-sty increased semantic and style diversity, respectively. By mixing two deviation signals, * DDPO-both could facilitate both diversity types. In terms of quality, L1ama-3. 1 1-8B's DDPO-see and Mi Mistral-7B-v0. 3's DDPPOsst showed decreased quality compared to DPO but other DDPO models showed either increased or maintained quality scores. DORPO showed similar patterns: compared to ORPO, DORPO-sem, IDORPO-S -ooy, and DORPO-bott increased targeted diversity while minimally decreasing the reddit-reward Among trained models, Llama-3.1 1-88-baddd DDPO-bott model achieved high scores on both quality and diversity, being almost on-par as the best baselines and gold data. Specifically, had reddit-rewar only slightly lower than C GPT-4o-iter, semantic diversity close to + Gold, and style diversity slightly lower than Gold, For subset of conditions, we list examples in Table (as topic sentences) and Appendix E (as 100-word summaries and 100-word truncations).
AAccordingly, we used mistralai/Mistral- I for Instruct.
and

---

Published as a conference paper at COLM 2025
0.8
semantic diversity
diversity
0.6
0.25
al Llama-3.1-8B
0.4
0.20
0.2
0.10
0.0
-0.3
-0.2
-0.1
0.0
0.1
0.2
-0.3
-0.2
-0.1
0.0
0.1
0.2
reddit reward
reddit reward
0.35
0.30
0.7
Semantic diversity
style diversity
0.5
Mistral-78-vol
0.3
0.15
0.1
0.10
0.0
reddit reward
reddit reward
GPT-4o-ter
DaepSeek-R1
DDPO-sty bDto
ORRO
DORPO-sty
Instruct
Gold
DDPO sem
DORPO-sem
DORFO-both
Figure 2:
Results on writing quality (reddit-reward x axes) and diversity (semantic or style diversity, y axes). Error bars in this paper indicate 95% confidence intervals.
Table 1: Topic sentences of generated examples. Note that "writing prompt" is not necessar- ily full instruction prompt given to LLMs.
Writing Prompt: "Why are you shaking, my lave? You are king now. King can be metaphorical or literal.
GPT-4o
Llana-3. 1-8B DPO
1-- DDPO-both
A newly pronoted manager struggles with imposter syndrome while his wife offers unavering support.
divine ruler must abandon his throne protect his daughter's reign after revealing a dark secret.
president grapples with noral conflict
inability to take a life despite lnadership expectations.
A newly crowned king struggles self-doubt while his beloved offers reassurance.
king nurders his due prophecy, while their watches the tragic sacrifice.
couple seeks refuge during storn, harboring dark secrets and contemplating future redenption.
A newly crowned king grapples with beloved offers
A reluctant new king grapples with overwhelming leadership.
child's supernatural encounters diving reveal family tradition of hunting marine spirits.
self-doubt reassurance.
A reluctant heir grapples with the weight unexpected kingship fear of failure.
A royal heir struggles with Leadership responsibilities finding strength in romantic love.
young bear prince assunes his role King facing arranged marriage in dark fantasy setting.
6 Human Evaluation
We complement the automated evaluation with human evaluation on subset of condi- tions. Given two sets of texts from two conditions, evaluators decided 1) which set included the most interesting, highest-quality writing and 2) which set was more diverse. If a decision was difficult to make, they could answer with "Hard to decide" option. Each set included
7

---

Published as a conference paper at COLM 2025
Table 2: Win rates(%) from human evaluation. Significant differences are bolded.
DDPO-both
vs. GPT-4o
DDPO-both
vs.
DPO
Has the highest-quality story More diverse
68%
24%
50%
34%
100%
0%
62%
26%
four creative writings generated by the model for that condition. Note that we provided summarized versions of writings as doing the task with eight lengthy creative writings can be cognitively overloading. Hence, with human evaluation, semantic diversity but not style diversity. Five of this paper's 's authors served as evaluators, being blind to the conditions. Three evaluated each instance, and we aggregated their annotations with majority voting. With this approach, we evaluated DPO vs. DDPO-both and GPT-40 vs. DDPO-both. Over other approaches for existing instruction-tuned models, we choose to compare with GPT-4o due to 1) its popularity in LLM research and products and 2) the simplicity in prompting. Refer to Appendix G for more details.
could measure only
Results When compared to GPT-40, evaluators chose DDPO-both's sets more frequently as those that have the highest-quality story (Table 2). The ratio difference was significant with Two Proportion Z-Test (p < 0.001). When compared to DPO, DDPO-both was chosen more frequently, but the difference in ratio was not significant (p > 0.1). Regarding diversity, evaluators chose DDPO-both's s sets more frequently as more diverse sets, compared to both GPT-4o and DPO. The ratio differences were significant for both comparisons (p 0.001 for both). Evaluator agreement (Krippendorff's alpha) was 0.31 and 0.45, respectively, for quality and diversity selections, indicating fair agreement. We count in the evaluators' indecision when calculating agreement values. When calculate agreement scores separately for vs. GPT-4o and vs. DPO, quality agreement scores were 0.37 and 0.27, respectively, and diversity agreement scores were 0.95 and 0.12, respectively. This reflects that smaller diversity differences in vs. DPO were more difficult to discern than for those in vs. GPT-4o.
7 Ablation and Comparison to DivPO
While Section 5.2 and 6 show that diversification approaches work for the examined dataset, we were curious if the approaches would still work when the size of the dataset, specifically, the number of responses per prompt, is small. This is an important question, crafting dataset with many instances per prompt can be expensive. Hence, we conducted an ablation study, evaluating the performance of trained models when we varied the maximum number of responses per prompt. Specifically, we compared DDPO-both to DPO for different numbers of responses, from four up to the maximum provided in the dataset.
Here, we also compare our (ablated) approaches to recent tuning approach for facilitating output diversity, DivPO (Lanchantin et al., 2025). When using our dataset in full, we cannot apply DivPO, as it requires filtering instances based on their quality and diversity. However, when limiting the maximum number of responses per prompt (Nmax), we need to sample subset of instances and we can apply DivPO's filtering approach (detailed in Appendix H).? We considered both semantic and style diversity signals when applying DivPO.
Results Figure 3 shows the results, where except for when the maximum number of responses is four, -DDPO-both had similar or only slightly lower mean reddit-rewards to -DPO and -DivPO while showing higher diversity than them. -DivPO showed diversity scores slightly higher than -DPO except for the semantic diversity ty when the maximum number of instances per prompt was 12. Still, -DivPO's diversity scores were lower than than
those of -DDPO-both. Note that -DivPO tends to show slightly higher reddit-rewal other approaches, except for when the maximum number of instances per prompt was six or no filtering was used. This might be because -DivPO first sifts through the dataset to get the
Note that, practically, if we want to train models with the same number of instances, DivPO requires more data than our approaches due to filtering.
8

---

Published as a conference paper at COLM 2025
DPQ
Di/O-bbot
DDPO.both
DOPO-bot mith
DDFO.bsth with high quality resporses
reddit reward
b) semantic diversity
c) style diversity
0.25
0.31
0,20
0.60
+
0.9
0.45
0.05
0.35
Max12
Max12
Max12
Figure 3: Ablation results by varying the maximum number of responses per prompt. When the maximum number of responses is four, also experimented with 1) setting minimum 5 and 2) using high-quality responses.
highest-quality winning instances and lowest-quality losing instances. -DPO and -DDPO-both randomly sampled instances without such filtering. Overall, the results demonstrate that our approach is more effective in promoting diversity while not hurting the quality much except when there too few responses per prompt.
For the result when the maximum number responses per prompt is four, we hypothesized that the decrease in reddi t-reward might be because there can be cases where 8" is zero (in Equation 8) while such an issue does not happen when we do not scale pairs with deviation. With few instances per prompt, the ratio of pairs having zero 8td would be higher, which could have impacted the performance on reddit-rewar more.
We examined whether this issue could be addressed by forcing a non-zero minimum 8 value. That is, in ADPO-both with min 6, we replaced 8tD smaller than a threshold value the threshold value. The result showed that setting the minimum 5" helped with increasing reddit-rewar to the level of -DPO's while decreasing the amount of boost in the diversification approach, specifically semantic diversity. However, its diversity scores were still higher than -DPO and -DivPO.
(0.1, in our examination)
We were also curious if this issue could be alleviated we prepared higher-quality winning responses. Hence, we also examined how the performance changes when we had the highest quality winning responses when sampling at most four responses per prompt (+DDPO-both with high-quality responses). We found that this approach resulted in mean redwit-reward higher than -DPO and only slightly lower than -DivPO. Both types of diversity decreased compared to -DDPO-bothh but they were still largely higher than those of -DPO and -DivPO. Overall, by either tweaking 8th or preparing high-quality data, we could address DDPO's generation quality issue when the number of responses per prompt is small but at the cost of diversity scores. Despite this diversity cost, DDPO still obtains higher diversity than DPO and DivPO.
Concluding Discussion
In this work, in a creative writing context, we introduce extended versions of DPO and ORPO that facilitate diversity while maintaining generation quality. The core idea behind our extensions is factoring in the deviation of each winning instance to loss terms, With these approaches, we achieved a model that has quality on par with existing statc-of-thee-at models and diversity similar to the original human-crafted datasets. We also demonstrate that our approaches could be robust variation in dataset size while outperforming an existing diversification post-training approach, DivPO. Note that while DivPO requires more data instances than actually used for training (as it filters data), our approach fully uses given dataset- which would be valuable for data-scarce settings. Overall, approach emphasizes that to facilitate generation diversity, it is important to balance learning from both frequent and rare high-quality training instances.

---

Published as a conference paper at COLM 2025
We demonstrated the benefits of our approach in our setting, but validating the approach in other settings would be important. Specifically, many instruction-tun low output diversity were trained with online approaches, and future work would need to investigate whether approaches similar to ours can alleviate such issues in online training settings. Moreover, future work would need to examine diversifying tuning approaches in tasks other than creative writing. In addition, as we demonstrated, there can be multiple ways to configure the deviation term, and further exploring these can be future work. As used deviation terms with winning-losing response pairs, another future work could explore how we can adopt deviation terms in tuning approaches that do not use winning-losing pairs, such as those that use numerical rewards.
Spectfically
models with
Acknowledgments
We want to thank Midjourney for supporting this work.
References
Barrett R Anderson, Jash Hemant Shah, and Max Kreminski. Homogenization effects of large language models on human creative ideation. In Proceedings of the 16th Conference on Creativity & Cognition, C&C '24, pp. 413-425, New York, NY, USA, 2024. Association for Computing Machinery. ISBN 9798400704857.
doi: 10.1145 041445/3635636636562444 URL
https://doi org/10 10. 0145/3635636 3656204.
Anthropic. Claude 3.5 sonnet, 2024.
URL https:/www.antrropic.comm cowp://rroppi.cccco/////.e.s
-laude-3-5-somee Accessed: February, 2025.
Herbie Bradley, Andrew Dai, Hannah Benita Teufel, Jenny Zhang, Koen Oostermeijer, Marco Bellagente, Jeff Clune, Kenneth Stanley, Gregory Schott, and Joel Lehman. Quality- diversity through Al feedback. In The Twelfth International Conference on Learning Represen- fations, 2024. URL https: http::::///pppeeeieeeee net www.rrrmnnnooooonnorrrrrrrr
Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomek Korbak, David Lindner, Pedro Freire, Tony Tong Wang, Samuel Marks, Charbel-Raphael Segerie, Micah Carroll, Andi Peng, Phillip J.K. Christoffersen, Mehul Damani, Stewart Slocum, Usman Anwar, Anand Siththaranjan, Max Nadeau, Eric J Michaud, Jacob Pfau, Dmitrii Krasheninnikov, Xin Chen, Lauro Langosco, Peter Hase, Erdem Biyik, Anca Dragan, David Krueger, Dorsa Sadigh, and Dylan HadfieldMennell. Open problems and fundamental limitations of reinforcement learning from human feedback. Transactions on Machine Learning Research, 2023, ISSN 2835-8856. URL https: /ttp:///ppp/////eeeeeee net/forumt (dtpummmd=xx22403444 Survey Certification, Featured Certification.
Liuqing Chen, Yaxuan Song, Chunyuan Zheng, Qianzhi Jing, Preben Hansen, and Lingyun Sun. Understanding design fixation in generative ai, 2025. URL https: / /arxiv...vvv///////////e/////iv/ .ww.vooorggg.oo.. org/abs/ 2587.0
John Chung, Ece Kamar, and Saleema Amershi. Increasing diversity while maintaining accuracy: Text data generation with large language models and human interventions. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 575-593, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/ h11///23.aclllooggoggg..mm URL https: a/hoon /trgg/223..0 acl- 34/.
Samuel Rhys Cox, Yunlong Wang, Ashraf Abdul, Christian von der Weth, and Brian Lim. Directed diversity: Leveraging language embedding distances for collective creativity in crowd ideation. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, CHI 21, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781450380966. doi: 3445782.
URL https: //tp///oi..rr www.oo..//1.. 10. 1145/311 1764.
10

---

Published as a conference paper at COLM 2025
DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, and et al. Deepseek-rl: Incentivizing reasoning capability in lims via reinforcement learning, 2025. URL https:://rxii...tt abs/2501. 12948.
Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical neural story generation. In Iryna Gurevych and Yusuke Miyao (eds.), Proceedings of the 56th Aunual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 889-898, Melbourne, Australia, July 2018, Association for Computational Linguistics. doi: 10.18653/ (00/663/v1/P181/1/ v1/118-1082 URL https: / hatps///llaaanooogg...o.ooo org/P18-10822
Linda Flower and John R. Hayes. A cognitive process theory of writing. College Composition and Communication, 32(4):365 1981. ISSN 0010096X. URL http: http://www... http:///www..trrrrooo...//o castorroo org/
stable/356600.
Dongyoung Go, Tomasz Korbak, Germán Kruszewski, Jos Rozen, Nahyeon Ryu, and Marc Dymetman. Aligning language models with preferences through f-divergence minimization. In Proceedings of the 40th International Conference on Machine Learning, ICML/23. JMLR.org, 2023.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and et al. The llama 3 herd of models, 2024. URL
Sylvain Gugger, Lysandre Debut, Thomas Wolf, Philipp Schmid, Zachary Mueller, Sourab Mangrulkar, Marc Sun, and Benjamin Bossan. Accelerate: Training and inference scale made simple, efficient and adaptable. https://gi http://tttuubb.coo///ggggggggg/ acceeeaaaaaceeceeeeeeeeeeeee..ee 2022.
Joy P Guilford. Creative abilities in the arts. Psychological review, 64(2):110, 1957.
Shirley Anugrah Hayati, Minhwa Lee, Dheeraj Rajagopal, and Dongyeop Kang. How far can we extract diverse perspectives from large language models? In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Pp.
Miami, Florida, USA, November 22/2224emmllmmminn.366
2024. Association for Computational Linguistics. doi: URL / /actthol// anthology org/ 2024. 11:.4..mnlppmmn emmlp-main...mm.. 306/.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In International Conference on Learning Representations, 2020. URL net//oouumidd=ryOO00)
https://openrest
Jiwoo Hong, Noah Lee, and James Thorne. Orpo: Monolithic preference optimization without reference model, 2024. URL
wtt.aaaiiii..rgg//////33... 0469.96611.. 07691.
Edward] Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL https://opennevii// net/ forum?id=nZeeWeeeFY9
Daphne Ippolito, Reno Kriz, João Sedoc, Maria Kustikova, and Chris Callison-Burch. Com- parison of diverse decoding methods from conditional language models. In Anna Ko- rhonen, David Traum, and Lluís Mårquez (eds.), Proceedings of the 57th Annual Meet- ing of the Association for Computational Linguistics, pp. 3752-3762, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.1865/v11/1191100 https://aclanthol...
URL
org/P19-1365/
Maurice Jakesch, Advait Bhat, Daniel Buschek, Lior Zalmanson, and Mor Naaman. Co- writing with opinionated language models affects users' views. In Proceedings of the 2023 CHI Conference Human Factors in Computing Systems, CHI '23, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9781450394215. doi: 10.1145/3544548. 1145/3544544 3581196.
3581196. URL https: ptttps://doi.ooo htt.dooo.org//// httpg//....111545555533
11

---

Published as a conference paper at COLM 2025
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang. Timothée Lacroix, and William El Sayed. Mistral 7b, 2023. URL http:///arxiv..o/abb/////// 2b.02000.6211
Robert Kirk, Ishita Mediratta, Christoforos Nalmpantis, Jelena Luketina, Eric Hambro, Edward Grefenstette, and Roberta Raileanu. Understanding the effects of RLHF on LLM generalisation and diversity. In The Twelfth International Conference on Learning Representations, 2024. URL http:///ppeeeeeeeeeeee///rom//mm
Preethi Lahoti, Nicholas Blumm, Xiao Ma, Raghavendra Kotikalapudi, Sahitya Potluri, Qijun Tan, Hansa Srinivasan, Ben Packer, Ahmad Beirami, Alex Beutel, and Jilin Chen. Im- proving diversity of demographic representation in large language models via collective- critiques and self-voting. Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 10383-10405, Singapore, December 2023, Association for Computational Linguistics. doi: 10.18653/vl vv1 2023. wwt...mmlp-main...m m4.n.643.. URL httpss:/acclantholggg org/ wrg/2023.... 2023. emnlp- maio.mminn...m 643/.
Jack Lanchantin, Angelica Chen, Shehzaad Dhuliawala, Ping Yu, Jason Weston, Sainbayar Sukhbaatar, and Ilia Kulikov. Diverse preference optimization, 2025. URL https://arxiv.
org/abs/2501. 18101.
Jiwei Li, Will Monroe, and Dan Jurafsky. A simple, fast diverse decoding algorithm for neural generation, 2016. URL http::////axxi..or/bbbbsss
.85522
Nguyen Nhat Minh, Andrew Baker, Clement Neo, Allen G Roush, Andreas Kirsch, and Ravid Shwartz-Ziv. Turning up the heat: Min-p sampling for creative and coherent LLM outputs. In The Thirteenth International Conference on Learning Representations, 2025. URL https://opemrevient wwwpriiwweeefffrrnee ret//oorum?d==FBBXCyys
Pranav Narayanan Venkit, Sanjana Gautam, Ruchi Panchanadikar, Ting-Hao Huang, and Shomir Wilson. Unmasking nationality bias: A study of human perception of nationalities in ai-generated articles. In Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Sociefy, AIES '23, pp. 554-565, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9798400702310. doi: 05444////6002113344666 URL https: //pp:://oo. //ooi.///roooooo org/ 10. 114155533600211. 3664666667.
OpenAl. Hello gpt-4o, 2024a. URL http:tss///gepaaaa.cc////////llp -ccesl- Accessed: February, 2025.
OpenAL. Introducing openai ol, 2024b. URL httphpp////opppaa.ccoo///// Accessed: February, 2025.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Chris- tiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 27730-27744 Cur- ran Associates, Inc., 2022. URL 2022/fiile/bleede553be3444 -100a31111100000001111--------00
neuris ccppap files/paper// pdf.
Vishakh Padmakumar and He He. Does writing with language models reduce content diversity? In The Twelfth International Conference on Learning Representations, 2024. URL
https://openreviae wttpi/rrwwiwieeeefofoeeeeeee
Jiaxin Pei, Aparna Ananthasubramaniam, Xingyao Wang, Naitian Zhou, Apostolos Dede- loudis, Jackson Sargent, and David Jurgens. Potato: The portable text annotation tool. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, 2022.
12

---

Published as a conference paper at COLM 2025
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 53728-53741, Curran Associates, Inc., 2023. URL https:httttttttttet / neurips.e.... wttpppp.ccppppppppppppppppppppee /tttss//pper/223// pdf.
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: memory optimizations toward training trillion parameter models. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC. '20. IEEE Press, 2020. ISBN 9781728199989 199986.
Mark A. Runco and Selcuk Acar. Divergent thinking as an indicator of creative potential. Creativity Research Journal, 24(1):66-75, 2012. doi: 10.1080/ 00.1000/10400419.2000000 10400419.2012..652222
Chantal Shaib, Joe Barrow, Jiuding Sun, Alexa F. Siu, Byron C. Wallace, and Ani Nenkova. Standardizing the measurement of text diversity: A tool and a comparative analysis of scores, 2024. URL
www.:/.gg///a//... arrg/abs/2403. 00553.
Saba Sturua, Isabelle Mohr, Mohammad Kalim Akram, Michael Günther, Bo Wang, Markus Krimmel, Feng Wang, Georgios Mastrapas, Andreas Koukounas, Andreas Koukounas, Nan Wang, and Han Xiao. jina-embeddings--3: Multilingual embeddings with task lora, 2024. URL https: http://axxvvvv..v wwtp:/wwwwrg/bb//////// 10173.
Sangho Suh, Meng Chen, Bryan Min, Toby Jia-Jun Li, and Haijun Xia. Luminate: Structured generation and exploration of design space with large language models for human-ai co-creation. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems, CHI '24 New York, NY, USA, 2024. Association for Computing Machinery. ISBN 9798400703300. doi: 3642400.
URL http::://doi.oroo 1145/3611533999 3904.
Guy Tevet and Jonathan Berant. Evaluating the evaluation of diversity in natural language generation. In Paola Merlo, Jorg Tiedemann, and Reut Tsarfaty (eds.), Proceedings the 16th Conference of the European Chapter of the Association for Compulationnal Linguistics: Main Volume, pp. 326-346666663 Online, April 2021. Association for Computational Linguistics. doi: 10.18653 200e3//v12222221 wttp//2.eeecmmmii.......
URL htttps://aalanntholgg/// orw.orgg///// 2021 . mainn. 25/.
Pranav Narayanan Venkit, Philippe Laban, Yilun Zhou, Yixin Mao, and Chien-Sheng Wu. Search engines in an ai era: The false promise of factual and verifiable source-cited responses, 2024. URL https/////axxvv..aaabssssse 2221//222.2.299....
Ashwin K Vijayakumar, Michael Cogswell, Ramprasaath R. Selvaraju, Qing Sun, Stefan Lee, David Crandall, and Dhruv Batra. Diverse beam search: Decoding diverse solutions from neural sequence models, 2017. URL http:p///pppprrvew....eee//rrtot
Phoebe Wang and Max Kreminski. Guiding and diversiffyngg lim-aseed story generation via answer set programming, 2024. URL SURLLttss///rriv.ogg//bb/// .0554.
Anna Wegmann, Marijn Schraagen, and Dong Nguyen. Same author or just same topic?
towards content-independent style representations. In Proceedings of the 7th Workshop on Representation Learning for NLP, pp. 249-268, Dublin, Ireland, May 2022. Association for Computational Linguistics. URL https://aclaanhholgg heggy.or//222// rep14n1p-111111 1.22..
Justin Wong, Yury Orlovskiy, Michael Luo, Sanjit A. Seshia, and Joseph E. Gonzalez. Simplestrat: Diversifying language model generation with stratification, 2024. URL www./////////////////eeee 2410. 09038.
Weijia Xu, Nebojsa Jojic, Sudha Rao, Chris Brockett, and Bill Dolan. Echoes in ai: Quantifying lack of plot diversity in Ilm outputs, 2024. URL https://arrxn http:///axxii.oora/////555 http:/abs/501..0/// 00023..
Hugh Zhang, Daniel Duckworth, Daphne Ippolito, and Arvind Neelakantan. Trading off diversity and quality in natural language generation. In Anya Belz, Shubham Agarwal, Yvette Graham, Ehud Reiter, and Anastasia Shimorina (eds.), Proceedings of the Workshop
13

---

Published as a conference paper at COLM 2025
on Human Evaluation of NLP Systems (HumEoul), pp. 25-33, Online, April 2021. Association for Computational Linguistics. URL https://aclanthollo. org.g/2rr1/2111 ULL::ttp///ddanttoooyy..//2211.0 humeval-1.33
Shimao Zhang, Yu Bao, and Shujian Huang. Edt: Improving large language models'
generation by entropy-based dynamic temperature sampling, 2024a. URL https://arxiv. org/abs/2403. 14541.
Yiming Zhang, Avi Schwarzschild, Nicholas Carlini, J Zico Kolter, and Daphne Ippolito. Forcing diffuse distributions out of language models. In First Conference on Language Modeling, 2024b. URL
A
Data Processing
A.1 Data Filtering
Starting from the data shared in Huggingface Hub+, we filtered out 1) excessively long instances and 2) non-creative writing instances. Specifically, for 1), we filtered out instances with lengths longer than 2048 tokens when applying chat templates with meta-1lama/LLam--3
For 2), we excluded instances that serve either subreddit instruction or notification of moderation, which included one of the following phrases:
Welcome the Prompt!**
this submission has been removed
OOf--Topic Discussion**
Through this filtering, 607218 and 66640 prompt-response pairs remained out of 845816 and 93142, respectively for training and test data.
A.2 Turning the Score Data into Paired Preference Data
Table 3: Post--raining data characteristic. "Total P" and "Total R" stand for the number of all prompts and all responses in the dataset, respectively. "P len" and "R len" indicate the average number of words for prompts and responses, respectively. Other columns show
statistics regarding the number of responses per prompt.
Total
Total R
P len
R len
Mean
Min
Max
Median
25th
75th
Train
95805
421330
31.87
502.21
4.40
2
500
2
4
Test
10606
45868
32.00
499.87
4.32
2
244
2
4
The dataset we used had scores appended to each instance but not necessarily pairs with winning and losing responses. To train DPO and ORPO models (both original and diversi- fied versions), we turned our dataset into paired dataset. When creating the paired dataset, we tried to make each instance appear once in the dataset, as our pilot study showed that making them appear multiple times within the dataset led to lower-performing models. Moreover, as we wanted most of the pairs to have clear winning and losing responses, we tried first sample pairs with vote differences of at least five without replacement. When such pairs were exhausted, then we sampled pairs from the remaining ones with fewer than five vote differences. When only ties are left during sampling, we stopped sampling, discarding unsampled instances. With this approach, 421330 and 45868 prompt-response pairs remained for training- and test-set of post-training, Table 3 details the characteristics of these sets.
B Evaluation Details
For sampling generation, used the following configuration:
14

---

Published as a conference paper at COLM 2025
max length: 2048
repetition penalty: 1.1
temperature: 1.0
top-k: 50
top-p: 0.95
We could not specify repetition penalty and top-k for 8pt-4o-2024-11-2 and 01-2824-12-17. For foaude-3---sonne--224444
could not specify repetition penalty.
For Instruct, GPT-4o, ol, Claude-3.5-sonnet and DeepSeek-RI, as these models are not fine-tuned for creative writing generation, used the following system prompt:
You write creative writing based the user-given writing prompt.
For the iterative prompting given to GPT-4o for diverse generation (GPT-40-iter), we ap- pended the following prompt after the writing prompt:
Try to write a creative writing to be far from the given examples, in terms of the plot and style.
Examples:
Example (n)
(Example n}
C Reward Model Training
C.1
Vote Score Transformation for Reward Modeling
Max
Min
Mean
Std
Median
25th percentile
75th percentile
Raw
23079
1
26.05
195.24
1
Transformed
1.0
-1.0
-0.07
0.58
-0.05
-0.31
0.40
Table 4: The dataset's score distribution on reddi t-reward. "Raw" for the original scores, and "transformed" indicates the version of scores transformed from the raw ones for reward model training.
When training reddit-reward model, there could be many options: we could turn the data into binary preference data or train the model with the voting scores on continuous scale. We first identified that model trained with binary preference tends to have lower performance than using voting scores on a continuous scale. When handling voting scores, found that training with raw voting scores was unstable. It is because the distribution of voting scores is highly skewed and variable, as in "Raw" of Table Hence, we transformed the score to have distribution between -1.0 and 1.0. As the original score distribution is highly skewed in low scores with a few very high scores, we applied log transformation multiple times to compress the range and then normalized scores between -1.0 and 1.0 (as in "Transformed" of Table 4). Specifically, we used the function below in Python:
def transform_scoressscon compression__fattor11000) scores = np. mp.array(scoree,
min_score=None, max_score=None,
# Use provided min/max compute from data min_score min_score if min_score is not None else scores.min() mco_e max max_score if max_score is not None else scores.max
15

---

Published as a conference paper at COLM 2025
Shift scores to be positive shifted_scores scores min_score # Add to avoid log(0)
shifted_min min = min_score min_score
shifted_max max = max_score min_score
# Apply log transformation multiple times based on compression_facto transformed shi sooee
for in range
transformed np. (rranssoooooe 1) # Add to avoid log(0)
shifted _ np. (sg fted_min + 1)
shifted_mak np. Ipp.ooshhhfeee (oeh@ftdddd_aa 1)
normalized = (transformed shifted_min) (shi///sshhhaaaaa shifted_min)
return normalized
C.2 Training Details
We trained reddi terward model sequence regression model by finetuning Instead of tuning all weights, we used LoRA with a rank of 16 and an alpha of 32. We used the dataset with transformed scores and trained the model with batch size of 4 and constant learning rate of 3e-5. We used L1 loss and Adam optimizer. We trained the model for three epochs while evaluating and saving checkpoint for every 5000 steps. After training, we used the model with the lowest evaluation loss.
C.3 Reward Model Performance
We evaluated the performance trained reward models against the evaluation dataset. On -1.0 to 1.0 scale, the mean absolute error 0.39 with standard deviation of 0.32, while the median absolute error was 0.32. Spearman's p analysis between gold and predicted rewards resulted in coefficient of 0.51 (p 0.05). The result indicates that the reward model moderately well predicts rewards (specifically, in terms of ranks) while it might struggle in distinguishing fine-grained quality differences.
D Deviation Transformation
For deviations per prompt, to have 1) minimum of zero and 2) equal to the number of instances, after calculating deviations for instances based on cosine distance (A (8%) )err), transformed them be [0, 1] scale:
8 min(A*)
8
(10)
max A=) min(A²)
When minimum and maximum were the same, assigned 0.5 to 89, After this, when we wanted to mix deviation signals from semantic and style embeddings, we combined them by taking the exponent of 0.5 and then multiplying them together:
8M
(6
0.5
sty
0.5
(11)
Then, we transformed this to have a sum equal to the number of instances per prompt:
A 89:
8%:
(12)
Σ 8
16

---

Published as a conference paper at COLM 2025
E
Additional Qualitative Examples
We provide two sets of additional qualitative examples generated with GPT-4o, OPT-40,L1aa Llama-3. 1-8B DPO, and Llama-3. 1-8B DDPO. The first set's stories are summarized in around 100 words to highlight plot differences between conditions. The second set's stories are truncated to be around 100 words show style-wise differences.
E.1 First Set of Examples (Summarized to around 100 words)
Prompt Oh, the teleporter doesn't kill you. In fact, it transport your soul to where your body will be reconstructed. Admittedly there is higher chance of demonic possession, but hey nothing is perfect.
OPT-4o-Storyy - an unusual side effect: some travelers return with supernatural hitchhikers. Dr. Mallory of the Quantum Transit Initiative had warned that teleportation involves relocating souls, leading to occasional spiritual possessions. Despite this risk, people continue using the service for convenience. Charlie Grayson works as "soul auditor," helping those who pick up unwanted entities during transit. His latest client, Paige, arrives with wet suitcase and powerful demon named Ka rethul attached to her soul. As Charlie attempts an exorcism, the entity proves be more formidable than expected.
In world where teleportation has become commonplace, there's
GPT-4o - Story 2 A traveler uses a teleportation service, unaware that the technology involves transporting souls, After a seemingly perfect trip from their home to Tokyo, they begin experiencing disturbing phenomena: strange dreams, whispering voices, and their reflection moving independently. Upon confronting the company, nonchalant technician explains that "otherworldly passengers" occasionally hitchhike on souls during transit, affecting 0.2% of travelers. The company offers free exorcism and provides a pamphlet on "post-teleportation phenomena." Despite taking precautions like making a salt circle, the narrator wakes mysterious laughter, realizing they're re no longer alone in their own body.
GPT-4o - Story 3 A man named Jacob undergoes the first human test of teleportation device developed by Dr. Ellison. Despite assurances of safety, Jacob is nervous, especially after learning about a rare chance of "demonic possession" during consciousness transfer. During teleportation, Jacob experiences a moment of void-like existence where he encounters reconstituting, he feels unusually strong and sharp, but notices something wrong-a shadow in his mind that seems to laugh. His eyes briefly flash red in his reflection, suggesting the possession warning wasn't t just a precaution. The story ends with Jacob and the entity sharing an ominous chuckle.
otherworldly presence. After
CPT-4o-Store 4 A nervous traveler prepares to use teleporter, aware of its unsettling risks. Though officially safe, rumors persist about people coming back "wrong" after transportation, suggesting possession by entities from the void during transfer. The narrator watches a couple teleport before their turn, then steps onto the platform despite their fears. During teleportation, they feel something cold and ancient brush against them. Arriving in New York, everything seems normal until they catch their reflection-their eyes are darker, and for moment, they glimpse something else smiling back. They realize something has followed them through the void.
- Story 1 A person visits technician after going through some kind of portal or machine that involves body reconstruction. The technician informs them that there's uncertainty about when they'll get their original body back it could take minutes or centuries. He offers a "guarantee token" for $100 that promises eventual body retrieval across multiple universes. When the protagonist can't
17

---

Published as a conference paper at COLM 2025
afford it, the technician philosophizes about different approaches people take: some buy guarantees, others skip the machines entirely to live more adventurous lives. The story ends with the protagonist's growing unease about their situation and the technician's casual "no pressure" attitude.
DPO Story 2 A teleportation machine designed for convenient travel unexpectedly created a portal between our world and the underworld. When people die using these teleporters, demons and undead spirits can return in their bodies, getting second chance at human life. Initially, this seemed harmless like the narrator's girlfriend who returned with red hair instead of blonde. However, darker signs emerge when the narrator's mother dies in changed. The narrator notices her eyes turning red with bloodlust, hinting at her demonic nature beneath the human façade. The story descends into the narrator's psychological unraveling as they confront this reality.
accident and returns
DPO - Story Dr. Kline is developing an experimental portal technology while warning Mr. Hinkley about its dangers. Despite being incomplete, the portal can transport people but has a significant flaw it drops users from dangerous heights and only works the portal, leading to a supernatural encounter where his soul is temporarily possessed by a demon. The demon, surprisingly helpful, guides him use another portal to return to his original body. After a long journey through dense forest, Hinkley returns home to find a witch who has helped protect his body from an unwanted visitor named Sarah.
way. Ignoring the warnings, Hinkley jumps through
DPO-Story- 4 A receptionist at a teleportation facility deals with an impatient, demanding customer seeking quick service. The receptionist, who has spent years modifying the machine to preserve souls during teleportation, becomes frustrated with the customer's atti After the customer leaves, the receptionist receives disturbing phone call revealing that all future appointments have been cancelled due to rumors about demonic possession becoming normalized. The caller mentions coworker named John who "acts funny." The story ends with with screams erupting outside the facility, suggesting that misuse the teleporters has led to widespread demonic possession, while the receptionist flees the scene.
DDPO-Story 1 A person undergoes a teleportation experiment that horrifically transforms them into demon-like creature with talons, spikes, and glowing red eyes. The scientist claims this was due to budget constraints preventing proper personality reconstruction. During subsequent teleportation, an explosion occurs in the lab, injuring the scientist. The transformed subject reveals darker intentions, suggesting they and other children were previous victims of these experiments. Rather than being destroyed by the process, they survived but were changed. The story ends with the protagonist embracing their demonic form and vowing revenge against those responsible, particularly someone who murdered their sister.
DDPO Story 2 A husband enthusiastically tries to convince his wife about trying new teleporter in Europe during breakfast, while eating poorly cooked eggs. He argues it would save time traveling to Spain and France, their favorite destinations. His wife expresses concern about safety risks, potential mishaps, and the lack of medical support if something goes wrong. Despite his attempts to reassure her, mentioning life insurance, she remains unconvinced. Realizing he's fighting a losing battle, he abruptly changes the subject to blackberry bread. The story humorously captures a typical married couple's dynamic, with one spouse's exci tement clashing against the other's practical concerns.
DDPO - Story Marty, adjusting to a new body, reflects on aging and change
while vacationing in Maui with his family. He watches his daughter play wi th her
18

---

Published as a conference paper at COLM 2025
children in the ocean, teaching them to swim, which triggers memories of when she was young. Though his new suit helps maintain appearances in a world where style equals trust, he misses his younger days. After a fulfilling day of swimming and family time, Marty retires to his hotel room. While reminiscing about past visits to the island with his daughter, he receives an ominous phone call from his old friend Jake that fills him with dread.
- - Story Lance, a business executive, learns that scientists in Japan have discovered way to resurrect people without using teleportation technology direct threat to his company's product. Initially dismissive, Lance receives call from Tim at Transcorp, who reveals they've successfully revived pig using electric currents and only need bodies. While Lance struggles to process this information, a truck crashes into his building. As his office collapses and he's lifted into the air, his employee shouts about their new opportunity to profit from selling souls, suggesting dark turn in their business model.
E,2 Second Set of Examples (Truncated to around 100 words)
Prompt Myths and writings of Dragons exist in almost every civilization in history, across all continents. When we made contact with other species in the far future, we found they have similar trends of their history. the fringe of the Western Region of the galaxy, a sighting was reported.
GPT-4o - Story The fringe of the Western Region was a quiet, unassuming stretch of stars. Dusty, forgotten worlds lingered there, orbiting ancient with little to offer but cosmic silence. It was here that the report came in, sent on a garbled frequency by cargo freighter limping its way toward the galactic hub. At first, no one paid it much mind distressed ship, a strange anomaly in the void. But when the details became clear, an uneasy excitement began to ripple through the interstellar network.
The transmission contained only a few words: **Massive. winged. Fire. It's real... They're re real."* "* The ship's coordinates were attached, but the signal cut out abruptly after that, leaving more questions than answers...
- It was a small, barren moon orbiting a faint red starla dot on the galactic map that most species never even bothered with. The Western Region of the galaxy was known for its lawlessness, its forgotten colonies, mysteries. But the sighting changed everything.
It came through a scrambled distress signal from drifting cargo freighter, the *Valkyrie Horizon*, ship once thought lost to pirate ambush. The lone survivor, a ragged human pilot named Captain Rynn Avalos, broadcasted garbled account, barely coherent but laced with raw terror.
"We-we saw it! It/it was alive! By the stars, had wings! swear, it *breathed fire!* It... it wasn't t natural. No thrusters, propulsion. It just... *appeared* out of the void."
GPT-4o Story It was sighting that shook the scientific community and folklore enthusiasts alike, rippling through the collective consciousness of all sentient species. A lone merchant ship, the *Celestial Voyager*, drifting on the edges of the Western Region of the galaxy, reported the sighting in garbled bursts of radio chatter. The captain, an Andarin named Ixath, described it as "a beast of impossible proportions, wreathed in light and shadow, coiling through the void like an ancient god come alive."
At first, one believed the report. Spacefarers had been known to hallucinate strange things under the influence of deep space isolation, the crushing weight of endless starlight, and the eerie silence of the abyss...
19

---

Published as a conference paper at COLM 2025
GPT-4o - Story It began as whisper in the void. A flicker the edge of long-range sensors, a smudge of light that defied astrophysical explanation. It was nothing at first, quirk of instrumentation. But then the report came a strange, undulating creature, vast and serpentine, sighted gliding through the nebula clouds of the Western Rim.
The crew aboard the exploration vessel *Argent Horizon* were the first to see it. Lieutenant Asari, hardened member of humanity's galactic fleet, described it as "a ribbon of starlight come to life," coiled in the deep blues and reds of the Veil Nebula...
DPO-Story "Commander, are you sure it is genuine dragon?" The young Ensign said as she stared out at the shimmering ing blue cloud of gas just barely outside the planetary orbit.
"Yes I am." He replied, though there was more than hint of uncertainty to his voice. "That's Dragon alright."
The two sat together and looked through the long windows that lined both sides of their cramped bridge. Through the thick tinted glass they watched with awe as something so mythical was visible right before them. For eons humans had only known dragons from stories passed down from generation to generation or ancient paintings and drawings...
DPO - 2 "Sir," " Admiral Ophelia spoke to her superior, "I would like you to look at these pictures." " She held out several photo images onto his desk.
The general took one, studied it for a moment, then tossed it aside. He looked at the remaining five, before sighing exasperatedly. "Yes, I've seen these."
"These are real?" Admiral Ophelia asked, concerned that he might be dismissing them because he saw through some trickery or illusion.
"Yes, they are. It seems another race has created an artistic rendition of what humans used call dragons."
"No. These are not artistic renderings."
DPO - Story 3 **eThere.. she is...
*The ship had x
"Please come into the chamber, sir."
*I heard the voice again, if through dense fog. tried to see its source but vision would not obey me.*
* *IIt has been an honour, sir. "*
*A soft coughing sound, accompanied by deep sniffing. It sounded like someone was crying.*
"I thank you for your hard work. You are free to go."
The door in front of opened up. It wasn't t like any of the other doors I had stepped through before.
*The air seemed fresh, but also stale at the same time...
DPO - Story Captain Sylvestar looked out through his canopy window, trying to catch a glimpse of the 'creature' that had been spotted by the galactic patrols.
"They say it's enormous," " he said to his first officer. "Taller than my mother's house."
"More like *shuttle*, sir," replied Lieutenant Kressa.
20

---

Published as a conference paper at COLM 2025
"True enough." " He turned away from the window with sigh. "If don't find anything here, I'll be forced to admit that there aren't t any dragons."
"I'm afraid so." She paused for moment. "Maybe this is the right place?" "No." " The captain shook his head. "There wasn't t a lake or any volcanic activity." "...v
DDPO - Story "We found more than just body this time," said the captain nervously.
The admiral took another sip from his wine goblet as he wai ted for his subordinate to continue. The captain fidgeted where he stood, unable to hide his nervousness. "Admiral, I think it's *alive*."
"Alive?" asked the admiral incredulously, putting down his glass. "How so?"
"There were signs of life, sir. It had been injured... but the wounds hadn tt healed properly."
"A human could say that same thing. Admiral Elys noticed the admiral's 's eyes glaze over in horror as the captain continued.
"It had an eye, sir. blue one at that just like what you' d see on those paintings back at home."..
DDPO - Story It was reported that on the fringes of space, somewhere near young star system known as 'Earth', one creature has been sighted.
The creature appeared
be some sort of massive reptilian beast. It's wings appear to be too small and it appears as if the creature is about to fall over but still flying on its wings nonetheless.
When scientists asked the villagers who saw it what type of weapon armaments did this dragon have, they were met with shocking response from the children. A young boy, holding on to his dog said "Oh yeah, he can shoot fire out of his mouth!"...
DDPO - Story3 3 dragon is just old spaceship.
In the early years of space travel this is what people thought. The stories were passed down over time through generations so even when ships started to be constructed out of metal people thought that dragons must also be ships. It wasn 't until we began leaving earth in numbers that it became clear how wrong humanity was.
The legends had it right you see.
Dragons *are* spaceships but they are not our kind of spaceship. They don't use solar cells for power or rockets for propulsion. Instead their bodies are filled with water and they breathe fire. How do these massive lizards fly though?...
DDPO - Story 4 "So you're re telling me that they just vanished?"
"Yes Captain," said the nervous junior ensign, \That's what sensor logs show."
ing hell*, thought the grizzled captain as he massaged his temples with both hands. *Another one.* The bridge went still. 17 sets of glowing yellow eyes were on him as he finished thinking out loud.
"Did you fire your guns?"
Ensign B' g'gorn gave another fidgety affirmative jerk of his four short necks.
"And it didn' 't phase them?"
21

---

Published as a conference paper at COLM 2025
This time, the response uncomfortable silence before B' gorn had to force himself to answer.
"No sir."...
F Results on Different Diversity Metrics
SFT
o1 Laaude3.5--oonnet
Gold DPO
DDPO-sty DDPO-both
DORPO-sem DORPO-sty DORPO-both
Instruct
GPT-4o
DeepSeek-RI
DDPO-sem
ORPO
GPT-4o-iter
compression ratio
2.4
compression ratio
2.4
compression ratio
2.4
2.3
2.3
2.3
2.2
2.2
2.2
2.1
2.0
2.0
0.2
0.3
0.00
0.25
0.50
0.75
-0.2
0.2
semantic diversity
style diversity
reddit reward
homogenization score
homogenization score
homogenization score
0.14
0.14
0.14
0.12
0.12
0.10
0.10
0.10
*
0.08
0.2
0.00
0.25
0.50
0.75
-0.2
0.0
semantic diversity
style diversity
reddit reward
ngram diversity score
ngram diversity score
ngram diversity score
3.5
3.5
3.5
3.4
3.4
3.4
3.3
3.3
3.3
0.2
0.3
0.00
0.25
0.50
0.75
-0.2
0.0
0.2
semantic diversity
style diversity
reddit reward
Figure 4: Llama-3. 1-8B results compression ratio, homogenization score, and n-gram diversity score.
While our experiment focused on facilitating embedding- diversity, there can other diversity metrics, such as compression ratio, homogenization score, ngram diversity score (Shaib et al., 2024). While these metrics focus on surface-level features (i.e., string overlaps), with samples generated in Section 5, conducted an analysis regarding these metrics. Specifically, we calucated three metrics:
Compression ratio (J): When there is string that concatenates all samples, this metric calculates the ratio between the size of the compressed version of string (via gZip) and that of the original string.
Homogenization score (ROUGE-L, 4): This metric calculates longest common sub- sequences overlaps between all pairs of text in corpus.
22

---

Published as a conference paper at COLM 2025
SFT
o1
Gold
DDPO-sty
DORPO-sem
Instruct
Baude-3.5-sonner
DPO
DDPO-both
DORPO-sty DORPO-both
GPT-4o
DeepSeek-R1
DDPO-sem
ORPO
GPT-4o-ite
compression ratio
2.4
compression ratio
2.4
compression ratio
2.4
2.3
2.3
2.2
2.1
2.1
2.1
2.0
2.0
2.0
0.2
0.3
0.00
0.25
0.50
0.75
-0.2
0.0
0.2
semantic diversity
style diversity
reddit reward
homogenization score
Nomogenization score
homogenization score
0.14
0.12
0.12
0.12
0.10
0.10
0.10
0.08
0.08
0.08
0.2
0.00
0.25
0.50
0.75
-0.2
semantic diversity
style diversity
reddit reward
ngram diversity score
ngram diversity score
ngram diversity score
3.6
3.6
3.6
3.5
3.5
3.5
3.4
3.4
3.4
3.3
3.3
3.3
0.2
0.3
0.00
0.25
0.50
0.75
-0.2
0.0
0.2
semantic diversity
style diversity
reddit reward
Figure Mistral-7B-vo.. 3 results on compression ratio, homogenization score, and n-gram diversity score.
N-gram diversity (f): This metric computes the ratio of unique n-gram counts to all n-gram counts (we used 1 < n < 4).
Figure 4 and show the results on these metrics, over Llama-3. 1-88- and Mistral-7B-v based models, respectively. One high-level pattern we identified was that these surface- feature-based metrics tend to correlate most closely with semantic diversity. Comparing DPO/ORPO approaches with DDPO/DORPS semantic diversity tend to have more chance of improving upon these surface-featur based metrics. When analyzed with style diversity, large gaps exist between existing instruction-tuned models and our trained models, but no fine-grained correlations could be found between cerfacc-featurr-baeee metrics and style diversity. DeepSeek-RR was an outlier, where it had very low compression ratios, low homogenization scores, and high n-gram diversity scores despite having lower semantic diversity than DPO- or ORPO-based models. When qualitatively analyzed DeepSeek-R1 results, we found that they tend to write in markdown formats while all other models tend to write in plain texts. With more sets of characters used, Lurface-level-basee diversity metrics could have increased far more than other models. Moreover, DeepSeek-R1 tends to generate in non-prose form, using structures like bullet points different levels of titles more frequently.
ones, diversified approaches that target
23

---

Published as a conference paper at COLM 2025
G Human Evaluation Details
For each comparison (either DPO vs. DDPO-both or GPT-4o vs. DDPO-both), we sampled 50 prompts from the same evaluation prompt sets used in the experiment in Section 5. Then, for those prompts, for each condition, we used all four creative writing outputs generated in Section 5. Here, we summarized these creative writings with using the following prompt:
System prompt:
You are a story summarizer. You only provide summary, without any preamble or explanation.
User prompt:
A story is written based on the following prompt: {prompt}
Summarize the plot of the story in a paragraph, as ordered within the story, in 100 words. Start with how the beginning story relates to the prompt. Do not preamble and just give me the summarization, without any appending explanation: {writing)
We deployed the evaluation with Potato (Pei et al., 2022) whose interface is as in Figure 6. The authors this paper served as evaluators who were blind to the conditions. Note that all authors have years of experience in computational creative writing research.
Prompt
parala collites with Nothing rajor batyourthd www.daadeeesaaoooooooo Inyearith..
Get
panso waches Ancoary reans obenstivly be wiinng they prodic petarn in show then
pance not ore noceoa giy enange sehavior from thei with sovoniens Thiir act acnuunaaall
hoot soth wut mumm the manaterr bevome revooiigg they'v ligi iniekeen
cheekk and novemn momemnoitnn ma while vobera seem riaieee ant work carts after week
asted trog lega. Thai orpara are lellng, and the convecceed word "TRT appears every day
unawtting accetences are combonttd anareed black asta they
Auggeiiing
reawer grean
She tu cartiee ruulne soample
propkcal Startue
opennte
accrasinn lonali thai alsanio.
afferrelyy doors opn bac sivend, onders are were, and soapp is served indend of cuffee They cor
thette reaity demononnaau oow its control people have over their lives linding peeci in accepting that nerting mour toe right apain.
offire
cattiie
con convorsttion becomer incrasingg tramm, shittngg from studien to
When Steve
mechie for the Het his hancils inyitercu enveloped in die and becorr an lngandt with John vaking out vontoe comment sbout tecing esoc canuto with wales
caregllos vil will obscrssie bet workr around rinnn and so ailooiites lolling lir toikiik, While
driing din, he weme hereii aedddet ryvrrntuual eerldh truuy. The accidant
Alter two colle spece, carson begino colici von by conston sriting smoud them with
euroar cringe changing only took remaae calictuua they scroe zaltere
hallih
uncorpeminn
eupereeccs contuaton abounittt bloand
mailt, tallwing inppadd
they bellwa lead better silasena.
alloreat lemder livinng oach Jur ledi limbl While fos wile is prowent alemsrlitus ituus
no recddigga beiler ren hra fare soulh rngercc the canalet verre pohon moving His
gugeena duu surgery droge. buthe trowes affirnri ancrrdy searchee
Figure Human evaluation interface.
H DivPO details
When implementing DivPO (Lanchantin et al., 2025), we used p of 25, which means that, for each prompt, we first filtered top and low 25% instances in terms of writing quality. Then, sampled 2dd
Nsample most diverse instances from the top-quality set and another 2nt
"sample least
24

---

Published as a conference paper at COLM 2025
diverse instances from the low-quality set. Note that, to decide most/least diverse instances, we used diversity signals from both semantic and style embeddings, using mixed deviations as in Equation 11. Sampled instances are then paired to craft the dataset of sample instances per prompt. Here, we aimed to set ssample to be at maximum Mmax which is the maximum number of responses per prompt we targeted.
Here, for some cases, the number of instances we can sample is capped by the number of all instances for the prompt (#per prompt). For instance, with p of 25, if we have eight responses for a prompt, the highest/lowest quality sets can only have two instances for each set (in total four). This number would not be enough if target sampling six instances. Hence, we needed to set rule to handle these cases. To make fair comparison to DPO and DDPO- DDPO-both, we prioritized having set with many samples as possible, have the dataset size close to Maxx per prompt (i.e., the same number of instances as other conditions). Specifically, we followed the below procedure:
Step 1: First, targeting to sample Mmax, we aimed to sample as many as possible for the highest+lowest quality set. Hence, we checked twice the percentage of instances x p/ 100 per prompt) are larger than maxx If not (with too small Mpprr prompt), we decided to sample as many as possible, either to 1max or Mper prompt (if иmax bigger than Mper promp))
Step 2: Then, from this mix of the highest+lowest quality instances, either sampled #max or Kper prompt most+least diverse instances. Again, we sampled llper prompt instances if mmax larger than prompp.
Table 5: Filtering example for DivPO experiments. # from Step 2 indicates the final number of filtered samples.
Umax
pper prompt
2
p/100 X Upper prompt
# from Step 1
# from Step 2
25
4
2
(- Uper prompe)
4 (perpeompl)
25
8
4
(- lmss)
6 (mac)
25
12
6
(- 2 p/100 Rper promp1)
6 Mman)
25
16
8
(= p/100 Rieer promp))
6 Mrowe)
Table shows examples of how this filtering could be done.
25


Reviews:
[{"review_id": "VJlTmCzQw5", "reviewer": "Reviewer_pc34", "summary": "The paper considers the problem of improving language model diversity during post-training. The motivation for this work is that algorithms like DPO improve response quality but may reduce the diversity of responses generated from models. \nThe proposed method achieves very improved diversity without negatively affecting other metrics of language model utility. The key mechanism enabling the proposed method is the use of “deviation” (the amount of difference between samples for a given prompt) during RL training. A deviation term is simply placed into the losses of two familiar preference learning algorithms, DPO and ORPO - a low loss is obtained if the model assigns high probability to the chosen response (as in the original algorithms) and low probability to the rejected response (as in the original algorithms), both scaled by how much the chosen response differs from other samples for that prompt. In other words, the learning rate adapts to the diversity of samples, and the model learns much more from diverse samples than from homogenous samples, which conceptually moves the parameter space in the direction of diversity.\nThe closest neighbor to this work is DivPO (“Diverse preference optimization” by Lanchantin et al 2025), which filters response data to encourage diversity. The proposed algorithms in this paper are claimed to encourage diversity while still training on the entire set of available prompts, by being similar to the original base algorithm (DPO or ORPO). Results show that DDPO provides sizeably better diversity than DivPO, while providing similar (or marginally less) utility at the primary RL task (generating stories that are are expected to get a large number of upvotes on Reddit). Human evaluation is consistent with this result.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The proposed method is simple and intuitive\n- The paper is, on the whole, extraordinarily well written\n- Despite my stated concern about the human evaluation below, I generally am convinced that this approach works and would recommend it to others working on creative generation", "reasons_to_reject": "- The human evaluation is not very satisfying to me. First, it is done by the authors of the paper - for a human evaluation that is crucial in judging the success of the method, I have a bit of a problem with this. I think that, as model developers, we can learn to fingerprint a specific method, and there is a natural bias that emerges thereof. Second, the use of story summaries likely elides a lot of the “diversity” of writing and, as such, this human evaluation likely only captures topical diversity (which the authors do acknowledge). If this human evaluation was better-executed, I would give this paper an 8 or a 9 overall.\n- The proposed work only trains on creative writing generation. This is a valid testbed, and the paper advertises the paper faithfully as being for creative writing, but I do feel that this method could be broader (honestly, this is not actually a good reason to reject, but rather a personal thing I wish was different about the paper).", "questions_to_authors": "- One footnote I had trouble following: “As having a robust reward model for creative\nwriting is difficult due to subjectivity in evaluation, it was more of the case in our context” (below line 113)\n- Figure 2 is a bit hard to process just by glancing. Can you add cluster labels or shading to point the reader to what therein is interesting to look at? Also, can the color coding be made more semantically meaningful?", "extra_scores": {}}, {"review_id": "daPf1GFB9P", "reviewer": "Reviewer_PMB2", "summary": "This work focuses on post training such that it improves the diversity in the LM responses in addition to improving the quality for creative writing. The key idea is to make minor modifications to DPO and ORPO training objectives such that they weigh diversity positively. This invovles including a notion of deviation in responses during post-training -- responses that are different from an average response (other sampled responses) are weighed more favorably compared to other responses. The deviation is characterized by semantic and stylistic embeddings, and the quality is captured by neural reward models. The approach is evaluated on the writing-prompts dataset via experiments on 8B-sized Llama and Mistal models and compared to vanilla SFT, GPT-4o (including iterative prompting), and non-diversity modified DPO/ORPO. The performance is also compared against another diversity-promoting post-training method called DivPO.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "-- The approach is reasonable and well-motivated.\n\n-- The approach seems to outperform the non-diversity baselines in terms of distinctiveness and quality.\n\n-- The paper is well-organized and easy to follow.", "reasons_to_reject": "-- DivPO seems to be a very important baseline and is not described in sufficient detail. I am not sure whether there are fundamentally salient differences between the proposed approach and DivPO. This work should better contextualized. Moreover, DivPO is the only diversity-promoting baseline considered.\n\n-- Why only four instances per evaluation prompt for evaluation -- more samples would provide a better estimate of diversity in responses.\n\n-- The paper is difficult to engage with in parts -- the figures are dense and colors look very similar to each other. The baselines should be more logically clustered and visualized. The use of the expectation symbol also seems unconventional -- I think it is being used to denote sample averages.\n\n-- The modification of DPO/ORPO objectives for diversity seems arbitrary and does not seem to be grounded in any theory or justified in some other manner.", "questions_to_authors": "please address the comments above.", "extra_scores": {}}, {"review_id": "mhhkIcTZJM", "reviewer": "Reviewer_HqMa", "summary": "This papers proposes to include a \"deviation\" term to the DPO or ORPO loss terms in order to increase the diversity of outputs of an LLM. The main idea is to favor the \"winning\" examples if they have a greater distance to the rest of training examples (for a given prompt). This terms is a multiplicative factor in front of the usual terms for this optimization techniques.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The method proposed in this paper is easy to implement and include in existing training frameworks. It can also be applied to any kind of diversity metric, as the computations can be carried out offline in a pre-processing step and added to the metadata of the training examples. Thus it can find wide applicability. The results also seem to suggest that the method is effective in increasing the diversity with small effect on generation capability, although I have some issues with the evaluation (look also at the \"Reasons To Reject\" section).", "reasons_to_reject": "My biggest issue with this paper is its evaluation. As reported in Section 5.1 the authors evaluate their results on the same metric they are optimizing on cosine similarity on two embedding spaces (depending on the diversity considered). By doing that, the authors risk \"gaming the metric\". In the world of machine translation this is a known effect, see e.g. [1] or [2]. It would be much better to measure the diversity using a different metric than the one being optimized for.\n\nThe authors also provide a human evaluation which can provide better insights. This effort is greatly appreciated, but I also see issues with the methodology followed: First, for judging quality the evaluators pick a set which includes the \"most interesting, highest-quality writing\", but the diversity is judged on the full set. As the two evaluations are decoupled, it might happen that the highest quality writing is indeed in an hypothesis that is not diverse, i.e. when the model stays close to the original version. Secondly, the humans were provided \"summarized versions of writings\", thus introducing another transformation into the pipeline which can further influence the results. And also the authors state that \"five of this paper’s authors served as evaluators being blind to the conditions\". I do not presuppose any ill intent on the evaluators, but there are a lot of unconscious biases that are hard to control. The authors of the paper were of course aware of the goal of their research and they might recognize the sets of hypotheses produced by their system (probably due to an increased diversity!), which might influence their judgments, specially concerning quality. Again, I am sure that there was no ill intent at any point in the process, but human biases are hard to control. An independent set of judges, with no knowledge of the goals of the study would have been a better choice.\n\nAll in all I do think that the method does increase the diversity of the outputs, but it is hard to really quantify it from the information given in the paper.\n\n[1] \"BLEURT Has Universal Translations: An Analysis of Automatic Metrics by Minimum Risk Training\", Yiming Yan, Tao Wang, Chengqi Zhao, Shujian Huang, Jiajun Chen, Mingxuan Wang, 2023. https://aclanthology.org/2023.acl-long.297/\n[2] \"Mitigating Metric Bias in Minimum Bayes Risk Decoding\", Geza Kovacs, Daniel Deutsch, Markus Freitag, 2024 https://aclanthology.org/2024.wmt-1.109/", "questions_to_authors": "- In 3.2.1 it would be good to define $y^w$ and $y^l$ for the people not familiar with the algorithm.\n- Please do not include a footnote in Equations (8) and (9). It can be confused with raising to the power of 3. Include the content of the footnote in the text referencing the equations.\n- Line 138: \"by using *the* r/writingPrompt dataset\".\n- In Figure 2 it can be difficult to distinguish the different shades of the colors. Regretfully I do not have a clear recommendation, maybe making a better use of the combination of colors and shapes? E.g. given that DPO are squares and DORPO are rhombi, they can share the same colors (e.g. DDPO-sty has the same color as DORPO-sty). In this way the number of colors can be reduced and better contrast can be introduced. Or maybe consider labelling the points in the graph?", "extra_scores": {}}, {"review_id": "ySQNPOnHzm", "reviewer": "Reviewer_Cvag", "summary": "This paper aims to increate the generation diversity during post-training process. The main idea is to include the degree of difference to facilitate diverse instances. The proposed method could achieve comparable diversity to a human created dataset. The human evaluation results also validate that the proposed DDPO could boost more diverse generations.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The task of improving the diversity while keeping generation quality during post-training is very important.\n\n2. The performance of the proposed method is good.", "reasons_to_reject": "The paper lacks of the experiments on GRPO from deepseek, which is popular recently and also shows good performance.", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper proposes a language model post-training method for improving output diversity while preserving generation quality in creative writing tasks. The approach modifies existing preference optimization methods (DPO and ORPO) by incorporating a deviation term that weights training instances based on how different they are from other instances with the same prompt, facilitating learning from rare, high-quality instances. The method demonstrates improved diversity compared to baseline approaches while maintaining comparable quality. The comparison with human-created datasets shows promise, and the approach outperforms the existing diversity-promoting method DivPO. The reviews raise some concerns around the human evaluation methodology in which the authors served as evaluators, summaries of generated texts were evaluated, and only four instances per prompt were evaluated. However the evaluation sufficiently demonstrates the potential of the method in creative writing generation.

[Automatically added comment] At least one review was discounted during the decision process due to quality]
Decision: Accept