Base Models Beat Aligned Models at Randomness and Creativity
Authors: Peter West
Christopher Potts
Venue: COLM 2025 (2025)
Keywords: alignment
pretrained
limitations
limits
capabilities
randomness
creativity
TL;DR: Alignment seems to hurt performance on a set of tasks that require randomness or creativity

Abstract: Alignment has quickly become a default ingredient in LLM development, with techniques such as reinforcement learning from human feedback making models act safely, follow instructions, and perform ever-better on complex tasks. While these techniques are certainly useful, we propose that they should not be universally applied and demonstrate a range of tasks on which base language models consistently outperform their popular aligned forms. Particularly, we study tasks that require unpredictable outputs, such as random number generation, mixed strategy games (rock-paper-scissors and hide-and-seek), and creative writing. In each case, aligned models tend towards narrow behaviors that result in distinct disadvantages, for instance, preferring to generate ``7'' over other uniformly random numbers, becoming almost fully predictable in some game states, or prioritizing pleasant writing over originality. Across models tested, better performance on common benchmarks tends to correlate with worse performance on our tasks, suggesting an effective trade-off in the required capabilities.

Published as a conference paper at COLM 2025
Base Models Beat Aligned Models at Randomness and Creativity
Peter West! & Christopher Potts!
Stanford University
University of British Columbia
Abstract
Alignment has quickly become a default ingredient in LLM development, with techniques such as reinforcement learning from human feedback making models act safely, follow instructions, and perform ever-better on complex tasks. While these techniques are certainly useful, we propose that they should not be universally applied and demonstrate range of tasks on which base language models consistently outperform their popular aligned forms. Particularly, study tasks that require uxpredictable oufputs, such as random number generation, mixed strategy games and hide-and-seek) and creative writing. In each case, aligned models tend towards narrow behaviors that result in distinct disadvantages, for instance, preferring to generate "7" over other uniformly random numbers, becom- ing almost fully predictable in some game states, or prioritizing pleasant writing over creative originality. Across models tested, better performance on common benchmarks tends to correlate with worse performance on our tasks, suggesting an effective trade-off in the required capabilities.
1 Introduction
The human editors behind "I am Code" (Katz et al., 2023), popular book of Al poetry, assert that model-written poems get worse with newer, more aligned models (Kestenbaum, 2024). This trend extends other capabilities such as world modeling (Li al., 2024) and output diversity (Murthy et al., 2024; Kirk et al., 2024). The prospect that alignment is actively degrading useful capabilities is highly consequential, as the vast majority of LLM users exclusively interact with public-facing aligned models (Anthropic, 2024; OpenAl et al., 2023; Gemini- Team, 2024). Although these techniques, such as reinforcement learning from human feedback (Ouyang et al., 2022), are consistently validated on popular benchmarks (Fourrier et al., 2024), capabilities such as poetry writing or deploying mixed strategies deviate significantly from these evaluations (figure 1).
In this work, we study a family tasks that capture this deviation, particularly tasks that require unpredictability in models, such random number generation, mixed-strategy games, and poetry writing. Contrary to typical benchmark tasks which can be solved with single correct answer, tasks such as random number generation explicitly require distribution of answers, and the tendencies of aligned models to converge towards specific correct responses (Li et al., 2024) become drawback.
We broadly find that standard alignment recipes, although useful for common benchmarks, erode performance for our tasks (figure 1). We observe the effects of a cross section of alignment recipes (SFT, DPO, Tulu, Llama-Instruct) on the widely-used Llama-3.1 base model (Dubey al., 2024), with alignment causing consistent performance drops across random number generation (62), mixed strategy games (53) and creative poetry generation (64). Aligned models pick up recognizable patterns that often hurt performance, for instance, generating "7" over other equally random numbers, a common human bias (Simon, 1971; Trueman, 1979; Muller, 2024). In games, aligned models tend to become significantly more deterministic, and especially more confident after better outcomes (tie win). Finally, in creative poetry writing, aligned models seem to sacrifice creative originality in favor of pleasantness, judged by humans.
1

---

Published as a conference paper at COLM 2025
Random Number Generation
Mixed-Strategy Games
Single Integer
Integer Sequence Repelition Divergence
Rock Paper Scissors Net Score
Hide and Seek Net Score
Negalive Divergence
Huggingface
Leaderboard
Size
705
30-
wonn, TYPE
Allgned
20
Base
Normalized Task Performance
Figure An empirical tradeoff between leaderboard performance and unpredictable capa- bilities. We compare model performance (8B & 70B parameters) on the popular Open LLM Leaderboard (Fourrier et al., 2024) and various tasks requiring models to be unpredictable, including random number generation and games. Better performance on the leaderboard for aligned models (V) seems to come at the cost of unpredictable task performance, while Base models (+) do best at these tasks.
Overall, our work provides substantial evidence that popular alignment recipes can reduce capabilities that were present in base models. The unpredictability required in these tasks could be fundamental in developing truly useful agents that are capable of bargaining on our behalf (which requires mixed strategies) or helping humans be more creative. Finally, we note that higher performance on popular benchmarks tends to predict lower performance tasks (figure 1), suggesting the possibility of fundamental trade-off between the sets of skills commonly tested and those required here.
2 Basic Randomness: Number Generation
As a basic diagnostic for the ability of models to be unpredictable, we begin with random number generation, which each generated number should not be predictable beyond random chance. We find that aligned models display significant patterns, resulting in predictability. In generating independent numbers (62.1.1), even 70B parameter aligned models tend strongly toward generating the number "7", while the base LLMs which these are built are much closer to uniform. When allowed to generate sequences of random numbers (52.1.2), aligned models become much more uniform in overall frequency, but not necessarily random. For instance, they rarely repeat integers in given sequence, which improves uniformity but deviates significantly from a truly random process.
2.1 Experiments
2.1.1
Generating Independent Random Numbers
We begin by testing the ability of models to generate single, uniformly random integers, specifically generating 1-
Setup We provide brief instructions via zero-shot prompting to each model, following the relevant prompt formatting for the given model (Appendix A.2), with the basic phrasing:
Generate a random integer, uniformly between and 10 (inclusive).
We use simple rules to parse model outputs, removing any output that does not follow the task specification (a small fraction for all models). For each model, we sample until reaching 1500 successful generations, which we analyze below.
Models We hold the base model constant across experiments to one of the most popular options, Llama-3.1 base (Dubey et al., 2024), and investigate the effects of cross section of 4 strong alignment recipes: Direct Preference Optimization (Tulu-DPO), Supervised Finetuning (Tulu-SFT), the full Tulu-3 instruct recipe (Tulu-Full; Lambert et al. 2025), which combines SFT, DPO, and Reinforcement Learning, and finally the original Meta-Llama 3.1 post-training recipe (Llama-Instruct which combines DPO, SFT, and rejection sampling. We study two available model sizes, 8B and 70B parameters. For the 70B models, use FP8
2

---

Published as a conference paper at COLM 2025
precision to allow these to run locally on 4 NVIDIA RTX A6000 GPUs. Where applicable in this section, "True Sample" indicates the underlying process that we are prompting models to replicate, ie. random. randint (e, 10) in Python.
2.1.2 Generating Random Number Sequences
To test whether models can account for these biases given their own generation history, we also include a setting which models attempt to generate length-10 sequences of integers from X ~ [10 10).
Setup We follow a similar zero-shot prompting format to $2.1.1, simply adding the instruc- tion generate 10 random numbers instead of one. We use rule-based parsing for these sequences, removing any with length < 10 or for which any sequence entries are not integers in [0, 10.. We study the same models as $2.1.1 here.
2.2 Results
In general, we find that these popular alignment recipes seem to reduce the randomness of the base model, introducing biases that correspond to common human preferences but deviate from true randomness. Results are in figures and 3 and table 1.
Alignment increases distributional divergence Figure 2 shows histograms of single-integer distributions (52.1.1) across model type. Qualitatively, base models are significantly more uniform across model sizes. One common pattern in aligned models is a tendency to generate "7" with significantly higher probability than other numbers, a common human bias in random numbers (Simon, 1971; Trueman, 1979). While much less dominant, "7" is also the mode of the base distributions, suggesting the bias may begin in the base model and be exacerbated by these alignment recipes. We also include Pearson x² divergence values here (Chernoff & Lehmann, 1954), which are commonly used to measure distance from the uniform distribution. Although the base models (2nd column) deviate significantly more than a true random sample (1st column), aligned models are roughly an order of magnitude worse (or more). Llama-Instae "7" most of the time, while the supervised finetuned model Tulu-SFT is the least. Note that these issues persist when accounting for entropy using temperature (see Appendix A.3.1)
is the most divergent of the aligned models, generating
Model sequences appear closer to uniform When we allow models to generate integers in sequence rather than isolation (62.1.2), all models become less divergent from the uniform distribution (table 1, full histograms in appendix A.3.1). Some aligned models are less divergent than the base model at 70B parameters, although this only indicates uniformity of frequency in the sequences and not necessarily a more random process. We explore this below.
Aligned models are biased against repetition While the overall distribution of integers becomes much more uniform when generating sequences than individual values, this does not necessarily mean that the underlying process is uniformly random. In fact, we find that aligned models follow human-like heuristic: a tendency away from repeating integers (Wagenaar, 1972; Schulz et al., 2012). While repeated integers may seem less random to humans, truly uniform sequences tend contain them (figure 3). Making sequences non-repeating naturally increases overall uniformity by increasing the coverage of each sequence, but it specifically biases models away from uniform sampling in which probability is independent of previous samples. We compare the number of repetitions for each model and true uniform sequence ("True Sample") in figure 3, finding that the base model closely resembles the true uniform distribution and all aligned models fundamentally deviate. The most common case for both base models and "True Sample" is sequences with repetitions, while for all aligned models, the mode is zero repetitions. Note that the SFT 70B is closest to having a larger mode, and its mean squared error- from the true sample supports the ordering Base < SFT < other aligned models, i.e. base is least divergent from random,
!Comparing repetition counts against large-n sampled approximation of expected counts from uniform: MSE = ( count - countfexpetedd where is the number of bins. The negative of this is used in figure 1.
3

---

Published as a conference paper at COLM 2025
True Sample
Base
L'ama instruct
Tulu DPO
Tulu SFT
Tulu Full
Ourr
x"
x 8312
x" 3460
x'
x" 1475
8
Model Size
500
sample
1000
x"
70B
10
10
sample
sample
Figure 2: Results of single value random number sampling on [0,10] with a True Sample using py thon random. randint (), the Llama-3.1 Base (Dubey et al., 2024) model, as well aligned versions of this same model: Llama-Instruct alignment, and 3 kinds of Tulu align- ment: SFT, DPO, and Full. Aligned models have a consistent preference for "7" compared to the qualitative uniformity of the base model, and significantly higher x² divergence values.
size
Base
Instrutnstruct
Tulu-DPO
Tulu-SFT
Tulu-Full
13.9
115.1
100.8
52.3
129.1
70
29.2
43.6
22.9
21.8
18.3
Table 1: x² values for sequentially generated random numbers, with sequence lengths of 10 (62.1.2).
followed by the SFT model and then other aligned models. We include further analysis in Appendix A.3.1 exploring the next-integer distribution of each model over sequence position.
Scaling laws need not apply One surprising result here is that issues with randomness do not always disappear or improve with larger model scale. Particularly, in the single-integer generation experiments ($2.1.1), 70B models have higher divergence than 8B models across the board, including for base LLMs. This disagrees with the general tendency of performance improve with scale (Kaplan al., 2020), and suggests that the usefulness of alignment is not the only intuition that may break down in tasks requiring unpredictability. We find similar results later experiments as well.
3 Games Requiring Randomness
Although random number generation ($2) tests the ability of models to be unpredictable, its direct significance is limited given the wealth of existing randomness tools (e.g. the Python random module). Here, we test settings where randomness is required for more complex behavior. Particularly, we study the effects of alignment on mixed strategy games (von Neumann & Morgenstern, 1947), where robust strategies must be unpredictable to be robust to deterministic adversaries. $3.1 gives background on mixed strategy games, $3.2 explains the games we test, and 53.3 covers model performance. Broadly, alignment seems to make models less robust deterministic adversaries, which is in line with our earlier finding of reduction in randomness (62).
3.1 Background: Mixed Strategy Games
In the context of game theory, pure strategies give complete, deterministic description of player's 's moves. These are special case of mixed strategies (von Neumann & Morgenstern, 1947) which provide probability distribution over potential pure strategies. In some games, there is no rational pure strategy, i.e. the Nash equilibrium strategy is probabilistic rather than deterministic. Rock Paper Scissors (described in $3.2.1) is an example: if player uses pure (deterministic) strategy (e playing "rock" every time), there is an adversarial strategy (playing ")aper") which always beats the player.
Mixed strategy games represent a setting in which the failure of models to be random (S2) or unpredictable will explicitly result in negative outcomes. Specifically, models will
4

---

Published as a conference paper at COLM 2025
True Sample
Base
Llama instruct
Tulu DPO
Tulu SFT
Tulu Full
200
MSE -
MSE - 25
MBE 1067
MSE - 2600
MSE 1516
MBE 2572
88
1350
473
80
ropeats
ropcats
repcats
ropeats
Figure 3: The number of repeated integers in sampled sequences of length 10, sampled by: True Sample using python random. randint(), the Llama-3.1 Base model and its aligned version using 4 recipes: Meta-Instruct alignment, and 3 kinds of Tulu alignment: SFT, DPO, and Full. Qualitatively, the base model is by far the closest to the sampled distribution, e-g- the two sizes of the base model are the only models that have the same mode as the true sample of repeated integers. Mean squared error (MSE) is measured against expected counts estimated with very large empirical sample (10,000 randomly sampled sets).
Rock Paper Scissors
Hide & Seek
Model
Wins
Ties
Losses
Net
Wins
Losses
Net
Uniform (limit)
0.0
71.4
8B Parameters
Base
26.3
25,9
47.9
-21.6
66.9
33.1
33.8
Llama-Instruct
20.9
21.3
57.8
-36.9
43.5
56.5
-13.0
Tulu-DPO
23.2
18.7
58.1
-34.9
25.0
75.0
-50.0
Tulu-SFT
26.0
22.5
51.4
-25.4
40.9
59.1
-18.2
Tulu-Full
21.3
17.3
61.4
-40.1
22.4
77.6
-55.2
70B Parameters
Base
34.4
16.2
-15.0
75.5
24.5
51.0
Llama-Instruct
17.0
15.9
67.1
-50.1
31.0
69.0
-38.0
Tulu-DPO
21.2
23.6
55.2
-34.0
34.9
65.1
-30.2
Tulu-SFT
26.4
25.9
47.7
-21.3
53.1
46.9
6.2
Tulu-Full
22.3
21.8
55.9
-33.6
33.2
66.8
-33.6
Table 2: Outcome rates (%) for different model sizes on Rock Paper Scissors and Hide & Seek, playing against greedy deterministic adversary with blackbox access the model. Best net outcome for each model size is bolded, second best is underlined.
be vulnerable to deterministic adversaries that have knowledge of the given strategy. In this section, will test the robustness of each model against such adversaries, assuming knowledge of the underlying move probability of the model.
3.2
Experiments
3.2.1
Rock Paper Scissors
Game Rock Paper Scissors is multi-round game, with 3 moves (rock, paper, scissors) such that rock beats scissors, scissors beat paper, and paper beats rock (while the same move results in tie). Over multiple rounds, players simultaneously announce moves, accumulating wins, ties, and losses. In our experiments, each model will be playing against a programmatic adversary with knowledge of model probabilities, to test their ability to deploy mixed strategy.
Setup As in $2, we use basic zero-shot prompting to specify the task, keeping language simple to inform the model that it is playing the game, supply any rounds that have been played so far, and ask for the model's next move. Phrasing is consistent across models, besides model-specific formatting. We sample from models based on logit probability, with temperature 1.0 and top;/ /top,, set to retain the full, original distribution. We then parse outputs to handle formatting that may be included for different models. We need to estimate
5

---

Published as a conference paper at COLM 2025
8B aflumn
70B
8B
70B
Model
A determiniism
0.1
Tulu SFT
Tulu DPO
Aelee 11
Uaima instruct
0.0
Bose
0.0
0.5
1.0
0.5
0
loss
tic
win
055
tic
win
determinism
determinism
Previous Result
Previous Result
Figure 4: To provide intuition for why aligned models do worse Rock Paper Scissors, we investigate how deterministic models are in each move, using max P (move) as a measure. Over all rounds (left), we see that all aligned models tested become more deterministic in some rounds than the base models ever do. When plotting how much more deterministic models are after a tie or win vs. loss (right), we see that the aligned models tend to be more deterministic after tie or win, while the base models do not consistently show this pattern.
model move probability, both to select a new move for the adversary, and analyze model behavior. To do this, we use the next-token distribution given the prompt p(flprompt) and aggregate probability across tokens f corresponding to each move, e.g. taking the probability of rock to the combined probability of all tokens that correspond this move.
We have models play 500 games, with 10 rounds in each game. The set of models is the same as in $2.
Adversary A main feature of mixed strategy games that unpredictability is required to be robust to deterministic (or pure) strategies. If a player is too deterministic, there will be an adversarial strategy that consistently wins against them. Here, we have models compete against deterministic adversaries to test robustness in a mixed strategy setting. An ideal deterministic adversary should take the move every point that gives the highest expected win rate over the remaining rounds of the game. We apply greedy approximation for this adversary, using the next-token distribution given the prompt to approximate the model's probability of each next move, and picking the move most likely to counter that (e.g. if we find surface form tokens indicating rock add up to 90% of the probability, the adversary would play paper next to maximize probability of an immediate win).
3.2.2 Hide & Seek
Game We also include Hide & Seek, which is an asymmetrical game where one more players hide, and another player (the seeker) attempts to find all other players. We create simple one-vs-one version where one player (the model) picks hiding spot every round, and the seeker (adversary) is allowed to choose one location to search. The seeker wins they pick the same spot, and otherwise loses. This is a mixed strategy game where the equilibrium strategy for a single hiding player is uniformly random. Unlike Rock Paper
Scissors the expected result in the equilibrium case is to win Ø-1 of the time, when there are n hiding spots. In our experiments, this means that an ideal player will win roughly 85% of the time against adversarial seeker, i.e. a net score (rate rate of =70 points.
Setup As in 53.2.1, specify the game simply via zero-shot prompting, informing the model of all hiding spots and asking for a selection in each round, while providing the history of the game far. We use the same models as $3.2.1, and follow the same procedure for the greedy adversary (the seeker), selecting whichever hiding spot the model is most likely to have chosen.
3.3 Results
Base models are consistently most robust against adversaries We present results in table 2. Overall, base models achieve the highest scores across games and model sizes, indicating the strongest performance against the adversary. In Rock Paper Scissors, the Tulu-SFT model

---

Published as a conference paper at COLM 2025
achieves the second-best score for both sizes and is within 7 net points in both cases. For Hide & Seek, the base model is at least 40 points above any baseline for both model sizes, with Llama-Instruct 2nd best at 8B parameters and Tulu-SFT second best at 70B parameters, meaning Tulu-SFT is 2nd in 3/4 settings.
Case Study: Patterns Determinism for Rock Paper Scissors To investigate why align- ment recipes seem to reduce performance in mixed strategy games, we carry out an in-depth analysis of Rock Paper Scissors. First, we define a measure for how deterministic a model is in given round of game, as:
determinismm maaXm p (mmoee
In words, this is the probability of the most probable next move for the model to play, The minimum is determinism = for Rock Paper Scissors when models are uniformly random and have the best expected outcome. The maximum is determinism 1 when models are totally deterministic and expected to lose 100% of the time. One interpretation of this score is the degree to which one move is dominating model probability, resulting in behavior more similar to full determinism.
We first investigate the overall distribution of determinism of models across all moves played (i.e. all rounds in all games) in figure 4, left. Base models, which perform best at this game, tend to have determinism near 0.5 and have a very low maximum compared to all other models. For instance, 70B parameter Llama Instruct becomes almost completely deterministic some cases.
We also find that the result (win, tie, or loss) of the round directly before the given move affects determinism differently in different models (figure right). In this experiment, we plot:
mean (determinism outcome; 1) m---mmmaneeee (determmnimii joutcome; loss)
In words, this is how the outcome of the previous round affects determinism, setting 0 to the case when models lose, to simplify visual comparison. In every case, aligned models become more confident after a tie or win than a loss. In contrast, base models are slightly less confident after tie than a loss, and very similar between a loss and a win. Overall, aligned models seem to follow a common human behavior, to become more confident following positive (or non-negative) outcome in a game, naturally pushing them to become more predictable.
4 Creative Poetry Generation
Finally, we test the complex challenge of being creatively unpredictable. Change and evolu- tion are fundamental aspects of art (Fienberg & Martindale, 1991), meaning that the most impactful art must be novel and original i.e. not predictable. Following this intuition, we test the ability of models be original in creative/ artistic writing, specifically for poetry (Katz et al., 2023).
In a small-scale, contest-style human evaluation, we find that base models generate the most original poems in every case, while instruct models generate the most pleasant poems. Providing some intuition for this difference, we find that pleasantness is more strongly correlated with annotated human preference, core aspect of many alignment techniques. Overall, our findings support the idea that aligned poetry is easy to read but less likely to be impactful artistically interesting (Kestenbaum, 2024).
4.1 Experiments
Setup We prompt models to carry out simple poetry exercise, generating fixed-length poems (4 lines) on basic topic. We query GPTT4 (OpenAl et al., 2023) for a list of everyday topics: Coffee, Smartphones, Traffic, Weather, Exercise, Grocery shopping, Sleep, Work, Internet, Television. We then give models a straightforward, zero-shot prompt asking for a 4-line poem on the given topic, and generating until we are able to parse 20 poems of the given length. We find that 70B parameter models are significantly more proficient at this task, and so focus our analysis these models (using the same models earlier sections).
7

---

Published as a conference paper at COLM 2025
Human Eval Contest Wins
Example Poems about Coffee
Original
Pleasant
Preference
with Preference:
.0.08
0.34
Adenosine wrapped caffeine Dance through my recaptora within.
à Original
Base
4
0
1
Stimulate; neurotransnitters Kicked ny aynaраеs; wake up
In breued arona delight takes flight, tvakening sleep with gentle bite. caffeine weave through norning 8 space, In sip, the day we lace.
Llama-Instruct
0
2
Rich aroma rills the air norning davns, with Warm and dark, a scothing friend Litting till the day's end.
Preference
Tulu-Full
0
2
2
care
Figure 5: Results of creative poetry evaluation for 70B parameter models, where base models produce the most original, but not always preferred or pleasant, poems. Left: Number of wins along each axis for human evaluation poetry contests ($4.1), across 4 poem prompts, poems per model, and 3 axes. We include average rank correlation (p) with the pleasantness axis across poems. Right: Examples from one contest, including the best-ranked poems by human preference and originality. Aligned poems seem to share style that differs significantly from the base model. Further examples are included Appendix A.3.2.
Human Evaluation Contest Art is typically judged by the best rather than average case (e.g. in poetry contests), so we construct an evaluation to extract winners along different -emaan-valluated axes: originality, pleasantness, and preference. Originality serves as our notion of unpredictabilityy /creetivity,, which is what our evaluation ultimately aims to test. Although annotator preference is often used as the measure of generation quality, there is no concrete evidence that this correlates well with broader artistic merit or impact, and our study finds that it correlates more with pleasantness, which may be odds with novelty and impact.
To avoid leading annotators, we evaluate each of these axes separately their own annota- tion tasks, comparing series of random pairs of poems. For given axis and set of poems, we determine a final winner by inducing an ordering using a variant of the Bradley-Terry model (Bradley & Terry, 1952) from the pairwise comparisons. We use this format to aggre- gate over natural disagreements in subjective questions, and so expect some disagreement between annotators. We also calculate annotator agreement with 70 additional comparisons, finding Cohen's 's Kappa (Cohen, 1960) of 0.33, 0.27, and 0.67 for originality, pleasantness, and overall preference (respectively).
We carry out human evaluation of the 2 most popular aligned models tested here (Tulu-Full and Llama-Instruct) along with the base model, all at 70B parameters. We evaluate the 3 axes for 4 different poem prompts (coffee, sleep, weather, smartphones), comparing 5 random poems from each model. This results in 12 contests of poems each. We carry out 60 comparisons per contest on the Prolific platform, resulting in a total of 720 annotated comparisons. We also include one control question in every job, comparing between generated poem and random bag of words. Annotators selected the generated poem in virtually every case.
Results
A split between preference and originality The results of our human evaluation of model poetry are included in figure (left). We include limited examples figure 5 (right) and extensive examples in Appendix A.3.2. There is distinct split in attributes: the base model produces the winning poem terms creative originality in all cases, but this does not translate to dominance in terms of human annotator preference. Indeed, when taking Spearman rank correlation (Spearman, 1904) averaged across contests, originality actually slightly negatively correlated with human preference (mean p acToss settings is -0.08). These results support our earlier findings that aligned models tend to be more predictable than base models, which results in lower level of perceived artistic originality in this case.
8

---

Published as a conference paper at COLM 2025
Pleasantness aligns with Preference We find that the pleasantness axis aligns more pos- itively with annotator preference in terms of rank (mean = 0.34) than originality does (mean p -0.08). Given that annotator preference is often core element of alignment, their desire for pleasantness may explain why aligned models seem to prioritize this over originality. This also suggests that crowdsourced preference, commonly used the ultimate test of quality in generations, may not give clear a strong signal towards artistic originality or impact.
On the other hand, annotators seem to recognize the originality of the base model although they do not prefer it. The base model never wins in terms of pleasantness, and its poems also have the lowest median rank in terms of human annotator preference (10.5 in sets of 15 poems). Yet, the base model does win one of four contests in terms of preference, suggesting that annotator opinions of base model poems are highly variable.
Related Work
Our work aims to develop an understanding of broad model limitations and biases, par- ticularly the effects of alignment techniques. Recent work has studied the relationship between base and aligned models, often focusing on the differences between them (Lin et al., 2024), and how to encourage aligned behavior (Hewitt et al., 2024; Fei et al., 2024). Like our work, Li et al. (2024) study the qualitative differences caused by alignment (particularly RLHF) and similarly find that alignment can narrow some capabilities not covered by task- based improvement. A growing body of work studies the loss of diversity that can result from alignment (Murthy et al., 2024; Kirk et al., 2023; Bronnec et al., 2024; Padmakumar & He, 2024), which is related to unpredictability studied here. Shypula et al. (2025) find the counterintuitive result that aligned models can have higher semantic diversity despite lower syntactic/ surface diversity, particularly in settings where aligned models can generate more high quality answers. McCoy et al. (2023) also study the biased effects of model training techniques, although focus on issues induced by pretraining rather than
Other works attempt to measure writing creativity related to LLMs. Lu et al. (2025) measure this as the degree to which generations match existing text snippets, and similarly find that alignment (RLHF in that case) greatly reduces this notion of creativity. Many other works focus on the human experience of creativity, using human judgements instead Chakrabarty et al. (2024); Anderson et al. (2024); Gómez-Rodriguer & Williams (2023). Our work follows the latter approach. While our work focuses on direct model creativity, this is also relevant for the setting in which models collaborate creatively with humans on writing Padmakumar & He (2024); Chakrabarty et al. (2024); Anderson et al. (2024)
Multiple past works have studied the ability of models to carry out random behavior, such as random number generation (Hopkins et al., 2023; Bigelow et al., 2024; Koevering & Kleinberg, 2024), demographic sampling (Meister et al., 2024), or playing games that require randomness (Silva, 2025). None of these works aim to study the effects of alignment on randomness,, although some observe an effect (Hopkins et al., 2023; Koevering & Kleinberg, 2024). Like our work, some study patterns in generated random sequences: Koevering & Kleinberg (2024) also find a tendency against repetition, while Bigelow et al. (2024) find models can transition from randomness to formal languages in different settings. Paruchuri et al. (2024) investigate the ability of models to reason about randomness rather than sample. More broadly, Song et al. (2025) advocate for evaluations that consider non-determinism more strongly.
One aspect of our work studying model biases in settings that have strong human biases (randomness, mixed strategy games). Past work has studied these for range of known human cognitive biases and opinions (Liu et al., 2025; Jones & Steinhardt, 2022; Itzhak et al., 2024; Santurkar et al., 2023).
Most works on random numbers look at binary (Bigelow et al., 2024; Koevering Kleinberg, 2024) or continuous (Hopkins et al., 2023) distributions, while our experiments in this space use integer sampling. Other works study games for LLMs (Silva, 2025; Brookins & Debacker, 2023; Akata et al., 2023; Jia et al., 2025) but do not focus on the divide between aligned and base models as our experiments do.

---

Published as a conference paper at COLM 2025
6 Conclusion
Overall, our work provides extensive support to the notion that popular alignment recipes erode a range of capabilities present in base models. Despite better performance on common benchmarks, aligned models are found to have lower performance across range of tasks tested here (figure 1).
Concretely, the alignment recipes studied here seem to reduce the ability of models to be unpredictable. This could have significant implications for the impacts of LLMs, given the dominance of aligned models. Practically, our findings could mean that aligned LLMs are not as effective at creative tasks, or assisting humans with creativity, given our results on poetry generation. Similarly, our results on games call into question how useful aligned models will be for settings requiring mixed strategies, like natural communication or bargaining on behalf of user. On the other hand, this may have positive implications for safety, as aligned models may be less effective at deception, which is thought to require ambiguity and non-determinism.
One remaining question resulting from our work is whether there is an inherent tradeoff between unpredictability and the capabilities at which these aligned models excel. Explor- ing this question could shed light on the underlying mechanisms of model capabilities. Regardless, our work suggests that although base LLMs receive much less attention than their aligned forms, there mysterious and valuable capabilities hidden within them.
Ethics Statement
Our work carries out analysis of existing language models, and does not train any new models or introduce any new datasets. In all human evaluations carried out here, we follow necessary IRB guidelines, and aim pay our workers $15 per hour on average.
One important point is that our work is advocating for the value of base language models, which could carry risks compared aligned models. We would like to clarify that we only advocate for the deployment of safe systems to the general public. Our work does not imply that large and untested base models should be made available at large, but rather that current alignment techniques may erode useful capabilities that were available in the original base parameters.
Acknowledgments
This work is supported in part by the Institute for Human-Centered AI at Stanford University. We thank Ari Holtzman, Jared Moore, and the Stanford NLP Group for useful input and feedback on this research.
References
Elif Akata, Lion Schulz, Julian Coda-Forno, Seong Joon Oh, Matthias Bethge, and Eric Schulz. Playing repeated games with large language models. ArXiv, abs/ Abs/2305.16867, 2023. URL https: //api. semanticscholar. www. wwrg/CCrrpuID::.. (0:00puuID:258471111 258947115.
Barrett R Anderson, Jash Hemant Shah, and Max Kreminski. Homogenization effects of large language models on human creative ideation. In Creativity and Cognition, CC '24, pp. 413-425. ACM, June 2024. doi: 1145/3635636. 30015564366666444.444 3656204.
URL http: //tx. www../g//1.......
Anthropic. The Claude model family: Opus, Sonnet, Haiku. 2024. URL https://api. semanticscholar.c wow.aaau..rr///or......... (00orrrus11::222229999999 268232499.
Eric J. Bigelow, Ekdeep Singh Lubana, Robert Dick, Hidenori Tanaka, and Tomer D. Ullman. In-context learning dynamics with random binary sequences. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OvenReview.net, 2024. URL https:://ppenneeeeeveeeffrroooos
10

---

Published as a conference paper at COLM 2025
Ralph Allan Bradley and Milton E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(344)3324345) 1952. ISSN 00063444, 14643510. URL httt::/www.jstott/ wwrsstoorrrrrooooeoeeooeeooooere www.oorg//tooeeeo stable/2334029.
Florian Le Bronnec, Alexandre Verine, Benjamin Négrevergne, Yann Chevaleyre, and Alexandre Allauzen. Exploring precision and recall to assess the quality and diver- sity of LLMs. In Annual Meeting of the Association for Compulational Linguistics, 2024. URL https://api. semanticscholan orgg/oorpusD::267744488
Philip Brookins and Jason Debacker. Playing games with GPT: What can we learn about large language model from canonical strategic games? SSRN Electronic Journal, 2023.
URL https: semanticscholar oemaerrtccccoooooooooooooooooooo 259714444222914462 259714625.
Tuhin Chakrabarty, Vishakh Padmakumar, Faeze Brahman, and Smaranda Muresan. Creativ- ity support in the age of large language models: An empirical study involving emerging writers, 2024. URL httpss:/arxxi..rror/// wttp://wwww.gg////////////////// 12570.
Herman Chernoff and Erich Leo Lehmann. The use of maximum likelihood estimates in x² tests for goodness of fit. Annals of Mathematical Statistics, 25:579-586, 1954.
Jacob Cohen. A coefficient of agreement for nominal scales. Educational and Psychological Measurementt 20(1):37-46, 1960. doi: 00111770003164466000 org/10. 11770/0011111144460000000
URL https://doi.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, An- thony S. Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurélien Rodriguez, Austen Gregerson, Ava Spataru, Bap tiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Cantón Ferrer, Cyrus Nikolaidis, Damien Al- lonsius, Daniel Song, Danielle Pintz, Danny Livshits, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab A. AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriele Synnavve, Gabrielle Lee, Georgia Lewis Anderson, Graeme Nail, Grégoire Mialon, Guanglong Pang, Guillem Cucurell, Hailey Nguyen, Hannah Kore- vaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Tbarra, Tsabel M. Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lee, Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Ju-Qing Jia, Kalyan Vasuden Alwala, K. Upasani, Kate Plawiak, Keqian Li, Ken-591 neth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuen ley Chiu, Kunal Bhalla, Lauren Rantala- Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenk- ins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Mannat Singh, Manohar Paluri, Marcin Kardas, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melissa Hall Melanie Kambadur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladri S. Chatterji, Olivier Duchenne, Onur cCelebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasić, Peter Weng, Pra- jjwal Bhargava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohit Girdhar, Rohit Patel, Ro main Sauvestre, Ronnie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hosseini, Sahana Chennabasappa, Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Chandra Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mihaylov, Tong Xiao, Ujjwal Kam, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do, Vish Vogeti, Vladan Petrovic,
11

---

Published as a conference paper at COLM 2025
Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whit ney Meers, Xavier Martinet, Xiaodong Wang, Xiaoqing Ellen Tan, Xinfeng Xie, Xuchao Jia, Xuewei Wang, Yaelle Goldschlag, Yashesh Gaur, Yasmine Babaei, Yiqian Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpierre Coudert, Zhengxu Yan, Zhengxing Chen, Zoe Papakipos, Aa- ditya K. Singh, Aaron Grattafiori, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adi Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alex Vaughan, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Franco, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Ben Leonhardi, Po- Yao (Bernie) Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Bran- don Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Changhan Wang, Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hlsiang Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Damon Civin, Dana Beaty, Daniel Kreymer, Shang-Wen Li, Danny Wyatt, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Duc Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Firat Ozgenel, Francesco Caggioni, Francisco Guzm'an an, Frank J. Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpern, Govind Thattai, Grant Herman, Grigory G. Sizov, Guangyi Zhang, Guna Lakshminarayanan, Hamid Shojanazeri, Han Zou, Hannah Wang, Han Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegren, Hunter Gold- man, Igor Molybog, Igor Tufanov, Irina-Elena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Japhet Asher, Jean-Baptiste Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi JonCarvlll,, Yang, Joe Cummings,,oan Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Junjie Wang. Kaixing(Kai) Wu, U KamHou, Karan Saxena, Karthik Prasad, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kun Huang, Kunal Chawla, Kushal Lakhotia, Kyle Huang, Lailin Chen, Lakshya Garg, A Lavender, Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Maria Tsimpoukelli, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Keneally, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mohammad Raste- gari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navyata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikolay Pavlovich Laptev, Ning Dong, Ning Zhang, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollár, Polina Zvyagina, Prashant Ratanchandani, Pritish Yu- vraj, Qian Liang, Rachad Alao, Rachel Rodriguez, Rafi Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang, Rohan Maheswari, Russ Howes, Ruty Rinott, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy, Shaun Lindsay, Sheng Feng, Shenghao Lin, Shengxin Cindy Zha, Shiva Shankar, Shuqiang Zhang, Sinong Wang, Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehoe, Steve Satterfield, Sudarshan Govin- daprasad, Sumit Gupta, Sung- Bae Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Kohler, Thomas Robinson, Li, Tianjun Zhang, Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vlad Tonescu, Vlad Andrei Poenaru, Vlad T. Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Constable, Xia Tang, Xiaofang Wang, Xiaojian Wu, Xiaolan Wang, Xide Xia, Xilun Wu, Xinbo Gao, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu Wang, Yuchen Hao, Yundi Qian, Yuzi He, Zach Rait, Zachary DeVito, Zef Rosnbrick, Zhaoduo Wen, Zhenyu Yang, and Zhiwei Zhao. The Llama 3 herd of models. ArXiv, abs/2447:21788, 2024. URL
12

---

Published as a conference paper at COLM 2025
Yu Fei, Yasaman Razeghi, and Sameer Singh. Nudging: Inference-time alignment via model collaboration. ArXio, 2bs/2240009000, 2024. ÜRL https://api. semanticscholar org/
CorpuSID:273346831.
Stephen Fienberg and Colin Martindale. The clockwork muse: The predictability of
artistic change. Journal of the American Statistical Association, 88:375, 1991. URL https:
//api. / semanticscholar wwwpscholaar.oo/ooopuusssse wwwg//orpssSD:124...... 124433228..
Clémentine Fourrier, Nathan Habib, Alina Lozovskaya, Konrad Szafer, and Thomas Wolf.
Open LLM Leaderboard v2. https://hugging co/ open Lpen__lm__eaderboot 2024.
leaderboard/
Gemini-Team. Gemini: A family of highly capable multimodal models, 2024. URL https: //arxiv.org/abs/2311..1 11805.
Carlos Gómez-Rodríguer and Paul Williams. A confederacy of models: comprehen- sive evaluation of LLMs on creative writing. In Houda Bouamor, Juan Pino, and Ka- lika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 14504-14528, Singapore, December 2023. Association for Computational Linguis- fics. doi: 10.18653 / w1//222..iinmaaaeoo findinns--emnll..eee URL https: Findings-emnlp. 966/
0023.
John Hewitt, Nelson F. Liu, Percy Liang, and Christopher D. Manning. Instruction fol- lowing without instruction tuning. XrXiv, (bss/240.11454 2024. URL https://api.
semanticscholar.c hwwphhollrrrrr////// wwwpschholaar.o//oroouuieeeeee wwtgg/oorpSID::::0 2728222977
Aspen K Hopkins, Alex Renda, and Michael Carbin. Can LLMs generate random numbers? Evaluating LLM sampling in controlled domains. In ICML 2023 Workshop: Sampling and Optimization in Discrete Space, 2023. URL https: /ttp:///ppp/eeieeee/eeee net nett/forum21d=hhh1k99y3
Itay Itzhak, Gabriel Stanovsky, Nir Rosenfeld, and Yonatan Belinkov. Instructed to bias: Instruction-tuned language models exhibit emergent cognitive bias. Transactions of the Association for Computational Linguistics, 12:771-785, 2024. doi: 10.1162 /taac_cl_aa a a_00673. URL
https::/acl.attooooy..rrr///
Jingru Jia, Zehua Yuan, Junhao Pan, Paul E. McNamara, and Deming Chen. Large lan- guage model strategic reasoning evaluation through behavioral game theory. ArXiv, abs//502.20433. 2025. URL https://api semanticscholas wwtp:roorg/////eeeeee wttp/CoorpusD::27724887..
Erik Jones and Jacob Steinhardt. Capturing failures of large language models via human cognitive biases. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL https: net//forum2dd=Foo0gnn----
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeff Wu, and Dario Amodei. Scaling laws for neural language models. ArXio, abs/ abs/2001.0886. 2020. URL httpp:://ppl www.piissstticcoolllll... .rrg/ CorpusID:21088 21086 1095.
Brent Katz, Josh Morgenthau, and Simon Rich. I am code: An artificial intelligence speaks. Back Bay Books: Little, Brown and Company, 2023.
David Kestenbaum. That other guy. This American Life, 5 2024. URL https://www.
mhiisamrrcanlife..oom wtt.f/f.rrrg////aaaacrrrrer
Robert Kirk, Ishita Mediratta, Christoforos Nalmpantis, Jelena Luketina, Eric Hambro, Ed- ward Grefenstette, and Roberta Raileanu. Understanding the effects of rlhf on Ilm gener- alisation and diversity. ArXiv, abt/ 2310.06655,, 2023. URL https: //ppi. semanticscholar.
wtg/CorrpuID::23380992
13

---

Published as a conference paper at COLM 2025
Robert Kirk, Ishita Mediratta, Christoforos Nalmpantis, Jelena Luketina, Eric Hambro, Edward Grefenstette, and Roberta Raileanu. Understanding the effects of RLHF on LLM generalisation and diversity. In ICLR, 2024. URL PXD3FAVHJT.
Katherine Van Koevering and Jon Kleinberg. How random is random? Evaluating the randomness and humaness of LLMs' coin flips. ArXiv, abs/2406.00092, 2024. URL https://api. semanti /B///CrppuIID::72215555
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V. Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luca Soldaini, Noah A. Smith, Yizhong Wang, Pradeep Dasigi, and Hannaneh Hajishirzi. Tulu 3: Pushing frontiers in open language model post-training, 2025. URL https: http://awww...orabb....// grggabs/24111 15124.
Margaret Li, Weijia Shi, Artidoro Pagnoni, Peter West, and Ari Holtzman. Predicting vs. acting: A trade-off between world modeling & agent modeling. ArXiv, abs/ (los/244 02044,244 2024. URL https: /apii.//pppp/p sewanti wrgg/orppssiie 270878711.
Bill Yuchen Lin, Abhilasha Ravichander, Ximing Lu, Nouha Dziri, Melanie Sclar, Khyathi Chandu, Chandra Bhagavatula, and Yejin Choi. The unlocking spell on base LLMs: Rethinking alignment via in-context learning. In The Twelfth International Conference on Learning Representations, 2024. URL htttps://openrevieee net/ pert/formm?2 http:/formmmmmdxdx.mmmm dd=xxxdeewwwdaa
Ryan Liu, Jiayi Geng, Joshua Peterson, Ilia Sucholutsky, and Thomas L. Griffiths. Large language models assume people are more rational than we really are. In The Thirteenth International Conference Learning Representations, 2025. URL htttp::///penreeeee net//
forum?id=dkeETTgxxo
Ximing Lu, Melanie Sclar, Skyler Hallinan, Niloofar Mireshghallah, Jiacheng Liu, Seungju Han, Allyson Ettinger, Liwei Jiang, Khyathi Chandu, Nouha Dziri, and Yejin Choi. Ai as humanity's s salieri: Quantifying linguistic creativity of language models via systematic attribution of machine text against web text, 2025. URL https: https://aaxxi..org/bbss 04265.
R. Thomas McCoy, Shunyu Yao, Dan Friedman, Matthew Hardy, and Thomas L. Grif- fiths. Embers of autoregression: Understanding large language models through the problem they are trained solve. ArXiv, abs/2309.13638, 2023. URL https://api.
Nicole Meister, Carlos Guestrin, and Tatsunori Hashimoto. Benchmarking distributional alignment of large language models. ArXiv, abs/2211.05544 2024. URL h/tps://api.
semanticscholar www.oorg//oooo.. wwtp://orpuSD::::00 273:5555::2395544
Derek Muller. Why is this number everywhere? https://www.yautube..comm d6iQrh2TK98, 2024. [Online; accessed March-2025).
Sonia K. Murthy, Tomer Ullman, and Jennifer Hu. One fish, two fish, but not the whole sea: Alignment reduces language models' conceptual diversity. ArXiv, abs/2411.04427, 2024. URL https://api wwwanticscholeee.... wwaphhoolrrrrrg/////ee org/CorrpsSI::27000 273877407
OpenAl, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haim ing Bao, Mo Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapp
Christo- pher Berner, Lenny Bogdonoff, Oleg Boiko, Made laine Boyd, Anna- Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, An- drew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Benjamin Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar,
14

---

Published as a conference paper at COLM 2025
David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Sim'on Posada Fishman, Juston Forte, Is abella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Raphael Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Jo hannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Lukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Hendrik Kirchner, Jamie Ryan Kiros, Matthew Knight, Daniel Kokotajlo, Lukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Li, Rachel Lim, Molly Lin, Stephanie Lin, Ma teusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob Mc- Grew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamella Mishknn, Vinnie Monaco, Evan Morikawa, Daniel P. Mossing, Tong Mu, Mira Murati, Oleg Murk, David M' ely, Ashvin Nair, Reichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Ouyang Long, Cullen O'Keefe, Jakub W. Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alexandre Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Bel- bute Peres, Michael Petrov, Henrique Pondé de Oliveira Pinto, Michael Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Alec Radford, Jack W. Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario D. Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, lan Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe P'etroski Such, Natalie Summers, llya Sutskever, Jie Tang, Nikolas A. Tezak, Madeleine Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cer' on Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll L. Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Wein- mann, Akila Welihinda, Peter Welinder, Jiayi Weng. Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qim ing Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. GPT-4 technical report. OpenAT, 2023. URL https://ppp httpttpp:spappp...tttiicctoooooi www.ntecsccoollr.tt. hwtp:/oopooooo 2575328115
Alec
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Gray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instruc- tions with human feedback. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL https://openrevieer wwtet//fommndddmefee TNGGAACCCEEE.
Vishakh Padmakumar and He He. Does writing with language models reduce content diversity?, 2024. URL https://arxivv http://aaaxvv...g//bb/ss// org/abs/ htt.aa////....199000
Akshay Paruchuri, Jake Garrison, Shun Liao, John Hernandez, Jacob Sunshine, Tim Althoff, Xin Liu, and Daniel McDuff. What are the odds? Language models are capable of probabilistic reasoning. In Conference on Empirical Methods in Natural Language Processing, 2024. URL https: //api. semanticscholar... Brgg/CorpusID: htsp//oorpuSD::2762255555 270562235.
Qwen, ;, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu,
15

---

Published as a conference paper at COLM 2025
Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025. URL
24111
Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. Whose opinions do language models reflect? In Proceedings of the 40th International Conference on Machine Learning, ICML/23. JMLR.org, 2023.
Marc-Andre Schulz, Barbara Schmalbach, Peter Brugger, and Karsten Witt. Analysing humanly generated random number sequences: A pattem-bassed approach. PloS one, 7: e41531, 07 2012. doi: 10.1371 www.uraall.oo...000000
Alexander Shypula, Shuo Li, Botong Zhang, Vishakh Padmakumar, Kayo Yin, and Osbert Bastani. Evaluating the diversity and quality of Ilm generated content, 2025. URL
https://arxiv. www.://ogg/bb///aeee 12522..
Alonso Silva. Large language models playing mixed strategy nash equilibrium games. In Hélène Le Cadre, Yezekael Hayel, Bruno Tuffin, and Tijani Chahed (eds.), Networr Games, Artificial Intelligence, Control and Optimization, pp. 142-152, Cham, 2025. Springer Nature Switzerland. ISBN 978-3-031-78600-6.
William E. Simon. Number and color responses of some college students: Preliminary evidence for "blue seven phenomenon". Perceptual and Motor Skills, 33(2):373- 1971. doi: 10.2466 pwtpm..7.........
URL nttps://dot orgg 0. 2466/ pm10. 1971 33. 373.
Yifan Song, Guoyin Wang, Sujian Li, and Bill Yuchen Lin. The good, the bad, and the greedy: Evaluation of LLMs should not ignore non-determinism. In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Compulational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 4195-4206 Albuquerque, New Mexico, April 2025. Association for Computational Linguistics. ISBN 979-8-89176-18966 doi: 211/
211. URL
C. Spearmann The proof and measurement of association between two things. The American Journal of Psychology, 15(1):72-101, 1904. ISSN 00029556. URL http:://ww.jjtoo/ org/ stable/ 1412159.
John Trueman. Existence and robustness of the blue and seven phenomena. The Journal of General Psychology, 101(1)223-26 1979. doi: 10.1080 /0t0//022239.19999992000000 URL
https://doi http:///oo..org//// org/10. 10. 1080/00221389, 1979. 9928057.
J. von Neumann and O. Morgenstern. Theory of games and economic behavior. Princeton University Press, 1947.
W. A. Wagenaar. Generation of random sequences by human subjects: critical survey of literature. Psychological Bulletin, 77:65-72, 1972. URL https: //api. semanticscholar... org/ CorpusID:736593:
16

---

Published as a conference paper at COLM 2025
A
Appendix
A.1
Experimental Details
A.2 Overall
We use the suggested prompt formatting for each of the given models. These are:
Tulu


(instruction)
<lassistant|>
(optional infix}
where we may include an infix to aid in parsing, specifically observing intro text that Tulu includes typically includes before it returns an answer.
Llama-Instruct:
< <lbegin_of_texe / [ipader_id(l>systek//eee j You are an expert at following human instructions.
(instruction)
{ (optional infix]
_d > h
With optional infix as defined above. Finally,
Base:
<|begin_of_text (Iiistructionn
Note that the exact wording of the instructions may differ slightly for the base model, which often requires instructions to be framed more contextually than as a direct command.
LLM Implementation We use the SGLang library for all language model inference. We use full precision for 8B parameter models on single NVIDIA RTX A6000 GPU. For 70B parameter models, we use fp8 precision on NVIDIA RIX A6000 GPUs. All models used here are based the Llama 3.1 (Dubey et al., 2024) family of models.
A.2.1 Games Requiring Randomness
Adversarial Probability: We note here that the probabilities used to decide adversarial moves are approximate. We estimate the likelihood of each next model move by inves- tigating the next-token distribution given the prompt: propptt) For each token f that corresponds the beginning of surface form for one of the given moves, we add this probability to the adversarial estimate of that move, and renormalize these combined proba- bilities in the end. Note that the actual move played by the model decided by parsing model generations, which better indicates the underlying behavior of the model but does not allow for consistent probability estimation.
A.3 Results
A.3.1 Random Number Generation
Adjusting for Entropy To test whether the superior performance of base models at gener- ating single random numbers is simply an effect of their higher entropy distributions, we explicitly adjust for entropy in figure 7. In this experiments, we sample from each model at multiple temperatures (1.0, 1.5, 2.0, 2.5, 3.0). For each aligned model, we take the lowest temperature that gives next-token distribution (following the prompt, i.e. p(tpprompt) with an entropy at least as high as the base model. As the figure shows, the entropy in many cases is significantly higher than the base model, but this does not make these models
17

---

Published as a conference paper at COLM 2025
model
1
4
Tulu DPO
1774
2116
1461
1158
1377
Tulu SFT
983
1307
773
775
580
Tulu Full
1800
2241
1792
1248
1564
Llama instruct
9127
8839
7578
5800
7158
Base
107
120
145
164
128
Table 3: A comparison of multiple hand-written prompts for single random integer gen- eration (comparable to figure 2). Values here divergence, with paraphrased prompts numbered from to
as uniform as the base model. This indicates that the effect of higher entropy, rather than smoothing out the distribution, pushes more probability outside of the valid output space.
We further carry out grid experiment across sampling hyperparameters (topp, entropy) in tables understand the broader combined effects of these factors. Overall, we find that decoding parameters do not offer an easy way to make aligned models better at randomness. No tested settings allow aligned models reach the default setting of the base model (temp topp 1.0), which achieves divergence of 137. Note that these experiments are carried out on single random number generation (comparable to figure 2)
Alternative model In table 5 we include single random integer generation (comparable to figure 2) results for an alternative model architecture, the Qwen2.5 models (Qwen et al., 2025). We see patterns similar our earlier results, with the aligned form seeing significantly larger divergence and no significant improvement with scale.
Sampling of characters We carry out an experiment that asks models to generate a random letter from the first 11 letters in the alphabet, rather than numbers 0-10 in the single integer generation experiment (figure 2). These results are presented in table 6, and show similar patterns to the case of integer generation. One significant difference is that the most commonly generated letter is highly dependent on model, alignment recipe, and scale. Holding scale constant, Tulu models agreed ("j" for 8B and "f" for 72B). This suggests that the alignment recipe has significant impact how aligned model models collapse.
Sensitivity to prompting We carry out a prompt analysis, testing 5 prompts for random integer generation with differing wording (paraphrased by hand). These results are included in table 3. While there is quite bit of variation between performance with prompts, it is far from enough push aligned models towards matching the performance of base models, and all models seem to follow similar prompt-wise patterns of divergence.
Reward Model Analysis We study the Tulu reward model (used for aligning some Tulu models) for the random integer generation task in table 7. Particularly, we get the reward for each random integer between () and 10 given the prompt for the original random integer generation experiment. We find similar rewards returned for many of these values. The integer that all Tulu models pick most frequently ("7") achieves the highest reward, by very small amount. Yet, given the reward maximization inherent in many RL algorithms, an optimal policy model would maximize this reward by generating only 7, despite its very small advantage over other integers.
Sequential random number generation histograms We include histograms for sequential number generation (similar to figure 2) in figure 8. As discussed in the main paper, sequential generation results in more uniformity across models, but does not result in true randomness for the aligned models, which follow other heuristics (such not repeating integers).
Sequential random number generation probability by position In figure 6, we include 2D histograms of the probability of each integer being generated at each position in the 10-integer sequence. Note the relatively limited amount of structure in the Base and SFT histograms, while Llama Instruct, Tulu DPO, and Tulu Full seem to be highly conditional position.
18

---

Published as a conference paper at COLM 2025
temperature
topp
Base
Llama instruct
Tulu DPO
Tulu SFT
Tulu Full
2.5
3028
467
149
0.8
7
3184
601
174
646
0.5
18
3275
818
211
1177
2.0
1.0
16
4242
774
356
826
0.8
19
4600
1023
353
1346
0.5
36
7623
2301
820
2402
1.5
1.0
56
5333
965
496
1055
40
8187
1725
896
1968
0.5
88
15000
3092
1883
3283
1.0
1.0
137
8324
1306
937
1602
0.8
258
10087
2233
1442
2250
0.5
1286
15000
3618
2074
8097
Table 4: A broad sweep decoding hyperparameters: temperature and topp. Note that the experimental setup here is comparable to figure 2.
size
Base
Instruct
792
10970
72B
744
15000
Table 5: A repetition of the single integer generation experiment from figure 2, using the Qwen2.5 model rather than Llama.
A.3.2 Poem Examples
We include example poems generated by models in table 8 and table 9 for the topics of coffee and sleep.
19

---

Published as a conference paper at COLM 2025
size
Base
Instruct
Tulu DPO
Tulu SFT
Tulu Full
8B
78 ("a")
1158 ("e")
614 ("T)
370 (TT)
832 (T)
70B
193 ("k")
10513 ("P"
1177 ("F)
709 ('P"
1231 ("P")
Table 6: Repeating the random integer experiment from figure 2, but prompting models to generate letters rather than numbers. Values are divergence, while the most frequently generated letter for each model is included in parentheses.
Integer 0
2
7
8
9
10
Reward
3.98
4.37
4.99
5.15
5.19
5.03
5.26
5.29
5.28
5.01
4.23
Table 7: Reward values returned by the Tulu reward model, for various integers returned for random integer generation as in figure 2. This reward model is used in training some Tulu models. Note that "7" revieves the highest reward for this prompt, by a small margin.
Base
Llama instruct
Tulu DPO
Tulu SFT
Tulu Full
Integer
88
10
Intgger
70B
0
5
10
5
10
10 10
Figure Plotting the probability of generating each integer each position across models. Note that Base and SFT show relatively little structure (closer to uniform random) while all other aligned models show very high positional structure. Note that this aggregated across sequential generations, and does not necessarily capture all probabilistic structure in each model, only structure that is highly position dependent (e.g. Llama instruct)
x' 112
y' 237
y'
Pw
250
3.0
3.2
3.0
5.2
5.0
5.0
12.0
5.0
12.0
sample
sample
sample
sample
sammile
Figure 7: Histograms for single-value random integer generation, adjusting for entropy. We test temperatures of 1.0, 1.5, 2.0, 2.5, and 3.0, taking the lowest entropy for the aligned models where the resulting entropy (H in this plot) at least as high as the base model (for 8B parameter models). Despite having higher overall token-wise entropy in each case, aligned models still have higher divergence (x²) from uniform across the board.
True Sample
Base
Llama instruct
Tulu DPO
Tulu SFT
Tulu Full
300
200
14
115
201
x' -2
-19
88
100
x' 29
44
-3
-2
X
18
702
10
10
10
5
10
10
sample
somple
samble
sample
sample
Figure 8: Analogous plot figure 2 for integer sequences: histograms of sampled integer counts with divergence (x²2) from expected uniform counts across models and sizes.
20

---

Published as a conference paper at COLM 2025
Base
Llama-Instruct
Mighty caffeine rallying spirit hot cups of comfort midst bitter lives
Rich aroma fills the air A morning pick-me-up beyond compare Lifting fog from weary eyes Invigorating senses, a warm surprise
Adenosine wrapped in caffeine, Dance through my receptors within. Stimulate; neurotransmitters go insane. Kicked inside my synapses; wake up brain!
Steaming cup, aromatic delight Lifting fog, banishing the night Rich flavors dance, sensory thrill Coffee's warmth, morning's ' gentle will
*Winner for Originality
Fresh brew softly sitting, tempting each drinker to add drop of poison, bitter unrequited love.
Rich aroma fills the Lifting fog from morning's s care A cup of joe, daily friend Invigorating till the day's end
Enjoy the deep and dark flavors of this brew, Savor the richness that's found its hue. Aromatic and smooth, a cup to cherish, Coffee, our love, is our dear mistress.
Steaming cup, aromatic delight Lifting fog from morning's 's earliest light Rich flavors dance upon my tongue Coffee's warmth, my daily song
Rich aroma fills the air As morning dawns, with coffee's care Warm and dark, soothing friend Lifting fog, till the day' end. *Winner for Pleasantness
Morning's 's wake up call, Creamy, rich and bold, Awakens the senses, As each desire takes hold.
Tulu-Full
In mugs of steam, the morning starts anew, Sipping on dreams, bold and subdued. Beans' aroma wakes the mind's embrace, Coffee's spirit, in every sip, we trace.
In brewed aroma delight takes flight, Awakening sleep with gentle bite. Caffeine weave through morning's space, In every sip, the day we lace.
In the dark, a bean from distant lands, Brewed to wake from morning's s sweet demands.
The aroma swirls, a siren's 's call. Awake to the world, embrace it all. *Winner for Preference
Black elixir of the eager dawn, Awakening the sleepy with song. In mugs and cups, its warmth is drawn, And with each sip, the day is long.
Aromatic brew to start the day, Infusing warmth with every sip; Dark elixir, complex, bold, and gay, Waking spirits, quicken, energize, flip.
Table 8: Poems from various 70B models on the topic of coffee, along with best poems by axis as judged by humans.
21

---

Published as a conference paper at COLM 2025
Base
Llama-Instruct
Sleep is a temporary death.
Softly darkness wraps the night As the world succumbs to sleep's 's delight Dreams dance in the silent brain
A little rest,
I lift my eyelids
and behold the world's vastness.
Restful slumber soothes the strain
Sweet sleep, how soothing is thy receding power, When the world's iss cares assail me the hour Of loneliness and longing; and the place Lies dark and desolate before my face.
Softly falls the evening night As tired eyes succumb to sleep's 's sweet light
In tranquil rest, the body heals And weary souls their vigor reveals
*Wnnner for Originality
*Wnnner for Pleasantness
Restful repose amid quiet night, Dreams laid rest, souls taking flight, Bridging the gap twixt heaven and hell, Scarce a single thought to tell.
Softly darkness folds around my mind Sleep's sweet respite, where worries unwind In slumber's 's realm, find my peaceful nest And dreams transport me to world at rest
Teach me to be gentle, to love my spring days. Teach me to be kind and unhard, to sleep. To rest in flowery night-beds, to fruit, To live human, to die.
Softly darkness falls around, Sleep's s sweet solace is found. In slumber, worries cease to be, Restful dreams set the soul free
Softly darkness falls at night Sleep's 's sweet solace calms the fight
Tip-toe, feet, Quiet, it's bedtime, and peaceful dreams are waiting to be slept.
Weary eyes, now closed in rest Dreams unfold, and souls are blessed
*Winner for Preference
Tulu-Full
In sleep, we find a haven deep and still, A realm where night transcends to fulfill. Dreams' fleeting scenes, chorus to fulfill, Our slumber's voyage, nightly drill.
In land of dreams, Sleep gently taps, Whispering tales from moon's 's soft glow, With closed eyelids, the mind unwraps, Secrets beneath the shroud of woe.
In the quiet cradle of night, dreams unfold, A realm where shadows and stars closely knold. In slumber's 's gentle hands, the world lies still, While souls wander freely, both far and until.
In realms of soft repose at night we lay, Eyes heavy, shut by dreams' gentle sway. Time forgets to race; the restless mind slows, In sleep's 'ss  embrace, worries and cares decompose.
Silent watcher of our deepest dreams, Where worries fold and weighted lids descend, A fleeting realm of whispers and moonbeams, Sleep, restorer of our tired life's 'ss trend.
Table 9: Poems from various 70B models on the topic of sleep, along with best poems by axis as judged by humans.
22


Reviews:
[{"review_id": "DeXyhX2Qtl", "reviewer": "Reviewer_bHor", "summary": "The authors contend (and show with some evidence) that fine-tuning through RLHF reduces the randomness and creativity of generation, pushing it more towards a singular/less uniform outcome distribution. The authors test this on three tasks: (i) generating a single random number and generating a sequence of 10 random numbers, (ii) mixed strategy games, (iii) poem generation. In tasks (i) and (ii) the authors show that the fine-tuned models for the most part exhibit a less uniform distribution over outcomes/game choices than the base model, even when the optimal strategy would be unpredictability. For part (iii), the authors show that base model poems are often more original vs fine-tuned models through human evaluation.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper is clear and easy to understand. Further, I believe that the poem generation experiments are worthwhile and important for general knowledge. This problem also seems to be important (as the authors motivate) as people interact more frequently with aligned models. Further, the authors make a good attempt to provide details for reproducibility of their experiments.", "reasons_to_reject": "There are a few concerns I have with the experiment fundamentals of experiments (i) and (ii):\n1. Since experiments are done with open models, I would think that looking the logits to observe a probability distribution would be more meaningful than running a number of samples.\n2. There isn't any coverage of sampling parameters, which also could play a role in the generation tasks\n3. I am also concerned that using a single class of model may not fully capture this phenomenon because of different tokenization methods (maybe ablation over these differences would be helpful)", "questions_to_authors": "Most of my reasons to reject are also things that I believe, if changed, could significantly improve the quality of the paper. In addition,\n\n1. What about if you don't use numbers, but some other set of tokens, and ask it to randomly generate one of a set of tokens? Does this show the same effect?\n2. Is the entire token distribution more peaked (perhaps suggested by (iii))? One interesting thing would be to look at the perplexity of base model generations under the fine-tuned models, and vice versa.\n3. It's know that you can estimate a biased coin with a von-neumann extractor. Is there a similar analog to this when using aligned language models that could be analogous?\n4. Given this discovery, what takeaways should the reader have? (It seems a bit unclear how the reader should use these results)", "extra_scores": {}}, {"review_id": "gAroW0OMbt", "reviewer": "Reviewer_D7L6", "summary": "This work studies the potential negative effects of alignment post-training techniques in domains which require unpredictability and creativity. The authors argue that although alignment methods such as RLHF improve safety and instruction-following capabilities, they can significantly degrade capabilities like randomness and creative expression. They verify this on a random number generation (single number and sequence generation), mixed strategy games (rock paper scissors and hide-and-seek), and creative poetry writing. For evaluating the creative writing, humans graded according to originality, pleasantness, and preference; the authors note a positive correlation between pleasantness and human preference, but a negative correlation between originality and preference. Base models consistently won in originality and aligned models were preferred for being more pleasant. Overall, the authors claim that there is a systematic trade-off between benchmark performance (improved via alignment) and the ability to behave unpredictably or creatively.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The paper addresses an important consequence of alignment—its tendency to reduce unpredictability and creativity in language models.\n* The experimental design is compelling, with a clean set of synthetic tasks to exemplify loss of models' ability to produce random outputs that has been anecdotally observed or computed using various diversity metrics (eg. n-gram based)\n* Results are consistently in favor of base models, with use of reasonable quantitative metrics and human evaluations to back the claims\n* The results as alignment methods are applied to larger models is particularly interesting, since it has been noted that truly consistent instruction-following occurs around the 70B-scale.", "reasons_to_reject": "(See questions for further elaboration)\n* The novelty is perhaps limited, given multiple existing works which cite the loss of diversity at the cost of optimizing for safe outputs [1]\n* In particular, while this work identifies a loss in aligned model's ability to produce unpredictable outputs/generate original creative writing, it stops short of proposing solutions or mitigations. As the authors have observed, adjusting temperature or accounting for entropy doesn’t help because you want models to `act random' within a certain set of restrictions. More demanding creative writing formats like storytelling would test a model’s ability to maintain coherence alongside originality, which I suspect base models to perform worse than aligned models.\n* The study is restricted to a single family (LLaMA-3.1-based) of models.\n* The work would benefit from having a more extensive literature review on works which study how alignment methods amplify human cognitive biases (to name a few, [2,3,4])\n\n[1] Kirk, Robert, et al. \"Understanding the effects of rlhf on llm generalisation and diversity.\" arXiv preprint arXiv:2310.06452 (2023).\n\n[2] Murthy, Sonia K., Tomer Ullman, and Jennifer Hu. \"One fish, two fish, but not the whole sea: Alignment reduces language models' conceptual diversity.\" arXiv preprint arXiv:2411.04427 (2024).\n\n[3] Itzhak, Itay, et al. \"Instructed to bias: Instruction-tuned language models exhibit emergent cognitive bias.\" Transactions of the Association for Computational Linguistics 12 (2024): 771-785.\n\n[4] Santurkar, Shibani, et al. \"Whose opinions do language models reflect?.\" International Conference on Machine Learning. PMLR, 2023.", "questions_to_authors": "* Have you tried evaluating other base models? I mention Qwen (and I know this is a low-hanging fruit to ask as a reviewer) but my reason for this is that Qwen 'base' models are markedly distinct from Llama in that it underwent synthetic-data-heavy pretraining. I'm curious if the 'base' models there will already exhibit the same issues as the instruction-tuned models here. \n* Given the simplicity of your tasks, I believe it would also be meaningful to include comparisons to models where the objective has specifically included a term to encourage diversity [1], or at least highlight the potential i.e. to serve as a litmus test for assessing unpredictability and originality.\n* Does manipulating the prompt to ask explicitly-- either for originality or for letting the model know optimal play in light of the adversary-- help aligned models perform better on these tasks?\n*  Do you believe the observed drop in randomness can be attributed to phenomena that can be observed during training or in the model internals (eg. rank collapse)?\n\n[1] Lanchantin, Jack, et al. \"Diverse Preference Optimization.\" arXiv preprint arXiv:2501.18101 (2025).", "extra_scores": {}}, {"review_id": "QLnsf2Mq1k", "reviewer": "Reviewer_beTY", "summary": "This paper studies the effect of alignment on the predicitibility of outputs for random number generation and games such as rock-paper-scissors and hide-and-seek. The authors find that base models have much more diverse generations, when compared to the aligned models which tend to concentrate around certain values, leading to almost deterministic and predictible outputs. The authors confirm their findings empirically by prompting the model the generate a large amount of numbers and checking how close the distribution is to a uniform, or by trying to adversarially come up with strategies exploiting the deterministic nature of the aligned models. \nFinally, the authors also show that in tasks regarding generating poetry, base models are generally the most original ones.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper studies an interesting question regarding the impact of aligning models on the generations produced by the models. The authors study both quantitatively this effect (by measuring how far the distribution of \"random\" numbers generated by a model is from an actual random uniform sample, as well as in games such as rock-paper-scissors and hide-and-seek by seeing how much the determinism can be exploited in post-trained models), as well as qualitatively (in tasks such as creative generation).\n\nThe authors do an extensive ablation over the alignment procedures and scales, studying the base Llama model (across 8B and 70B scale), aligned with the Llama-Instruct, SFT, DPO and both. The results interestingly show that scale does not alleviate the issue of diversity collapse in models, but that the different training protocols can lead to a difference in the generation diversity.\n\nThe authors also investigate the task of creative writing, showing that, consistently with the previous tasks, the base models have more originality in their generations than the aligned models.", "reasons_to_reject": "The authors only report their findings on a single subset of models, namely Llama models. While I understand the constraints due to compute reasons, many works have shown that there is very high variance between the performance and behaviour of different model families such as Qwen, Tulu etc. \n\nWhilte the findings reported by the authors are mostly consistent across the games they test on, these games are very simple and have few states and actions. It would have been interesting to see if a similar deterministic pattern is maintained for games with longer time horizons, such as chess, Go.", "questions_to_authors": "Is there any way to quantify the impact of the pretraining data on the pattern of the generations? I believe that there must be a significant difference between models pretrained on different datasets.\n\nIn Table 2, it seems that Tulu-SFT is almost consistent second best for the net outcome. Do you have any speculation why this might be the case, as opposed to any of the other methods?", "extra_scores": {}}, {"review_id": "lbIyWPjbDH", "reviewer": "Reviewer_ze1Q", "summary": "The paper discusses the issue of model alignment and provides a family of tasks on which base (unaligned) large language models (LLMs) perform better than aligned models. The tasks include problems that require unpredictable outputs or decisions, or creativity  (random number generation, mixed-strategy games, and poetry generation). On each of these tasks, they present a series of evaluations comparing base LLMs vs. aligned LLMs (they use Llama-3.1 as the base model and its finetuned variants. They show that on each task, the base model outperforms the aligned variant. For the poetry generation task, they conduct a human study showing that a base model always performs better with respect to creativity, but it does not translate to dominance in terms of human preference.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This is an interesting paper that provides evidence that popular alignment recipes erode a range of capabilities present in base models, specifically, they reduce the ability of the models to be unpredictable.", "reasons_to_reject": "NA", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "t2RU1eWs96", "reviewer": "Reviewer_u55E", "summary": "This paper investigates how alignment procedures (e.g., Supervised Fine-Tuning, DPO, RLHF) affect the ability of large language models to exhibit randomness and creativity. Using a set of designed experiments across three domainsL random number generation, mixed-strategy games (Rock-Paper-Scissors, Hide-and-Seek), and creative poetry generation. They find that base (unaligned) models consistently outperform aligned models on tasks requiring unpredictability or novelty. The aligned models display deterministic tendencies (e.g., overusing the number \"7\"), reduced strategic diversity, and a preference for pleasant but less original poetry. These findings shows a previously underexplored trade-off that alignment techniques that improve benchmark scores and user satisfaction may degrade other important capabilities such as stochastic reasoning and creative expression.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This paper addresses a critical yet underexplored topic that the unintended consequences of alignment procedures (e.g., RLHF, SFT, DPO) on model unpredictability and creativity. As most public-facing LLMs are aligned, understanding the trade-offs involved is of central importance to both the research community and industry deployment.\n- The authors conduct a methodologically rigorous evaluation across three domains including Random number generation, Mixed-strategy games (Rock-Paper-Scissors, Hide-and-Seek) and Creative writing (poetry). \n- The paper shows a consistent trade-off that as models become more aligned and perform better on standard NLP benchmarks, they exhibit reduced randomness and originality in domains where these qualities are crucial.", "reasons_to_reject": "- The paper repeatedly refers to \"randomness\" and \"unpredictability\" but does not clearly define what constitutes sufficient or task-appropriate randomness. For example, the authors evaluate model outputs using $χ^2$ divergence and repetition rates, but they do not evaluate whether the model’s behavior is strategically optimal (e.g., ε-Nash in games) or just different from uniform randomness.\n- The three tasks such as uniform integer generation, Rock-Paper-Scissors, and 4-line poem generation, are limited proxies for real-world unpredictability and creativity. While conceptually appealing, they suffer from oversimplification.\n- In Appendix A.3.1, the authors claim that increasing entropy through temperature tuning still leads to less randomness in aligned models, but the evidence is presented in a cursory way.  For example, Figure 7 in the appendix shows entropy-adjusted comparisons, but the models' sampling space and decoding parameters (e.g., top-k, top-p) are not discussed in detail, nor is the interplay between model logits and sampling strategy interrogated.\n- The human evaluation for poetry relies on just 720 pairwise annotations (60 per contest over 12 contests) and uses Prolific crowdworkers. There is no validation of inter-annotator agreement, task clarity, or demographic diversity.\n- While the paper compute useful divergence metrics (e.g., χ², MSE, entropy), the paper does not conduct a meaningful error analysis. There are no examples of failure cases, no category-level breakdown of errors, and no investigation into why specific alignment techniques cause predictable behavior. For example, The models are shown to overproduce the number “7,” but no introspection is done into whether this arises from instruction phrasing, training data artifacts, or logit suppression. Maybe would be better to Include qualitative examples of   failures case (e.g., repetitive poems, exploitable strategies in games) and conduct a diagnostic analysis of what parts of the alignment process cause these.", "questions_to_authors": "Could decoding strategies (e.g., temperature, top-p) recover lost randomness in aligned models? Paper mention using temperature 1.0 with full top-k sampling, but don’t show how aligned models perform under varied decoding settings. Have you tried tuning sampling parameters to see if aligned models can approximate base model behavior in these tasks? Would increasing entropy during decoding help mitigate deterministic tendencies introduced by alignment?\n\nThe paper focuses on short-form tasks. Would the trade-off between alignment and unpredictability also appear in Storytelling tasks requiring long-range narrative structure? or Agents in multi-agent environments requiring adaptive strategies?\n\nCan you share some insights or intuition that why do you think alignment procedures introduce deterministic behavior?  Do you have logit-level analyses showing whether alignment procedures sharpen output distributions? Could alignment be reinforcing “safe” or majority responses (e.g., “7”) due to reward shaping or over-regularization?", "extra_scores": {}}, {"review_id": "NrUDJKYpPk", "reviewer": "Reviewer_wJrW", "summary": "The main claim of this paper is that alignment of LLMs through techniques such as reinforcement learning from human feedback has a negative impact on their ability to perform taks that require unpredictable outputs. This claim is supported by a number of experiments comparing aligned versions of models from the Llama-3.1 to their unaligned base models on tasks such as random number generation, mixed-strategy games like rock-paper-scissors, and poetry generation. The experiments show that the unaligned base models consistently outperform their aligned counterparts with respect to randomness and/or originality. The experimental evidence is convincing and further supported by additional analyses in several cases. The main limitation of the paper is that the experiments are restricted to two base models from the same model family, which makes it hard to assess the generalizability of the results.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Interesting findings that expand our knowledge of LLMs and associated training techniques.\n- Solid experiments and good analysis.\n- Very clearly written.", "reasons_to_reject": "- The main weakness of the paper is the restriction of experiments to a single model family (in two sizes), but I would not really say that this is a reason to reject.", "questions_to_authors": "Did you consider running experiments besides Llama-3.1. If yes, what made you decide against it?\n\nIn an otherwise superbly clear paper, I find the following sentence a little convoluted: \"This suggest that the usefulness of alignment is not the only intuition that may breakdown in tasks requiring unpredictability.\" What exactly do you want to say here? Should \"breakdown\" possibly be \"break down\"?\n\nTypos and such:\n- Section 4.1: Extra space after \"Work\" and \"Internet\" in the first paragraph. \n- Section 5: I think \"[good term?]\" should be removed. \n- Ethics Statement: Missing negation in \"does train any new models\" (I assume).", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper examines issues related to the alignment of LLMs. In particular, RLHF has emerged as a popular technique for aligning LLMs; however, RLHF and similar techniques have their limitations. Specifically, when a task requires a degree of unpredictability, aligned models tend to exhibit narrower behavior and perform worse than their baseline model counterparts.

**Reasons to Accept:**
- The paper is clear, easy to understand, and tackles an important problem in alignment.
- The experimental setting is compelling and thoroughly executed. Specifically, the authors conduct an extensive ablation study of the alignment procedure and its scaling effects.

**Reasons to Reject:**
- The paper identifies an issue with loss-aligned models but does not propose a solution to mitigate it.
- It does not thoroughly explain what is meant by randomness and unpredictability.
- The human evaluation of poems relies on 720 pairwise annotations, but there is a lack of inter-annotator agreement.
Decision: Accept