Roll the dice & look before you leap: Going beyond the creative limits of next-token prediction
Authors: Vaishnavh Nagarajan
Chen Henry Wu
Charles Ding
Aditi Raghunathan
Venue: ICML 2025 oral (2025)
Keywords: next-token prediction
multi-token prediction
creativity
Primary area: deep_learning->large_language_models
TL;DR: We design a series of open-ended algorithmic tasks inspired by creative tasks and show that multi-token prediction and seed-conditioning lead to much more creative planning than next-token prediction.

Abstract: We design a suite of minimal algorithmic tasks that are a loose abstraction of _open-ended_ real-world tasks. This allows us to cleanly and controllably quantify the creative limits of the present-day language model. 
Much like real-world tasks that require a creative, far-sighted leap of thought, our tasks require an implicit, open-ended _stochastic_ planning step that either (a) discovers new connections in an abstract knowledge graph (like in wordplay, drawing analogies, or research) or (b) constructs new patterns  (like in designing math problems or new proteins). In these tasks, we empirically and conceptually argue how next-token learning is myopic;  multi-token approaches, namely teacherless training and diffusion models, comparatively excel in producing diverse and original output. 
 Secondly, to elicit randomness without hurting coherence, we find that injecting noise at the input layer (dubbed _seed-conditioning_) works surprisingly as well as (and in some conditions, better than) temperature sampling from the output layer. Thus, our work offers a principled, minimal test-bed for analyzing open-ended creative skills, and offers new arguments for going beyond next-token learning and temperature sampling. We make part of the code available under https://github.com/chenwu98/algorithmic-creativity

O
leap of thought Roll the dice & look before you leap:
Going beyond the creative limits of next-token prediction
Vaishnavh Nagarajan" Chen Henry Wu 2
Charles Ding?
Aditi Raghunathan'
Abstract
surprising and fresh connections never seen before. For instance, consider responding to highly under-specified prompts like "Generate a challenging high-school word problem involving the Pythagoras Theo- rem." or "Suggest some candidate therapeutic antibodies targeting the HER2 antigen." or "Provide a vivid analogy to differentiate quantum and classical mechanics." Creativity in such tasks requires generating responses that are not just correct or coherent, but are also diverse across responses and are original compared to the training data. These currently-sidelinet desiderata will rise prominence explore LLMs for open-ended scientific discovery (Gruver et al., 2023; Romera-Paredes et al., 2024; Hayes et al., 2025), for generating novel training data ( (Yu et al., 2024; Yang et al., 2024c; Wang et al., 2023), and as we scale up test-time compute approaches that benefit from diversity in exploration, such as best-of-N (Cobbe al., 2021; Chow et al., 2024; Dang al., 2025) and long chain-of-thouu reasoning (OpenAI, 2024; DeepSeek-AI, 2025; Snell et al., 2024; Wu et al., 2024).
We design suite of minimal algorithmic tasks that are loose abstraction of open-ended real- world tasks. This allows to cleanly and control- lably quantify the creative limits of the present- day language model. Much like real-word real-world tasks that require a creative, far-sighted leap of thought, our tasks require an implicit, open-ended stochas- tic planning step that either (a) discovers new con- nections in an abstract knowledge graph (like in wordplay, drawing analogies, research) or (b) constructs new patterns (like in designing math problems or new proteins). In these tasks, we empirically and conceptually argue how next- token learning is myopic; multi-token approaches, namely teacherless training and diffusion mod- els, comparatively excel in producing diverse and original output. Secondly, to elicit random- ness without hurting coherence, we find that in- jecting noise at the input layer (dubbed seed- conditioning) works surprisingly as well as (and in some conditions, better than) temperature sam- pling from the output layer. Thus, our work offers principled, minimal test-bed for analyzing open- ended creative skills, and offers new arguments for going beyond next-token learning and temper- ature sampling. We make part of the code avail- able under
Unlike simple open-ended tasks like generating names and basic sentences (Zhang et al., 2024b; Hopkins et al., 2023; Bigelow et al., 2024), many creative tasks (like designing clever Olympiad problem) are said to involve random flash of creative insight termed variously
(Wang et al., 2024a; Talmor et al., 2020; Zhong et al., 2024), a "cureka" moment (Bubeck et al., 2023), a mental leap (Holyoak & Thagard, 1995; Callaway, 2013; Hofstadter, 1995) or an incubation step (Varshney et al., 2019). The thesis of this paper is that learning to solve such creative leap-of-thought tasks (defined shortly) is misaligned with the current language modeling paradigm (a) in terms of next-token learning, and (b) in how randomness is elicited. We articulate these two concerns by designing a suite of algorithmic tasks inspired by such creative tasks. We then demonstrate how the creativity of language models suffers in these tasks, and how this can be alleviated (to an extent, within our tasks).
com/ cchnnu99/ algori cetorittmm-cccaaiviii
1. Introduction
Not all forms of intelligence are solely about being correct or wrong. In open-ended tasks, what also matters is finding creative ways to satisfy a request, making
'Equal contribution 'Google Research, US *Carnegie Mel- lon University, Pittsburgh. US. Correspondence Vaish- navh Nagarajan <vaishnavh@@ waiisnavh@@ooolll.comm google.com>, Chen Henry Wu cchenwu2@cs.cmm..dd
Concretely, for the scope of this paper, creative leap-of- thought task refers to tasks that involve a search-and-ple process; crucially, this process orchestrates multiple random
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Going beyond the creative limits of next-token prediction
decisions in advance before generating the output. Typically, such leap of thought is highly implicit in the text to infer it, one has to deeply engage with the text and detect higher-order patterns. We can think of tasks like designing satisfying math problems, generating worthwhile research ideas, or drawing surprising analogies as examples of such tasks.
Generate: "g, f, x"
Cenerate: "a, b, c"
8.t.
s.t.
(in wen-wa graph)
(in-weights graph)
(a) Sibling Discovery
(b) Triangle Discovery
Ideally, one would directly study these real-world tasks to quantify the limits of language models. Indeed, a flurry of recent works report that LLM-generated research ideas tend to be rephrased from existing ideas (Gupta & Pruthi, 2025; Beel et al., 2025) and that LLM outputs tend to be less creative than humans e.g., Chakrabarty et al. (2024); Lu et al. (2024b) (See $K). While assessing real-world tasks a lofty goal, the evaluations are subjective (Wang et al., 2024;; Runco & Jaeger, 2012), and when the model has been exposed to all of the internet, originality is hard to ascertain. Thus, the conclusions will inevitably invite debate (such as Si et al. (2024) vs. Gupta& & Pruthi (2025) or Lu et al. (2024a) vs. Beel et al. (2025)).
Figure I. Minimal tasks inspired by combinational creativity: Skills like research, humor and analogies often require identify- ing novel multi-hop connections from known pair-wise relation- ships in a knowledge graph. For instance, creating the word- play "What kind of shoes do spies wear? Sneakers." requires searching over semantic graph, and carefully planning pair of words (shoes, spies) that lead to mutual neighbor (sneakers). Inspired by this, we define tasks where a symbolic graph is stored in the model weights; the model is exposed to example node sequences that form a specific multi-hop structure (like sibling or triangle) during training. The model must in- fer this structure from training, during inference, the model must implicitly scall-search-and--le to generate novel and diverse node sequences obeying the same structure in the in-weights graph. Pictured are two example tasks with a symbolic graph each, and corresponding example sequence obeying a sibling (g, f, Y) or triangle structure (a, b, c). More details in $2.3 and Fig 9.
In search of more definitive conclusions, we approach from a different angle: we study controllable tasks that are loose abstractions of real-world tasks and yet allow one to objec- tively quantify originality and diversity. This follows recent works that have studied diversity of models in (Khona et al., 2024) and challenging context-free grammars (CFGs) (Allen-Zhi Zhu & Li, 2023b). We broadly term these as open-ended algorithmic tasks. Our aim then is design minimal instances of these tasks, akin to basic arithmetic for reasoning, so to tease apart the bare minimum compu- tational skills required for creativity. Then, we can examine how the current modeling paradigm may be limited even with this basic skills, and what alternative paradigms may be desired.
Generate:
Generate: "c-a, bd, d e-b"
"a-b, c-d, d-e, b-c, e-a"
s.t.
a.t.
(a) Circle Construction
(b) Line Construction
Figure 2. Minimal tasks inspired by exploratory creativity: Skills like designing problem sets, novel proteins and plots re- quire devising patterns that can be resolved in novel ways through some general rules. Inspired by this, we design a task where during training, we expose the model to "adjacency lists" that implicitly resolve into specific structure (a circle a line graph) under some permutation. The model must infer this higher-order struc- ture: during inference, the model must generate adjacency lists resolving to the same structure, but under novel and diverse per- mutations. Pictured are example sequences and the corresponding implicit structure they would resolve to. See $2.4 and Fig10.
As our first main contribution, we draw inspiration from cognitive science literature (Boden, 2003) (see also Franceschelli & Musolesi (2023)) design algorithmic tasks isolating two distinct types of creative leaps of thought. The first class of tasks involves combinational creativity: drawing novel connections in knowledge, like in research, wordplay or drawing analogies (see Fig 1 for task descrip- tion). The second class of tasks involves exploratory creativ- ity: constructing fresh patterns subject to certain rules, like in designing problems and suspense (see Fig 2). In these tasks, we can precisely evaluate models for the fraction of generations that are coherent, unique and original (not present in training set). We term this metric "algorithmic creativity" to denote that it solely evaluates the computa- tional aspects of creativity.
rithmic creativity compared to some multi-token approach, namely, either teacherless training (Bachmann & Nagarajan, 2024; Monea et al., 2023; Tschannen et al., 2023) or diffu- sion (Hoogeboom al., 2021; Austin et al., 2021; Lou et al., 2023) (see Fig 3 and Fig 4). Our argument is that in all our tasks, inferring the latent leap of thought requires observ- ing global higher-order patterns rather than local next-token patterns in the sequence.
Within this framework, we articulate two creative limits of the current language modeling paradigm. First, we em- pirically find that next-token learning achieves lower algo-
Next, we turn to how we elicit randomness from a Trans- former. While the de facto approach is to elicit randomness
2

---

Going beyond the creative limits of next-token prediction
Training Objective:
Standard (Next-Token
Teacherless (Multi-Token,
prediction the optimal thing do is globally learn higher-order patterns implicit in the whole future sequence. This presents challenge to recent proposals that aim to fix next-token prediction via permutations (Pannatier et al., 2024; Thankaraj et al., 2025) or partial lookaheads (Bavar- ian et al., 2022; Fried et al., 2022; Kitouni et al., 2024; Nolte et al., 2024).
Creativity
8.4
1.0
0.4
0.75
8.2
0.5
0.2
0.50
0.25
0.0
Sib Ling Discovery
0.0
Triangle Discovery
0.0
Circle Construction
0.00
Line Construction
As a second direction of progress, we hope our work pro- vides foundation to think about open-ended tasks which are extremely hard to quantify in the wild. This may spur more algorithmic explorations on improving diversity (such as our approach of ceed-conditioning) and on curbing ver- batim memorization in language models.
Nemorization
0.75
0.75
8.75
8.3
0.50
0,50
8.58
0.25
0.25
0.25
0.1
Our contributions:
0.00
Sib Ling Discovery
0.00
Triangle Discavery
0.00
Circle Construction
0.0
Line Construction
1. We create minimal, controlled and easy-to-quantinn open- ended algorithmic tasks. These tasks isolate, and loosely capture two fundamental modes of creativity.
Figure 3. Multi-token teacherless finetuning improves algo- rithmic creativity (top; Eq 1) and reduces memorization (bot- tom; fraction of generations seen during training) on our four open-ended algorithmic tasks for Gemma v1 (2B) model.
2. We find that multi-token prediction through one of teacherless training or diffusion, results in significantly increased algorithmic creativity in our tasks compared to next-token prediction.
at the output- - temperature sampling - we contrast this
with injecting randomness into the model. Concretely, we study seed-conditioning, where we train and test with a ran- dom prefix string per example. Surprisingly, we find that seed-conditioning induces non-trivial algorithmic creativity even with deterministic decoding; it is in fact, comparable to temperature sampling (and in some cases, better). We intuit that maximizing diversity at the output-level results in cognitive overload: it requires simultancously processing many leaps of thoughts to compute a marginalized token distribution. Input-level noise injection could sidestep this by articulating one leap of thought per seed. We also view this exploration as controlled study solidifying prior in- dications that prompt variations induce diversity (Li et al., 2023; Lau et al., 2024; Naik et al., 2024; Li et al., 2022).
3. Our argument provides new support for multi-token pre- diction, going beyond B&N'24. We show gap in cre- ativity in an open-ended task (rather than correctness in deterministic one). in much simpler 2-token- lookahead tasks, and in tasks where no token permutation is friendly to next-token-learning.
4.
We find that input-randomizattie via ceed-conditioning achieves algorithmic creativity comparable with conven- tional output-randomizztion via temperature sampling.
2. Open-ended algorithmic tasks & two types of creativity
We are interested in designing simple algorithmic tasks that are loosely inspired by endeavors such as generating sci- entific ideas, wordplay, narration, or problem-set design, where one needs to generate strings that are both "inter- esting" and never seen before. In all these tasks, before generating the output, one requires a (creative) leap of thought, a process that (a) is implicit i.c., is not spelled out in token space (or is even inherently hard to spell out), (b) involves discrete random choices (c) and together, those choices must be coherent in that they are carefully planned to satisfy various non-trivial, discrete constraints. These constraints fundamentally define the task and make it inter- esting e.g., a word problem should be solvable by arithmetic rules, or a pun must deliver a surprising punchline. The goal in such open-ended tasks is not just coherence though, but also diversity and novelty - generations must be as varied as possible and must not be regurgitated training data.
Overall, we hope to advance the field in two directions. First, we provide new angle advocate for multi-token approaches, orthogonal the "path-star" example in Bach- mann & Nagarajan (2024) (or B&N'24 24 in short). Whereas, the path-star example portrays a gap in correctness of rea- soning, ours shows a gap in diversity of open-ended think- ing. We note though that B&N 24 is an impossibility result where next-token learning breaks down spectacularly (un- less there is exponential data or compute), while ours is a data-inefficiency result (where next-token learning occurs but is mediocre). Next, the gap we show appears even in 2- token-lookahead tasks as against the -an--ooken-lookaaaad path-star task. Third, and most conceptually important is the fact that, while the path-star task is amenable to next- token prediction upon reversing the tokens, we identify tasks where no re-ordering is friendly towards next-token
3

---

Going beyond the creative limits of next-token prediction
Before we design tasks that require the aforementioned leap of thought, we first clarify what tasks do not require it.
setup intend to capture the human values or the subjective aspects of these endeavors (Schmidhuber, 2009); nor do they capture the rich array of creative acts that Boden (2003) discusses within these categories. (See limitations in ŞA).
Open-ended tasks that do not require leap of thought. One simple open-ended task that may come to mind is gener- ating mniformly-random known entities, like celebrity names (Zhang et al., 2024b). However, there is no opportunity to create a novel string here. more interesting example may be generating grammatically coherent probabilistic CFG strings following subject verb object format e.g., the cat chased rat (Hopkins et al., 2023). While novel strings become possible now, no sophisticated leaps of thought are involved; each token can be generated on the fly, satisfying a local next-token constraint to be coher- ent. In light of this, we can rephrase our goal as designing open-ended, creative tasks where coherence requires satis- fying more interesting, "global" constraints. To build this systematically, we draw inspiration from literature in cog- nitive science (Boden, 2003). Boden (2003) argues that fundamentally, there are three forms of creativity in that order: combinational, exploratory and transformative. We elaborate on the first two (the last, we do not look at).
2.2. The basic setting and notations
In all our tasks, we assume the standard generative model setting: the model must learn an underlying distribution D through a training set S of m independent samples 8; ~ D. The distribution is over a space V of L-Iength strings composed of tokens from the vocabulary V. The tasks are open-ended in that there is no one correct answer at test-time. The goal is to produce random string from D, much like responding to the query Design high- school word problem. We hope that this setup loosely resembles pretraining for language or in applications like drug discovery or protein (Meier et al., 2021; Madani et al., 2023; Watson et al., 2023) and genome modeling (Rives et al., 2021; Nguyen et al., 2024), where the model is trained set of varied example sequences, and is later used to produce novel outputs during inference.
Coherence: Each task is defined by a boolcan coherence function coh V + (true, false) which is true only on the support i.e., (D) = {s E V coh(s)) The exact form of coh will be defined each algorithmic task but broadly, we are interested in scenarios where determining coherence requires a global understanding of the whole string. This is inspired by the fact that wordplay must have preplanned punchline connecting what comes before, or a word problem must be solvable. We can think of 'D to be a simple uniform distribution over all coherent strings.
2.1. The fundamental types of creativity (Boden, 2003)
Combinational creativity. Consider rudimentary word- play of the form "What musical genre do balloons enjoy? Pop music." or "What kind of shoes do spies wear? Sneakers." " There is global structure here: two unrelated entities (genre& & balloons) are re- lated eventually through punchline (pop); the punchline it- self a mutual neighbor on semantic graph. More broadly, Boden (2003) argues that many tasks, like the above, involve "making unfamiliar combinations of familiar ideas" or the "unexpected juxtaposition of [known] ideas". Other tasks include drawing analogies, or finding connections between disparate ideas in science. All these tasks involve a leap of thought that in effect searches and plans over a space of known facts and combines them.
A Algorithmic creativity: Upon witnessing a finite set of ex- amples, the model must learn generate only strings that are (a) coherent, (b) original (not memorized) and (c) di- verse (covers the whole support). An exact quantification of this is computationally expensive in our tasks. Instead, we approximate it by sampling a set T of many independent generations from the model and computing the fraction of T that is original, coherent and unique. Let the boolean mems(s) denote whether an example s is from the training set S and let the integer function uniq(X X) denote the num- ber of unique examples in a set X. (The exact definitions of these quantities vary by tasks, as we will see). Then, we define our (empirical) algorithmic creativity metric:
Exploratory creativity. Consider the act of developing a mystery or designing logical puzzles. These endeavors are not as seowledge-heavy. What they crucially require is constructing fresh patterns that satisfy some highly non- trivial global constraint e.g., being resolvable as per some rules (e.g., logic). Such endeavors fall into a second class of exploratory creativity in Boden (2003). This includes much grander forms of exploration e.g. exploring various forms of outputs within a stylistic constraint, or exploring various corollaries within theoretical paradigm in physics or chemistry. The leap of thought here requires searching over all possible sequences, constrained by set of rules.
CI (7) uniq({s E T mems(s) A coh(s)))
(1)
T
Admittedly, we are looking at a simple form of novelty that is in-distribution. This notion, while simple, finds its rele- vance in applications like modeling proteins and genomes. We will also see that this notion is non-trivial enough to demarcate the limits of language models.
In the upcoming sections, we will capture some core com- putations of rudimentary instances within the two creative skills above. By no means does this minimal algorithmic
4

---

Going beyond the creative limits of next-token prediction
2.3. Tasks inspired by combinational creativity
ous permutations of the same triangle are counted one (see details in $D, including the exact formatting of the string). Note that the leap of thought in this task is much harder to learn and execute as it requires co-ordinating three edges in parallel, from memory.
Combinational creativity requires recall-and-search through entities from memory, constrained to relate with each other in an interesting way. We abstract this through tasks that discover structures from an in-weights graph i.e., a graph stored in the model weights, not reveal in context.
This type of a higher-order planning task can be thought of an abstraction of more complex wordplay (like antanacla- sis, where a word must repeat in two different senses in a sentence, while still being coherently related to the rest of the sentence) or creating word games (like crosswords) or discovering contradictions or feedback loops in a body of knowledge, an essential research skill - see $C.5.
2.3.1. SIBLING DISCOVERY
This task involves an implicit, bipartite 9 with parent ver- tices V {A, B, ..... each neighboring set of children nbr(A) = {ai, 02 .... : }, nbr(B)
{d1, b.. and so on. We define coh(s) to be true on sibling-parent triplets of the form = 77, D) such that 7, E nbr(T). We then consider a uniform distribution 2 over all coherent strings (7, r) for a fixed graph g. The model witnesses i..d samples from D. During test-time, the model must maximize algorithmic creativity (Eq 1) by generating novel parent-sibling triplets based on its in- ln-weights knowledge of G. Note that the model is not provided the graph in-context as this would sidestep a core computational step in com- binational creativity: recalling facts from a large memory (see $C.4). The hope is that the model infers and stores the pairwise adjacencies of 9 in its weights (given sufficient data). Full dataset description is in $D and Fig 9.
2.4. Tasks inspired by exploratory creativity
We are also interested in the sanowledge-intensee cre- ativity involved in tasks like designing word problems that demand novel solutions. We capture this skill minimally through tasks that construct new structures. No knowledge graph is involved in these tasks.
2.4.1. 1. CIRCLE CONSTRUCTION
In this task, the model must generate adjacency lists that can be rearranged to recover circle graphs of N vertices. Let the generated list be == (Vi, 0ig), (Vig. Mi.).... We define coh(s) ()) = true iff there exists a resolving permuta- tion π such that ((s) = (U), V)2); (1),. "))):)))). uy)).... ("),,.......)))))))) v,)) for distinct j2, jn- i.e., each edge leads to the next, and eventually circles back to the first vertex. We define uniq and mem such that different examples with the same resolving π are counted as the same, even if they have dif- fering vertices. As always, the learner is then exposed to a finite set of uniformly sampled coherent strings. Note that the latent leap of thought here requires constructing a novel permutation π before generating the sequence.
We view this task an abstraction of the wordplay example. One can think of the parent I' as the "punchline" that delivers a connection between otherwise non-adjacemi vertices, in the same way sneaker surprisingly connects the otherwise non-adjacent words, spies and shoes.
What is a leap of thought? The natural order of genera- tion is to pick the parent vertex (i.e., punchline) first, and the siblings after (conditioned on the parent). Thus, if the task demanded producing (T, 7'),, the tokens we observe betray the way they are naturally generated. However, the wordplay example presents the punchline (the parent, T) last while implicitly must be planned ahead. We term this implicit (random) planning step a leap of thought. Parallel- ing this, we define our sibling discovery task to generate the triplets s = (7, T [T), where the parent appears last. We hypothesize that this (sibling-first construction is adversarial towards next-token learning, while reversed (parent-first) dataset is friendlier towards next-token learn- ing. (More on this in $2.6.)
Loosely, we think of the resolving permutation as how a conflict in story or a word problem or puzzle is solved; the vertices as characters or mathematical objects; the rules of rearranging an adjacency list as rules of logic, math or story- building. The creative goal in this task is to create novel dynamics in the conflict, or equivalently, novel dynamics in how the conflict is resolved. Thus if only the entities differ, but the plot dynamics remain unaltered, count them as duplicates. See details in $D.
2.3.2. TRIANGLE DISCOVERY
2.4.2. LINE CONSTRUCTION
Next, we design a task that requires more complex, higher- order planning: generating triangles from an appropriately- constructed knowledge graph g (V,E) (see graph con- struction in SD). Thus, in this task con((v,, U2, vz)) = true iff all three edges between (1i, U2 03) belong in g. Furthermore, we define uniq(-) and mem(-) such that vari-
A simple variant of the above task is one where the edge list is of a line graph. The resolving permutation क satisfies T(s) (Vj, "; (1z2. Ujs)... ..... (Ujn-1.">, for distinct j1 ... Jn. i... each edge leads to the next until a dead-end.
5

---

Going beyond the creative limits of next-token prediction
2.5. Leap of thought is obscured at the token-level
in the prefix; B&N' 24 term this a "Clever Hans cheat" since the model witnesses and exploits part of the ground-truth it must generate (the siblings). Such cheats, being simpler than even the true generative rule, are quickly picked up during learning due to the well-known simplicity bias of gradient-dessentt-riin
Rarely does human--witten creative output come annotated with an articulation of the background creative process. Nor does each protein or molecule come with an enumeration of the laws that generated it- - in fact, it is because those laws are not fully known do we delegate the task of inferring those to a machine. Likewise, in our setting, the training signal consists only of the observed tokens; the model must infer the underlying leap of thought.
neural networks (Shah et al., 2020), Subsequently, any gradient supervision from the parent I is lost (also known as gradient starvation (Pezeshki et al.. 2021)), leaving no guidance learn the latent plan, 2 r.
After the Clever Hans cheat is learned, we conjecture that the NTP- NTP-Iearner learns the second sibling not through the plan- conditioned distribution p(7'|= 2 == D) but through the next- coken-condiiional, p(7'l7). This is a complex distribution: learning this may require witnessing every sibling-sibling pair totalling O(m n²²) many training data - larger by a factor of compared to the natural rule.
In fact, in last three tasks, the model must infer deeper form of implicit structure than what exists in many typical algorithmic tasks, like addition (Lee et al., 2024), or the path- star task (B&N'24) or Sibling Discovery. Whereas in those tasks, the leap of thought is perceptible at the token- level, modulo some re-ordering, our last three tasks have no such token ordering. These task are permutation -nvtrriannn token is more privileged than the other, and all tokens must be "simultaneously learned" to infer the underlying process. We view this as an abstraction of tasks where the creative process is not immediate from the surface of the text. These tasks offer a test-bed even for non-next-token approaches that rely on re-permuting the tokens (Pannatier et al., 2024; Thankaraj et al., 2025) or predicting only a part of the future (Kitouni et al., 2024; Nolte et al., 2024; Bavarian et al., 2022; Fried al., 2022).
Summary of argument. Abstractly, in a creative leap-of- thought task, it is most efficient to learn well-planned random latent p(z) and a subsequent latent-conditioned distributiinn p( p(s[z). However, NTP factorizes this into pieces of the form p(s $<s, 2). Consequently, on the later tokens, the model may be lured by Clever Hans cheats and learn uninformative latents. Conversely, the model may learn the earlier through complex rules, bereft of latent plan. While this may not lead to complete breakdown of learning as in B&N 24 in our tasks, we hypothesize it must lead to data-hungry learning.
2.6. How next-token learning may suffer in our tasks
Much like in sophisticated creative tasks, in our tasks, the most natural way to generate the string is by planning var- ious random latent choices (say 2) in advance and by pro- ducing plan-conditioned distribution p(s[2) 2) over coherent strings s. However, next-token prediction (NTP) - or next- token learning to be precise - we argue, is myopic and may struggle to learn such a latent plan. Our argument extends that of B&N' 24 to our even simpler tasks.
3. Training and Inference
For our next-token-trained (NTP) models, we use the standard teacher-forcin objective used in super- vised finetuning. Given prompt p and ground truth sequence s, the model is trained to predict the i'th token s, given as input the prompt and all ground truth tokens up until that point, (p. s<i). We write the objective more explicitly in $B Eq 2. For the multi-token Transformer models, we use teacherless training (Monea et al., 2023; Bachmann & Nagarajan, 2024; Tschannen et al., 2023), where the model is trained to predict s; simultancously for all i, only given the prompt p (and some dummy tokens masking the s that was once given as input). Since the exact details of this is irrelevant to our discussion, we describe this in Eq 2. To train our models, we use hybrid of this objective and the next-token objective.
Transtormers.
Consider the sibling task where we must generate sibling- parent triplets (7:7', r)). The most natural generative rule is to plan the last token (the parent) first and decide the children last. Think of this as learning a latent plan x := r. Then, learning the plan-conditioned generator 77,,7' T|z) factorizes to learning the distribution of children conditioned on parent as p(>|= := T) and p(7'l= := T) (due con- ditional independence), and the trivial p(T := r). This requires only as many parent-sibling edges as there are in the graph, i.e., ((m-n) n) many points, if there are m parents, each with n children. This is optimal.
Diffusion models. We use the score entropy discrete dif- fusion model (SEDD (90M), Lou et al., 2023). Loosely, we can think of the objective here as a hybrid of the teacherless objective, where the input is fully masked or corrupted (and the model must unmask or uncorrupt all of it), and various easier objectives with partial maskings or corruptions of the input. During inference, the model starts with a fully
Things should proceed differently with NTP. We argue that a lea-leamer would fail to learn the plan x := r. The key in- tuition is that NTP-learner learns the parent I' witnessing the siblings (7,2)) as input. This is trivial to fit: the parent is simply the mutual neighbor of the two siblings revealed
6

---

Going beyond the creative limits of next-token prediction
masked or corrupted sequence of tokens, and iteratively corrects subsets of them.
Training Objective
Standard [Next-Token! Teacherless (Multi-Toker)
Diffusion -bssobb (Nulti-Token)
Di fiffuioonUniffon (Multi-Token)
Inference. We extract independent samples from the model. For Transformers, during inference, we perform standard autoregression in both the next- and multi-token trained settings. We do this either with greedy decoding or with nucleus sampling (Finlayson al., 2024).
Creativity
8.
0.5
0.05
Sibling Discovery
Triangle Discovery
circle Construction
Line Construction
3.1. seed-conditioning
While temperature sampling is the standard way to elicit diversity from a Transformer, we speculate that this can lead to cognitive overload: the model must process mul- tiple thoughts to compute diverse softmax distribution. Alternatively, we propose conditioning on random seed as input to the model, hoping that thought can be fleshed out with singular focus. Thus, during training, each point is associated with an arbitrary (meaningless) prefix (e.g., of random characters) called a seed (note that there is no prefix in our tasks besides the seed). During test-time, novel seeds are used extract test data. We also view seed- conditioning more neutrally as controlled way to simulate variations in prompt-wordings for a fixed task (e.g., de- sign a word problem vs construct a problem). We elaborate on these intuitions in SC.1.
Memorization
0.2
0.50
0.05
0.02
0.25
0.08
Sibting Discovery
0.80
Triang Le Discovery
Circle Construction
Line Construction
Figure 4. Multi-token diffusion training improves algorithmic creativity (top; Eq 1) on three of our four open-ended tasks on CPT-2 (86M) and similarly-sized diffusion model, SEDD (9ON). We report the best performance after tuning the sam- pling hyperparameters - temperature from {0, 0.5, 1, 2), with or without lop-K where = 50.
the smaller models, in Fig 4, we find that the diffusion model improves over next-token training of similar-sized Transformers on all tasks, except on Sibling Discovery where it is mildly worse. With teacherless training, the gains are absent; however we find that the creativity of the top-K samples is generally improved (Fig 24), indicating a less peaky distribution. Regardless, the overall trend echoes prior findings that for smaller Transformers, teacherless training hard to optimize and other multi-token objectives even hurt (Gloeckle et al., 2024) smaller models.
4. Experimental results
Key details. Part of our experiments performed for a Gemma v1 (2B) pre-trained model (Gemma Team et al., 2024), averaged over 4 runs. For diffusion, we use a 90M (non-embeddinng parameters Score Entropy Discrete Dif- fusion model (SEDD (90M); Lou et al., 2023). For a fair comparison against NTP, we use 86M -oneeddddd parameters GPT-2 (86M) model (Radford al., 2019). In all our experiments, we finetune the models until it is clear that algorithmic creativity (Eq. 1) has saturated. All values are reported from this checkpoint. Finally. since our best Transformer results were under seed-conditioning (for both next- and mult-token training), our main results are reported under that training setting; we provide various ablations without that as well. Please see ŞE for more experimental details, and $D for precise dataset details (e.g., how the graph is constructed, how sequences are formatted etc..).
Multi-token prediction reduces memorization for the larger model. Algorithmic creativity may suffer either due to either incoherence or mode collapse or memorization. For our larger model, is the last reason that dominates: across the board (in Fig 3), next-token prediction is signifi- cantly prone to memorizing the data, while the multi-token method is highly resistant. As foreshadowed in $2.6, we hypothesize that this is because NTP memorizes the earlier training tokens without global plan, having fit the later tokens via local coherence rules (because of Clever Hans cheats la B&N'244. In contrast, the smaller models have much lower absolute memorization values; the multi-token objective preserves mildly worsens this, in way that does not always hurt creativity proportionally. here, we find that amongst the top-K samples, teacherless training has reduced memorization for some tasks (see Fig 25).
4.1. Observations
Multi-token prediction improves algorithmic creativity significantly. In all our datasets, the algorithmic creativity of the Gemma v1 (2B) model increases significantly under multi-token prediction ( (Fg 3), with nearly a 5x factor for the discovery datasets. Note that for this, we have selected the learning rate favorable towards next-token prediction; tuning for multi-token yields further gains (Fig 15). For
We point the reader to $C.6 for further empirical evidence supporting our argument about NTP from $2.6, including

---

Going beyond the creative limits of next-token prediction
Training
Standard
Teacherlese
Temperature for sampling (trained with NTP)
Objective:
(Next-Token)
(Multi- Token)
greedy
temp8.5
temp1.0 8
temp2.
Siblingbbsyy
TriangleDiscovery
Sibling Discovery
Triangle Discovery
Crestivity
Creatvstt
1.00
0.100
Creativity
0.75
Creativity
0.075
0.050
seed10
nu
seed2
greedy
greedy
terp2.0 terp2.0
greedy
growdy
Temp2.0 temp2.0
0.25
0.625
Inference Type
Inference Type
0.80
8.888
CircleConstruction
LineConstruction
seed
null
seed
null
Creativity
Circle Construction
Line Construction
Creat
Creativity
0.4
Creativity
0.4
seed10
greedy
greedy
50a04
terp2.0 8 terp2. 0
5eed2
nult
seed gready
greedy
seeed
temp2.a
seed2
null temp2.6
0.2
0.2
Inference
Type
Inference
Type
0.0
Figure 5. Seed-condittoning significantly improves algorithmic ereativity of both next- and multi-token prediction on Gemma v1 (2B) model. The X-axis labels denote the prefix (at training and inference) and the temperature (at inference).
seed
nuLL
Figure 6. Seed-conditioning achieves comparable algorithmic creativity in the GPT-2 (86M) model: All models trained with NTP. The color denotes temperature, while the X-axis, the pre- fix. Remarkably, seed-conditioning achieves comparable creativity even with greedy decoding. and improves creativity particularly in our exploratory creativity tasks (bottom).
experiments on boken-reordent and experiments ruling out other hypotheses.
Seed-conditioning improves algorithmic creativity for Transformers. Orthogonal to the effect of multi-token vs. next-token objectives, we point out the effect of seed- conditioning a Transformer. First, and most remarkably, with seed-conditioning, there is no need for temperature: even deterministic greedy decoding generates diverse out- puts and highly non-trivial creativity (Fig 5, Fig 6). This means that, even though the seeds we provide are arbitrarily paired with the training datapoint, the models have learned to transform the arbitrary noise into meaningful random- ness. Next, increasing the seed lengths consistently boosts algorithmic creativity for both next-token and multi-token approaches (Fig 5, 23), presumably up to a limit. Thus, for Transformers, we propose viewing geed-conditioning as a knob for diversity distinct from temperature sampling.
hypotheses for when and why seed-conditioning may help Transformers in $C.1. Note that we do not see gains of seed- conditioning when it comes to diffusion training ($ G.3).
Robustness to hyperparameters. In SF and Fig 22, we do sensitivity analysis all the datasets. We report how our above findings are robust to the choice of learning rate, batch-size, number of training steps, weight given to the multi-token objective, varying sampling conditions and rea- sonable changes to the complexity of the dataset and train- ing set size (as per our argument in $2.6, we do expect the next-vs, multi-token gap to diminish for larger dataset size).
4.2. An exploration of real-world summarization
For a more realistic examination, we finetune GPT models NTP and the multi-token teacherless objectives on summa- rization tasks (XSUM, CNN/DailyMailn diversity of model for any given prompt by generating 5 different completions and computing a Self- -Bleu metric (Zhu et al., 2018). Admittedly though, a summarization task is not as open-ended as we would like: a higher quality model (i.e., higher Rouge; Lin, 2004) necessarily means lower diversity. To account for this, we plot how diversity evolves over time then find in Fig that for given model quality, the larger multi-token models achieve higher diversity (albeit only by slight amount). This increase does not hold for smaller mod- els and is not always noticeable for CNN MMN/DailyMal (see $J). Interestingly, teacherless training consistently shows an increase summarization quality, measured by Rouge.
Next, conditinii results in algorithmic creativity comparable temperature sampling. For the larger models (Fig 5), seed-conditioning generally outperforms the base- line "null" conditioning (which appears mode-collapsed, see Fig 18. 19, 20). For the smaller model as well (Fig 6), we find gains in all tasks except for Sibling Discovery. Besides, for any fixed temperature, prefixing a seed only improves performance over null prefix (Fig 6, 18).
We measure the
In S1.2, we find that the improved creativity from seed- conditioning comes from improved diversity, rather than from reduced memorization (unlike with multi-token train- ing), All these behaviors of seed-conditioning require fur- ther study as they are non-trivial, especially given that the seeds are naively, arbitrarily chosen in relationship to the output. As starting point, we provide some preliminary
function of generation quality; we
8

---

Going beyond the creative limits of next-token prediction
Going beyond next-token prediction (NTP). There has been recent emerging discussion surrounding the role of NTP as foundational piece in developing intelligent mod- els. On the critical side, arguments have been made about the inference-time issues with auto-regression (Dziri et al., 2024; LeCun, 2024; Kääriäinen, 2006; Ross & Bagnell, 2010), Others have reported the planning and arithmetic lim- itations of next-token trained models (McCoy al., 2023; Momennejad et al., 2023; Valmeekam et al., 2023a;b;e; Bachmann & Nagarajan, 2024) where the goal is accuracy, not diversity. As for diffusion as an alternative to NTP, our findings parallel that of Ye et al. (2024) who show that their variant of diffusion is able to solve the challenging path-star task of 24. We provide references to more lines of multi-token prediction work in SK. There are also other Transformer failures such as the reversal curse (Allen- Zhu & Li, 2023a) or shorteut-learning (Dziri et al., 2024; Zhang et al., 2023; Liu et al., 2023; Young & You, 2022;
Training 0ojective:
Standard (Next-Token)
Teacherless Mutts-Token)
XSUN-GPT-XL
XSUM-GPT-LARGE
-35
Quality (ROUSE)
Quality (ROUGE!
Figure 7. Multi-token training improves diversity scores for XSUM summarization for large GPT-2 models: We observe gaps diversity for a fixed quality over the course of finetuning.
5. Related Work
Open-ended algorithmic tasks. Directly related to us are Khona et al. (2024); Allen-Zhu & Li (2023b) who study diversity of eext-toke-trrineee models on an open- ended algorithmic task. Khona al. (2024) demonstrate a for
Lai et al., 2021; Ranaldi & Zanzotto, 2023), however these are out -ittt--distriuttion failures; the sub-optimality we show is in-distribution, like in B&N
evversity-accurase tradeoff under temperature-scalln path-connectivity on on knowledge graph. We show how this tradeoff can be improved under alternative training methods. Allen-Zhu & Li (2023b) empirically demonstrate that next-token predictors are able to learn a synthetic, chal- lenging CFG, in the "infinite" data regime (<s 100m tokens). Our datasets are not CFGs, with the exception of Siblini ing Discovery. Our negative result does not contradict theirs since what we show is sub-optimality of NTP in a smaller data regime. Our work also extends the above works by studying limitations in much more minimal tasks that re- quire as little as 2-hop lookahead. There are other works that study Transformers on non-open-ended graph-algorithmic tasks, discussed in SK.
Injecting noise into a Transformer. Most to seed- conditioning is DeSalvo et al. (2024) who induce diversity by varying a soft-prompt learned using a reconstruction loss. Our approach requires no modification to the architecture or the loss; however, train the whole model, which is more expensive. A concurrent position paper (Jahrens & Martinetz, 2025) conceptually suggests injecting noise with the same motivation as us. Seeddcoonitioning may also be viewed as a controllable way to vary prompt wordings which is known to induce diverse outputs (Li et al., 2023; Lau et al., 2024; Naik et al., 2024; Li et al., 2022).
6. Conclusions
This work provides a minimal test-bed of tasks abstract- ing distinct modes of creativity. While these tasks are ad- mittedly an extreme caricaturization of real-world tasks, they enable us quantify otherwise elusive metries like originality and diversity. They also enable us to control and investigate distinct parts of the current apparatus for language modeling (next-token learning and temperature sampling) and advocate for alternatives (multi- token learn- ing and seed-conditioning). Next, one can investigate the effect of length generalization, scaling and the effect of in-context learning vs. finetuning. The surprising effec- tiveness of ceed-conditioning raises various open questions (SC.3). Other profound questions arise, such whether geasoning-enhanche methods like RI. and CoT are optimal for enhancing open-ended diversity and originality ($C.2). We hope our work inspires discussion in the various direc- tions of multi-token prediction, creativity and planning. We present more discussions in $C and limitations in $A.
Diversity in generative models. Generative diversity has long been a major goal, at least until the revolution in (de- terministic) reasoning of language models. Much work has gone into concerns such as mode collapse (Che al., 2017) or posterior collapse (Bowman et al., 2016) and memoriza- tion (Carlini et al., 2020; 2023; Nasr et al., 2023). Temper- ature sampling has been known to have weak correlation with creativity, often inadvertently introducing incoherence (Peeperkorn et al., 2024; Chen & Ding, 2023; Chung et al., 2023). Our results on renditondiio are also reminiscent of a line of work on reinforcement learning (RL) showing that adding noises to the policy model parameters enables more efficient exploration than directly adding noises to the output space (Plappert et al., 2017; Fortunato et al., 2017). We defer discussion of theoretical studies of di- versity and memorization and with empirical studies on natural-language creativity in SK.

---

Going beyond the creative limits of next-token prediction
Impact Statement
serence on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 2296-2318, 2024.
This paper presents work whose goal is to advance the field of Machine Learning through the study of simple algorith- mic tasks inspired by creativity. There are many potential societal consequences of our work especially if one ap- plies Al to real-world creative endeavors- none which we feel must be specifically highlighted in our focused algorith- mic study.
Bavarian, M., Jun, H., Tezak, N. A., Schulman, J., McLeavey, C., Tworek, J., and Chen, M. Efficient training of language models fill in the middle, ArXiv, abs/2207.14255,, 2022. URL https://api. semanticacholar org/Corpuu 251135228.
Beel, J., Kan, M.-Y., and Baumgart, M. Evaluating sakana's ai scientist for autonomous research: Wishful thinking an emerging reality towards 'artificial research intelli- gence' (ari)?, 2025.
Acknowledgements
We wish to thank Gregor Bachmann, Jacob Springer, and Sachin Goyal for extensive feedback draft of the paper. We are grateful to Vansh Bansal whose significant effort in an independent reproduction of our GPT-2 (86M) ex- periments helped zero in on the role of top-K sampling in seed-conditioning for some of our datasets which we had overlooked in initial versions of the paper. We wish to thank Quentin Fournier for excellent references to work on Al for scientific discovery. We thank Garret Tanzer and Elan Rosenfeld for helping arrive the term "seed-conditioning", We also wish to thank Mike Mozer, Suhas Kotha, Clayton Sanford, Christina Baek, Yuxiao Qu, Ziqian Zhong for valu- able discussions and pointers. The work was supported in part by Cisco, Apple, Google, OpenAI, NSF, Okawa foundation and Schmidt Sciences.
Berglund, L.., Tong, M., Kaufmann, M., Balesni, M., Stick- land, A. C., Korbak, T., and Evans, O. The reversal curse: Lims trained on "a b" fail learn "b is a". In The Twelfth International Conference on Learning Represen- tations, ICIR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https://openreviev. net/forum?id-GPKTII
Bigelow, E. J., Lubana, E. S., Dick, R. P., Tanaka, H., and Ullman, T. D. In-context learning dynamics with random binary sequences. In The Twelfth International Confer- ence on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024.
Boden, M. A. The Creative Mind Myths and Mechanisms (2. ed.). Routledge, 2003.
References
Bowman, S. R., Vilnis, L., Vinyals, O., Dai, A. M., Józe- fowiez, R., and Bengio, S. Generating sentences from continuous space. In Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learn- ing, CoNLL 2016., pp. 10-21. ACL, 2016.
Alabdulmohsin, I. Tran, V. Q., and Dehghani, M. Frac- tal patterns may unravel the intelligence in next-token prediction, 2024.
Allen-Zhu, Z. and Li, Y. Physics of language models: Part 3.2, knowledge manipulation. CoRR, abs/2309.14403 2023a. doi: 10.48500ARXXI.2300..44A 14402. URL https: //doi.org/10 htte//1..4855///00 48550/arXiv.... http:/awwwvvv..........
Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y., Lundberg, S., et al. Sparks of artificial general intel- ligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.1277 2023.
Allen-Zhu, Z. and Li, Y. Physics of language models: Part 1, context-free grammar. CORR, abs/2305.13677 2023b. doi: 10.465500ARXXII22055..55 13673. URL https://doi. org/10. 10/0
Callaway, E. Cognitive science: Leap of thought. Nature, 502, 2013.
11.11.48550 405.0/8550500 1/550/arXiv.....///// 2305. 13673.
Anderson, B. R., Shah, J. H., and Kreminski, M. Ho- mogenization effects of large language models on hu- man creative ideation. In Proceedings of the 16th Con- ference on Creativity & Cognition, Chicago, II, USA, June 23-26, 2024, pp. 413 425. ACM, 2024. URL 3656204.
Carlini, N., Tramèr, F., Wallace, E., Jagielski, M., Herbert- Voss, A., Lee, K., Roberts, A., Brown, B., Song, D. X., Erlingsson, Ú., Oprea, A., and Raffel, C. Extracting training data from large language models. In USENIX Security Symposium, 2020.
https:/ //doi. org/ 10. 1144/11355660 010236666.36622444
Austin, J., Johnson, D. D., Ho, J., Tarlow, D., and van den Berg, R. Structured denoising diffusion models in discrete state-spaces. NeurlPS, 2021.
Carlini, N., Ippolito, D., Jagielski, M., Lee, K., Tramèr, F, and Zhang, C. Quantifying memorization across neural language models. ICLR, 2023.
Bachmann, G. and Nagarajan, V. The pitfalls of next-token prediction, In Proceedings of the 41st International Con-
Chakrabarty, T., Laban, P., Agarwal, D., Muresan, S., and Wu, C. Art or artifice? large language models and the
10

---

Going beyond the creative limits of next-token prediction
false promise of creativity. In Proceedings of the CHI Conference on Human Factors in Computing Systems, CHI 2024, Honolulu, HI, USA, May 11-16, 2024, pp. 30:1 10:1-30:34. ACM, 2024.
DeepSeek-AI, Liu, A., Feng, B., Xue, B., Wang, B.-L., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., Dai, D., Guo, D., Yang, D., Chen, D., Ji, D.-L., Li, E., Lin, F. Dai, F., Luo, F., Hao, G., Chen, G., Li, G.. Zhang, H., Bao, H., Xu, H., Wang. H., Zhang, H., Ding, H., Xin, H., Gao, H., Li, H., Qu, H., Cai, J. L.,, Liang, J., Guo. J., Ni, J., Li, J., Wang. J., Chen, J., Chen, J., Yuan, J., Qiu, J., Li, J., Song, J.-M., Dong. K., Hu, K., Gao. K., Guan, K., Huang. K., Yu, K., Wang. L.,, Zhang, L.., Xu, L.., Xia, L.,, Zhao, L., Wang, L., Zhang, L.,, Li, M., Wang. M., Zhang, M., Zhang. M. Tang, M., Li, M., Tian, N., Huang, P., Wang. P., Zhang, P., Wang, Q. Zhu, Q., Chen, Q. Du, Q. Chen, R. Jin, R. L., Ge, R., Zhang, R., Pan, R., Wang, R., Xu, R., Zhang, R., Chen, R., Li, S. S., Lu, S., Zhou, S., Chen, S., Wu, S.-P. Ye. S., Ma, S., Wang. S., Zhou, S., Yu, S., Zhou, S., Pan, S., Wang. T., Yun, T., Pei, T., Sun, T., Xiao, W. L., Zeng, W., Zhao, W., An, W., Liu, W., Liang, W., Gao, W., Yu, W.-X., Zhang, W., Li, X. Q., Jin, X., Wang. X., Bi, X., Liu, X., Wang, X., Shen, X.-C., Chen, X., Zhang, X., Chen, X., Nie, X., Sun, X., Wang, X., Cheng, X., Liu, X., Xie, X., Liu, X., Yu, X., Song, X., Shan, X., Zhou, X., Yang. X., Li, X., Su, X., Lin, X., Li, Y. K., Wang, Y. Q., Wei, Y. X., Zhu, Y. X., Zhang, Y.. Xu, Y., Huang, Y., Li, Y., Zhao, Y., Sun, Y.. Li, Y., Wang, Y., Yu, Y., Zheng, Y., Zhang, Y., Shi, Y., Xiong, Y., He, Y., Tang, Y., Piao, Y., Wang. Y., Tan, Y., Ma, Y.-B., Liu, Y. Guo, Y., Wu, Y., Ou, Y., Zhu, Y., Wang, Y., Gong, Y., Zou, Y., He, Y., Zha, Y., Xiong, Y., Ma, Y. Yan, Y., Luo, Y.W.,, mei You, Y., Liu, Y.. Zhou, Y., Wu, Z. F., Ren, Z., Ren, Z., Sha, Z., Fu, Z., Xu, Z., Huang. Z., Zhang, Z., Xie, Z., guo Zhang, Z.,, Hao, 7.,, Gou, Z., Ma, 7.,, Yan, Z Shao, Z., Xu, Z., Wu, Z., Zhang, Z., Li, Z., Gu, Z., Zhu, Z., Liu, Z., Li, Z.-A., Xie, Z., Song, Z., Gao, Z., and Pan, Z. Deepseek-v3 technical report. In ArXiv, 2024
Che, T., Li, Y., Jacob, A. P., Bengio, Y., and Li, W. Mode regularized generative adversarial networks. In 5th Inter- national Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017.
Chen, H. and Ding, N. Probing the "creativity" of large lan- guage models: Can models produce divergent semantic association? In Findings of the Association for Computa- tional Linguistics: EMNLP 2023, Singapore, December 6-10, 2023, pp. 11881-1288.. Association for Computa- tional Linguistics, 2023.
Chow, Y. Tennenholtz, G., Gur, I,, Zhuang, V. Dai, B., Thi- agarajan, S., Boutilier, C., Agarwal, R., Kumar, A., and Faust, A. Inference-aware fine-tuning for best-of-n sam- pling in large language models. ArXiv, abs/2412.15287, 2024. URL https://app //tp://appi... wtt/pppssssaatccccoolaaa.ooo.o CorpusID:274965000
Chung. J. Y.. Kamar, E., and Amershi, S. Increasing diversity while maintaining accuracy: Text data genera- tion with large language models and human interventions. In Proceedings of the 61st Anmual Meeting of the Asso- ciation for Computational Linguistics (Volume I: Long Papers), ACL, pp. 575-593, Association for Computa- tional Linguistics, 2023.
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training verifiers solve math word problems. arXiv preprint arXiv:2110.144 14168, 2021.
DeSalvo, G., Kagy, J.-F, Karydas, I.., Rostamizadeh, A., and Kumar, S. No more hard prompts: Softsrv prompt- ing for synthetic data generation, 2024. URL https: // //arxiv. org/ brgg/bss/2ae a2s/ 16534.
Csikszentmihalyi, M. Creativity: Flow and the Psychology of Discovery and Invention. HarperCollins Publishers, New York, NY, first edition, 1996.
Du, L., Mei, H., and Eisner, J. Autoregressive modeling with lookahead attention. arXiv preprint arXiv:2305.12272. 2023.
Dang, X., Baek, C., Wen, K., Kolter, Z., and Raghunathan, A. Weight ensembling improves reasoning in language models. 2025. URL https: / http:/pppp..aaaiaatiiiiaaaa sew.ntccccooola.r.........cc arg/CorpusIDD:27781112
Dziri, N., Lu, X., Sclar, M., L. X. L.,, Jiang, L., Lin, B. Y., Welleck, S., West, P., Bhagavatula, C., Le Bras, R., et al. Faith and fate: Limits of transformers on compositionality. Advances in Neural Information Processing Systems, 36. 2024.
Dawid, A. and LeCun, Y. Introduction to latent variable energy-based models: A path towards autonomous ma- chine intelligence. arXiv preprint arXiv:2306.02572, 2023.
Feng, G., Zhang, B., Gu, Y., Ye, H., He, D., and Wang. L. Towards revealing the mystery behind chain of thought: a theoretical perspective. Advances in Neural Information Processing Systems, 36, 2023.
DeepSeek-AL. Deepseek-rl: Incentivizing reasoning ca- pability in 11ms via reinforcement learning. ArXiv, hls/2501.119444 2025.
Finlayson, M., Hewitt, J., Koller, A., Swayamdipta, S., and

---

Going beyond the creative limits of next-token prediction
Sabharwal, A. Closing the curious case of neural text degeneration. In The Twelfth International Conference Learning Representations, ICLR 2024, Vienna, Austria, May 7-11. 2024. OpenReview.net, 2024.
Gruver, N., Stanton, S., Frey, N. C., Rudner, T. G. J., Hotzel, I., Lafrance-Vanasse, J., Rajpal, A., Cho, K., and Wil- son, A. G. Protein design with guided discrete diffu- sion. ArXiv, abs/2305.20009, 2023. URL https://api. semanticacholar pewpaatiicccoollo.oo//oooopppppe (B 258987335.
Fortunato, M., Azar, M. G., Piot, B., Menick, J., Osband, I., Graves, A., Mnih, V., Munos, R., Hassabis, D., Pietquin, O., Blundell, C., and Legg, Noisy networks for ex- ploration. ArXiv, 0bss/776..10999 2017. URL https: //api. slapiissemanticscholles. org/ wwrg/CorpusID.... 5176587.
Gu, J., Bradbury, J., Xiong, C., Li, V. O. K., and Socher, R. Non-autoregressivve neural machine translation. In 6th International Conference on Learning Representa- tions, ICLR 2018. Conference Track Proceedings. Open- Review.net, 2018.
Franceschelli, G. and Musolesi, M. On the creativity of large language models. CoRR, abs/2304.00008, 2023.
Gupta, T. and Pruthi, D. All that glitters novel: Pla- giarism in ai generated research, 2025. URL https:
Fried, D., Aghajanyan, A., Lin, J., Wang, S. I., Wallace, E., Shi, F., Zhong, R., tau Yih, W., Zettlemoyer, L., and Lewis, M. Incoder: A generative model for code infilling and synthesis. ArXiv, abs/2204.05999, 2022.
//arxiv. wtt.oorg////aeeeeeoooeee abs.///ss222.... 12250....87....
Hayes, T., Rao, R., Akin, H., Sofroniew, N. J., Ok- tay, D., Lin, Z., Verkuil, R., Tran, V. Q., Deaton, J., Wiggert, M., Badkundri, R., Shafkat, I., Gong, J., Derry, A., Molina, R. S., Thomas, N., Khan, Y. A., Mishra, C., Kim, C., Bartie, L. J., Nemeth, M., Hsu, P. D., Sereu, T., Candido, S., and Rives, A. Simu- lating 500 million years of evolution with language model. Science, 387(6736)8850-8 858, 2025. doi: 10.1126/ science.ads0018. URL https: :/vp::://wwwrt www.wwwwccnc...o... org/ doi/abs/100 10. 1126/sciiecceeeeee adw.m00 ads0018.
Frydenlund, A. The mystery the pathological path-star task for language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, Miami, FL,, USA, November 12-16, 2024, pp. 12493-12516. Association for Computa- tional Linguistics, 2024.
Gemma Team, T. M., Hardin, C., Dadashi, R., Bhupatiraju, S., Sifre, L.,, Rivière, M., Kale, M. S., Love, J., Tafti, P., Hussenot, L., and et Gemma. 2024. doi: 10.34740/ KAGGLE/M//301. URL https:///ww/i http:::/www..aggle... waogglee.com com/ m/3301.
Hofstadter, D. A review of mental leaps: Analogy in cre- ative thought. Al Mag., 16(3):75-80, 1995. doi: 10. 1609/AIMAG. V1613. 1154. URL https://doi.org: 10. 10.1609/aimag v1613. 1154.
Gloeckle, F., Idrissi, B. Y., Rozière, B., Lopez-Paz, D., and Synnaeve, G. Better & faster large language models via multi-token prediction. 2024.
Holyoak, K. and Thagard, P. Mental leaps: analogy in creative thought. MIT Press, Cambridge, MA, USA, 1995. ISBN 0262082330.
Gong, S., Li, M., Feng, J., Wu, Z.,, and Kong, L. Diffuseq: Sequence to sequence text generation with diffusion mod- els. In The Eleventh International Conference on Learn- ing Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023.
Hoogeboom, E., Nielsen, D., Jaini, P., Forr'e, P., and Welling, M. Argmax flows and multinomial diffusion: Learning categorical distributions. In Neural Information Processing Systems, 2021.
Goodfellow, I. J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A. C., and Bengio, Y. Generative adversarial networks. Commun. ACM, doi: 10. 10/1442/342262.. URL 1145/3422622.
Hopkins, A. K., Renda, A., and Carbin, M. Can LLMs generate random numbers? evaluating LLM sampling in controlled domains. In ICML 2023 Workshop: Sampling and Optimization Discrete Space, 2023, URL https: ///ppe/reevieeeeee///o/oomr/eeee wwwt:/ooruuiid--/hkk......
63(11))139-144, 2020.
https://doi. orgooo//////// 10. http///......1.2211111
//opnreview
Goyal, A., Sordoni, A., Côté, M.-A., Ke, N. R., and Bengio, Y. Z-forcing: Training stochastic recurrent networks. NeurlPS, 2017.
Hu, E. S., Ahn, K., Liu, Q., Xu, H., Tomar, M., Langford, A., Jayaraman, D.. Lamb, A., and Langford, J. The belief state transformer. In The Thirleenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net, 2025.
Goyal, S., Ji, Z., Rawat, A. S., Menon, A. K., Kumar, S., and Nagarajan, V. Think before you speak: Training language models with pause tokens. The Twelfth International Conference on Learning Representations, ICLR 2024, 2024.
Hua, H., X., Dou, D., Xu, C., and Luo, J. Fine-tuning pre-trained language models with noise stability regular- ization. CoRR, 2022.
12

---

Going beyond the creative limits of next-token prediction
Jahrens, M. and Martinetz, T. Why Ilms cannot think and how fix it, 2025. URL https://arxivv wetreriiivoorvveeee org/abs/ 2503. 2503.09211.
Lai, Y., Zhang, C., Feng, Y., Huang, Q., and Zhao, D. Why machine reading comprehension models learn shortcuts? In Findings of the Association for Computational Lin- guistics: ACL/IJCNLP 2021, Online Event, August 1-6, 2021, volume ACI/I/CNLP 2021 of Findings ACI,, pp- 989-1002. Association for Computational Linguistics,
Jain, N., Chiang, P., Wen, Y., Kirchenbauer, J., Chu, H., Somepalli, G., Bartoldson, B. R., Kailkhura, B., Schwarzschild, A., Saha, A., Goldblum, M., Geiping, J., and Goldstein, T. Neftune: Noisy embeddings improve instruction finetuning. The Twelfth International Con- ference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024.
2021.
Lau, G. K. R., Hu, W., Liu, D., Chen, J., Ng. S.-K., and Low, B. K. H. Dipper: Diversity in prompts for producing large language model ensembles in reasoning tasks, 2024. URL https http//axxivv.. www.iiiw................ httg//bs/2200 /4122. 15238.
Jansen, P. A., Cot'e, e, M.-A., Khot, T., Bransom, E., Dalvi, B., Majumder, B. P., Tafjord, O., and Clark, P. Discov- cryworid: A virtual environment for developing and eval- uating automated scientific discovery agents. NeurIPS, 2024.
LeCun, Y. Do large language models need sensory ground- ing for meaning and understanding? University Lecture, 2024.
Lee, N., Sreenivasan, K., J. D., Lee, K., and Papail- iopoulos, D. Teaching arithmetic to small transformers. In The Twelfth International Conference on Learning Representations, ICLR 2024. OpenReview.net 2024.
Kääriäinen, M. Lower bounds for reductions. In Atomic Learning Workshop, 2006.
Kalai, A. T. and Vempala, S. S. Calibrated language models must hallucinate. In Proceedings of the 56th Annual ACM Symposium Theory of Computing, STOC 2024, Vancouver, BC, Canada, June 24-28, 2024, pp. 160-171. ACM, 2024.
Li, Y., Choi, D. H., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Lago, A. D., Hubert, T., Choy, P., de Masson d' Autume, C., Babuschkin, I., Chen, X., Huang, P.. Welbl, J., Gowal, S., Cherepanov, A., Molloy, J., Mankowitz, D. J., Rob- son, E. S., Kohli, P., de Freitas, N., Kavukcuoglu, K., and Vinyals, O. Competition-level code generation with alphacode. 2022.
Kalavasis, A., Mehrotra, A., and Velegkas, G. On the limits of language generation: Trade-off' between hallucination and mode collapse. abs/2411 1.09642, 2024.
Kamb, M. and Ganguli, S. An analytic theory of creativity in convolutional diffusion models, 2024. URL https: // //taxxiv.orgg////e/ /21p./222...2 2022.
Li, Y., Lin, Z., Zhang. S., Fu, Q., Chen, B., Lou, J.-G., and Chen, W. Making language models better reasoners with step-aware verifier. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Toronto, Canada, July 2023, Association for Computational Linguistics. URL https: //aclanthology. wrg/2023.3... acl-lcnlaaaal 291/
Khona, M., Okawa, M., Hula, J., Ramesh, R., Nishi, K., Dick, R. P., Lubana, E. S., and Tanaka, H. Towards an understanding of stepwise inference in transformers: A synthetic graph navigation model. Forty-first Inter- national Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net 2024.
Lin, C.-Y. ROUGE:
A package for automatic evaluation of summaries. In Text Summarization Branches Out, pp. 74-81 Barcelona, Spain, July 2004. Association for Com- putational Linguistics. URL org/w04- /0W/1000111..
Kingma, D. P. and Welling, M. Auto-encoding variational bayes. In 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014. URL http: //a///a//// /arxiviv werxiv.or///a.... ats//3///1111 6114.
Liu, B., Ash, J. T., Goel, S., Krishnamurthy, A., and Zhang, C. Transformers learn shortcuts to automata. In The Eleventh International Conference on Learning Repre- sentations, ICLR 2023, 2023.
Kitouni, O., Nolte, N., Bouchacourt, D., Williams, A., Rabbat, M., and Ibrahim, M. The factorization curse: Which tokens you predict underlie the reversal curse and more. CORR, abs/2406.0511 001888 2024. doi: 10. 48550/ARXXVV2406.0051
URL https://doi.orr 10. 48550/ arXiv wwt.xii..406..0 04406.05183.
Lou, A., Meng, C., and Ermon, S. Discrete diffusion mod- eling by estimating the ratios of the data distribution. In International Conference on Machine Learning, 2023.
Kleinberg. J. M. and Mullainathan, S. Language generation in the limit. CORR, abs/2404.06777 4.6507, 2024. doi: 10. 48S50/ARRXVV2244000777 10. 48550/ arxiv. 2404.06757.
Lu, C., Lu, C., Lange, R. T., Foerster, J., Clune, J., and Ha, D. The ai scientist: Towards fully automated open-ended scientific discovery, 2024a.
URL hattp::/ddel htte:://dooooooo org/
13

---

Going beyond the creative limits of next-token prediction
Lu, X., Sclar, M., Hallinan, S., Mireshghallah, N., Liu, J., Han, S., Ettinger, A., Jiang, L.., Chandu, K. R., Dziri, N., and Choi, Y. AI humanity's ss salieri: Quantifying lin- guistic creativity of language models via systematic attri- bution of machine text against web text. abs/2410.04265,, 2024b.
Nagarajan, V., Raffel, C., and Goodfellow, J. Theoretical insights into memorization in gans. In Neural Information Processing Systems Workshop, volume 1, pp. 3, 2018.
Naik, R., Chandrasekaran, V., Yuksekgonul, M., Palangi. H., and Nushi, B. Diversity of thought improves reasoning abilities of Ilms, 2024. URL wtwpt:tps///aatt www.aaiwww...o.aaoaaaaaa//a
Madani, A., Krause, B., Greene, E. R., Subramanian, S., Mohr, B. P., Holton, J. M., Olmos, Jr., J. L., Xiong, C., Sun, Z. Z., Socher, R., Fraser, J. S., and Naik, N. Large language models generate functional protein sequences across diverse families. Nature Biotechnology, 41(8), 2023. doi: 00.1088/41157--0220011333
2310. 2310.07088. 07088.
Nakkiran, P., Bradley, A., Zhou, H., and Advani, M. Step- by-step diffusion: An elementary tutorial, 2024.
Nallapati, R., Zhou, B., dos santos, C. N., Gulcehre, C., and
Xiang. B. Abstractive text summarization using sequence- to-sequence rnns and beyond, 2016. URL https://
Malach, E. Auto-regressive next-token predictors are uni- versal learners. arXiv preprint arXiv:2309. 06979, 2023.
arxiv. wwwpg///////aaae/ 0atpb///162.00000 06023.
Narayan, S., Cohen, S. B., and Lapata, M. Don't give me the details, just just the summary! topic-aware convolutional neural networks for extreme summarization, 2018. URL attps://arxxe
McCoy, R. T., Yao, S., Friedman, D., Hardy, M., and Grif- fiths, T. L. Embers of autoregression: Understanding large language models through the problem they are trained to solve. arXiv preprint arXiv:23309. 102009113333 13638, 2023.
brrg/bb//an 180/ 100ps/18888...85....
08745.
Nasr, M., Carlini, N., Hayase, J., Jagiclski, M., Cooper, A. F., Ippolito, D., Choquette-Choo, C. A., Wallace, E., Tramèr. F., and Lee, K. Scalable extraction of training data from (production) language models. ArXiv, 2023.
McLaughlin, A., Campbell, J., Uppuluri, A., and Yang, Y. Aidanbench: Stress-testing language model creativity open-ended questions. In NeurlPS 2024 Workshop Language Gamification, 2024.
Nguyen, E., Poli, M., Durrant, M. G., Kang, B., Ka- trekar, D., Li, D. B., Bartie, L., J., Thomas, A. W., King, S. H., Brixi, G., Sullivan, J., Ng. M. Y., Lewis, A., Lou, T.,
Meier, J., Rao, R., Verkuil, R., Liu, J., Sereu, T., and Rives, A. Language models enable zero-shot prediction of the effects of mutations protein function. In Advances in Neural Information Processing Systems, volume 34, pp. 29287-29303, 2021.
A., Ermon, S., Baccus, S. A., Hermandez-Boussaan Ré, C., Hsu, D., and Hic, B. L. Sequence model- ing and design from molecular to genome scale with evo. Science, 386(6723)eead093336 2024. doi: 10.1 1226 science.ado9336. URL https://ww/w http:://wwwwwwwwneee/eeeooee org/ doi/ 10. www./11/cccciie swtcccciiiccc.eeo www.ieeneeeddd33....oo.. ado9336.
Merrill, W. and Sabharwal, A. The expressive power of transformers with chain of thought. The Twelfih Inter- national Conference on Learning Representations, 2024.
Nolte, N., Kitouni, O., Williams, A., Rabbat, M., and Ibrahim, M. Transformers can navigate mazes with multi-step prediction. CORR, abs/2412.05117, 2024. doi: 1044850/ARKIVV2412201111 URL https://ddo org/10. /0. 48550/ 2212.. 05117.
Mirowski, P. W., Love, J., Mathewson, K. W., and Mo- hamed, S. A robot walks into a bar: Can language models serve as creativity support tools for comedy? an evalua- tion of lIms' humour alignment with comedians. CoRR, abs/2405.20956, 2024.
Okawa, M., Lubana, E. S., Dick, R. P., and Tanaka, H. Com- positional abilities emerge multiplicatively: Exploring diffusion models on a synthetic task. In Advances in Neural Information Processing Systems 36: Annual Con- ference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 16. 2023, 2023.
Momennejad, Hasanbeig, H., Frujeri, V., Sharma, H., Ness, R. O., Jojic, N., Palangi, H., and Larson, J. Eval- uating cognitive maps and planning in large language models with cogeval. Advances in Neural Information Processing Systems, 36, 2023.
Monea, G., Joulin, A., and Grave, E. Pass: Parallel spec- ulative sampling. 3rd Workshop on Efficient Natural Language and Speech Processing (NeurIPS 2023), 2023.
OpenAI. Openai ol system card. ArXiv, 2024.
Padmakumar, V. and He, H. Does writing with language models reduce content diversity? In The Twelfth Inter- national Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https: //pp:reviee/eiere//// net/forumiid- Feiz5HtCDO.
Muennighoff, N., Yang, Z., Shi, W., Li, X. L., Fei-Fei, L., Hajishirzi, H., Zettlemoyer, L., Liang, P., Candès, E. J., and Hashimoto, T. sl: Simple test-time scaling. abs/2501. 19393, 2025.
14

---

Going beyond the creative limits of next-token prediction
Pannatier, A., Courdier, E., and Fleuret, o-gpts: A new approach to autoregressive models. In Machine Learn- ing and Knowledge Discovery in Databases. Research Track European Conference, ECML PKDD 2024, Vil- nius, Lithuania, September 9-13, 2024, Proceedings, Part VII, volume 14947 of Lecture Notes in Computer Science, pp. 143-159. Springer, 2024.
Saparov, A., Pawar, S., Pimpalgaonkar, S., Joshi, N., Pang. R. Y., Padmakumar, V., Kazemi, S. M., Kim, N., and He, H. Transformers struggle to learn to search, 2024, URL https://arxiv httplaaiivv....gaaaa hr/gabb//4411 04703.
Schmidhuber, J. Driven by compression progress: A sim- ple principle explains essential aspects of subjective beauty, novelty, surprise, interestingness, attention, cu- riosity, creativity, art, science, music, jokes. In Computa- tional Creativity: An Interdisciplinary Approach, 12.07. 17.07.2009, Dagstuhl Seminar Proceedings, 2009.
Peeperkorn, M., Kouwenhoven, T., Brown, D., and Jor- danous, A. Is temperature the creativity parameter of large language models? abs/2405. 00492, 2024.
Pezeshki, M.. Kaba, S.. Bengio, Y., Courville. A. C.,, Precup, D., and Lajoie, G. Gradient starvation: A learning procliv- ity in neural networks. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurlPS 2021, De- cember 6-14, 2021, virtual, pp. 1276-1272, 2021.
Schnitzler, J., Ho, X., Huang, J., Boudin, F,, Sugawara, S., and Aizawa, A. Morehopqa: More than multi-hop reasoning. abs/2406. 13397. doi: (0.48550/ 13397.
Shah, H., Tamuly, K., Raghunathan, A., Jain, P., and Netrapalli, P. The pitfalls of simplicity bias in neural networks. In Advances in Neural Information Pro- cessing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020. URL https:
Plappert, M., Houthooft, R., Dhariwal, P., Sidor, S., Chen, R. Y., Chen, X., Asfour, T., Abbeel, P., and Andrychowicz, M. Parameter space noise for explo- ration. ArXiv, abs/1706.01905, 2017. URL https: //api. seappsisemaniccccoolle htt.:oolar..rrrrrrusai::::1:111 29716655
/ /pppoeediing..nnrrss.ccccc
6cf nwwrips. www.erppp...ppppp////////./ html.
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. Language models unsupervised multitask learners. 2019.
Shannon, C. E. A mathematical theory of communication.
Ranaldi, L. and Zanzotto, F. M. Hans, are you clever? clever hans effect analysis of neural systems, 2023.
The Bell System Technical Journal, 27(3):379-433, 1948.
Shannon, C. E. Prediction and entropy of printed english. The Bell System Technical Journal, 30(1):50 64, 1951.
Rives, A., Meier, J., Sercu, T., Goyal, S., Lin, Z., Liu, J., Guo, D., Ott, M., Zitnick, C. L., Ma, J., and Fergus, R. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. Proc. Natl. Acad. Sci. USA, 118(1))e220) ((15))2016239111 2021.
Shlegeris, B., Roger, F., Chan, L., and McLean, E. Language models are better than humans at next-token prediction. arXiv preprint arXiv:2212.11281, 2022.
Si, C., Yang, D., and Hashimoto, T. Can Ilms generate novel research ideas? A large-scale human study with 100+ NLP researchers. 2024.
Romera- Parede,, B., Barekatain, M., Novikov, A., Balog, M.. Kumar, M. P., Dupont, E., Ruiz, F. J. R., Ellenberg, J. S., Wang, P., Fawzi, O., Kohli, P., and Fawzi, A. Math- ematical discoveries from program search with large lan- guage models. Nat., 625(7995), 2024.
Snell, C., Lee, J. Xu, K., and Kumar, A. Scaling I1m test-time compute optimally can be more effective than scaling model parameters. ArXiv, abs/2408.00314 2024. URL https://aappp htt.lttgg///ppp.smmaiiccccooooot www.ntssscooooa..coooo org/ IorpuIDD:2111::0
Ross, S. and Bagnell, D. Efficient reductions for imitation learning. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, AIS- TATS 2010, Chia Laguna Resort, Sardinia, Italy, May 13-15, 2010, volume 9 of JMLR Proceedings, 2010.
Talmor, A., Tafjord, O., Clark, P., Goldberg, Y., and Berant, J. seap-of-thought Teaching pre-trained models to sys- tematically reason over implicit knowledge. In Advances in Neural Information Processing Systems 33: Annual Conference Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
Runco, M. A. and Jaeger, J. J. The standard definition of creativity. Creativity Research Journal, 24(1):92-96, 2012.
Sanford, C., Fatemi, B., Hall, E., Tsitsulin, A., Kazemi, S. M., Halcrow, J., Perozzi, B., and Mirrokni, V. Un- derstanding transformer reasoning capabilities via graph algorithms. abs/2405.1 18512, 2024.
Thankaraj. A., Jiang, Y., Kolter, J. Z., and Bisk, Y. Looking beyond the next token, 2025. URL https: //txxvarxvvv/// org/ hrggabbs25004 11336.
15

---

Going beyond the creative limits of next-token prediction
Tschannen, M., Kumar, M., Steiner, A., Zhai, X., Houlsby, N., and Beyer, L. Image captioners are scalable vision learners too. In Advances in Neural Information Pro- cessing Systems 36: Annual Conference on Neural In- formation Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10- 16, 2023, 2023.
language models with self-generated instructions. In Pro- ceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pp. 13484- 13508. Association for Computational Linguisties, 2023.
Wang, Y., Luo, X., Wei, F., Liu, Y., Zhu, Q., Zhang, X., Yang, Q. Xu, D., and Che, W. Make some noise: Unlock- ing language model parallel inference capability through noisy training. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, Miami, FI, USA, November 12-16, 2024, pp. 1291 14-1499.... Association for Computational Lin- guistics, 2024c.
Valmeekam, K., Marquez, M., and Kambhampati, S. Can large language models really improve by self-critiquing their own plans? arXiv preprint arXiv: 200.283 2023a.
Valmeekam, K., Marquez, M., Olmo, A., Sreedharan, S., and Kambhampati, S. Planbench: An extensible bench- mark for evaluating large language models on planning and reasoning about change, 2023b.
Watson, J.. Juergens, D., Bennett, N., Trippe, B., Yim. J., Eisenach, H., Ahern, W., Borst, A., Ragotte, R., Milles, L.. Wicky, B., Hanikel, N., Pellock. S., Courbet, A., Shef- fler, W., Wang, J.. Venkatesh, P., Sappington, I., Torres, S., Lauko, A., (De Bortoli), V., Mathieu, E.,, Ovehin- nikov, S., Barzilay, R., Jaakkola, T., DiMaio, F., Baek, M., and Baker, D. De novo design of protein structure and function with rfdiffusion. Nature, 620:1089-1100, 2023.
Valmeekam, K., Marquez, M., Sreedharan, S., and Kamb- hampati, S. On the planning abilities of large language models- A critical investigation. In Advances in Neu- ral Information Processing Systems 36: Annual Confer- ence Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10- 16, 2023, 2023c.
Varshney. R., Pinel, F., Varshney, K. R., Bhattachariya, D., Schörgendorfer, A., and Chee, Y. A big data approach to computational creativity: The curious case of chef watson. IBM J. Res. Dev., 63(1):7:1-7:118 2019.
Wei, J.. Wang, X., Schuurmans, D., Bosma, M., Ichter, B. Xia, F., Chi, E. H., Le, Q. V., and Zhou, D. Chain-of- thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, 2022.
Walsh, M., Preus, A., and Gronski, E. Does chatgpt have a poetic style? In Proceedings of the Computational Hu- manities Research Conference 2024, Aarhus, Denmark, December 4-6, 2024, volume 3834 of CEUR Workshop Proceedings, pp. 1201-1219. CEUR-WS.org.
Wies, N., Levine, Y., and Shashua, A. Sub-task decom- position enables learning in sequence to sequence tasks. In The Eleventh International Conference on Learning Representations, ICLR 2023, 2023.
Wang, H., Zhao, Y., Li, D., Wang. X., Liu, G., Lan, X., and Wang, H. Innovative thinking, infinite humor: Humor research of large language models through structured thought leaps. abs/2410.10314 2024a.
Wu, Y., Sun, Z., Li, S., Welleck, S., and Yang, Y. Infer- ence scaling laws: An empirical analysis of compute- optimal inference for problem-solving with language models. 2024. URL https: / /ttpppp..smaaiicccollla.tt htg/CoorpsID::2716000
Wang, H., Zou, J., Mozer, M., Goyal, A., Lamb, A., Zhang, L., Su, W. J., Deng, Z., Xie, M. Q., Brown, H., and Kawaguchi, K. Can ai be as creative as humans?, 2024b. URL https://aaaaee //tp:////vv.////////a/eeeeaaaav/ . httg/abs/24011.. 01623.
Xu, M., Jiang, G., Zhang, C., Zhu, S.-C., and Zhu, Y. In- teractive visual reasoning under uncertainty. In Neural Information Processing Systems, 2022. URL https:// // org/CorpusID: N.NDD::299899.. api. semantcccchooooarr wwt.s/schhoaaooooooorooorpllllla
Wang, J. X., King, M., Porcel, N., Kurth-Nelson, Z., Zhu, T., Deck, C., Choy, P., Cassin, M., Reynolds, M.. Song, F, Buttimore, G., Reichert, D. P., Rabinowitz, N. C., Matthey, L., Hassabis, D., Lerchner, A., and Botvinick, M. M. Alchemy: A benchmark and analysis toolkit for meta-reinforcemmnt learning agents. In NeurIPS Datasets and Benchmarks, 2021. URL https: / semanticschon
Yang, S., Gribovskaya, E., Kassner, N., Geva, M., and Riedel, S. Do large language models latently perform multi-hop reasoning? In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pp. 10210-10229, Association for Computational Linguistics, 2024a.
wttp/CorpuuID::2901992 239019925.
Wang, Y., Kordi, Y., Mishra, S., Liu, A., Smith, N. A., Khashabi, D., and Hajishirzi, H. Self-instruet: Aligning
16

---

Going beyond the creative limits of next-token prediction
Yang, S., Kassner, N., Gribovskaya, E., Riedel, S., and Geva, M. Do large language models perform latent multi-hop reasoning without exploiting shortcuts? abs/2411.16679 2024b. doi: 1044850/AARX1V2241116677 URL https: //doi. //doi.org//i. 10. 48550/arXiv. 2411 2461.16679.
2024. URL http://ppp..enntiiicccccoo..r.oo CorpusID:267094800
Zhong. S., Huang, Z., Gao, S., Wen, W., Lin, L., Zitnik, M., and Zhou, P. Let's think outside the box: Exploring leap- of-thought large language models with creative humor generation. In IEEF/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, 2024.
Yang, Z., Hu, Z., Salakhutdinov, R., and Berg- Kirkpatrick, T. Improved variational autoencoders for text modeling using dilated convolutions. ICML, 2017.
Zhu, Y., Lu, S., Zheng, L.., Guo, J., Zhang, W., Wang, J., and Yu, Y. Texygen: A benchmarking platform for text generation models. The 41 International ACM SIGIR Conference Research & Development in Information Retrieval, 2018
Yang, Z.,, Band, N., Li, S., Candès, E. J., and Hashimoto, T. Synthetic continued pretraining- CoRR, abs/2409.0744 2024c. doi: 1148500ABRXI.22400004753 URL https: //doi.org/10 48550/arXiv. 201.500/XX...229...000
Ye, J., Gao, J., Gong, S., Zheng. I.,, Jiang, X., Li, Z.,, and Kong, L. Beyond autoregression: Discrete diffusion for complex reasoning and planning. 2024. doi: 10.48550/
ARXIV.2410.14157.
Young, T. and You, Y. On the inconsistencies of condition- als learned by masked language models. arXiv preprint 2:Xiv:2301.00000 2022.
Yu, I.,, Jiang, W., Shi, H., YU, J. Liu, Z., Zhang, Y., Kwok, J., Li, Z., Weller, A., and Liu, W. Metamath: Bootstrap your own mathematical questions for large language models. The Twelfth International Confer- ence on Learning Representattions, 2024. URL https:
//openreview haa.lee..eeettfoommmmdnansooooe
Zhang, H., Li, L. H., Meng, T., Chang, K., and den Broeck, G. the paradox of learning to reason from data. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, IJCAI 2023, 19th- 25th August 2023, Macao, SAR, China, pp. 3365-3373. jjcai.org, 2023.
Zhang. J., Jain, L., Guo, Y., Chen, J., Zhou, K. L., Suresh, S., Wagenmaker, A., Sievert, S., Rogers, T. T., Jamieson, K., Mankoff, R., and Nowak, R. Humor in Al: massive scale coordesourced preferences and benchmarks for cartoon captioning. CoRR, abs/2406. 10522, 2024a.
Zhang, Y., Schwarzschild, A., Carlini, N., Kolter, Z., and Ippolito, D. Forcing diffuse distributions out of language models. abs/2404. 10859, 2024b.
Zhang, Y., Diddee, H., Holm, S.. Liu, H., Liu, X., Samuel, V., Wang, B., and Ippolito, D. Noveltybench: Evaluating language models for humanlike diversity. 2025. URL
set....smattccccollarr .rrg/// CorpusID:277621515
Zhao. Y., Zhang, R., Li, W., Huang, D., Guo. J., Peng, S., Hao, Y., Wen, Y., Hu, X., Du, Z., Guo, Q., Li, L., and Chen, Y. Assessing and understanding creativ- ity large language models. ArXiv, abs/2401.12491.
17

---

Going beyond the creative limits of next-token prediction
A. Limitations
We enumerate in detail the limitations of our work in terms of our experimental conclusions and in terms our general approach to an abstraction of creativity.
A.1. Limitations of our experimental conclusions
1. There may be many ways to improve upon next-token prediction for a minimal task. Unfortunately, success here does not necessarily guarantee success more complex tasks. Conversely, minimal tasks more valuable failure base case: failure here guarantees failure in more complex tasks.
2. Our examples do not preclude the existence of tasks where next-token prediction will outperform multi-token prediction; multi-token prediction is simply more ceneral-purposs objective suitable to lookahead tasks.
3. The teacherless multi-token prediction technique we explore as an alternative is generally harder to optimize than next-token prediction, especially for smaller models.
4. Even if multi-token approaches outperform next-token prediction relatively, in some of our simple tasks, all algorithms far from delivering a sufficiently diverse model.
5. Although our tasks are minimal, we note that there is a certain range of hyperparameters (e.g., high degree or edge count) beyond which the models struggle to learn them. We find that Triangle Discovery in particular is challenging task, especially for smaller models. We also note that the models are curiously sensitive to the way the edges are formatted (see $G.5).
6. While seed-conditioning appears to be competitive alternative to temperature sampling, there are cavcats here:
(a) In our GPT-2 (86M) experiments for Sibling Discovery and Triangle Discovery, the gap between seed- conditioning and samplia is contingent on top-K sampling (which is turned on by default in the library as it is standard practice in settings (and is immaterial for the other datasets which have a smaller vocabulary), it is not clear how crucial this factor would be for demonstrating the same gap in real settings.
applications). Although this does not seem to be requirement in our other
(b) It is not clear how well this idea may generalize to realistic data.
(c) Unfortunately, condicondii requires special training of the model, which is not as appealing as the far simpler temperature sampling.
7. There are many ways in which our observations have not been fully characterized understood even in minimal settings. For example, we do not thoroughly report on the effect of model size, model family, or pretraining; we do not analyze when and why seed-conditioning helps.
A.2. Our approach to creativity
Below, we enumerate some important limitations of our approach towards building abstract and minimal models of creative tasks.
1. The skills we capture in our tasks are only (a subset of) computational skills necessary for creativity; these are far from being sufficient.
2. The type of algorithmic tasks we study capture only a tiny subset of creative tasks that fall under the taxonomy in Boden (2003). There is yet another class called transformative creativity that we do not look at, and also other important taxonomies such as the Big- -llttllitee-- creativity (Csikszentmihalyi, 1996). Big-C Creativity corresponds breakthroughs and world-changing ideas; what we focus is adjacent to class of little-c creativity tasks. Relatedly, many real-world creative tasks appear to be
in nature, which we do not capture.
3. Real-world creative tasks also apply over much larger context length and require drawing connections from a signif- icantly larger memory (literally, the set of all things human may know about). Our algorithmic tasks are tiny in comparison (although deliberately so).
4. Our empirical measure of creativity for algorithmic tasks is only a computationally--fficcin proxy. Achieving an absolute high algorithmic creativity score does not imply a complete coverage of the space.
18

---

Going beyond the creative limits of next-token prediction
5. As stated earlier, we study abstract tasks that are inspired by the computations involved in creative tasks. Our study is not intended to capture the subjective aspects like interestingness (Schmidhuber, 2009) and other social, cultural and personal values integral to many creative tasks.
B. Transformer Training Objectives
Let LMg be our language model, parameterized by 8, for which LMg(3; = s;: $25) ) is the probability it assigns to the ith output 8; being 8t. given as input a sequence 8çi- Let (p,r) be a prefix-response pair. standard next-token finetuning, we maximize the objective:
Lewapp
Inext-tokan (0) = ED
log LMg -f:=1:0 (F=:r:P,re))
In teacherless (multi-token) training (Monea et al., 2023; Bachmann & Nagarajan, 2024; Tschannen et al., 2023), we make use of an uninformative input string $ that simply corresponds to series of dummy tokens
Droup
Imulti-tokon (0) ED
log LMo (f+ r;: $ci)
(3)
C. Further discussion
C.1. Intuition for seed-conditioning
Seed-conditioning produces non-trivial algorithmic creativity even with greedy decoding. Perhaps, one could view the seed prefixes as a simpler alternative to varying the wordings a prompt (Li et al., 2023; Lau et al., 2024; Naik et al., 2024; Li et al., 2022) or tuning a soft-prompt (Wang al., 2024c), both of which are known induce diversity. However, is unexpected that the seeds provide any meaningful randomness at all, given they arbitrarily picked with semantic relationship to the training point. We do not understand this presently.
Another pertinent question is to why seed-conditioning works better than temperature sampling in the settings it does. To this, we offer three potential explanations. The simplest explanation is that in the initial version of our paper, top-K sampling was enabled by default with K = 50 for the GPT-2 (86M) settings. This would have restricted the degree of freedom the support induced by the output-layer randomness; thus, providing different seeds can help vary this support and produce more varying samples. However, even upon removing top-K sampling, we see gains in our exploratory creativity tasks for GPT-2 (86M) (after all their vocabulary is smaller than K 50), and in all our Gemma v1 (2B) results. This is unexplained.
Thus, we speculatively put forth two more arguments. First, is a representational one. Fixing random seed upfront may help the model flesh out (i.c., compute the tokens of) one thought per sample, as against maintaining running set of multiple thoughts and computing a distribution over all their tokens at each step. A similar point made in a concurrent position paper (Jahrens& & Martinetz, 2025). The second argument is specific to next-token prediction on open-ended planning tasks: fixing random seed upfront may help the model co-ordinate multiple interlocking random decisions in advance rather than deciding them on the fly.
Finally, there are also optimization aspects of how seed-conditioning works that we do not understand (see SC.3). Regardless, it remains to be seen whether reed-conditioning is useful in tasks beyond the minimal ones we design.
C.2. Effects of reasoning-enhancing methods.
Our argument is limited to learning open-ended tasks in supervised manner which directly corresponds how perform pretraining or apply models for protein or drug discovery. While we do not comment how well other powerful approaches like RL (DeepSeck-AI, 2025), chain-of-thoughe (CoT, Wei et al., 2022), and scaling test-time compute (OpenAI, 2024) would address the limitations we bring up. we make few remarks. First, in cases where post-training only elicits pre-existing skills in the base model (Muennighoff et al., 2025), our limitations and suggestions about a next-token token-trained trained base model remain relevant. Next, regardless of the benefits one may get from pre-training, our arguments imply that
19

---

Going beyond the creative limits of next-token prediction
pre-training squanders its rich supervision, when it could be vastly compute- and data-efficient. Finally, there is profound question to whether exploration through sparse rewards learn the creative process during training-time or whether articulation in the token space (with a thinking model) can efficiently maximize diversity during test-time (which may require brute-force enumeration of all candidates).
C.3. Style of noise-injection
Our technique of injecting noise into the model is somewhat different from how noise is introduced in traditional VAEs (Kingma & Welling, 2014) or GANs (Goodfellow et al., 2020), and this difference is worth noticing. In traditional approaches, although the model learns a noise-output mapping, this mapping is enforced only at a distribution level i.e., the distribution of noise vectors must mäp to a distribution of real vectors. However, in our approach we arbitrarily enforce what noise vector goes to what real datapoint, at a pointwise level. This raises the open questions of why seed-conditioning works in the first place- - surprisingly, without breaking optimization or generalization and whether there is a way to enforce it at distribution-levell, and whether that can provide even greater improvements.
C.4. In-weights vs in-context graphs for combinational creativity
Combinational creativity requires searching through known entities. In abstracting this, there is an interesting choice to be made to whether the relevant search space is retrieved and spelled out in-context or whether it remains in-weights (like in Sibling Discovery and Triangle Discovery). We argue that the in-context version does not capture the creative skills required in many real-world tasks. For instance, discovering fresh and surprising analogy necessitates noticing similarities from sufficiently distinct parts of one's vast, rich space of memory. Thus, the core challenge here lies in retrieving from the entirety of one's memory. If one were to faithfully simulate this an in-context version of this in a model, one would have to provide the entirety of the model's pretraining data in context.
C.5. Examples of Triangle Discovery
Although we presented this task as a more complex, higher-order counterpart to Sibling Discovery, we retrospectively identify some real-world examples that resemble the higher-order search skill involved in this task.
1. Discovering contradictions: Consider identifying non-trivial contradictions within (a large body) of knowledge (like legal system, or proof based on many lemmas, or the literature spanning many papers in a certain field). This may require identifying two or more facts that together result in an implication that contradicts another fact.
2. Discovering feedback loops: Fields like biology, ecology, climate science and economics may involve discovering non-trivial feedback loops. Unlike feedback loops where two events encourage each other, a non-trivial loop would be one where an Event A encourages Event B, that in turn encourages Event that in turn encourages Event A.
3. Antanaclasis: An antanaclasis involves using word in two different senses in a sentence, while still ensuring that each sense has coherent relationship with the rest of the sentence. Consider Benjamin Franklin's quote, Your argument is sound, nothing but sound. Here, the two senses are sound; as in "logically correct", soundy as in "noise". This sentence encodes an pairwise relationship between three entities (argument, sound), individually. While the last two entities (the two senses) themselves must be related to each other (through the common word, sound), for a coherent sentence, both senses must also be appropriate descriptors for the first entity, argument. Thus, constructing this sentence requires searching through one's vocabulary to discover three words that satisfy these three relationships simultaneously.
4. Word games: Some word games require identifying a set of words that simultancously have pairwise relationships with each other.
(a) For example, standard crosswords would require identifying sets of 4 or more words that have various simultancous pairwise intersections in the letters used.
(b) Devising "& Lit." clues in cryptic crosswords are an altogether different, yet compelling example that require discovering a satisfying triangular relationship. Consider the clue "Some assassin in Japan" " whose answer is Ninja. Here the phrase Some assassin in Japan participates in two senses. First, the direct semantic sense as a definition of what Ninja is. But there is second, indirect
20

---

Going beyond the creative limits of next-token prediction
sense: the word Some indicates that the solution lies as some substring of the phrase, namely "as- sassi (n in Ja)p Thus, constructing the clue requires identifying triangular relationship between [Ninja, (Some assassin in Japan) (Some assassin in Japan) } just like in an antanaclasis. This is true generally of any & Lit. clues as these clues must perform "double duty" in pointing to the answer.
C.6. Further evidence of our argument in $2.6
Below we provide two more pieces of evidence affirming the failure mechanism of next-token prediction outlined in $2.6.
Improved algorithmic creativity is not due to some form of capacity control. While $2.6 argues that multi-token prediction should help creativity by providing critical lookahead capabilities, it is also possible that it simply acts as form of capacity control that prevents memorization. We rule this out in Fig 8: even as memorization computed on unseen seeds is controlled, the multi-token model perfectly reproduces the training data on seen seeds. We term this seed-mmemoriuttin. An exact equivalence of this phenomenon was noticed in GANs in Nagarajan et al. (2018), where the generator can be trained on specific latent vectors to memorize the mapping on those, and yet produce fresh samples outside of those latent vectors.
Training Obj ective:::
Standard (Next-Token)
Teacherless (Multi-Token)
Seed Memorization Memorization
1.0
0.5
0.0
50k 100k
Num Train Step
Figure 8. Even if multi-token prediction reduces memorization (on unseen seeds), it has enough capacity memorize training data on the seen seeds (denoted by meed-iimoizzation.. Note that the best algorithmic creativity for NTP and MTP are achieved at step 10k and 40k, respectively, which are the checkpoints we used to report metrics in Fig
Effeet of token reordering. The implication of our argument in $2.6 is that next-token learning would benefit from reversing the token ordering of the Sibling Discovery task (i.e., parent appears before siblings). Indeed, we find this to be the case in Fig 12 and Fig 22. Interestingly, we find that the reverse-traines NTP model is still far from the original multi-token teacherless model. More surprisingly, a teacherless model trained on the reversed data, achieves even higher algorithmic creativity of all training methods here. Note that in all other datasets, no reordering of the tokens should make any change to the training.
D. Description of datasets
D.I. Datasets inspired by combinational creativity
Dataset 1: Sibling Discovery (Fig 9a)). This task is based off bipartite graph G made of parent vertices =
(4, B, C.... each neighboring a corresponding set of children (br(A) = {a1, 42 Q2,..., }. We set the number of parent vertices [ to be small and the number of children for each parent vertex [nbr(A) to be large. For example, |V] = 5 and [nbr(A)] 500. We define coh(s) to hold on "sibling- paringg-paett" triplets of the form
(7, ,,, such that y, 7 E nbr(T).
Next, ensure that the training set is large enough for the model infer all the edges in the graph. Let m [|] and lbb (T) (for all I e V). This means S 2(m n). At the same time, to keep the task non-trivial, the training set must be small enough to not cover all the coherent sibling- parent triplets. Thus, we ensure S = o(m - n??).
For the default version of this dataset, we set [ = 5 and (T) = 500 for all I E V.
Dataset 2: Triangle Discovery (Fig 9b)). This task based off an undirected graph g = (V, E) which contains many triangles. Since a triangle is a symmetric structure, the problem remains the same even upon reordering the vertices. Thus, in this task coh(("), 02, vz)) true iff all three edges between {M1, 02, v3) belong in G. To make this task interesting
21

---

Going beyond the creative limits of next-token prediction
Generated samples
(A) "b, f, Y" (incoherent)
(B) "b,
(memorized (4))
(in (in-weights graph)
(C) "e, d,
(memorized (2))
(D) "e, d, x" (incoherent)
(E) "x, i, b" (incoherent)
Training data
(F) "b, c, x"
(g)",, f, Y"
(G) "e, Y"
(2) ee, d, Y"
(II "e, Y" (duplicated (G))
(3) "h, c, x"
(4) "b, i, x"
Algorithmic creativity = 2
(a) Sibling Discovery
Generated samples
(in (in-weights graph)
(A) "tri: ab, be, ca" (memorized ~
(1))
Training data
(B) "tri: ab, bf, fa"
(incoherent)
"tri: ab, be, ca"
(C) "tri: be, ca, ab" (memorized (1))
"edge: cd, de"
(D) "tri: ca, ab, be"
(memorized (1))
(3)
"edge: ab, ba"
(E) "tri: cd, de,
(4)
"edge: be, cb"
(F) "tri: de, ea, ad"
(duplicated (E)
(5) "edge: fb, bf"
"tri: da, ce, ed"
(duplicated (E)
(6) "edge: ac, ca"
(1) "tri: be, ce, eb"
(incoherent)
(7) "tri: da, ac, ad"
(T) "tri: de, ca, ad"
(memorized (7))
"edge: ce, ec"
(J) "tri: ac, ce, ea"
(incoherent)
"edge: ed, de"
(10) "edge: ad, da"
Algorithmic creativity
10
(b) Triangle Discovery
Figure 9. Minimal tasks inspired by combinational creativity:
The in-weights graph represents the underlying knowledge graph used generate the training data (not provided in-context). Based definition of algorithmic creativity in Eq. (1), generated samples that are incoherent or memorized, or duplicated are not counted as valid samples. Note that sequences that are permutations of each other are considered identical when computing duplicates and memorization.
(neither too trivial nor too non-trivial) for our models to learn, we enforce several constraints on the graph. First, we try to keep the degree deg of each vertex to be sufficiently small. On the one hand, this is so that no vertex requires too much computation to find a triangle it is part of; on the other, we also do not want very dense graph where most random triplets are a triangle. In addition to this degree requirement, we ensure that each vertex has a minimum number of triangles.
Thus to create graph that is neither too trivial nor too non-trivial, we define a two-step graph generation procedure. In the first step, iterate deg is small, such as 3 or 10). To avoid creating high-degree vertices inadvertently, we only select neighbors with degree < 1.2 deg. This alone may not ensure sufficient number triangles in each vertex; so we iterate over the vertices to explicitly create tri random triangles on each vertex (where tri is small, such as or 10), We do this by selecting pairs of a vertex's 's neighbors and drawing an edge between them.
the vertices, and add deg many edges from that vertex to other vertices in the set (where
Next, we want a training dataset such that (a) the model can infer all the edges from the graph and yet (b) not all triangles appear in the dataset. This necessitates training on dataset that consists not only of a subset of the triangles, but also of edges from the graph. Our training data consists of two parts: (1) 1/3 random triangles from the graph, (2) 2/3 random edges from the graph. In the training set, the triangle and edge samples are distinguished by prefix "triangle:" or ""dge:". During test-time, we ensure that the model is prompted with "triangle:". A triangle (u, v, w) is tokenized as "tri: (u, v), (v, w), (w, u)" and an edge (u, v) as "edge: (u, v), (v, u)". We provide both the directions of edge to potentially avoid any issues with the reversal curse (Berglund et al., 2024; Allen-Zhu & Li, 2023a).
For the default setting of the dataset, we set [V 999, deg = 3, tri
22

---

Going beyond the creative limits of next-token prediction
Vocab: a, b, c, d, e, f,
(constructed graph)
(constructed graph)
Training data
Generated samples
"c-,,,, e-c, b-a, E-e, a-g" (memorized (2))
(1) ee-b, d-a, b-d, a-,,
(B) "e-g, g-f, b-e, d-b, f-e" (incoherent)
(2) -, g,e, f. a f-a
" g-f, f-ce a-g, d-a"
(3) "a) b-c, c-d, d-a, e-a"
re-d, c-a, a-e, b-c, d-b" (duplicated (C)
(4) ",) 2-a, g-o, c,b, b.f"
4
Algorithmic creativity
(a) Circle Construction
Vocab: a, b, c, d. e, E,
Training data
(constructed graph)
Generated samples
(constructed graph)
(1) "caa, b-d, d.a, e.b"
(A) aac, E-g, b-a, e-E" (memorized (2))
(2) nd g,, dd, g"
3
(B) e-,, a-b, f-g, b-d" (incoherent)
1
(3) bb b-c d-d,
re b
2
(4) "gf, e-g, b-d, f.b"
"f-b, e-c, c-f, b-a" (duplicated (C))
Algorithmic creativity
(b) Line Construction
Figure 10. Tasks inspired by exploratory creativity: The constructed graph visualizes the graph induced by the training or generated sample. Edge indices represent the order of edge appearing in the string. Based definition of algorithmic creativity in Eq. (1), generated samples that are incoherent, memorized, or duplicated are not counted as valid samples. Note that sequences that correspond to the same permutations hut with different participating vertices are considered identical when computing duplicates and memorization
D.2. Datasets inspired by exploratory creativity
Dataset 3: Circle Construction (Fig 10(a)). In this task, the generated strings must be randomized adjacency lists that can rearranged to recover circle graphs of N vertices. The vertices come from a fixed vocabulary of M tokens. Specifically, let the generated list be s (V, viy), (Viy vi.).... We define coh(s) permutation such that (s)) (t), (22); (122, U..).... (1j. Uj for distinct j1 f1. In. i.e., each edge leads to the next, and eventually circles back to the first vertex. In our experiments, set M to be larger than N.
true iff there exists resolving
Our default experiments are reported for N = 9, M = 15.
Dataset 4: Line Construction (Fig 10(a)). This task is minor variant of the above where the edge set E corresponds to line graph. The details are same here except for coherence to hold, we need resolving permutation such that (s) (U),, 1i (1),, v..)..., (Uj..1: U),) for distinct ........v. i.c., each edge leads to the next, stopping at a end. We use the same set of hyperparamters as Circle Construction.
Our default experiments are reported for 9, M 15.
23

---

Going beyond the creative limits of next-token prediction
Table
Hyperparameter details Gemma
(2B) model.
Sibling Discovery
Triangle Discovery
Circle Construction
Line
Hyperparameter
Construction
Max. Learning Rate
5 x 10-4
5 x 10
10
5 x 10
Model Seq. Len.
32
32
2048
2048
Training steps
7500
10k
15k:
15k
Training size
50k
15k
10k
10k
Weight given to multi-token obj.
0.5
0.5
0.75
0.75
Table 2. Hyperparameter details GPT-2 (86M) and SEDD (90M) model.
Sibling Discovery
Triangle Discovery
Circle Construction
Line Construction
Hyperparameter
Max. Learning Rate
x 10-4
1 10-4
x 10
x 10
Model Seq. Len.
128
128
128
128
Max training steps (GPT-2 (86M)
20k
80k
40k
40k
Max training steps (SEDD (90M)
80k
80k
80k
80k
Training size
50k
15k
10k
10k
Weight given to multi-token obj.
0.5
0.33
0.75
0.75
E. Further experimental details
Details for Gemma v1 (2B) model. In Table 1, we provide the hyperparameter details for each of our datasets. We note some common details here. First, the batch size is 4, but each sequence is packed with multiple examples; thus the model sequence length (divided by the input length) can be treated as multiplicative factor that determines the effective batch size. The learning rates are chosen favorable to next-token prediction (not multi-token prediction). The training steps were chosen roughly based on a point after which the model had saturated algorithmic creativity (and exhibited decreasing creativity). We use a learning rate with linear warm up for 100 steps, followed by cosine annealing upto a factor 0.01 of the maximum dataset T of 1024 datapoints.
learning rate. To measure creativity, sample
We represent the main tokens in our tasks with integers (ranging upwards of 0 to many distinct integers as required), In the ceddcconiiiooing setting, we use seeds of default length 10, using randomly sampled uppercase characters from the English alphabet. In all datasets, we space-separate the vertices in a string, and comma-separate the edges.
Details for GPT-2 (86M) model. In Table 2, we provide the hyperparameter details for each of our datasets. We use GPT-2 (small) with 86M non-embedding parameters when we are comparing Transformers with diffusion models. We train these models with learning rate 10- and batch size of 64, to convergence in terms of the algorithmic creativity, In temperature sampling, results qualitatively. We used top-k sampling with parameter of 50. We also explore temperatures above 1.0 for these models since temperatures low than 1.0 only did worse. We provide sensitivity analysis of learning rate in $G.
nucleus sampling with parameter of 0.95, although observed this does not affect the
Details for SEDD (9OM) model. We use SEDD's "absorb" variant, which begins denoising with a fully masked sequence and iteratively refines tokens over 128 denoising steps. This variant achieves the best language modeling performance in the original paper. Same as GPT-2 (86M), we train these models with learning rate of 10-4 and a batch size of 64, to
24

---

Going beyond the creative limits of next-token prediction
convergence in terms of algorithmic creativity. We provide sensitive analysis of learning rate in &G. For sampling, we used the default function parameters, which turned out to involve top-K sampling with K 50.
use codehase Lou et al. (2023) at http:///.ahlllccoooeee////reeaea
25

---

Going beyond the creative limits of next-token prediction
F. Sensitivity analyses for Gemma v1 (2B)
In this section, we report that our observations are robust to the choice of various hyper-parameters. First, we present a series of plots for the Gemma v1 (2B) model; each group of plots reports varying one hyperparameter for all the datasets. Fig 11 for train set size, Fig 12 for task complexity, Fig 13 for the weight given the multi-token objective (and Fig 14 correspondingly for memorization), Fig 15 for learning rates, Fig 16 for number of training steps and Fig 17 for batch size. In SF.I, we report analyses for varying sampling conditions. It is worth noting that the occasional exceptions to our trends generally come from Line Construction, suggesting that this task is most friendly towards next-token prediction of the four we study.
Note on tosk-compl.xity. Fig 12, we report robustness of our results to variations in the task complexity (e-g., degree, path length etc... Note that the variations we have explored are within reasonable factors. If we vastly increase certain factors (e.g., increase the degree of the vertices), we expect learning to become either highly trivial or non-trivial (see $D for some reasoning). Besides, as discussed in the main paper, teacherless training is hard objective to optimize especially for smaller models; thus, we expect increasing the task complexity beyond a point to hurt the teacherless model for fixed model size (crucially, for optimization reasons, not generalization reasons).
Training
Standard
Teacherless (Multi-Token)
Objective:
(Next-
SibiingDyyyyvery TriangleDiscovery
LineConstruction
CircleConstruction
Creativity
1.00
0.4
0.8
Creativity
0.4
Creativity
Creativity
0.6
0.50
8.2
0.4
0.2
0.25
8.1
0.0
50000
200000
7500
15000
10000
200000
0.0
10000
50000
Num Train Data
Num Train Data
Num Train Data
Num Train Data
Figure Training size and algorithmic creativity for Gemma v1 (2B): Algorithmic creativity increases under multi multi-token prediction across various training set sizes. Note though that, in our examples, except the gap to diminish eventually with sufficiently many training datapoints (this unlike the failure of next-token prediction B&N)224).
SiblingDiscovery TriangleDiscovery
CircleConstruction 8.4
LineConstruction
Creativity
Creativity
1.0
Creativity
Creativity
1.0
0.5
0.2
0.5
0.0
0.0
parent1e parent5 parent5
vert500 vert999 deg10
0.0
child999 child500 child500
vocabl0 vorab15 Length9 length9
vocab vacab15
reverse
tril0
tri6
length9 length9
Task Complexity
Task Complexity
Task Complexity
Task Complexity
Figure 12. Task complexity and algorithmic creativity for Gemma v1 (2B): Algorithmic creativity increases under multi-token pre- diction across (reasonable) variations the dataset parameters (as described in $D).
F.1. Varying sampling methods
Fig 18, Fig 19, and Fig 20 report creativity, memorization and coherence (i.c., fraction of generated strings that are coherent) for various sampling methods (greedy decoding and nucleus sampling) with various prefix conditionings. Here not only look the baseline (denoted by null) and seed prefix (denoted by seed), we also look using prefix of pause tokens (Goyal et al., 2024) (denoted by pause). This can be viewed as way to provide extra computation to the model before it emits its outputs. If this had the same effect of seed reed-conditioning. then we would conclude that the gains of seed seed-conditioning come purely from prepending extra computation to the model - not from the act of injecting noise at the input. However, in our experiments, we do not find noticeable gains from pause-conditioning. establishing that noise- insseeiittiiinn is in itself beneficial.
26

---

Going beyond the creative limits of next-token prediction
Siblingbiscavery
Trivngge
Cireletonst
LineConstruction
1.0
Criativity
0.50
Creativity
Crestivity
0.5
0.25
0.5
0.2
0.00
0.0
0.0
0.0
8.75
0.25
Wt. of Teacherless MTP
Mt. of Teacherless MTP
wt. of Teacherless MTP
Wt. of Teacherless MIP
Figure 13.. Weight given to multi-token objective and algorithmic creativity for Gemma v1 (2B): Algorithmic creativity increases under multi-token prediction across various weights given to multi-token component of the objective, barring some deviations for Line Construction.
Sib Dingisiiiyy
TriangleBiscovee
CireleConstruution
LineConstruction
Menorization
Memo Menorizan 1wattatttoooooo
Memo werttiotta om
0.5
0.25
0.0
0.0
0.0
0.00
MTP 6. 8. 0.58 75 0.90
NTP 8.18 0.25 8.50
0.75
8.
MTP 0.18 0.25 0.58 8.
0.90
NTP
8.18
0.25
8.56 0.75
Wt. of Teacherless MTP
Wt. Teacherless HTP
Teacherless MTP
Wt. of Teacherless
Figure 14. Weight given to multi-token objective and memorization score for Gemma v1 (2B): Memorization reduces under multi- token prediction across various weights given to the multi-token component the objective.
SiblingDiscovery
TriangleDiscovery
0.6
1.00
Creativity
0.4
0.25
0.0
0.00
5e-3
1e-3
5
1e-4
5e-5
1e-5
5e
1e-3
Se-4
1e-4
5e-5
1e-5
Learning Rate
Learning Rate
CircleConstruction
LineConstruction
Creativity
8.6
Ereativity
8.8
0.6
8.4
0.2
0.2
0.0
0.0
5e-3
1e-3
5e-4
1e-4
5e-5
1e-5
5e-3
1e-3
Se-4
1e-4
5e-5
1e-5
Learning Rate
Learning Rate
Figure 15. Learning rate and algorithmic creativity for Gemma v1 (2B): Algorithmic creativity increases under multi-token prediction across various learning rates.
SiblingDiscovery
TriangleDiscovery
CircleConstruction
LineConstruction
1.00
0.8
Creativity
0.6
Creativity
Creativity
Creativity
8.75
0.4
0.2
0.4
0.2
8.25
0.2
0.0
0.00
0.0
7508
25880
10000
30000
15000
30000
15000
38000
Num Train Steps
Num Train Steps
Num Train Steps
Num
Train Steps
Figure Training steps algorithmic creativity for Gemma v1 (2B): Algorithmic creativity under multi-token prediction across lengths of training.
27

---

Going beyond the creative limits of next-token prediction
SiblingDiscovery
TriangleDiscoveer
CircleConstruction
LineConstruction
0.8
Creativity
0.4
Creativity
Creativity
0.4
Creativity
0.3
8.75
0.6
0.2
0.50
0.2
0.1
0.2
0.0
8.00
0.0
0.0
32
3072
3072
32
8192
2048
8192
2048
Model Seq Len
Model Seq Len
Model Seq Len
Model Seq Len
Figure 17. Batch size and algorithmic creativity for Gemma v1 (2B): Algorithmic creativity increases under multi-token prediction across various batch sizes. finetuning examples into the sequence.
here batch size effectively proportional to the model sequence length, since we pack multiple
SiblingDiscovery
Creativity
0.4
0.2
seed1e greedy
seed10
seed10 temp5.0 0
seed4 greedy
seed2 temp2.0
seed2 temp5.0
null
null
pause18 temp2.0
pause10 tempS.
temp2.8 0
temp2.0
temp5.0
Inference Type
TriangleDiscovery
1.0
Creativity
0.5
seed1e greedy
seed10
seed10 temp5.4 0
seed4 greedy
seed2 temp2.0 0
seed2 temp5. 8 temp2.0
pause10 temp2.0
pause10 temp5.0
temp2.0 0
temp5.0
Inference Type
CircleConstruction
8.4
Creativity
0.2
0.0
seed10 greedy
seed10
seed10 temp5.. .
seed4 greedy
seed2 temp2. e
seed2 temp5.6
null temp2.0
null temp5.0
pause10 temp2. 8
temp2.0 0
temp5.
Inference Type
LineConstruction
Creativity
0.75
0.25
0.00
seed10 greedy
seed10 temp2.0
sced10 temp5.
seed4 greedy
seed2 temp2. Inference Type
seed2 temp5.0
null temp2.8
null temp5..
pause10 temp2.0
pause10 temp5.
Figure 18. Algorithmic creativity under various sampling conditions for Gemma v1 (2B) Across all conditions, and in almost all datasets (with a few exceptions in Line Construction) multi-token prediction improves creativity. Furthermore, seed-ou conditioning achieves best algorithmic creativity, with longer seed helping more.
28

---

Going beyond the creative limits of next-token prediction
SiblingDiscovery
rizatizoo
seed16
sced10
seed10 temp5.0
seed4 greedy
seed2
seed2 temp5.
null temp2.0
null temp5.0
pause1e temp2.0
pause18 temp5.0
greedy
temp2.0
temp2.8
Inference Type
TriangleDiscovery
Menorization
0.75
0.50
0.25
0.00
seed10 greedy
seed10 temp2.0 0
seed10 temp5.0 8
seed4 greedy
seed2 temp2.0
seed2 temp5.0 0
null temp2.8 8
null temp5.0
pausel0 temp5. temp2.0
pause10
Inference Type
CircleConstruction
Nemorization
1.0
8.5
8.0
seed1e greedy
seed10
seed18 temp5.
seed4 greedy
seed2 temp2. 8
seed2 temp5.0 8
null
null temp5.0
pause1e temp2.0 0
pause10 temp5.0 0
temp2.0 0
temp2.0
Inference Type
LineConstruction
Memo rization
8.4
0.0
seed10 greedy
seed10 temp2.0
seed10 temp5. 0
seed4 greedy
seed2 temp2.
seed2
null temp2.
null temp5. e
pause10 temp2.
pause10 temp5.0
0
temp5.
Inference Type
Figure 19. Memorization under various sampling conditions for Gemma v1 (2B): Barring few conditions, the most prominent trend that memorization reduces under multi-token prediction for various sampling conditions. Observe that the null and pause- conditiond models do produce memorized output while their creativity was non-existent.
29

---

Going beyond the creative limits of next-token prediction
Siblingiiscovery
Coherence
1.0
8.5
8.0
seed1e g reedy
seed10 temp2.8 0
seed10 temp5.0 0
seed4 greedy
seed2 temp2.0
seed2 temp5.0
nulL temp2.0
nuLL temp5.0
pausele temp5. 0
temp2.8
Inference Type
TriangleDiscovery
0.8
seed10 greedy
seed10 temp2.0 9
seed10 temp5.0 0
seed4 greedy
seed2 temp2.0
seed2 temps.0
null temp2.0
null temp5.0
pausel0 temp5.
temp2.0
Inference Type
CircleConstruction
1.0
seedle greedy
seed1e temp2. 0
seed10 temp5. 0
seed4 greedy
seed2 temp2.0 0
seed2 temps.0 8 temp2.0
null temp5.
pause10 temp2.0
pausel0 temp5.
Inference Type
LineConstruction
Coherence
1.0
0.5
seed1e greedy
seed10 temp2.8
seed10 temp5. e
seed4 greedy
seed2 temp2.0
seed2 temp5.0
null temp2.0
null temp5.0
pause10 temp2.0
pause10
temp5.0
Inference Type
Figure 20, Coherence under various sampling conditions for Gemma v1 (2B): Surprisingly, coherence of all models is high or at least noticeable, across various sampling conditions. This suggests that the low algorithmic creativity of the null-conditioned models in the previous plots arises from model collapsing to single original point.
30

---

Going beyond the creative limits of next-token prediction
G. Additional experiments in SEDD (90M) vs. GPT-2 (86M)
G.I. Ablation studies
In this section, we first provide additional ablation studies for SEDD (90M) vs GPT-2 (86M) with different training and dataset settings (Fig and Fig 22).
Training Objective ive
Standard (Next-Token)
Di Llfffuiin-Abseee (Multi-Token
Teacherless (Multi-Token)
Di Mussion-Uniforo (Multi-Token)
Sibling
0.75
Creativity
0.50
0.25
0.00
5e-4
1e-4
5e-5
1e-5
Learning Rate
Triangle
Creativity
0.10
0.05
0.00
5e-4
1e-4
5e-5
1e-5
Learning Rate
Circle
0.6
Creativity
0.4
0.0
5e-4
1e-4
1e-5
Learning Rate
Line
Creativity
0.
0.50
0.25
0.00
5e- 4
1e-4
5e-5
1e: 5
Learning Rate
Figure 21. Learning rates and algorithmic creativity for the SEDD (9OM) model vs. GPT-2 (86M): MTP achieves higher algorithmic creativity than NTP when both are trained at their optimal learning rates. We use greedy decoding in this plot.
G.2. Effect seed length
We provide an ablation study on the seed length for NTP vs MTP on the Sibling Discovery task (Fig 23 (left)). We see that longer seeds lead higher algorithmic creativity.
31

---

Going beyond the creative limits of next-token prediction
Training Objective
Standard (Next-Token)
Diffusion IMulti-Token)
Sibling
Sibling
Creativity
Creativity
0.75
0.5
0.50
0.25
0.0
parent5 children500
parent5 children500
parent children1000
0.00
50000
400000
reverse
Num Train Data
Task Complexity
Triangle
Triangle
Creativity
0.10
Creativity
0.10
0.05
0.05
0.00
0.00
999
500
tri=6
tri=8 Triangle Density
tri=10
tri=25
Num Nodes
Circle
Creativity
0.5
0.0
N9M10 10000
N9M12 10000
N9M15
N9M10 5000
N9M12 5000
N9M15
10000
5000
Task Complexity
Num Train Data
Line
Creativity
0.5
0.0
N9M10
N9M12 10000
N9M15 10000
N9M18 5000
N9M12 5000
N9M15 5000
18000
Task Complexin Num Train Data
Figure 22. Task complexity and algorithmic creativity of SEDD (9OM) model vs. GPT-2 (86M): MTP consistently outperforms NTP
under varying task configurations, with some exceptions Line Constructionand Circle Constructionnatasets We use greedy decoding in this plot.
G.3. scedicoonitioning for diffusion training
In Fig 23 (right), we apply conditondii to the SEDD (90M) on the Sibling Discovery task. Unlike what we see on transformers, we not observe improved creativity when applying seed-conditioning diffusion training.
32

---

Going beyond the creative limits of next-token prediction
Sibling
Sibling
1.0
Creativity
1.0
Creativity
0.5
Training Objective:
Standard (Next-Token
Teacherless (Multi- Token)
seed
0.5
null
seed4
seed18
0.0
Seed
Len
Di f fusiio
Figure 23. GPT-2 (86M) Transformer achieves higher algorithmic creativity with longer seeds (top): We report algorithmic ativity with seeds of length and 10, with both NTP and teacherless MTP (with greedy decoding). We do not see gains of seed- conditioning when it comes to diffusion training (right): We report algorithmic creativity of the SEDD (90M) model with without seed-conditioning.
G.4. Effect of top-K sampling
Figure reports the algorithmic creativity scores NTP vs teacherless MTP training with seed-conditioning and top- sampling where K=50.
Sibling Discovery
Triangle Discovery
0.04
0.8
Creativity
Torpe rature
Creativity
Tenperature
greedy texpa.5
greedy
temp8.5
0.4
temp1.8
templ. 8
terp2.0
temp2.0
0.2
0.8
0.00
NTP top- k
Teacherless w/ tap-k w/ seed
NTP
Teacherless w/ top w/ seed
w/ top- k seed
seed
Circle Construction
Line Construction
0.3
0.4
Creativity
Tenperature greedy tempe
Creativity
Temperature
temp0.5 5
temp1.
templ. 8
0.1
temp2. B
temp2.8
8.1
0.0
Teacherless w/ top-k w/ seed
Teacherless top-k w/ seed
M/ top-k
w/ seed
w/ seed
Figure 24, Creativity scores for NTP vs teacherless MTP training with seed-conditioning + top-K sampling where = 50.
G.S. Format sensitivity for Triangle Discovery
Recall that our input format for Triangle Discovery follows the edge list representation triangles ($D. Fig. 10). For instance, triangle ABC is represented as AB, BC, CA. This format explicitly lists the edges of the triangle, making easier for the model to attend to edge-level patterns during leamning
We also experimented with an alternative node-based representation, where triangle ABC is represented more compactly as ABC, without making the edges explicit. We note in Fig 26 that the models are curiously sensitive to the way the triangles are formatted. Models trained on the node-basee based format perform equally badly with all training objectives, while the diffusion model outperforms by large margin with the edge list representation.
33

---

Going beyond the creative limits of next-token prediction
Sibling Discovery
Triangle Discovery
0.20
0.8
Masorization
0.15
Temperature greedy
Mcnorizati
8.6
Temperature
greedy
tempa.5
tempa.5
0.10
terp1.0
temp1.8
temp2.0
temp2 0
0.85
0.2
TeacherLass
Teacherless
top-k
top-
Circle Construction
Line Construction
0.10
0.015
Monorization
0.08
Temperature greedy terp0.5
Menorization
Temperature greedy
terp8. 5
0.84
temp1.0
terp2.0
0.085
temp2.
0.82
0.80
0.080
NTP
Teacher lass
NTP
Teacherless w/ top-k w/ seed
w/ top-k
top-k w/ Seed
w/ top - w/ Seed
w/ seed
Figure 25. Memorization scores for NTP vs teacherless MTP training with seed-conditioning + top-K sampling where K = 50.
Training Objective
Standard (Next- Token)
DLffusion-ABel -bsooon-bbbbbbbbooooo (Multi-Token)
Teacherless (Multi-Token)
Diffusion-Unie forr (Multi- Token)
Triangle
Creativity
0.
0.05
0.00
Edge rep
Edge rep 500
Node rep 999
rep
999
500
Format / Nodes
Figure 26. Sensitivity to formatting of the sequence in Triangle Discovery: We find that all small models perform equally poorly with node-wise representation of the input sequence, whereas there was stark difference in performance with the edge-wise representation. We use greedy decoding this plot.
G.6. Additional ablations
Figure 28 reports the ablations GPT-2 (86M) for (1) NTP vs MTP, (2) effect of seed conditioning, (3) effect of top-K, and (4) effect of temperature sampling.
H. Additional experiments with medium-sized Transformer and SEDD
We replicate our SEDD (9OM) and GPT-2 (86M) experiments on larger model size (~400M parameters). Fig 29, we similar trends to the smaller model sizes (Fig 4).
Decomposing creativity
Through following experiments on the GPT-2 (86M) model in the Sibling Discovery task, we try to understand the dynamics between two important quantities that affect algorithmic creativity: diversity/duplication and originality/memo- rization.
34

---

Going beyond the creative limits of next-token prediction
Sibling Discovery
trestivity
w/ seed Best w/o seed
0.4
8.0
NTP
MTP w/o t0p
Teacherless N/ top-k
Teacherless w/o top
w/ top
Triangle Discovery
Creativity
w/ seed Best w/o seed
MTP w/ top- k
NTP we top-k
Teacherless W/ top-k
Teacherless we top-
Circle Construction
Crastivity
8.2
Best w/ seed Best seed
NTP
Teacherless
Teacher w/o
w/ Top-k
w/o
w/
Line Construction
8.4
Crestivity
8.3
Bost w/ seed
0.2
0.1
Teacherless
baachrrless top-k
Figure 27. Seed conditioning improves creativity. "Best with seed" and "best without seed" stand for the best creativity score in each setting after we tune sampling temperature from (0, 0,5,
L.1. Diversity score
Equation (1) defines our algorithmic creativity by rewarding samples that are both unique and novel. A higher score can be achieved either by enhancing diversity or by reducing memorization. In the following section, we examine this decomposition using the Sibling Discovery task. Formally, we define the diversity score as:
dvN uniq((s (s E T|coh(s)))
(4)
[
We first demonstrate that creativity and diversity not necessarily correlated, and next, that MTP particularly improves creativity (while achieving lower diversity than NTP). To show this, report the algorithmic creativity and diversity scores along training in Fig 30. We see that for NTP, the diversity score keeps increasing and stays high, while algorithmic creativity increases in the first 10k steps and starts to decrease. For teacherless training, both scores increase throughout training. While the diversity MTP is surprisingly lower than NTP throughout training, the creativity of MTP surpasses NTP at 20k steps.
35

---

Going beyond the creative limits of next-token prediction
Sibting Discovery
Creatrvtty
Terpersture
grwdy
Leepl
ceep1.0
Cemp2.0
0.7
0.0
NTP seed
VTP
Teacheriess
Teacterless
Teachortess w/ seal
Teacheriess seed
uead
of seed
Triangle scovvryyy
9.
Creativity
Terpersture
0.43
greedy
tesph.1
9.82
temp1.0
temp2.
0.01
MTP
NTP
NTP who tap-k
NTP win top.
Teacherless tapak
Teachertess top-a
Teacherless top-k
Teacherless wit top-k
</ top-k
Construction
0.3
Torpersture proody
Creatvity
0.2
twap1.0
(wmp2.00
0.0
top
top
w/o top w/ soed
we top K w/o seed
w top w/ seed
top we seed
w/o top k N/ sood
top we seed
W/ Seed
we sood
Line Construction
Crattivity
Terperature
preedy
teepo.
teep1.0 temp2.0
0.0
Teacherless w/o sead
tap-k W/ seed
top-k w/o seed
Lop
tap-k we seed
top-k of seed
we tap-k w/ seul
tup-k w/o seed
w/ seed
Figure 28. Ablations GPT-2 (86M) for NTP teacherless NTP training, (2) effect seed conditioning, effect of top-K,
and (4) effect of temperature sampling
1.2. Decomposing algorithmic creativity as diversity and memorization
Better creativity can be achieved either by enhancing diversity or by reducing memorization we try to disentangle these factors this section. Fig 31, we plot the algorithmic creativity, diversity, and memorization scores at the checkpoint of best algorithmic creativity. We see that cedd-condiiioning contributes to higher diversity does little bring down memorization; however, teacherless MTP training contributes higher diversity and also reducing memorization. In Fig 30, see that the best creativity and best diversity achieved the same checkpoint.
I.3. Data scaling for algorithmic creativity
How does algorithmic creativity change as we increase the amount of training data? Intuitively, more training data helps the model learn the true distribution, but also makes harder generate unseen samples (since the uncovered space becomes rarer). To understand this, plot how models perform relative theoretically expected maximum algorithmic creativity. This is computed by assuming an oracle that samples generated set T (in Eq. (1)) uniformly with replacement from the true underlying distribution, and then computing algorithmic creativity Eq. (1. In Fig 32, we see that as we increase the training data (for a fixed underlying graph), the theoretically expected creativity decreases as expected, while the theoretically expected diversity stays the same (since this quantity does not care about being original with respect to the training set). Interestingly, as training data increases, MTP narrows the gap between NTP and the theoretically expected creativity and
36

---

Going beyond the creative limits of next-token prediction
Training Objective
Standard (Next-Token)
Dif Eaffuiio-Absonn (Multi- Token)
Teacherless (Multi- Token)
Diffusion Uninn (Multi-Tode
Creativity
1.0
1.0
0.050
0.50
0.5
0.5
0.025
0.25
0.0
0.000
0.00
0.0
Sibling Discovery
Triangle Le Discovery Memorization
Circle Construction
Line Construction
0.2
0.02
0.2
0.5
0.1
0.1
0.01
0.0
0.0 Triang le Discovery
0.0 0
0.00
Sibling Discovery
Circle Construction
Line Construction
Figure 29. On medium-sized (~-400M model, multi-token diffusion training improves algorithmic creativity from Eq 1 (top) on our four open-ended algorithmic tasks. We use greedy decoding in this plot.
Creativity Diversity
1.0
Training Objective:
Standard (Next-Token)
Teacherless (Multi-Token)
0.5
0.0
10k
20k
30k
40k
Num
Train
Step
Figure Algorithmic ereativity diversity are not necessarily correlated, exhibiting distinct dynamics: We find that a high diversity score through training, even higher than MTP. However, its algorithmic creativity reaches only mediocre peak before descending, when MTP starts surpassing it. We use greedy decoding this plot.
almost achieves the theoretically expected performance in the high data regime.
J. Experiments on summarization
Experimental Details. Table we provide the hyperparameter details for the GPT models finetuned on both XSUM (Narayan al., 2018) and CNN Mul//ailyMail (Nallapati et al., 2016) for one epoch. We use learning rate with linear warm up for 0.05 of the total steps, followed by linear decay to 0. To measure Rouge and Self-Bleu, generate and average across 5 summarizations per document, on test dataset 7 of 250 datapoints. We finetune our models with either the NTP objective (Eq 2) or the teacherless MTP objective (Eq 3), with equal weight to both.
To measure quality, we compute the average of Rouge-1, Rouge-2, Rouge-L as Rouge. For measuring diversity, we generate five different summaries per test example, and compute Self-Bleu. This computes average pairwise sentence Bleu-2 scores with weights (0.5, 0.5, 0, 0) on 1- and 2-tuples.
37

---

Going beyond the creative limits of next-token prediction
Sibling
1.0
1.0
0.6
Creativity
0.8
Diversity
0.8
Memorizate iiatiioioo
0.5
0.4
Training Objective
Standard
0.6
0.6
(Next-Token)
0.3
Teacherlese
0.4
0.4
0.2
(Multi Token)
0.2
0.2
0.1
0.0
seed4
Seed10 50000
seed4 408060
seed4
seed10
seed4 400000
seed4 50000
seed18
seed4
50000
50000
408006
Data
Sibling
1.0
1.0
0.20
0.8
0.8
Creativity
Diversity
Meno rization
0.15
0.6
0.6
Prefix
0.10
null
0.4
0.4
seed
0.05
0.2
0.2
0.0
0.0
0.80
NTP
NTP
NTP
NTP
NTP
NTP
NTP
NTP
NTP temp2.6
temp8.
temp1.0
temp2.0
tempê. temp1.0 temp2.0 e
temp0.5
temp1.0
Objective temperature
Figure 31. Decomposition of algorithmic creativity for GPT-2 (86M) in Sibling Discovery: We report algorithmic creativity, diversity and memorization the checkpoint of best algorithmic creativity. We see that seed-conditioning contributes higher diversity but does not help bring down memorization; teacherless training helps both diversity and in bringing down memorization.
Sibling
Sibling
1.0
1.0
Diversity
Creativity
0.8
Standard (Next Tooken)
0.9
Teacherless
0.6
(Multi-
Theoretically
0.4
Expecter ted
0.8
1x
2x
4x
8x
16x
32x
1x
2x
4x
8x
16x
32x
Num
Train
Data
Num
Train Data
Figure 32, Data scaling curve for algorithmic creativity and diversity: As we increase the training data (for a fixed underlying graph). the theoretically expected maximum algorithmic creativity decreases as expected, while the theoretically expected maximum diversity stays the same. NTP tails achieve the theoretically expected algorithmic creativity, while MTP almost achieves the theoretically expected performance at scale.
J. Additional graphs for effect of multi-token training
Fig 33 shows the diversity and quality graphs on the smaller-sized GPT-2 models on XSUM, and Fig 34 for CNN/DailyMail While we consistently see improved quality from the multi-token model across the board, we don't see an increased diversity for fixed Rouge scores anymore.
38

---

Going beyond the creative limits of next-token prediction
Table 3. Hyperparameter details for summarization experiments.
Hyperparameter
XSUM
CNN CNN/DailyMail
Batch Size
32
32
Max. Learning Rate
10-3
10-6
Warmup Steps
338
124
Training Steps
7778
2486
Training Size
248906
79552
Training Obj ective:
Standard (Next-Token)
Teacherless (Multi-Token)
Self- -LEE))
XSUM- MED-UM
Self Self-BLE (BBEE))
XSUM-GPM --T-GPT--MMM SMALL
0.9675
0.980
8.9659
0.955
0.9625
8,9608
Diversity (1
Diversity
0.945
8.955
0.9525
0.948
6.13259. 13580.
13750 14800, 14258. 14588. 147.9.47.....0 1508 Quality (ROUGE)
0.128
8.130
0.134
0.138
Quality (ROUGE)
Figure 33. Multi-Token Objective has no effect on diversity for smaller CPT models on XSUM.
Training Objective:
Standard (oket-Token)
Teacherless MMulti-Token
CNN-GPT-XX -XL
CNN-GPT- -AAGE
Siversity (BLEU)
Diversity (1 (ALf-BLEU)
0.994
0.062
0,944
3,40
0.240
3.340
0.242
1:444
3.24
0.268
0.364
0.148
2.140
5.1.0
6.152
6.254
Qualiiy (ROUGE)
Quality (ROUGE)
CNN-GPT MEDIIUM
CNN-GPT GAT-GPPTSLSLLLL
662
0.606
Sel.
8.300
0.500
0.602
Diversity
Diversity
0,962
0.406
6.132
6.154
0.228
6.242
6.244
5.228
0.328
0-132
1.138 0.
Quality (ROUGE)
Quality (ROUGE)
Figure Multi-Token Objective increases diversity for GPT-L and GPT-X for GPT-XL GPT-S CNN/DailyMail
39

---

Going beyond the creative limits of next-token prediction
J.2. Effect of seed-conditioning
We also conducted seed-conditioning experiments as described in $3.1. The seed strings we use are 10 randomly sampled uppercase characters from the English alphabet. We report the quality plots in Fig 35 (for next-token prediction XSUM) and Fig 36 (for multi-token prediction XSUM). As such, do not find any changes in diversity, perhaps because this is not sufficiently open-ended task.
Training Objective:
Null (Next-Token)
Seed (Next Token)
SSUM-GPT-X XL
Self-BLEUU
XSUM OPT-LARGE
SeLf-BLEUD
0,950
0,365
00,05
9,25
09.96
Diversity
Diversity
0.24
3.254
0.128
8.135
0:208
122 ROUGE
0.344
0,36
0.343
0.132
9.354
0.55
8.138
0.342 0.34 0.14 ROUGE
3:58
XSUM XSUM-GPT-T MEDIIM
XSUM-GPT SMALL GPMLLLALL
Diverssty
6.555
0.354
0.130
0.330
336
0.190
6.342
12559,166,, 127 2200 1298 100 131 3020.33
ROUGE
ROUGE
Figure 35. Seed-conditioning has no effect diversity for GPT models on XSUM summarization with next-token prediction.
Training Objective:
Null (Multi-Token)
Seed (Multi Token)
Self BLEU)
XSUM- GPT--PPT-P XL
Diversity Self-Bu -BLEU)
XSUM GPT-MGPP-LLAA -LAGG
0,
0,961
0.000
0.220
0.925
0.445
0.006
Diversity
0.945
0.35
0.044
0,430
0.905
6.24200 1i00 24150
2500 1515
Losee 15153 2003
0.248
344
2.152 154 ROUGE
169 348
ROUGE
RSSUMGPPT- MEDIUM
XSUM- GPT SMALL GPM-LLALL
6.550
Diversity
Daveres
6.554
0.255 8:258
8:248
8.240 6:144 6:36 6,29 0.138
0.138
0.332
3:334 ROUGE
7.135
8:128
ROUGE
Figure Seed-conditioning has no effect diversity for GPT models on XSUM summarization with multi-token prediction.
40

---

Going beyond the creative limits of next-token prediction
K. More related works
Empirical studies of creativity in LLMs. There is a long line of recent works that measure novelty and creativity of LLMs and LLM-assisted users. (Chakrabarty et al., 2024; Lu et al., 2024b) quantitatively evaluate and report that models vastly underperform under expert human evaluation against human writers. Zhang et al. (2024a) argue that finetuning methods such as RLHF and DPO, are limited when applied to creative GPT4 and Claude currently underperform top human contestants in generating humorous captions. In poetry, Walsh et al. argue that there are certain characterstic styles that ChatGPT restricts itself to. Even assisted-writing can reduce diversity (Padmakumar& & He, 2024) or produce bland writing (Mirowski et al., 2024). On the positive side, Si et al. (2024) report that LLMs surprisingly generate novel research ideas, although these are less feasible. Anderson et al. (2024) find that users tend to produce more divergent ideas when assisted by ChatGPT (although at a group level, ideas tend to homogenize). Another line of works (Wang al., 2024a; Talmor et al., 2020; Zhong et al., 2024) has proposed algorithmic improvements involve creative leaps-of of --thuught for real-world tasks.
tasks. Likewise models like
Other studies have proposed benchmarks for evaluating creativity. AidanBench (MeLaughlin et al., 2024) and NoveltyBench (Zhang et al., 2025) evaluate LMs on their ability to produce diverse and coherent responses by penalizing repetition across generations. However, they do not measure originality relative to training data, leaving open whether outputs are genuinely novel or simply unseen paraphrasss/rcomminatioo of Creative Thinking, a standard in human psychometrics. Another line of work such as Alchemy (Wang et al., 2021), IVRE (Xu et al., 2022), and Discovery World (Jansen et al., 2024) present simulations with hidden facts and rules, requiring LMs to explore, hypothesize, and test through interaction. While these simulations focus on pretrained models rather than examining how training shapes creative capabilities, they serve as valuable and realistic benchmarks for assessing the role of creativity in scientific discovery.
Zhao et al. (2024) evaluate LM creativity using the Torrance Tests
Finally, we refer the reader to Franceschelli & Musolesi (2023) for a rigorous treatment of philosophical questions surrounding creativity in LLMs. We also a refer to Wang et al. (2024b) for a theoretical treatment of how to formalize subjectivity in creativity.
The next-token prediction debate. In support of next-token prediction, there are arguments (Shannon, 1948; 1951; Alabdulmohsin et al., 2024) that claim that language is captured by NTP with models even superceding humans (Shlegeris et al., 2022) at NTP. There are also theoretical results emphasizing the expressivity (Merrill & Sabharwal, 2024; Feng et al., 2023) and learnability (Malach, 2023; Wies et al., 2023) of autoregressive Transformers as long as there is sufficiently long chain of thought.
Multi-token training. While multi-token methods employ diverse strategies, a common feature is their reliance on objec- tives that capture dependencies across entire sequences. Representative examples include teacherless training (Bachmann & Nagarajan, 2024; Monea et al., 2023; Tschannen et al., 2023) and independent output heads or modules (Gloeckle et al., 2024; DeepScek-AI et al., 2024) or inserting lookahead attention (Du al., 2023). Another line of research is discrete diffusion models (Hoogeboom et al., 2021; Austin al., 2021; Gong al., 2023; Lou al., 2023), which avoid strict left-to-right factorization by iteratively refining entire sequence at multiple positions. There are other models as well, such as eneryy-based models (Dawid & LeCun, 2023) and non- -oo--utooreerssvvee models or (Gu et al., 2018). Frydenlund (2024) report gains from non-autoregressive models on the path-star task (Bachmann & Nagarajan, 2024), while Hu et al. (2025) propose a novel method call the Belief State Transformer that shows gains on these tasks. While we only report the effect of teacherless training on our tasks due to its simplicity, we speculate that these methods would show similar (or better) results as teacherless training in our settings.
Transformers and graph algorithmic tasks. Graph tasks have been used to understand various limitations of Transform- ers in orthogonal settings. Bachmann & Nagarajan (2024); Saparov et al. (2024) report that Transformers are limited in terms of learning to search tasks on graphs, while Sanford et al. (2024) provide positive expressivity results for a range of algorithmic tasks that process an graph. These works differ from our study of combinational creativity since their graphs are provided in-context and the tasks have unique answer. Other works (Schnitzler et al.; Yang et al., 2024a;b) study multi-hop question answering on a knowledge graph; however, this does not require planning.
Diversity of generative models. One line of work relevant to us in the history of generative models is RNN-based VAE for text data (Bowman et al., 2016). The motivation, like in our work, was to learn high-level semantic features rather than

---

Going beyond the creative limits of next-token prediction
next-token features with the hope of producing more novel sentences. However, this suffered from posterior collapse, where the model ignores the latent variable altogether inspiring various solutions (Yang et al., 2017; Goyal et al., 2017). Our results on seed-conditioning are also reminiscent of a line of work exploration in reinforcement learning (RL), where it has been shown that adding noise to the policy model parameters enables more efficient exploration than directly adding noises to the output space (Plappert et al., 2017; Fortunato et al., 2017).
Learning- studies of diversity in LLMs. Various theoretical works provide rigorous arguments for how pre- venting hallucination and maximizing the model's coverage are at odds with each other abstract settings (Kalai & Vempala, 2024; Kalavasis et al., 2024; Kleinberg & Mullainathan, 2024). We clarify that this tension does not apply in our concrete settings. In those abstract settings, the strings in the support can be arbitrary and adversarially chosen whereas, our strings generated by simple rule (which can be learned).
Another theoretical question underlying generative models is that the optimum of their objectives are attained at perfect memorization; yet they tend to produce novel examples e.g., this question has been posed for GANs in Nagarajan et al. (2018) and for diffusion in Nakkiran et al. (2024) (see "remarks on generalization") or Kamb & Ganguli (2024). Okawa al. (2023); Kamb Ganguli (2024) study the creativity ability image diffusion models combining various concepts create an image. Unlike our setting, these do not require planning.
Injecting input noise. We note that the concept of injecting input noise into Transformer has been explored before, but for other functions (such quality, robustness (Hua et al., 2022; Jain al., 2024) efficiency (Wang et al., 2024c).), and in other forms (e.g., inducing Gaussian noise).
42


Reviews:
[{"review_id": "3AQuDftRFc", "reviewer": "Reviewer_Lrgc", "summary": "In this work, the authors aim to study the failure of the next-token prediction (NTP) objective at open-ended creative tasks, where the goal is to generate a diversity of outputs satisfying some constraint. Motivated by the fact that many such tasks require learning to form a latent plan that is not captured from learning the distribution one token at a time, they hypothesize that multi-token prediction (MTP) objectives would be more suited to open-ended learning.\n\nMotivated by work in the creativity cognitive science literature, the authors design a set of 4 algorithmic tasks whose distribution they aim to model. They define the creativity score as the number of unique non-memorized outputs that satisfy the task-dependent constraint.\n\nThe authors consider 2 MTP objectives from the literature for evaluation against NTP on these tasks: teacherless training and discrete diffusion. They consider 2 sets of models: Gemma 2b v1 models for NTP vs teacherless training, and an additional ~90m parameter setup for NTP vs Diffusion (and teacherless).\n\nThe authors find that across tasks and models, NTP leads to low creativity and high memorization, whereas MTP objectives consistently exhibit higher creativity score. The authors additionally find that a novel training method (hash-conditioning), wherein a model is trained with the hash of the sequence it is supposed to model as a prefix and at inference conditioned on a novel hash, consistently leads to higher creativity, even in the NTP setting.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed and encouraging feedback. We are happy that you find our approach elegant and timely given how it is near-impossible to rigorously measure creativity in the real-world. We are also pleased that you find (a) our proposed hash-conditioning approach interesting (b) our claims well-supported, the model sizes reasonable and (c) our related work discussion extensive.\n\n### Reg. CoT, RL etc.,\n\n> A second counterargument: using chains of thought… to capture the latent plan needed to produce creative outputs\n\nWe _completely_ agree with your counterargument (and glad you acknowledge this to be out of scope of this paper!).\nNevertheless, there are some profound doubts that arise when considering CoT in creative planning tasks which differ from standard reasoning tasks:\n\n1. **Current paradigms like CoT, prompting, RL, instruct-tuning -- which are evaluated on math/coding tasks -- are purely designed for one goal: *coherence/correctness of a single, generated plan.*** None are explicitly designed for (i) diversity across various generations and (ii) originality of a generation compared against the training set. Importantly, it is highly unclear how to redesign these paradigms (the rewards, the prompts etc.,) to optimize originality or diversity! \n   - Arguably, the “SFT on iid samples from a distribution” setup — despite being less fancy — seems to be the most natural way to teach a model to produce diverse & original outputs. \n2. **Human-generated data rarely contains the explicit, step-by-step trace behind creative thought.** For example, authors rarely document their internal thought processes while conceptualizing creative works like a research paper or a clever olympiad problem. **Given that the creative process is highly complex compared to standard CoT use-cases,** (see below!) to what extent can a model produce such traces with originality & diversity? \n3. Even if one may hope such CoT emerges, **fundamentally, it is unclear if CoT traces even exist for many creative tasks.** \n   - For instance, in the triangle discovery task, is there a CoT-style approach for maximizing creativity? Would that involve laboriously enumerating all triplets in the graph as CoT and filtering them out? Is such an approach even scalable in real-world tasks? What would the CoT trace for generating all possible original, and diverse completions for “a horse walked into the bar” look like?\n   - In contrast, we speculate that in many real-world tasks, creative _leaps_-of-thought come from \"quick, internal, latent heuristic leaps\" that do _not_ occur in token space. (Our non-CoT setup seems better suited for modeling this.)\n\nEvidently, these are profound questions that future research must tackle.\n\n---\n\n> The larger-scale tasks might require more explanation and exposition (why Self-Bleu as an evaluation metric?) but unfortunately space is lacking. \n\nWe will improve this discussion!  We choose Self-BLEU as it’s widely used as a diversity metric in language generation [1]. It measures BLEU scores across outputs generated by the same model to measure distinctiveness among outputs.  \n\nA full fledged future investigation would certainly strengthen our findings.\n\n[1] Texygen: A Benchmarking Platform for Text Generation Models\n\n> The result on hash-conditioning is very interesting and should be validated with larger-scale empirical work; how usable is it in practice?\n\nAbsolutely — validating hash-conditioning at scale would have  great impact! **Currently, the roadblock is large-scale evaluation: we need _open-ended_ benchmarks where diversity & originality can be precisely measured alongside correctness/coherence.** While there are evaluations of creativity like in [2] [3] [4] [5], unfortunately, these either use debatable proxies (e.g., n-grams, sentence embeddings) or only measure the quality of a standalone generation without grounding to the training data - none of them are accurately evaluating originality and creativity.\n\nAs for the algorithm itself, there are obvious ways to extend hash-conditioning to practice e.g., allocate a reserve of “hash-tokens”, and pretrain/finetune with such hash-tokens inserted in the model. We hope that our paper can inspire an active exploration of such ideas & also settle the benchmark challenge above.\n\n[2] Hu et al., The Belief State Transformer\n\n[3] Lu et al., Quantifying linguistic creativity of language models via systematic attri-bution of machine text against web text\n\n[4] Peeperkorn et al., Is temperature the creativity parameter of large language models?\n\n[5] Si et al., Can LLMs generate novel research ideas?  A large-scale human study with 100+NLP researchers\n\n> Fig 5 Temperature 2.0 might be too much for Gemma, did you try lower temperatures (1.0)?\n\nGood point; we in fact started with lower temperatures which only performs equivalently if not worse in terms of creativity, which is why we moved to exploring larger temperatures. We will make sure to add.", "claims_and_evidence": "All claims in the paper are supported by adequate evidence.", "methods_and_evaluation_criteria": "The paper uses a set of simple algorithmic tasks to test for the failure cases of the NTP objective. While I am usually quite wary of toy tasks, here its actually quite useful to have a simplified setting. This allows the authors to measure creativity where, in natural language scenarios, this would be challenging (exact uniqueness would not work because of superficial difference, string distance would not capture conceptual differences, and metrics based on embedding distances are less interpretable and sensitive to choice of representation). \n\nI mostly agree with the authors that this task is a good failure test of NTP prediction: models failing on these tasks at a small scale can reasonably be expected to fail at larger scale. A counterargument might be that large-scale models exhibit more capabilities than small-scale ones, but the experiments at 2b scale with pretrained models somewhat mitigate this concern. A second counterargument might be the existence of chain-of-thought capabilities in large models: using such chains of thought the model might be able to capture the latent plan needed to produce creative outputs. This is shortly discussed in the appendix of the paper and warrants its own followup investigation, nevertheless I think it is outside the scope of the current paper.", "theoretical_claims": "No theoretical proofs in this paper.", "experimental_designs_or_analyses": "The models used were reasonable, and all experiments with the 2b model were performed 4 times with variation plotted in the charts. The appendix contains detailed hyperparameter studies.", "relation_to_broader_scientific_literature": "The relation to the broader literature is excellent, with the main paper detailing the closest works with algorithmic tasks, and the supplementary related work section delving in studies of LLM creativity in more realistic settings (notably creative writing) as well as multi-token objective in contemporary large-scale training (such as the work of Gloeckle et al 24 or the training of DeepSeek V3). The relationship between the teacherless objective and these large-scale efforts might be worth discussing in the main text. \n\nThe reference of the classical work of Boden as an inspiration for the tasks is most welcome.", "essential_references_not_discussed": "None found.", "other_strengths_and_weaknesses": "* This paper is exceptionally well-written, and the text addresses most questions that come up while reading;\n* This paper is very timely, given the growth in interest in MTP methods recently;\n* The tasks are quite elegant models of creative tasks and the relationship to combinatorial and exploratory creativity is well-drawn;\n* The larger-scale tasks might require more explanation and exposition (why Self-Bleu as an evaluation metric?) but unfortunately space is lacking. This could make for a good follow-up study.\n* The result on hash-conditioning is very interesting and should be validated with larger-scale empirical work; how usable is it in practice? \n* Related work is extensive and presents an overview of the creativity literature, the MTP literature, the literature on strengths and weaknesses of NTP, and the theoretical literature on creativity in models.", "other_comments_or_suggestions": "* Sec 3.1: typo “from an prompt-free autoregressive” -> “ from a prompt-free autoregressive”\n* Sec 5: typo  “We defer discussion of theoretical studis of diversity”", "questions_for_authors": "Question 1: Fig 5 Temperature 2.0 might be too much for Gemma, did you try lower temperatures (1.0)?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TWadacY6x4", "reviewer": "Reviewer_gG8C", "summary": "This paper focuses on addressing the issue of insufficient creativity in traditional next-token prediction (NTP) when applied to open-ended algorithmic tasks. The authors have designed a series of minimalistic algorithmic tasks (such as Sibling Discovery, Triangle Discovery, Circle Construction, and Line Construction) to simulate real-world scenarios that require creative thinking. To this end, the paper proposes a multi-token prediction approach, primarily realized through Teacherless Training and Discrete Diffusion Models, and introduces a Hash-Conditioning strategy to enhance the diversity of generation. Experimental results demonstrate that, compared to traditional NTP methods, the multi-token approach exhibits significant advantages in boosting creativity and reducing the model's tendency to memorize and reiterate.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time & insightful feedback! We address your points below:\n\n### Major points\n\n> The mechanism underlying the role of hash-conditioning has not been clearly elucidated\n\nHere's our current intuition which we'll add to the paper. \n\nRecall that hash-conditioning explicitly injects randomness in the input; in softmax sampling, randomness is elicited from model output.  **Hash-conditioning may then help via two mechanisms:**\n\n- **More representational power:** In softmax sampling, if the model has to maximize diversity, the model must maintain multiple thoughts in its representations so that one of them can be sampled at output.  Hash-conditioning allows the model to fixate on just “one thought for a given hash”. This way the model can carefully represent one thought while being able to produce diverse thoughts for different hashes.\n- **Better planning:** Fixing randomness ahead of time allows the model to plan and co-ordinate multiple random decisions in tandem.\n\nWe emphasize that hash-conditioning is a novel approach proposed as part of our broader exploration of creativity (as Reviewer `Lrgc` and `16p5` remark as very interesting). **Thus, \nwe invite you to view hash-conditioning as a novel algorithmic contribution (in a larger paper) with reasonable intuition behind it,** opening up interesting future directions (such as verifying the above mechanisms & generalizing it). \n\n> Although the minimalistic tasks in the paper facilitate quantification and analysis, there exists a gap between these tasks and real-world applications, casting doubt on their generalizability\n\nIt is absolutely right to acknowledge the gap between our very simple tasks and ambitious real-world tasks. \n\nAs the first step, we provided summarization as a practical example demonstrating real-world applicability.\nHowever, the challenge with generalizing real-world tasks is that there is no pre-existing benchmark where creativity, originality and diversity are even quantifiable e.g., it’s **impossible** to judge originality when the dataset is the whole of the internet. This would require a non-trivial effort that is worth a few other papers’ work! \n\nRegardless, we’re happy that you (and also the other reviewers) acknowledge the value in studying our algorithmic tasks: they help us cleanly measure creativity/diversity/originality and analyze how different training methods can affect creativity.\n\n> The performance of large-scale NTP models (only Gemma-2b and GPT2-86M were used in the study) in this context,..., has not been evaluated.\n\nWe agree that an ideal study would study a range of larger models. \n- But it’s worth noting that, **for our very minimal dataset sizes/complexities, the 2B scale is very, very large!** (Indeed, Reviewer Lrgc notes that these models are “reasonable”).\n- Additionally, we have observed limited absolute performance gain with increasing model size. \n- **We have new experimental results that reproduce the benefit of diffusion models over NTP at the 400M parameter scale (the largest open-sourced one we can find)**. We will add detailed scaling plots for each task in the final version of the paper.\n\n> as well as the effectiveness of reasoning enhancement methods on these tasks\n\nThis is a completely valid question that we are curious about too. However, there is doubt as to whether “reasoning enhancement” methods would translate to “creativity enhancement\". This is because we not only care about “correctness/coherence of planning/reasoning”, but the “coherence **+ originality + diversity** of a plan”. We kindly refer you to **our response to Reviewer Lrgc (first quoted question)** where we provide three arguments substantiating our point. These are profound questions that require multiple papers of future research.\n\n---\n### Other points \n> Incorporate additional evaluation metrics... human studies...; demonstration of how the global perception capability of MTP (Multi-Token Prediction) contributes to the task... use case studies.\n\nThank you for these wonderful suggestions! \nEvaluation of creativity in the wild is hard and requires careful consideration worth the effort of a whole new paper. \nThe current creativity metric we chose, Self-BLEU, is widely used as a diversity metric in language generation benchmarks [1]. It quantifies diversity by measuring BLEU scores across outputs generated by the same model, thus showing the distinctiveness among outputs. We are also exploring other diversity metrics (such as distinct n-gram). \nHuman evaluations are beyond our current scope & expertise but remain a key future direction for the community.\n\nWe will certainly keep these in mind for future follow-ups!\n \n[1] Texygen: A Benchmarking Platform for Text Generation Models\n\n---\n\n**We sincerely hope we've addressed your key concerns (model size & other reasoning methods) in a way that allows you to re-evaluate the paper in more positive light, thank you!**", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "make sense", "theoretical_claims": "Yes", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "Key contributions of the paper related to the topic of Multi-token prediction of LLM and Next-token prediction.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strength:\n\n(1) The investigation into the issues surrounding NTP holds substantial theoretical significance and practical application prospects. (2) Controlled experimental design: By devising simplified algorithmic tasks, the study has clearly quantified two metrics—creativity and memorized repetition—thereby facilitating the demonstration of the method's advantages.\n\nWeakness:\n\n(1) The mechanism underlying the role of hash-conditioning has not been clearly elucidated. (2) Although the minimalistic tasks in the paper facilitate quantification and analysis, there exists a gap between these tasks and real-world applications, casting doubt on their generalizability. (3) The performance of large-scale NTP models (only Gemma-2b and GPT2-86M were used in the study) in this context, as well as the effectiveness of reasoning enhancement methods on these tasks, has not been evaluated.", "other_comments_or_suggestions": "Suggestions:\n\n(1) Expand the scope of experiments to other practical tasks to assess the generalizability of the proposed method. (2) Incorporate additional evaluation metrics to measure creativity, and conduct human studies to evaluate the effectiveness. (3) Provide a more detailed demonstration of how the global perception capability of MTP (Multi-Token Prediction) contributes to the task at hand, supported by use case studies.", "questions_for_authors": "See **Suggestions**", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "R2ksfkw82v", "reviewer": "Reviewer_16p5", "summary": "The paper analyzes the effectiveness of multi-token prediction in open-ended algorithmic tasks that require combinatorial and exploratory creativity. It argues that transformers trained with next-token prediction struggle in these tasks, whereas transformers using multi-token prediction or diffusion models perform better.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and effort in evaluating the paper! We are excited that you appreciate the analogy between wordplay and our tasks! We are also happy that you found (a) the tasks well-designed, (b) hash-conditioning interesting and (c) did not find any major drawbacks with the work!\n\nWe respond to your questions below. \n\n\n> the role of the random hash prefix is to retrieve a random sibling-parent triplet (or triangle, circle, etc.) from the input graph. Does this interpretation make sense?\n\nExactly! Intuitively, we view it as a random seed that guides the model in its random search over the graph to identify the siblings/triangles. \n\nFor what it’s worth, more abstractly, we believe hash-conditioning may help via two mechanisms:\n- **More representation:** In softmax sampling, if the model has to maximize diversity, the model must maintain multiple diverse thoughts in its representations so that one of them can be sampled at output.  Hash-conditioning allows the model to fixate on just “one thought for a given hash”. This way the model can carefully represent one thought while being able to produce diverse thoughts for different hashes.\n- **Better planning:** Fixing the randomness ahead of time could allow the model to plan and co-ordinate multiple random decisions in tandem.\n\n> While this paper does not introduce any new methods, \n\n\nWe wish to gently highlight that hash-conditioning is a novel method we put forth even if we only explore it on our algorithmic test bed. We believe this opens significantly new algorithmic possibilities for creativity in generative models e.g., could one pretrain language models with such hash strings (or by randomly inserting “hash tokens”) to encourage diversity?\n\n> What exactly does a null prefix refer to? … no additional tokens between the input and output? \n\nYou’re right! A null prefix indicates there are no additional tokens between the input and output sequences. Sorry about the lack of clarity -- we will fix it!\n\n> If possible, could you run experiments where the transformer predicts the output while attending only to the hash prefix? This experiment would help clarify why the hash prefix enhances performance in transformer models while having minimal impact on diffusion models.\n\n\nIf we understand correctly, we believe that one of our experimental setups already does this: specifically, teacherless training with hash-conditioning. Here, the model attends only to the hash prefix and some dummy tokens. We are curious if you have thoughts about the implications of this!\n\n> I also attempted to review Appendix D to find an example of the data, but it was not provided. It would be better to include an example for each task\n\nThanks for the feedback! We will improve the clarity of the task data with more illustrated examples, but for now, we note that we provide one example for each task in Figure 1&2, and **the full description of dataset generation** in Appendix D.", "claims_and_evidence": "The claim is well supported by experiments conducted across three types of architectures: the pre-trained Gemma, the SEDD diffusion model, and GPT-2, and four types of tasks.", "methods_and_evaluation_criteria": "While this paper does not introduce any new methods, I believe that designing new benchmark tasks is a valuable contribution. The authors also provide diversity and memorization scores, along with performance metrics, which I consider to be appropriate evaluation criteria. However, it would be better to define the memorization score in the main text, not in the caption in Figure 3.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The proposed tasks are well designed, and drawing an analogy with the wordplay “What kind of shoes do spies wear? Sneakers.” helps me better understand their purpose. The authors effectively present the experimental results using appropriate plots and clear visualizations.", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Overall, the paper is well written and well organized. I did not find any major drawbacks.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "I am very interested in the hash-conditioning experiment and have a few questions:\n1. What exactly does a null prefix refer to? Does it mean that there is no additional tokens between the input and output?\n2. From my understanding, the role of the random hash prefix is to retrieve a random sibling-parent triplet (or triangle, circle, etc.) from the input graph. Does this interpretation make sense?\n3. If possible, could you run experiments where the transformer predicts the output while attending only to the hash prefix? This experiment would help clarify why the hash prefix enhances performance in transformer models while having minimal impact on diffusion models.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper studies multi-token prediction for creativity in open-ended algorithmic tasks, introduces a novel testbed, and proposes hash-conditioning. Reviewers appreciate the novel and well-designed testbed, the rigorous quantification of creativity, strong empirical results showing MTP benefits, and the timely topic. However, initial questions were raised about clarifying hash-conditioning, the generalizability of tasks to real-world, and evaluating larger NTP models/reasoning methods. Overall, the paper is recommended for acceptance, as the authors' comprehensive rebuttal addressed concerns, confirming the work's significant contribution.
Decision: Accept (oral)