AI as Humanity’s Salieri: Quantifying Linguistic Creativity of Language Models via Systematic Attribution of Machine Text against Web Text
Authors: Ximing Lu
Melanie Sclar
Skyler Hallinan
Niloofar Mireshghallah
Jiacheng Liu
Seungju Han
Allyson Ettinger
Liwei Jiang
Khyathi Chandu
Nouha Dziri
Yejin Choi
Venue: ICLR 2025 Oral (2025)
Keywords: Machine Creativity
Large Language Model
Science of LLM
Machine Text Detection
Primary area: interpretability and explainable AI
TL;DR: We present CREATIVITY INDEX, a metric that quantifies the creativity of a text by reconstructing it from existing web snippets, supported by a novel dynamic programming algorithm, DJ SEARCH, for efficient computation.

Abstract: Creativity has long been considered one of the most difficult aspect of human intelligence for AI to mimic. However, the rise of Large Language Models (LLMs), like ChatGPT, has raised questions about whether AI can match or even surpass human creativity. We present CREATIVITY INDEX as the first step to quantify the linguistic creativity of a text by reconstructing it from existing text snippets on the web. CREATIVITY INDEX is motivated by the hypothesis that the seemingly remarkable creativity of LLMs may be attributable in large part to the creativity of human-written texts on the web. To compute CREATIVITY INDEX efficiently, we introduce DJ SEARCH, a novel dynamic programming algorithm that can search verbatim and near-verbatim matches of text snippets from a given document against the web. Experiments reveal that the CREATIVITY INDEX of professional human authors is on average 66.2% higher than that of LLMs, and that alignment reduces the CREATIVITY INDEX of LLMs by an average of 30.1%. In addition, we find that distinguished authors like Hemingway exhibit measurably higher CREATIVITY INDEX compared to other human writers. Finally, we demonstrate that CREATIVITY INDEX can be used as a surprisingly effective criterion for zero-shot machine text detection, surpassing the strongest existing zero-shot system, DetectGPT, by a significant margin of 30.2%, and even outperforming the strongest supervised system, GhostBuster, in five out of six domains.

Published as a conference paper at ICLR 2025
AI AS HUMANITY'S SALIERI: QUANTIFYING LINGUISTIC CREATIVITY OF LAN- GUAGE MODELS VIA SYSTEMATIC ATTRIBUTION OF MACHINE TEXT AGAINST WEB TEXT
Ximing Lu
Melanie Sclar
Skyler Hallinan &
Niloofar Mireshghallah
Jiacheng Liu
Seungju Han *
Allyson Ettinger Liwei Jiang"
Khyathi Chandu
Nouha Dziri
Yejin Choi
University of Washington [lux32, yejinj washington.edl teshiggtssssssssinnnto edu
Allen Institute for Artificial Intelligence
ABSTRACT
Creativity has long been considered one of the most difficult aspect human in- telligence for Al to mimic. However, the rise of Large Language Models (LLMss,, like ChatGPT, has raised questions about whether Al can match or even surpass human creativity. We present CREATIVITY INDEX the first step quantify the linguistic creativity of a text by reconstructing from existing text smippets on the web. CREATIVITY INDEX is motivated by the hypothesis that the seemingly remarkable creativity of LLMs may be attributable in large part to the creativ- ity of human human-written written texts the web. To compute CREATIVITY INDEX effi- ciently, we introduce DJ SEARCH, a novel dynamic programming algorithm that search verbatim and near-verbatim matches of text snippets from given doe- ument against the web. Experiments reveal that the CREATIVITY INDEX of pro- fessional human authors on average 66.2% higher than that of LLMs, and that alignment reduces the CREATIVITY INDEX of LLMs by an average of 30. %. In addition, we find that distinguished authors like Hemingway exhibit measurably higher CREATIVITY INDEX compared to other human writers. Finally, demon- strate that CREATIVITY INDEX can used as a surprisingly effective criterion for zero-shot machine text detection, surpassing the strongest existing zero-shot sys- tem, DetectGPT, by a significant margin of 30.2%, and even outperforming the strongest supervised system, GhostBuster, in five out of six domains.
INTRODUCTION
Creativity has long been considered of the most challenging "holy grail" of human intelligence for AI to mimic (Hasselberger & Loti (023). However, Large Language Models (LLLMs) such as
ChatGPT have taken the world by storm with their creative power. From generating poetry (Sawicki
ct al.
Deng et al.] 2024bb Sawicki et al. 2023) and composing music (Ding et al.) [024 Deng et al.,
2024a
Liang et 2024) to designing artwork (Makatura et al. 2024| Jignasu al. 2023 Lim
et al.
2024) and cratting compelling narratives (Yuan et al.) 2022 Mirowski et al. 2023a Ippolito al.
2022p. LLMs take only seconds produce outputs that would rival or even surpass the work of human creators. This proficiency has even sparked growing trend of using LLMs for content creation in industrial settings. For example, major studios in Hollywood have integrated L.LMs into production processes such as movie scriptwriting (Carnevale 2023). While studio executives are optimistic about using LLMs to streamline production and reduce costs, Hollywood writers are deeply concerned about being replaced by the rapid integration of LLMMs in the industry, leading to a five-month writers' strike (Koblin & John 2023).
While science fiction writer Ted Chiang characterizes LLMs blurry JPEG of the web (Hubert et al. 2024), many others wonder whether AI can indeed match or surpass the creativity of humanity. After all, LLMs have consumed orders of magnitude more works of writing than any single human

---

Published as a conference paper at ICLR 2025
could ever read, thus it may seem possible that LLMs could consequently reach a new level of literary sophistication and creativity beyond that of humanity at large.
To answer this question, the first step is to assess the level of creativity in machine texts compared to human texts. Creativity is a complex and ambiguous process that is challenging to define and quantify (Csikszentmihalyy 1997, Glaveanu et al., 2020; 2 Eagleman & Brandt 2017 Paeth). Several previous studies have attempted to quantify creativity in writing by developing specific rubrics and asking human evaluators score the writing based on these criteria. Vaezi & Rezaei (2018) devel- oped a comprehensive rubric to assess fiction writing, while [iigs & Collis ( (1982) used a taxonomy of structural complexity to categorize creative writing. More recently. [Chakrabarty et al. (2024) ap- plied the Torrance Test of Creative Thinking to evaluate the creativity of short stories generated by LLMSs terms of fluency, flexibility, originality and elaboration. While these rubric-based methods are valuable, scaling them up to evaluate large amounts of texts generated by LLMs is impractical due to the reliance on human evaluators.
In this work, we propose CREATIVITY INDEX, a novel statistical measure of creativity in text. The key intuition underlying CREATIVITY INDEX is to quantify the degree of linguistic creativity of a given text by reconstructing that text via mixing and matching of vast amount of existing text snippets on the web (See Figure Ir 24 additional examples in Appendix Fig. 5 to Fig. (3). The underlying premise of our work is that the seemingly remarkable creativity of LLMs may be in large part attributable to the remarkable creativity of human-written texts on the web. This contrasts with distinguished human authors such as Hemingway, whose original content and unique writing style cannot be easily replicated by simply assembling snippets from other works. To test this, we provide novel computational approach to systematically attribute machine text to web texts. Specifically, we introduce DJ SEARCH a novel dynamic programming algorithm that can efficiently search for verbatim and near-verbatim matches of text snippets from given document against the web. Here, near verbatim matches are defined as close paraphrases. characterized by high semantic similarity. Our algorithm combines strict verbatim matching using Infini-gram (Liu et al. (0024). which allows for fast retrieval of any existing sequence of words, with near-verbatim semantic matching achieved through a novel application of Word Mover's Distance (WMD) (Kusner et al. 2015) computed on the word embeddings of text snippets.
The contribution of our work is threefold: First, we introduce the CREATIVITY INDEX to reveal novel insights about machine creativity and human creativity. We find that the CREATIVITY INDEX of human authors- specifically professional writers and historical figures- is on average 66.2% higher than that of L.LMs. This creativity gap is consistent across various domains-novel snippets, modern poems, and speech transcripts- at both verbatim and semantic levels. Moreover, we notice that Reinforcement Learning from Human Feedback (RLHF), widely used alignment method, dramatically reduces the CREATIVITY INDEX of LLMs, by an average of 30.1%. This reduction more significant at the verbatim level than the semantic level, indicating that LLMs may have converged to certain linguistic style preferred by humans during alignment. Furthermore, we explore creativity differences among various groups of humans. Despite in- g-our variance, famous authors of classic literature, like Hemingway and Dickens, exhibit the highest levels of creativity, consistent with their levels of renown.
Second, we introduce DJ SEARCH as an efficient algorithmic tool to trace the usage of existing text snippets from the web that LLMs incorporate to compose new generations. The power of LLMs arises from training exhaustively existing human-written texts on the web, and it is meaningful to trace back and acknowledge the human writers whose work empowers these models' outputs-just as credit original composers when enjoying DJ's remix.
Finally, we demonstrate a novel use of CREATIVITY INDEX as a surprisingly effective criterion for zero-shot black-box machine text detection. Our method is ready to deploy out-of-the-bi requiring no training or prior knowledge of the text generator. It not only surpasses the strongest zero-shot baseline, DetectGPT (Mitchell et al. 2023a). by significant margin of 30.2%, but also outperforms the strongest supervised baseline, GhostBuster (Verma et al. 2024)-hiich requires expensive data collection for supervised training-in five out of six domains.
We believe that our study will enhance the understanding of LLMs and guide informed usage of con- tent created by LLMs. by providing an interoperable and scalable measurement assess creativity
'The name DJ SEARCH is inspired by the way a DJ creates remix by blending pieces of existing music.
2

---

Published as a conference paper at ICLR 2025
Please draft an abstract for a paper with the following title: "Postry Will Not Optimize, or, What Is Literature to AIP"
with Qualcomm
That partnership (ed the integration artfic intillalleeeeeeeeee sencee (A] varlouslapps,
Reforence
Vertatioo Mation
making them smarto and easler to use....
The aevancament of tachnolog logy has/lad tojthe integration (of artificial intelligence (AI) in various Tields, including literature, (However, the question remains: can Al] truly understard and poetry? This paper delves the complexities of poelry and the limilations/of (in its ability to compreheno and replicate the nuanced (mmttoons muanceclemotions
frrati
use artiicaa lgence in
Vanous fielos, including onguage
penerasion. One such language model
ChatGPT
Artificial Inialigenns (AO, such as dhat GPT33 to assist in the process. (However question
current generation poople with a
intel veclual brain. (The adhancement
Aljully rop
technology has led 10
and intr cacies of language,
elcctronic device
capable
Through afcritical anelysis of the ronshoo between poetry and this paper(e (eguees that while Allmay be able to generaate text that resembles] lacks the ability to Iruly... ...
point About the perception poetry the Irritationaa that office and readers impose
but there
but too far namowry limited infoxible (n ability to co morchend and apply all relevve facts in order to serve the process of selection, which better...
upon if Bayy understand powtry
existing research in field femily
Literature, poatry, and other forms of) noncommercial creative expression challenge the Simmo-nnttrumenllllss epproaches to language, predictive language ganeration, informing NLP (arg natural language processing models) such as GPT-3 -I as well as, more ganerally, generative Al (text to image, video, audio). Claims that Al systems automate and expedite creetivity reflect industry and research prioritess of speed, scale, optimization, and fictiooleessees driving much artilicial intelligence design and application. But...
mansuewring billion puszle of psycho and eemation spirivallty language, thougg puzzle komepa changing as ange and
migration Through c tocal analysis of the relationship between fomily and (nnsecuurty the article offers nuanced insight into
coomination of the role of human
crualivity the age of Al. He largus that be able procccca creative is own, ultimatoly humans
Prof. Michele Elam
search for similar esamples
because abiliy of automated
wisdom art, (ltarature, poctry, and
systems/oo be ge generate that
other forms nd human diecoure resuli, when Tran across the address of
resembiiig what hummnn might way Fugee we can just improve question...
b
CREATIVITY INDEX
Σ Lmu
Luniqueness
L-uniqueness number of words outside of jt-grams 2 L) that occur the reference corpus
number of words in the text
1.0
Novel
1.0
Speech
1.0
Poem
Luunqqueness
0.8
Luniguenees
0.8
L-uniqueness
0.8
0.6
0.6
0.6
0.4
0.4
professional writers
historical figures
0.2
famous
ChatEPT
ChatGPT
ChatGPP
0.0
minimum n n-gram length
minimum n-gram length
minimum n-gram length
Figure a: Example outputs from DJ SEARCH. We asked ChatGPT to generate an abstract based on the title of Prof. Michele Flam's paper, "Poetry Will Not Optimize; or, What Is Literature to AI?" (Flam) 2023) The abstract generated by ChatGPT contains significantly more verbatim and near-verbatim matches with existing texts on the web compared to the original abstract written by Prof. Elam. b: Definition of CREATIVITY INDEX. CREATIVITY INDEX is mathematically equivalent to the area under the L-uniqueness curve across range of minimum n-gram lengths L. The L-uniqueness of ChatGPT is noticeably lower than that of proficient human writers across various context granularities (i.e., n-gram lengths) in all domains, leading to significantly higher CREATIVITY INDEX for human writers compared to ChatGPT.
3

---

Published as a conference paper at ICLR 2025
in machine texts. Additionally, we hope that the out-of-the-bon machine text detection enabled by the CREATIVITY INDEX can empower individuals to discern between human texts and machine texts, fostering more informed and critical engagement with information in the digital age.
METHOD
CREATIVITY INDEX The key intuition underlying CREATIVITY INDEX is quantify the degree of linguistic creativity of a given text by estimating how much of that text can be reconstructed by mixing and matching vast amount of existing text snippets on the web, shown in Figure D Specifically, CREATIVITY INDEX assesses the extent to which the content of the text can be traced back to similar identical contexts found in other existing texts. This metric is grounded in the notion of originality from creative thinking in psychology literature, which is defined as the statistic rarity of response or an idea (Torrance 1966 Crossley et al.] (0166).
Concretely, let x be a text whose creativity we aim to quantify, such as speech transcript or a poem, either human written or machine generated. Let an n-gram of be any contiguous sequence of words of x, and let X;iin be the n-gram of x starting in the i-th word. Let C' be a massive reference corpus of publicly available texts on the web, and let f be a binary function that determines whether an n-gram Xii+n occurs anywhere in the corpus C. We define the L-uniqueness of a text x as the proportion of words w E x such that none of the n-grams in x that include w occur in the corpus C for 2 L denoted uniq(x, L). Intuitively. L-uniqueness measures the proportion of x's words that are used in novel contexts (here, n n-grams), unseen across a vast text collection C. Thus, higher T-uniqueness implies a higher level of originality of Formally, uniq(x, T.) ΣiII {x C) = Vi E (k- - n, k3,, n 2 ||/||] where trivially uniq(x, L) E [0, 1].
Note that when fixing x, the function uniq L) is monotonically increasing L grows. Its im- an indicator of the overall uniqueness of x across various context granularities (i.e., 7-gram lengths), and because of uniq(x, L)'s monotonicity it indirectly measures uniqueness growth speed. We thus define CREATIVITY INDEX as En>i uniq(x, n), with higher CREATIVITY INDEX indicating greater linguistic originality with respect the corpus C, as shown in Figure Dp.
proper integral-En21 -n22 uniq(x,n)
When text x part of the reference corpus C, its CREATIVITY INDEX would trivially become zero. This issue often arises with works from famous authors, as their writings are widely available online. To address this, for human texts written before the cutoff date of the reference corpus. we exclude any document d E C that contains copies, quotations, or citations of x and compute CREATIVITY INDEX using this filtered corpus, detailed in Appendix A.3
DJ SEARCH
To enable the use of our CREATIVITY INDEX it vital to compute it efficiently. For the efficient computation, we introduce DJ SEARCH, a dynamic programming algorithm designed to radpily identify the set of all x's n-grams (n 2 L) that occur in the corpus C.
A brute force approach would independently check every n-gram of x occurs in C. performing a quadratic number of f evaluations with respect x's length, and thus making it too computationally expensive. Instead, we design a two-pointer method (Laaksonen) (020) that takes only a linear number of f evaluations, as illustrated in Figure 2 The key idea is to reduce finding all n-grams occurring in C' to identifying the longest n-gram occurring in C' starting at each index i: once those have been found, is trivial to deduce all the n-gram occurring in C by computing their subsequences. Concretely, we progressively analyze the whole document by iteratively searching for the longest n- g-grrm that starts at each index and occurs in C, using f as the assessment. Once we have found such longest n-gram starting at i, we crucially reuse computations for i + by noting that f( [x C) = implies fxx++1a+n, C) = 1. Thus, we always analyze n-grams starting and/or ending at a later endpoint than before, which upper bounds the number of analyzed n-grams (i.e.. the number of f calls) to at most 2 | The implementation is detailed in Appendix A.
In addition to minimizing the number f evaluations, DJ SEARCH optimizes the time complexity of each evaluation. f determines whether a n- n-gram X occurs in the corpus C' either exactly or in semantically similar way. a paraphrase of X exists in C. Semantic similarity is often computed using text embeddings, which are fixed-length vector representations of text meanings. This reduces measuring text similarity computing vector distance. Text embeddings, typically
4

---

Published as a conference paper at ICLR 2025
this paper argues that while Al may be able to generate text that resembbees
Algurithm
DJ SEARCH (x,
Input:
text x. reference corpus C,
End Position
gram matching function ((N..C)
while
generate
resembies
minimum gram length L
this
Y
X
Oulput: set of matched gram
i- 0, J + L.8
paper
x
while S. ir (xiy:CC then
Position
argues
S 8 (Nay)
that
X
j + j+1
else
X
1- i+
+ max(1 +
may
end while
Matched ni nigram
Found gram matches from start to and position
x No gram matches from start to and position
An illustration of DJ Search algorithm. A brute force approach would independently check every n-gram of x occurs C, performing a quadratic number of evaluations with respect to x's length (i.e,, checking every cell in the grid). DJ SEARCH is two-pointer method that takes only a linear number of f evaluations. By progressively analyzing n-grams starting and/or ending later endpoint than before, DJ SEARCH limits the total number evaluations 2 2||x||. this example, the minimum n-gram length L is set to 5.
Figure 2:
generated by complex models (e.g., BERT (Devlin et al. 2019). RoBERTa (Liu et al. 2019). Span- BERT (Joshi al., 2020)) lack linearity, requiring independent computation for each n-gram in and C. To alleviate this issue we use Word Mover's Distance (WMD) (Kusner et al. 2015). an metric that measures distance between two n-grams by combining word embedding distances between each n-gram's words. WMD enables optimizing f's computation, as pairwise distances between word embeddings can be pre-computed for every pair of words, and then be reused in every function call of to identify n-grams in C' that are semantically similar to the ones x. The implementation is detailed in Appendix A.2
optimal transport-inspirg
To further boost efficiency, and given that occurrences of Xiit- are more likely in texts similar to x, we estimate by computing WMD only for the texts C most similar to x, as identified by BM25 (Robertson & Walker 1994). Moreover, exact occurrences of X in C' represent a less costly special case in computing f. We further optimize f's computation by using Infini- gram (Liu et al.) 2024), which finds exact matches of Xgi+n in C' in milliseconds; WMD is com- puted only if no matches are found by Infini-gram.
3 EVALUATION
How does the creativity of language models compare to humans? We compute the CREATIV- ITY INDEX for machine texts and human texts across three creative writing tasks: novel writing, poetry composition, and speech drafting. For human texts,
book snippets in the Book- MIA (Shi et al. 2024) dataset, popular modern poems collected by |PoemHunter.com and famous speeches from the American Rhetoric|speech bank. For machine texts, we prompt LLMs generate several paragraphs of novels, poems, or speeches, starting with an initial sentence from existing hu- man writings in each category (see Appendix for details). We experiment with state-of-the-art LLMs, including GPT-3 (Brown et al. 2020), ChatGPT (Ouyang et al., 2022), LLaMA 2 Chat (Tou- vron al., 2023), Tulu 2 (Ivison et al. 2023), and OLMo Instruct (Groeneveld et al. 2024). For open-source and open-we weight models, we use the largest model size available from each model family. We use RedPajama (Computer 2023), a large-scale English corpus with 900 million web documents, the reference corpus. The models we analyze are primarily pre-trained on the web data available before the cutoff date of the reference corpus RedPajama. We will discuss later how to handle newer models, such GPT-4 (OpenAl et al. 2023), given that it was largely trained on more recent web data and third-party private data, both of which fall outside the reference corpus. We restrict the matching criteria verbatim matches only in the first experiment. We will ablate the effect of different matching criteria, prompt formats, decoding strategies, context length, and model sizes in later experiments.
5

---

Published as a conference paper at ICLR 2025
Creativty Betteri
Creativity index Betteri
Creativtty Betteri
a
b
Human
ChetGPT
CheiGPT
CheiGPT
GPT-3
GPT-3
GPT-3
Tuk2
Tuk-2
Tuku2
DUKO
DLMO
OUMo
d
Creativity
Luniquerass (f) v.s. min n-gram length
f
Creativity
1.0
Human
GPT Base
0.8
CharGPT
Luuiquueess
ChatGPT
GPT-3
Uama2-Suae
0.4
Lama2-Chat
L'aa2-CCaat
Tuk2
9.2
Human inebes samurticr
c1300 postutis siyt
OLMO Base
13
0LKO
minimum nigram length
OLMO
g
Creativily Index (1 Beller)
h
L-anqqueness (1) documents
Luniqueeess Beller)
Human
OFT.
CPT-Base
ChacGPT
Luniquends
Liberd Chat
ChanGPT
Lama2-88se
0.6
GPT-3
0.5
Uhaa
L'ma2-Chat
0.4
0.3
10
15
DLMG
# doctuments
DUMO
documents
Bettori
20
35
50
k
Creativity Index Gemma
Creativity Index Mixtral
Human
ChatSPT
GPT-4
GPT-6
Creativit index wist. Lioma3 (T)
Creatiity Index wintt LLMs (1)
Chat
Human
Human
Tulu2
GPT-3
GPT-4-
OLMO
Creativity Index Better]
DetectGPT
Creativity
Derection AUROC
Classic Literature
Popular Teen Fiction
Novel
Powm
Speech
News
Thetrem:
Figure 3: a-c: CREATIVITY INDEX in novel writing (a), poetry composition (b) and speech writing (c) based solely on verbatim matches. d: CREATIVITY INDEX in novel writing considering both verbatim and semantic matches. L-uniqueness in novel writing with respect the minimum n-gram length L for humans and OLMo. f-g: CREATIVITY INDEX of LLMs before and after RLHF in novel writing, based solely verbatim matches (f) and based on both verbatim and semantic matches (g). h: I.-uniqueness in novel writing with respect to number of documents in the reference corpus. L-uniqueness when search over the top 50 documents in novel writing. j: The number of reference documents required to keep L-umiqueness below 50% in novel writing. k-l: CREATIVITY INDEX GPT-4 compared humans in novel writing based on verbatim matches, using a machine- generated reference corpus sourced from the instruction-alignel version of Gemma- 7B, Lllma3-8B and Mixtral-7B. well combination all three. m: CREATIVITY INDEX different groups of human writers. n: Detection AUROC across various domains: our approach sets new state-of- the-art for zero-shot detection, even surpassing supervised baselines.

---

Published as a conference paper at ICLR 2025
a
Creativity/ Index (1 Better)
b
Creativity index (f Better)
3.5
35
4,0
Human ChatGPT
Human
ChatGPT
C
Creativity Index Better)
d
Creativity Index Better)
3.0
3.5
4,0
Seitences
Human
2.5
30
4,0
708
Human
ChatGPT
Uamaz-Chat
158
Tuluz
Figure 4: a-c: CREATIVITY INDEX of ChatGPT in novel writing based verbatim matches, with different prompt formats (a), p values in top-p decoding (b) and prompt length (c). d: CREATIVITY INDEX of LLaMA 2 Chat and Tulu 2 with different model sizes.
Our primary finding is that humans consistently exhibit much higher level of creativity compared to any LLM across all tasks (Fig. Ba-c).. Averaged across all models, the CREATIVITY INDEX of humans is 52.2% highe than LLMSs in novel writing (p- 6.9 10 by Mann-Whitmes U test unless otherwise specified; N = 600), 31.1% % higher in po-try composition (p = 1.5 x 10-1b; N = 600) and 115.3% higher in speech drafting (p- = 6.1 x 10-31, N = 600). This suggests that human writings are composed of far more unique combinations of words and phrases compared to model generations. On the other hand, the differences in model creativity are much smaller and show very low statistical significance (p 0.09; N 1500).
Furthermore, we experiment with different prompt formats on top of ChatGPT, intentionally encour- aging creativity in the model's generations by incorporating instructions such as 'push for creative ideas, unique emotions, and original twists,' 'be bold and creative," or 'you are a creative writer.' (Fig. Fa) For a full list of the prompts we used, please see Appendix [B.]] We found that the difference in the CREATIVITY INDEX of ChatGPT across different prompts is minimal, with no statistical significance (p 0.23; N = 600). We also experimented with different decoding strate- gies by varying the p value in top-p decoding (Fig. FD. Although higher p value resulted a marginally higher CREATIVITY INDEX, the difference was minimal and not statistically significant (p = 0.23; N = 600). Moreover, we ablate the effect of prompt length by varying the number of sentences from human writings included in the prompt (Fig. 4c). We found that longer prompts tended to result in a slightly higher CREATIVITY INDEX, likely due to the model copying more from the longer human text in the prompt. However, the statistical significance of these differences is very low (p = 0. 0.13; N = 600). Lastly, we analyze the effect of different model sizes for LLaMA 2 Chat and Tulu 2, but do not observe a consistent trend (p- 0.12; N- 600) (Fig.
How do different matching criteria affect creativity measurement? We experiment with stricting valid matches to verbatim only, and with allowing both verbatim and semantic matches. First, the creativity gap between humans and LLMs becomes even larger when considering semantic matches in addition to verbatim matches (Fig. 3i. Averaged across all models, the CREATIVITY INDEX of human, based on both verbatim and semantic matches, is 102.5% higher than LLMs in novel writing (p- 2.6 x 10 12, N = 600), whereas based on verbatim matches alone, the CRE- ATIVITY INDEX of human 52.2% higher than LLMs. Second, semantic matches provide more signal for analyzing the uniqueness of longer n-grams (Fig, Be.. For example, while the gap in L-uniqueness at L = 11 between human text and machine text from OLMo Instruct is 3.7% based on verbatim matches alone, this gap widens to 16.3% when considering both verbatim and semantic matches (p- = 3.1 x 10-7; N = 600). This indicates that although some of the longer n- n-grams in machine text may appear unique at the verbatim level, they are similar to certain text snippets in the reference corpus at the content level.
What impact does RLHF have on model creativity? RLHF aims to align model's outputs with human preferences, enhancing LLMs' ability to follow instructions and improving their safety and adaptability. To understand the impact of RLHF on model creativity, we compare the CREATIVITY INDEX of the LLMs before and after RLHF alignment. Specifically, we experiment with GPT Base (Brown et al.] 2020), LLaMA 2 Base (Touvron et al. 2023). and OLMo Base (Groeneveld al. 2024) and compare their creativity with their counterparts post- RLLt- alignment. Our main
The percentage difference computed using the formula: CREATIVITY INDEX (human) CREATVITY INDEX (model)
CRLATIVIYY INDEX (model)
7

---

Published as a conference paper at ICLR 2025
finding is that the CREATIVITY INDEX of models after RLHF alignment is much lower than those before RLHF (Fig. 3r.g). Based on verbatim match alone, the CREATIVITY INDEX of LLMs reduces by average of 30.1% 1% after RLHF (p = 1.3 X 10 12; N 600). Based on both verbatim and semantic matches, the CREATIVITY INDEX of LLMM decreases by an average of 8.9% after RLHF (p 0.01; N 600). We notice that the reduction of CREATIVITY INDEX after RLHF is noticeably larger when considering verbatim matches alone. We speculate that models might have learned certain linguistic styles preferred by humans during RLHF, leading to a decreased surface form diversity in its outputs.
How do overlapped n-grams distribute in the reference corpus? In addition to measuring the amount of matched n-grams in a given text, we also investigate the distribution of these n-grams in the reference corpus. We aim to understand whether these matched n-grams spread across many documents or concentrated in a few. Specifically, we identify the top N documents that contain the highest amount of matched n-grams and result in the minimum L-uniqueness for given text. This problem can be reduced to the maximum coverage problem (Nemhauser et al.] 1978) and approximated using a greedy algorithm. Here, we consider both verbatim and semantic matches. Our main finding is that the matched n- n-grams in machine texts are concentrated in fewer documents compared human texts (Fig. B---). When searching over the top 50 documents, the averaged L-uniqueness (L 5) for machine texts is 32.8%, which is 73.4% lower than human texts (mean: 56.6%: p = 3.9 x 10 10-19; 19; N 600). Conversely, keeping L-uniqueness below 50% requires searching through an average of 41.2 documents for human texts, which is 213.7% more than for machine texts (mean: 13.4; p = 1.6 x 10-22 N = 600). This implies that it's more likely to find some existing documents resemble models' generations than human writings.
How to measure creativity in LLMs trained on data outside of the reference corpus? The CREATIVITY INDEX of GPT-4 would be significantly inflated if computed using the RedPajama corpus, as RedPajama's cutoff date is two years earlier than GPT-4's knowledge cutoff, and GPT- 4 is additionally trained on third-party private data that we don't 't have access to. We hypothesize that LLMs pre-trained on similar web data are likely to memorize and replicate similar patterns. As a result, when comparing the generations of these models, we expect them to be more similar to each other than to human texts, which often contain long-tail patterns. Therefore, to compare the creativity level of GPT-4 with humans, we use a model-genent generreeee reference corpus from newer open-weight models with knowledge cutoff dates similar to GPT-4 including the instruction-aligned versions of Gemma-7B (Team et al. 2024). Llama3-8B (AI@ Metal 2024), and Mixtral-7B (Jing ctt al., 2023). Specifically, we randomly sample 150k sentences from the RedPajama corpus and prompt these models generate document-level continuations. Based on the model--eneratte ref- erence corpus, the average CREATIVITY INDEX of humans is 30.3% higher than GPT-4 in novel writing (p 2.3 10-12, N 600) (Fig. <I). This suggests that while newer LLMs like GPT-4 may appear more creative when compared to public data, they still learn common patterns from their private training data and tend to emit similar patterns as other LLMs trained on comparable data.
How does the creativity vary among different groups of human? We compare the creativity levels among three categories of writings: books published in 2023 from the BookMIA (Shi et al.) 2024) dataset, classic literature by famous authors, and popular young adult fictions, both sampled from Goodreads) book lists. Our main finding is that classic literature exhibits higher creativity level than the other two categories (Fig. (3m). The average CREATIVITY INDEX of classic literature is 21.6% higher than young adult fictions (p 2.7 10-90, N 3000), and 13.8% higher than books published in 2023 (p- = 4.3 x 10-120. 120. N = 3000). Though CR CAATVIIYYY INDEX may reflect some inherent differences in creativity, it could also be influenced by factors like writing style and era of composition. For instance, some classic literature is written in older English, making it harder to reconstruct from web texts that primarily use modern English. In addition to the differences across categories, we also observed noticeable variance in creativity within each category. For example, the CREATIVITY INDEX of 'The Hunger Games' is 35.4% higher than 'Twilight' (p- = 1.5x 10 19, N 200), even though both books belong to the category of popular young adult fiction.
Can we leverage differences in creativity for detecting machine-generaten text? Based on the creativity difference between humans and LLMs, we propose to use CREATIVITY INDEX as a cri- terion for zero-shot black-box machine text detection. Texts with higher creativity are more likely
8

---

Published as a conference paper at ICLR 2025
to be written by human. Our approach is ready to deploy out-of-the-bos requiring no training or prior knowledge of the text generator. In addition to creative writing tasks, we also test our method fake news and theorem proofs. Detecting fake news is crucial for genee-eeeee solutions is impor-
on detecting
protecting the public from misinformation, while identifying tant for regulating students' use of LLMs in their coursework. To obtain additional test data, prompt LLMs to generate news articles based on the fake news headlines from the Misinfo Reaction Frames (Gabriel et al. 2022) and compare them with the real news articles from the XSum (Narayan et al. 2018) dataset. Meanwhile, we prompt LLMs to generate proofs for theorems from the Natu- ralProofs (Welleck et al., 2022) benchmark, and compare them with the ground- -ruth numan-writter proofs. The baselines we compare against includes the state-of-the-art zero-shot detector, Detect- GPT (Mithhell et al. 2023a), which uses the curvature of log probability as the detection criterion, as well as several supervised methods. These include OpenAl's I' RoBERTa-based detector, fine-tuned on millions of generations from various GPT-2 sized models, and the state-of-the-a supervised detector, Ghostbuster (Verma al., 2024), fine-tuned thousands of generations from ChatGPT. We measure performance using the area under the receiver operating characteristic curve (AUROC), which represents the probability that a classifier correctly ranks sandomly-selecter human-writte example higher than a randomly selected machine-generater example. Our method achieves new state-of-the-arr performance in zero-shot detection: consistently surpasses DetectGPT and Ope- nAI's detector across all domains, with significant improvements in AUROC- 30.2% and 26.9%, respectively. also outperforms the strongest supervised baseline, Ghostbuster- which requires expensive training and data collection-in in five out of six domains, achieving an average AUROC improvement of 3.5% (Fig. 3).
4 DISCUSSION
This work investigates the level of linguistic creativity in texts generated by LLMS and written by humans. Our findings suggest that the content and writing style of machine-generatee texts may be less original and unique, as they contain significantly more semantic and verbatim matches with existing web texts compared to high-quality human writings. We hypothesize that this limited cre- ativity in models may result from the current data-driven paradigm used to train LLMs. In this paradigm, models are trained to mimic human-written texts during the pre-training stage, and to produce outputs aligned with human preferences during the RLHF stage. As result, models learn to generate fluent and coherent texts by absorbing and replicating common patterns observed in their training data. This reliance on existing text patterns can restrict their originality, as their outputs are inherently shaped by previously seen examples. In contrast, accomplished authors such as Hem- ingway go beyond simply mimicking the great writings of others; they craft their own narratives to express their unique opinions, perspectives, and insights, drawing from their personal experiences, emotions, and backgrounds, which translates to the more creative compositions of words and phrases that our method detects. Just as DJ remixes existing tracks while a composer creates original mu- sic, we speculate that LLMs behave more like DJs, blending existing texts to produce impressive new outputs, while skilled human authors, similar to music composers, craft original works.
This work also faces the following limitations. Firstly, the computation of the CREATIVITY INDEX is constrained by the reference corpus used for DJ SEARCH. While open-source LLMs such as OLMo rely on publicly available texts from the internet for their training data, major companies like OpenAl additionally curate private data to train their closed- source LLMs such ChatGPT. Without incorporating these private data into the reference corpus of DJ SEARCH, the CREATIVITY INDEX of closed-source LLMs may be somewhat inflated. Secondly, the overlap with existing texts identified by DJ SEARCH in models' generations may not conclusively indicate memorization of a specific document. It's possible that these text fragments, or their variations, appear in multiple documents that the model has been trained on, including those outside the reference corpus of DJ SEARCH. Thirdly, the human authors that this work focuses on are those with relatively high-quality writings available in existing public datasets. While some human writings can be mediocre, tedious and unoriginal, we aim to assess how the creativity levels in impressive LLM outputs compare against the high-quality writings produced by professional human authors. Lastly, we acknowledge that the discussion surrounding the use of LLMs in social and industrial settings is highly complex, and our work here speaks only considerations in this discussion include socioeconomic factors and ethical implications, which fall beyond the scope of this paper.
part of it. Besides the creativity of machine- -eneertted content, other

---

Published as a conference paper at ICLR 2025
5 RELATED WORK
Measuring Creativity in Ideas: Measuring creative thinking and problem solving takes root in early work in psychology (Torrance 1966), where researchers defined four pillars for creative think- ing: fluency, flexibility, originality and elaboration. [Crossley et al. (2016) later on developed this notion and built it to expand this to measuring creative writing in students, where they also adopted n- n-gram novelty for a measure of originality. However, these prior work focus on creativity in humans, and they also do not introduce any automated metrics or measurements.
Measuring Creativity in Machine-generated Text Using Expert Annotators: Closely related to CREATIVITY INDEX is recent line of work in the generative Al literature comparing the creativity of human writers to that of large language models in different domains such as story telling and journalism (Chakrabarty et al. 2023 2024 Anonymous, 2024). Similar to us, the approach in this direction often involves prompting an LLM to write an original story or news article, based on some existing premise press release, and then comparing the
text to the human- written counterparts. These works, however, take a rather subjective approach, where they define and measure creativity based human expert annotations and whether people perceive the text to be more creative, rather than an objective quantification of novelty that we provide.
Measuring Novelty of N N-grams: Finally, closely related to our work in terms of techniques is Nguyen| (2024) and Merrill et al. (2024). The former attempts at finding n-gram rules that would cover and predict generations from transformer models, showing that more than 70% of the times transformers follow some pre-set patterns and rules. The latter is more similar to our work as they also measure the novelty of generated n-grams and compare it to human-written text, however they differ from us in low major ways: (I) they only find verbatim matches, whereas we also match to approximate, semantically similar blocks of text and (2) they compute the percentage of n-grams of a certain length in a text that can be found in the reference corpus, whereas we measure how much of the text can be reconstructed by mixing and matching vast amount of existing text snippets of varying lengths from the web.
Machine Text Detection: Detecting machine machine-generater generated text has been explored for several years using a variety of methods (Jawahar al.] 2020; [Uchendu et al. 202T). Gehrmann et al. (2019)
and Dugan et al. (202)) demonstrate that even humans tend to struggle to differentiate between text written by humans and machines, highlighting the need for automated detection solutions.
Some approaches involve training classifier in supervised manner to identify machine-geneee generrted text (Bakhtin et al. 2019; Uchendu et al.) 2020), while others use a zero-shot detection method (So-) laiman al., 2019| Ippolito et al. (2020). Additionally, there is research on bot detection through question answering (Wang et al. [023]] Chew & Baird (2003). Recently, Miichhel et al.) (2023b) in- troduced DetectGPT, a zero-shot method based on the hypothesis that texts produced by large lan- guage model (LLM) are located at local maxima, and thus exhibit negative curvature, in the model's probability distribution. Follow-up work build on DetectGPT by making it faster (Bao et al. 2024) and using cross-detection when the target model is unknown (Mireshghallah et al. 2024).
CONCLUSION
We introduce CREATIVITY INDEX, an interoperable and scalable metric designed to quantify the linguistic creativity of given text by estimating how much of that text can be reconstructed by mixing and matching a vast amount of existing text snippets on the web. To efficiently compute the CREATIVITY INDEX, we developed DJ SEARCH, a novel dynamic programming algorithm that can search verbatim and near- near-verbatim matches of text snippets from a given document against the web. We find that the creativity index of professional human writers is, on average, 66.2% higher than that of L.Ms.. Notably, RLHF dramatically reduces the creativity index of LLMs by an average of 30.1%% %. Furthermore, we demonstrate that CREATIVITY INDEX can be used as a surprisingly effective criterion for zero-shot black-box machine text detection. Our method not only surpasses the strongest zero-shot baseline, DetectGPT, by a significant margin of 30.2%, but also outperforms the strongest supervised baseline, GhostBuster, in five out of six domains. We hope that this study enhances the understanding of LLMs through the lens of linguistic creativity, and fosters informed usage of content created by L.MMs in real-world applications.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Sahar Abdelnabi and Mario Fritz. Adversarial watermarking transformer: Towards tracing text provenance with data hiding. 42nd IEEE Symposium on Security and Privacy, 2021.
Al@Meta. Llama 3 model card. 2024.
URL
lama/
1lama3/blob wwwooo//mainn. main/MODEL CARD md
Anonymous. Do LLMs plan like human writers? comparing journalist coverage of press re-
leases with LLMs.. In Submitted ACL Rolling Review June 2024, 2024. URL https:
//openzeview. www.wwww..eeeeeeeeee..ee...eee.e met//founme t E3VS455xx under review.
Mikhail J. Atallah, Victor Raskin, Michael Crogan, Christian F. Hempelmann, Florian Kerschbaum, Dina Mohamed, and Sanket Naik. Natural language watermarking: Design, analysis, and proof-of-concept implementation. In Information Hiding, 2001. URL https://api.
Anton Bakhtin, Sam Gross, Myle Ott, Yuntian Deng, Marc' Aurelio Ranzato, and Arthur Szlam. Real or fake? learning to discriminate machine from human generated text, 2019. URL https:
//axxiv.org/aba. 1966. 03351
Guangsheng Bao, Yanbin Zhao, Zhiyang Teng, Linyi Yang, and Yue Zhang. Fast-detectgpt: Ef- text via conditional probability curvature. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL
ficient zero-shot detection of
net forum?id=
Bpcgcr8E82
John B. Biggs and Kevin F. Collis. The psychological structure of creative writing. Australian Journal of Education, 26:59 70, 1982. URL htt.tttss//pa....cacccccoooooo.. oorp
CorpusID: 1413811:4111111
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert-Voss Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam Mc- Candlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few- shot learners. In Advances in Neural Information Processing Systems 33: Annual Con- ference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020. URL http:ps///prcccc.nggss hew. neurips. cc/paperr/222/hash 1457c0a6bfcb496744 htt.a//ceeee..aaaaa.aaaaaaaaaaa.
Robert Carnevale.
The
hollywood
writers'
strike
may actually be aid-
ing ai's
takeover,
May
2023.
URL https://www thewrap. com/
Tuhin Chakrabarty, Vishakh Padmakumar, Faeze Brahman, and Smaranda Muresan. Creativity support in the age of large language models: An empirical study involving emerging writers. arXiv preprint arXiv:2309.1255 2023.
Tuhin Chakrabarty, Philippe Laban, Divyansh Agarwal, Smaranda Muresan, and Chien-Sheng Wu.
Art or artifice? large language models and the false promise of creativity. In Proceedings of the CHI Conference on Human Factors in Computing Systems, CHI 2024, Honolula, HI, USA, May 11-16, 2024, pp. 30:1-30:34 ACM, 2024. http:t/:: /twii /ttp:///....ro/...111//333333333 3642731
Monica Chew and Henry S. Baird. Baffletext: human interactive proof. In IS&T/SPIE Electronic Imaging, 2003.
John Joon Young Chung, Shiqing He, and Eytan Adar. The intersection of users, roles, interac- tions, and technologies in creativity support tools. In Proceedings of the 2021 ACM Designing Interactive Systems Conference, pp. 1817-1833, 2021.

---

Published as a conference paper at ICLR 2025
Together Computer. Redpajama: open dataset for training large language models, 2023. URL
https: hithhb..ttt..t... httpaaa/.occcootteeeeccooroeeeee RRedP a
Scott A Crossley, Kasia Muldner, and Danielle $ McNamara. Idea generation student writing: Computational assessments and links to successful writing. Written Communication, 33(3):328- 354, 2016.
Mihaly Csikszentmihalyi Flow and the psychology of discovery and invention. HarperPerennial, New York, 39:1-16, 1997.
Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. In Neural Information Processing Systems, 2013. URL ott. orgg
ID:
Qixin Deng, Qikai Yang, Ruibin Yuan, Yipeng Huang, Yi Wang, Xubo Liu, Zeyue Tian, Jiahao Pan, Ge Zhang, Hanfeng Lin, Yizhi Li, Ying Ma, Jie Fu, Chenghua Lin, Emmanouil Benetos, Wenwu Wang, Guangyu Xia, Wei Xue, and YinTing Guo. Composerx: Multi-agent symbolic music composition with Imss ArXiv preprint, abs/2404. 18081, 2024a. URL bottps://arxiv.ors abs/2404 18081
Zekun Deng, Haoxia Yang, and Jun Wang. Can ai write classical chinese poetry like humans? an empirical study inspired by turing test. ArXiv preprint, abs/2401.04952, 2024b. URL https:
//arxiv.org/ abw/////.111 04952
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 4171-4186 Association for Computational wtwg/1111-- 1423
Linguistics, 2019. URL httpt:::///mm httttp::/aclannthllee
Shuangrui Ding, Zihan Liu, Xiao wen Dong, Pan Zhang, Rui Qian, Conghui He, Dahua Lin, and Jiaqi Wang. Songcomposer: A large language model for lyric and melody composition in song generation. ArXiv preprint, abs/2402. 2024.
17645
Liam Dugan, Daphne Ippolito, Arun Kirubarajan, Sherry Shi, and Chris Kallison-Burche Real or fake text?: Investigating human ability to detect boundaries between human-written and machine-generatee text. In Thirty-Seventh AAAI Conference on Artificial Intelligence, AAAI 2023, Thirty-Fifih Conference on Innovative Applications of Artificial Intelligence, IAAI 2023, Thirteenth Symposium on Educational Advances in Artificial Intelligence, EAAI 2023, Wash- ington, DC, USA, February 7-14, 2023, pp. 12763-12771. AAAI Press, 2023. URL https:
1609/ awt.:/aaa..... www.131111..... 26501
David Eagleman and Anthony Brandt. The runaway species: How human creativity remakes the world. Catapult, 2017.
Michele Elam. Poetry will not optimize, or what is literature to ai? American Literature, 2023.
URL mnttps://app http://///ppps..asieeeeeeeee semant http:oooa...o.rroooors/002222520
Tina Fang, Martin Jaggi, and Katerina Argyraki. Generating steganographic text with LSTMs. In
Allyson Ettinger, Spandana Gella, Matthieu Labcau, Cecilia Ovesdotter Alm, Marine Carpuat, and Mark Dredze (eds.), Proceedings of ACL 2017, Student Research Workshop, pp. 100-106, Vancouver, Canada, July 2017. Association for Computational Linguistics. URL https://
velanthology.oror
Jonas Frich, Lindsay MacDonald Vermeulen, Christian Remy, Michael Mose Biskjaer, and Peter Dalsgaard. Mapping the landscape of creativity support tools in HCL In Proceedings of the 2019 CHI Conference Human Factors in Computing Systems, CHI 2019, Glasgow, Scotland, UK, May 04-09, 2019, pp. 389. ACM, 2019. URL https::/ //di org 1t/g/10.100 212/10..145/32906055 3300619
12

---

Published as a conference paper at ICLR 2025
Saadia Gabriel, Skyler Hallinan, Maarten Sap, Pemi Nguyen, Franziska Roesner, Eunsol Choi, and Yejin Choi. Misinfo reaction frames: Reasoning about readers' reactions to news headlines. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguisties (Volume I: Long Papers), pp. 3108-3127. Association for Computational Linguistics, 2022. URI https: http::///ec.btttooooooooo/g/2222
222
Sebastian Gehrmann, Hendrik Strobelt, and Alexander Rush. GLTR: Statistical detection and vi- sualization of generated text. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pp. 11 111-116. Association for Computational Linguisties, 2019. URL http::t:/aaaaattttoooaao
org. 0000119-311199911111110101111111
Katy Gero, Alex Calderwood, Charlotte Li, and Lydia Chilton. design space for writing support tools using a cognitive process model of writing. In Proceedings of the First Workshop on Intelli- gent and Interactive Writing Assistants (In2Writing 2022). pp. 11-24. Association for Computa- tional Linguistics, 2022. URL BtLhttts::10 /
wrg. www.:222.nwwiriiinnn 1.2
Katy Ilonka Gero, Tao Long, and Lydia B. Chilton. Social dynamics of Al support in creative writing. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, CHI 2023, Hamburg, Germany, April 23-28, 2023, pp. 255:11252444 ACM, 2023. URL https: //doi. 110.0.115443500 111111144444555 4388444588838882 3580782
Vlad Petre Glaveanu, Michael Hanchett Hanson, John Baer, Baptiste Barbot, Edward P Clapp, Gio- vanni Emanuele Corazza, Beth Hennessey, James C Kaufman, Izabela Lebuda, Todd Lubart, al. Advancing creativity theory and research: A socio-cultural manifesto. The Journal of Creative Behavior, 54(3):741-744) 2020.
Dirk Gronnvvel, Iz Beltagy, Evan Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Au- thur, Khyathi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, Tushar Khot, William Merrill, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crys- tal Nam, Matthew Peters, Valentina Pyatkin, Abhilasha Ravichander, Dustin Schwenk, Saurabh Shah, William Smith, Emma Strubell, Nishant Subramani, Mitchell Wortsman, Pradeep Dasigi, Nathan Lambert, Kyle Richardson, Luke Zettlemoyer, Jesse Dodge. Kyle Lo, Luca Soldaini, Noah Smith, and Hannaneh Hajishirzi. OL.Mo: Accelerating the science of language models. In P Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pp. 15789-15809, Association for Computational Linguistics, 2024. URL https: http//aaaaannnnoco..oaococ..c.aa wwwp://...rrg.ee
202/ 2024 Long.ll--ooonn.. 841
William Hasselberger and Micah Lott. "where lies the grail? ai, common sense, and human practical intelligence". Phenomenology and the Cognitive Sciences, pp. 1-22, 2023. URL https://
api. www.ntisccccccaa.nccommmm httpsshcc..a..oooooooooospaaaaa.
Gao Huang, Chuan Guo, Matt J. Kusner, Yu Sun, Fei Sha, and Kilian Q. Weinberger. Supervised word mover's distance. In Neural Information Processing Systems, 2016. URLhttss://aap
semanticscholee htt.a//aa...ageeooooussiiii1-1-.
Kent F Hubert, Kim N. Awa, and Darya L. Zabelina. The current state of artificial intelligence generative language models is more creative than humans on divergent thinking tasks. Sci- entific Reports, 14, 2024. URL lnttps:: //aps:://ap.. semanticscholar. wwwphh/ww.........oooooooooooooe http://cco.oaaaoooo/ooosspssaao Cotg//orppssD:
267616181
Daphne Ippolito. Daniel Duckworth, Chris Callison-Burch, and Douglas Eck. Automatic detection of generated text is casiest when humans are fooled. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 1808-1822 1822. Association for Computational Linguisties, 2020. URL
org. 2000 aci www.amcc.mm.....................
Daphne Ippolito, Ann Yuan, Andy Coenen, and Sehmon Burnam. Creative writing with an ai-powered writing assistant: Perspectives from professional writers.
ArXiv preprint,
abs/2211 11.211.0555 1.05030, 2022. URL httl:tttss/:.s....ooo waarr///b///aa/ httg//bb/2211..00//
Hamish Ivison, Yizhong Wang, Valentina Pyatkin, Nathan Lambert, Matthew Peters, Pradeep Dasigi, Joel Jang, David Wadden, Noah A. Smith, Iz Beltagy, and Hannaneh Hajishirzi. Camels
13

---

Published as a conference paper at ICLR 2025
in a changing climate: Enhancing Im adaptation with tulu 2. ArXiv preprint, abs/2311.10702. 2023.
Ganesh Jawahar, Muhammad Abdul-Mageed, and Laks Lakshmanan, V.S. Automatic detection of machine generated text: A critical survey. In Proceedings of the 28th International Confer- ence on Computational Linguistics, pp. 2296-2309. International Committee on Computational Linguisties, 2020. URL htttp:::s//aaaatlllleeoo
org/ wwwinnooninneeeeeo
Albert Qiaochu Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lu- cile Saulnier, L'elio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b. ArXiv, abs/2310.06825 2023. URL http:s///aai.ssmmtitsssstteeei wtwpaaa/oo/ooooorrriiaaaa 263830494
Anushrut Jignasu, Kelly O. Marshall, Baskar Ganapathysubramanian, Aditya Balu, Chinmay Hegde, and Adarsh Krishnamurthy, Towards foundational ai models for additive manufactur- ing: Language models for g-code debugging, manipulation, and comprehension. ArXiv preprint, 0t//2339.024465 2023. URL /ttp:ttts///:xx..rrrrr http///..//233 02465
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. Span- BERT: Improving pre-training by representing and predicting spans. Trunsactions of the Associ- ation for Compulational Linguistics, 8:64-77, 2020. URL https::/aclarr: loy. org/
2020. ta20..a
John Kirchenbauer, Jonas Geiping, Yuxin Wen, Jonathan Katz, Ian Miers, and Tom Goldstein. A watermark for large language models. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th Inter- national Conference Machine Learning, volume 202 of Proceedings of Machine Learning mmlr.
Research, pp. 17061-17088 PMLR, 23-29 Jul 2023. URL pettps://proceeiie presss/202/kkaeecc httss//wo222kkrrcheebbee2222222 html
Brooks Barnes Koblin and John. Hollywood writers strike: Agreement gives writers guild most of what wanted, Sep 2023. URL htttps::/wwwwwttttttttt wttpttime..comeieeeee com/ ccmm/Livee2222/09/2/1
business mossiness/bolywwod-wwwo
Matt J. Kusner, Yu Sun, Nicholas I. Kolkin, and Kilian Q. Weinberger. From word embeddings to document distances. In Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015, volume 37 of JMLR Workshop and Conference Pro- ceedings, pp. 957-966. JMLR.org, 2015, URL http: wttp:prreeeeiiin...........e.eee kusnerb15. html
mlr.
Antti Laaksonen. Guide to competitive programming. Springer, 2020.
Xia Liang, Xingjian Du, Jiaju Lin, Pei Zou, Yuan Wan, and Bilei Zhu. Bytecomposer: a human-like melody composition method based on language model agent. ArXiv preprint, abs/2402.17785,, 2024. URL arxiv...vvv org abs 22102220222777
Jonghan Lim, Birgit Vogel-Heuser, and Ilya Kovalenko. Large language model-enabled multiia- manufacturing systems. ArXiv preprint, abs/2406.01 893, 2024. Ut..httpss//aaxxw..rrg
abs abs/ /006/240.cco 01893
Jiacheng Liu, Sewon Min, Luke Zetrlemoyer, Yejin Choi, and Hannaneh Hajishirzi. Infini-rram: Scaling unbounded n-gram language models to trillion tokens. ArXiv preprint, abs/2401.17377,, 2024. URU/itpp:://arivi.orr///bs
17377
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. ArXiv preprint, abs/1907.11699 2019. URL http:htt::/// arxiv. wttp://..rg/bb///199990 11692]
14

---

Published as a conference paper at ICLR 2025
Liane Makatura, Michael Foshey, Bohan Wang, Felix Hähnlein, Pingchuan Ma, Bolei Deng, Megan Tjandrasuwita, Andrew Spielberg, Crystal Elaine Owens, Peter Yichen Chen, Allan Zhao, Amy Zhu, Wil J. Norton, Edward Gu, Joshua Jacob, Yifei Li, Adriana Schulz, and Wojciech Matusik. Large Language Models for Design and Manufacturing. An MIT Exploration of Generative AI, 2024. http:://mit-genai.uububhh pwwppuh...g/ubb/myyymmtt
William Merrill, Noah A Smith, and Yanai Elazar. Evaluating n-gram novelty of language models wrg.
using rusty-dawg. ArXiv preprint, abs/2406. 13069, 2024. URL
2406. 13069
Niloofar Mireshghallah, Justus Mattern, Sieun Gao, Reza Shokri, and Taylor Smaller language models are better zero-shot machine-generann generated text detectors. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 278 293. Association for Computational Linguisties, 2024. URL https://acc www. 2aw.. 25.
Piotr Mirowski, Kory W. Mathewson, Jaylen Pittman, and Richard Evans. Co-writing screenplays and theatre scripts with language models: Evaluation by industry professionals. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, CHI 2023, Hamburg, Germany, April 23-28, 2023, pp. 355:1-355:34 ACM, 2023a. htttss/ts:///et org/10. 2145/3544548 3581225
Piotr Mirowski, Kory W. Mathewson, Jaylen Pittman, and Richard Evans. Co-writing screenplays and theatre scripts with language models: Evaluation by industry professionals. In Proceedings of the 2023 CHI Conference Human Factors in Computing Systems, CHI 2023, Hamburg, Germany, April 23-28, 2023, pp. 355:1-355:34, ACM, 2023b. URLhttps: //oi. .ogg/1.. 10. 1145/354454 3581225
Eric Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher D. Manning, and Chelsea Finn. De- tectgpt: Zero-shot machine-generated text detection using probability curvature. In International July 2023, Honolulu, Hawaii, USA, vol- PMLR, 2023a. URL mlr wwwss/eee222 v202 mi tcchthehele. html
Conference on Machine Learning, ICML. 2023, 23-29
ume 202 of Proceedings of Machine Learning Research, pp. btttps:/pproceednngg
Eric Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher D. Manning, and Chelsea Finn. De- tectgpt: Zero-shot machine-generated text detection using probability curvature. In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, vol- PMLR, 2023b. URL mlr www...r.ess.............emmmee.e press v202 mi tche 1123a. html
ume 202 of Proceedings of Machine Learning Research, pp. htttps:/pprocedding
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. Don't give me the details, just the sum- mary! topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 1797-1807. Association for Computational Linguistics, 2018. URL httpsss//claahhoooogg...
D18- 118-1206
George L. Nemhauser, Laurence A. Wolsey, and Marshall L. Fisher. An analysis of approximations for maximizing submodular set functions i. Mathematical Programming, 14:265-299 1978.
Timothy Nguyen. Understanding transformers via n-gram statistics.
abs/2407.12034 2024. URL httthttts///axttt wrw.w.oorrr/oo. /argg//b///a 0240712204 ArXiv preprint,
OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Moham- mad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapiro, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brock- man, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux,
15

---

Published as a conference paper at ICLR 2025
Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elic Georges, Christian Gib- son, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hal- lacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Lukasz Kaiser, Ali Ka- mali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirehner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Fukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung. Daniel Levy. Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning. Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing. Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reücchir Nakano, Ra- jeev Nayak, Arvind Neelakantan, Richard Ngo. Hyeonwoo Noh, Long Ouyang, Cullen O' Keeefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filippe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Prochl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross. Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Sel- sam, Kyla Sheppard, Toki Sherbakov, Jessica Shich, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine B. Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Pre- ston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vi- jayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang. Ben Wang. Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Work- man, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao Tianhao Zheng, Juntang Zhuang. William Zhuk, and Barret Zoph. Gpt-4 technical report. ArXiv preprint, /ws/2333..0877,, 2023. URL https: /ttppii///axvv../// org/ att///ba/22333 08774
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kel- ton, Luke E. Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Francis Christiano, Jan Leike, and Ryan J. Lowe. Training language models to follow instructions with human feedback. ArXiv, abs/2203.02155 2022. URL http::::///pp..semmtttsstcooooae (CorpusDD:2442658))
Scott Paeth. 2 human creativity and a doctrine of creation.
Srishti Palani, David Ledo, George W. Fitzmaurice, and Fraser Anderson. 'i don't 't want to feel like i'm working in a 1960s factory":: The practitioner perspective on creativity support tool adoption. In CHI 22: CHI Conference on Human Factors in Computing Systems, New Orleans, IA, USA, 29 April 2022 5 May 2022, pp. 379:1-379:18, ACM, 2022. URL https:://oot 3501933
Stephen E. Robertson and Steve Walker. Some simple effective approximations to the 2-poisson model for probabilistic weighted retrieval. In Annual International ACM SIGIR Confer- ence on Research and Development in Information Retrieval, 1994. URL httpp:://aat
semant Beemaatt_cccccclaaaroooo/ooooppo 8552
16

---

Published as a conference paper at ICLR 2025
Piotr Sawicki, Marek Grze's 's Fabrício Góes, Daniel Brown, Max Peeperkom, Aisha Khatun, and Simona Paraskevopoulou. On the power of special gpt models to create and evalu- ate new poetry in old styles.
269589029
Piotr Sawicki, Marek Grzes, Fabrício Góes, Daniel Brown, Max Peeperkorn, and Aisha Khatun. Bits of grass: Does gpt already know how to write like whitman? ArXiv preprint, abs/2305. 1006, 2023.
Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Teraa Blevins, Danqi Chen, and Luke Zettlemoyer. Detecting pretraining data from large language models. In The Twelfth International Conference Learning Representations, ICLR 2024, Vienna, Austria, net fforumirummi
May 7-11, 2024. OpenReview.net, 2024. URL htttp:://openeew// zæqr 3MQuNs
Irene Solaiman, Miles Brundage, Jack Clark, Amanda Askell, Ariel Herbert-Voss, Jeff Wu, Alec Radford, Gretchen Krueger, Jong Wook Kim, Sarah Kreps, Miles McCain, Alex Newhouse, Jason Blazakis, Kris McGuffie, and Jasmine Wang. Release strategies and the social impacts of language models, 2019. URL http:: arxia. org/ abs /1908 09203
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Léonard Hussenot, Pier Giuseppe Sessa, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Alex Castro-Ross Ros, Ambrose Slone, Amélie Héliou, Andrea Tacchetti, Anna Bulanova, An- tonia Paterson, Beth Tsai, Bobak Shahriari, Charline Le Lan, Christopher A. Choquette-Choo Clément Crepy, Daniel Cer, Daphne Ippolito. David Reid, Elena Buchatskaya, Eric Ni, Eric Noland, Geng Yan, George Tucker, George-Christian Muraru, Grigory Rozhdestvenskly, Hen- ryk Michalewski, Ian Tenney, Ivan Grishchenko. Jacob Austin, James Keeling, Jane Labanowski, Jean-Baptinn Baptiste Lespiau, Jeff Stanway, Jenny Brennan, Jeremy Chen, Johan Ferret, Justin Chiu, Justin Mao-Jones, Katherine Lee, Kathy Yu, Katie Millican, Lars Lowe Sjoesund, Lisa Lee, Lucas Dixon, Machel Reid, Maciej Mikula, Mateo Wirth, Michael Sharman, Nikolai Chinaev, Nithum Thain, Olivier Bachem, Osear Chang, Oscar Wahltinez, Paige Bailey, Paul Michel, Petko Yotov, Rahma Chaabouni, Ramona Comanescu, Reena Jana, Rohan Anil, Ross Mcllroy, Ruibo Liu, Ryan Mullins, Samuel L Smith, Sebastian Borgeaud, Sertan Girgin, Sholto Douglas, Shree Pandya, Siamak Shakeri, Soham De, Ted Klimenko, Tom Hennigan, Vlad Feinberg, Wojeiech Stokowiec, Yu hui Chen, Zafarali Ahmed, Zhitao Gong. Tris Warkentin, Ludovic Peran, Minh Giang, Clément Farabet, Oriol Vinyals, Jeff Dean, Koray Kavukcuoglu, Demis Hassabis. Zoubin Ghahramani, Douglas Eck, Joelle Barral, Fernando Pereira, Eli Collins, Armand Joulin, Noah Fiedel, Evan Senter, Alek Andreev, and Kathleen Kenealy. Gemma: Open models based on gemini research and technology, 2024. URL
08295
E Paul Torrance. Torrance tests of creative thinking. Educational and psychological measurement, 1966.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuming Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. ArXiv preprint, abs/2307. 00002300700288 2023. URL
org/ abs/
Adaku Uchendu, Thai Le, Kai Shu, and Dongwon Lee. Authorship attribution for neural text gen- eration. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 8384 8395. Association for Computational Linguistics, 2020. URL https://aci antho ogy. org/0020 emn emn p- n. 673
17

---

Published as a conference paper at ICLR 2025
Adaku Uchendu, Zeyu Ma, Thai Le, Rui Zhang, and Dongwon Lee. TURINGBENCH: A
benchmark environment for Turing test in the age of neural text generation. In Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 2001-2016. Associa- tion for Computational Linguistics, 2021. URL https: / antho wotthlloy...o org/2011. Findings-emn 172
Maryam Vaezi and Saeed Rezaci. Development of a rubric for evaluating creative writ- ing: a multi- research. New Writing, 16:303 317. 2018. URL https::/aap)
semant cschollall ar.
98
Vivek Verma, Eve Fleisig, Nicholas Tomlin, and Dan Klein. Ghostbuster: Detecting text ghostwrit- ten by large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Vol- ume 1: Long Papers), pp. 1702-1717. Association for Computational Linguistics, 2024. URL
htwp:2002..aaa11oooo... 95
Hong Wang, Xuan Luo, Weizhi Wang, and Xifeng Yan. Bot or human? detecting chatgpt imposters with a single question, 2023. Utthttss: //axiv org/ abs/ / 2205. 06424
Sean Welleck, Jiacheng Liu, Ximing Lu, Hannaneh Hajishirzi, and Yejin Choi. Natural- prover: Grounded mathematical proof generation with language models. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Pro- cessing Systems 2022, NeurIPS 2022. New Orleans, LA, USA, November 28- December 9, 2022, 2022. URL 1Fc548a8243adD
html]
Kevin Yang, Yuandong Tian, Nanyun Peng, and Dan Klein. Re3: Generating longer stories with recursive reprompting and revision. In Proceedings of the 2022 Conference on Empirical Meth- ods in Natural Language Processing, pp. 4393 4479, Association for Computational Linguistics, 2022. BtLpttts:::/ hta.atttss//ccclacoooooogo wwwppoyy..gg22223/ emn emnlp-main. 296
Ann Yuan, Andy Coenen, Emily Reif, and Daphne Ippolito. Wordcraft: Story writing with large lan- guage models. Proceedings of the 27th International Conference on Intelligent User Interfaces, 2022. URL
Zachary Ziegler, Yuntian Deng, and Alexander Rush. Neural linguistic steganography. In Ken- taro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Confer- ence on Natural Language Processing (EMNLP-IJCNLP),, pp. 1210-1215, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10. (0.18853/v1/DI19 URL
18

---

Published as a conference paper at ICLR 2025
A METHOD DETAILS
A.1 IMPLEMENTATION DETAILS OF DJ SEARCH
As discussed in the main text, the deployment of the CREATIVITY INDEX relies on efficiently deter- mining whether each n-gram X;+i+n E can be found anywhere in the massive reference corpus C of publicly available texts. The function fxx++s-n C) is a binary indicator that determines whether an n-gram X occurs in C. In line with the definition of CREATIVITY INDEX, only consider the n n--grams Xiti+n such that n 2 for some fixed constant L.
While naive approach to checking whether X appears in C' for every n-gram Xi+i+n would take O(Ix 2) to f (see Algorithm 1]. using a two-pointer approach we can radically reduce this O(xx)) calls (see Algorithm 2). Note that two-pointer approach does O[xx]) calls to f since in each iteration we advance at least one of the two pointers and j by 1, and 0 S i, S [x].
Algorithm Naive Computation
NGramsFound; j + False Vi [0..|x|) and j E [0..|x]) >> matrix to store n-gram occurrence for i [0, 1, x L) do
for j E i L. xx)) do
NGramsFound (i, + (xxaj C)
end for end for return NGramsFound
Algorithm 2 Efficient computation of DJ SEARCH(X,C)
NGramsFound:; False V E [O../x)) and j E [0..|x]) matrix store n-gram occurrence i + 0, <- L
while [x| do NGramsFoundd(..) = f(x;j,, C)
NGramssound then
j + +1
we will search for Kz:++1 next
else
i i -+i-1 j max(i + L,j)
I since K.ÿ was not found, Xaj+s will not be found for all k 0 D- we only explore L-grams and beyond
end if end while
return NGramsFound
A.2 IMPLEMENTATION DETAILS OF WORD MOVER'S DISTANCE
Let w be an n-gram. Let fww, C)) be the function that determines whether w appears in any text E C, either exactly or as a phrase that is highly similar in meaning to w (e.g.. a paraphrase of w). Trivially, f(w,CC C) == Uaec fww, d), and here on we will only discuss how to compute f(w, d).
An established approach for finding semantically similar phrases given n-gram w is compute its embedding enbedding--embedding embedding(()) and and then independently compute its similarity to the embeddings of all other n-grams to analyzed. An embedding of a n-gram a vector that represents the mean- ing of such n-gram in k-th dimensional space of fixed size, enabling the comparison of similarity between concepts expressed in different surface forms. This comparison is typically done using cosine similarity, the scaled dot product between the two embeddings being compared. Text embed- dings are generated by models specifically trained to this effect (e.g., BERT (Devlin et al.) 2019), RoBERTa (Liu et al. 2019), SpanBERT (Joshi al. (020)) making their computation expensive a large scale. Notably, text embeddings usually do not possess linearity, i.e. the embedding con- catenating n-grams and cannot be deduced from knowing embedding(w) and embedding(r)). and instead needs to be computed from scratch.
'There ([x) [xx[ T.+ 3)/2 spans to analyze if L. is the minimum n-gram length to be considered.
19

---

Published as a conference paper at ICLR 2025
Since goal is to find the n-grams of d that highly similar to w, using the traditional approach would entail comparing embedding( (w) with the embeddings of all n-grams in C, which are approx- imately Edec [d]2 in number. Note that this also implies independently computing s Edec |d] emheddings, which increases the computation costs significantly. Instead we use Word Mover's Distance (Kusner al.) 2015) (WMD)), method to estimate similarity between
n-grams by
combining comparisons between pairs of word embeddings. This enables lifting the requirement independently computing the embedding for each r--ram in C. Concretely, the Word Movers'
Distance between two n-grams and v is defined as follows:
Dw-
min 1 cosine cosine.simlartyymmmdddieggg))))) embedding(w:)) [ol) (e()
cosine_iimlarriyemmmddddngggg))) e sE(O..[o)
w
iE(o. [e))
WMD(w, v) max D De D D
WMD also pre-filters the words considered in w and v to only include the content words in the analysis (i.e, discards stop-words, such as the, a, an, it, on, ...).
Note that D D 's definition is asymmetric wD # Do-ww). Thus, consider the Word Movers' Distance of two n-grams w and v as the maximum of D w- and Du- w and v are highly similar if their distance is below threshold 5 for both D Dw-te and De-ne (See Algorithm 3):
WMD(w, u) max(D D De-ww <
Algorithm 3 Conceptual writeup of f [w, d) using Word Mover Distance (WMD) find the n- grams of single text d E C that are highly similar to the n-gram and are of length 2 L.
procedure DIRECTIONALWMDE
v)
return
Σ Eje(o.|o)) max;c]o.1[l) cosine aaaiiisiiiiiarrlyyemmedddnggug hatp:///mmddddooooommmmmmeeenn))
embedding(w;))
end procedure
for E [0, 1, ..., dd) do for b b [a + L.... [d]] do
WymmetrieWMD&
max dindiiiiccion w), directionnallMMD d(a b)))
if WymmettricWDD < 5 then
return True
end if end for end for return False
Avid readers may notice that Algorithm 3 repeatedly computes the maximum over the same set,
and sums of contiguous similarity scores; these can be pre-computed. Algorithm 4 shows these optimizations, resulting in algorithm of time complexity O(Id, |w| + |d] w|l O([d)(uw)) assuming already computed word embeddings. Note that because there is a fixed vocabulary, all word embeddings well cosine similarities of word embedding pairs can be pre-computed.
We described how to compute f(w, d) for a single document E C, as we have already established that (w, C) = Uaec f d). To accelerate computation, and given that similar n-grams to X are more likely to occur in texts similar to x, we select C's top most likely documents to contain using BM2 & Walker (1994) index, denoted C². We then approximate f(w, C*) s
Uaeo f(w, d).
As a final optimization, we note that it is unnecessary to compute the costly fww, C) for finding semantically similar matches for in the case where appears exactly in C. To check if appears exactly in C, we can leverage the existing, less expensive approach Infini-Gram (Liu et al, (024) and search for the semantic similar matches only if Infini-Gram could not find any exact matches.
20

---

Published as a conference paper at ICLR 2025
Algorithm 4 Efficient Computation of f(w, d) (optimization of Algorithm
token similarity,, 4.j t- cosine sinilaritylembeeddnggw))) hamp://dddnggtt.mmmmmmmmmddddd)) embedding(d,))
4y i E [O..|w]) and
[0..|d])
for j E [L...., [d]] do ceo.prffixsimmilriii - doc. preffiii.immllaitty... + max;e(0..Jr)) token.similarity...
end for
for [0, 1,.,, d do
for b 6 T..... [d) do
Computed_WMD(CC doc flc_pprefi_ssiie www.ssiiiiiaa/////...aaal (/ a)
b)
(doc-prefix pono-prefxxssmilarity..
computed Crmputed_WMD(eee aa b)) - 1 Eie (o......i mn)) maxje(a..b) tokent similarity,,
w
symmetric. WMD max (computed LMttttedddddMddddd WMD(d 6), w), computed WMD(w, d(a b)))
symmetric WMD < then
return True end if end for end for return False
A.3 DEDUPLICATION OF THE REFERENCE CORPUS
When text is part of the reference corpus C, its CREATIVITY INDEX would trivially become zero. This issue often arises when analyzing the works of famous authors, as their writings are frequently copied, quoted, before the cutoff date of the reference corpus, we exclude any document E C that contains copies, quotations, citations of the text x from the reference corpus C, and compute CREATIVITY INDEX of x using this filtered reference corpus.
cited online. To address this, when analyzing human texts written
Specifically, measure the degree of overlap between x and d by calculating the length of the longest common subsequence (LCS) between them, normalized by the length of x. Formally, ((x.d) [CC(xx))]] If and d have a high degree of overlap (i.c., S(x, d) > a), it's very likely that |x]
contains an exact copy of x. If x and show a moderate amount of overlap (i.e., 8 S(x, d) < a), prompt LLM to determine whether contains copies or quotations of using in-context exam- ples provided below. Additionally, if includes the author name or title of x, is highly likely that d contains citation of x. In practice, set the values of and / to 0.9 and 0.3, respectively, and use LLaMA Chat as the L.M to check for copies and quotations.
To determine the optimal value for and 3, analyzed the relationship between the degree of overlap S(x, d) and the likelihood that the reference document d contains an exact copy or quotations of the input text
Degree of Overlap S(x, d)
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
Percentage of Exact Copy
0.0
0.0
0.0
0.03
0.12
0.28
0.49
0.77
0.94
1.0
Percentage of Quotation
0.0
0.02
0.04
0.11
0.15
0.13
0.05
0.02
0.0
0.0
We found that when S(x, d) < 0.3, there text x in the reference document d. Conversely, when S(x, d) > 0.9, the reference document almost always contains exact copy of For cases where 0.3 S(x, d) 0.9, all three scenarios (exact copy, quotation, or none) are plausible. Therefore, we set the values of a and 3 to 0.9 and 0.3 ensure accurate deduplication.
exact copies, and almost quotations of the input
21

---

Published as a conference paper at ICLR 2025
Please check if paragraph A contains any ccpies quotations from. paragraph
Here are some examples:
Paragraph A: In the end though, I did the required reading, complained bitterly about being bared, wrote the requisite essay, and promptly forgot all about the Gulf Stream and he had gone eighty-four days now with taking a fish. In the first forty days
old man who fished alone in skiff
Paragraph B: He was an old man who fished alone in a skiff in the Gulf Stream and he had gone eighty-four days without taking fish. the first forty days a boy had been with him. But after forty days without a fish the boy's s parents had told him that the old man was now definitely and finally salao
Answer: Yes
Paragraph A: He was an old man who fished alone in a lobster boat off the Maine coast and he had gone 117 days without taking crustacean. His luck not baa, rather his judgment winter). Then
good (don't fish the Atlantic and for all know his luck changed. R1 Campion is due for change of luck Paragraph B: He was an old man who fished alone in a skiff in the Gulf Strean and he had gone eighty-four days new without taking a fish. In the first forty days boy had been with him. But after forty days without a fish the boy's s parents had told him that the old man was now definitely and finally salao Answer: No
Paragraph Santiage, the "old man who fished alone, in Hemingway's s "The 0ld Man and the Sea" appears as one who has an unde character, loving, cheerful character, and very humble. The writer describes him in this way: "Everything about him eyes, and they were the same color as the sea Paragraph B: He was an ald man who fished alone in a skiff in the Gulf Stream and he had gone eighty-four days new without taking a fish. In the first forty days boy had been with him. But after forty days without fish the boy's parents had told him that the old definitely and finally salso Answer: Yes
old except his
Paragraph He was an old man who could see the form of his god, and monk, moreover. Izzie had limited ability to communicate directly with her own deity. Much of her life she had proceeded by vague impressions and only glimpsed the great god's image briefly in the depths of meditation ...
Paragraph B: He was an eld man who fished alone in a skiff in the Gulf Stream and he haa gone eighty-four days now without taking a fish. In the first forty cays a boy had peen with him. But after forty days without fish the boy's parents had told him that the old man definitely and finally salao Answer: No
Here is the test example:
Paragraph A: [A]
Paragraph B: [B]
Answer:
22

---

Published as a conference paper at ICLR 2025
B EVALUATION
B.1 MACHINE TEXT GENERATION
We experiment with state-of-the-arr LLMs: GPT-3 (Brown et al.
2020) meext-davincee 003)),
ChatGPT (Ouyang al. 2022) (gpt-3. 5 5-turbo), LLaMA Chat Touvron al. 2023 Tulu
2 Ivison et al. 2023) and OLMo Instruct (Groeneveld et al. 2024) along with their base model betore RLHF: GPT Base (Brown et al. 2020) (davinci-002 LLaMA Base (Touvron et al. 2023) and OLMo Base (Groeneveld et al.) 2024). These models are primarily pre-trained on the web data available before the cutoff date of the reference corpus RedPajama (Computer 2023). We
additionally discuss how to handle newer models, such as GPT-4 (OpenAl et al. (203)), which are largely trained on more recent web data and third-party private data, both of which fall outside the reference corpus RedPajama.
To obtain machine texts, we prompt LLMs generate several paragraphs of novels, poems, or speeches, starting with an initial sentence taken from existing human writings in each category. To construct test data for machine text detection, we further prompt LLMs to generate news articles
based on the fake news headlines from the Misinfo Reaction Frames (Gabriel et al. 2022) and to
generate theorem proofs for questions from the NaturalProofs (Welleck et al. 2022) benchmark. The prompts used for each task are illustrated below. For all generations, with = 0.9 and set the maximum length the generated texts to 288 tokens.
nucleus sampling
Please write few paragraphs for novel starting with the follewing prompt: [PROMPT SENTENCE]
Please write a poem starting with the follewing line:
[PROMPT LINE]
Please write a speech starting with the following sentence: SENTENCE]
[PROMPT
Please write a news article based on the given headline: NEWS HEADLINE
Please provide proof for the following theorem:
(TEORREM QUESTION]
To obtain model-generated reference corpus to compare the CREATIVITY INDEX of GPT-4 with humans, we randomly sample 150k sentences from the RedPajama corpus and prompt open- weight LLMs with knowledge cutoff dates similar to GPT-4 generate document-level contin- uations. The models we use are the instruction-aligned versions of Gemma-77 7B (Team et al, 2024) Ccemma-b-itt
(AI@ Meta 2024 and Mixtral-7B Siistral-/B-vo. (Jiang et al. 2023). The prompt used to generate continuations illustrated below. We use nucleus sampling with 0.9 and set the maximum length of the generated texts 2048 tokens.
Llama3-80 8B -Meta--Laam--3-e
Please generate a continuation for the fellowing sentence: SENTENCE
[PROMPT
We additionally experiment with different prompt formats, intentionally encouraging creativity in models' generations by incorporating instructions such as 'push for creative ideas, unique emotions, and original twists." 'be bold and creative,' or 'you are creative writer.' Please see blow for full list of the prompts we tried.
Write a few paragraphs for novel from the following prompt, pushing for creative iceas, unique emotions, and criginal twists. Prompt: [PROMPT SENTENCE]
23

---

Published as a conference paper at ICLR 2025
Use the following prompt to write few paragraphs for a novel with creative, ungiue perspectives twists. Let your originality shine. Prompt: [FROMPT SENTENCE]
Create a few paragraphs from the following prompt for a novel, focusing on novel ideas, emot smcticns, or perspectives. Be as creative as possible. Prompt:
[PROMPT SENTENCE]
Write Few paragraphs for novel based the following prempt, exploring unexpected twists, emotions, or unique perspectives. Be bold and creative.
Prompt: [PROMPT SENTENCE]
Based the follewing prompt, and write few paragraphs for novel that explore unexpected twists, deep emeticns, or unique perspectives. Let your creativity flow, and don't be afraid to experiment with unconventional ideas or characters Prompt: [PROMPT SENTENCE]
As creative agent, write few paragraphs for novel based the following prompt, bringing your novel ideas and original emotions to 1ife.
Prompt:
[PROMPT SENTENCE]
You are creative writer, write few paragraphs for novel based the following prompt. Explore unique perspectives and unexpected twists, and 1et your creativity guide you. Prompt: [PROMPT SENTENCE]
You are a creative agent, free to shape this story in any direction. Write few paragraphs for novel based the following prompt, using surprises and depth.
your imagination to Prompt: [PROMPT SENTENCE]
As a creative writer, your task is to write a few paragraphs for à novel based on the following prompt. Dive into original ideas, explore emotions, and surprise yourself. Prompt: [PROMPT SENTENCE]
You are a creative writer who brings stories to 1ife. Write few paragraphs for novel based on the follewing prompt, letting your imagination take bold, unexpected turns. Prompt: [PROMPT SENTENCEI
B.2 DATASET DETAILS
Reference Corpus: We use RedPajama (Computer) 2023). the largest web data collection avail- able at the time of this study, as our reference corpus. RedPajama contains 100 billion text docu- ments with 100+ trillion raw tokens from 84 CommonCrawi dumps.
Novel: For human-written novels, we use book snippets from the BookMIA (Shi et al., 2024) dataset. The BookMIA dataset contains approximately 10k book snippets, with an average length of around 650 words per snippet. We randomly sample 100 book snippets from the BookMIA dataset and select the first K sentences of each snippet such that their total length exceeds 256 words, to use as human text. Since novels we use were published after the cutoff date of RedPajama, there's need for deduplication before DJ SEARCH.
24

---

Published as a conference paper at ICLR 2025
Speech:
For the transcripts of human speeches, randomly sample 100 speeches from the fa- mous speeches available in the American Rhetoric speech bank. For each speech, we randomly sample continuous K sentences such that their total length exceeds 256 words, to as human text. Since these speeches were made before the cutoff date RedPajama, deduplication is needed before DJ SEARCH.
Poem: For human- poems, we randomly sample 100 poems from the popular modern po- ems collected by PoemHunter.com com Since these poems were published before the cutoff date of RedPajama, deduplication is needed before DJ SEARCH.
We use news articles from the XSum (Narayan et al. 2018) dataset as the human text for the machine text detection task. The Xsum dataset contains around 200k new articles, with an average length of around 380 words per article. We randomly sample 500 articles to use as human text. Since these news articles were released before the cutoff date of RedPajama, deduplication is needed before DJ SEARCH. For machine- generated fake news, we randomly sample 500 fake news headlines from the Misinfo Reaction Frames (Gabriel et al.) (022), and based on these headlines, LLMs are asked to generate corresponding news articles.
News Article:
Theorem Proof: We use the ground-truth human- proofs from the NaturalProofs (Welleck) et al. 2022) dataset as the human text for the machine text detection task. The NaturalProofs dataset contains approximately 24k theorems and their corresponding proofs. We randomly sam- ple 500 theorem-proof pairs and use the ground-truthh proofs as human text. Since the NaturalProofs dataset was curated after the cutoff date of RedPajama, there's no need for deduplication before DJ SEARCH. For machine-generated math proofs, we prompt LLMs to write proofs for the 500 theorems we sampled.
B.3 PARAMETERS OF DJ SEARCH
We set the minimum n-gram length L in DJ SEARCH to 5, and set the threshold for Word Mover's Distance to 0.95 for semantic matches. We observe that the L-uniqueness is close to zero for most human and machine texts when L < 5 and close sum up the L-uniqueness for 5 < S 12 when computing CREATIVITY INDEX.
when
12. Therefore, in practice
The only experiment with slightly different parameters is compare the creativity of GPT-4 with humans. We observed that the i-uniqueness close to one when I. 2 7 based on the model- generated reference corpus. Therefore, sum up the T-uuniqueness for 5 s when computing CREATIVITY INDEX.
B.4 RUNTIME OF DJ SEARCH
To verify linear complexity, we measured the runtime of DJ SEARCH with respect to increasing input text lengths, using RedPajama as the reference corpus, restricting the matching criteria to verbatim matches only and averaging the results over 500 examples.
Input Length
64
256
512
Runtime (seconds)
9.2
16.9
30.7
62.3
We observed that the runtime of DJ SEARCH indeed increases linearly with the length of the input text. We also measured the runtime of DJ SEARCH with respect to increasing sizes of the reference corpus while keeping the input length fixed 256 words. In addition to RedPajama. which contains approximately 1.4 trillion tokens, we experimented with the Pile (380B tokens) and DOLMA (2.6 trillion tokens).
Reference Corpus Size
380B Tokens
1.4T Tokens
2.6T Tokens
Runtime (seconds)
24.1
30.7
37.1
We found that the runtime increase for DJ SEARCH is minimal the size of the reference corpus grows. This is likely due to the high optimization of the Infini-gram, which enables it to retrieve matched n-grams within milliseconds.
25

---

Published as a conference paper at ICLR 2025
B.5 VARIATION OF WORD MOVER'S DISTANCE
We experimented with different variations of transport- inspoieed distances, including relaxed WMD, supervised WMD (Huang et al. 2016), and Sinkhorn distance (Cuturi) 2013).
We computed the CREATIVITY INDEX for ChatGPT and human writing in the novel domain based on semantic matches using relaxed WMD, supervised WMD, and Sinkhorn distance. Regardless of the distance metric used, human writings consistently exhibited a significantly higher CREATIVITY INDEX than ChatGPT, aligning with our main results. Specifically, based on relaxed WMD, the CREATIVITY INDEX of human authors was 97.6% higher than ChatGPT (p 2.5 x 10 13, N 100). Based on supervised WMD, the CREATIVITY INDEX of human authors was 79.3% higher than ChatGPT (p- 1.9 10-7; N 100). Based on Sinkhorn distance, the CREATIVITY INDEX of human authors was 91. 133% higher than ChatGPT (p = 3.5 x 10-12; N = 100).
As mentioned in the main text, the motivation for using transport-inspired bine word embedding distances for each n-gram's words rather than directly computing the Eu- clidean distance between embeddings of two n--grams-is rooted in efficiency. The latter requires computing n-gram embeddings for every n-gram in the input text and every document in the ref- erence corpus during DJ SEARCH, which is computationally infeasible. Ideally, a good transport- inspired distance should approximate the Euclidean distance between the embeddings of two n- grams. To evaluate this, we computed the Pearson correlation between our implementations of WMD, relaxed WMD, supervised WMD, and Sinkhorn distance with the Euclidean distance be- tween the embeddings of two n-grams to assess how closely they correlate.
com-
MWD
Relaxed MWD
Supervised MWD
Sinkhorn Distance
r-value
0,842
0.825
0.716
0.808
p-value
5.7 10-11
4.1 x 10 10
8.3 10
9.1 10
We found that all transport-inspired distances correlate highly with Euclidean distance, showing strong statistical significance. The slightly lower correlation with supervised MWD may be due to its training objective, which focuses on distinguishing between different categories of documents.
B.6 COMPARE WITH EXISTING CREATIVITY METRICS
A recent study by Chakrabarty et al. 2024 evaluates the creativity of stories written by LLMs and professional human authors through expert evaluations based on carefully designed rubric inspired by the Torrance Test of Creative Thinking (TTCT). This work introduces the Torrance Test of Creative Writing (TTCW), which consists of 14 binary questions grouped into four dimensions: Fluency, Flexibility, Originality, and Elaboration. To conduct the evaluation, ten expert creative writers were recruited to evaluate 48 stories-writter either by LLMs or published in The New York Times-using the TTCW rubric.
For each story, three expert evaluators are tasked with answering 14 binary questions, each designed to quantify specific dimension of creativity. For example, within the Fluency dimension, "Un- derstandability & Coherence" is assessed by asking. "Do the different elements of the story work together to form a unified, engaging, and satisfying whole?" In the Flexibility dimension, "Emo- tional Flexibility" is evaluated with the question, "Does the story achieve a good balance between interiority and exteriority, in a way that feels emotionally flexible?" Similarly, within the Originality dimension, "Originality in Thought" is measured by asking, "Is the story an original piece of writing without any clichés?" For each of the 14 binary questions, the final answer is determined by majority vote among the expert evaluators.
To evaluate the correlation between our CREATIVITY INDEX and [hakrabarty al. TTCW, we compute the CREATIVITY INDEX for each of the 48 stories. For each binary question in the TTCT (c.g., "Understandability & Coherence" or "Emotional Flcxibilityy), we calculate the point biserial correlation between the CREATIVITY INDEX values and the corresponding binary expert ratings of that question across all 48 stories.
We found that the CREATIVITY INDEX positively correlates with all dimensions of TTCW, showing particularly strong correlations with high statistical significance in dimensions such as "Emotional Flexibility." "Narrative Pacing," "Structural Flexibility," and "Originality in Thought."
26

---

Published as a conference paper at ICLR 2025
Emsotional Flexibility
Narrative Pacing
Structural Flexibility
Originality in Thought
Scene vs Summary
Rhetorical Complesity
Language
r-value p-value
0.6826
0.6524 0.0051
0.5968
0.5968 0.0355
0.5812 0.0135
0.5810
0.5702
0.0028
0.0135
0.0135
0.0204
World Building Setting
Narrative
Originality in Form Structure
Understandabilit Coherence
Perspective & Voice Flexibility
Originality in Theme Content
Choracter
Ending
Development
r-vulue p-value
0.5170 0.0426
0.4871 0.0617
0.4626 0.0816
0.4495
0.4467 0.0969
0.3582 0.1231
0.2299
0.0941
0.1453
C RELATED WORK
Measuring Creativity in Ideas: Measuring creative thinking and problem solving takes root in
early work in psychology [Torrance 1966). where researchers defined four pillars for creative think- ing: fluency, flexibility,, originality and elaboration. [Crosslyy et al. (2016) later on developed this notion and built on expand this to measuring creative writing in students, where they also adopted n n-gram novelty for measure of originality. However, these prior work focus on creativity in humans, and they also do not introduce any automated metrics or measurements.
Measuring Creativity in Machine-generater Text Using Expert Annotators: Closely related to CREATIVITY INDEX is a recent line of work in the generative AI literature comparing the creativity of human writers that of large language models different domains such as story telling and journalism (Chakrabarty et al. 2023 2024 Anonymous 2024). Similar to us, the approach in this direction often involves prompting an LLM to write an original story or news article, based on some existing premise press release, and then comparing the machine-generan
text to the human- written counterparts. These works, however, take a rather subjective approach, where they define and measure creativity based human expert annotations and whether people perceive the text to be more creative, rather than an objective quantification of novelty that we provide.
Measuring Novelty of N-grams: Finally, closely related to our work in terms of techniques is Nguyen (2024) and Merrill et al.] (2024). The former attempts at finding n-gram rules that would cover and predict generations from transformer models, showing that more than 70% of the times transformers follow some pre-set patterns and rules. The latter is more similar to our work they also measure the novelty of generated n-grams and compare it to human-written text, however they differ from us in tow major ways: (1) they only find verbatim matches, whereas we also match to approximate, semantically similar blocks text and (2) they compute the percentage of n n-grams of certain length in text that be found in the reference corpus, whereas measure how much of the text can be reconstructed by mixing and matching a vast amount of existing text snippets of varying lengths from the web.
Machine Text Detection: Detecting geeerattee text has been explored for several years using a variety of methods (Jawahar et al.] 22200] [Cchnnu et al.] 2021). Gehhmann et al. (2019) and Dugan al. (2023) demonstrate that even humans tend to struggle differentiate between text written by humans and machines, highlighting the need for automated detection solutions. Some approaches involve training classifier
supervised manner to identify machine-generated text (Bakhtin et al. 2019 Uchendu et al.) 2020), while others use zero-shot detection method (So-
laiman al. 2019 Ippolito et al. (2020) Additionally, there research on bot detection through question answering (Wang et al. 2023 Chew & Baird 2003). Recently, Mitchell et al. (202bb) introduced DetectGPT, a zero-shot method based on the hypothesis that texts produced by large language model (LLM) are located at local maxima, and thus exhibit negative curvature, in the model's 8 probability distribution. Follow-up work build on DetectGPT by making it faster (Bao) ct al. 2024) and proposing to use cross-detection when the target model is unknown (Mireshghallah ctt al. 2024).
Various strategies have been developed to detect machine-generated text in real-world settings. One notable approach is watermarking, which embeds algorithmically detectable patterns into the gen- erated text while maintaining the quality and diversity of the language model's outputs. Initial watermarking techniques for natural language were proposed Atallah et al. (200T) and have been adapted for neural language model outputs (Fang al. 2017 Ziegler al. 2019). Recent advance-
27

---

Published as a conference paper at ICLR 2025
ments include Abdelnabi & Fritz (2021) work adversarial watermarking transformer (AWT) for transformer- based language models. Unlike methods dependent on specific model architectures, Kirchenbauer et al. (2023) introduce watermarking technique applicable texts generated by any common autoregressive language model.
Application of LLMs Creative Writing: Recent advancements have highlighted the potential of LLMs in supporting various creative writing endeavors, ranging from short stories (Yang et al. 2022) to screenplays (Mirowski et al., 2023b). Enhancing LLMs to produce text that aligns more closely with human preferences has made them adept at following user instructions, thereby turn- ing them into valuable tools for individuals without technical expertise. This progress has boosted the commercial viability of LLMs as writing aids. which can continue a narrative, describe scenes, or offer feedback. [Chung et al. (202T) conducted a review of literature on creativity support tools across various arts, leading the development of taxonomy that includes roles, interactions, and technologies. In contrast, Frich et al. (2019) and Palani et al. (2022) focused on how creative prac- titioners select new tools, highlighting their emphasis functionality, workflow integration, and performance, and noting that personal recommendations often guide their choices, Additionally, Gero et al. (2022) created space based the cognitive process model of writing, influencing inter- face design decisions. Gero et al. (2023) further explored the social dynamics of AI in creative tasks, revealing disconnect between writers' objectives and the support provided by computer tools.
28

---

Published as a conference paper at ICLR 2025
Reference Copus
Please write a few peragraphs for a novel starting with the following prompt: tumned flarce scow! inward."
Verborin Match
Masas Virtiation
turned fierce scowl inward. The Su Sultana's nonsense lovers had clearly scrambled my obvious
Arin attractive and indisputable is as the sun. But I had spont nearly twenty-ona years capable of acknowledging attract veness without being attracted myself. had never yeamed for the physical relationships Marak chased. finally empathized with the girls keep. Espacially Gana. fanciful used dress in Raya's finest gowns avery week, sing warbling ballads while dabbiig fragrances fon her wrists and banind her ear. "Thare is power conquering the unconquerable; Gana had year, rejecting yet another fellow's advences. The keep had gone to Zeila's for celebratory tea and alwa after a successful market. Zeils laid reed rugs on the floor, and we set on beaded cushions, wooden table wobbling feet. I'd been few cushions down with Sefa and Marek, sisping my
wanted aryone,
Sara Hashem
and stuffy nose. love wearing perfume- day is not complete if carnor dal perfuree whess, behind eer/and my
... don't want to be around people. A lot of it bolls down dealing with anger that's been bublling up naide. I like feeling don't like
anger...
turned fierce scowl inwerd, feeling (he anger bubbling up inside me. llooudn'' tibelieve had winesssd, My bes: friend, the one person trusted more than anyone else, had just betrayed me in the worst way possible. had] caught her red handed, lying straight my and now didn't knowlff could ever forgive her)
right. I shared cashier there minute because just couldn't be leve what just witnessed) All could say was what
GPT-3.5
-the summeer after freshman lanon broke up because he pressunng lost was one person llooked up
poing to hard get
again,
37. instte her with all my heart, but she botroyed me the wonst way possible. You willineer forgiven for this.
As I stormed out her hause, the weight of her my chest. How could she do this me? had always been there for her, through thick and thin, and now it seamed! like she had thrown it all away inja instant. mix of emotions betrayal, hurt, loss. Our friandship had bean one of thel
was hornlled see hor real personality, complete lack empatiyy the lying striight lacell same subject...
deception
heavy
better myself. my depression was intensel heavy on my chest My awereness was
I never made the
aftermath
the girl haldd dept with She called me and sold she hadn't known Mickey had gir and could forgue Ferlj) teld har wain' her feal. gol
and above deep
... living. My wife, Elzabeth, and I needed an occasion flxed on some praximity to her to focue the awful sense loss, Our frinccl ip had been
seconds everal. looked like thrown away the gates, with what ooked like a elight hapse in concentration, but tha American.
Figure 5: Example outputs from DJ SEARCH based both verbatim and semantic matches. We prompt LLMs to generate a few paragraphs of a novel, beginning with a first sentence taken from a human-writter novel snippet.
29

---

Published as a conference paper at ICLR 2025
Please write few paregraphs for novel starting with the following prompt: "I turned fierce scowl inward."
Reference Copus
Verberin Match
Or rather, my jeans couldn": ftt into mo! felt like I've been punched in the storrach. AA staned rellection the mirror I asked mysel: How could this happen?
Mosa Nhet-thettatinn
As stared Bt my reflection in the mirror, [turned a fierce scowl inward (My frustration and anger were directed at mmysellf allowing others control myy life. was fired of constandy/ppttingg on smile and pretending that everything wes okay. But in Jeality, was drowning in my own misery.
honustly angry about that happoned in the past A few things, but not a lot Much of my frustration and anger directed towerdd mysel but I reelize...
GPT-3
stains, to the betroom, just my my other sull unrianed, reacceed hand out the back the mimor, refeccion turned hand
ann before you know there been a lieemme of tomoroos. was tired of not reall living try life was tined of constartlyy worrying about things that have no
me,
glass....
lind of deep reforms that Western states want to see. Bahrain was bacome
hac always been people pleaser always putting others before myself, But in doing so.
accomplishod od atiputting on a smile and
had lost my own identity. didn't tevvn recognize the person looking back
weigh the pros/cons of going of your way the moming Ive needed this lvv fshoays been a people
pretending that everything fine...
the miiror. Who was she? What clid she wart? cuestiors swirled in my mind as Joontinued to scowl at my reflaction.... ...
time fronting. ... Good people protec:
pleaser. I was always teased growing...
people they love even if that meons that pretanding that everything okuy Kirk
Lazanus: Man were lost, were super looe!...
...bbvve and beyond the call of duty in life I son't know who am any longer.
To know that you
looking back at me the mirorr
recogniee
person cuitivatedd that smile. It's the greatest feeling. am stways putting others before mysel and that jmakes me...
-idl like her or did hate her? Or did
love hate [hese questions swifled my (read book. story slar led simply with Emily and Paolo...
... -years, For while I cried whon was soon recogrize phatsprroon the minor looking back me. Butjall I turned on the IV. ...
tureed a fiecce scowl inward, directing (my
rry major decoession is accompanied by
anger and frustration atymyslll. couldn't !
quito lat
and frustration
"I had the biggest crush him" know"
eve that had things get Tists were clenched, and my
point My
mysei aie Self-contempt.
walked the bethrooms, (jew tight, fists dlenchecl jristy eyes... I ssa people looking at me with..
tight paced the length of the small, cramped room. Howw could have been so toolish? trusting?
knowing deep down that wull never
eating and self leathing I started crying, becausel coulddr ebbelivel I let things
echojot my heavy footstops was only sound the room as Itried to tamp down the]
point. was cryingg stopped my body so much.
hated
Tulu2-70B-dpo
fear and disbelief that threatened to consume mo.
was doing it myeelf, the hole would close up and no power could come through. Then everything Icould do would be foolish Alspoon
Slowly, with Feasy footatops eccho, the jiatus enters. Another word for footed The footsteps are light and springy.....
shudents
The dim light overhead cast ominous snadows across the walls, making the space feel even more claustrophobic. My thoughts raced through my mind, a Ehaotic
so he was often at friend's 's's house and not home. One night, I woke up to the sound of hasvy foorsteps his After) rying go
What's the way out of here? As surveyed
... drink. To have come for in.
olame and
sleep
room, tried to tamp down myddread
recrimination. I should have seen this..
reading FAST FUTURE, the themes of ertilement, lasiness, and inflated salf esteoms faced through mind. These thoughts through my mindi
Which door should choose? A hint....
remorse or recrim nation. (25) Perhaps unsuprisingly die Boor
voice amid the daaour criblame ation. (We must join in
Figure 6: Example outputs from DJ SEARCH based on both verbatim and semantic matches. We prompt LLMs to generate a few paragraphs of novel, beginning with a first sentence taken from human-writte novel snippet.
30

---

Published as a conference paper at ICLR 2025
Please write few paragraphs for novel starting with the following prompt: "I turned fierce scowl inwerd."
Reference Carus
Verbeii Match
other people talk anymore, I uat have this Insane hatred brewwng inside day
dehumanizing, ccnn't stand hearing
turned fierce scowl inwerd, trying to mask the turmoilthet was brew inside me. Myees narrowed aadd lips pressed into thin line, struggled to keep my emot in check Il had always been a person who wore their heart on their sleeve, in this nomentt, knew had to keep my feelings hidden.
much mask
OO
acceptted beer slid way
glance swung toward
Brown
whose lips pressed into thin line You ve already get one, gave soft luug,,
Llama2-708 Chat
My mother was just starting wake
and knww had no chance with her
has wronged me. feel but I'm tending to Fiche the allects thow amotion falwws been who heart their
sinuggled keep my mumbled "oour wooks" as check]
The weigght of the world seemed to bel bearing cown on me, and felt like on the verge collapse. My mind uncortainty and fear, as tried to make sense of the chaos that was unfolding around ma. falt likel was stuck in nightmare that couldin't wake from, and dion)...
outsidoo the Inn, Harnah saw Robin
Park come out of the Jazz Lounge [The weightt of the world seemedt to bel puching down his namoo shour dees he
racing with thoughts of
But sisie''s devastating bet ayo rivealad, Parge con't kees her feelinss hidden any longer I've wanted Paige and
soor-to-be
the streets, I'd slready experienced far middle age
... hatedl heatstroke. Samehow, dragged my ass to the airport and got on the plane, even though I felt like was the verge of vary lashionallle
bearing down on me, and like lwae bearing up under it precty well. And ...
collapuse.
want to lose you tool" she said her voice Emma watched the snarky commant leavelher mouth before she realized what she had just said.
slihtly reised the and. The words came ofther before reallaed said. Damn, didn'tt want
the room, and wake sister up. explain to (Buck nightmere to wake un. Sinjowat differently everytime, but bhe never...
just waken up from a fitful sleep. and the intense, angry-making sensations that filled her were like bridge connecting her to the dark, chaotic whidlpool that lay just beneath her daily struggles. Shee knew she had to wrestle the monster back down; she didn't want win, not again.
Wind. "Wiut in ihe world happarnng
over there? John asked he rubbed his eyes like he wokkn after fifull sleep.)
L-Mo-rB-Instruct
'Gei out of the John" Arty
with Baartlein shooting. Flack sprawling and the two finising stalemate
Emma turned fierce scowl inward. It wasn't 't her form of self-defence, but nothing. She had been fighting for her imental health for a long time now, She knew she could cefeat these strong./cestructive (destructive emotions. She
symptoomttic of National's failed approach health whole. The warning been sounding (mental health long time now:.... ...
than
position. 'I knowww Just wrsttal) smart, teer
When she
... way oring posttive change to
your family claseroom. You can learn
the many ways expressing anger rape, while exploring how these (eseructivee emorions could be).
learned gender
their expression,
do it. took while the starm, but she)..
how to find your calm the stomm, and transform m emotional breekdowns....
Figure 7: Example outputs from DJ SEARCH based on both verbatim and semantic matches, We prompt LLMs to generate a few paragraphs of novel, beginning with a first sentence taken from a Human-writter novel snippet.
31

---

Published as a conference paper at ICLR 2025
Reference Copus
Flease few peragraphs novel staring with the following prompt: "This was the of the first bird ever pushed from its nest, tentative wings stratch flight."
Verborin Match
Masa- Virtaation
This was cry first bird ever pushed from its nest, tentative wings stretching for flight The finst thunder of restless sky. The waters moving under the Awaleen as they rested on their thrones beneath Sirauk, kept alive by thair magic and trapped by too. was flustaring toward the surface of the sun, burning from the mere flight, "Baira's seal is for her Sultanas alonel not breach her
Sara Hashem
foo to hook her chair where it by on its
side. Like scccer player, she kicted it
feitblulees to the truth: the law God
ba former posttion, and with flik of foot, rocked
brokken, and all the blood shed them is
commandimen: again." Thejthing primly returndito its fomee position, and with à last smile at me, Its eyes rolled forward. Valda collapsed into same position from which had risen (There were footsteps
lees.
breach of this comennmmnnt Agein,
nolvio ation this
i. and the und metiooan-- "Saderl she sat this state of doubi and despalr noice foctstee was heard at the door, with whieppeiigg and brokenn ejaccl
door, and Marek's teasing voice trying to cajole the guard away. At loss, I <ikeed the ring under the dresser and prayed Vaida would think she knocked from nail during drunken stupor. I squeezed outside. Marek framed
After Miritter li Lily lay instaad sleap fult as if she hooteres on the edge of vasa) chasm. No teare left, she told herself
This was cry first bird ever pushed from its tentative wings stretching for flight.
The young chick trembled as it teetered on the
last looking in the di d'stance like a child's spinning pinwheel The wind murmured wokk my eyes was was tvallve...
ChatGPT
edge, the vasi expanse of the sky beckoning it forward (With deep breath, closed eyes and leaped, the rush of wind beneath [ts wings liftnng it higher and higher. whrl below blurred as the bird soared, the thrill of freedom coursing through veins.
closed
but a lanush of alamm fils you and you launch yourself back into yourfes purfaings lifing you grer cools around
You watch
you are obiterated. in the courtroams of heaven, the gavel now corre es down you favor: Do sense the freedom
abovn Joreed.
through your
France has beautiful land. ushh green forests, spardding rivers, and mountains, The highest point Mount
As circled above, the chick marveled at the beauty of the world around it. (hh ush green forests, sparkling rivers, mountains loom ing the horizon. [[t falt sense of wonder and awe at the vastness of all, rallzing that theme was so much more to explore beyond the confines of itsinest.
the purpose providing honest review and the image featured above own, e have al marveled at the beauty of the world around) uak
range
sensibility, the sence awe, s look ups un a cleat night think woeryore in every culture has felt sense of awe and wander) loking the Thank you,
Grand Canyon, type murder just unusuni, and fevt them was much mom to explons beyand the thejsimple
that's vielent
sun eagle windless west Atop the spint-clits builds him crimson hest. The sun does not round the earth, but the earth...
But the sun
Figure 8: Example outputs from DJ SEARCH based both verbatim and semantic matches. We prompt LLMs to generate a few paragraphs of a novel, beginning with a first sentence taken from a human-written novel snippet.
32

---

Published as a conference paper at ICLR 2025
Releranc Carpu
Pleese write few peragraphs for starting with the following prompt: "This was the cry of the first ever pushed from its nest, tentative wings stretching for flightt.
Verberm Match
Mese- Verb-teratttn March
The young bird's heart raced with fear and excitament testerad on edge its nost, the only home had known. Its) itllsioling ngniinn had already taken their first flights, but this was the first time it mustarad (tee courage to leave the safety of nost With a doep breath, [I spread its wings and took the leap of faith.
... stuuiis bufare they were handed back
to the developers and her work responds to the inherent tension of the site gs it
GPT-3
bewtaredd on the sdge of iiddemsse...
you are used seeing. Think of baby's first At only weeks old. Bedglings leave onlyhome they ve ever inown, jooaedd by the calls...
... for particular species, whare and when to site the m. and more. A fledg ing
chiccadeas musters the courage to leavn
For a aommmet, time seemed to stand still as the) bird tumbled through (the eir, unsure of what to do next, But then, with flutter its wings, cought the wind and soared. [The feeling of weightlessness and freedom) was exhilarating, and the bird let out a cry of joy and triumph.
the safety of nast
artempts...
tracks, unsure what to do. The creature nobiedd hm and stared at him with its glowing For mommentt time soamed to
world with new eyes stretching her wings and over guided by instinet
the locked.
she With wing spread wide
edge and (ook hoap
llight? While we may never know exacily
jostled about Like have turned powedrr, the mouth, now lineer on air unsure what to do next What baouut gnie ...
ih't like to fly like a bird, we can still enjcy the feeling of weightessness and freeddmm that jormss with d dowering
As flew further and higher the bird marveled
at the world below. The trees and Fields that had
tan down to railing birds never come back their old bedroom after venttrinng into the larger /world. It was over. In the tressjichiind the house...
seemed so
Guidan Bakoye village Nigar
seemedi rocical Thay wex longer di youung frins out of ther "elds bafora lanting seecs,
This was the cry of the first bird pushed from its nest, tentative wings stretching for flight. (The sound echood through the forest, hounding trees and filling the with sense anticipation and excitament It sound that
har feet, but didn't
seconc
..., ....pplying increase pressure, dencched until one fiber witer another syndhvonized into loud break that harfechoed through the forest Same sound as that...
was on knees, gasping sounc rang (hruuhh bouncing off the trees and makinng the
forest.
for Twilight Ziss after dark, zips
haraldad the start of new journyy a new
night the great things about
chapter in the life of this small, fragile creature.
Daiby is the barrew exci iemiittent and
anticipation w which joui ds as the course
Tulu2-70B-dpo
The birkllooked down the ground far below) its heart pounding in its chest [Tis was it the moment it had been weiting for, the moment
for hours, while the slowly dipped
west lengtheiing the tree. Every hoked down at the ground for below) he felt dazzy and
... Today as write the first blog post for Rustic Mottis, feel ittle emot ooaal Alter
had been preparing for Ity life. The bird knew
all, is the start a new journey a new phose lives, we hope
hed that had spreed its wings and
anviaioning what would bes all those visions that wc talked about, all of the worting the precaaing it is moment, that they bagn to las..
take leep of faith into the unknown.
the si ence that falls over then so total I can't hear anything not the nush-hiss eliny pounding my souncs around us; insects, wind,.. ...
Tentatively, the bird pushed off from the safety of the
being cha stalenged to ta leap of farth because we cannot roar we do not fly mottffy wings take lcap Babyy ...
Figure 9: Example outputs from DJ SEARCH based on both verbatim and semantic matches. We prompt LLMs to generate few paragraphs of novel, beginning with a first sentence taken from human- written novel snippet.
33

---

Published as a conference paper at ICLR 2025
Refaronce Carua
Please write few paragraphs for novel start with the following prompt: "This was the cry of the first bird pushed from nest, tentative wings stretching for fight."
Verbalim Match
Meus Veribation Meerch
This was cry of from its nost, tantative wings stretching for flight
ever pushec
... ...yarrs Thump, thamp. felt the blood
The bird, small, fluffy creature with spackled chest and bright, curious eyas, looked up at the sky and at out hesitant chirp. had never known (he thrill soaring through the of feeling the wind rushing beneath its wings, off seeing the world from a bird' s-eye view (But it was eager to learn.
moving quicky through my body, It's heat once again present. Thump, thump. albraat.....
Llama2-708- Chat
(loored the sky and
landl achvettum? This tour gives you all runs of the signatura plina course combined with ATV adventure After the scaring through the anjonnise....
LLOYD GIFT Surprise your ones and give away uuique experience of seeing the world from bied s-ey view jnit one our
we from, was geared-up.
with Lew Strasburg for five years and became member of Actors Studio.
Without Femitation, and jumsed off the mountain, Instantly the wind catch our vings. We wereffy scaring...
With alten hop, the bird pushed off from
the branch, Teeling the wind catch its wings ledd]
My stucents have lot tolleam from
lift up into the ait It was al alwobbly takeofff but
eager share I'm really..
on the breeze. You are stretching your nasigete the strong neck and [itt of Youjehive aga the brisk upturn downy Teachers.
the bird determined mester the of
immadiataly takeoff was 'wobibly'. How is this showing something wann 71 It was a litt windy rodayy, so (wobibly takeoff is to be expected.
llight. Riaoped its wings fur
with mothrer scane becomme mven
It was sunny day in the village when a small, fluffy bird emerged from nest for the very first) time wings
cays after hatching for birds to accomplis their firet actempt at fight Typically smaller binds(master the of fightbearler than larger
more compeling whan the flodging
suddenly spread its wings and flew leaving nest for the very fine time. Scheerer
still soft and untested, but it stretched them out confidently, peering over the edge of the branch to take in its new world. The village was bustling with activity, and the bird hopped from branch branch) exploring its new surroundings. But even was learning the basics of life, there wesls sense that sometthing wes about to change.
Bald
and yellod, "Hey Anku what's up!?!"
O
well, obviously Cerise though. following him to (wage the brench and peering over it. Instantly she
OLMo-7B-Instree
comes to you called 'maturity. And as you gain this maturity you realize you do belong this lower(branch, That bird hopped hightr branch
like London and wander ng arourd and Furopp. They hald off bookking based
As the days passed, the little bird grew stronger, its wi nin incressing in size and strenith. But something was still not quite right. Despite its excitement prospect of flying, seemed to be avoiding the ideaa altogether. Every time hopped closer to the edge of the branch, would ...)
the news andia about change
something
weeks
the parent's squaack put off, the chick's back paws slipped. Shara dashed forward as diuurteerers slid over jedge of the brarch chi inslwings.... ...
Figure 10: Example outputs from DJ SEARCH based on both verbatim and semantic matches. We prompt LLMs to generate few paragraphs of a novel, beginning with a first sentence taken from human-written novel snippet.
34

---

Published as a conference paper at ICLR 2025
Reforanco
Pleese write paragraphs a novel starting with the following prompt: "A sickening trunk fist crashed into his temple, he slumped to the ground."
Verburin Match
sickaning thunk crashad into his temple; he slumped to the ground. Someone grabbed Lome's braid, fallen from its jet pins, and jarked backward. She snarled, Mont' arms closed around her, kept har confined, took two to do the same to Bast an; the Sun Prince thrashed, shouting curses that echoed through the slowly darkening atrium. One of the monks struck with degger; the sharp edge sliced through Bastian's eyebrow, sheeting blood and shocking him into enough stillness be subdued, arms twisted behind back. The shadow the The Presque Mort who held Lore steered her toward August's throne. The Sainted Kiny stood molionless and alool, hands behind his back. Another Presque the one from the leak, walking almost normally on prosthetic appraacchc the dais and handed the King a dagger, cast in silver and scrolled
Presque
PONGLOWT
Hannah Whitten
moved closer
the low-hanging
When this hapoens. he seems litt discriented, (his give undemeeth him, and falls. How should he deal with this?
A sickening thunk echoed through the air
fist creshed into his temple, sending searing pain through his skull. felthhis legs give from underneeth (him, and he slumped to the ground his vision blurring aihis heed hit the hard earth.[He could feel his pulse racing in his temples, gasps.
... had lquor
cupboard when they were for too young be drnking it. Fie slumped to the ground.
Llama2 70B chat
his vision Buuning as hears formed at
Sandstream staggered few tail-lengths
tewaeds the
breath coming ragged
"Don't move," sald Jude. Sebartian haoze his gaze lock with Hero's, (Ha could teel his pulse racing mis) nock, foel wind buffeting his
... _Chapter 59,
(umm stumped ground, histtl dangl ig the water...
lesped to his feet along with the other clearing, eyes and breath coming ragged garpe "Pitrel" Fim whewred,
As ne lay there, could feel the weight his
sttacker's footsteps approaching him,
the door. I trieed ignare them, g glanced sc Lugh slumped
groundi trembling beneath hisffect. He tried lift his head to see who had attacked him but) the pain was too much, he fell back onto ground, hyseeyes slipping
opposite corner No hands, No feer. triee to hesc and choked back cry as saw the
stretching row from Alton East could feel back shir, damo aweat, even though the sun was pale and barely
(Ax lay
sippedd
to ground He tried stand, tried to look back tn sue who had attac him, but boot pressed against his temple..
tractments to avail, Songala to participate in the training camp practicn lest manth Richmondd but the pain was too much, and has not...
It was before. He folt the world starting darks and had no solha full back conscious siipee away
fight
Figure 11: Example outputs from DJ SEARCH based on verbatim matches. We prompt LLMs to generate a few paragraphs of novel, beginning with a first sentence taken from human-written novel snippet.
35

---

Published as a conference paper at ICLR 2025
Please write few paragraphs for a novel starting with the 'ollowing prompt: "A sickening thunk fist crashed into his temple; he slumped to the ground."
Reference Capus
Varberin Match
Mass Vertiation March
envisiioned is simply not in the cards. At the beginning of this campaign, Ifad always known that this was possblity) And many of you know. passed
James had a always known that this was possibility Growingfup in rough streets of New York City, he had learned ways ba his guard. But (oo matter how prepared he thou ght he was nothing could have prepared him the sickening thunk that echoed through his head fist crashed into his temple.
Gundam Muster and representative Neo-America, His lighting style boxing After the loss of his mother. he grew in the rough streets of New York City....
GPT-3
and the baby formuls that she coultitt
manths bottles. But matter how prepared he thought he was for times like this, she...
"Onnee Floard ordered. It not Unsurg g practioe hall. But surprsing that fwas, nothing prepared him the) who
He sumbled backwards, his vision blurring and ears ringing (He could Teel warm blood
Shock gripped Caldinn stared fel. surpr sod look his face as sumped to the ground, blood seoping from the...
done.
trickling down side of facel he slumped
goddamned bottle while I wasn't
to the ground, his body uneble support itself. He knew he haad to get up, had to fight beck, but his limbs felt heavy and week,as aas if they were
looking? ? His heart thundered! (He could blood ddingg facel. had been replaced by ...
... she would be. She moved from between Adam and Darek and stood un. Her logs
were weak, if they were made ruborr
off ead.....
McLandon know hefladi to get up, had do something before Brautigan sorembbled
She stumbled over where her clothes
him again. staggerec hiss feet looked
A sickening thunk echoed through the dark alley as a crashed into his temple, gending reeling the ground The rough concrete scraped against his cheek he struggled to stay conscious, his vision blurring and (twisting into kale keleidoscope of colors. Hejoould hear the sound of foolsteps his allackers circled around him, their laughter and jeers ringing his ears.
thief came running out of the bank's front Cobalt
with from both hands, sanding him reeling to the ground. The man was
Tulu2-70B-dpo
instant, was suckedd
way insida, my body (wisting into kalaidocopp of colors. Hejat eejatrppp ripeed through me from my arms to my chest
and then the blade came forword and he could leel the warmih of his own bloud tricking down hig nack in the cool air of the
throat: Duh crew a sharp
Every (bresth he took like fire his lunsel and he could feel the warmth of his own blood trickling down hisface (He tried to move, to push himsel) up, but his limbs felt heavy and uncoordinated, as was trapped nightmare. He cauldn't even...
The back of his head and every breath he took felt lungs) He was weery, es something had used up all the el eiemental energy he
throbbing
cold moonlight, his open eyes weemee unfooused, and his pale lios trebiedd jas
nighemare. "Die you He walked...
the It ley si paces aony Survival invind to him ha must get to it. (Ha tried to move, to push himself The hool of his sandbl slided on blood.
Figure 12: Example outputs from DJ SEARCH based on verbatim matches, We prompt LLMs to generate a few paragraphs of a novel, beginning with a first sentence taken from a human-written novel snippet.
36

---

Published as a conference paper at ICLR 2025
Please write a few paragraphs for nove starting with following prompt: "A sickening thunk fist crashed into his temple; he slumped to the ground."
Pallernce Carpu
Mess Verbbtiinn Merrc
The tonsion was starting to get to him, his was makking hard focus, could what was being down below, ...
A sickening thunk as fist crashed into his temple; he slumped to the ground. The world around him spun as tried to shake off the blow. The pain saared through his skull, mmaking it hard to focus. (He could hear the murfled sounds of a scuffle around /him, but his vision was blurred and his thoughts scattered.
... He looked up and down the hall. No one was there (He could hear the murteed sounds of affow TV sets coming through the doors, well what reemed
ChatGPT
garbled sounce sagged his knees (Aa str struggled to
last thing saw was someone
regain his bearings, she sidestepped and rammed a kree into his side, toopling....
leaning overthim. but his vision was blumed and he couldi't tell who it was. "Rave.." he slumed, say ng the only name that
As he struggled to regain his bearings. he felt
was that no omagas had ever been hore, not oven by accident. But as ha stra ned hislgyes, he saw the sihouutte of patite body, diresee in pasteels and...
hane grab his collar and pull him upright.
Through bleary eyes, he saw the silhouette of a
The icy cold of
with
figure looming over
menacing presence
weaksned stato, worked agains: him, and he sank furthe: Grstefully he felt hard grabs his collar and pull him) up
in the dimily lit a leywey, The smell of sweat and alcohol filled his nostrils as he braced himself for another blow.
Blair blurted
mouth fell open
in cisbeliel and his eyes glittered bwhind his glaises. Helspoke in law, monasing tone] Youu we nt through my garbage?
But instead of another punch, the figure in a low, menacing Loree) "You shouldn't have crossed us, Johnny. Now...
Through this money, he became The Kingpin. But, this feteful day, with the kingsin in pool of bloed. lookod up What was...
threw drive punched the
gas, Rasputin flipee over hood
followed byfthe sound of his heed hiting
Jacoo woke to the sound of his head hitting the pavement He groggily sat up and found himselff in pool his own blood, He looked down
the paveenee)) t-m ach like wetermelon
0 o
collspped one day...
day, after an
18-hour shift, he saddinll passed out and fel. Wher he woke up, he found himseff in pool of ris own blood sporting
fist-siced lump where his right temple used to be, and his head was throbbing (ike he hed been hit with e hemmer.)
OLMo-7B-Instruct
look on retrospect might have been harsher than needied to be. Begus winced like his had boon hit with hammer Gorso
"What!" shot
Jacoo fried stand, but his legs like jelly. stumbled to his feet and stumbled down the sidewalk. People were starting to notice him how, and he could see their fsces contort in concern as they noticed blood.
doonway before Jon's strongth gave out.
As Jenny pushed open the door, Joe's knees bidkled again. Heltried to sta rd, but legs felt like olly. le saw the cabin...
Willie blinked;; that second leap would be difficut for any athlete. The honded figures were close (now. and he could see their faces) crossed sits replacing.....
... Datsuun pulled up along road stopped It was packed with teenagers.
("ey man, are you okay?" passerby yelled.
One rolled down the window and said. okay?") ln ow the...
[Hoy man,
Jacoo friied nod, his wasjspinning and was hard folform words.
this information, S Surviver, then will. to hac or will die trying Spookk fried to nod, but his head was held tight by bandages wrappigs He
... South Lucroy harbor. Had they gotten out of there and sack their Intll farm?
("What happaned? the man asked)...
She wasn't esactly sure. Herlhed was sprning and was hard tolget her...
I feel gulty for money, so would like call it day." "What aappored the man asked) Pointig to the street right in front of the
Figure 13: Example outputs from DJ SEARCH based on verbatim matches. We prompt LLMs to generate a few paragraphs of a novel, beginning with a first sentence taken from a human-wrinte novel snippet.
37

---

shadows and Published as a conference paper at ICLR 2025
Please write few paragrapht for novel starting the following prompt: "The sufficiently warren-like that she could sneak upstairs thout being obsarved."
Reforence
werv
Varberen Match
The lower offices sufficiently warren-like that she could sneak upstairs without being observed. She entered the front hall first. felt satisfying to start somewhere forbidden. There cathedrai-like nush, light coming down through glass dome above. Palms and ferns great veses. floor mace of white marble. Gold on the door panels and crystal in the doorknobs. lot of very disgusting that Mrs. Bone rather liked: paintings nude ladies, foxes stuffed till their eyes popped, stags screaming ming silently from their plinths. It wasn't exactly the breath. was the upward, all glass and iron and light. It seemed frasted, iced, lickable, kizsable house. Her envy made
HOUSSE On HEEPFRS
expensive things
Alex Hay
place caught the way flowed
grow hot. connacted the gardens lang,
was
colonnaded passage and several glass-franted doors. Sho remambered from the
The lower offices were sufficiently warren-like she could sneak upstairs without being observed. The nerrow curridors and cramped workstations created
o
Llama2 70B chat
hiding spots, ellowing her to move stealthily through the space. She knew that upper floors reserved the higher-ups, ones who made the important decisions and commanded the respect of their colleagues. She had always been curious about what went on) behind those
being paid a bribe a finnccal gain being made, but then vary and nothing about
behiidd
closed
doors,
curcuuuuu what was in that briefcese the Yauze had paid her to deliver the Triack, and Her
chance find
find out) Butterfly,
Seturdays pmn Since Spring quarter my 2nd year. inttialyy got involved because Ihad always beer curicus about what went on the
(As she made her way up the stains, Shhe couldn't t healthy dose /Se had dea what
sense excitement
to claim his price, Elizabeth could see the look of power and control in his eyes. She couldn't help but feel wrep
walk toward the h house, the aroma People lined
blunts
fille
the
drivewsy (Au made her way up the stairs she
unwelzome but the mage did not react simply every muscle her happaned, heart shumping
magnified thousand times, seamedi to stare directly nto hers. Exctemmntt shivered down her spine, mixed with dase fear full-grown
had idea
Figure 14: Example outputs from DJ SEARCH based on verbatim matches. We prompt LLMs to generate a few paragraphs of novel, beginning with a first sentence taken from human-written novel snippet.
38

---

Published as a conference paper at ICLR 2025
Please write few paragraphs for nove starting with the following prompt: "The lower offices were sufficiendl warren like that she could sneak upstairs without being obeerved."
Reforence Capus
Verberi Match
... packed floor of room
red ated cold even through her heavy
woolen wore soces As she made her way
through the dimmly L0 room to the door
As she made her way through the dimiy lowar offices, Elizabeth couldn't it help but feel a thrill of excitement t, (Sha had been planning this for weaks, meticulllyy studying the office layout and schedules find the perfect opportunity sneak upstairs is unnoticad (Her hear pounderd with each step she took hands clammy with nervousness.
whole couldn't board-
not population, actual town
Srill, the ioelc'un''' help out feel a
GPT-3
excittrr ent, She 'd saloc befora, but
town was Inowing where wouldint be
botharad: Once they movedi to the back,
the front section of the barn where several grooms were busy brushing horses and checking hooves. Heer heart pour ded with each stap she took, lard stran go
finally Wondy know this was She had been planning this weeks
He passeci the book Masia "Thank you much." She took the open book and waked to desk, (canefll make little poss ble. Sho look
Finally, she reached the sta rivell and quickly ascended the stairs,carell careful to make es little noise as possible, She had always been curious about the upper floors of the office lkiing the reserved for thereigeerupp and executives The thought of being able to explore those forbidden territories filled her with a sense of rebellion.
the reopening of the concourse the public with retail restaurants,
residantial convarsion oftthe upper of the office building, ot last bringing....
She ran off of the porch and stood under
the tree, wetching the branches for sign
When that boy wasn't around, she walked
satisfied
stainwell that led down the parking garage
lights along
steps had been turmed she reaches the top of the stains Johe heard running...
As she reached the top of the stairs, Ellizabeth paused, her ears straining for any signs (of movement. [When she was satishid that the clear)..
light to them streett before then looking right When shn was satisfiad that the coast was clear She
through
looked first,
o bscrvar might think that we d spent the whole period wandering through the mane of Eorridons thai marde up the ooer levels of the High Fano. have
The owwr offices were sufficiently warren like that she could sneak upstairs without being observed. was a blessing and a curse, the) labyrinthine corridor that made up the lower levels of the company's 's headquarters. On one hand, made her job junior amployee that
rivelty had, its own way. know helpwd to make us g grwet, (t
was a blessing g and a curse, the price we
Tulu2-70B-dpo
hac pay for our genius For now
much (nore difficult; on the other, it itallowed her the freedom to move about unnoticed when the situation called it.
affected word learning. On
hand, might be distracting for
and make learning imor difficut; on the other rrgght help word herring....
Dytha could until she was blue
the but days, she rally fooling the mood (loca was one those cays: She waved hand....
Today was one of those days. With folder full of sensitive documents clutched tightly to her chest, the made her way through the Bimily hellways, her soft-soled shoes silent on the worn carpet. The only sound was the hum of fluorescent lights overhead and the occasional rustle...
brack bell-ash hoped to heaven would't have use. With hor heart pouncing in (ber chest, she made her way through the ace via
The hard-gueceed ced dirt floor of her room radiated cold even through her hakvy woolen work socks. As the made her way thruuhh the dilly lifroom
Figure 15: Example outputs from DJ SEARCH based on verbatim matches. We prompt LLMs to generate a few paragraphs of a novel, beginning with a first sentence taken from a human-writter novel snippet.
39

---

Published as a conference paper at ICLR 2025
Please write few poragraphs for nove starting with following prompt: "The lower affices were sufficiendly warren-like like that she could sneak upetairs without baing obsarvad."
Reference Carus
Verbernn Match
"excited to start something new. and
Masa Virtzation
(el: more comiortablee making the
transition since She has been working building for yearsjand she was
The lower offices sufficiently tly warren-like that she could sneak upstairs without being observed. lvy knew this well, as ahe had been working the building for years The dimily lit corridors and maze-like layout/made easy for her slip away away)unnotice unnoticed (As she made her way up the sta her heart reced with anticipution. She was about to do something risky, something that could potentially change life forever.
oblivious to her and hor bookish seft, so
much so that they let her wonder away ball, imade easy away with Nathanicl, Anyway. the...
ChatGPT
the reality show Uirdscovered Down was nervous for her daughter. This e-raa contained neve (tht could potentiely change her li forewer) Dawn opened
night, Cord." 1 "Abigail," was he said, could feel his syes
kochen. (Ai she maae her way up the stains, the men's vo ces corree to
Hyacinth wh redippeed as they approached
the corner The Sun is Also Star I'm vory? geared around camer, making
Reaching the top floor, lvy paused tio catch her
body tight against the trunk an altempt to make herself ilvis es possible as she took the time catch her breath. She peered around to check if...
How you know...
breath. Shel peered around the corner, making sure coast was clear before continuing on.
right path egain but with the impanding trap, that very licely Details
(hhe had mission accomplish, and she determined to see it through. As she crept closor to her dastination, the sound muffled voices grow louder, causing freeze in her tracks. [She pressed harself against the wall. listening.... )..
(She and she was already going do wall,..
accomplissh
... Whatever the reason, an add pricding sensarion can the lengtn of Black's spine, (causing her freese her tracks. the roof of
Clucking her tongue
contrast, made temible racket, through his for Narmora the first reach the gates, (She pressed heme! against the wall, fistening) interily to
... Henry crossed the bridge into Oakand.
No lights the windows of the house.
0 olmo
She snuck through the hallway, careful not make a sound))he had be quiet, she had to be quick, or she would be cought (Her heart pounded her chest as maee her way up to higher offices of the building. her destination hhoden from view the dark corners of the lower floors.
Insida, down the hallway, careful to scunc) Thomas as was ssleep on...
OLMo-78-Inttruct
at all No matter the outcome, didit work way
If Janx won, ithad duick, or the would bad for taking so long If Blsse won,
her just poweriese. A whispered requiett his lip shald
melt into and bag more heart pounded in her chest she) sttred into...
It have been to puncture boy't 's illusion to remind him those who lived in obscarity/ dark corners of the metropolii, the
Afice said, then fall bahind the other
She had plan, a risky plan, could change her life forever. wes plan that cou'd d make her rich and give her power to do good in the world Butt knew that came with price, (if caught, would be) arrested, her dreams of wealth and power shattered like glass prism (under the light of the sun.
from view in the
woman she led her to Mr Davenport's office, and the intersiew that could change
her lila forever. was andly purprisin....
techtical/eectorl br dge," Zeichner says. Aside from being good for business, Zeichner believes loT has power de
learning in the Despbice consumed huge volumes of resources, Even though he had the subsidy or 30,000 coins, he knnw that it came with a price Jt looked....
good the
emphasises....
She reachecc the top of stairs pushed itheideorrrst the highest office in the
French scil, she thought with shiver fear, enemy torrttry, Now wes crimiral terrorist, a sny (I the was Gaught she would be jovouttd She put
Fack time travel, fuck Setsuna, fack
Island Loser: UzaMaidi
underdland how Camoi welke freely finderr the light of the sun. She jhould be in
... words. "We are said, room at the top of the stairs Or so she hoped (She reached the top of the stairs and pushed open the dlllle doors. To
provincial premier. With his eveering-in, Mabuza finger jamb ohe to Nighest off co in the ladd Ms... ....
har
Figure 16: Example outputs from DJ SEARCH based on verbatim matches. We prompt LLMs to generate a few paragraphs of a novel, beginning with a first sentence taken from a human-written novel snippet.
40

---

Published as a conference paper at ICLR 2025
Please draft speach with the opaning sentence: and much in mind that before you today."
Parlerncc Carpus
Meae Verbatin Metch
editorialist trade press wrate that "the FCC of the New Frontier is gaing to be of the toughest FCC's in the history of broadcast regulation. If he meant that we intond to the
carton- motoincutting.o goa's. "Our building soctor is gainng to be one of the toughent to fully decarbenze, said Deborah Denovan, Massachusefts diecttrr...
Newton Minow
public interest, let make perfectly clear that he right: We do. he meant that wa intend to muzzle or cansor broadcasting, he dead wrong, It wouldn't surprise me some of you had expected me to here today and say you effect, "Clean your own house or the government
2030> polica resporsibiityy
enforce the law in public interest
Achually and interestinlly you're te mataken.
And, actually, but for less...
will do for you. Well, In limited sense, you
.and the Liberals didn't anything
would right because I've just said (t.
E. But want say government, and this very serius, that have been lving through some very...
say to you eamestly can that not that spirit that come before
RRgulltee industry, governeent will do for you With that ultimauum on the table, several major players the media industry...
As stand before you, am filled with deep sense of responsibility and purpose For understand that every word speak today holds the power shope future, to inspire change and to bring acout progress.
Jm delighted have this opportundyy distinusshed aucianzal, such stallar event, such lovely evening...
GPT3
our team," McNeil said. "It is litth a deep sense respons bility and purpose that wwil make certain that plan delivare....
this opportunity to adoress juch distinguished audience want take this moment exoness my gratitude for your presence and attention.
hanored
The holldbys are upor us, and wont to take this moment far/s many mearing
gratitudl
goal. Once again have this apportunity addrees you and look fonward to the rest of the seesions today...
We living challenges Our world constantly evolving, and with it, so are the issues we face. From climate change to social inequality, from political divisions economic disperities, society grappling with complex problems thet require immediate ate attention.
of great uncertainty
the appointment in lutter dated 29th 2019. would like gratitude your presence valuable
riaul and headendii upp role el first
pendemic has put everyone on edie We are living a time of great undertainty and almost year iater, there's stil no pandemic end.
The crypto is constantly avaving.
and are the trading alarforms. Laver? la such platform that is making waves the incustry I...) Share your videos
guarantee
.Murshy to tie it all together in the end. Even with the chaos and confusion, there sure thing about the returning glad
Howeves amidst (ill the chaos confusion, there is...
Howevar the dam given "poor" rating by See below for the original documents.
they found nelproblems that inmedd
Figure 17:
Example outputs from DJ SEARCH based on verbatim matches. We prompt LLMs to generate a speech starting with the opening sentence of a human speech transcript.
41

---

Published as a conference paper at ICLR 2025
Flease draft speech with the opening sentence: and much more mind that come before you toclyy"
Reference Copus
learned that GIS has stll chellenges conscantly charging. these changes cover cimate, public safety, transportation, pollution...
Verbernn Match
Masa- Virbatiio Match
... As we gather here, is crucial reflect on challanges and opportunities that shead Our world constantly changing, presanting us with newjobstadess to overcome and new paths to explore. is up to us tojrise to the occasion and meat these challanges headlon.
... observes and asis quest one about
world. Our undenttadinng conatantyy
Changing, presenting us with new
ChatGPT
questions and answers Science gives
for working familiess" he said "However, we can fise the occasion and meet those challenges need on) we've done generation gemeenttin
for C.P's Chair of the
Neil Wallack
challangas and appartuntiess Shoad in his two year torm...
One of the key issues facing us today (s climate change The planet (s warming at an jelarning rete, with devastating consequences for our environment and communillees, is imperative that take action now to reduce our carbon foctprint and protect our planet for future genterations
Pentagon has recog for years
we have yetho exclons. la up to change what do,
now that one the most prwsing fseuus lacing us today is climato change. Fortunately. we have a wealth
make identiyy how take Therefore, OGI, and...
the world is warming at an si slarning rall, with devaslaiing ng corseqqusntss)
... evoolves, laws charges, and audiences movelon. One of the bey haues facing) a adu't performers is that global
But it's notjus just environmental issues that must address. Incquality and injustice are pervasive in our society. and we must/work together create more equitable world This moons fighting for socil justice...
While climate change,
helping take
governments shunting
acrrsss irriicttonn, governmente organ zations, and communitios is imperot ve that we wolwork together to improse diete ther ri and compenibiilyy.
... the greatest of we climate change.
The planet is wa ming at an jalammingy fast pace. putting pressure on natural resources
and chinging the that land used In
Air Arrericans should lead the way in the mission to reduce our carbon foctprint and protect planet for future generations.) Abandoning agreement ignores the
and poverty "(t il action now protect and rerwo soile ensure e more lood secuure fulure. lucklly thene are
of people
imperative
... action to achieve ger der parity Let's all take so time todlyy to appnreciate the wamen in our lives and jeark tngether to create a more equiable world for al)...
The effects of this insquaiity are pervesive our seciety, and we must) arknowledie with peritence that they parvaiive organisaiion... "We
Figure 18: Example outputs from DJ SEARCH based on verbatim matches. We prompt LLMs to generate a speech starting with the opening sentence of a human speech transcript.
Please draft speech with the opening sentence: "It is with this and much more mind that come befora you today."
Reeeeenee Carput
the funding and materialse the projects?
Verberim Match
The recovery plan is great example of how can workk together to achieve grest Bhings individual and countries.
Moar Vertatto
... to speak aboout the incredible potemtal our commun ty and the ways which can) cankwork together to achieve great things.
0O
Hamaa-70B-Char
and Popular Prosest conference and the open joumnl. Interface. (Lavv privlage wo king with some the most intoresting activis....
As a helpful assistan: have had the privilege of working with many of you, and I hevelseen firsthend the pession, dedication, and expertise that you bring to your work. Whether it's in the
take consiceration the health
our community and the ways which we canjdo our best to protect each other,
... compensation, med cal and dental and
which means we not be able to...
field of healthcare, education, technologyl or any
4018) benefits, If you you have the passion, desication, and rspert tive that) match our unique company. apply...
other area, have been constantly impressed by
privlege meeting and
the innovative solutions and creative problem- solving that I have witnessed.
warking with many you, and havee) witnessed firat-hand your immarse contribution. Whether you are part of....
for the surpose for which it created, belleve we can even more My noble fr fned Lord Hunt of Kings Heath wil hear for the fini,, and probably.... ...
believe that we can even more, By another,
collaborsting and supporting can achieve far more than we could ever accomplish alone) Phat'swyy/ I...
for numerous Danish and international field of
organisations, among haalthcaro, educat on, tachnology employ employabilty, energy, and food
... believe that what we can accompiish as far more than we could over
We belimve that by working together andls-pportng one anpthee, wn can sch ova anything) hhank you far considering us
accomplich alons) Are You loring Full tage Your Buying Group
Figure 19: Example outputs from DJ SEARCH based on verbatim matches. We prompt LLMs to generate a speech starting with the opening sentence of a human speech transcript.
42

---

Published as a conference paper at ICLR 2025
Reference Carus
Flease draft speech with the opening sentence: "It with this and much more in mind that I come before you todayy"
you Irinnds stand bafons you with heavy heart, a heart weighed down by news that carn to my attention, news of such...
Friends, and don't
Verbernn Match
Mase Virtatioo
Honorable guests, di stiinuissedd colleegues, and dear friends, I stand before you with al heart hope land burning passion make cifference in world.
Presentation at City Varsity She grackuated 1995 heart of and miidd dreams, Abera Ayhor career After graduating, she auditiored
Tulu2-70B-dpo
used to explain explain away much world mulsfactoria strees: the economy:
the opponent himseff. So, with a vision make
We live in a time of glgreet challenge end uncertainty, but also
in time
mind and a burning dillerence Highlight injuatice dons and countless other people... ...
boundless potential and opportunity. The decisions make toclyy will shapel not only our own futures, but futurel of generations to come.
keeping, losing or gerting jobs
... the professiona world ato time of great challenge and uncertainty but will be up them on and ensure that everyona gets fair chance...
the way communicate the way oppcrunityy decisions own malks toriyl is country- - and as individuals might not maxe
We are here today because believe brighter tomorrow, one where every person has the opportunity to thriva and roach their full potential. A world where/we we prisritize the wall. being of people and the planst we Ical (home.
respionsible, corporate chtizens
convergence of the Manhattan and
ensure future deggeerattn ons to come.
Apolio projects to protect (not only our own stturiyttolllty glozal trading partners. We to
We ane committed to strengthening our
local commun bis and anriching the
this is bari or to visio of the future
is explict to tay, (We are today) because owe belleve inja visoo of new Nigeria. We are here today because we strongly belleve that fixing Nigeria...
We know that this not a alvison that can be...
lehore every
has the apportunity
thrive healiny Davis
authors hope the report inspire
Our good to live and work wholsheartedly way that honors the well-beigg of people shure W we Use virus "il, third
a world where each individusl can reach their Tall potemtal A world where) each pe empowered cultivate gifts
planetiwe
Leaders...
to advantage of the nat and all the people. Burithis not a vision can balsold straight - has be accessorized undil It looks like the opposite of...
wide range belongings and items in your home. (We know that this not a) good sitution are here provide extensive and...
but technicions
Figure 20: Example outputs from DJ SEARCH based on verbatim matches. We prompt LLMs to generate a speech starting with the opening sentence of a human speech transcript.
Fleese draft speech with the opening sentence: this and much more mind that come before you today."
Reference Corpus
Verberim Match
Moss Verbatiirtttnnn Match
would Hew to bring You into the New
year with a firm belief in the power of unity thatknows no bouncories obstacles. For You going through the whole country.
Ladins and gontlemen, respacted guests, and fellow citizens:
O 0
OLMo-7B-Instruct
It with profound sense of gratitude and aa firm belief in the power of unity that) stand before you today. Asil look out at this audience, filled with diverse individuals united by a shared aspiration for progress, am deeply grateful for the bond that binds us.
is here with me is largely because of horihatt stancl before you today, As a young introduced
senior high school,
... of customers, stafi, vendors, and schisors, is with profound sense of gratituds and cincore heart that say thank you! The Accullemp Community..
tuning this meeting tonight hold the Aaranic Pristtoodd As look at audiones) sec many of you stttnn next to mature men, perhapss your
... momentt in history and offer music and roflections that aow us laugh, cry, and imagine à brighter f ture for ourselies, our cormmuniiees and world).... ...
For this shared spirit brotherhood and sisterhoud that we find the founcitiion upon which we can buid a brighter future for curselves, our commun ittes and our worls, This is why here, and why am so passionate about this cause. Iamm here to champion your creams, to amplify your voices, and to act as bridge that connects uus helping us to realize the incredible potential that lies within each and every of
allon that we beie They amithe foundation upon
build strong, correct undorstanding Cod's Word. Otherwise we be building on
the cha riubll car maiigiing that the Coffee Music Project does why hero, and why to iied the final: think oee can all recognise...
resilency of the Afghan Refugen Lartly. to exclain (ely I am so pasa onate about this cause. (have not only fought brave my mother
... keen to gain knowledge, hence this platform will play an mporrant role help them and act bridge that connects)
bled with
very easily requiee information.
... serves as a reminder of the incrddble potential that las within wach and every one ugl and the impact that person have the world. 18th
Figure 21: Example outputs from DJ SEARCH based on verbatim matches. We prompt LLMs to
generate a speech starting with the opening sentence of a human speech transcript.
43

---

Published as a conference paper at ICLR 2025
Refaronce Capua
Fleese draft speech with opening sentence:
"Green fields and dense forests are dissopearing."
Match
Meus Veribatin Mibrrsh
few years ego we were greatly concernd about the "Ugly American." Today must act to prevent an ugly America. For once the battle lost, once our natural splendor destroyed, never recaptured. And man can longer walk with besuty wender rature his spirit will wither and his sustenance be wasted. A third place to build the Great Society the classroms America. There your children's lives will be shaped. Our society will not great until every young mind is set free to scan the farthest reaches of thought and imaginarion. are
Lyndon Johnson
the neaches thought and imagination. We are ztil far from that geal Today, millon adult Americans, more than the entire of...
that goal. Today, 8 million adult Amaricans, population of M Michiaan, have not finished 5 years of school. Nearly 20 million have not finished years school. Nearly million- than quartar of all
than the entire
THE MODERATOR: Ladies
gentlereen distingussee guests, frands, media hianns, very warm welcome to everybodly and welcome to the
Ladies and gentlemen, distinguished guests, and fellow neture entthssasst,, it with heavy that Ilstand before you today address pressing issue that threatens the very fabric of our planet. The green fields and danse forests that once adomned our landscapes are rapidly disappearing,
CHURCHILL dear friends] stand you today address aanttor grave importarce The 1929
Reapportionment Act, also known....
fenthuseasts, is with haavy that must remind you that Wue headed to that great console in the sky at the end Jenuary Herw's the new round
govermmntt alfairs Robert McCaw. supromacis: violence Sace that threstens the very fabric of our nation and must be adidressd in
our duty before it's too late.
action
control, teering the K-Man family spart and threatening to destroy the very fabric of planet The film the most intense senetionnal X-Man
The destrruction of our natural habitats taas been an ongeing battle for decades, but recont years, the rate which our forasts and fields vanishing has reached alarming levals. It estimated every minute, an area of forest equival ent to football fields deforestation. And this just limited forests, as aagriccuuurall expansion, urbanization, and other human activitias to encroach upon the remaining green spaces on our planet.
including school gystem. have just take action before it's too late. Tholoils ars:" those who live there, many....
public
Health Care Financing. "The safety children is paramount, and duly asiizr protect them tooic substances found in flavored...
The fight be recognized and acceptedl been ongoing bartle for decadea, pperhaps longer, and while have made progrest today there are
due
ready 85% of thei in faci, ievery minute, an area forest Bauhvllnnt 27 soccer fidlss lost globally.1 l'atts like these are hard...
to berter technology, your after year.
Electronics have always produced
years, rate which
But why should
use discard deviccss has repicly...
can vary graatly Land-use and land- cover changes such as deforestation, legicultural expantion, urbanization, and other human activities wow a direct....
Figure 22: Example outputs from DJ SEARCH based on verbatim matches. We prompt LLMs to generate a speech starting with the opening sentence of a human speech transcript.
44

---

Published as a conference paper at ICLR 2025
Reforence Copus
Please draft speech with the opening sentence:
..aa it up to us to take action these words, Worthiping tegether in context of larger body provides benafit and protections to at
"Green fields and dense forests are disappeering."
Verberin Match
Mess Wertbtiin Mercch
at an alamnnng rate, is up to to action before is too late. The loss these vital natural habitats (oot only impacts the envirenment but also has far-reaching consequences for our health, (well-being, and the future planet
are taken to protect and preserve the rainfores: before It too late.
biggest calast ophe for mankind la
tropical
ChatGPT
did yestenday and expect different Delayed action only impacts the enviroment but also Jinacial implications The longer occupants
large and capable wild native to destruction wher frees are out dowr and land is cleared) the jagaar popu ation...
hunting and habitat
As frees are cut down and land is cleared]
egriculture and development, we are losing valusble carbon sinks that help regulate our climate. Deforestation contributes to greenhouse gas emissions andd exacerbates the effects of climete change) leading to more frequent and severe netural disesteri such wildfires, floods, and droughts.
and duration heatwaves, Heat fawreeching consequences health, ceraing or contribute buuting to dozera of debilttting or deady llnesses....
continuud Burning of fosell fuels wtible
heal-trapping gases to the abmoghhere and exacercates the effects of climate
... top of the piee cal the shots. We are
pay ng for these profits with our health, our woall being, and the future our planet) The tricle down environmental
chergee he going
dlimate change, you
may ask? Well, cliaae change contributed to more frequent and severe natural disesters such as tropical
Furthemoree, the destruction forests and fields threstens biodiversity and disrupts ecosystems, /putting.//gggggi.... countiess species risk extinction. These natural habitats provide assantial services such as clean air and water, pollination, soil fertility, which
The en incresse nen disweters luch as wiidires foods, and droughts) Chico, whole, the affected by
Northen
Emiieddied in the worldh largest and ninforests hnbutthsaittts and water, soil stabiltatioo, flood... ...
... humans, our welfae and cconony, but ittlputting count less species at rek exti rntiiiiion, My artwork partras the dilemma created climato change
years. Lengue of Lagends is spreacing across the globa jtt an alarming rato, and it's prime) mover in the incressinlyy luzrotiv...
... smartest
jtt alarming rate, and it's trend that must reverse. The impact of deforestation and land degradation is not only fel: by the env rovmrnt bui by communities and individuuls (ho rely on these ecosystems for their livelihoods]
Into sea. Effect On Human The effects of global warming arefnct only. felt by the environment buglty humans as wall, Higher temperatures can load to
o
Hamma--7B-Chatt
and lke Turkans as well as the people for their Hachoway,
lhh raly these
Ineelooods.Acoo According director Intemationsl Rivers
Forests provide us withlclean) air and water, fertile soil] and a wealth of biodiversity. They are alsols le critical source of income for many communities) providing wood, fruit, and other forest products. But sadly. these valuable resources are being lost a rate thousands of hectares per day.
Jihn 1.6 billion people the world ody
... also work to protect and onhance farmland so that future generations will have glean air and water, fortile soil, and for healthy delicicus food...
heautly on forest resources for their livel hoods, Forests provide with dewn)
waller, regelatts diinele and protects....
of
detrimental impact Palestinian
personsily? The forests represent national heniaaa wealth of bddivesiity Toy are als0 spacos relesse of oxygen and they provide
livelooods, agriculture wasla
source income for many communiiossnn in the West Dank. It was Israells legal..... ...
The consequences of deforestation are far reaching devastating. Soil crosion ...
who thousends of dollars digitie bookslat rate volumes year? soams those ask such quessoos have litlle..
Figure 23: Example outputs from DJ SEARCH based on verbatim matches. We prompt LLMs to generate a speech starting with the opening sentence of a human speech transcript.
45

---

Published as a conference paper at ICLR 2025
Caruu
Flease draft speech with the opening sentence: een dansa are disappearing."
Vercerim Match
Moar Verpation
for chibushua stolen in SF smash-
at an alarming rate. Our once lush and thriving andscapes are transforming into barren and desolate wastelands, devoid of life and color. The unchedked expansion of industries, urbanization and deforastation have resuhed in devastating
grab The subsidence (ss not just
ansthatic issue. Thejanking sidewaks have
danger pedestrians
Tulu2-70B-dpo
for betwwen 70-80%
rainforest Forests are the lungs our planet, providing us with oxygen and also sedt motter planei-warring
our natural haritage.
But the loss of greanery (s not just an aesthotic issue. The consequences of this deplation are cire and far-reaching. Forests lungs planet. providing oxygen we need to breathe and absorbingg the carbon cloxide that our existence, Without them, we face clanger climate
"Wherever we are we are connected the sea it prowided in with the asygan we need to breathe and pupprts the dimata that live off We need healthy
fields are the
may remember from school, plants photosynt producing ooygen breathe and abso bing carbon dodde that dangerous for
have the opportunity transform the
nature public particpption government. WNithout tham, we danger butting the people out
THE MODERATOR Cadios
[Ladies and gentieman, distinguishad guests. and Jeveryone watching or listening us today: As we stand here, on this grand and important day, (wo faced our atrentttn and calls upon our shared responsibility for the future.
gentlerenn distingisshed guests, and
frendd media friends, very warm welcome to everybodly and welcome to the press
0 M
challenge that demands
OLMo- 7.Mn-rB-Inttruct
... goal of curlng an five pounds three add addtional repowwe are facod with challenge that demands our attension. Losing lecu under such conditions.
Gverburrdeneee Grean
danse
forests are diaappcaiing few years ago we were greatly concemned about the Uglyy Amerr can'. Today must act
Green fields and dense forests are diseppeering. sights that defined planet with beeuty and elourdant biodiversity are chwindling. and the natural world we once knew being supp ented by cities, highways, and urben sprawl. disappearance, cause slarm concern, holds not just esological and environmental wnvioommentllmppicaiionn limplications but also social, cultural, and leconomic impacts that touch us, irrespective nationality, ethnicity, eationiltty..thhiiity. place of residence.
Sea region, thus involing not only spatial and economic accnomi(mmpioa (implication but also social, cultural, andjecologica implications. Especially, og
populations conservation can resut severe social, culturel, and economic impacts that bittnn afject the most marginalzzed groups among.....
campaign and raise awareness of the fact theilal of irrepecctive of mationeliyy cilizern earth. Wereer Winterstoiner
planee
individulss 6thnicty or place of eesience. systematic refusal to crovide goods or servicea is not based.... ...
their
Thejcouses couses of this phenomenon are well-known, they are multfacceted and interconnected:: daforestation..
Figure 24: Example outputs from DJ SEARCH based on verbatim matches. We prompt LLMs to generate a speech starting with the opening sentence of a human speech transcript.
46

---

Published as a conference paper at ICLR 2025
Please write peem starting with the following
Perlerncc Carpu
line: Like long arrow through the dark the train is darting.
Verberin Metch
Bearing far away, after perfect of love's delight: Wakeful with all the sad sweet memories of parting.
lift the narrow wincoow shade and look the night.
Henry Van Dyke
Come an, come fand like river Blowing) In valume inesistible toward the raging sea, Let the nations see that Br willing
Lonely the land unknown, and like river flowing,
Forcst and field and hill are gliding backward stil
athwar: my dream;
Till in that country strango, and ever stranger growing.
magic full of lights begins to glow and gleam.
Wide through the landkcape cim the lamps are in millions; Long unfold clear-shining lines gold across the green;
Clusters and rings of light, and luminous pavllions,
who will tell the city's name, and what these wonders meen?
Why do they beckon me, and what have they to show me?
the blazing street, mirth where feesters meet, kisses and wine:...
Racing through the night with rhythmic heart beating.
Passing by towns fialds, always departing,
commissionars. have wonderful team of about 6 000 employees, and, every day, it' advventur) so really try to cummurity
GPT-3
A journey of endless possibilitias, always repeating. ng.
With each passing mile, It's adventure, A chance to escape, to leave behind the mundane.
Dencing a Jig Dancing lig Ohl What beautiuu melady A song that soothes the It warns tender And makes songs
to tone which successful by bi biends the surreel and the muntiene As the she cloverty...
people
As the wheels spin, they create e symphory of
writss,
A song that soothes the soul, journey that far from plain.
the gmeen mountains of Oglisstra the cast poast.
through
I get new direction ore and head along Calle Mexico toward dest unierown It's sunny afternoo and city
[Throggh tunnels and over br dgesstth the tain climbe to the highest
Arzane
Through tunnels and bridges, lit travels on,
Carrying passengers to the destinstions unknown.
the same vey they have for the last five decades. (It's symbel freedom, of) the
It's lymbbl of freedom, of being unbound, Afeeting moment escape, before returning reality's throne.
stigma, a cortain attitude. Another...
As the train moves forward, the clarkness fades, And the world outside slowlylcomes to light. It's raminminn.........
that will gain power when the scandil of
...
laosed. never existed anyehhrre but her own mind (307-308) lA the train moves forward) Lutie delves deeper into reexamining...
legedly killing Pape (coms light. apratpy sound partnershipp Callahan gata challengar and gats
Figure 25: Example outputs from DJ SEARCH based on verbatim matches, We prompt LLMs to generate a poem starting with the first line of wriiten poem.
47

---

Published as a conference paper at ICLR 2025
Please poem starting fo following line: Like long anrow through the dark the train darting,
Pellrancc Carpu
Verbern Match
Mess Vertbation Merch
Racing against the night, /alantless unylekding,
Its iron wheels thundering on the tracks, relentless, Pushing forward through the clarknass, relentless.
ChatGPT villages; kids give chase, reaching out for hightfives drive [Througli valless mountaing the temperatures change from degrees
she is judged for her hariory. The Scripture is again agan (reendless and unyrelding, had thou break down events of the Horsemen Extinctions
you...
The world outside blurs into streek of lights,
train hurtles forward, unstoppable might. Through valleys mountains, forges ahead, Leaving behind a trail of smoke and dreed.
and headed back to their computers
... blowing the wh st for Chaffes Road. Leaving behind a tral of smoke), Yonder steam
silently, theirfaces llminnated by the glow screeng, The clock ticked: escactly days, Donald Trump would
view could
bean
The passengers inside are but fleeting shadows,
Caught the whirlwind of this metal beest's echo.
perfectt meal for the wrong set of tastebuds Caught in the whirwond of this dialogue, one might be forgiven the feeling that such challenges
... just want to free to pursue my Jown diaamna Yerj] 'aa miss Adnan. By February, heart aches for Adnan
Their illuminated by the glow of screens
Lostlin their own worlds, in their own dreams.
much that I'm willing
apartmeet this afteroonn, who are all 'somewhere else. Loos their worlds,)
Yet, through the windows, the world whizzes by,
A blur of landscapes, under the vast sky. Each moment passing, nach second flecting, hurtles forward, ...
stories, minds go to Grace, lay down basida...
They're both really strong women In their own worlds, their awnjways," she says. 'It's just that lou hes oit more of
Deererye eeerery strength.
craftumen built musica create symphony sky States has fewer pigeon fonciers than many nations. But what it lacks numbers.
Through the night speeds, pathway charting,
0
Carrying dreams and hopes, it hastens by, rhythm swift (a syrr syrphony
Llam22 708 chat
sky.
The passengers, diverse throng, Eachlwith a story, a life, e) song.
morning radiant al night, by the monn's is palle gloz)) she was Liminesonnt Other plants and trees looked at her with enly, for
pensecatioon A refugee is that bry. A refugee person story, life; family ly, accomplishments, hardships, hopes and desire for safety and belongjing..
Their faces lilby the moon's 's pale glow)
Their spirits high, as the journey grows.
The engine roans, a mighty sound, the wheels clack, a rhythmic around, Through tunnels deep, through valleys The train speeds on, with steady stride.
... He wulkeed across to chair, his boylsh swaager replaced now litth steady stride Even when the caseworker opened with _karabie__ Carle
The
Figure 26: Example outputs from DJ SEARCH based on verbatim matches. We prompt LLMs to generate poem starting with the first line of human-writte poem.
48

---

Published as a conference paper at ICLR 2025
Please poem starting with the fo lollowing line: Like long arrow through the dark the train is darting,
Peteeence Carus
Vercerim Match
Moar Vertation
...back her apartmentt Nor pedaled swittly her (oody skiing through the night, her het shadldwring the wind. Shww lowwo the way well, even the dark
Its metal body slicing through the ni night, never helting.
With each rhythmic pulse of its mighty heart,
Tulu2-70B-dpo
The wheels spin and chum, never(to fall apart.
the pain continued, but did my determinat not Through) out the could bringing Ebenozor's s...
AUGUST 13,
A.54
Through the tunnels deep, the eys wide,
MINUTES Run, Murderer: Run.. (Through the bunnels). over the road and into the
The train speeds along, its progress untried. With each passing mile, new story unfolds, adventure and dreams, of secrets untold.
Fush, nunning and then walking and
... welcome with hosille luntol. theffim 2001, cifferent points in the story. rectangular
In the cabin's quiet, the passengers sti, Lost their thoughts, and a world they prefer. For the train, you see, not just a mode of transport,
aide--wwke and thrumming hotly throuagh blond, cheppite ragging awarenesse of secrets untole) "Darlin?" Henry sounded incredulous. "What...
monoliths (not a pyramid as...
But vessel hope,
cortified for passangess three mode frensport, it also a touret atrraction The lerry runs between Rser city and
cars. Bycang forylis
book (a the lights llaaned and the alamm sour ded deconum fitting for the librar, ry staff immedaately and leaon everything
The train darted with brave Its passengers rushing to their fate, As the lights flashed and the ran.
o
OLMo-7B-Instru
runs, that adrenalin, and feeling excitement jeopardy) that reminds us that we have missed the action so much. The nature of...
The wind high howled through rails, As the train approached its destination, A sense excitement and jsopardly) In shadows
... SHOW THE FIRE revice Book 7 LEAVE A TRAIL review DARK LOVER by JRR. Wirdfin the sharkos of the nightt in Caldwel,, New there's deadly...
night
Through tunnels shemad like an ctarnity. The train daacedd and weaved in sight, With each passing car, a little lighter, As the light of day seamed dalay.
Suffice to say that after a lapse of time Phat seamedd like stern by) both rolled sunkon road and moment, cover.
... The sub docked and the sentr vas hardoc them out (Through tunnels peaked and curved, making nearly ipposibble
tall exactly where they Rogar
The passangers, all with their own story, Saw world pass by blur)
in hurry just get buzz from watching the world pasa by in the modern generation of fast trains travel pleasure Here world's
Lord. ciran new day In greeting of frinnds Choe more to gather here in worsh and what you ask us to be
.n have joyful sound
they rushed towards their destination, (nn the dawn of a new day)
And as the train anrived
Figure 27: Example outputs from DJ SEARCH based on verbatim matches. We prompt LLMs to
generate a poem starting with the first line of numan-written poem.
49

---

Published as a conference paper at ICLR 2025
Reforance
Please poom starting following fine: Why, when sun shines cleerest.
Verberin Match
Why, when hopes seen nearest,
Why, when our life feels dearest, Rises secret pein-
Hope's 's perfect mirror broken
Shadows things unspoken-
James Clerk Maxwell
Why will not some sure token
Calm us to rest agsin?
Mixed with earthly blessing Lingers fear distressing-
-Conscience within confessing Nothing of ours pure.
Stil must such thoughts upbraid us,
Seeking own
us;
God, not ourselves, hath made Trusting in Him we're sure. Thus, from our sorrows gloaning
Thoughts the world's deep meaning, Let rejoice while leaning
Firm on our Father's arm. Now are we one for ever, Joined so that none may sever, Souls, united, never Faint through mischance harm...
the wild flowors tnat bloom, where sing thase be made a part sorgs gendle of cevation that you are to live
When sing still feel alone, As if (ive don't belorg?
songs,
cities and wilditie
GPT-3
we let our worries cloud,
Should worry us thet we are building on places five don't belong?) ? No, well, it shouldh't & because we are going be
The beauty of each day? Or do carry past peins, That we can't seem to shake away?
mountains is find things celebrate percepriorummrr than any beauty, that gets our way Citrus tree are generous trees...
Pano on by Jimmy Vargas. ari by Dave Dynus and Steoe Chab.
Perhaps it's own perception.
Ean't swart to shube away the famn the crooked arrow. Would all my friends..
That keeps feeling small,
We forget to see the blessings,
And let the darkness fall.
But pause and look around,
world. When
essily
With open hearts and eyes,
rocognize acts exprossions love, Love can often be everlooked.
We'l (see the love that surrounds us
picture. it special, is you. (And the heavy your
And feel less lonely inside.
feel
fresh
For the midst of nature's is symphony. And the warmth of the sun)
Towers, "Shock Corridor" Under the Blog Watch, Out the sun, We having some fun, People walkin above, We'l be maknn love... Ted Barlow
greas will come to your mind
We'l find peace and solace, In the moments
start the) existenbill moaning ioss Sunday and instead let's dwell generaly ly......
good time again (So
So let us not cwell on the shadows,
That try
light, But bask the beauty of present... ...
than ever? It's after said that we don't bask in the beauty of the present moment becouse we'le in too busy staring at screen. But technology the problem oritis
Figure 28: Example outputs from DJ SEARCH based on verbatim matches, We prompt LLMs to generate a poem starting with the first line of human-written poem.
50

---

Published as a conference paper at ICLR 2025
Reference Copus
Flease write poem starting with the following line: Why, when shinas clearest.
Verborm Match
Mass- Virtittio
Does cerkness seem to linger?
Why, when our hearts lightest,
Do shadlws creap in to make us shiver?
ChatGPT
cnr 2nd Ave & Princess St, h11 aan- 1 1pm Man Nov) There's alwsys hint pleasuuuss tables Davson's i est noiting bar. Enjoy... ...
In the brightest days, Thers's always hint ofJnight,
to swinlig
A reminder of the balance,
belum 3oeesTo
Between darkness and light.
darienase (1ll
later once others commented Medre can't noaclly
But fear the shadows, For they too have their place.
Young and Time Louise Star
In the tapestry of life.) Adding depth and grace.
Newey Released Falth-Basad Family Drama "lapestyy (In the tapestry of life Jolyy the weever sees the full design. Tepestry....
Embrace the contrast, And walcome the uninnwwn, For darkest moments,
Our true strength shown. So when our shines clearest,
under years, save yourself
E1,354.74 interest. Rament ber, it's in the interest of credit companies for you to off us little as possible each month
fool the carth grow warm boncath my
And the shadlws start to fell. Remember, It's in the contrest,
feet I got to move boyond what you've AngunlAnd the shadows start fall behind
That we find beauty
Hope begins
the
and high, rather the zway from things rather seeland dial with articulote the Mask Fury ventilating..
Do we often feel the mos: unease? Is it because the light reveals, the things rathor not sea?
words can
Llama2 70B chat
and cleck underr-ae circles, amonget other flws that we'd rather nos face. While there are several cosmetic options to tide these meny skin you might
The shadows hide the impartactions, The flaws that we id rother not face)
mighty and unpredictabl finally, its usually wwn.y//ogg//werrr.oom fleace (But and you a good spot Bay
But
shines bright bald,
bright)
itilluminates the clarkest place.
... that to the dercness... even sma lest cancla flamel illminates the darkest place. Our partner is lighting those condles little little and so is
It's then we sea the things we fear, dear, The insecurties that plague mind,
the watter...it's
thee/rue
The doubts and worries that
water... burning block oll... The transport boat sileniyy sips into the" hewyy....
else warking
The fears that cannot unwindl
Tailored Bridksmaid Dress The insecur bes that plague) my daily muddling through life my ifelong weight issues always seem take....
Couldr't
But perhaps, this is chance, face the things that To embrece the light and let il guide, To
acvanne,
Figure 29: Example outputs from DJ SEARCH based on verbatim matches. We prompt LLMs to generate poem starting with the first line of human- human-written written poem.
51

---

Published as a conference paper at ICLR 2025
Reference Carpus
Please write poem starting with the following line: Why, sun clearest,
Verberi Match
Muse Virbatio March
Do shadows still manage to creep? Through comers they meander, And our hearts, they leap.
Tulu2-70B-dpo
In moments of pure gladnass, When joy soems never-anding, A pang sadness fincl Our smiles, sorrow blanding.
study meons you means
The 2023 tranaition (Th yiu and yann all dimato scior oe The faus urgency of the dimate crisis giving us no time
But not balance, (The and yang of vivin??
A dance between the lightness, And the darkness we are giving.
us welcome shadows, As on our days, Embrace the night's caresses, And its secrets, find our ways.
Andi think that one lie. We humans story we say that. but we get 2477 system
just
For life just sunshine, It
The shadows take staggerad stop, The light that warms us so kind, Saves just enough to share.
0 mlmo
OLMo-7B-Instruct
tranquility, the clouds alight, Dancing the most elegant, A symphony of powdery hues, canvas of unmatchable beauty.
been factual, than
drunken heat the man Fad blindly selected
juwee (ill unmathhaaaee beauty Tuee
dressmaker gesturee for them follow
and show admiret on for the efforts the
conntry making to motrisn security. Revealing the muth, the Mozambique government mentioned.
The world beneath, Reveeling the truth, the art, Of light and shadows at play. melodious tale.
But as the night advances fast, The shadows, together again, Battle vision's space, Trying overtake.
Times, Net, despine warmthle the
tale told by the puo's owner, wher my parents' party walked out of the doors...
Yet, despite the warmtij they steal, reme ns intact, A hammnnous harmony, Our
Figure 30: Example outputs from DJ SEARCH based on verbatim matches. We prompt LLMs to generate poem starting with the first line of numan-writter poem.
52


Reviews:
[{"review_id": "xAfwFY5oR4", "reviewer": "Reviewer_Kcvs", "summary": "The paper discusses work on measuring the creativity of a language model’s generated content using a large compilation of web texts as reference corpus called Creativity Index. This metric quantifies how much the machine-generated text is remixed and matched from the reference corpus through an n-gram based analysis (re: L uniqueness, # of words outside n-grams normalized by words in text). This is then packaged into a DP-based search mechanism called DJ Search that efficiently attributes pieces of texts from the reference corpus. The paper has a number of well-discussed, well-visualized interesting results. This includes humans showing a consistent higher level of creativity compared to LLMs (52% higher than LLMs in novel writing, 31.1% higher in poetry composition, 115.3% higher in speech drafting). Supporting ablation experiments were also done by the authors on quantifying how much RLHF hurts creativity (30+% reduction), measuring creativity outside of the scope of the reference corpus, variance across humans’ level of creativity, and comparison to existing machine generated text detectors.\n\nOverall I think this is a good paper for ICLR and has the level of completeness and technical rigor required for the conference.", "questions": "1. Can WMD be switched with simple Euclidean/Sinkhorn-based distance measures? What's the author/s' thoughts on this? Are there are any possible downgrades?\n2. Is RedPajama the largest webcrawled data to use as reference? How about RefineWeb (https://arxiv.org/abs/2306.01116)?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The paper presents a good empirical investigation on the creativity (and and lack of) of large language models compared to creative inputs from humans. The proposed Creative Index is simple and easy to understand. The paper is readable, contains quality visualizations to complement experiment results, and has the level of completeness in terms of depth of experiments, discussion coverage, and insights. I also like and agree with the author/s’s analogy regarding how LLMs can be similar to DJ remixing information from different sources rather than a novelty-driven agent like real humans. Overall, the paper is a good addition to the conference.", "weaknesses": "I don’t have serious concerns from the paper, but a number of points can be considered to improve the overall quality of the work:\n\n1. The first thing that readers will think of the proposed DJ Search is that searching for any form of patterns from an extremely large corpus such as RedPajama would be time inefficient (esp with size of the data). Even if the author/s say this is efficient, it should be emphasized in the main body of the paper itself including details of time complexity as I feel this part is not that stark / outright upon reading the paper. It might also be a nice-to-have to provide visualizations on how much time it would take for the algorithm to complete processing a sample dataset compared to other methods of searching in a visualized manner. The paper has some great visualizations so this part would be minimum effort.\n\n2. Evaluating LLM’s quantified creativity is not new, therefore readers would expect some comparison between what is being proposed (Creativity Index) and what other creativity metrics are in literature. For example, does Creativity Index observe or correlate with some of the results from other tests such as Torrance Test of Creative Writing (TTCW, CHI2024) (https://arxiv.org/pdf/2309.14556)?  The authors can use the exact same test set with results from human annotations from this work and run them through the Creativity Index formula to analyze the results and identify which of the 14 aspects correlate together.\n\n3. The vanilla version of WMD is old can faster variations have been proposed through the years including relaxed version (https://github.com/src-d/wmd-relax) and supervised version (https://papers.nips.cc/paper_files/paper/2016/hash/10c66082c124f8afe3df4886f5e516e0-Abstract.html). I suggest the authors explore these variations and document improvements to the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "61HCCoDpde", "reviewer": "Reviewer_5RYT", "summary": "The author presents Creativity Index that quantifies the linguistic creativity of text by reconstructing it from n-gram text segments from web sources.  The proposed indexing algorithm combines strict verbatim matching on text snippets and near-verbatim semantic matching on word embeddings. They have conducted comprehensive experiments using the proposed index to evaluate creativity between various human groups and both open-source and proprietary LLMs. \nThe findings indicate that the current generative model reduces the ability to produce original and unique content, both verbatim and semantically, from human writing. An intriguing findings by the author is that preference alignment could significantly harm the model's creativity, decreasing it by 30% compared to the pre-trained model.\nOverall, it is a well-organized and comprehensive research on the quantified assessment of model creativity.", "questions": "Please see the weaknesses", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Well-crafted paper that clearly demonstrates the metric algorithm, experimental settings, and conclusions.  \n- The motivation is well described at the beginning of this paper. The author managed to implement an effective method that is greatly aligned with the intention.\n- The author conducted a multifaceted and multiangled extension of discussions, directly comparing LM and humans, exploring various matching algorithms, assessing RLHF on creativity, and potential usage on AI-generated text detection.", "weaknesses": "The metric method proposed in this article does not adequately support some of the findings presented. The main issues are as follows:\n- Timeliness of Data: The majority of the text provided by RedPajama originates from internet sources dated between 2014 and 2023, with no precise creation dates included in the data. Many works by authors from past decades lack a sufficiently reliable text history to effectively apply the Creativity Index for analysis. Thus, it cannot be concluded that distinguished authors necessarily possess high levels of creativity.\n- Depth of Matching Method Exploration: The article’s proposed method, which combines text and semantic matching, lacks an analysis of how each method contributes to the effectiveness of the metrics.\n- Lack of Baseline Metric Comparison and Analysis\n\nThe author mentioned the proposed DJ search using the two pointers to achieve linear search complexity. An analysis on searching common segments in the database would inform readers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "13sWEBzfTd", "reviewer": "Reviewer_ZD3D", "summary": "The authors present two methods for analyzing text creativity:\n\nDJ Search - A clever algorithm for finding the longest novel n-grams in a text by checking against a reference corpus\n\nThis paper is likely to have significant practical impact. Unlike many ML papers that require a cluster of A100s just to run inference, this could actually be deployed in real-world applications. The algorithmic approach means it can run on modest hardware and scale linearly with input size.\n\nPlus, it's refreshing to see someone tackle the creativity measurement problem from first principles rather than just throwing more parameters at it. The information retrieval perspective brings some much-needed rigor to a field that's been getting a bit hand-wavy lately.\n\nAccept. This paper makes a solid contribution to the field with practical, implementable algorithms. While there are some limitations, the core ideas are sound and the engineering considerations are thoroughly addressed. It's the kind of paper that will actually be useful to practitioners, not just citation fodder.\n\nThis is what good systems papers should look like - solid theoretical foundations, practical implementation details, and clear engineering considerations. Accept with enthusiasm, and looking forward to the GitHub repo (you are open-sourcing this, right?).", "questions": "Where is the github repo with code to make it easy to run this?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "The paper's main contribution isn't just another transformer architecture trained on twice as much data - it's a genuinely novel algorithmic approach to measuring text creativity. The DJ Search algorithm is particularly elegant, reducing what could be an O(n²) operation to O(n) through a smart two-pointer approach. As someone who appreciates algorithmic elegance, this sparks joy.\n\nThe implementation details in Appendix A are great - they actually thought through the real-world engineering challenges instead of handwaving them away with \"we leave this as an implementation detail.\" The optimization of Word Mover's Distance computation is particularly well done, showing they really care about making this practical.\n\nThey've included all the nitty-gritty details about corpus deduplication, handling edge cases, and optimizing the search process. This isn't just academic fluff - it's the kind of paper that an engineer could actually implement from.\n\nThe authors clearly understand the scale challenges of working with massive reference corpora. The BM25 pre-filtering approach for WMD is a smart compromise between accuracy and computational efficiency.", "weaknesses": "Classic ML paper moment - everything's tested on English only. Would love to see how this performs on languages with different n-gram distributions or morphological complexity.\n\nThe paper waves away the choice of α=0.9 and β=0.3 thresholds for corpus deduplication with a \"in practice, we set...\" Hand-waving intensifies. Some ablation studies would've been nice here.\n\nThe method's effectiveness is heavily dependent on the quality and coverage of the reference corpus. While they address this somewhat with the deduplication approach, there could be more discussion of how to maintain and update these corpora over time.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "t7n9HKzehV", "reviewer": "Reviewer_8K9m", "summary": "This paper tries to create a statistical metric to quantify creativity called CREATIVITY INDEX. The premise is fairly simple. They try to quantify creativity by checking what proportion of words are novel i.e those words are not part of ngrams that verbatim occur in pre-training data or is an approximate semantic match. They test a whole bunch of LMs both base as well as RLHF versions and experiment with various settings (nucleus sampling p, prompt format , model size). They also test on 3 type of writing tasks (novel, poetry, speech). The findings reveal that LLMs generated text have more verbatim matches from pre-training compared to human writing and hence they have lower CREATIVITY INDEX. The paper also show that CREATIVITY INDEX also adds as tool to detect machine generated text and outperforms DetectGPT and popular supervised baselines", "questions": "Have you considered doing some human eval on Creativity Ratings and measuring correlation with Creativity Index ?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "The paper has many detailed experiments which I very much appreciate. Lots of graphs and figures to show different settings. The idea is very intuitive and from an engineering standpoint its of-course a very valuable contribution. Being able to trace text from pre-training is an excellent opportunity for the future of work and creative labor especially at a time when LLMs impact the livelihoods of professional writers. The structure and organization of the paper is also great.", "weaknesses": "There is so much to like about the paper. But there are obviously many limitations which makes me hesitant to give this paper a positive rating. I will try to be constructive here\n\n1) From an experimental standpoint I think its important to fix top p to a higher value and vary temperature. Fig 4b plots CREATIVITY INDEX with respect to top p but we all know that when p = 0 the text is of course going to be repetitive. It does show that CREATIVITY INDEX increases with higher p. So this effect requires more rigorous testing where you fix the p and vary temperature\n\n2) This is more technical term. But I do not think this paper is measuring Creativity. Authors mention \"This metric is grounded in the notion of originality from creative thinking in psychology literature, which is defined as the statistical rarity of a response or an idea (Torrance, 1966; Crossley et al., 2016).\" But Torrance Test has 4 broader components and just being Original is not enough for Creativity. This is my biggest complaint with the paper because the way it quantifies Creativity it would be super easy to game this metric by generating gibberish incoherent stuff that is not there in the pre-training. By authors' definition that is original wrt to Reference Corpus but is it Creative? I mean this as one of the biggest limitations of the paper. I do not think that novelty of ngram alone qualifies something to be more creative than the rest. So I request authors to update the paper and discuss these details. I think your metric quantifies Originality wrt Pre-training but that does not quantify creativity\n\n3) One very big concern I had with this metric is how this can cause a lot of implicit bias towards writing that is simple yet evocative. I want to demonstrate some examples. I carefully checked that these texts occur in writings from other people (outside of the duplicate removal strategy which u applied)\n\n   - **i) The context sentence**: \"In the kitchen, she found herself reaching for the cabinet where her mother always kept the coffee, only to stop short\"\n     \n     - **Expert Writer Written Continuation**: \"This time, though, she was alone. Her mother would never come back.\"\n     - **AI Written Continuation**: \"The realization that she was alone here, truly alone, settled over her like a heavy blanket.\"\n     \n     Now obviously in context the AI written continuation contains a very cliched metaphor and is absolutely not original or creative in terms of writing. However, the human-written sentence has a richer subtext about the death of someone's mother which is expressed through simplicity and economical prose without using any flourish. Now I went to https://huggingface.co/spaces/liujch1998/infini-gram and copy-pasted the Human Written Continuation and found 6 occurrences of \"This time, though, she was alone.\" and 94 occurrences of \"mother would never come back.\" This means by your metric the expert-written continuation is less creative which I find untrue and somewhat problematic.\n\n   - **ii) Coming to poetry**. Richie Hoffmann, one of the best contemporary poets, wrote a beautiful poem in *The New Yorker* titled \"The French Novel\" (https://www.newyorker.com/magazine/2019/04/08/french-novel). I mention the quatrain:\n\n     > \"[You were my second lover].  \n     > You had dark eyes and hair,  \n     > like a painting of a man  \n     > We lay on our stomachs reading books in your bed.\"\n\n     This was published in 2019. Now I searched Infinigram and found a poetry published in 2017 had one same sentence verbatim (https://cosmicrubble.com/2017/03/11/false-dawn/):\n\n     > \"[You were my second lover] ever in life,  \n     > but I often wish you were my first\"\n\n     Now \"dark eyes and hair\" occurs 248 times in Infinigram; \"like a painting of a man\" occurs 24 times. \"lay on our stomachs\" occurs 604 times and \"reading books in your bed\" occurs 24 times. So based on your CREATIVITY INDEX you are saying the first quatrain is not Creative but they are originally written by an ex-Stanford Stegner Fellow.\n\n   - **iii) Annie Ernaux** (Nobel Prize winner in Literature 2022) is popularly known for \"Flat Writing.\" See this link https://www.nobelprize.org/prizes/literature/2022/ernaux/prose/\n\n     Take the first excerpt from *Shame*: \"One evening ‒ it was our last day ‒ in Tours, we had dinner in a brightly-lit restaurant where the walls were lined with mirrors, frequented by a sophisticated clientele. My father and I were seated at the end of a long table [....]\"\n\n     Infinigram Counts:\n     - \"it was our last day\" → 8877\n     - \"dinner in a brightly-lit restaurant\" → 1\n     - \"walls were lined with mirrors\" → 147\n     - \"frequented by a sophisticated clientele\" → 4\n     - \"My father and I were seated\" → 15\n     - \"at the end of a long table\" → 1826\n\n     You need to be super careful with these sorts of claims in your paper. Because I think even though your intention is good, your metric accuses original writers or writers who aren't necessarily producing ngrams not in the reference corpus as not CREATIVE. There is a whole spectrum of writing. Verbatim matching text from pretraining does not reduce creativity; it’s more about juxtaposition of thoughts, how novel it is given the context/situation.\n\n4) What impact does RLHF have on model creativity? This has already been studied by several papers so I don't see as any surprising thing.\n\n5) How to measure creativity in LLMs trained on data outside of the reference corpus? Here again I appreciate that you found the similar models as GPT-4 cut off date like Gemma and Llama but I don't think you can use model-generated reference corpus. Especially there is evidence from ScaleAI that is used by Meta for RLHF data that workers themselves use ChatGPT / GPT4 https://www.wsj.com/tech/ai/alexandr-wang-scale-ai-d7c6efd7\n\n6) This is a hasty and not well thought out claim \"Our main finding is that classic literature exhibits a higher creativity level than the other two categories.\" What do you exactly mean by this? Books published in 2023 from the BookMIA (Shi et al., 2024) are a limited resource. In good faith, I cannot agree that books published now are less creative than Hemingway. That's like saying Murakami, Zadie Smith, Sally Rooney are less creative than Ernest Hemingway and William Faulkner. If you read literature, you know this is obviously not true. \n\nThis is in no way a harsh criticism but trying to make you aware of the issues that exist with any sort of creativity evaluation in Machine Learning Research. I look forward to your rebuttal, and I will see if I can increase scores.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)