Death of the Novel(ty): Beyond N-Gram Novelty as a Metric for Textual Creativity
Authors: Arkadiy Saakyan
Najoung Kim
Smaranda Muresan
Tuhin Chakrabarty
Venue: ICLR 2026 Poster (2026)
Keywords: creativity
creative writing
evaluation
creativity evaluation
machine creativity
n-gram novelty
Primary area: foundation or frontier models, including LLMs
TL;DR: Study with expert writers cautions against using n-gram novelty for creativity evaluation. Open-source LLMs tend to sound less pragmatic as n-gram novelty increases. Evaluation of close reading skills of frontier and fine-tuned LLMs.

Abstract: $N$-gram novelty is widely used to evaluate language models' ability to generate text outside of their training data. More recently, it has also been adopted as a metric for measuring textual creativity. However, theoretical work on creativity suggests that this approach may be inadequate, as it does not account for creativity's dual nature: novelty (how original the text is) and appropriateness (how sensical and pragmatic it is). We investigate the relationship between this notion of creativity and $n$-gram novelty through 8,618 expert writer annotations of novelty, pragmaticality, and sensicality via \emph{close reading} of human- and AI-generated text. We find that while $n$-gram novelty is positively associated with expert writer-judged creativity, approximately 91% of top-quartile $n$-gram novel expressions are not judged as creative, cautioning against relying on $n$-gram novelty alone. Furthermore, unlike in human-written text, higher $n$-gram novelty in open-source LLMs correlates with lower pragmaticality. In an exploratory study with frontier closed-source models, we additionally confirm that they are less likely to produce creative expressions than humans. Using our dataset, we test whether zero-shot, few-shot, and finetuned models are able to identify expressions perceived as novel by experts (a positive aspect of writing) or non-pragmatic (a negative aspect). Overall, frontier LLMs exhibit performance much higher than random but leave room for improvement, especially struggling to identify non-pragmatic expressions. We further find that LLM-as-a-Judge novelty ratings align with expert writer preferences in an out-of-distribution dataset, more so than an n-gram based metric.

Published as a conference paper ICLR 2026
DEATH OF THE NOVEL(TY): BEYOND n-GRAM NOVELTY AS A METRIC FOR TEXTUAL CREATIVITY
Arkadiy Saakyan Columbia University
Najoung Kim Boston University
Smaranda Muresan
Tuhin Chakrabarty Stony Brook University
Columbia University
a. saakyan columbia.edu
ABSTRACT
N-gram novelty is widely used evaluate language models' ability to generate text out- side of their training data. More recently, it has also been adopted as metric for measur- ing textual creativity. However, theoretical work on creativity suggests that this approach may be inadequate, as it does not account for creativity's s dual nature: novelty (how orig- inal the text is) and appropriateness (how sensical and pragmatic it is). We investigate the relationship between this notion of creativity and n-gram novelty through 8,618 ex- pert writer annotations of novelty, pragmaticality, and sensicality via close reading of human- and Al-generated text. We find that while n-gram novelty is positively associated with expert writer-judged creativity, approximately 91 % of top-quartile n-gram novel ex- pressions are not judged creative, cautioning against relying on n-gram novelty alone. Furthermore, unlike in human-wriin written text, higher n-gram novelty in open-source LLMs correlates with lower pragmaticality. In an exploratory study with frontier closed-source models, we additionally confirm that they are less likely to produce creative expressions than humans. Using our dataset, we test whether zero-shot, few-shot, and finetuned models are able to identify expressions perceived as novel by experts (a positive aspect of writing) or non-pragmatic (a negative aspect). Overall, frontier L.LMs exhibit performance much higher than random but leave room for improvement, especially struggling identify non-prag praggaaaaaiic expressions. We further find that LLM-aa---Judge novelty ratings align with expert writer preferences in out-of -iitti--iitiittiion dataset, more than n-gram based metric.
1 INTRODUCTION
Advances in large language models (LLMs) have led to their widespread applications in writing. In fact, recent studies (Handa et al. 2025 Chatterji et al. 2025) show that writing assistance remains one of the main use cases of LLMs. At the same time, researchers have been raising concerns about how writing assistance tools can reduce collective human creativity via homogenization effects (Doshi & Hauser 2024
Kobak et al.) 2025 Zhang et al. (2025), proliferation of AI slop (Chakrabarty et al. 2025a Shaib et al. 2025
or copying from training data (McCoy et al. 2023
Such challenges lead to growing need for robust textual creativity evaluation. Recently, tools like WiMBD (Elazar et al., 2024). Rusty-DAWG (Merrill et al.. 2024). aafini-gram-mming (Liu et al. 2024 Xu et al.
2025 have been developed to efficiently search L LMs" pretraining corpora and evaluate the novelty of
their generations. Building on these tools, Lu et al. (025 introduced a metric called CREATIVITY INDEX, which places significant weight n-gram novelty lack of occurrence of textual fragments in some large (several trillion tokens) corpora- for measuring creativity of text. However, literature on the psychology of creativity would consider such approach as fully adequate: based the widely adopted definitions
We will release the collected dataset and models github. com/ wttp////aaaaaaa//aaaaaaaaaa//eaa

---

Published as a conference paper ICLR 2026
of creativity, novelty a necessary but not sufficient criterion (Sawyer Henriksen 2024; Runco & Jaeger
2012 Csikszentmihalyi et al. 1997. Amabile 1983 Jackson & Messick 1965). An expression generated
by an LLM may be novel with respect to the pretraning corpora but make little sense (e.g., stitched the frayed edges of cloaks and ragged seams of fear in Figure I precluding it from being judged as creative. At the same time, an expression may not be n-gram novel with respect to pretraining but still be creative. For instance, that's 'ss' the bottom of the heart, where blood gathers has low n-gram novelty, but was rated as creative in the context of the given passage by an expert writer because of how it comes across as emotionally foreshadowing (sce Table Tp.
While prior work CChakrabarty et al., 2024b) has relied expert human evalu- ation for measuring creativity of long form text via fine-grained rubrics, LLMs struggle to imitate nuanced expert judgments, mak- ing this approach difficult to scale and au- tomate. Instead, we turn to the standard definition of creativity (Runco & Jaeger 2012), operationalizing it as a combination of novelty and appropriateness. We further decompose appropriateness into sensicality (making sense on its own) and pragmati- cality (making sense in context). Figure [ shows an example of sensical but non- pragmatic expression (suddenly mentioning sacrifice at the end of the passage without providing appropriate context).
ruieed hed and nevert Nasscrilei
Consenallly
Yes
No
What secreect to the gi''s mect on redding aboutic, that leening about
vader doewn't know eweer cricatt
QUAU
lec Serve De to Oumnicaat
Nakiz Seroe Conkentally (oegratic) No
No
To formalize the failure modes of novelty a metric, we conduct study on the re- lationship between n-gram novelty low occurrence trillion token-level corpora and creativity. Expert writers were asked close read AI- and human-written pas- sages and provide expression-level sensical- ity, pragmaticality, and perceived novelty ratings. Mixed-effects regression model analysis accounting for individual rater vari- ation and other confounding factors re- vealed a negative relationship between n- gram novelty and pragmaticality in open-source models, with x 91% of highly n-gram novel expressions not judged as creative. This cautions against relying on n-gram novelty alone to evaluate creativity. To under- stand whether LLM-as-a-Judge frameworks could be used instead, we use the collected dataset to evaluate how well frontier LLMs and finetuned models can emulate expert human judgments. We further validate our best performing model on out-of-distribttion data, finding strong alignment between the LLM--as--Judke novelty scores and expert preferences.
endedded cluake aber contasas iasiiinggl
desci iplan cloak
rapped
Faurdeer where her b veterncing fow dbes witching the ciata abich theauur t'tar?? mestaphhr a velus body widhis eefervele The owrsene" loe raily
Figure I: Examples of non-pragmatic expressions (top), and expressions that are both non-sensical and non-pragmatic (bottom).
2 DATA
OPERATIONALIZING CREATIVITY We operationalize creativity through appropriateness and perceived novelty components (Runco & Jaeger 2012). We decompose appropriateness into two subcomponents:
2

---

Published as a conference paper at ICLR 2026
sensicality and pragmaticality. Sensicality is whether the expression makes sense standalone (for example, an expression like "he tended at cloud finudfinger" does not make sense by itself, or in other words, semantically infelicitous). Pragmaticality refers to the expression's 's fit within the context of the passage, capturing a range of errors, including logical incoherence (e.g., "Alice felt greaf" immediately followed by "Despite being sad, Alice...)) "). or sounding awkward or odd in that context
Fi] In addition, we collect perceived novelty ratings by asking whether expression is unusual, surprising or otherwise origina Creative expressions are those thut are simultaneously judged by a human as sensical. pragmatic, and novel. Text that is creative in this sense may not necessarily be n-gram novel (see Table I.
SURVEY INSTRUMENT To obtain the annotations, we took inspiration from the practice of close reading (Smith) 2016), a literary analysis technique of carefully examining small parts of the
net ven thon alon and any of contan tler dnading Bargon Usec und Kallag CT viall
vedlccon docido affaria Desalt
cacre pne uou your ane coveaive
time aftert bur perfeceds die did from are came was the ther
eightees
Your Creathve Hightights
passage to make sense of it as a whole, "putting the author's 's choices under a mi- croscope" (Brummeti 2018). We first split the passage by punctuation into "atomic" expression units, and then sam- ple roughly 50% of them for annota- tion based on the percent of novel n- grams within the expression (see Ap- pendix [A. These expressions are pre- highlighted. Then, we ask the annota- tors to rate each of the pre-highlighted expressions in the passage for their sen- sicality, pragmaticality, and novelty (see the user interface in Figure 22. In addi- tion, the interface allows experts to high- light any creative expressions (which satisfy the sensicality, pragmaticality and novelty requirements) that were not already pre-highlighted. The annotators were required to provide rationales for why the expression was creative or perceived novel, and could leave an optional comment on other aspects. We provide detailed instruction and examples to the an- notators with simplified terms for sensicality ("Makes sense on its own?") and pragmaticality ("Makes sense contextually? (Flows naturally)""> ""turalll"")scee Appendix A.3 An example of an annotation is shown in Figure 2 where the pop-up window displays the annotation of a pre-highlighted expression, and in green the annotator highlighted creative expression.
the clecs cire tou aandikis sontic in leds decatis boloral vas tor back coud
coautt/ cart the Bechher Reserveer
ghort merail Paatt wes en skon the more the fistumm cost
being dieicctiit bookk Thu cougle sharsstesk cans get mediiiction suat. and per de
brah on sice ewcr deyee beee
This
bunch sone ast
of siy right out hore unier thom The mer book ais bikio
Lani on" "Rack in ani day.
sinee and coeene alo
Viiaes Serse On
Puted dight
and
Noser
Thomis oo
Fale fultur Wirooe tiewu.
30 (ene
Faee Car vertudlyy to loraliy
Lovn' th var, and Mitute simendy soart
declualiiny bunle ped
mhe checs"
Cante Corde Cordersilly pnd Hitoonly
bast vec ulll soping wut tor but k fovard the form duce in
aloui Biue
Eaee Car deriadyllyy to loralt)
Figure 2: Example of the annotation interface and an expert writer's annotation.
RECRUITMENT We recruit professional writers (n = 26) via listservs of top Master of Fine Arts writing programs in the US, as well published writers with a Masters in English Writing Literature through the Upwork platform. Each annotator first underwent training on at least one passage during which any instruction misunderstandings were corrected. Annotators were paid 100 USD per annotated batch of 10 passages, which took approximately 2-3 hours.
ANNOTATION The fiction domain best suited for evaluating textual creativity, as it does not impose factuality constraints unlike technical or journalistic writing, and is less subjective in creativity evaluation than poetry (Kaufman et al. 2008 2009). Hence, for seed human data we collect 50 passages of z 400
Note this allows for deliberate violations of pragmatic norms, e.g. in surreal writing.
Pragmaticallity is more stringent requirement than sensicality (an expression cannot be pragmatic if nonsensical). We make this distinction to analyze more nuanced effects of context rather than atomic expression-level non-sensicality. OOur annotations suggest that pragmaticality is a precondition for human-judged novelty, discussed later.
3

---

Published as a conference paper ICLR 2026
words each from fiction published in the New Yorker We use fully open-source (both weights and training data) LLMs: OLMo (7B) (Groeneveld et al. 2024) and OLM (32B) (Team OLMo et al. 2023 to generate LLM passages, considering both the older and the improved, larger models understand whether progress in language modeling and benchmark performance led to improvements in writing quality. The passages were chosen such that they are not present the corresponding open-source LLM training corpora (see Appendix A.2). We assign each model 25 passages from the human sample (seed passages), and prompt them to generate a similar length passage based on the content obtained from the summary of the human text (see details in Appendix C.2) To ensure reasonable completion time for each annotator task, we divide the total 100 passages into 10 batches of 10 passages each, containing 5 human and Al passages. Three annotators are assigned per batch. To gauge the ceiling LLM performance,
conduct a smaller-scale follow-up study_on on 2 frontier models: GPT-5 and Claude 4. 1, each generating 5 passages to make up one additional batch of 10 texts assigned to 4 randomly selected annotators from our pool.
STATISTICS Overall for the main and frontier model study, 8618 pre-highlighted expression annotations were collected for 2783 unique expressions, with 589 unique expressions rated as perceived novel by at least one annotator, 722 as non-pragmatic and 274 as non-sensical. Pragmaticality and sensicality were largely preconditions for perceived novelty judgments, as only 3% and 5% of perceived novel annotations were also marked as non-sensical and pragmmatee,, respectively. In addition to the pre-highlighted expression ratings, the annotators highlighted 241 new expressions as creative (satisfying the sensicality, pragmaticality, and perceived novelty criteria at the same time). Due to low prevalence of expressions annotated as perceived novel (7%) and non-pragmatic (8%), traditional agreement metrics would suffer from prevalence bias and would not be suitable (Eugenio & Glass 2004 Brennan & Silman
1992 Feinstein & Cicchetti] 1990).
Hence, we report the free-marginal multirates Kappa K free (Randolph 2005). ). Across the 10 batches, the mean K free for perceived novelty was 0.78 (sd 0.11), and for pragmaticality 0.72 (sd 0.12).
3 METHODS
MEASURING n-GRAM NOVELTY We use the infinigram package (Liu et al. 2024 estimate n- n--ram novelty with respect to LLM training corpora. Specifically, we use co-probability, which allows to assign a probability to any expression using backoff' to the longest expression present in the corpus D:
cut cnt(w (ut(W;-(a-Daa-1u0) D)
Pso(w;) w1s-1)
where n = max(n' E [1,
(D) > 0}
cnt (W: (n- (n D)
We can then use the consecutive probabilities to compute the perplexity of an expression as a proxy for n--rrmm novelty[ We use the respective OLMo and OLMo-2 training corpora a the reference D.
MODELING Annotation of textual creativity is inherently subjective. To account for annotator variation as well as other confounding factors (topic of the passage, generation model), we turn to hierarchica//multieeve modeling (Gielman & Hill, 2007) commonly used to handle nested data in behavioral and social sciences (Baayen et 2008| Yarkoni [2022 Kaufmann et al. 2025) and account for confounders in language model evaluation (Lampinen et al. 2022). Since our target variables are binary judgments of sensicality, prag- maticality, and perceived novelty, fit logistic regression models. For instance, model the relationship
*One of the most prestigious literary magazines for publishing fiction.
'Our main study focuses on LLMs for which pretraining corpora is known and hence the only models that allow accurate n-gram novelty estimation.
Defined in a standard way as [II (ws wis-1)
DDolm-vil (2.6T tokens) for OLMo and v4_olmo 032003225----33 febb-instuutt (4.2T tokens) for OLMo-2.
4

---

Published as a conference paper at ICLR 2026
between n-gram novelty (measured by perplexity) and creativity, we fit the following model for the proba-
bility of the expression being labeled as creative (i.e. sensical, pragmatic, and novel simultaneously):
(P (P [creative; 11] Bu + 31 ppl.log.stte + 8 Pgennsourca(j] gen + // + (para)
(mater)
where pp1 log st.d is log-standardized (to reduce skew and ease interpretation) perplexity, gen source is a categorical variable representing the generation source (human, OLMo, or OL.o-2), and afi), p(i] are the indices of the rater and seed passage id for expression i. Variation in each rater's baseline threshold for rating something creative is represented by the varying intercept (rater) while variation in prevalence ((aaaa of creative expressions by different topies (seed passages) is represented by the varying intercept u "p[i] Random intercepts are assumed to be normally distributed with a mean of zero and their respective group variances. See detailed specifications in Appendix B
4 RESULTS
n-GRAM NOVELTY PREDICTS CREATIVITY BUT IS NOT A RELIABLE METRIC We find that standard- ized log perplexity is significantly associated with creativity (OR (Odds Ratio) & 1.96 per SD (Standard Deviation), p 0.001). However, & 91% (n 3928, Wilson 95% CI: [0.90, 0.92)) of top-quartile n- gram novel expressions are not judged as creative (sensical, pragmatic, and novel at the same time), with 79% (n 1384, Wilson 95% CI: [0.77, 0.81]) of unique such expressions not judged creative by any of the annotators. Further, a substantial portion of creative expressions have very low perplexity: z 25% (n 605, Wilson 95% CI: [0.21, 0. 0.28) of unique creative expressions fall below the mean perplexity, and 8% (n = 605, Wilson 95% CI: [0.06, 0.10)) are in the lowest quartile. This demonstrates that while high perplexity is predictive of creativity, a non-negligible fraction of creative expressions are not n-gram novel, and the vast majority of highly n-gram novel expressions are not creative, cautioning against relying solely on n-gram novelty to capture creativity. In Table we provide examples of expressions that were judged as creative by the annotators along with their justifications, but have very low -og-standardized perplex- ity (among the lowest the dataset). Annotator justifications indicate the importance of contextual factors rather than r--gram novelty for judging an expression creative (more examples in Appendix A.44.
n-GRAM NOVELTY NEGATIVELY IMPACTS PRAGMATICALITY IN OPEN-SOURCE LLMs We fit a
logistic regression model whether expression rated as pragmatic using standardized log perplexity, generation source, and their interaction as predictors, with varying intercepts for annotators and seed passage IDs. Instances that were labeled not sensical were removed to focus cases where expressions do not make sense in the context, rather than simply not felicitous standalone.
Figure 3 demonstrates how n-gram nov- city affects the probability of an expres- sion being rated as pragmatic according to our model. We find negative slopes for both OLMo and OLMo-2 generated text, while we see no such effect for Human--ritten text. Linear hypothesis tests show no evidence that n-gram nov- elty affects pragmaticality for human- written text (B = 0.01, SE 0.09, p = 0.92). In contrast, higher n-gram novelty negatively affects pragmatical- ity of Al-generated text: OLMo (3 = 0.17, (1) 4.89, p 0.027) and OLMo-2 (8 = -0.48, (1) = 13.80, p 0.001).
Probsgiit
5.6
Standdrriized on Perplesity (nol_ 100 2015
Figure 3: Predicted probability of being rated pragmatic for different values of log-std perplexity, by generation source. Bands indicate 95% Cls. Annotator and paragraph intercepts cor- respond to population-Ievel fixed effects.
5

---

Published as conference paper ICLR 2026
This indicates that as open-source LLMs try generate more n-gram novel text, they are less likely generate expressions that make sense contextually. In Table | expressions that were rated as non-pragmatic.
we show examples of highly n- n-gram novel
Expression
Justification
PPL.
Source
Low re-gram Novelty but Creative
Even when you'le 're here, you 're not here
This is creative way of showing the emotional disconncction of the narrator and how that is the source of problems with his partner.
-2.3
human
knowing that he wouldn't tt.
found this to be a creative expression because it offers insight into Tongsu's 's mind and surprises the reader with its blunt honesty.
-2.1
human
That's the bottom of the heart, where blood gathers.
The metaphor vivid and physiological, merging the act of writing with the body in a tactile, somber way. It's also emotionally foreshadowing.
-1.5
human
High n-gram Novelty but Con-Pragmatic
and said morning blessings in a whisper that embarrassed the choirs
2.37
GPT-5
the child who was at once present fundamental fact but
In isolation and in context, this is misadventure of poor word choice, punctuation, and structure.
2.36
human
each word spoken thread in the slow weaving of trust tope suspended between her fingers like frozen gesture
[...] saying act weaving is made up physical threads is mixing metaphors:[....
2.09
OLMo-2
1,40
Claudc-4,1
Table 1: Top: expressions judged creative despite low n-gram novelty (negative log-standardized perplexity).
Bottom: non-pragmatic expressions with high n-gram novelty (positive log-standardized perplexity).
AN EXPLORATORY STUDY WITH FRONTIER CLOSED MODELS study with two frontier models: GPT-5 and Claude-4.1 (see details in Appendices C.3 B.4
We conduct an exploratory follow-up
Expanding the original dataset, compare the probability of expression creativity across generation sources (hu- man, OLMo, OL.o-2, GPT-5, Claude-
Contrast
Odds Ratio (OR)
95%
p-value
Claude-4.1 Human
0.521
[0,289, 0.939]
0.024
GPT-5 / Human
0.511
[0.279, 0.938]
0.024
4.1) by fitting logistic regression with generation source as the predictor and random intercepts for annotators and seed passages. Table 2 shows that the probability of an expression to be judged creative is significantly higher for hu- mans compared to frontier and open- source LLMs. Although not statistically significant (likely due to small sample of the exploratory study and imprecise estimation of n-gram novelty have access the models' pre- training data), we observe a similar trend a negative impact of n-gram perplexity when comparing frontier model and human expressions: the interaction terms between perplexity and generation source nega-
OLMo / Human
0.500
[0.370, 0.676]
<0.001
OLMo-2 Human
0.588
[0.439, 0.788]
0.001
Table 2: Estimated marginal means (EMM) contrasts compar- ing creativity of Al and human expressions. OR < indicate model-generated expressions were less likely to be judged cre- ative compared to human. P-values adjusted for multiple com- parisons using the method Dunnett (1955).
6

---

Published as a conference paper ICLR 2026
tive for both Claude (3 p 0.44) and GPT-5 (5 p 0.32). but slightly positive for the human baseline (3 = 0.13. p = -0.49). Future work could utilize our close reading annotation scheme to conduct& a
study.
LACK OF EVIDENCE THAT AI LIKELIHOOD PREDICTS NON-CREATIVITY One plausible hypothesis is that if a text seems more Al-generated, it is judged by humans as less creative. We fit logistic regres- sion predicting the probability of the expression being judged creative or pragmatic given the AI likelihood scores from one of the leading Al Al-generated text detectors, Pangram (Emi & Spero 2024). The detector perfectly classified AI texts assigning all of them a score of 100%; human texts were assigned varying small likelihoods of sounding like AI (see Figure D. Log- standarddndd AI likelihood scores were used as a passage-level predictor in the logistic regression model with varying intercepts for annotators and passages. We did not find evidence that passages with higher (or lower) Al likelihood scores systematically differed in the average novelty or pragmaticality of their expressions. A one-standard deviation increase in a passage's s Al likelihood was associated with small, non-significant increase in the odds that expression was judged creative (3 = 0.05, SE = 0.111, = 0.624) and a non-statistically significant decrease in the probability that it was judged pragmatic (B -.225, SE = 0.15, p 0.09).
WRITING QUALITY REWARD MODELS PREDICT CREATIVITY AND PRAGMATICALITY Chakrabarty et al. (2025a) recently explored training scalar reward models on corpus of <AI generreee
Expert-edited> pairs of text with implicit preference judgments (edited > original) (Chakrabarty et al. 20256). Unlike for Al-likelihood scores, logistic regression on the passage-level writing quality reward scores showed significant positive effect of higher reward model scores for both creativity (3 0.26, SE 0.04. OR = 1.30, < 0.001) and (separately) pragmaticality (3 = 0.28, SE = 0.05, OR x 1.33, p < 0.001). This indicates that existing reward models can be predictive of positive and negative aspects of writing. However, these models output a single passage-level score, making them hard to interpret. In the following section, we explore whether LLMS can provide expression-level creativity judgments.
5 CAN LLMS REPLICATE HUMAN CLOSE READING JUDGMENTS OF CREATIVITY?
To understand whether LLMs can be used as expression-level reward models for textual creativity, we eval- uate their ability to find both the positive aspects of writing thuden-judged perceived novel expressions), as well as the negative ones (non-pragmatic expressions). We formulate close reading task similar to the annotation setting using our dataset containing passages P, as well as set of ground-truth perceived novel expressions (or ground-truth non-pragmatic expressions, respectively) W provided by 3 annotators. For prompts and hyperparameters, see Appendix C
CLOSE READING TASK DEFINITION Given a passage, the model needs to extract perceived novel (or non non-pragmatic) expressions N. We consider expression á E N to be an approximate match with expression n E W if one a subset of the other or their normalized indel similarity (Bachmann) 2025) Is 2 90%. We compute the F1 score as follows: an expression ñ is true positive if it has an approximate match with some expression in N, and false positive otherwise. In addition, all that do not have a match in A are considered as false negatives. Note that achieving a perfect Fl score on this task would mean perfectly modeling every annotator's 's (subjective) rating. Hence, our goal is not to reach the maximum possible F1 but rather compare the models and check if there is any useful signal in their judgments.
ALM--SSA-JUDDG (LLM-J) SETUP For reasoning model evaluation, we test 5 frontier models: GPT- 5. Claude 4.1, Claude 4.5, Gemini 2.5 Pro, and Gemini 3 Pro. In the zero-shot setting, we provide a prompt analogous to the instructions we gave to the human annotators. the few-shot setting, we additionally pro- vide examples from 3 passages set aside for few-shot prompting (ss 15 novel expression and non- con-pragmatic expression examples) chosen randomly among passages with a median number of novel or non-pragmatic annotations. For finetuned model evaluation, we finetune a set of smaller open-source models (OLMo2- Instruct 7B, Qwen3 8B (Yang et al., 2025). Llama-3.1 3.1-...11mmm ---nsrruct 8B (Metal 2024)) using LoRA (Hu et al.) 2022). well G and femini-2.5. 2.5-Pro using API-based finetuning on 60% of our dataset (including
7

---

Published as conference paper ICLR 2026
the 3 passages used for few-shot examples), using the rest for evaluation. An empirical random baseline also provided: for each passage, we extracted n expressions of length m words, where and m are sampled from negative binomial distributions, with parameters estimated from human annotations using the method of moments.
RESULTS Figure shows F1 scores with 95% confidence intervals for few-shot and finetumed models. There little difference among models each task, as well as little improvement from few-shot prompting. The non-pragmatic expression identification task appears to be significantly harder, with Fl scores below 20, whereas model performance on the novel expression task was above 40, cautioning against using LLM- as-a-Judge for identification of writing issues (non coon-pragmatic expressions). This aligns with finding earlier that LLMS tend to produce non-pragmatic expressions at high n-gram novelty level. All finetuned models lag behind large reasoning models, suggesting that the close reading task is very difficult even with task-specific adaptation. Overall, reasoning models demonstrate impressive performance compared to the empirical random baseline both tasks. For novelty, random Fl score 9.6 (95% CI [5.3, 14.0), which roughly 4 times lower than reasoning model scores (few-shot GPT-5 Reasoning model Fl zs 41.3). Similarly, for pragmaticality, the random F1 score is 2.3 (95% CI: [0.0, 4.9]). outperformed by reasoning models by roughly 6 times (few-shot GPT-5 Reasoning model F1 = 13.5).
Random
Base
Finee-laned
Few-Shot
Random
Base Rine-Tuned
Fow-Shot
OLMo 278
Owen 36
Owan-389
Uama-1.1
FT
88
Uama-3.1
88
FTT
Deminis
2.5 Pro
Pro
2.5
Gemnl
Gemin
Pro
10
15
20
25
30
35
50
30
F2 Score 196)
Score (%)
(a) Perceived Novel expression identification
(b) Non-pragmatic expression identification
Figure 4: Comparison of finetuned, zero-shot and few-shot model performance across (a) perceived novel and (b) non-pragmatic expression identification tasks (F1 scores with 95% Cls).
close of the best performing models reading tasks, few-shot GPT-5, to explore how LLM-as-a-Judge (LLM--J scores align with expert prefer- ences. Given dataset of preferences (passageA, passageB. 1 |A preferred over B)), we obtain the number of novel (Nov A, Nov B) and non-pragmatic (Prag Prag expressions identified by the LLM-J for each passage. We normalize each score by the number of words Nov
ALIGNMENT WITH EXPERT PREFERENCES We
09 in the passage and take the difference between Pray
them to obtain ANoV = 3 words in logistic regression model fit on the differences between the scores for every pair passages predict- ing expert preference for that pair. We use the Style Mimic dataset containing preferences of three expert writer annotators comparing MFA authors' imitations of famous authors against LLM A.LM-genrrates imitations
# words in F7. APrAgAB = woods in A words in B. A hicrarchical
Nov
Prag
Not the number of atomic expressions, since LLM-J is allowed to choose expressions regardless of punctuation mirror the creative highlight setting in the annotation.
8

---

Published as a conference paper at ICLR 2026
Chakrrbbrry et al., 2025a). Random intercepts were included for each annotator, seed famous author, and for MFA authors nested within seed authors (c.f. Appendix C.6). The model showed that LLM-J novelty score differences were a significant positive predictor of expert preference (BaNOW 0.63, SE= 0.26, OR =1.88, 0.014), while pragmaticality scores were not predictive (РдPrаRлв -0.05, p = 0.832), as expected due to weaker LLM-as-a-Judge performance on this task.
ALIGNMENT WITH CROWD PREFERENCES We explore whether similar association is present in the dataset of crowd-sourced writing preferences released by Chakrabarty et al. (2025a) obtained from LMArena Chiang al. (024).. Each pair contains text generated from 2 language models (model A and model B). Data was sampled such that both model A and model B are in the top 15 most popular models, and so that it contains the same number of comparisons as the Style Mimic data (450). Similarly, we fit a logistic regres- sion model adding random intercepts for the model A and model B. We find that novelty score differences were marginally predictive of the crowd preference (3ANOw AB 0.21, SE= 0.11, OR 1.24, p = 0.054), and pragmaticality scores had significant negative coefficient (ЗaPmEee = -.26 SE= 0.11, OR = 0.77, p = 0.020). Together with Style Mimic findings, this indicates that while LLM-Judge novelty scores align with both expert and crowd preferences, pragmaticality scores only align with crowd preferences. This may indicate a misalignment between expert and crowd preferences stemming from (mis-) dideniffinn non- pragmatic expressions. Figure 5 visualizes the difference of the LLM-J score relationship with expert vs. crowd preferences.
di 0677. c 001
d= 050 0.812
Redddtdd probabitty Human Spiteeme
LLM- Novelty Score (scated)
&.LM- Pregmati: Score (scated)
Prodiceed probublity THOMLE 5 pretturee
1.5
J.LLM- Novelly Scote (scalet)
LLM. Pregmetic Scare (oaleet)
Figure 5: Comparison of how predictive LLM-J novelty (ANov AB) and pragmaticality (APragAB) score differences are of expert vs. crowd preferences, bands indicate 95% confidence intervals.
COMPARISON WITH CREATIVITY INDEX We investigate how well LLM-J novelty scores align with expert preferences compared to CREATIVITY INDEX scores. We compute the CREATIVITY INDEX for each paragraph and use the difference between each pair as predictor for preference (similarly to the ANov AB, APrag AB scores). While CREATIVITY INDEX was predictive of expert preferences (ВaCтл = 0.51, SE= 0.24. OR- 1.66, p 0.038), LLM-J novelty scores had slightly stronger effect (BaNoVaR 0.63, = 0.014). In a model including both predictors, LLM-J novelty scores showed a trend-level effect (BANoy مu = 0.48, p 0.010), while CREATIVITY INDEX did not contribute significantly (BACIA p 0.358). Likelihood ratio tests confirmed that adding CREATIVITY INDEX as a predictor to a model containing novelty scores did not improve model fit (p 0.35), whereas adding novelty scores to model
0.26,
9

---

Published as a conference paper ICLR 2026
containing CREATIVITY INDEX showed trend toward improved fit (p 0.096). Model comparison using AIC (Akaike Information Criterion Akkaike (1774), a measure used for statistical model selection) also favored novelty: novelty-only had the lowest AIC (471.16) compared with the CREATIVITY INDEX- only model (473.07) and the joint model (472.30), indicating that LLM-J novelty scores provide the most parsimonious explanation of expert preference.
6 RELATED WORK
McCoy et al. 2023) investigates LLM copying from training data, finding that GPT-2 tends to copy large chunks of training corpora and has issues with coherence high 7-gram novelty, aligning with some of our findings. Recent advances in training data exploration (Elazar et al. 2024, Merril et al. 2024 Liu
jet al. 2024 Xu et al.] 2025) have enabled textual creativity metrics based n-gram novelty with respect to trillion-token sized corpora (Lu et al. 2025). Our study cautions against using n-gram novelty alone as a metric for creativity based on the standard definition of creativity in psychology requiring the text not only be novel but also pragmatic. In addition, we make a distinction between human-judged and n n-gram novelty.
There have been many efforts in quantifying issues in AI writing. Chakrabarty et al. (2025b) consulted with writing experts to create a taxonomy of idiosyncrasies in AI writing. They further collected span level edits from expert writers following the proposed laxonomy to improve AI writing in an automated pipeline. More recently (Shaib et al. (204)) collected annotations from experts in terms of what qualifies as "slop" in both human and Al text and found that correlates with latent dimensions such as coherence and relevance. In concurrent study Russell et al. (2025) found that people who use Al to write are better detectors of Al writing. Such users typically rely on specific lexical clues ("AI vocabulary"), as well as more complex phenomena within the text such formality, originality and clarity to discriminate Al from human writing.
Recent work has shown that pervasive LLM use may cause negative socictal effects like homogenization (Zhan et al.] 2025 Doshi & Hauser 2024). with RLHF-trained models producing less diverse outputs (Padmakumar & He 2024). Paadakkmarr et al. (2025) proposes a novelty metric balancing originality and quality, and corroborates prior finding that LLM text is less novel than human writing (Chakrabarty al., 2024a). They also find that inference-time methods can boost novelty, though they increase originality at the expense of quality. Unlike prior work, our study investigates the relationship between n-gram novelty and human-judged creativity through expert close reading annotations. We demonstrate that high n-gram novelty often correlates with reduced pragmaticality in LLM outputs, suggesting that optimizing for novelty alone may not lead to genuinely creative text.
7 CONCLUSION
We propose an operationalization of textual creativity beyond n-gram novelty. Rooted in the standard defi- nition of creativity, our definition requires assessing both novelty and appropriateness (sensicality and prag- maticality) of text. We conducted a close reading study of human and Al-generated text collecting an- notations from professional writers (n = 26), obtaining dataset of 8618 annotated expressions and 274 creative expression highlights with justifications. Our analysis reveals that as open-source LLMs generate more novel text, they tend to generate less pragmatic expressions, and that R 91% of top-quartile n-gram novel expressions are not judged as creative, cautioning against the use of n-gram novelty metric alone for creativity evaluation. We confirm similar trends in an exploratory study with frontier models, as well as identify their lower propensity to produce creative expressions compared to humans. We show that reason- ing models can replicate some of the human judgments on creativity, aligning with expert preferences in an nut-of-distributie dataset, more so than n-gram based CREATIVITY INDEX.
10

---

Published as conference paper ICLR 2026
REFERENCES
H. Akaike. A new look at the statistical model identification. IEEE Transactions Automatic Control, 19 (0)7716-223, 1974. doi: 10.109/TAA.19744100075.
Teresa M Amabile. The social psychology of creativity: componential conceptualization. Journal of personality and social psychology, 45(2):357, 1983.
R Harald Baayen, Douglas J Davidson, and Douglas M Bates. Mixed-effects modeling with crossed random effects for subjects and items. Journal of memory and language, 59(4))390-412, 2008
Max Bachmann. rapidfuzz: Levenshtein.ratio levenshtein similarity ratio. battps:/rapidfuuzz
github. hwwp:///....eeessstaeeeeeeeeeese
html fratio, 2025. Accessed: 2025-09-07.
Douglas Bates, Martin Mächler, Ben Bolker, and Steve Walker. Fitting linear mixed-effects models using 1me4. Journal of Statistical Software, 67(1):1-48, 2015. doi: 10. 10.186337/ss.ww067.11.
Paul Brennan and Alan Silman. Statistical methods for assessing observer variability in clinical measures. BMJ: British Medical Journal, (04(6840):1491, 1992.
Barry Brummett. Techniques of close reading. Sage Publications, 2018.
Tuhin Chakrabarty, Philippe Laban, Divyansh Agarwal, Smaranda Muresan, and Chien-Sheng Wu. Art or artifice? large language models and the false promise of creativity. In Proceedings of the 2024 CHI Conference on Human Factors Computing Systems, CHI '24, New York, NY, USA, 2024a. Association
for Computing Machinery. ISBN 9798400703300. doi: 10. 10.1145726139044644773.
URL https:
//doi.org/// 10. 1145/36111
3642731
Tuhin Chakrabarty, Philippe Laban, Divyansh Agarwal, Smaranda Muresan, and Chien-Sheng Wu. Art artifice? large language models and the false promise of creativity. In Proceedings the 2024 CHI Conference on Human Factors in Computing Systems, pp. 1-34, 2024b.
Tuhin Chakrabarty, Philippe Laban, and Chien-Sheng Wu. Ai-slop to ai-polish? aligning language models through edit-based writing rewards and test-time computation. arXiv preprint ww.Xiv:2504.05::3 2025a.
Tuhin Chakrabarty, Philippe Laban, and Chien-Sheng Wu. Can ai writing be salvaged? mitigating idiosyn- crasies and improving human-ai alignment in the writing process through edits. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems, pp. 1-33, 2025b.
Aaron Chatterji, Thomas Cunningham, David J Deming, Zoe Hitzig, Christopher Ong, Carl Yan Shan, and Kevin Wadman. How people use chatgpt. Technical report, National Bureau of Economic Research, 2025.
Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E. Gonzalez, and lon Stoica. Chatbot arena: An open platform for evaluating iims by human preference, 2024. URL https: / arxiv. org/ abs/2403
04132
Mihaly Csikszentmihalyi al. Flow and the psychology of discovery and invention. HarperPerennial,, New York, 39:1-16, 1997.
Anil R Doshi and Oliver P Hauser. Generative ai enhances individual creativity but reduces the collective diversity of novel content. Science Advances, 10(28):eadn5290, 2024.
Charles W Dunnett. A multiple comparison procedure for comparing several treatments with a control. Journal the American Statistical Association, 506272) 1955.

---

Published as a conference paper ICLR 2026
Yanai Elazar, Akshita Bhagia, Ian Magnusson. Abhilasha Ravichander, Dustin Schwenk, Alane Suhr, Evan Pete Walsh, Dirk Groeneveld, Luca Soldaini, Sameer Singh, Hannaneh Hajishirzi, Noah A. Smith, and Jesse Dodge. What's my big data? In The Twelfth International Conference on Learning Rep- resentations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https: 7 //openreview. net wwwpeeeeeeeeeeeeeeeeeeeeeeeeeeee dow.ommidd-rrPPooO..
Bradley Emi and Max Spero. Technical report the pangram ai-generated text classifier, 2024. URL
https:/ / a aaa/arxiii.../ wttp:///ggaa/a//////o//a 14873
Barbara Di Eugenio and Michael Glass. The kappa statistic: A second look. Computational Linguisties, httphttps://at
30 3091) 03 2004, ISSN 0891-2017. doi: 10.166/8992201047733344 1tte://0.1112/011211111111111
Alvan R. Feinstein and Domenic V. Cicchetti. High agreement but low kappa: I. the problems of two
paradoxes. Journal of Clinical Epidemiology, 43(6):543-549, 1990. ISSN 0895-4356. doi: https://
0hlagg/00.001100055-566666000000 URLhttps www.:./ceeeeeireece.......oo htw.::/ccddrret....coco/c....e articlee/pt/0854455003111
John Fox and Sanford Weisherg. An Companion to Applied Regression. Sage, Thousand Oaks CA, third edition, 2019. URL hts /www. john-fox ca. ca/Companicn)
Andrew Gelman and Jennifer Hill. Data analysis using regression and multileve//hierrrcchici models. Cam- bridge university press, 2007.
Jacob Gildenblat. A python library for confidence intervals. confidenceintervaa 2023.
Dirk Groeneveld, Iz Beltagy, Evan Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khyathi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, Tushar Khot, William Merrill, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew Peters, Valentina Pyatkin, Abhilasha Ravichander, Dustin Schwenk, Saurabh Shah, William Smith, Emma Strubell, Nishant Subramani, Mitchell Wortsman, Pradeep Dasigi, Nathan Lambert, Kyle Richardson, Luke Zettlemoyer, Jesse Dodge, Kyle Lo, Luca Soldaini, Noah Smith, and Hannaneh Hajishirzi. OLMo: Accelerating the science of language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 15789-15809, Bangkok, Thailand, August 2024, Association for Computational Linguistics. doi: 10.18653/ 841.
acl acl-long. 841/
Kunal Handa, Alex Tamkin, Miles McCain, Saffron Huang, Esin Durmus, Sarah Heck, Jared Mueller, Jerry Hong, Stuart Ritchie, Tim Belonax, et al. Which economic tasks are performed with ai? evidence from millions of claude conversations. arXiv preprint arXiv:2503.0476) 2025.
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL litpttt::/// httphttt///peeeetee //tp:://///ppeevieww net mnee/forumt "eoormmiidnneeiiee nde
Philip W Jackson and Samuel Messick. The person, the product, and the response: conceptual problems in the assessment of creativity. Journal of personality, 1965.
James C Kaufman, John Baer, Jason C Cole, and Janel D Sexton. A comparison of expert and nonexpert
raters using the consensual assessment technique. Creativity Research Journal, 20 2008.
12

---

Published as a conference paper ICLR 2026
James C. Kaufman, John Baer, and Jason C Cole. Expertise, domains, and the consensual assessment tech- nique. The Journal of creative behavior, 43(4):223-2333 2009.
Talia Kaufmann, Gabriel Agostini, Trivik Verma, and Daniel T. O'Brien. Inequalities in mobility patterns: Reconciling access and travel in american cities. In Susannah Greeba,,,,, Adam Dennett, and Chen Zhong (eds.), The 19th International Conference on Computational Urban Planning and Urban Munagement, London, UK, June 2025. ACM Press. doi: doi.org/10. 100011717700000
Dmitry Kohak, Rita González-Márquez, Emōke-Agnes Horvát, and Jan Lause. Delving into Ilm-assisted writing in biomedical publications through excess vocabulary. Science Advances, 11(27)xedd38833 2025.
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang. Hamish Ivison, Faeze Brahman, Lester James V. Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luca Soldaini, Noah A. Smith, Yizhong Wang. Pradeep Dasigi, and Hannaneh Hajishirzi. Tülu 3: Pushing frontiers in open language model post-trainne traininiinnn. NA, 2024.
Andrew Lampinen, Ishita Dasgupta, Stephanic Chan, Kory Mathewson, Mh Tessler, Antonia Creswell, James McClelland, Jane Wang, and Felix Hill. Can language models learn from explanations in con- text? In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 537-563, Abu Dhabi, United Arab Emirates, Decem- ber 2022. Association for Computational Linguistics. doi: findings emn 1p.1pp...... 387
URL
https: / http://cc..aatcelo...orro///////
Jiacheng Liu, Sewon Min, Luke Zettlemoyer, Yejin Choi, and Hannaneh Hajishirzi. Infini-gram: Scaling unbounded n-gram language models to a trillion tokens. In First Conference on Language Modeling, 2024.
Ximing Lu, Melanie Sclar, Skyler Hallinan, Niloofar Mireshghallah, Jiacheng Liu, Seungju Han, Allyson Ettinger, Liwei Jiang, Khyathi Raghavi Chandu, Nouha Dziri, and Yejin Choi. Al as humanity's salieri: Quantifying linguistic creativity of language models via systematic attribution of machine text against web text. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singa- pore, April 24-28, 2025. OpenReview.net, 2025. URL https::/ httts:///peeneevee..r net htte//ormmmiiom 110E0Iqp10
R. Thomas McCoy, Paul Smolensky, Tal Linzen, Jianfeng Gao, and Asli Celikyilmaz. How much do lan- guage models copy from their training data? evaluating linguistic novelty in text generation using raven. Transactions of the Association for Computational Linguistics, 11:652-670, 06 2023. ISSN 2307-387X. doi: 10.1162/tacl a 00567. Utt.ttts::///o1. wtw.iooorg/1... /22 /tcc/taacl..mm 00567
William Merrill, Noah A. Smith, and Yanai Elazar. Evaluating n-gram novelty of language models using rusGy-DAWG. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 14459-14473, Miami, Florida, USA, November 2024, Association for Computational Linguistics. doi: 10. 100116665.11/22222mmmmmmmaaaa 800. URLhtttps. / acl antho org /20gg/2224..22 wmplp-mmmllppmmmmmmm
Meta. The llama 3 herd of models, 2024. URL http:p::/aaa.....ooo/aas////////
Vishakh Padmakumar and He He. Does writing with language models reduce content diversity? In The
Twelfth International Conference on Learning Representations, 2024. URL httphhtttts://ppeeeee/
het/Eorum?a wtw.mmmmdddeeiiisteeeem
13

---

Published as a conference paper ICLR 2026
Vishakh Padmakumar, Chen Yuch-Han, Jane Pan, Valerie Chen, and He He. Beyond memorization: Map- ping the originality-qualin frontier of language models, 2025. URL https://azxiv.org/
2504. 09389
Justus J Randolph. Free-marginal multirater kappa (multirater k [free]): An alternative to fleiss' fixed- marginal multirater kappa. Online submission, 2005.
Mark A Runco and Garrett J Jaeger. The standard definition of creativity. Creativity research journal, 24 (1):92-96, 2012.
Jenna Russell, Marzena Karpinska, and Mohit Iyyer. People who frequently use ChatGPT for writ- ing tasks are accurate and robust detectors of Al-generated text. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.). Proceedings of the 63rd Annual Meet- ing of the Association for Computational Linguistics (Volume I: Long Papers), pp. 5342-5373, Vi- enna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-89176-251-0, doi: wwtge/or.///2. acl long.
10.18653/v1/2023
accls///oo........o.gg..... acl-long..67. URL htLLttts://ccaatthaat
Robert Keith Sawyer and Danah Henriksen. Explaining creativity: The science of human innovation. Oxford university press, 2024.
Chantal Shaib, Yanai Elazar, Junyi Jessy Li, and Byron C Wallace. Detection and measurement of syntactic templates in generated text. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceed- ings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 6416-6431, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/v 2024.emnlp-main.36.
httttt/t:://att htt.ttppp//////aaaooooooo.../// orl.orrg/////// 2024. emn 1p-1lpp-ma-- ma n. 368/
Chantal Shaib, Tuhin Chakrabarty, Diego Garcia-Olano, and Byron C Wallace. Measuring ai" slop" in text.
arXiv preprint arXiv:2509.19163 2025.
Seabold Skipper and Perktold Josef. statsmodels: Econometric and statistical modeling with python. 9th Python in Science Conference, 2010.
Barbara Herrsstein Smith. What was "close reading"? " a century of method in literary studies. the minnesota review, 2016(87):57-77,, 2016.
Kanae Takahashi, Kouji Yamamoto, Aya Kuchiba, and Tatsuki Koyama. Confidence interval for micro- averaged 1 and macro-averaged f scores. Applied Intelligence, 52(5):4961-499 2022.
Team OLMo. Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, Nathan Lambert, Dustin Schwenk, Oyvind Tafjord, Taira Anderson, David Atkinson, Faeze Brahman, Christopher Clark, Pradeep Dasigi, Nouha Dziri, Michal Guerquin, Hamish Ivison, Pang Wei Koh, Jiacheng Liu, Saumya Malik, William Merrill, Lester James V. Mi- randa, Jacob Morrison, Tyler Murray, Crystal Nam, Valentina Pyatkin, Aman Rangapur, Michael Schmitz, Sam Skjonsberg, David Wadden, Christopher Wilhelm, Michael Wilson, Luke Zettlemoyer, Ali Farhadi, Noah A. Smith, and Hannaneh Hajishirzi. 2 olmo furious, 2025. http:t:s::
org/ abs/
2501.00555
Hao Xu, Jiacheng Liu, Yejin Choi, Noah A. Smith, and Hanna Hajishirzi. Infini mini: Exact n- gram search at the internet scale with fm-index. ArXiv, abs/2506. 12229, 2025. URL htttts:/p//app
semanticscholas:
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang.
14

---

Published as a conference paper ICLR 2026
Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025. URL https:
7/arxiv. www..orgaabao abs/ 2505. 09388
Tal Yarkoni. The generalizability crisis. Behavioral and Brain Sciences, 45:e1, 2022.
Yiming Zhang, Harshita Diddee, Susan Holm, Hanchen Liu, Xinyue Liu, Vinay Samuel, Barry Wang. and Daphne Ippolito. Noveltybench: Evaluating language models for humanlike diversity. arXiv preprint arXiv:2504.05222 2025.
A
DATA
A.1
SAMPLING EXPRESSIONS FOR ANNOTATION
To select pre- highllgihghhed expressions, a passage was divided into atomic expressions via splitting by punc- tuation and heuristic rules. Then, we computed % of novel n-grams for each expression as follows:
F: the expression
(E)): all n-grams length in T
D: reference set of n-grams
1(+): indicator function
n" min n 2 1: ::39 E Gn(E) with g $ D
NovelPet(11)) = Epeg.. (F) 1g $ D)
99n- (F)
That is, we compute the percentage of novel n-grams for n s.t. there at least one novel n-gram. Among the various metrics tested, this was the most interpretable and had the best balance between correlation with expression length (as we do not want to select only long expressions) and perplexity. We set a threshold for novelty very liberally 15% which allowed to get large number of expressions annotated (roughly 50%).
A.2 SELECTING PASSAGES FOR ANNOTATION
We ensured that passages were not present in the respective OLMo model training corpora by manually checking that large n-grams from the passage as well as seemingly rare n-grams (e.g., unique proper nouns) had zero counts in the pre- trainini corpora. In addition, we had an automatic verification mechanism where a sample of 5 15-grams from the beginning, 5 from middle, and from the end of the text were checked to have a O count in the pretraining data.
To sample the 5 passages for the follow-up exploratory study with frontier models, split the 50 passages from the main study into novelty quintiles according to estimates from the model novel ~ 1 + (1/para_id) + (1 |annotator). We randomly pick one passage per quintile.
15

---

Published as conference paper ICLR 2026
A.3 ANNOTATION INSTRUCTIONS
We provide the annotation instructions in Figure 6
A.4 DATASET EXAMPLES
Table 3 shows additional examples of low n n-gram novelty creative expressions.
A.5 ANNOTATOR AGREEMENT COMPUTATION
We use stat.mmoeel package (Skipper & Josef 2010) to compute the free-marginal Kappa.
Expression
Justification
PPL
disappearing from one room and reappearing in another
As in the previous clause, this metaphor defamiliarizes mundane experience comparing it to the magical, heightening the surreality/rrama of the narrator watching their daughter.
-1.6
you'le 're the ugliest ever seen
I've the days of the week whisk by like ponties
This sentence marks surprising description a date up to this point.
from what has heen innocuous
-1.5
Inventive simile; surprising and humorous, central to the thematic conceit of the passage. Strong voice.
-1.4
lay across the roof of the house like the severed hand of a giant
Describing felled pine bough similar to "the severed hand of a giant" -1.4 delightfully fresh and original.
Table 3: Examples of expressions judged creative with low n-gram novelty (log-standardized perplexity), along with the expert writers' justifications.
LINEAR MODELS
All linear models herein were estimated with the 1me4 package (Bates al. 2015).
B.I DATA PRE-PROCESSING
Due instability the infinigram API, we intially were not able to detect two passages as part of the OLMo pretraining data (IDs 14520 and 7644). We therefore removed those passages at the pre-pre peeppocesssing stage for the linear models, which did not affect outcomes of the research questions. addition, we filter out 37 unique expressions that received varying perplexity estimates from the infinigram API due to being computed at different times.
B.2 N-GRAM NOVELTY AND CREATIVITY
We incorporate creativity highlights in the following manner: we add all highlighted expressions and their perplexity scores to the pre-highlighted ratings data, exclude expressions that were subsets, supersets over 90% similar by normalized indel similarity |acchmann| (2025). Given that highlights allowed the an- notators to select any expression, pre-highlighted set) as non-novel by any annotator that did not highlight it. As a result, our dataset contains enlarged number of 15, 982 annotations.
consider expressions that were not highlighted (and not in the
16

---

Published as conference paper ICLR 2026
creative ppl__og_ssd + gen_source + (1 annotator)
(1 seed passage id)
(Intercept)
3.29
3-664; -2.4
ppl.logssd
0.67
[0.59; 0.76]*
gen sourceolmol
0.63
-0.87; -0338
gen sourscecclmm
0.61
0.85; 0.37
R2 (theoretical)
0.12
R (theoretical)
0.30
AIC
5066.55
BIC
51 5112.53
Log Likelihood
2527. .88
Num. obs.
15725
Num. groups: seed passage id
50
Num. groups: annotator
26
Var: seed_passage id (Intercept)
0.36
Var: annotator (Intercept)
0.53
outside the confidence interval.
Table 4: Mixed-effects logistic regression predicting creativity from standardized PPL
B.3 N-GRAM NOVELTY AND PRAGMATICALITY
To focus on pragmaticality, we exclude expressions rated as non-sensical (so we know they do not make sense in the context, not just because they are ill-formed). The full specification of the model is available in Table
For frontier model study, we did exclude these expressions since the sample size of non-pragmatic expressions was already very small (12 and 43 for Claude and GPT-5). In addition, since our research question concerns head-to-head comparison of humans and frontier LLMs, we exclude passages generated by OLMo and OLMo-2. The model is available in Table
We tested the following null hypotheses regarding the effect of perplexity (ppl.log.std) on pragmatic judgments within each generator:
Human (reference group): LLM:
Ho 3ppl. sid 0
Hu 3ppl log std + 9ppl log stdigen sourceLLM 0
The tests were carried out with the function 1 LinearHypothesis () from the car package Fox & Weis- berg (2019). applied the fitted glmer model.
B.4 CREATIVITY AND GENERATION SOURCE
To compare creativity along all generation sources, add the frontier model annotations to the original data, adding the highlights in the same manner as above. The model is available in Table
B.5 WRITING QUALITY REWARD MODELS AND CREATIVITY
The model for creativity is available in Table 8 and for pragmaticality in Table 9
17

---

Published as conference paper ICLR 2026
pragmatic ppl_log_ste std gen_source
(1 annotator) +
seed passage id)
(Intercept)
3.93 4.51;
ppl log std gen sourceolmol
0.01 -0,16; 0.18
1.50 -1.11;
gen_sourceolmo2
0.43 00.07:0.78
h___lg__sdgeen.sourreelloo
-0.18 -0.42
gp..og.sdd.een.ssurrooommm
0.49 -0.18)
R2 (thcoretical)
0.10
R (theoretical)
0.47
AIC
2752.73
BIC
2808.07
Log Likelihood
1368.37
Num. obs.
7455
Num. groups: seed_passage.it id
50
Num. groups: annotator
26
Var: seed passagaa_seeeeee id (Intercept)
0.31
Var: annotator (Intercept)
2.00
outside the confidence interval.
Table 5: Mixed-effects logistic regression predicting pragmatism from standardized PPL. and gen, source
pragmatic " ppl_log_std gen_source
(1 annotator) + seed sasd_passage_id)
(Intercept)
2.70 3.6773.73
ppl_log_std
0.13 - 0.24:0044
gen_source
1.47
3.04
gen_sourcegpt5
0.04
1.46; 1.53
opl.log.stdgge
wtt.:stdedssuureeeeeeeeeeeeeeeee sourceclaude
00.35
1.02, 0.32
ppl.log_std:ger wtt.sstgen.suureegeteee
0.30
0.80; 0.20
R2 (theoretical)
0.10
R (theoretical)
0.41
AIC
613.98
BIC
656.38
Log Likelihood
298.99
Num. obs.
1481
Num. groups: annotator
12
Num. groups: seed -aesagee id
5
Var: annotator (Intercept)
1.62
Var: seed passage id (Intercept)
0.09
outside the confidence interval.
Table Mixed-effects logistic regression predicting creativity from standardized PPL (comparing humans and frontier models)
18

---

Published as conference paper ICLR 2026
creative gen. source
annotator) +
(1 seed passage id)
(Intercept)
-3.08
-2.74"
gen_sourceclaude
-0.65
1112; -0.18
sourceers
-0.67
1.16;
-0.19
gen sen_sourceolmo
00.69
0.93: -0.45
gen sourceolcmm2
0.53
0.76; -0.30
R (theoretical)
0.02
R (theoretical)
0.22
AIC
5898.85
BIC
5953.43 443
Log Likelihood
29422.42
Num. obs.
18004
Num. groups: seed passage id
50
Num. groups: annotator
26
Var: seed_passage_ id (Intercept)
0.33
Var: annotator (Intercept)
0.51
0 outside the confidence interval.
Table Mixed-effects logistic regression predicting creativity from generation source (comparing humans and all models)
creative WQRM score
annotator)
(1 seed_passage.ing
(Intercept)
5..36 -6.01; -4.71]
WQRM score
0.26 [0.19; 0.33)
R2
(theoretical)
0.02
R (theoretical)
0.22
AIC
5339.09
BIC
5369.74 74
Log Likelihood
2665.54
Num. obs.
15725
Num. groups: seed passage id
50
Num. groups: annotator
26
Var: seed_passage_in id (Intercept)
0.34
Var: annotator (Intercept)
0.49
outside the contidence interval.
Table 8: Mixed-effecte logistic regression predicting creativity from passage-level WQRM scores
19

---

Published as conference paper ICLR 2026
pragmatic WWRM score
+ (1 annotator) +
(1 seed_ passage_id
(Intercept)
1.48 [0.56; 2.41]*
WQRM_scoM
0.28
00.19; 0.38
R (theoretical)
0.02
R (theoretical)
0.45
AIC
2.16.08
BIC
2843.75
Log Likelihood
1404.04
Num. obs.
7455
Num. groups: seed-passage id
50
Num. groups: annotator
26
Var: seed passage id (Intercept)
0.74
Var: annotator (Intercept)
1.85
outside the confidence interval.
Table 9: Mixed-effects logistic regression predicting pragmaticality from passage-level WQRM scores
B.6 AI-LIKELIHOOD SCORES
The model for creativity is available in Table and for pragmaticality in Table [T]
creative std _ai likelihood
+ (1 annotator) +
seed passage id)
(Intercept)
3.08 3.43; -2.74]
std std_log_ai likellioo
0.05 0.116; 0.27]
R2 (theoretical)
0.00
R (theoretical)
0.20
AIC
3127.20 29
BIC
3154.94
Log Likelihood
- 1559..4
Num. obs.
7430
Num. groups: seed_passage
passage id
49
Num. groups: annotator
26
Var: seed peesaggss id (Intercept)
0.31
Var: annotator (Intercept)
0.52
outside the confidence interval.
Table 10: Mixed-effects logistic regression predicting creativity from passage-level Al likelihood scores
20

---

Published as conference paper ICLR 2026
pragmatic std_log_at likelihood
+ (1 annotator)
(1 leed_passae__lee _id
(Intercept)
3.85 33.27:4.43
std std_log.ai. ai Jikelihood
0.25 0..53; 0.04]
R (theoretical)
0.01
R (theoretical)
0.37
AIC
1042.47
BIC
1067.07
Log Likelihood
517.24
Num. obs.
3466
Num. groups: seed_passage- id
49
Num. groups: annotator
26
Var: seed passage id (Intercept)
0.37
Var: annotator (Intercept)
1.48
outside the confidence interval.
Table 11: Mixed-effects logistic regression predicting pragmaticality from passage-level WQRM scores
LLM-JUDGE EVALUATION DETAILS, PROMPTS AND HYPERPARAMETERS
C.1
CONFIDENCE INTERVAL COMPUTATION
All 95% Cls computed with with conf serffiidnneenteege package Giilennllat 2023) using the Takahashi
al. 2022 method.
C.2
OLMO VERSIONS, PROMPTS AND GENERATION LYYPERPARAMTERS
To generate LLM passages for annotation, use the following simple prompts for OLMo (OLMo-7B-072--nattree and OLMo-2 (0LMo-2-0325-32--nne in Table T2
Row
Prompt
Summary Prompt
(atory)\n/nSSummall ze in sentence.
Passage Generation Prompt
(ommaryy]n)/Wrilnn a fragment of a story around {n words) words long in the style of [author] based on the summary above.
Table 12: Prompts for summary and passage generation.
We sample with temperature and truncate output by paragraphs if they were too long, regenerat- ing if they were too short (not within 10% of the original passage). For summaries, manually ensured the overall topic was consistent with the passage (in 2-3 rare cases, the summaries were completely off-topic). For stories,
manually examined the outputs and regenerated the story in case of output that did not all adhere to the prompt (for example, sometimes OLMo model would generate a play script instead of a passage, and once the model generated story fully in Spanish).
21

---

Published as conference paper ICLR 2026
We the same generation prompt for Claude-4.1 and GPT-5 in the exploratory frontier model study, and use default API inference hyperparameters. We used the same summaries as for the OLMo models to exclude summary content effects.
C.3 LLM-AS-A-JUDGE VERSIONS AND GENERATION HYPERPARAMTERS
GPT-5 Hyperparameters:
Model version: gpt-5-2025-08
verbosity: high
reasoning effort: high
Claude-4.1 Hyperparameters:
Model version: claude-opus-44
temperature
Gemini 2.5 Pro Hyperparameters: Default hyperparameters
C.4 PROMPTS FOR MODEL EVALUATION
We based our prompts very closely on annotator instructions (see Figure used for zero-shot and few-shot model evaluation.
6). Figure 7 shows the prompts
C.5 FINETUNING HYPERPARAMETERS
Below are the hyperparameters used for model fine-tuning:
GPT-4.1
Model version
Fine-tuning type: full parameter (API-based)
Batch size: 2
Learning rate multiplier:
Epochs:
Seed: 42
OLMo-2-7B
Model version: 01M6-2-1124-7R-Imne
Fine-tuning type: LoRA
LoRA rank: 8
*
LoRA alpha:
*
lora dropout 0.2
per device batch size:
gradient accumulation steps: 2
learning rate 2c-4
scheduler type linear
warmup ratio 0.03
weight decay 0.01
epochs 3
22

---

Published as conference paper ICLR 2026
max seq length 4096
Qwen3-8B
Model version: Qwen3-88
Fine-tuning parameters: same above
LLama-3.1
-Model version: Llama 3. 3.1-...... 8----------- Instruct
Fine-tuning parameters: same as above
All open model fine-tuning for fine-tuning script Lambert et al. (2024).
2 A.0 40GB NVIDIA GPUs. We used the Instruutt codebase
C.6 LINEAR MODELS FOR EXPERT AND CROWD PREFERENCE ALIGNMENT
The linear model fit esimtate alignment between LLM-J scores and expert references in presented in Table [3 For crowd preferences, see Table 14 AIC comparison and model specifications for Creativity Index comparisons can be found in Table 15
23

---

Published as conference paper ICLR 2026
Annotation Instructions
you clottig study!
passage creativity annotation, Your gool identity expressions contribute
tee's creative
highlight Ratiings
Rate whather the aiready ligh kghted expross oma adhere to the lotlowing three crieria of creativity:
VERY POORTANTT Foreach expression, eva unto: does it makes sonse? does it Tlow natura typ it novelt Provide uxlpements on three of these cotogories, not just one them. when you turn the toggle on, e the Making Sense quest means you annotate the expressiion as Makinn Semse (anowertng
Makes Sense On ths Own
Whither rxpressioon makes sense itsel term grammer evanal tuenny. it the exprossion uid make some contest, mark making sense by Isell. if there is no content in which the expression makes samoe, mark as not makk sensa by tsal, and also as not making serse in connect.
exampple,
towerds top sence, our "comermation towards
iighht
Simi "hrom
maried
Streett something
completely ungrammatical nonsens ica, no matter what contect put in,, shou id marked not.
Naturally)
exprossion makes
tox, logical condrnitt issues, sound nooheernn rest
the sect.
For example,, it the previou text describes that Wilc''s Aile decame sad and glvl and the empression following & "And sp Alice fburisied", describing happy
logicaly
previous passage.
Note: Generaly, the exprecsion does not make sence on its own, itatoo sbes not make sense conteat.
Novel
Whether or the expression is unusual, surpriaing, or
the connect of the noxt.
technique could contribute contertual Forexsmple, pesser pessurlication licationn tex the tables heve beoorn tenvie sisce ahe wse kad unuld be naddendd aurpe sing and unusuel.
powertul metapher, interesting detail character
important: You wil be asked prowide justiiiaaion for why you find the exprossion novel it you check this box.
Important: The exprossion
has novel, ununual surprinng
not
(espression pre-highlighttd could
tamaly Noventber
heresitery diseace siread cisimed people
Drog
Justification: Austaposiion
"und" this
think cyclica
the
rest of the passags. The suthor hints that while some events the fall of dynasty here, lnter sease a heart stracko may deen as the dealine final stage somethingg they are neoo begarnange.
Part Creativity Highlights
Using your curser, select any expressions that ware not high ightod, but which you findd creative sccording the defiiition below: You can creative expressione the came you completing Par1
Selection Criteria
expressione,
simutameeuu eeoutty
serse
Be nove
Important: Do select "cad" eugr repressional Focus identirying creative
exprossione
THREE
percehved
may select expressioon concurentty with
getted expressione
can vlso light partx the prav pra-tight lepresssione.
a highlligteed expression:
stwnding in other coomens weetching other paopple move fornitue around indkor apwowss
The para le and ant pora structure of the semtence letherlocior sim larisini lar' works to characterize the stuttion van finds himself apwece threrowl this the ides that furiitur nocur along with the coneeptusllaa liaatio,, univer salloo process that the character app ying to the seene.
Jusotfication:
peopie
Iou'' bwadi-heeaa underslanc
Figure 6: Annotation Instructions
24

---

Published as a conference paper ICLR 2026
Below will find passage for creativity annotation. Your goal to identify expressions that contribute to this text's uniqueness and creative value through close reading of the passage.
Find expressions that are novel the passage. novel expression unusual, surprising, original given the context of the text. powerful metaphor, interesting detail about character, or insightful literary technique could all contribute to contextual novelty. For example, a personification "as if the tables have become heavier since she walked in' could be considered surprising and unusual. Find as many expressions as you can but be sure to select only those that are truly novel.
Important: You will be asked provide justification for why you find the expression novel.
Novel
Important:
The expression select has to be novel, unusual, surprising in some way. not simply "strong". -
Expression
Judge Prompt
Example:
sixth November, 2015, my had sudden heart attack, the result of hereditary disease that had already claimed five six people in my family. the first of them at the end of the Qing dynasty.
Passage: Then,
Output:
(" I"expression" "the first them at the end of the Qing dynasty". "justification": "the juxtaposition "first" and "end" in this fragment makes reader think about the cyclical nature of time, which passage. The author hints that while some events (the fall of dynasty here, later a disease or heart attack) may be seen as the decline or final stage of something, they are also new beginnings."" Passage: (passage)
big theme the rest of the
Below will find passage for annotation. Your goal to identify expressions that may not make sense or flow naturally through close reading of the passage. Find all expressions that do not make sense contextually or flow naturally. That is, find all expressions that do not make sense the context of the text, have logical or continuity issues, or sound awkward, odd, incoherent with rest the
Find as many expressions you can be to select only those that truly not making sense context or flowing naturally.
Non-pragmatic Expression Judge Prompt
For example, the previous text describes that 'Alice's life became sad and dull" and the expression following it is And so Alice flourished", describing how happy she is, could be considered incoherent since logically contradicts the previous passage.
Another example: 'her fingers tended his fire' may sound awkward and unnatural.
Output the expressions and your justification in the following json format: [[ ("exxpressoon": "your answer", "justification": "your answer"!]
Passage: (passage)
Figure 7: Prompts used for few-shot and zero-shot model evaluation.
25

---

Published as conference paper ICLR 2026
preference _bin scaled_delta_nov. dellta_nov_sscore scaled_delta.ppe pea.tttarpasooooo
(1 annotator) +
seed_author / mmfa.author
(Intercept)
1.67 0.977 2.38)*
ccaled_delta_nee
0.63
0.133 1.13]*
scaled_delta.prag..oeo
-0.05
[-0.52; 0.41]
Ra (theoretical)
0.04
R (theoretical)
0.62
AIC
473.11
BIC
497.77
Log Likelihood
230.56
Num. obs.
450
Num. groups: mffa_authorseeddauthrr
150
Num. groups: sced-author
50
Num. groups: annotator
21
Var: mía author:seed author (Intercept)
3,49
Var: seed author (Intercept)
0.92
Var: annotator (Intercept)
0.51
O outside the confidence interval.
Table 13: Mixed-effects logistic regression predicting expert style preference from novelty and pragmaticity score differences
26

---

Published as a conference paper ICLR 2026
preference_bie _bin scaled_delta deltta_ovv_scorr
scaled www.eddee--aaassoooo
model a) +
(1 model.b)
(Intercept)
-0.07
[0.477
0.22
-0.00;
gagledddeltapprgg.eee
0.26
[- 0.49; -0.49;00.04)
R² (theoretical)
0.02
R (theoretical)
0.14
AIC
609.06
BIC
629.61
Log Likelihood
299.53
Num. obs.
450
Num. groups: model_a
15
Num. groups: model_b
15
Var: model_a (Intercept)
0.10
Var: model b (Intercept)
0.36
outside the confidence interval.
Table 14: Mixed-effects logistic regression predicting crowd preference from novelty and pragmaticality score differences
27

---

Published as conference paper ICLR 2026
(1) Nov. only
(2) CR-index only
(3) Nov. + CR-index
(Intercept)
1.67 (0.97; 2.38)
1.71 [1.00; 2.43
1.69 (0.98:2.40)
dealedddett..ov..oo score
0.62 (0.13; 1.11]
0.48 -0.09; 1.05]
detoodddela..ciddeee score
0.51 [0.03;0.99]*
0.26 -0.29; 0.80]
R (theoretical)
0.04
0.03
0.05
Rº (theoretical)
0.62
0.62
0.62
AIC
471.16
473.07
472.30
BIC
491.70
493.61
496.95
Log Likelihond
230.58
-21.533
230.15
Num. obs.
450
450
450
Num. groups: mfa authorsseed author
150
Num. groups: sethhr
50
50
50
Num. groups: annotator
21
21
21
Var: mfa_ aeff_aathoorscedd..k author (Intercept)
3.51
3.82
3.53
Var: seed authhr (Intercept)
0.90
0.80
0.89
Var: annotator (Intercept)
0.51
0.52
0.53
Nall hypothosis value ousiide the confidence interval.
Table 15: Mixed-effects logistic regression predicting expert style preference: novelty only, CR-index only, and combined. All models include random intercepts for annotator and seed author/mfa author.
28

---

Published as conference paper ICLR 2026
D ADDITIONAL FIGURES
1.00
Prob. of being rated creative
0.75
0.50
0.25
0.00
Standardized log Perplexity [op__lo__std)
Figure 8: Effect of n-gram novelty (log-standardized perplexity) probability of the expression being rated as novel.
Histogram of licelihood for human-generated expressions
Count
Log-standaraan sized likathood
Figure 9: passage-level log Al-likelihood distribution for human-written expressions.
29

---

Published as conference paper ICLR 2026
Writing Quality (WORM Score)
almoz
Generation Source
Figure 10: Writing quality reward model distribution by generation source.
Geman- 255P
Gemtri- 2.5-Ph
Caude 4.1
Cluuee411
GP15
C275
25
(a) Novel Expression Identification: few-shot zero-shot results.
(b) Non-Pragmatic Expression Identification: few- shot and zero-shot results.
Figure 11: Comparison of zero-shot and few-shot model performance across (a) novel and (b) non-pragmatic expression identification tasks on the full test set of 97% of the data excluding 3 few-shot passages only (Fl scores with 95% CIs computed with confidence int erval package (Gildenblat (023)) using the Takahashi et al. (2022) method.
30


Reviews:
[{"review_id": "JAeltl6xGA", "reviewer": "Reviewer_WN29", "summary": "The paper tests whether n-gram novelty is predictive of creativity. The taxonamize creativity into sensibility and pragmatically and build an expert-annotated dataset to study this (~8000 samples are annotated). They conclude n-gram novelty is not a reliable predictor of creativity and worse of higher n-gram novelty is also correlated with loss of sensicality.", "questions": "- In Ln 217, you mention that ‘Instances that were labeled as not sensical were removed to focus on cases where expressions do not make sense in the context’; Will this not alter the logical flow of the entire text ? \n- You mention in Appendix D, that the pragmaticality scores consistently know a significant negative effect with the crowd preferences - Couldn’t this be indicative of the pragmaticality definition being ill-suited/divergent from what a naive-user/lay crowd prefers ?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The study between the n-gram novelty and creativity judgements from experts engenders a useful conclusion.  Sensibility vs pragmatically distinction is also very interesting; \n- The resulting corpus is substantial, providing scope for running large scale analysis for future work.", "weaknesses": "- Deviation from layman perception of high quality creativity versus an expert perception of high quality creativity: Since the definitions of creativity are expert curated the claims in the paper seem very plausible; However, it is unclear if the authors expect these conclusions to hold with less expert audiences who perceive creative texts through less varying rubrics (some of which may include allegedly spurious factors like novel n-grams)\n\n- Conditional dependence of sensicality/pragmaticality on the length of the snippet: It is unclear if the decision of the in-context sensicality or pragmaticality can vary if additional context is provided to the annotators. I understand that this is non-trivial to ablate due to the substantial variation in the induced cognitive burden to annotators but an ablation for this even a small control experiment that maintains the stability of the judgement with varying lengths over the snippet will help confirm the legitimacy,  of especially the sensible judgements.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "k6n97hogK7", "reviewer": "Reviewer_rBr1", "summary": "This paper re-examines n-gram novelty metrics which have been previously used as proxies for creativity. Specifically, they collect a pool of annotators to measure the creativity of llm-generated text and human gold text via close reading, finding that while n-gram novelty correlates with creativity, it also corresponds to lower pragmaticality compare to human texts. Their findings suggest defining better metrics for creativity that incorporate multiple factors.", "questions": "* Given the findings, what is an alternative metric for creativity?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "* The paper is well-written and easy to follow\n* A large amount of effort went into collecting a diverse set of annotations across many annotators and passages. The definition and task for annotators are clearly defined. In general, Section 2 is strong. The experiments are designed well and a lot of thought was put into them.\n* The findings about n-gram novelty relation to creativity (human judgement) are interesting and novel. There is a lot of analysis done that goes into specific details around this behavior which are interesting, like the decreasing pragmatism. \n* A variety of models are tested", "weaknesses": "* Minor: Formatting on line 2 is odd \" (Handa et al., 2025) (Anthropic) and (Chatterji et al., 2025) (OpenAI)\"\n* The main study comparing human and text generation is only based only on one source - The New Yorker. While this source is high-quality, it would be nice to compare LLM/Human texts in other domains to see how well n-gram novelty findings hold.\n* Minor: The findings could be stronger by including more models to analyze (in addition to the task mentioned above). \n* A conceptual limitation is that the paper assumes creative expressions must be both sensical and pragmatically appropriate. While this is reasonable for narrative prose, some creative language intentionally violates pragmatic norms (e.g., surreal or poetic writing). Thus, the finding that high n-gram novelty often corresponds to non-pragmatic text may not be a fundamental flaw of the metric on its own.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "68irnf0OuA", "reviewer": "Reviewer_NcHT", "summary": "This paper declared death of n-gram as a measure of creativity, with detailed study into rare n-gram in both human & AI generated passages and painstakingly annotated dataset. Findings are cool: rare n-grams are overwhelmingly not creative. The author continue to investigate if AI can replace human in such annotations.", "questions": "Please see weakness.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- This is a well-motivated and complete investigation into whether n-gram can be a good metric.\n- The author conducted rigorous annotations on n-gram quality and creativity, with the annotation guideline elaborate and well defined. Annotations of the samples are really well thought and detailed. Statistical analysis are thorough.\n- The author investigated various interesting research quesitons that flows well with the ideas, including if LLM can replace human in close reading and if experts are needed to create the best annotations.", "weaknesses": "- The scope of the work is clear but limited to New Yorker passages and n=26.\n- This result is intuitive but expected, once one understand the context. While the work is solid, its insights are not very novel by iclr standards.\n- The paper's writing can benefit from placing some details (e.g. python packages used) in appendix and devote more space to more analysis.\n- LLM as a judge's F1 performance is overly low. Some more insights would be helpful. The current statement \"it suggests it may be stemming from failure to recognize non-pragmatic expressions\" is unconvincing. \n- Similarly, finetuned pragmatic or novelty identification models perform poorly, with error bars swing to as bad as x = 0. A further invesitgation on why, or finetuning larger model might be needed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QJOvdWxn2D", "reviewer": "Reviewer_Z5SQ", "summary": "This paper challenges the use of n-gram novelty as a sufficient metric for evaluating textual creativity in language models. The authors argue that true creativity requires both novelty (how original the text is) and appropriateness (how sensical and pragmatic it is), based on established creativity theory. Through a study with 26 expert writers annotating 7542 expressions from human and AI-generated text, they find that 91% of top-quartile expressions by n-gram novelty are not judged as creative, and that higher n-gram novelty in open-source LLMs in facts correlates with lower pragmaticality. Furthermore, they found that LLMs are less likely to produce creative expressions than humans and even struggle to identify creative expressions (a positive aspect of writing) and non-pragmatic ones (a negative aspect).", "questions": "What do you think it's a scalable method to measure creativity beyond n-gram metrics and expert annotation?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well motivated, well written, and addresses an important research question on measuring creativity.\n2. The authors conduct high-quality expert annotations, recruited 26 professional writers from top MFA programs with rigorous training, achieving good inter-rater agreement (κ_free = 0.78 for novelty, 0.72 for pragmaticality)\n3. The finding is somewhat surprising: 91% of top-quartile expressions by n-gram novelty are not judged as creative by experts, highlighting the risk of over-relying on n-gram based metrics as a proxy for creativity.", "weaknesses": "1. The scale of the study is relatively small. For example, the human text samples include only 50 passages of roughly 400 words each.\n\n2. The domain is limited to creative fiction from The New Yorker, findings may or may not generalizes to other domains such as technical writing, journalism, or poetry, where measuring creativity is also very important. \n\n3. The authors rely on Infinigram and open-source OLMo/OLMo-2 corpora to compute n-gram novelty. However, the training data of frontier models (e.g., GPT-5, Claude 4.1) are not publicly available, raising the question of whether the observed limitations of n-gram novelty metric comes from the n-gram metric itself or from an imprecise estimation of n-gram novelty due to incomplete corpus coverage.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)