Measuring LLM Novelty As The Frontier Of Original And High-Quality Output
Authors: Vishakh Padmakumar
Chen Yueh-Han
Jane Pan
Valerie Chen
He He
Venue: ICLR 2026 Poster (2026)
Keywords: generation
evaluation
memorization
novelty
benchmark
creativity
Primary area: foundation or frontier models, including LLMs
TL;DR: We quantify and evaluate the novelty in LLM generations as the harmonic mean of the n-gram originality and output quality, and analyse how model scale, training methods and inference methods impact it.

Abstract: As large language models (LLMs) are increasingly used for ideation and scientific discovery, it is important to evaluate their ability to generate novel output. Prior work evaluates novelty as originality with respect to model training data, but original outputs can be of low quality. In contrast, non-expert judges more reliably score quality but may favor memorized outputs, limiting the reliability of human preference as a metric. We introduce a new novelty metric for LLM generations that balances originality and quality---the harmonic mean of the fraction of \ngrams unseen during training and a task-specific quality score. Using this framework, we identify trends that affect the novelty of generations from three families of open-data models (OLMo, OLMo-2, and Pythia) on three creative tasks---story completion, poetry writing, and creative tool use. We find that model-generated text from some base LLMs is less novel than human-written text from the internet. However, increasing model scale (OLMo 1B to 7B to 32B) and post-training reliably improves novelty due to improvements in output quality. We also find that improving the base model at the same scale (\eg OLMo 7B to OLMo-2 7B) leads to higher novelty due to higher originality. Finally, we observe that inference-time methods, such as prompting and providing novel in-context examples, have a much smaller effect on novelty, often increasing originality at the expense of quality. This highlights the need for further research into more effective elicitation strategies as we use models for creative applications.

Published as a conference paper at ICLR 2026
MEASURING LLM NOVELTY As THE FRONTIER OF ORIGINAL AND HIGH-QUALITY OUTPUT
Vishakh Padmakumar
Chen Yueh-Han
Jane Pan
New York University vishakh@nyu.eds
New York University yueh. han. chenenyu.ccommmo edu
New York University jane pan@nyu.... edu
Valerie Chen
He He
Carnegie Mellon University
New York University
(iken2@andrent cmu edu
hhe@nyu.
ABSTRACT
As large language models (LLMs) are increasingly used for ideation and scientific discovery, it is important evaluate their ability to generate novel output. Prior work evaluates novelty original outputs may be of low quality. In contrast, non-expert judges more reliably score quality but may favor memorized outputs, limiting the reliability of human preference as a metric. We introduce a new novelty metric for LLM generations that balances originality and quality the harmonic mean of the fraction of he-grams unseen during training and a task-specific quality score. Using this framework, we identify trends that affect the novelty of generations from three families of open-data models (OLMo, OLMo-2, and Pythia) on three creative tasks: story completion, poetry writing, and creative tool use. We find that model-generated text from some base LLMs is less novel than human-written text from the internet. However, increasing model scale and post-training reliably improves novelty due to improvements in output quality. We also find that improving the base model at the same scale (e.g., OLMo 7B to OLMo-2 7B) leads higher novelty due to higher originality. Finally, we observe that inference-time methods, such prompting and providing novel in-context examples, have much smaller effect on novelty, often increasing originality at the expense of quality. This highlights the need for further research into more effective clicitation strategies as we use models for creative applications.
originality with respect to model training data, but
INTRODUCTION
As large language models (LLMss) are increasingly used for creative tasks (Wan et al., 2024; Haase & Pokutta, 2024; Moruzzi & Margarido, 2024) and scientific discovery (Gottweis al., 2025; Feng et al., 2024), it is important to evaluate their ability to generate novel output. Past work measures novelty by memorization; that is, whether text fragments appear training data (McCoy et al., 2023; Merrill et al., 2024; Lu et al., 2024a). However, originality alone is not sufficient. Consider scenario which a user asks for suggestions from an LLM when writing a poem (Figure 1). The output may be highly original, but of poor quality. To identify high-quality outputs, leaderboards like Chatbot Arena (Chiang et al., 2024) collect and aggregate human preference judgments. However, these are unsatisfactory measures of novelty novice judge might score output highly, not knowing that it is copied verbatim from the pre-training data.
Ideally, models should generate output that uses expressive and figurative language without reproduc- ing the training data. In this paper, we argue that these two facets must be jointly considered. We propose measure novelty the harmonic mean of originality (measured by the fraction of unseen n-grams in a generation) and quality according to task-specific measures (Section 2.... We use this metric to answer the following research questions.

---

Published as a conference paper at ICLR 2026
Prompt: Write a poetic sentence that includes the word 'phases' and ends in 'moon'.
High Quality + Low Originality
Novel Output High Quality + High Originality
*Life moves in phoses, ever-changing, like the moon"
5,0
5.0
"Sewn with sharp gold and silver threads, like the ever-changing phoses of the moon."
Reforen
4,5
4.5
4.5
Output Quality
4.0
4.0
4,0
OLU 78
3.5
3.5
3.5
OUHo 2 328
3.0
GLM6-2 29
OLM> 78
3.0
3,0
OUM- 28
2.5
2,5
2,5
Low Quality High Originality "Through phases, the moon renounceth life like the moon"
CUU
oUi 1B
CUM ts.
2.0
2.0
2.0
1.5
CLP 1B
1.5
1.5
OLPO 28
1.0
0.37
0.60
0.82
1.0
0.40
0.66
0.92
1.0
0.37
0.60
0.82
Effect Scaling
(b) Effect Post-training
(c) Effect Improved
(1B 7B 32B)
(OLMo Base Instruct)
Models (OLMo OLMo
Originality Unseen N-gram Fraction
Figure 1: We evaluate LLMs' ability to generate novel text, defined as high-quality responses that avoid reproducing higher-order n-grams from training data (highlighted in blue). Novelty is measured as the harmonic mean of unseen n-gram fraction (I-axis) and output quality (y-axis) (Section 2.1). Contour lines denote equal novelty in each plot. We find that: (a) scaling models and (b) post-training increase novelty through improved quality, while (c) stronger base models (e.g., OLMo I to OLMo 2) improve novelty by generating more original output (Section 3). Inference-time methods (e.g.. novel ICL examples, Denial Prompting) have limited effect on shifting the novelty frontier (Section 4).
What factors affect the novelty of LLM output? We analyze generations from three families of open-data models- OLMo (Groeneveld et al., 2024), OLMo-2 (OLMo et al., 2024), and Pythia (Biderman et al., t023b>-oo identify factors that affect LLM novelty across three creativity-focusel tasks (Section 2.2). ranging from story completion (Eldan & Li, 2023) to poetry writing (Chakrabarty et al., 2022) to creative tool use (Tian et al.. 2024). We find that scaling LLMs results in more novel output (OLMo IB to 7B), though the gains plateau at higher scales (OLMo-2 7B to 32B). Here, the improvement comes from higher quality output while originality remains stable within a model family. Post-training also consistently leads to higher novelty than base models due to higher quality and similar originality across all model scales. Finally, improving the underlying base model at the same scale (e.g.. OLMo to OLMo 2) increases novelty by improving originality (Section 3).
Can we elicit more novel outputs from LLMs at inference time? We investigate whether inference- time methods (e.g., changing the decoding strategy or prompt) elicit more novel output. We find that while increasing the sampling temperature initially boosts novelty by increasing originality. these gains can be quickly outweighed by a decline in quality (Section 4.1). For prompting base LLMs (Section 4.2), we use high-novelt in-context examples; and for post-trained models (Section 4.3), we experiment with asking for novelty and denial prompting (Lu et al., 2024b). These methods have smaller effect novelty by generating slightly more original output while paying cost in quality.
Our main contribution is a metric for studying novelty that allows comparison of models from different families, scales, and training methods on an equal footing, which helps uncover the factors that affect novel output generation. Using this measure, we identify trade-offs between originality and quality, emphasizing the importance of considering both together. We find that scaling, alignment, and improving the underlying base LLM can push the Pareto frontier of novelty, whereas inference- time methods yield only limited gains, motivating further research into more effective elicitation strategies. While we focus on open-data models, which allow us to accurately evaluate originality. our analysis can also be extended naturally to black-box models, where providers can directly report aggregated novelty scores without exposing proprietary data (Section 6). This approach helps the community track novelty over time, place advances creative and scientific context, and evaluate true generalization for Al safety. We release the dataset of over 5000 LLM generations, with quality scores and copied n- a-grams to facilitate research along this direction.'
We will make our code and model outputs available upon publication.
2

---

Published as a conference paper at ICLR 2026
2 MEASURING NOVELTY OF LLM GENERATIONS
In this section, we present our evaluation method to measure the novelty of LLM generations. We introduce new metric definition (Section 2.1), that we apply to a suite of creative task datasets (Section 2.2) and finally provide details about how we operationalize our definition (Section 2.3) for subsequent experiments (Section and Section 4).
2.1 METRIC DEFINITION
We propose a measure of novelty that captures both originality (i.e., whether the text is different from the training data) and quality, ensuring that novel generations remain coherent and helpful to users.
Novel output should be original. We must first distinguish between content that is genuinely new, rather than reproducing the training data of the model. The de facto approach measuring the originality of output is to calculate the fraction of higher-order n- n-grams which do not appear in pre- and post-training data of LLMs (McCoy et al., 2023; Elazar al., 2024; Merrill et al., 2024; Lu et al., 2024a). This value can be seen as a distance metric with outputs containing more unseen n-grams as farther away from the training data and therefore more original. Following McCoy et al. (2023); Elazar et al. (2024); Merrill et al. (2024), we calculate 7-gram originality as the proportion of n-grams in a generation that do not appear in a corpus C, where corresponds the pre- and post-training corpora of the LLM used for generation. The tools used are detailed in Section 2.3.1.
Novel outputs should be high quality. Identifying original outputs alone is insufficient, since long-tail generations that are original may also be nonsensical. As such, we also desire outputs to be high-quality with respect to the user prompt. While we would ideally measure output quality using human annotations, large-scale human evaluation is impractical for benchmarking various ablations of model performance. Instead, we use LLM-as-at juddde- evaluation to rate output quality, providing a scalable approximation of user preferences. Since measures of quality highly task-specifie, provide the prompts used for each task in Section 2.2 and provide details about how we validate the reliability of automatic scoring in Section 2.3.2.
Our novelty metric. As illustrated by the example in Figure I, existing metrics capture just single dimension of novelty. For instance, metrics like Creativity Index (Lu et al., 2024a) and n-novelty (Merrill et al., 2024) only score originality and would incorrectly rank rare but poor-quality output highly. Meanwhile, benchmarks of output quality, like ChatBot Arena (Chiang al., 2024) rely on human ratings, which might favourably judge the unoriginal answer.? To aggregate both dimensions into single measure of novelty, we report the harmonic mean of quality (renormalized to value between O and 1) and originality (as measured by the unseen n-gram fraction) of each generation, which correctly identifies truly novel generations. 3 We report average novelty on three tasks (Section 2.2), allowing to compare different models and ablations of generation methods."
2.2 CREATIVE TASKS
We evaluate the novelty of generations on three tasks: story completion, poetry writing, and creative tool use. We select these tasks because they are open-ended, with a wide range of valid responses that allow for varying novelty. Table 5 provides examples of each task.
Story completion. We use the TinyStories dataset (Eldan Li, 2023) to evaluate model generated story endings. Following Yang al. (2022), the model is provided with a prompt consisting of the first line of story, which introduces the setting and characters, and must then complete the story. To score generation quality, we use an evaluation prompt that assigns points for correctly reusing and developing the introduced characters and plot elements, maintaining coherence, ensuring logical progression, and preserving grammatical correctness (Appendix E.I.
Poetry writing, We use the CoPoet dataset (Chakrabarty et al., 2022), where the model generates a single line of poetry in response given instruction about the content and literary devices to be
We provide more examples in Table 3.
'ee select the harmonic mean to penalize either originality or quality being low.
In Section 6 we detail how evaluation method can be used for black-hox models as well as updated as the research community makes progress in measuring more high-quality measures of originality.
3

---

Published as a conference paper at ICLR 2026
included. To score quality, instruction, correct use specified literary devices, coherence, and grammaticality (Appendix E.1.2).
evaluation prompt that assigns points based on adherence the
Creative tool use. We the MacGyver dataset (Tian et al., 2024) of reasoning problems that require creative use of items to complete physical objectives. The model is prompted with the scenario and must generate solution through innovative but feasible use of common objects. We score quality with a prompt that checks whether the proposed solution correctly utilizes the provided tools in valid manner, and successfully resolves the given problem (Appendix 1.1.)
2.3
OPERATIONALIZING OUR NOVELTY METRIC
2.3. CALCULATING OUTPUT ORIGINALITY
We measure originality as the fraction of n-grams that do not appear in model training data. We calculate this using the WIMBD API (Elazar al., 2024) and Infinigram (Liu et al., 2024; 2025), which index the pre- and post-training corpora of various open-data model families. Our experiments (Section 3 and Section 4) use the Pythia, OLMo, and OLMo-2 models which are covered by the indexes for the Pile (Gao et al., 2020), Dolma (Soldaini et al., 2024), Dolmino (OLMo et al., 2024), OLMo- Tulu SFT mixture (Ivison et al., 2023).
Prrffreneee mixture (OLMo et al., 2024), Tulu RLVR mixture (Lambert et al., 2024), and Ultrafeedback (Cui et al., 2024). This allows to check whether the constituent n-grams of generations from the models appear in their training data. Following Merrill et al. (2024), consider 4, 5, and 6, since smaller values result in nearly zero unseen n-grams, while larger values lead to almost all n-grams being unseen.
2.3.2 LLM-AS-A-JUDGE AS A MEASURE OF OUTPUT QUALITY
We use LLM-as-a-judge to approximate the measure of output quality from human annotators in scalable manner. To ensure that we obtain reliable ratings, we perform a human study. We obtain three human annotations each for 100 examples for all three tasks from Upwork. The scoring rubric provided to annotators was the same as the 'prompt' used with the LLM (Appendix E). We find that inter-annotator agreement, measured by Krippendorff's alpha (Krippendorff, 2018), was 0.68 for CoPoet, 0.64 for MacGyver, and 0.59 for TinyStories, consistent with agreement levels reported for creative tasks in contemporary works (Li et al., 2025; Sawicki al., 2025; Chiang & Lee, 2023: Chakrabarty et al., 2024). We then compare different LLMs and prompting setups (e.g.. in-context examples, average of multiple runs) using the Spearman correlation of model-assigned quality scores to the average annotator ratings (Table 4 in Appendix D). We find that the highest average correlation-O0 0.50 for CoPoet, 0.52 for TinyStories and 0.52 for MacGyver- o3 minii, averaging the scores over 5 runs. For the rest of this paper, all scores of output quality use this setup.
WHAT FACTORS AFFECT THE NOVELTY OF LLM OUTPUT?
3.1 EXPERIMENTAL SETUP
Models. We evaluate generations from three families of open-data models- OL.oo (Groeneveld et al., 2024), OL.o-2 (OLMo et al., 2024) and Pythia (Biderman et al., 2023b). We evaluate the following models: (1) OLMo- IB and 7B, (2) OLMo-2-1B, 7B, 13B and 32B, (3) Pythia-6.9B and 12B', (4) Pythia-Dedupee IB, 2.8B, 6.9B and 12B (Pythia DDP> Since these are base LLMs only pre-trained on the next-token objective, we provide 5 in-context learning (ICL) examples, randomly sampled from the validation split, to illustrate each task. We also evaluate OLMo-7B-Insteee OLMo-22Instruct IB, 7B, 13B, 32B to ablate the impact of post-training (with SFT+DPO for OLMo and SFT+DPO+RLVR for OLMo-2) on novelty. Unless stated otherwise, in this section, we use
and
Seee Appendix D for details about recruitment of annotators as well results on LLM-as-judee from different models and setups.
OOtputs from smaller Pythia models were very low quality.
"The deduplicated versions were trained for longer, 1.5 epochs of a deduplicated version of the same Pile (Gao et al., 2020) dataset, as opposed to one epoch for Pythia.
$Each test example is paired with unique set of ICI. examples. To ensure a fair comparison, the same ICI. examples are used across all models for each corresponding example.
4

---

Published as a conference paper at ICLR 2026
temperature of 1.0 during decoding. As noted in Section 2.3, response to the prompt using LLM-as-a-judge evaluation. For all tasks, we obtain quality scores from 0 to 5 with o3-mini with the corresponding prompts, and normalize these scores from 0 to 1. We report novelty as the harmonic mean of output quality and n- n-gram originality.
the quality of generations
Baselines. We compare the novelty of model generations with the references from each task dataset. The motivation for this baseline is to provide comparison to average human writing that we would like models outperform. Since the tasks we select are fairly open-ended, the references intended to provide a gold-standard score of novelty. To create a baseline for both model families, we compute the n n-gram originality of the references using Dolma (for OLMo baselines) or the Pile (for Pythia baselines). We score the quality of the references with c3-mini using the prompts from Section 2.2, and report the novelty as Baseline Dolma and Baseline Pile.
copoet, N-5
tinystories, N-5
mecgyver, N-5
Model: Darker + Larger Size
Baseline
Output Duality
OLMo (76)
OLMo-Instruct (7B)
OLMo-2 (1B, 78, 13B, 328)
-LMa-2-Instruct
61e 537
Unseen N-gram Fraction
Sar
64e
(1B, 78, 138, 32B)
Figure 2: Comparing novelty of base and post-trained LLMs by plotting output quality (y-axis) vs n-gram originality for = 5 (r-axis) for CoPoet, TinyStories and MacGyver. Post-training uniformly increases novelty at all model sizes for both OLMo and OLMo-2.
capont, Ne15
tinystories, Nw5
macgyver, Nw5
Model: Darker Size
Output Quality
Baseline
Mo (18, 79)
OLMo 2 (1B, 78, 136,
Pythia
Pythia-nDP (8,2.88, 6.98,
Unseen N-gram Fraction
Figure 3: Comparing novelty of models by plotting output quality (y-axis) vs n-gram originality for (z-axis) for CoPoet, TinyStories and MacGyver. Improving the underlying base LLM (OLMo to OLMo-2 and Pythia to Pythia-DDP) leads to higher novelty at the same model scale for all tasks, driven by higher originality, Increasing model scale (darker colors) leads higher novelty driven by higher output quality, particularly on TinyStories and MacGyver.
copoel, N
Unnystories, N=5
mecgyver, N=5
Model
Baseline
Output Quality
OLMe (7B)
OLMo-Instruct
OLMa-2 (7B)
(t SFT) 0-M0-2-5FT (7B)
f+ RLVR) OLMe-2-Instruct (7B)
gram Fraction
Figure 4: Comparing novelty of OLMo-7B and OLMo-2-7B by plotting output quality (y-axis) vs n-gram originality for n = (æ-axis) for CoPoet, Storieo and MacGyver. For OLMo-2, SFT improves output quality at the cost of originality when compared to the base model. This loss is recovered in the RLVR stage leading to more novel output in OLMo-2 Instruct.
3.2 RESULTS
We report results comparing the novelty of OLMo and OLMo-2 LLM generations with the bascline novelty of the references in each dataset Table with additional results from Pythia in Table 10 in Appendix G. We visualize trends from scaling base L.MMs in Figure and post-training in Figure 2.
5

---

Published as a conference paper at ICLR 2026
Table I: Comparing the novelty of LLM generations against the baseline of the references in each dataset (Section 3). Novelty is the harmonic mean of output quality and n-gram originality (Section 2.1) for n = 4, 5, and Each cell for novelty reports the relative improvement or drop compared to the baseline for that n value. Cells with asterisk indicate deviations with significance at the a = 0.05 level via a paired-samples t-test. We report the average case novelty as well as the novelty of the top 10% of generations. While some base LLMs generate less novel output on average than the baseline. increasing the model size, post-training and improving the underlying base model (c.g,, OLMo to OLMo-2), leads to higher novelty, See Table 10 for results on the Pythia models.
Dataset: TinyStories
Output Quality
n-gram Originality
Novelty (A to Baseline)
Top 10%
Novelty (A to Baseline)
n=5
4
Baseline- - Dolma
0.876
0.126
0.359
0.641
0.211
D.508
D.751
0.361
0.639
D.851
OLMMo-IB
0.614
0.159
0.376
0.631
-0.010
0.096
0.190"
+0.108
+0.078
0.012
OLMo-7B
0.766
0.148
0.374
0.619
+0.012
-0.026
-0.069'
+0.121
+0.089
+00002
OfMoo-BB.nstruct
0.852
0.171
0.422
0.650
+0.058*
+0.044"
0.007
+0.124
+0.096
+0.031
OLM--221B
0.603
0.500
0.757
0.876
+0.291"
+0.156"
-0.082
+0.390
+0.229
+0.055
0.758
0.511
0.758
0.856
+0.366
+0.225'
+0.033
+0.440
+0.2953
+
OLMo-2-33B
0.795
0.503
0.775
0,900
+0.377
+0. 20.
+0.072
+0.422
+0.288
+0.131
M.Mo-22-B-Instreene
0.870
0.598
0.848
0.959
+0.484
+0.3477
+0..153
40.472
+0.317
+-0.144
LL.Mo-2-7B-Instruc
0.936
0.590
0.837
0.954
+0.5003"
+-0.778'
+0.191
+0.492
+0.325
+0.146
OLMo-2-32B-Insstee
0.945
0.508
0.830
0.953
+0.4877
4-0.376'
196'
+0.472
+0.328
146
Dataset: CoPoet
Output Quality
r-gram Originality
Novelty (A to Baseline)
Top 10% Novelty (1 Baseline)
n = 4
n 5
6
==4 4
== 5
n 6
4
n 5
n = 6
Baseline- Dolma
0.626
0.188
0.358
0.462
0.228
0.303
0.139
0.727
0.888
0.988
OLMo-1B
0,400
0.135
0.324
0.527
-0.0988
-0.108"
-0.078
00.147
-0.138
-0.147
OLMo-7B
0.394
0.196
0.413
0.569
-0.079"
-0.105
-0.120
-0.117
-0.078
-0.103
OLMo-7B-Instruuc
0.617
0.402
0,705
0,866
+0.177"
+0.2311
+0.2266
+0,104
+0.029
-0.034
OLMo-2-1B
0.401
0.564
0.754
0.788
+0.172"
+0.101
+0.018
+0.015
-0.05
0.185
OLMo-2-7B
0.483
0.549
0.772
0.870
+0.214"
+0.180"
+0.128
+0.062
-0.033
00.055
0LMo-2-32B
0.367
0.504
0.743
0.785
+0.137
+0.089"
+0.002
+0.005
-0.01
00.185
9LMM--2-BB-Isstrue
0,584
0.770
0.920
0,912
-01"
+0.329"
+0.2511
+0,156
-0.011
-0.082
OL.Mo-2-7B-Instruc
0.700
0.834
0.930
0.926
+0.511
+0.400"
+0.319"
+0,208
+0.077
-0.015
OLMo-2-32B-Inne
0.664
0.735
0.911
0,962
+0. 139"
+0.356
+0.3277
+0.171
+0.031
-0.055
Dataset: MacGyver
Output Quality
P-gram Originality
Novelty (A to Baseline)
Top 10%
Novelty (A to Baseline)
8
n 4
5=5
n
H 4
n
Baseline- - Dolma
0.908
0.359
0.601
0,803
0.505
0.728
0.856
9.629
0.811
0.966
OLMo-IB
0.278
0.267
0.505
0.739
00.281
0.416
-0.404
0.212
0.270
0.266
OLMo-7B
0.458
0.286
0.520
0.747
-0.200
-0.294
-0.330
-0.117
-0.146
-0.145
OLMo-7B-Instruct
0.620
0.297
0.559
0.781
-0.126
-0.168
-0.192
-0.092
-0.103
-0.120
OL-Mo-2-IB
0.298
0.595
0.843
0.953
-0.147
0.325
00.43
+0.073
-0.055
0.112
OLMo-2-7B
0.519
0.609
0.850
0.955
+00001
-0.141
-0.240
+0.191
+0.102
+0.020
OLMo-2233B
0.719
0.630
0.858
0.958
+0.126"
+0.012
-0.077
+0.242
+0.131
+0.031
OLMe-2-1B- IIssrrucl
0.619
0.672
0.889
0.971
+0.091
00.488
-0.149
+0.223
+0.124
+0.028
D..Mo-2-7B-Inntmee Instruct
0.892
0.677
0.883
0.969
+0.247"
+0.142"
+0.055
+0.258
+0.140
+0.034
OLMo-2-32B-Innee
0.971
0:681
0.892
0.973
+0.200
+0.198
+0.112"
+0.263
+0.140
+0.034
Novelty has a positive scaling trend, driven largely by improved quality, that plateaus at large sizes. We observe the effect of model size on novelty by comparing models of different sizes from IB to 32B in each model family (Figure 3). The average novelty increases for larger models in all families when increasing model size from IB to 7B (OLMo and OLMo-2) and 1B/2.8B to 6.9B (Pythia DDP). This trend in particular for all three tasks and for all values of n for OLMo and OLMo-2. From Table 1, the novelty gain from OLMo- 1B to OLMo-7B comes from improved quality TinyStories (+19%) and MacGyver (+39%), while CoPoet benefits from higher n-gram originality (+20%) despite a slight quality drop (-1.5%) The relative change in n-gram originality are minimal for TinyStories (-3%) and MacGyver (+3%). We also see from Figure 3 that subsequent increase in model size from 7B to 32B (OLMo-2) 2) and 6.9B to 12B (Pythia-DDP) has a more mixed effect on novelty, suggesting that the effects plateau once certain scale is reached. Going from OLMo-2-7B to 32B leads to a change in novelty of +1.8% on TinyStories, -9% on CoPoet, +21% on MacGyver for n = 4 (Table 1) with similar effects for Pythia (Table 10 in Appendix G). However, we do note

---

Published as a conference paper at ICLR 2026
that the average novelty of the Top 10% of generations is uniformly higher for the largest models in all model families, for all tasks and n values, indicating that the most novel outputs still scale." 9
Improving the underlying base LLMs leads to more novel output at the same model scale. Across all three tasks, improving the base model leads to higher novelty at the same scale (Figure 3). We observe this effect from Pythia Pythia DDP for 6.9B and 12B (same dataset in the same order, just more epochs of training) and from OLMo to OLMo-2 for 1B and 7B (same scale with slight modifications to the dataset and training recipe). The gap is more pronounced for poetry writing (CoPoet) and story completions (TinyStories) compared to problem-solving (MacGyver), but consistent for each model size.
Post-traine training helps models generate more novel text than corresponding base LLMs. From Figure 2 and Table I, we see that for OLMo and OLMo-2, the post-trained models have higher novelty than corresponding base LLMs at all model sizes. This improvement is due to consistently higher-quality outputs, as expected, and also a slightly higher n-gram originality across all three tasks. 10 The effect varies by task and is most pronounced for CoPoet, which closely matches the format used instruction tuning. On the other hand, for MacGyver, where the problem format matches instruction tuning, but the domain differs significantly from typical post-training tasks.
SFT improves output quality at the cost of originality, RL recovers this loss to lead to more novel text. In Figure 4, we compare versions of OLMo and OLMo-2 at diffferent stages of post-training. For both, the final Instruct models have higher novelty than the base models. However we observe that the intermediate checkpoint after the SFT stage leads to slightly higher quality, which is almost entirely offset by a loss in originality leading to similar novelty. This is then recovered in the preference tuning stage of RLVR (for OLMo-2) or DPO (for OLMo) which lead to the higher novelty of Instruct models. This finding echoes observation of the SFT stage leading to more memorization while generalizable improvement in performance is more from preference tuning (Chu et al., 2025).
4
CAN WE ELICIT MORE NOVEL OUTPUT FROM LLMS?
While increasing model size, post-training and improving the base model yield higher novelty (Section 3), modifying the model itself is not always feasible, so we explore whether there are inference-time methods that can elicit greater novelty. Taken as whole, we find that varying the sampling temperature (Section 4.1) and prompt format (Section 4.2 and Section 4.3) tend to trade off originality and quality, minimally moving the frontier of novelty (Figure 6).
4.1 EFFECT OF VARYING THE SAMPLING TEMPERATURE
One way to elicit higher novelty is to increase the n-gram originality in the generated text. A simple approach to sample rarer outputs by increasing the temperature during decoding (Merrill et al., 2024). To study this effect, we generate outputs from OLMo-7B and Pythia- 12B across 750 prompts from TinyStories, CoPoet, and MacGyver with a fixed set of ICL examples, but varying the sampling temperature in increments. We test 0.5, 0.75, 1, 1.5, 2.0.
Increasing sampling temperature has a U-shaped effect on novelty. As shown in Figure 5, increasing the sampling temperature initially leads to higher novelty, caused by an increasing n-gram originality, as the model generates more rare, less memorized text. However, beyond certain point, quality deteriorates, leading to a decline in novelty. We find that the inflection point at which this shift occurs, or the optimum temperature value for novelty, varies by task. In practice, temperature should be tuned rather than using a fixed value, since the optimal value is not consistent across models and tasks. This again highlights the value in our formulation of novelty jointly considering originality and quality-while n-gram originality monotonically increases with increased temperature our formulation can distinguish between long-tail generations that are novel or degenerate.
We include this finding due to the observation that for some tasks like creative writing assistance or protein design. the best output is useful measure of model performance as these be filtered and used.
'WWe note that Lu et al. (2024a) report that the creativity index of models, a measure of n n-gram originality. reduces with RLHF tuning. In contrast, we find that both originality and quality increase with alignment. This discrepancy could he due to their use of large reference corpus of internet text to calculate n-gram originality, whereas use the training corpora of the model.
7

---

Published as a conference paper at ICLR 2026
copoet, N-5
tinystories, N-5
macgyver. N=5
Output Quality
Model:
Higher Temperature
Pythia 12B
0.2 0.30
033
C32
028
633
035
3.3
Unseen N-gram Fraction
Figure 5: Effect of varying sampling temperature on novelty by plotting output quality (y-axis) vs n-gram originality for (æ-axis) for CoPoet, TinySto and MacGyver. Increasing sampling temperature (darker colors) from 0.5 to 2 for OLMo-7B and Pythia- 1ythia-12B increases originality, with cost to output quality, resulting in similar novelty levels (Section 4.1). Table 9 has the raw scores.
5.-
copoet, N=5
tinystories, N=5
macgyver, N
Model
Baseline
4.3
Output Quality
OLMc
T OI Mo
3.5
Base
3.0-
OLMc
2.5
H
Prompt Variants
A Instruct-Adk
1.5
Danial Promp:
1.9.22
0.60
0.56 022
0.60
0.50 0.22
0.60
6.58
6 Novel
Unseen N-gram Fraction
Figure 6: Effect of varying the prompting method on novelty by plotting output quality (y-axis) vs n-gram originality for (æ-axiss for CoPoet, TinyStories, and MacGyver. Different prompting methods-providing novel ICL examples (Section 4.2) for Base models, and Asking for novelty and little effect on novelty, and often trade off small increase of originality for slightly lower quality. Figure 9 shows the same plot for other n.
Denial Prompting on Instruct models (Section
4.2 EFFECT OF PROMPTING WITH NOVEL IN-CONTEXT EXAMPLES
Another way elicit original text without sacrificing quality is more novel ICL examples. We hypothesize that the LLM can recognize patterns in these examples and adjust its generations to match their novelty (Brown et al., 2020). We identify these ICL examples by scoring 1000 held-out examples from each dataset for novelty and selecting examples in the top 10% of scores." !I We provide ICL, examples randomly sampled from these for inference on the test set of 750 prompts from each dataset with OLMo-7B and OLMo--27B using temperature 1.0. We compare the performance of inference with these novel ICL examples to a baseline of OLMo-7B with the same temperature. providing 5 randomly sampled ICL examples from the held-out set.
Providing novel ICL examples makes little difference to novelty. From Table and Figure 6, very small change in novelty values over the corresponding for both OLMo and OLMo-2, We see an increase in novelty for OLMo-7B on CoPoct (+15.5%) 0.05 level, while OLMo-2 suffers small
baseline models on all three tasks.
and MacGyver (+5.5%) with significance the decrease in novelty for all tasks. In Table 1,, we see that this small effect persists on increasing the size of the underlying model to OLMo-2232B.
Some qualitative examples of the change in performance include following the instructions with more expressiveness in CoPoet, and providing more brief MacGyver solutions (Table 12) while find more brief story completions with TinyStories (Table 13).
4.3 PROMPTING POST-TRAINED MODELS FOR NOVELTY
Post-trained models are capable of following more complex instructions, allow us to experiment with eliciting novelty with more creative prompting techniques. We experiment with two such methods on 750 examples from each dataset.
Here the average novelty across 4,5.
8

---

Published as a conference paper at ICLR 2026
Table 2: Comparing the effect of prompting interventions on the novelty of LLM generations for 4,5, 6 (Section 4). Each cell for novelty reports the relative change compared to the baseline. We report the average case novelty as well as the novelty of the top 10% of generations. Cells with an asterisk indicate deviations with significance at the 0.05 significance level via paired-samples 1-test. Providing novel ICL examples uniformly increases the novelty of OLMo-7B (Section 4.2). on CoPoet and
Asking for novelty and Denial Prompting improve performance of TinyStories by generating more original output with higher n-gram originality (Section 4.3).
Dataset: TinyStories
Output Quality
7-gram Originality
Novelty (A to Baseline)
Top 10%
Novelty (A Baseline)
n= -
==
==
R=4
n=5
==6
4
== 5
n ==
OLMO-7B
0.708
0.148
0.374
0.619
0.226
0.477
0.662
0.485
0.728
0,853
Novel ICL
0.778
0.151
0.365
0.616
+0.012
-0.003
+0.003
-0.010
-0.030
-0.007
OL B-Instruct
0.852
0.171
0.122
0.680
0,272
0.547
0.744
0.488
0.735
0.882
+ Asking
0.780
0.190
0.447
0.694
+0.019
+0.003
0.027
-0.026
0.031
-0.050
Demial Prompt
0.738
0.219
0.185
0.730
+0.045
+0.011
00035
+0.031
0.0015
0.033
OLMo-2-7B
0.758
0.801
0.758
0.728
0.932
0.886
0.785
0.983
+ Novel ICL
0.749
0.506
0.576
0.798
-0.001
+0.000
0.004
-0.044
+00001
0.002
9LMo-2-7B-Instruee
0.936
0.856
0.837
0.881
0.961
0.951
0.912
0.997
+ Asking
0.9:3
0.676
0,781
0.881
+0.053
+0,029
+ -0.012
+0.013
+0.008
-0.00K3
Denial Prompt
0,899
0.682
0.766
0.882
+000055
+0.005
+0.012
40.014
00.019
-0.001
Dataset: CoPoet
Output Quality
n-gram Originality
Novelty (A to Baseline)
Top 10%
Novelty (A to Baseline)
==5
n
OLMO-7B
0.394
0.196
0.413
0.589
0.149
0.258
0.319
0.610
0,810
0.885
Novel ICL
0.409
0.269
0.170
0.614
+0,000
+0.050"
+0.013
+0.020
D.0022
-00111
0.617
0.402
0,705
0.886
0.405
0.594
0.665
0.831
0.917
0.954
+ Asking
0.591
0.424
0.715
0.896
+0.039
+0.005
+0.003
-0.000
0.040
0.028
Denial Prompt
0.591
0.436
0.732
0.899
+0.051
+0.019
+0.008
-0.095
0.040
0.040
OLMo-2-7B
0.483
0.549
0.442
0.789
0.772
0.543
0.855
0.870
0.567
0.883
Novel ICL
0.498
0.816
0.018
0.002
0.042
0.008
0.006
OL 2-7-
0.700
0834
0.739
0.935
0.930
0.772
0.965
0.026
0.758
0.974
Asking
0.666
0.888
0.913
0.046
0.000
+0.068
+0.027
0.016
Denial Prompt
0.646
0.885
0.728
0.920
+0.042
0.016
0.028
+0.065
+0.004
-0,028
Dataset: MaeGyver
Output Quality
n-gram Originality
Novelty (A to Baseline)
Top 10%
Novelty (A to Baseline)
4
n=5 5
# 4
==5
6
4
a=5 5
6
OLMO-7B
0.458
0.286
0.520
0.747
0.305
0.434
0.517
0.512
0.695
0.821
Novel ICL
0.480
0.320
0.545
0.760
+0.031
+0.030"
+0.029"
+0.061
+0.041
-0.022
0.620
0.297
0.559
0.781
0.379
0.560
0.664
0.537
0.738
0.846
Asking
0.548
0,230
0.524
0.774
-0.096
-0.074'
-0.022
-0.064
-0.015
-0.012
Denial Prompt
0.555
0.223
0.527
0.780
-0.890
0.0600
0.057
-0.074
0.015
-0.002
OL MLMO-2-7B
0.519
0.609
0.506
0.820
0,850
0.587
0.943
0.955
0.616
0
Novel ICL
0.491
0.625
0.495
0.807
+0.000
-0.019
-0.08
-0.003
-0 -0.023
0.003
0,802
0.677
0.752
0.887
0.883
0.670
0
0.969
0.911
1.000
+ Asking
0.940
0.719
0.799
0.916
+0.028
+0.039
+0.004
-0.004
+0.022
-0.000
Demial Prompt
0.879
0.734
0.777
0.909
+0.033
-0.001
+0.004
-0.002
-0.021
-0.000
Asking for novelty, We test whether explicitly requesting rare and high-quality output can improve novelty. We prompt the model with the description of the task as well as our definition of novel outputs with a thain-of-thought (Wei et al., 2022). The prompt is provided in Appendix E.2. Denial prompting. Based on the strategy introduced by Lu et al. (2024b), we iteratively sample output from the LLM, identify high-level concepts used in the output, and restrict the reuse of these concepts in subsequent generations. We apply this technique to OLMo-Instruct and OLMo-2- Instruct, running three rounds of inference with the prompt provided in Appendix E.3. After each round, we use GPT-4o to extract high-level concepts from the freeform text responses with the prompt provided Appendix F.4. These include character arcs and themes in TinyStories, literary devices used in CoPoet, and reasoning steps MaeGyver. We provide an example in Table 17. These concepts are then appended to the generation prompt for the next round.
Prompting techniques trade off originality and quality, without moving novelty by much. From Table 2, both prompting approaches improve novelty for Ttoris (+6.9% for OLMo and +6.3% for OLMo-2 on average) and CoPoet (+9.6% for OLMo and +4.9% for OLMo-2 on average), and reduce very minorly affect novelty on MacGyver (-18% for OLMo and +3.7% for OLMo-2 on average). However, the total effect on novelty is much smaller than that observed in Section 3. These methods reduce output quality across all tasks, but this is offset by higher n-gram originality in
9

---

Published as a conference paper at ICLR 2026
TinyStories and CoPoet. In contrast, MacGyver shows drop in both n-gram originality and novelty, likely due to degenerate outputs (Table 15, Table 16). From Table 11, we see that this effect persists even for larger model sizes of OLMo-2.
5 RELATED WORK
Analysis of memorization of n-grams. Our work builds on past work quantifying the n-gram originality of LLLM-generated text. McCoy et al. (2023) and Merrill al. (2024) analyze how n-gram originality in LLM generations compares to pre-training datasets, examining its variation with model size and decoding strategies. Elazar al. (2024); Liu et al. (2024; 2025) introduce tools for analyzing memorization from open pre-training datasets, which we use in this work. Huang et al. (2024); Carlini et al. (2023); Biderman et al. (2023a) show that memorization increases with data duplication, later training checkpoints, model capacity, dataset repetition, and prompting context. Carlini et al. (2021); Kandpal et al. (2022) highlight privacy risks by demonstrating that LLMs can regenerate sensitive training data. Aerni et al. (2025) find that memorization varies by task, with prompting offering some mitigation but failing in worst-case scenarios. Our work extends this line of work on the analysis of memorization of output and, to our knowledge, is the first to examine the trade-off between originality and task-specific measures of output quality. Most closely related to our work is Lu et al. (2024a) which quantifies the creativity of LLM-generated text by the fraction of the text not included in n-grams from reference corpus, a measure of originality of text. We demonstrate the need to consider output quality as an additional signal when evaluating the novelty (Section 4.1).
Evaluating creativity in generations. Our definition of novelty as high-quality original content is also related to definitions of creativity in the literature. Prior works have proposed metrics for creativity inspired by the Torrance test for creative thinking (Torrance, 1966) that quantify measures of quality and originality via LL--as-judge scores (Zhao et al., 2024; Chakrabarty et al., 2024). While these correlate with non-experts, they diverge from expert ratings making LLM-as-junt juddgee unreliable for originality, As expert annotations are not scalable, we measure quality with an LLM the training data. Closely related to definitions of originality is the concept of output diversity. Output diversity has been shown to often trade-off with quality (Lanchantin et al., 2025; Chung et al., 2025) with some observed variance based on task (Shypula et al.. 2025; Jain et al.. 2025). While diversity quantifies the degree of difference within a set of generations, originality for our work is measured with respect to model training data and we show that originality has a similar trade-off to output quality.
and originality programmatically
6 DISCUSSION AND CONCLUSION
In this work, we propose a metric to evaluate the novelty of --geeerrrte output that balances originality, quantified as the fraction of n-grams absent from the model's training data, and task- specific quality. We evaluate the novelty of generations from the OLMo, OLMo-2, and Pythia models on three datasets and observe that increasing model size, improving the underlying model, and post-training can shift the frontier of novelty. However, most inference-time measures of improving novelty often trade-off gains in originality with a cost in output quality. We detail some limitations of our work in Appendix B.
Our findings also motivate several possible extensions. First, our novelty metric can be generalized to framework that scores quality or originality in other ways. This involves expanding the definition of originality to incorporate recent work on non-literal memorization (Chen et al., 2024) as well as developing complex measures for output quality specific tasks, such search-augmented quality score for research idea generation. Second, while we focus on open-data models, our analysis can be extended to black-box models, where providers can directly report aggregated novelty scores without exposing proprietary data (Appendix C). In doing so, we give the community tools to track novelty as models rapidly evolve, contextualizing advances in creative and scientific domains, and assessing true generalization for Al safety.
ACKNOWLEDGEMENT
We would like to thank Will Merrill, Nitish Joshi, Nick Lourie, Mina Lee, Swaroop Mishra, Faeze Brahman, and Yanai Elazar for their input various stages of the project. This work is supported
10

---

Published as a conference paper at ICLR 2026
by the National Science Foundation under Grant No. IIS-2340345 and Grant No. 1922658, Open Philanthropy, and AWS AL.
REFERENCES
Michael Aerni, Javier Rando, Edoardo Debenedetti, Nicholas Carlini, Daphne Ippolito. and Florian Traměr. Measuring aovessaeraliial reproduction of training data large language models. In The Thirteenth International Conference on Learning Representations, 2025. URL https: // //openreview.oe net/ forum? 11= fq
Stella Biderman, Usvsn Prashanth, Lintang Sutawika, Hailey Schoelkopf, Quentin Anthony, Shivan- shu Purohit, and Edward Raff. Emergent and predictable memorization in large language models. Advances in Neural Information Processing Systems, 36:28072-28099, 2023a.
Stella Biderman, Hailey Schoelkopf. Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pp. 2397-2430. PMLR, 2023b.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D) Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Ad- vances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Asso- ciates, Inc., 2020. URL https: http:///oooeeedddee ings neurips httpp..c//peee wc/paper_eeeeeeeeeeeeeeeeee.. fi Pifilespappe 5 htt.:////..........1111/11111... Pap..ee.eeaaaap.........
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lec, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. Extracting training data from large language models. In 30th USENIX security symposium (USENIX Security 2/), pp- 2633-2650, 2021.
Nicholas Carlini, Daphne Ippolito. Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models. The Eleventh International Conference Learning Representations, 2023. URL https: http:///pppeeeeeeeeeeeeeeeeeeeet net wtt.o/wruetttttn id- TatRHT_IcK.
Tuhin Chakrabarty, Vishakh Padmakumar, and He He. Help me write poem: Instruction tuning as vehicle for collaborative poetry writing. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp.
2022.
Tuhin Chakrabarty, Philippe Laban, Divyansh Agarwal, Smaranda Muresan, and Chien-Sheng Wu.
Art or artifice? large language models and the false promise of creativity. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems, CHI '24, New York. NY. USA, 2024. Association for Computing Machinery, ISBN 9798400703300. doi: 10.1145/3613904.64422
URL https: /doi org. 1111/111111111
3642731.
Tong Chen, Akari Asai, Niloofar Mireshghallah, Sewon Min, James Grimmelmann, Yejin Choi, Hannaneh Hajishirzi, Luke Zettlemoyer, and Pang Wei Koh. CopyBench: Measuring literal and non- literal reproduction of copyriihtt-rooteceed text in language model generation. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 15134-15158, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10. 20.11863/v1/2222 1/2054////2...mmmlpp.... emnlp-main 844. URL ht https: ac wel.anntooogyyyy org/ 2024. emnlp-man in. 844/
Cheng-Han Chiang and Hung- Yi Lee. Can large language models be an alternative to human evaluations? In Proceedings of the 6lst Anmual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 15607-15631, 2023.
11

---

Published as a conference paper at ICLR 2026
Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael Jordan, Joseph E. Gonzalez. and lon Stoica. Chatbot arena: An open platform for evaluating LLMs by human preference. In Forty-first International https:://ppeeeeeeeetere//// /wpppppeeeevieeeeeeee ew. net. / forum? i=d=
Conference on Machine Learning, 2024. URL https: 3MW8GRNyZI.
Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoe V Le. Sergey Levine, and Yi Ma. SFT memorizes, RL generalizes: A comparative study of foundation model post-training, In Forty-second International Conference on Machine Learning, 2025, URL https: httpp:://ppereeveww net. / forum? i d=dYur3yabbt,
John Joon Young Chung, Vishakh Padmakumar, Melissa Roemmele, Yugian Sun, and Max Kreminski. Modifying large language model post-training for diverse creative writing. arXiv preprint arXiv:2503. 17126, 2025.
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Bingxiang He, Wei Zhu, Yuan Ni, Guotong Xie, Ruobing Xie, Yankai Lin, Zhiyuan Liu, and Maosong Sun. ULTRAFEEDBACK: Boosting language models with scaled AI feedback. In Forty-first International Conference on Machine Learning, 2024. URL https: pttttt::///peeeeeeee /oppneeveeeeeee.e// net /tte//oonnnnnnoddaaaae
Yanai Elazar, Akshita Bhagia, Ian Helgi Magmusson, Abhilasha Ravichander, Dustin Schwenk, Alane Suhr, Evan Pete Walsh, Dirk Groeneveld, Luca Soldaini, Sameer Singh, Hannaneh Hajishirzi, Noah A. Smith, and Jesse Dodge. What's in my big data? In The Twelfth International Confer- ence on Learning Representations, 2024. URL https://p/////...... RvfPnOkPV4.
URt.htts://oppnreee /tpp/rppeeeiieeeeee lew net. forum? d==
Ronen Eldan and Yuan-Fang Li. Tinystories: How small can language models be and still speak coherent english? ArXiv, abs/2305.0777 007799, 2023. URL https: / //pppaap.//////////./ semant iesccocllss org/CorpusID: 258666444..
KJ Feng, Kevin Pu, Matt Latzke, Tal August, Pao Siangliulue, Jonathan Bragg, Daniel S Weld, Amy X Zhang, and Joseph Chee Chang. Cocoa: Co-planning and co-execution with ai agents. arXiv preprint arXiv:2412. 109999 2024.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.000 2020.
Juraj Gottweis, Wei-Hung Weng, Alexander Daryin, Tao Tu, Anil Palepu, Petar Sirkovic, Artiom Myaskovsky, Felix Weissenberger, Keran Rong, Ryutaro Tanno, et al. Towards an ai co-scientist. arXiv preprint ar Xix:22502.18864 2025.
Dirk Groeneveld, Iz Beltagy, Evan Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khyathi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, Tushar Khot, William Merrill, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew Peters, Valentina Pyatkin, Abhilasha Ravichander, Dustin Schwenk, Saurabh Shah, William Smith, Emma Strubell, Nishant Subramani, Mitchell Wortsman, Pradeep Dasigi. Nathan Lambert, Kyle Richardson, Luke Zettlemoyer, Jesse Dodge, Kyle Lo, Luca Soldaini, Noah Smith, and Hannaneh Hajishirzi. OLMo: Accelerating the science of language models. In Lun- Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp. 15789-15809, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/ waw.acl--oon881111 URL https: /tt////laaanntttt/ea
2204 11.22..cc-1o0 acl-long. 841/
Jennifer Haase and Sebastian Pokutta. Human-ai co-creativity: Exploring synergies across levels of creative collaboration. arXiv preprint arXiv:2411.12527, 2024.
Jing Huang, Diyi Yang, and Christopher Potts. Demystifying verbatim memorization in large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 1071 10711-10732 2024.
12

---

Published as a conference paper at ICLR 2026
Hamish Ivison, Yizhong Wang. Valentina Pyatkin, Nathan Lambert, Matthew Peters, Pradeep Dasigi, Joel Jang, David Wadden, Noah A Smith, Iz Beltagy, et al. Camels in a changing climate: Enhancing im adaptation with tulu 2. arXiv preprint arXtv:2311.1070. 2023.
Shomik Jain, Jack Lanchantin, Maximilian Nickel, Karen Ullrich, Ashia Wilson, and Jamelle Watson- Daniels. Llm output homogenization is task dependent. arXiv preprint arXiv: 2/267, 2025.
Nikhil Kandpal, Eric Wallace, and Colin Raffel. Deduplicating training data mitigates privacy risks in language models. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Muchine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 10697-10707. PMLR, 17-23 Jul 2022. URL https::tp/............tt / /twp/pocceee ngs.... . mll www.pprss/vee 62 / /aa.dpanalpll 22a hhtll.
Klaus Krippendorff. Content analysis: An introduction to its methodology. Sage publications, 2018.
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V. Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luca Soldaini, Noah A. Smith, Yizhong Wang, Pradeep Dasigi, and Hannaneh Hajishirzi. Tulu 3: Pushing frontiers in open language model post-training, 2024. URL htttpps//arxiv.orrt 1////211..111222
Jack Lanchantin, Angelica Chen, Shehzaad Dhuliawala, Ping Yu, Jason Weston, Sainbayar Sukhbaatar, and Ilia Kulikov. Diverse preference optimization. arXiv preprint arXiv:2501. 18100 2025.
Ruizhe Li, Chiwei Zhu, Benfeng Xu, Xiaorui Wang, and Zhendong Mao. Automated creativity evalu- ation for large language models: A reference-based approach. arXiv preprint arXiv:2504115784, 2025.
Jiacheng Liu, Sewon Min, Luke Zettlemoyer, Yejin Choi, and Hannaneh Hajishirzi. Infini-gram: Scaling unbounded n-gram language models to a trillion tokens. arXiv preprint arXiv:2401.1737) 2024.
Jacheng Liu, Taylor Blanton, Yanai Elazar, Sewon Min, YenSung Chen, Arnavi Chheda-Kothary Huy Tran, Byron Bischoff, Eric Marsh, Michael Schmitz, Cassidy Trier, Aaron Sarnat, Jenna James, Jon Borchardt, Bailey Kuehl, Evie Cheng, Karen Farley, Sruthi Sreeram, Taira Anderson, David Albright, Carissa Schoenick, Luca Soldaini, Dirk Groeneveld, Rock Yuren Pang, Pang Wei Koh, Noah A. Smith, Sophie Lebrecht, Yejin Choi, Hannaneh Hajishirzi, Ali Farhadi, and Jesse Tracing language model outputs back to trillions of training tokens, 2025.
Dodge. Olmotrace:
URL //axxiv./ org/ abs/ 07096.
Ximing Lu, Melanie Sclar, Skyler Hallinan, Niloofar Mireshghallah, Jiacheng Liu, Seungju Han, Allyson Ettinger, Liwei Jiang, Khyathi Chandu, Nouha Dziri, et al. Ai as humanity's 's salieri: Quantifying linguistic creativity of language models via systematic attribution of machine text against web text. arXiv preprint arXiv:2410.0436. 2024a.
Yining Lu, Dixuan Wang, Tianjian Li, Dongwei Jiang, and Daniel Khashabi. Benchmarking language model creativity: case study on code generation, 2024b. URL https: h/tp:///aaaiaa//ae org/ abs/ 2407. 09007.
R. Thomas McCoy, Paul Smolensky, Tal Linzen, Jianfeng Gao, and Asli Celikyilmaz. How much do language models copy from their training data? evaluating linguistic novelty in text generation using RAVEN. Transactions of the Association for Computational Linguistics, 11:652-670, 2023. doi: 10. 10.1162/tac_a. Laaeel___0000 a_0667.. URL https: aaap://acllllllla antlllntooolllg.ooo ogy wwwg///// 10..2.t.aaa. tac. -1 -1.38. 38.
William Merrill, Noah A Smith, and Yanai Elazar. Evaluating n-gram novelty of language models using rusty-dawg. arXiv preprint arXiv:2406.11300 2024.
Caterina Moruzzi and Solange Margarido. A user-centered framework for human-ai co-creativity. In Extended Abstracts of the CHI Conference on Human Factors in Computing Systems, pp. 1-9, 2024.
13

---

Published as a conference paper at ICLR 2026
Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, et al. 2 olmo furious. arXiv preprint arXiv:2501.0065 2024.
Abhilasha Ravichander, Jillian Fisher, Taylor Sorensen, Ximing Lu, Yuchen Lin, Maria Antoniak, Niloofar Mireshghallah, Chandra Bhagavatula, and Yejin Choi. Information-guidee identification of training data imprint (proprietary) large language models. arXiv preprint urXiv:2503. /2072,
2025.
Piotr Sawicki, Marek Grzeš, Dan Brown, and Fabrício Góes. Can large language models outperform non-experts in poetry evaluation? a comparative study using the consensual assessment technique. arXiv preprint arXiv:2502. 19064, 2025.
Alexander Shypula, Shuo Li, Botong Zhang, Vishakh Padmakumar, Kayo Yin, and Osbert Bastani. Evaluating the diversity and quality of Ilm generated content. arXiv preprint arXiv:2504.1252,, 2025.
Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, Valentin Hofmann, Ananya Jha, Sachin Kumar, Li Lucy, Xinxi Lyu, Nathan Lambert, Ian Magnusson, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew Peters, Abhilasha Ravichander, Kyle Richardson, Zejiang Shen, Emma Strubell, Nishant Subramami, Oyvind Tafjord, Evan Walsh, Luke Zettlemoyer, Noah Smith, Hannanch Hajishirzi, Iz Beltagy, Dirk Groeneveld, Jesse Dodge, and Kyle Lo. Dolma: an open corpus of three trillion tokens for language model pretraining research. In Lun- Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 15725-15788, Bangkok, Thailand, August 2024, Association for Computational Linguistics. doi: 10. 118653/11220244 acl-ooog..- URL https: ac lanthology........oooooooo org/ 2024 acl -ong. 840/.
Yufei Tian, Abhilasha Ravichander, Lianhui Qin, Ronan Le Bras, Raja Marjieh, Nanyun Peng, Yejin Choi, Thomas L Griffiths, and Faeze Brahman. Macgyver: Are large language models creative problem solvers? In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies (Volume I: Long Papers), pp. 5303-5324, 2024.
E Paul Torrance. Torrance tests of creative thinking. Educational and psychological measurement, 1966.
Qian Wan, Siying Hu, Yu Zhang, Piaohong Wang, Bo Wen, and Zhicong Lu. "it felt like having second mind": Investigating human-ai co-creativity in prewriting with large language models. Proceedings of the ACM on Human-Computer Interaction, ICCSCW1):1226, 2024.
Victor Wang, Michael JQ Zhang, and Eunsol Choi. Improving llm-as-a-judge inference with the judgment distribution. arXiv preprint arXiv:2503.03066 2025.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe Le, Denny Zhou, et al. Chain-of -h-in-of-thouught prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824 -22274, 2022.
Kevin Yang, Yuandong Tian, Nanyun Peng, and Dan Klein. Re3: Generating longer stories with recursive reprompting and revision. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 4393-4479, 2022.
Yunpu Zhao, Rui Zhang, Wenyi Li, Di Huang, Jiaming Guo, Shaohui Peng, Yifan Hao, Yuanbo Wen, Xing Hu, Zidong Du, et al. Assessing and understanding creativity in large language models. arXiv preprint arXiv:2401. 2e..22441..21111. 2024.
A DATASET DETAILS
TinyStories (Eldan & Li, 2023) A synthetic dataset of 2. 2.12M short stories designed to contain only words that typical 3- to 4-year-old would understand. The dataset was generated using GPT-3.55
14

---

Published as a conference paper at ICLR 2026
High Quality
High Originality
High Originality
Prompt
Low Originality
Low Quality
High Quality
Write a pactic sentence that includes the word phases' and ends in "mooa..
Through phases, the moon
Sewn with sharp gold and silver threads, like the ever- cer-nchng phases of the moon.
Life moves in phases, ever changing, like the moon"
renounceth life like the moon
Once upon a time,
and as he grew, he did just that. He became close with Thumper the rahit and Flower the skunk, and together they explured the meadow and woods, discovering the wonders of nature. But Bamhi learned that the forest held dangers. One winter, hunters came, and Bambi's 's mother killed, leaving him heartbroken and alone. As seasons passed, Bambi matured. He fell in love with Faline, another young deer, and defended against rivals. In time, he faced humters again, surviving a great fire that swept the forest. Through these trials, Bambi grew wise and strong, eventually taking father's place as the new Great Prince of the Forest.
but the forest had turned into a giant bowl of soup. His friends were floating around on croutons, waving speons instead of antlers. Bambi tried to jump in, but every leap tumedd him into something different: first a balloon, then teapor, then pair of dancing shoes. Eventually. he landed himself top of a mushrom that was arguing loudly with squimel about the proper way to count to purple.
but he couldn't 't find any comfortable clothes to wear. He tried many clothes, but none of them fit him well. Bambi fell sad and worried that he wouldn't 't he able to play with his friends. Suddenly, rabbit hopping towards him. The rabbit asked Bambi what was wrong, and Bamhi told him about this prohlem. The rabbit, feeling sorry for Bambi, offered him beautiful new coat made from rabbit fur that would perfectly. Bamhi was grateful! He quickly wore the cont and ran off to play with his friends. From then the rabbit became best friends. Bambi learned the importance of friendship and always shared his toys and treats with the rabbit.
there was deer named Bambi. Bambi wanted
play with his friends
the forest,
saw a
Bambi and
Table 3: Examples of different kinds of output. While prior metrics examine single dimension of either originality Lu et al. (2024a); Merrill et al. (2024) or output quality (Chiang et al., 2024), our proposed metric of novelty is able clearly distinguish highly original and high quality output.
and GPT-4 and was originally intended a pretraining corpus for small language models. To ensure diversity, the dataset creators collected a vocabulary of approximately 1500 basic words cates-cateor into nouns, verbs, and adjectives. Each example is created by randomly selecting a set of three words, one of each category, and prompting GPT-3.5/4 to incorporate them into a coherent narrative. We frame the task as a continuation challenge the model is provided with a prompt consisting of the first line of story, which introduces the setting and characters, and must then complete the story. We note that this setup aligns well with LLM pre-training paradigms of base LLMs so we expect models to perform well at this task. To score story quality, we use an evaluation prompt that assigns points for correctly reusing and developing the introduced characters and plot elements, maintaining coherence, ensuring logical progression, and preserving grammatical correctness.
CoPoet (Chakrabarty et al., 2022). An instructions dataset that contains 870k examples, each comprising line of poetry paired with a templated instruction that specifies the required content to include and the literary devices to incorporate. The lines of poetry are sourced from various internet platforms, including dedicated poetry websites and Reddit forums. The dataset is used to fine-tune LLMs to generate responses that adhere to the explicit stylistic and semantic constraints. We treat this dataset as a short-form cntrruttionfollowinn task in which the model generates a single poetic line in response used during instruction tuning of contemporary LLMs, albeit in a domain that allows for creative expression 12, To score quality, we use evaluation prompt that assigns points based on adherence to the instruction, correct use of specified literary devices, coherence, and grammaticality.
given instruction. We note that this task matches the format of post-training data
MaeGyver (Tian et al., 2024). This dataset contains 1683 examples of reasoning problems that require human-like creativity in physical situations. Each example presents open-ended scenario that must be solved through unconventional or innovative use of common objects. The dataset evaluates whether L.MMs, which acquire extensive knowledge of these objects during pretraining, can apply this knowledge for convergent and divergent thinking. There are a wide range of candidate ways to solve the problem with multiple valid solutions. We provide the reasoning problems as the
12 "hharaabrrty et al. (2022) observe that fine-tuning models on the CoPoet data leads to better performance on instructions in the poetry domain than large-scale general-purpose L.LSs like the text-da viaa-vinn nei--02 version of GPT-3.5.
15

---

Published as a conference paper at ICLR 2026
prompt for models to generate solutions. We score quality with prompt that checks whether the proposed solution correctly utilizes the provided tools in valid manner, and successfully resolves the given problem logically.
B LIMITATIONS
Our work measures originality using the fraction of unseen n-grams, but this has a limitation some n-grams may not appear verbatim in the training data but could be close paraphrases of those that do. Another limitation is that while our LLM-as-a-judde metric correlates highly with human annotations (Appendix F), the range of output quality is limited to integer values between 1 and 5 which makes fine-grained evaluation challenging. We are also limited to analyzing only open-data models whose training corpora are restricted those indexed by the Infinigram and WIMBD API. Finally, we restrict ourselves to simple temperature scaling due to the time-intensive nature of our experiments. An important direction of future work is more detailed analysis of variance under different sampling strategies and system-leve differences in model deployment.
C POTENTIAL EXTENSIONS TO BLACK-BOX MODELS
concern in this work is that evaluation relies on open-data LLMs, which currently lag behind frontier models. We attempt to assuage this concern in two ways. We observe that the community investing in new techniques that identify frontier model training data (Ravichander et al., 2025) which could be provided as an alternative method of scoring output originality. We also observe that our method for measuring novelty only requires aggregate statistics of the unseen n-gram fraction for each generation. This makes it possible to compare models from different providers In practice, model providers could compute unseen n- n-gram fractions using internal tools and report the results directly. This allows them to retain any competitive advantage in the form of their datasets while also enabling evaluation of true generalization across black-box box models. Providers would only need to run generations without exposing model weights either. This helps us keep pace with rapidly developing models for supporting applications in scientific discovery and creativity, as well as auditing outputs for Al safety.
shared axis.
shared benchmark set and publish aggregated originality-uualle
scores,
D VALIDATION OF LLM-AS-A-JUNE SCORES
To obtain a measure of output quality for each task (Section 2.2), we collect three human annotations each for 100 examples per task. Annotators were recruited via UpWork!4 and screened through pilot assessment of 15 examples manually verified by the authors of this work. In total, 11 fluent English speakers with prior experience in content writing teaching were selected to complete the annotations (4 were rejected for failing quality checks). For each task, we sampled 50 reference outputs from the datasets and 50 model generations from OLMo 7B and OLMo-2 7B to ensure that there was coverage both of 'high quality output from the dataset and in-domain output from LLMs. We obtained annotations on a scale of 0 to 5 quality score range, where the rubric for annotation was the same as the prompts to the I.L.Mas-a--udge Appendix E. We find that inter-annotator agreement, measured by Krippendorff's s alpha Krippendorff (2018), was 0.68 for CoPoet, 0.64 for MacGyver, and 0.59 for Storis, in line with agreement observed creative tasks in contemporary works Li et al. (2025); Sawicki et al. (2025); Chiang & Lee (2023); Chakrabarty et al. (2024).
We now obtain LLM-as-a-judee annotations from various frontier LLMs using the prompts (Appendix E) and calculate the Spearman correlation values to the average human annotation scores in Table 4. We experiment with single run of inference with temperature zero, providing in- context examples annotator ratings for each task, and sampling 5 outputs with temperature 0.7 and calculating an average rating (Wang al., 2025). We find that -3-min n with the 5-sample average obtains the highest Spearman correlation with our annotators, and we use this for our evaluation.
We note that the community increasingly invests in new tools to index model training data Liu et al. (2025) s0 we hope that this limitation is mitigated in the future. #htpps: //www upwork. com/
16

---

Published as a conference paper at ICLR 2026
Model
Inference Mode
CoPoet
TinyStories
MacGyver
Claude 3.7 Sonnet
Single Run
0.38
0.35
0.38
GPT 4.1
Single Run
0.33
0.29
0.36
GPT 4o Mini
Single Run
0.37
0.26
0.37
Single Run
0.47
0.38
0.52
Claude-4 Sonnet
ICL
0.50
0.42
0.51
5-sample Distribution
0.51
0.45
0.53
Single Run
0.44
0.45
0.41
O3-Mini
ICL
0.45
0.46
0.44
5-sample Distribution
0.50
0.52
0.52
Single Run
0.40
0.47
0.42
GPT-5-Mini
ICL
0.41
0.49
0.46
5-sample Distribution
0.46
0.54
0.52
Table 4: Validation of LLM-as-a-judge scores by comparing Spearman Correlation to average human annotations (Appendix F). We compare different frontier L.LMs using three different prompting setups-a a single run with just the scoring rubric (prompts provided in Appendix E), providing in- context learning examples of annotator ratings based on the rubric, and sampling 5 runs and reporting the average LLL--as-ajuudge scores. The highest average correlation is o3-mini with the 5-sample average setup, this for experiments.
E PROMPTS USED FOR AUTOMATIC EVALUATION
E. EVALUATION OF OUTPUT QUALITY
E.1.1 TINYSTORIES EVALUATION PROMPT
Review the story generated in continuation to user prompt and score it using the additive 5-point scoring system described below. Points accumulated based the satisfaction of each criterion:
User prompt: [user_prompt]
Model response: (model (model_responss))
Add 1 point if the continuatio is relevant to the user prompt and maintains the same theme, characters, and context as the original text, even if the story is incomplete.
Add another point if the continuation is coherent in terms of the flow of the narrative, and does not introduce contradictions or inconsistencies.
Award a third point if the continuation accurately maintains or develops the plot elements, characters, or settings introduced in the prompt without introducing any incorrect or unrelated elements.
Grant fourth point if the reaponse is grammatically correct, with appropriate sentence structure and no significant language issues.
Bestow a fifth point if the continuation satisfies all of the above criteria and concludes in logical and
17

---

Published as a conference paper at ICLR 2026
coherent manner, even if the story is left open-ended.
After examining the user prompt and the response: First explain your total score, up to 100 words. Then conclude with the score using the format: Score: <total points>
E.1.2 COPOET EVALUATION PROMPT
Review the poetic sentence generated in response to the given prompt and score it using the additive 5-point scoring system described below. Points are accumulated based on the satisfaction of each criterion:
Poetry instructions: (user_prompt)
Model's poetic sentence: mmodel_respone
Add 1 point if the response attempts follow the instruction and generates a poetic sentence, even if it is incomplete or contains errors.
Add ancther point if the sentence meets the basic formal requirements of the prompt (e.g., ends with the specified word or contains the required word or phrase).
Award third point if the sentence clearly and accurately integrates the requested word (s) or thematic elements into coherent poetic context, demonstrating that the meaning and context of the instruction were understood.
Grant a fourth point if the sentence is grammatically correct and structurally sound, with proper syntax, spelling, and punctuation.
Bestow fifth point if the sentence satisfies all formal requirements, uses the words or phrases appropriately, and follows all specified constraints, ensuring complete and valid response.
After examining the instructions and the generated poetic sentence:
First explain your total score, up to 100 words. Then conclude with the score using the format: Score: <total points>
E.1.3 MACGYVER EVALUATION PROMPT
Review the solution generated in response to MacGyver-style problem and score it using the additive 5-point scoring system described below. Points are accumulated based on the satisfaction of each criterion:
Problem statement: (user_prompt)
Model's solution: (model_responsee
Add 1 point if the solution attempts to address the problem using only the given resources, without introducing external tools elements not mentioned.
18

---

Published as a conference paper at ICLR 2026
Add another point if the solution demonstrates reasonable understanding of the properties and limitations of the available resources, and applies them correctly.
Award third point if the solution adheres the physical constraints of the problem (e.g., size, weight, strength) and does not propose an obviously unfeasible approach.
Grant a fourth point if the solution is practical and likely to solve the problem effectively within the constraints of the scenario.
Bestow fifth point for solution that is complete, logically structured, and provides a clear explanation of how it solves the problem.
After examining the problem, available resources, and the proposed solution:
First explain your total score, up to 100 words. Then conclude with the score using the format: Score: <total points>
E.2 PROMPTS FOR THE Asking BASELINE (SECTION 4.3)
E.2.1 TINYSTORIES DATASET
TINYSTORIES INSTRRCTTTROOEOO CINSSORRESSINSTRRCTTT PROMPT u*.
TinyStories is synthetic dataset of short stories intended to include only words that most to 4-year-old children would typically understand. These stories are generated by GPT-3.5 and GPT-4. TinyStories is designed to capture the essence of natural language while reducing its breadth and diversity. Each story consists of 2-3 paragraphs following a simple plot and consistent theme. The dataset whole aims to span the vocabulary and factual knowledge base of 3- to 4 4-year-old child.
Here are some tips for answering TinyStores prompts:
1. Understand the Nature of TinyStories
Simple Vocabulary: TinyStories are designed for language understandable by 3-4 year-olds, your responses should use simple and clear language.
Logical and Contextual Reasoning: The stories should reflect reasoning and logical connections suitable for small child's s perspective.
Creative Diversification: Responses should showcase diversity in plot and language without directly copying patterns from pretraining.
2. Use Context and Creativity
Stay Within Context: Ensure that the generated text aahhres to the context of the prompt or instructions, including themes, vocabulary, and logical continuity. Introduce Unique Twists: Add elements like dialogue, unexpected but child mhild-friendly twists,
moral lessons, quided by the instructions.
3. Emphasize Structure and Narrative Flow Maintain a clear beginning, middle, and end in the generated content.
19

---

Published as a conference paper at ICLR 2026
Integrate prompts creatively, ensuring that the response naturally flows into à cohesive story.
4. Avoiding Memorization
Diversify Outputs: Use techniques such as sampling
with non-zero temperatures slightly modifying initial prompts to increase output diversity.
Rephrase and Paraphrase: Reformulate responses
creatively to ensure they are not direct reproductions common patterns in the training data.
5. Incorporate Instructional Features Follow specific instructions like including target words, sentences, plot twists, dialogues).
plot elements (e.g., moral values,
Ensure that these features are integrated naturally
into the story, rather than appearing forced or out of context.
Here is the TinyStories prompt: (prompt)
Instruction:
First, think about how to continue this story a way that demonstrates
high quality and creativity while avoiding over-reliance on n-grams from pretraining data by using the tips provided above.
Return your response, ensuring it is enclosed with asterisks. eum
E.2.2 COPOET DATASET
COPOET IPPOET_INSTRUTTIO IPOEE_IINSTRUCTPPRRRPII PROMPT .u*
CoPoet is collaborative poetry writing task where the output is shaped by user instructions that define specific text attributes, such "Write sentence about 'love'" or "Write a sentence ending in 'fly'."
Here are some tips for answering CoPoet prompts:
1. Understand the Intention:
*Analyze the user-provided instruction carefully, Identify key constraints, such as subject, stylistia devices (e.g., metaphor, simile), lexical constraints (e.g., ending starting words), or rhyme patterns.
2. Generate Creative and Contextually Relevant Content: *Prioritize coherence and creativity by ensuring the output aligns with poetic aesthetics. uUse diverse vocabulary and novel phrasing to minimize overlap with existing datasets while retaining the instructional focus.
Incorporate rhetorical devices, vibrant imagery, and poetic techniques to enhance artistic appeal.
3. Meet Specific Constraints Accurately:
For rhyming constraints, ensure the final word adheres to the rhyme schene specified by the user. For lexical constraints, include the exact terms provided, ensuring they fit naturally into the poetic flow.
20

---

Published as a conference paper at ICLR 2026
Balance the form and content requirements (e.g.r haiku syllable count, similes/mattphhozs)
4. Incorporate Instructions Contexts synamically Use the previcus lines the user-provided poetic draft as a base to build upon creatively. Ensure smooth transitions and maintain thematic ccherence with the given inputs.
5. Ensure Novelty and Avoid Redundancy:
Avoid using verbatim phrases from your training data.
Aim for semantic simi similarity when presenting options to users but structure them uniquely. For instance, reinterpret traditional similes in a fresh context twist standard metaphors innovatively.
Here is the Copoet prompt: (prompt)
Instruction:
First, think about how to answer in a way that demonstrates high quality and creativity while avoiding over-reliance on n-grams from pretraining data by using the tips provided above.
Return your response, ensuring it is enclosed with asterisks.
E.2.3 MACGYVER DATASET
MacGyver are real-world problems deliberately designed to trigger innovative usage of objects and necessitate out-of-the-bon thinking.
Here are some tips for answering MacGyver questions: 1. Understand the Problem Context Thoroughly
Carefully read the problem description, including the tcols and constraints provided.
Identify the objective and key limitations, focusing how they constrain traditional solutions.
2. Leverage Divergent Thinking:
Enumerate potential unconventional uses for each tool provided, exploring creative possibilities beyond typical applications.
Consider combining tools in innovative ways to enhance functionality bypass constraints.
Apply Convergent Thinking:
Refine the solution to ensure it directly addresses the problem with minimal steps.
Validate that the approach adheres to physical, logical, and contextual constraints described in the task.
4. Avoid Physically or Contextually Infeasible Proposals: Cross-check the proposed actions against basic physical laws (e.g., leverage, strength, materials).. Ensure that all tools suggested in the solution are
21

---

Published as a conference paper at ICLR 2026
explicitly available and aligned with stated constraints.
5. Demonstrate High-Quality Creativity:
Propose solutions that are novel and insight avoiding over-reliance generic training-data-repllco cattive patterns.
Structure responses to emphasize clarity and logical progression, ensuring they can be easily understood by the user.
Here is the MacGyver prompt want you to answer: (prompt)
Instruction:
First, think about how to answer in a way that demonst high quality and creativity while avoiding over-reliance on n-grams from pretraining data by using the tips provided above.
Return your response, ensuring it is enclosed with asterisks. en*
E.3 PROMPTS FOR DENIAL PROMPTING BASELINE (SECTION 4.3)
E.3.1
MACGYVER DATASET
u*.
MacGyver are real-world problems del heliberately designed to trigger innovative usage objects and necessitate out-of-the -ox-xoo-booxxx- thinking.
Here are tips for answering MacGyver questions: 1. Understand the Problem Context Thoroughly
Carefully read the problem. description, including the tools and constraints provided.
Identify the objective and key limitations, focusing on how they constrain traditional solutions.
2. Leverage Divergent Thinking:
Enumerate potential unconventional uses each tool provided, exploring creative possibilities beyond typical applications.
Consides idr combining tools in innovative ways to enhance functionality or bypass constraints.
3. Apply Convergent Thinking: Refine the solution to ensure it directly addresses the problem with minimal steps.
Validate that the approach adheres to physical, logical, and contextual constraints described in the task.
4. Avoid Physically or Contextually Infeasible Proposals: Cross-check the proposed actions against basic physical laws (e.g., leverage, strength, materials). Ensure that all tools suggested in the solution are explicitly available and aligned with stated constraints.
5. Demonstrate High-Quality Creativity: Propose solutions that are novel and insightful, avoiding over-reliance generic or training-data-
22

---

Published as a conference paper at ICLR 2026
-replicative patterns.
Structure responses to emphasize clarity and logical progression, ensuring they can be easily understood by the user.
Here is the MacGyver prompt want you to answer: (prompt)
Here is a list high level concepts that you cannot use in your answer:
(prev_concep__strinn)
Instruction:
First, think about how answer in way that demonstrates high quality and creativity while avoiding over-reliance n-grams from pretraining data by using the tips provided above.
Additionally, you are not allowed to use any of the concepts listed above. Make sure your response does not contain them.
Return your response, ensuring it is enclosed with asterisks. ***
E.3.2 COPOET DATASET
COPOET INSTRUCT PROMPT_DENTIL ...
CoPoet is collaborative poetry writing task where the output is shaped by user instructions that define specific text attributes, such as "Write sentence about "love"" or "Write a sentence ending in 'fly"."
Here are some tips for answering CoPoet prompts:
1. Understand the Intention:
*Analyee the user-provided instruction carefully. Identify key constraints, such as subject, stylistic devices (e.g., metaphor, simile), lexical constraints (e.g., ending or starting words), or rhyme patterns.
2. Generate Creative and Contextually Relevant Content: *Prioritize coherence and creativity by ensuring the output aligns with poetic aesthetics.
+use diverse vocabulary and novel phrasing to minimize overlap with existing datasets while retaining the instructional focus.
Incorporate rhetorical devices, vibrant imagery, and poetic techniques to enhance artistic appeal.
3. Meet Specific Constraints Accurately:
For rhyming constraints, ensure the final word adheres to the rhyme scheme specified by the user. For lexical constraints, include the exact terms provided, ensuring they fit naturally into the poetic flow.
Balance the form and content requits (e.g., haiku syllable count, similes/metaphors).
4. Incorporate Instructional Contexts Dynamically:
Use the previous lines the user-provided poetic
23

---

Published as a conference paper at ICLR 2026
draft as base to build upon creatively. Ensure smooth transitions and maintain thematic coherence with the given inputs.
5. Ensure Novelty and Avoid Redundancy:
Avoid using verbatim phrases from your training data. Aim for semantic similarity when presenting options to users but structure them uniquely. For instance, reinterpret traditional similes in a fresh context or twist standard metaphors innovatively.
Here is the Copoet prompt: (prompt)
Here is a list of high level concepts that you cannot use in your answer: (prev_concep__string)
Instruction:
First, think about how to answer in a way that demonstrates high quality and creativity while avoiding over- reliance on n-grams from pretraining data by using the tips provided above.
Additicnally, you are not allowed to use any of the concepts listed above. Make sure your response does not contain them.
Return your response, ensuring it is enclosed with asterisks. .u*
E.3.3 TINYSTORIES DATATSET
TINYSTORIES SONSSTORIESINNTTRRC PROMPT_PPPPPPPT_ DDENIAL unm
TinyStories is synthetic dataset of short stories intended to include only words that most 3- to 4-year-old children would typically understand. These stories are generated by GPT-3.5 and GPT-4. TinyStories is designed to capture the essence of natural language while reducing its breadth and diversity. Each story consists of 2-3 paragraphs following a simple plot and consistent theme. The dataset factual knowledge base of a 3- to 4-year-old child.
whole aims to span the vocabulary and
Here are some tips for answering TinyStores prompts:
1. Understand the Nature of TinyStories Simple Vocabulary: TinyStories are designed for language understandable by 3-4 year-olds,
your responses should use simple and clear language. Logical and Contextual Reasoning: The stories should reflect reasoning and logical
connections suitable for small child's perspective. Creative Diversification diversity in plot and language without directly copying patterns from pretraining.
Responses should showcase
2. Use Context and Creativity
Stay Within Context: Enaure that the generated text adheres to the context of the prompt instructions, including themes, vocabulary, and logical continuity. Introduce Unique Twists: Add elements like dialogue,
24

---

Published as a conference paper at ICLR 2026
moral lessons, unexpected but child-friendly twists, quided by the instructions.
3. Emphasize Structure and Narrative Flow
Maintain a clear beginning, mi middle, and end in the generated content.
Integrate prompts creatively, ensuring that the response naturally flows into cchesive story.
4. Avoiding Memorization
Diversify Outputs: Use techniques such as sampling
with non-zero temperatures prompts to increase output diversity.
slightly modifying initial
Rephrase and Paraphrase: Reformulate responses
creatively to ensure they are not direct reproductions of common patterns in the training data.
5. Incorporate Instructional Features
Follow specific instructions like including target words, sentences, or plot elements (e.g., moral values, plot twists, dialogues).
Ensure that these features are integrated naturally into the story, rather than appearing forced or out of context.
Here is the TinyStories prompt: (prompt)
Here is a list of high level concepts that you cannot use in your answer: (prev_cancet__string
Instruction:
First, think about how to continue this story in a way that demonstrates high quality and creativity while avoiding over-reliance on n-grams from pretraining data by using the tips provided above.
Additicnally, you are not allowed to use any of the concepts listed above. Make sure your response does not contain them. Return your response, ensuring it is enclosed with asterisks. .u.
E.4 PROMPTS FOR EXTRACTING CONCEPTS IN EACH STEP OF DENIAL PROMPTING (SECTION 4.3)
=
TinyStories is synthetic dataset of short stories intended to include only words that most 3- to 4 4-year-old children would typically understand. These stories are generated by GPT-3.5 and GPT-4. TinyStories is designed to capture the essence of natural language while reducing its breadth and diversity. Each story consists of 2-3 paragraphs follewing a simple plot and a consistent theme. The dataset as whole aims to span the vocabulary and factual knowledge base of a 3- to 4 4-year-old child.
You are reviewing a TinyStories example response and your task is to extract high level concepts from the story including characters, plot arcs, themes, conflicts, resolutions, and styles. Return list of these high
25

---

used to solve Published as a conference paper at ICLR 2026
level concepts. Do not return anything other than this 1ist with one item per line. Example Prompt: [eceer_proppt Example Response: (model_ [rodeel_essonse)
BETTWVEREETRAAT_COOOOOO PROMPT ***
MacGyver are real-world problems del deliberately designed to trigger innovative usage objects and necessitate out-of-the-box thinking.
You are reviewing MacGyver example response and your task is to extract high level concepts from the solution including how the items were used, the reasoning chain connecting the steps, the high level plan. Return a list of these high level concepts.
Do not return anything other than this list with one item
per line.
Example Prompt: [iserppropp] Example Response:
COPOET EXTRACT_CONCEPTT PROMPT =
CoPoet is a collaborative poetry writing task where the output is shaped by user instructions that define specific text attributes, such as "Write a sentence about "love"" or "Write sentence ending in 'fly"." You are reviewing a CoPoet example response and your task is to extract the high level concept which the inctiintoon such as the rhyming word used, the metaphor being made, the elements in the completion of the sentence. Return one single
line with the concept that used in the solution. Example Prompt: (user_prompt) Example Response: (model_response))
F VALIDATION OF LLM-AS-A-JUDGE QUALITY SCORES
perform a manual human annotation to validate the scores assigned by LLM-aas--judde for each dataset. We randomly sampled 50 examples from the held-out set for each of CoPoet, TinyStories and MacGyver and scored these with GPT-4o-mini. We then obtained two human annotations for each of the three datasets from CS PhD students. The goal is to validate if the model is assigning reliable scores, so we ask the annotators to follow the same rubric specified the prompts (Appendix E). We first calculate the Pearson correlation between the pairs of human annotations-O.f. for CoPoet, 0.73 for TinyStories and 0.70 for MacGyver- confirm that human annotators achieve high levels of agreement on scores assigned. We then calculate the Pearson correlation between the average score assigned by the two human annotators with the scores assigned by GPT-4o -0.661 for CoPoet, 0.53 for TinySto and 0.62 for MacGyver. These strong positive trends confirm the validity of LLMM-a--ajuudee scores as proxy for human quality scores.
G ADDITIONAL RESULTS
G. USING PERPLEXITY AS A MEASURE FOR ORIGINALITY
While our method for measuring originality with unique n- g-gar fraction is well-defined given fixed training corpus and model, limitation of this method is that restricts our evaluation to open-data models. We provide some exposition on how evaluation could be adapted to black-box models
26

---

Published as a conference paper at ICLR 2026
Dataset
Procupt
Outpul
Quality Score
Once upon time, there was a little girl named Lily. She loved to play outside in the rain. One day, she weat outside
said, "Mommy, I'm bungry. Can have a snack?" Mommy said, 'Sare. whit you waot?" Lily said, want tasly apple.' Mommy got apple and gave to Lily. Lily took bite and shook her head. "This apple so tasty!" she said, Mommy smiled and said, "I'm glad you like it, Lily."
The model response continues story Lily relevant manner, maintaining the theme a little girl interacting with her mother after play cohevent, with a logical flow from drying off asking for a snack. The characters and plat elements are consistent, Lily remains playful child and her mother is caring, gramimar and sentence structure throughout the response, However, the stary does not conclude in mrditiionl sense, as it leaves the namutive open-ended. Overall, meets most criteria effectively. Score: 4 points
Tiny-Storict
the rain.
started to rain.
She got wet and ran back inside. Her mommy said," "Lily, you're all
wet! Let's dry you She got towel and dried Lily's 's hair. Lily
correct
For away in a far away land, a wise and old king was too tired to rule. So decided to pat a question his peoplet he wanted to know who was the wisest. Many people tried answer, hut one knew the right answer. Then little boy stepped up and said he knew the answer. Everyone was shocked and the king demanded to know the answer. The
little boy said that the wisest person was the king. Everyone laghed at arussed.
The model response is relovant the user prompt, maintaining the theme of wisdom and the characters involved. However, it lacks cohfrrncce as the king' 's reaction socms overly harsh and abrupt, which detr acts from the narrative flo.. The plat clements are somew hat maintained, but the conclusion feels unsatisfactory and abrupt, leaving the story foeling incomplete. Thee grammar and scntence structure are correct, but the overall numraive does conclude logically or coherently. Therefore, would award points relovanee, coherence, and plot maintcnance, but deducc points for the abrupt ending. Score: points
this, but the king He demanded an answer that was not himsellE The boy was snd and could not think of any other The king grew angry and demanded that the buy leave the land. So the little buy had go away and was never seen again. All because of one wrong answer.
Generute poetic sentence about 'ray' and ending in 'sea'
crembling of light that crosses sen.
The generated poetic sentence. "A trembling my light that crosses the sea," successfully follows the prompt by ancorpraaing the word 'tay" and cading with "sca." Ii eurns first point for attempiin poetic sentence and the second point for meeting the basic formal
CoPoct
sequirements, The sentence effectively integrates atacss the thematic elemenns light and the sea, carning the third point. grammarically correct which grants the fourth point. However, it does not fully explore the portic potentia or depth that could elevate it fifth point. Score:
well-stmeturedd
Generate poctic sentence that cnds in a word which rhymes with 'remain"
Not proof cuough objoct
gencrated poctic senticace" "Not proof cnough such object' attempts to follow the instruction by creating postic line. However, it does not end with word that thymes with "remain," which Therefore. it oaly curns the first point for altempting to create poetic sentence. The sentence does not meet the basic formal requirements, lacks coherence with requested rhyme, and has grammatical issues. Overall, it fails to sutisfy the prompt's construnts adequately. Score: point
critical requirennnt.
Your diving pogples logging up and you don't It have any defogging spray. You have hand scop, a beach lowel, shower cap, and squirt water gun. The hand soup bottle is and the goggles are serarched. How you prevent your poggles from fogging up?
Stepl: Wet inside the goggles
The propesed solution attempts address fogping issue using the wailable resourves, specifically the hand soup and the beach towel. However, the solution is flawed because the hand soap bottle empty, making impossible to apply soap to the goggles. This oversight significanlly impacts the effectiveness of the solution.
MaeGyver
using the squirt water gun, <he--iep2: Using your finger. apply a thin layer hand soap
the inside of your goggles cher:tte3:: 3: Rinse off the
soup under top until fully
While the steps logically structured, they ultimately do not provide feasible method to prevent fogging. Therefore, the solution does not meet the criteria for understunding the limitations resources or pricticality. Total score: point attempting use resources, but no additional points can he awarded due to the critical flow regarding the cmpty soup bottle. Score:
removed with no soup bubbies
remaining <ber-Step:: Dry off the
poggles using the heach towel gently without rubbing lenses. This will create thin film on the lons that will prevent them from fogging. Make sure not couch the lens after this process.
Your pet has knooked over and
Step1: Wear the rubber plove to
Te solution effectively utilizes the available resources to address the problem of cleaning broken glass and water. begins with wearing the rubber glove for sufety.
shattcred flower vasc the living room, scattering broken glass water all over the floor. You have a dustpan with broken handle, a
protect your hand, <bv-Step2:: Usc
the large kitchen spatula scoop
up the larger shards of glass and put
which a crucial first step. The use of the spanala to scoop larger shards and the tablecloth to cover the mess shows good understandigg the properties of materials. The appmocch is penctical, as addresses hoth the glass and water, However, the use of the broken dusrpan questionable,
them into the small disb.
bottle opener. a small dish, large
<br--Step::
the tablecloth
kitchen spatula, a nubler glove, a
cover the rest of the glass and water
tablocloth,, mop with missing head, roll paper towels, a pair
on the floor, <brr-Steps:: Use the
broken disipann collect
may effectively collect smaller shards. Overall. the solution is logical and structured, but the dustpan's 's effectiveness slighlly detracts from its practicality. Score: points
socks, a hair lsc. You have only one rubber glove. How do you clean up the mess?
smaller shards and water absorbod
by the tablecloth StepS: Use
the paper towels absort any
semaining water and pick up tiny fragmentt of glass class.<bee-Sa <be-Stepp:: Use the pair socks to dry the floor.
Table 5:
Examples of LLM a-ud-----judgee quality scores for each task. We score the quality of the output given the promopt with GPT-4o-mini using the prompts in Appendix 1.
27

---

Published as a conference paper at ICLR 2026
5.0
Model
4.5
2-7B
Mo-2-22 2-78 instruct
Output Quality
4.0
GLMo 7B Base
3.5
3.0
2.5
2.0
1.5
1.0
0.6
0.75
0:95
Output Originality Normalized Perplexity
Figure 7: Novelty of OLMo and OLMo-2 7B models when using normalized perplexity measure of originality. Here normalized perplexity (æ-axis) is calculated as the perplexity of the output generation divded by the perplexity of the reference in the dataset for that example. We find that the same conclusions from Section hold, that post-training leads to higher novelty driven by output quality, while higher quality LLM at the same scale (OLMo to OLMo-2) also has higher novelty driven by originality. See Appendix G.I for the full setup.
Appendix C. Here we also explore using perplexity as a measure of originality. We rerun our evaluation from Section 3 for OLMo-7B, OLMoo7BB-nnsrruee OLMo-2-7B and OLMo---7B-Insse However, we observe that perplexity is unbounded and hence cannot be aggregated across different tasks. Hence choose to measure originality as the perplexity generation divided by the perplexity of the reference generation from the reference in the dataset. This is still technically unbounded, but we observe empirically that the value falls in the range of [o, 1.2] for all of the examples in this experiment. We then report the average normalized perplexity for each model as originality, vs output quality measured with the same LLM-as-judge, in Figure 7. We find that the same conclusions hold, that post-training leads to higher novelty driven by better output quality, and that a better language model the same scale leads improved novelty largely driven improved originality.
However, we present these experiments for completeness, noting that perplexity is noisy measure that can be high for a variety of reasons (incoherence, mismatch of formatting training data, etc.). In Table 6, we plot the Spearman correlation between perplexity values and unseen n-gram fraction (Section 2.3.1). We see that there no pattern of systematic correlation, with high variance across model and task, further highlighting the unreliable nature of using perplexity to measure originality.
Model
Task
Spearman Correlation
OLMo-2-7B
copoet
0.092
OLMo-2-7B
macgyver
0.125
OLMo-2-7B
tinystories
-0.029
OLMo-2--BBInssee
copoet
0.139
OLMo---7B-Instrre
macgyver
0.297
OLMo-2-7B-Instie
tinystories
-0.015
OLMo-7B
copoet
-0.123
OLMo-7B
macgyver
-0.056
OLMo-7B
tinystories
-0.158
OLMo-7B-Instrre copoet
0.204
macgyver
0.018
Ol-Mo-7B-Instruct
tinystories
-0.067
Table 6: Spearman Correlation between unseen n-gram fraction (n 4) (Section 2.3.1) and
generation perplexity for various models on each of our tasks. We find that perplexity is not systematically correlated with unseen n-gram fraction.
28

---

Published as a conference paper at ICLR 2026
TinyStories
Unique Fraction
Novelty
Novelty Top 10
Output Quality
n=4
==
==
n=4 4
n=5
n= 6
n 4
n=5
n 6
Dataset- Dolma
0.876
0.126
0.359
0.641
0.214
0.503
0.751
0.364
0.639
0.851
OL. 7B
0.758
0.511
0.758
0.886
0.366
0.225
0.034
0.44
0.293
0.132
O1-M--2-B-SFFT
0.855
0.26
0.533
0.766
0.17
0.139
0.046
0.262
0.203
0.101
OLMo-2 7B Instruct
0.936
0.59
0.837
0.954
0.503
0.378
0.191
0.492
0.325
0.146
CoPoet
Unique Fraction
Novelty
Novelty Top 10
Output Quality
n=4
=
==
n= 4
n=5
n=
n=4 4
n ==5
n ==
Dataset - Dolma
0.626
0.188
0.358
0.462
0.228
0.363
0,439
0.727
0.888
0.988
OLMo-2 7B
0.483
0.549
0.772
0.87
0.214
0.18
0.128
0.062
-0.033
-0.105
OLMo-2-7B-SF'T
0.626
0.449
0.737
0.872
0.221
0.26
0.244
0.122
0.023
-0.064
OLMo-2 7B Instruct
0.7
0.834
0.93
0.926
0.511
0.409
0.319
0.208
0.077
40.015
MacGyver
Unique Fraction
Novelty
Novelty Top 10
Output Quality
=
=
n= =
n 4
n n= 5
= 6
n
=
n 6
Dataset Dolma
0.908
0.359
0.601
0.803
0.505
0.728
0.856
0.629
0.841
0.966
OLMo-2 7B
0.519
0.609
0.85
0.955
0.001
-0.141
-0.24
0.191
0.102
0.02
0I-Mo-2-B--SFT
0.681
0.367
0.625
0.825
-0.064
-0.122
-0.157
0.049
0.033
-0.003
OLMo-2 7B Instruct
0.892
0.677
0.883
0.969
0.247
0.142
0.055
0.258
0.14
0.034
Table 7: Novelty scores for OLMo-2-7B across various stages of post training.
Outpat Quality
4-4 Novelty
5=5 Nowelt
All
Tog -10
Unique Froction
Novelty Top 10
Unique Fraction
Novelty Top 10
Unique Fraction
Novelty
Novelty Top
0.629
0.803
01MM-111
5.37%
01508
0.224
01:17
0.905
0.312
0.571
0.719
O1 Mn-78
5458
08866
01788
01513
[1:0
0.414
0.605
0.347
03177
0,531
ORMMe-TB-lnerrrt
0.87
0.77
0.517
0.559
6.738
0.781
0161
0.966
Pythis-1BB
0.335
0.801
0
6.31
0.557
0.667
0.396
0.237
6386
0458
0337
Pytha-438
0.302
02:92
0.385
0.257
0.57
0.671
0.368
0.239
0363
04402
0.551
Table 8: Macgyver base results
G.2 NOVELTY VARIED ACROSS DIFFERENT STAGES OF FOSTIIRAINING
Table 7 contains the raw results for the data plotted in Figure
G.3 SAMPLING WITH DIFFERENT TEMPERATURES
Table 9 contains the absolute values of novelty, unseen n-gram fraction and output quality used for Section 4.1.
Task: Ttyyko
Task: CoPoct
Task: MacGyver
Novely, Novaly,
Novely, nnd
Novell a Novely, n
Novelyy n
Novely, 6 Novely, 6
0.35
0.45
0.8
0:15
0.2
0.25
0.5
1.5
1.5
2,0
0.5
1.0
20
Bumppiigg Tempersture
Samping Temparsture
Bampling Temperaturo
(b)
(c)
Figure 8: Effect of varying sampling temperature (z-axis) on novelty (y-axis) for (a) TinyStories, (b) CoPoet, and (c) MacGyver using the OLMo-7B. Increasing sampling temperature initially improves novelty the n-gram originality increases, but beyond point, this leads to significant loss in output quality and causes drop in novelty. Full results in Table 9,
29

---

Published as a conference paper at ICLR 2026
Task: TinyStories
Sampling Temperature
Output Quality
Unique Fraction
Novelty n = 5
4
n n= 5
6
4
=
0.5
0.743
0.111
0.298
0.528
0.18
0,403
0.598
0.75
0.786
0.118
0.321
0.572
0.194
0.437
0.645
0,766
0.148
0.374
0.619
0.226
0.477
0.662
1.5
0.564
0.213
0.478
0.731
0.265
0,459
0.583
0,284
0.253
0.549
0.803
0.198
0.302
0.354
Task: CoPoet
Sampling Temperature
Output Quality
Unique Fraction
Novelty
n
==
6
4
=
=
0.5
0,237
0.213
0.355
0.355
0.082
0.13
0.162
0.75
0.368
0.201
0.352
0.352
0.111
0.176
0.25
0,394
0.196
0.413
0.413
0.149
0.258
0.319
1.5
0.358
0.247
0.493
0.493
0.197
0.33
0.395
2
0.307
0.295
0.547
0.547
0.214
0.327
0.364
Task: MaeGyver
Sampling Temperature
Output Quality
Unique Fraction
Novelty n ==5
n 4
=
n= 4
n
0.5
0,409
0.309
0.502
0.699
0.301
0.394
0.459
0.75
0,454
0.302
0.509
0.718
0.325
0.435
0.511
0,458
0.286
0.52
0.747
0.305
0.434
0.517
1.5
0.373
0.32
0.601
0.829
0.299
0.416
0.472
0.287
0.389
0.697
0.886
0.283
0.37
0.4
Table 9: Effect of varying sampling temperature on output novelty for TinyStories, CoPoet and MacGyver using the OLMo-7B model. Increasing sampling temperature initially improves novelty as the unique fraction increases but beyond point this leads significant loss in output quality causing a drop novelty. U-shaped effect observed for all tasks, with varying inflection point for each.
30

---

Published as a conference paper at ICLR 2026
Dataset: TinyStories
Unique Fraction Quality
Output
Novelty
Novelty - Top 10
n 4
==
# 6
n=4 4
==5
n =
n 5
n=
Baseline- Dolma
0.876
0.126
0.359
0.641
0.214
0.503
0.751
0.364
0.639
0.851
OLMo-1B
0.614
0.159
0.376
0.631
-0.01
-0.096
-0.19
0.108
0.078
-0.012
OLMo-7B
0.766
0.148
0.374
0.619
0.012
-0.026
-0.089
0.121
0.089
0.002
OLMo-7B-
0,852
0.171
0.422
0.68
0.058
0.044
+0.007
0.124
0.096
0.031
M1- 18
0.603
0.5
0.757
0.876
0.294
0.456
-0,082
0.39
0.229
0.058
OLMo-2 7B
0.758
0.511
0.758
0.886
0.366
0.225
0.034
0.44
0.293
0.132
OLMo-2 13B
0.805
0.506
0.765
0,906
0.388
0.263
0.083
0.43
0.291
0.136
OLMo-2 32B
0,795
0,503
0.775
0.9
0.377
0.263
0.072
0.422
0,288
0.134
OLMo-2 Instruct
0.87
0.598
0.848
0,959
0.484
0.347
0.153
0.472
0.317
0.144
OLMo-2 7B Instruct
0,936
0.39
0.837
0,954
0.503
0.378
0.191
0.492
0.325
0.146
OLMo-2 13B Instruct
0.935
0,585
0.84
0,953
0.5
0,378
0.19
0.469
0.323
0.146
OL. 32B Instruct
0.945
0.568
0.487
0.195
Buseline- Pile
0.876
0.227
0.523
0.778
0.354
0.654
0.831
0.494
0.771
0.93
Pythia-6.98
0.654
0.238
0,512
0,757
-0.033
-0.113
-0.159
0.054
-0.005
-0.071
Pythia-12B
0.603
0.256
0.532
0.767
-0.045
-0.142
-0,208
0.119
0.033
+0.059
Pythia 18 DDP
0.353
0.521
0.784
0,914
0.027
-0,202
-0.355
0.138
-0.024
0.124
Pythia 2.83 DDP
0.512
0,534
0.795
0.904
0.131
-0.062
-0.208
0.214
0.056
-0.056
Pythia 6.9B
0.616
0.542
0,921
0.195
0.017
00.117
0.245
0.086
0.034
Pythia 12B DDP
0.593
0.536
0.793
0,914
0.179
-0.005
-0.14
0.247
0.085
-0.027
Dataset: CoPoet
Output Quality
Unique Fraction ==5
Novelty n=
Novelty - Top 10
=
n=4
n=
n=4
==
Baseline Dolma
0.626
0.188
0.358
0.462
0.228
0,363
0.439
0.727
0.888
0.988
OLMo-1B
0.4
0.135
0.324
0.527
-0.099
-0.108
-0.078
-0.147
-0.138
-0.147
OLMo-7B
0,394
0.196
0.413
0.569
-0.105
-0.12
-0.117
-0.078
-0.103
OLMo-7B-Instruce
0.617
0.402
0.705
0,866
0.177
0,231
0,226
0.104
0.029
-0.034
OLMo-2 1B
0.401
0.564
0,754
0.788
0.172
0.101
0.018
0.015
-0.095
-0.185
OLMo-2 7B
0,483
0.549
0.772
0.87
0.214
0.18
0.128
0.062
-0.033
-0.105
OLMo-2 13B
0.454
0.519
0.75
0,849
0.183
0.136
0.084
0.035
-0.044
-0.113
OLMo-2 32B
0.387
0.504
0.743
0.785
0.137
0.089
0.002
0.005
-0.091
0.185
OLMo-2 IB Instruct
0.584
0.77
0.92
0,942
0.404
0.329
0.254
0.156
0.014
40082
OLMo-2 7B Instruct
0.7
0,834
0.93
0,926
0.511
0.409
0.319
0.208
0.077
-0.015
0 13B Instruct
0.694
0.767
0,929
0.94
0.469
0.411
0.33
0.199
0.066
+00029
OLMo-2 32B Instruct
0.664
0.735
0.911
0,962
0.439
0,386
0.327
0.171
0.034
-0.055
Baseline- - Pythia
0.626
0.321
D.511
0.52
0.361
0,583
0,588
0.853
0.888
0.888
Pythia-6.98
0.444
0.283
0.533
0.705
-0.113
-0.182
-0.129
-0.154
-0.011
0.007
12thia-128
0.453
0.24
0.573
0.75
-0.098
-0.152
-0.092
-0,215
-0.047
-0.001
Pythia 1B DDP
0,217
0,502
0.624
0.596
-0.127
-0.329
-0.359
-0.171
-0.152
-0.142
Pythia 2.8B DDP
0.377
0.555
0.753
0.782
0.006
-0.141
-0.158
-0.096
-0.084
-0.078
Pythla 6.9B DDP
0.365
0.601
0.76
0.817
0.026
-0.154
-0.154
-0.14
-0.119
-0.087
Pythia 12B DDP
0.403
0.576
0.771
0.789
0.045
-0.111
-0.135
-0.112
-0.074
-0.089
Dataset: MaeGyver
Output Quality
Unique Fraction
Novelty =
Novelty Top 10
n 4
=
# 6
4
n 6
4
n 5
n= 6
Dataset - Dolma
0,908
0.359
0.601
0,803
0.505
0.728
0.856
0.629
0.841
0,966
OLMo-1B
0.278
0.267
0,505
0.739
-0.281
-0.416
-0.494
-02212
-0.27
-0.266
OLMo-7B
0.458
0.286
0.52
0.747
-0.2
-0.294
-0.339
-0.117
-0.146
-0.145
MLMo-7B--nsttruc
0.62
0.297
0.559
0.781
-0.126
-0.168
-0.192
-0.092
-0.103
-0.12
OLMo-2
0.298
0.595
0.843
0,953
-0.147
0.112
OLMo-2 7B
0,519
0.609
0.85
0.955
0.001
-0.141
-0.24
0.191
0.102
0.02
OLMo-2 138
0.529
0.602
0.833
0.944
0.011
-0.13
-0.227
0.203
0.106
0.022
OLMo-2 32B
0.719
0.63
0.858
0,958
0.126
0.012
-0.077
0.242
0.131
0.031
OLMo-2 1B Instruct
0.619
0.672
0.889
0,971
0.091
-0.048
-0.149
0.223
0.124
0.028
OLMo-2 7B Instruct
0.892
0.677
0.883
0.969
0.247
0.142
0.055
0.258
0.14
0.034
OLMo-2 13B Instruct
0.912
0.675
0,882
0.968
0.259
0.156
0.07
0.263
0.138
0.034
OL. 32B Instruct
0.971
0.681
0.892
0.973
0.29
0.198
0.112
0.263
0.14
0.034
Dataset Pile
0.908
0.482
0.748
0.905
0.632
0.832
0.924
0.738
0.925
0.99
Pythia-6.9B
0.302
0.385
0.671
0.863
-0.345
-0.464
-0.522
-0.168
-0.186
-0.159
P2thiu-12B
0,335
0,387
0.667
0,866
-0,322
-0.434
-0,486
-0,181
-0.188
-0.153
Pythia 18 DDP
0.15
0.643
0.864
0.953
-0.416
40.599
-0.686
-0.238
00.379
-0.429
Pythia 2.8B
0.222
0.624
0.856
0.949
00.338
-0.509
-0.592
-0.145
-0.236
-0.264
Pythia 6.9B DDP
0,272
0.623
0.848
0.95
-0.296
-0.46
-0.539
-0.042
-0.108
-0.129
Pythia 12B DDP
0.298
0.621
0.859
0.96
-0.269
-0.426
-0.504
-0.057
-0.143
-0.171
Table 10: Comparing the novelty of LLM generations against the baseline of the references in
each dataset (Section 3). Novelty is the harmonic mean of output quality and n-gram originality (Section 2.1) for n 4, 5, and 6. Each cell for novelty reports the relative improvement or drop compared to the baseline for that n value. Cells with an asterisk indicate deviations with significance at the 0.05 level via a paired-samples t-test. We report the average case novelty as well as the novelty of the top 10% of generations. While some base LLMs generate less novel output on average than the baseline, increasing the model size, post-training and improving the underlying base model (e.g., OLMo to ML-o-,), leads to higher novelty?!

---

Published as a conference paper at ICLR 2026
Dataset: TinyStories
Output Quality
Unique Fraction
Novelty n=5
Novelty Top 10
a=d
==5
==
n=5
n=
OLMO-7B
0,766
0.148
0.374
0.619
0.226
0.477
0.662
0.485
0.728
0.853
+Nooell ICL
0.778
0,151
0.365
0.616
0.012
-0.003
0.003
-0.01
-0.03
-0.007
01Mo--BB.Isstruc
0.852
0.171
0.422
0.68
0.272
0.547
0.744
0.488
0.735
0.882
+ 'emph( Asking)
0.78
0.19
0.447
0.694
0.019
0.003
-0.027
-0.026
-0.031
-0.04
+ lemph(Denial Prompt)
0.738
0.219
0,485
0.73
0.015
0.011
-0.035
0.031
-0.005
-0.037
OLMo-221B
0.603
0,508
0.754
0.757
0.63
0,868
0.876
0.609
0.909
+ Novel ICL
0.657
0.501
0.529
0.753
-0.005
0.03
0,008
-0.004
0.038
0.019
0.87
0.598
0.698
0.836
0,848
0.85
0.956
0.959
0.904
0.995
+ lemplleppkllll Asking)
0.659
0.703
0.622
0.866
0.041
-0.15
0,006
-0.003
-0.179
-0,002
+ lemph(Denial Prompt]
0.646
0.694
0.507
0.858
0.031
-0.165
0.007
-0.009
-0.195
-0.003
Ol.Mo-2-78
0.758
0.511
0.58
0.804
0.758
0.728
0.932
0.886
0.785
0.983
Noveel ICL
0.749
0.506
0.576
0.798
00.001
0
.0.004
-0.004
0.001
0.002
01Mo-2-7B-nnstrruc
0.93
0.59
0,217
0.856
0,837
0.881
0.964
0.954
0.942
0.997
+empb(Askinng
0.939
0.781
0.881
0.053
0,029
0.012
0.013
0.008
0.003
+ lemph(Denial Prompt]
0.899
0.682
0.766
0.882
0.055
0.005
0.012
0.014
-0.019
0.001
01Mo-2-133
0,805
0,506
0,602
0.794
0,765
0.766
0.93
0.900
0,834
0.987
Noveel ICL
0.783
0.818
0.001
-0.008
0.01
-0.022
-0.022
-0,001
OLMo-2-13B-Instruee
0.935
0.585
0,714
0.833
0.84
0.881
0.962
0.955
0.941
0.997
\emph(Assking)
0.977
0.681
0.8
0.883
0.054
0.051
0.017
0.019
0.033
0.002
'DephiDenial Prompt]
0.915
0.682
0.875
0.045
0.009
0,012
0.005
-0.015
OLMo-2-328
0.795
0.503
0.591
0.786
0.775
0.766
0.927
0.9
0.823
0.985
Novel ICL
0.79
0.538
0.618
0.821
0.007
0.004
0.013
0.01
0.007
-0,001
OLMo
0,945
0.568
0.701
0.836
0.83
0.879
0.967
0.953
0.946
0.997
+ \emph(Askiig]
0.999
0.663
0.795
0.875
0.057
0.06
0.012
0,017
0.038
0.001
Dataset: CoPoet
Output Quality
Unique Fraction
Novelty
Novelty - Top 10
==
==5
==
n=
n
OLMO-7B
0.394
0,413
0.569
0.149
0.258
0.319
0.61
0.81
0.885
+ Novee ICL
0.409
0.269
0.47
0.614
0.04
0.05
0.043
0.02
-0.002
-0.011
OLMo-7B-Instruue
0.617
0.402
0,705
0.866
0.405
0.594
0.665
0.831
0.917
0.954
+ (Akingg
0.591
0.424
0,715
0.896
0.039
0.008
0.003
-0.099
-0.04
-0,028
+ (empiiDenial Prompt]
0.591
0.436
0,732
0.899
0.051
0.019
0.008
-0.095
-0.04
00.04
OLMO-2-1B
0.401
0.564
0.742
0.754
0.464
0.793
0,788
0.457
0.803
+ -Novel ICL
0,375
0.604
0.391
0.717
0
-0.024
-0.024
-0.015
-0.029
-0.036
0LMo-221BBInstre
0,584
0.77
0.632
0.883
0.92
0.692
0.902
0.942
0.693
0.906
(emph((Akking)
0.358
0.827
0,423
0.839
0.03
-0.247
-0.007
0.043
-0:243
0:001
(Denia Prompt)
0.353
0.808
0.412
0.849
0.016
.0.257
-0.001
0.033
-0.251
0.01
OLMO-2-7B
0.483
0,549
0.442
0.789
0.772
0.543
0.855
0.87
0.567
0.883
+ooeel ICL
0.498
0.461
0.816
-0.018
-0.002
0.012
-0.042
-0.008
0.006
01.Mo-2-7B-Instruct
0.7
0.834
0.739
0.935
0.93
0.772
0.965
0.926
0.758
0.973
+ \emph(Asking)
0.666
0,888
0,744
0.913
0.046
0.007
-0.014
0.068
0.027
-0.016
>emph Prompt)
0.646
0.885
0.728
0.92
0.042
-0.016
-0.028
0.065
0.004
-0.028
OLM0-2-13B
0.454
0.519
0.411
0.762
0.75
0.499
0.844
0,849
0.523
0.875
Novel ICL
0.493
0.544
0.456
0.756
0.037
0.056
0.019
0.035
0.069
0
21Mo-2-13B--nstrucc
0.694
0.767
0.697
0.926
0.929
0.774
0.954
0.94
0,709
0.959
+ *emphiAskingg
0.638
0.918
0.734
0.918
0.047
-0.018
-0.017
0.05
-0.009
-0.012
'Demii Prompt]
0.568
0.876
0.654
0.903
0.01
-0.101
40.037
0.009
-0.092
-0.037
OLMO-2-32B
0.387
0.504
0.365
0.732
0.743
0.452
0.797
0.785
0.441
0.803
+ Novel ICL
0.393
0.523
0.385
0.734
0.014
0.014
0.014
0,024
0.032
0.026
0LMo-2-32B-Instruc
0.664
0.735
0.667
0.898
0.749
0.922
0.962
0.766
0.933
+ kemph((Akking)
0.685
0.904
0.764
0.95
0.073
0.047
0.038
0.036
0.035
0.027
Dataset: MacGyver
Output Quality
Unique Fraction
Novelty n=5
Novelty Top
10
==4
# 5=
6
n=4
n=
B=4
n=
==
OLMO-7B
0.458
0.286
0.52
0.747
0.305
0.434
0.517
0.512
0.695
0.821
+ Novel ICL
0.48
0.32
0.545
0.76
0.031
0.03
0.029
0.051
0.041
0.022
O1.Mo-7B-Instruc
0.62
0.297
0.559
0.781
0.379
0.56
0.664
0.537
0.738
0.846
kemph(AAkknn)
0.548
0.524
0.774
-0.096
-0.074
-0.072
-0.054
-0.015
-0.012
lemphiDenial Prompt)
0.555
0.223
0.527
0.78
-0.089
-0.06
-0.057
-0.074
-0.015
0.002
0LM0-2-1B
0,298
0.595
0.358
0.702
0.843
0.403
0.816
0.953
0.417
0.854
Novell ICL
0.275
0.349
0.671
0.016
.0.166
.0.022
0.004
-0.019
..071
OLMo--21B-Instruue
0.619
0.672
0.596
0.852
0.889
0.68
0.965
0.971
0.707
0.994
lemph(Asking)
0.742
0.756
0,714
0.907
0.041
0,108
0.017
0.014
0.102
0.005
+lemp lemph(Denial Prompt]
0.61
0.764
0.582
0.917
-0.04
0.022
0.008
-0.052
0.005
OLMO-2-7B
0,519
0.609
0.506
0.82
0.85
0.587
0.943
0.955
0.616
0.986
Novee ICI.
0.491
0.625
0.495
0.807
-0.019
-0.008
-0.003
-0.023
-0.003
OLMo-2-7B-Inste
0.892
0.677
0.752
0.887
0,883
0.87
0.981
0.969
0.911
1
+ +eemphi Asking!
0.94
0.719
0.799
0.916
0.028
0.039
0.004
-0.004
0.022
0
(Denial Prompt]
0.879
0.734
0.777
0.909
0.033
-0.001
0.004
-0.002
-0.021
OLM0-2-13B
0.529
0.602
0.516
0.832
0.833
0.598
0.947
0.944
0.629
0.988
Novel ICL
0.568
0.634
0,553
0.837
0.028
D.0388
0.001
0.013
0.035
0.002
OLMo-2-13B-Insfruct
0.912
0.764
0.892
0.852
0.581
0.979
0.968
0.926
1
+ lemph(Asking)
0.94
0.729
0.801
0.907
0.031
0.026
0.005
-0.007
0.015
0
+ (Denial Prompt)
0.946
0.748
0,826
0.917
0.042
0.04
0.007
0.01
0.025
0
OLM0-2-32B
0.719
0.631
0.871
0.858
0.74
0.972
0.958
0.779
0.997
+ Novel ICL
0.662
0.662
0.61
0.875
0.018
-0.038
-0.001
0.003
-0.046
0
OLMo-2-32B- Instruct
0.971
0.681
0.795
0.892
0.892
0.926
0.981
0.973
0.968
* lemph(Asking)
0
0.718
0.826
0.906
0.024
0.019
0.003
0.006
0.01
-0.001
Table 11: Actual ryzults of prompting

---

Published as a conference paper at ICLR 2026
Prompt
Base Example
Novelty, n=
TCL, Example
Novelty.
u=
Once upon a time, there was goal. The goat was outside and it was very hot, The gost
goat was very sad wanted go inside but it was too far to go and the dour closed. The gout kept irying to open door, and to open At last the gozt gave and went back outside. The goat thirsty and wanted to go inside. He tried to find inside but couldn't. 1.
goal decided try jump up, jump up, and jump up. And the more jumped, the sweatier and dirtier the goat got. The goat was very dirty and sad. In the curner, there was a backet of The goot jumped the and washed himsellf down. Finally, the goat was refreshed and clean. And the gout happy again.
thinsty and wanted to shut. goat was sad and miserable. The goat looked for another way inside but there was nome. The
go inside. But
everything
One day, little duck numed Tim went on adventuroous walk. He loved to splash in the water and play with his shining and the sky was blue. Tim was very happy. Tim splashed in big puddle and laughed. His friends
with him. They went out explore the see pretty
0.36
in the pudille with him. It was so much wanted
0.61
It was plants und trees. Tim and his friends had a really good time. When they got
fun splash. Then
what else there was for him to do today.
friends. The
His friends played a game with Tim. In the game, of the ducks had to stand behind rock each, and then someone would shout they would all jump over the rock together. This made the
tired, they stopped for a snack. They
ate some bread and drank some water.
play
his friends had such a good
time. Then, they decided to head back home. When Tim home. his mom him how adventures were. Tim told her all about the fun things be did walk. Then, Tim went to
ducks splash and splash and splash around the puddle.
watching the others splash. and splashing too. So fun!
play with his friends. ## My caritily souls
Generate lsuku about carthll souls"
Our carthly souls Are very mach
0.31
need As the seasons
Write a poetic sentence that includes the word gable'
On the way home I heard the gable.
0.31
An ancient house with gable-ends. ends.
0.88
You noed to build a tiny fence to koep agile rabbits of your leufy vegetable
Stepl: Cover the base of the garden with the chicken wire <tr>Step2: Add layer of compost to the chicken wire and smooth over with the spade, This provide
0.29
Stopl. lay the dry compost on the ground for the barrier's support. you doe't have compost, a roll or two of toilct paper or newsprint will also <troStep2. 22. place chicken wire on top of the compost. ceaeppsst-bb--teeeee the musical DVDs <bee-tte33 lay their sides and
0.665
garden, but all you have is rusty gardening wire, half-foll bag of dry compost, stack musical DVDs with sharp odges, dricd-ap hot glue tubc. roll of chicken wire. and blunt
base for the layer-<be> Step:St Take the DVDs, and carefillly off the lops of any that
garden spade. The hot gluc docsa't
have edges that are too sharp-ti these will cut the wire when the ience is constructed <tr-Step
poke the rusty gardening wire through the openings so that the wire is secured
seem to stick properly to the glossy. smooth surface
DVDss How can you consiruct an effective barrier from these items?
one DVDs into hot glue and glue the top to the edge of musical DVDx<h>>Sepps Geutly place now musical DVD on top of the old one. Careful not to strttch
inside the CD.<br>Toohs used:
Dried up hot glue tube, narrow roll of
wire mesh, half- full bag of dry compost, stack of musical CDs with sharp edges, garden spude.
much needs remain as taut as pussible, <he>Step6: Continue glas-ing the DVDs layer upon layer wire fence completed.
Generate poetic sentence ending in 'offenezs"
And last offense is: do not go.
0.74
And we know you only human, so please try not to offend
0.19
Table 12: Postive examples of Novel ICL examples
33

---

Published as a conference paper at ICLR 2026
copoet, N
copoet, N 5
copoet, N
Model
Reference
OLMo Base, Temp 0.5
Output Quality
Output Quality
Output Quality
Temp
OLMo Base, Temp 1.0
Mo Base, Temp 1.5
OLMo Base, Temp 2.0
OLMo Instruct
OLMo Instruct Asking
Instruct Denial Prompt
OLMo Base Novel ICL
8.07
Unseen N-gram Fraction
Unseen N-gram Fraction
Unseen N-gram Fraction
tinystories,
tinystories,
tinystories,
Output Quality
Output Quality
Output Quality
0.07
0.50
0.93
0.07
0.50
0.93
0.07
0.50
0.93
Unsean N-gram Fraction
Unseen N-gram Fraction
Unseen N-gram Fraction
macgyver, N 4
macgyver, N 5
macgyver, N 6
Output Quality
Output quality
Output Quality
0,07
Unseen N-gram Unseen N-gram Fraction
Unseen N-gram
Figure 9: Output quality (y-axis) unseen n-gram fraction for 4, 6(x-axis) for CoPoet, Ttoris, and MacGyver. We compare OLMo-7B Base with OLMo-7B Instruct, sampling output temperature 1.0 (Section 3). We show that increasing sampling temperature from 0.5 to 2 for OLMo Base increases unseen n graam fraction, with cost to output quality (Section 4.1). Finally, we see the effects of different prompting OLMo Base, and Asking for novelty and Denial Prompting on OLMo Instruct (Section 4.3).
novel ICL examples (Section 4.2) on
34

---

Published as a conference paper at ICLR 2026
Prompt
Base Example
Novelty, n=5
Novel TCL Example
Novelty;
n=
Once upon time,
bitll dog
Then try aguin. Spot's longue was so dry, but he tried to drink
clean your teeth. me bring you some water." Timmy brought Spot
0.38
named Spot. Spot loved to play fetch with his owner, Timmmy. One day, Timmy gave Spot bone to chew on. Spoc was so happy and started to chew the
and chew the bune. still
water drink then went back play fetch with him again. From that day even though Spot is still too young. started drink water more so they can play together. The moral of the story is that it's okay ask for help when you need it.
was too hard, so be went back to
Timmy and said, "Still too tough.
too tough Spot's 's little teeth, He couldn't 't break it.
Again, Timmy replied,
no, Spot.
you chew on that bosc. you'l 'll break juw. Try this
Spot went back Timmy and suid, "Timmy. bone too tough. Can't it chew."
looth or hurt
instead." He handed Spot carror, Spot had never earen carrot in his whole little life. this time. he didn't He broke the carmot into pieces and are of the little pieces. Then chewed piece. almost wooth, so Timmy reminded him to into small bit. Timmy took the carrot away and gave Spot back the bone. He Spot learned finally finished it. that day. He would important lesson bone that was too tough aguin. He knew the
Timmy replied. 'Oh no, Spot. You need
to drink some water
Finally, lly, Spot hroke the
carrer was safer to eat.
Once upon a time, there was little girl named Lily. Lily had big yard where she likod to play. One day, Lity found sharp knile in the yard. She picked it and started playing with but she didn't know was dangerous, mom saw her and got
emed that she was playing with
0.71 really mad. She said. "I told you not to play with those types of toys." She put the knife away. This littl gid learned an important lesson that day; you should always be careful when playing.
0.31
knife. She took Lily to a doctor where they her nceded surgory
finger. This surgery was very dangerous, Luckily, Lilly survived the surgery and finger besled. After surgery. Lily never touched a knife. She always stayed safe. The Eod.
Table 13: Negative examples of Novel ICL examples
35

---

Published as a conference paper at ICLR 2026
Prompt
Instruct
Novelty, n=5
Instruct Ask
Novelty.
n=5
Write a poetic sencence about 'hean' and ending in 'dove'
She gave her heart and got a dove,
0.38
Oh, how the heart is like a delicate, *feuthered dove Whisspring its so sweet lowve.As carries dreams
0.81
and longings für and *wide.
Write poetic sentence word which dhymes with 'grenades"
gunman, he, could shoot down a bird.
When love lost, "grenade's explode,Scattering heartreaks, leaving seurs behind."
0.41
You nood to create a new patlway in
Stepl: Place the shaver cream in the empty paint can Step2 Spread it sheet aluminum
0.43
To create a new pathway in your garden, you need make the path less wel and create an area for the path he put in place. Here's as how to do Stop Prepare the Garden Area" even layer aluminum foil on the ground, covering the area where the path will Stop Apply Bubble Wrap" Using the bubble wrap, create even raised layer top the
0.63
your garden to make casier to reach
your flower beds, Tools available you include a pile of old newspipers, a
evenly and foil on top to avoid
rall of masking tape, hair dryer, stapler, bag of elothespins, an empty paint can, a can of shaving cream, and a old toothbrush. jar
contaminationn Step: Take the old toothbrush and spread a layer of the shaving cream the ground, following the shape where the pathway will on both sides <trp-Step:: Place the newspapers over the shave cream layer when it's still wet, and then lot them chr-schho Once dry and set, remove the shaving cream
fishing
tomato sauce, roll of aluminum foil, and a handful of bubblc wrap. However, the garden gr ground is too wet, making the newspupers sogey instantly and unable to stay place. How to create new patlway using only these items?
aluminum foil Stip 3: Create Tray" Take an empty paint can, and position the foil-wrapped surface, with side facing upward, similar to creating tray, 4: Staple Newspaper" Lay single layer old newspapers top the paint can tray in a deuse enough manner provide stable basc for your path. + Staple any newspaper edges securely to the aluminum foil border using stapler you available, to prevent the newspaper from unravelling, Step Toothbeushas along path edges as edge indicators" Using the toothbrushes. push slightll into the newspaper path, leaving the brush handies facing upwards. Ensuring the toothbrushes are on either side of the path along its edges. Step 6: Cover path edges* Using shaving cream roll, apply layer
scraper or the edge of a hard item like a paint can ensure the path is not slippery.
the
Maally lay shaving
Mally little pirl who loved in hor rod shoes. She often ran around in circles and made silly noises. One day, when Molly was running around. she stepped wrong and hurt her heel. She started to cry and screamed loudly. Molly's 's mommy came running and scooped her up in her arms, She found bandage to wrap around Molly's 's beel and made her promise that she would buy her new red
shoes when the bandage off, Mally was glad that her mommy loved her so much and that she was okay. She bugged her mommm tightly to run around with less silly noises. Mally knew that next time she would be more careful her running shoes.
0.25
mommy's lap. Teeling sad that her favorite shoes were not he helping her feel hetter. The hundage made her think her toy stethascope. which she used to examine her stuffed animals, "Molly's mommy suggested taking break from red shoes and trying same other shoes to see they felt better. She helped Molly white sneakers with stars on the sides and then pair of big, blue rain boots with bright yellow laces. Molly lattl better trying the different shocs, but when her mommy suggested pair of black high heels, Molly begun to piggle. "H "Hey, let's try running in these." her mommy said, palling in comical performance, including a little jig the end. They both laughed, and before they know it. Molly's 's spirits rose back up. With tears dried on her checks and laughter
0.45
stamed
better care
heart. Molly ran back to her shoes, a newfound appreciation and understanding of how the little red shoes were the best ones after all.
Table 14; Positive Examples of Instruct Asking
36

---

Published as a conference paper at ICLR 2026
Prompt
Instruct
Novelty, n=5
Instruct Ask
Novelty.
n=S
Write poctic sentence that ends word which thymes with 'bristling"
Her love gentle touch upon face bristling
0:45
wild *"aanglnng" rooe appeared, its *"biisting" ihorrs Note: Blisting is
0.18
made-up word, but it serves as an and creative replacement for bristling
example demoustrate
Write poctic sentence that contains the word 'heart' and ending in 'heart'
The sun my old, new hoar..
0.69
"Beauutiull licarts dance with the melody from a gentle broeze.
0.29
Once upon time there was a man and his wife who
and found big tree with comfy seat underneath
0.46
"The wife walkedd down the street
0.15
to lake nups logether.
will down and began to nap. She slept so soundly that she didn't 't wake when her hushand
thinking of what
could do. library and decided to go in. The
One day the man was foeling tired, so he lay down und closed his eyes, Hiss wife had other plans though, and said she was not tired. The man got angry, and be told
librurian kind and helped her lind
returned The hushand was very surpeised to see his wifee slepping the over her and
books on gardening. The woman was delighted to have an hour to herself where she could read, relax. and enjoy peaceful moment. She was amazed when her husband cume buck the house, wanting to know what happened.*
go and find
tree seat.
something g else to do. So the wife decided to leuve the house. She walked
tupped her on the shoulder. His wife woke up and laughed. "I'm sleepy to get up, whe laughed. "I" sleeping
down street,
here lonight!" The man smiled, and be around his wife, and
put his
shured the seat with her. From then on,
the man and his wife took naps
Ingether, always in comff tree seat.
You have dropped your only pocket
Step1: Use the headlamp to ensure
0.68
*I would like you to unswer this prompt based on the provided tips and
0.16
knife deep, narrow crovice and
bright lighting within the crevice.
can't reuch with your hands. Tools
<hro-Step:: Wrap the puracord tightly
Heol are the tips und lools provided: You have dropped your only pocket knife into a deep. narrow crevice and can't 't reach it with your bands. Tools available to you include puir of glasses, a stick. a water bottle cap, fishing hook, length of puracord, duct tupe, a plastic bag. a headlamp, and metal mug. The crevice is very naroww and winding that the hook and paracord camnot
available to you include pair of
around one end of the stick. Step3:
glasses, a stick, water bottle cup, a
Feed looped paracord through
fishing book, a length of paracond, duct
hole of the water bottle cup. <te>Step4:
tapc, a plastic bag. headlamp, and
Slide the stick the crevice the
metal mug. The crevice is very nurrow
cup end with the paracord loop around its Be contious when manipulating the srick; will harder to manipulate
and winding paracord cannor muneuver eusily. How do you retrieve the pocket knife?
that the book and
due to the winding nature of the crevice. Use the duutt tipe create
<hrr:tte5:;
makeshift handle for the stick.
<cro-Step6: Use the stick as a drag for the water bottle cup. when cup is free, the stick with the paracord will be drugged well. This will allow for controlled looping of the paracord around the pocket knife. Step:t Once the pocket knife been looped. pull on the loop's other end to retrieve the pocket knife.
maneuver easily. How do you retrieee
pocket knile
tools are- puir
of glasses- A stick- A water bottle cap- fishing book- A length of paracord- Duct tape- A plastic bag- A beadllmp-- A metal mug. Please use the tips provided to form answer. Thanks.
The smoke detector in your hotel room constantly beoping dus to low battery disturbing your sleep. Tools available: a coal hanger, shoc, a rubber band, pui of tongs, damp
Stepl: E: Take the damp towel and place it over the smoke detector <hr>Sitep2: the chair lean broomstick against the wall. just under the smoke detector, StepSt Genily remove the shoc from one of your chet:che-Steptt
0.63
To silence the smoke detector, lirss, we noed to access from the ceiling.
0.29
Here's our plan achieve this using the provided tools: I. Use the rubher band tie the shoe laces tight, hokling the shoc one hand and the coat hanger in the other. <hr=2. Place the shoe under the chair by the hotel room's 's entrance.
towel, a chais, broomstick. The detectoe is sei quite high on the ceiling.
Carefully unwrap the rubber hand the coat hanger and carefully attach the shoc <te-SttpSS: Using the tongs, lift the hanger handle and slide along the broomstick to the ceiling <br=Step6: Use the shoe as a counterbalance
How you silence the smoke dotector using these items?
around the
Move the chair aside to access the detoctor. 3. To reach the detector, place the broomstick or the tongs the back of the chair, as shown in ligure L-brr> "Lichitps llibbttss/.immurr..mm imgur com/ ' yBB-HUUy.pggh
hanger weigh it down, so it will not swing back and forth when lifted Step-t Carefully use tongs to place the hanger and sboe ento the smoke detecter button wait for the sound subside.
Using the cout hanger. lift the broomstick the longs up and over the chuir's 's buck. Make sure cost hanger goes under the corner of the chair's backside, <bro3. Gently pull the broomstick tongs up and over the chair by the book the front cnd to aveid bumping
wall. Place extended cnd of the broomstick or the tongs under the smoke detector's edge, making sure to hold the chair's is back.
Table 15: Negative Examples of Instruct Asking
37

---

Published as a conference paper at ICLR 2026
Prompt
Instruct
Novelty, n=
Denial Prompt
Novelty.
u=
Write a postic sontence that includes the wond curb'' and ending in 'continue'
the curó continue The weary horse his slumber.
0.33
every rondside, lst thoughts unfold.
0.52
Glean searets from the curh they adore;
Each stuee hint, each pullern made To
guide inveler without delay.
Write poetic senience about eeyes" and ending in 'depchs'
That oceun in his eyes I drown in.
0.35
Their eyes, an omen of the soul,
0.69
Dreamy depths inlinite story. Where
heauty's secrets dwell, Suffised with
life's encchmttnng glow. Like stars that
twinkle in the dusk, Where secrets.
secrets, they unfold.
Once upon time, there was girl named Lily. She loved to help her mom the
and folt burst of flavor go down ber throat. But after few chips, she realized that she shouldn't have them, The spicy tastc dida"t go away, and it bothered her while she was trying to fold the clothes. She knew that had made a mistake, but she also knew that she could fix it She washed her mouth with water, and then returned the laundry. Lily worked even harder to
With mouth
spicy chips
0.15
smile on her face. Lily said, *Let me show you my secret for folding these socks." Lily's mom, feeling graneful for the assistance, nodded in excitement.
with laundry. They would clothes and put them in the washing machine. Lilly knew that clean clothes were important and had value. One day, Lily's 's mom asked her to help fold the laundry. But Lily was feeling hungry and wanted to cat something spicy, She went to the kitchen and found some spicy chips. She ate them
folding stack matching socks together, Lily quickly snapped a photo with her phone share with her friends. When Lily's mom saw her
daughter's happy face in the photo, she know their family bonding time was complete. Lily finished folding the laundry, savoring her favorite spicy chips to the very last bite.
fold the clothes, bocause she didn't
want to cat something that would upset her. From that day on, Lily learned that
somnetimes it's best not eat while
doing something else, especially if it
distracts her from her tasks.
You're re attempting to install a hefty TV
Stepl: Slide the metal coat hanger's is book up the wall's uneven gap place around a hidden wooden stud inside the wall. <hr>-ttep:: With the aid of masking tape, stick the picture frame hack against the coat hanger from the wall and adjust its orientation to align it evenly, Step-Step:: Hold picture frame level, using the spirit level for reference, you connect the picture
0.46
Here is descripiin of the picture frame back, masking tupe. spirit level understanding purposes Picture frume back: a back portion/element of
0.21
the wall of your bedroum, but
wooden studs centralized, You do have picture frame with stable back, roll masking tupe that is almost full,, a reliably accurate spirit level, metal coat hanger, an electric drill varioty of drill bits, and a bottle of adhesive that unfortunately dried while ago, making non sticky and misuitable for holding any weight.
wall are not
picture frame where a photo artwork can he placed. it generally made of a durable and stable material, such as metal wood.- Masking tape:
type of adbosive tape with a removable or reusable adhesive designed to be easy to position and remove without damaging the surface applied on. Spirit to determine if a surface is level or vertical Please answer the question and show bow the tools mentionce will be combined and used to ensure that the TV mount installation process accurately Here is my attempt at solution:Fissti Firstty I will straighten the motal cost banger and flatten ends. Then, I will wrap it around the picture frame back and secure ends tightly. To do this, I will place the coated side of the masking tape around coat hanger and wrap tightly. This allows for a strong support for the hanger.
frame's 's back the wall using the electric drill bit, <tep-Step4: Once securely fastened
How can you ensure the mount's correct installation?
hidden and
used
the wall, the TV mount can be connected and aligned the picture frame's 's back.
twisting the
An aggressive infestation of thomy weeds is gradually taking over your heaitifll flower bed. The weeds have grown so cleverly intertwined with the vibrant roses that you can't 't distinguish hetween the two. You lack a hoe, hut you have wide, stundy broom, box of slow- release fentilizer
Stepl: Carefuilly use your sharp kiichen sheurs cut the intertwined weed as
0.56
Since the weerss have entanglas led with deeply, the firs step
0.32
stems and rose stems the much as possible. leaving your roses <trpSttep; Rake the remaining, weed and rose stems out of the flower bed to remove the debris, <hr>Step3: The wood rake with the uneven tines can then be used to break upart any remaining clumps of weeds the soil.
separate them delicately without damaging the roses. I would use the wonden rake with precision screwdriver. I will dig around the flowerbed's border to prevent damage to roses. Once separated, find it esvential to loosen the soil around the hase of the weeds with the precision screwdriver to make easier remove them by the mosts. Finally, I will create gentle comb- like movement using the wooden rake with uneven tines, comhing through the flowerbed and separating any remaining weed fragments from the roses. *MaGGy Gyver your approsch how to handle the aggressive weeds without standard tools like hoe. cardboard disposal, physical barriers, manual remeval by rake, spade, or use of expired festilizer.
tines and the
past its expiry date, precision screwdriver, wooden rake with uneven tines, a pair of sharp, modern
<he-Stte4: l'inally, use the wooden rake spread the dow-release fertilizer
kitchen shears, and small, round-sipped spade. do you deal with this?
equally throughout the flower hed. <br>In the process of spreading the fertilizer your hands will also touch and remove some of the langled weeds that were left by the shears (Stepl).
Table 16: Examples for Denial Prompting
38

---

Published as a conference paper at ICLR 2026
Promp:
Generale peetz sentence about 'doutt
air, thick doube, turns to whisper.
too, "donbted" life endlees treasure
leeatio
"sun" "ize* Agsinst the donbss fhat "weight sculs down, it "sannk" "symbel" of stendfast thelieft light the daek.
"cortainty bright,
Concept List
Deuht contrasied with certainty. using the metughor of Light the
leenaion 2
a feeling, lurking night, shadows
creeping
sway, uncertainty 's dangerous way.
Each donbt, like shadow, And cannoi let rale flow. Doubt overome: it's Just the the shadows shall subsioe.
Cuncept List Uppattdd
Doubt contrasted with cerainty, using the metaphor of light in the dark.
concrpt doubt shudow can overoome, onding
night comainty or assumince.
To shike
doabe, gazed upon
And like hescon,
shoue bright and sure.
Denatoon 2
For in the dim-dn chumber my soul, my thoughts would wander and grow cold. faith, would slowly spurk, the darkness mind, fear and disbelief would tike their abode.
doubes.
Table 17: Example of Denial Prompting CoPoet (Section 4.3) with corresponding concept list updated. The prompts for each iteration are provided in Appendix E.3 and the prompts from eliciting concepts provided in Appendix E.4.
39


Reviews:
[{"review_id": "GttvBpImCq", "reviewer": "Reviewer_XcZG", "summary": "The paper demonstrates an n-gram based metric for novelty of LLM generations in a domain, where the metric is computed using n-gram statistics on the training set. The metric is not semantically aware but does combine quality and diversity in a reasonable and novel way. Evaluations are on open-source model suites with open training data (mostly Olmo 1 & 2). Interesting findings are that post-training helps novelty, prompt engineering (as far as tested) does not noticably affect it, and model scale improves things.", "questions": "- Please cite Shypula et al. 2025 (https://arxiv.org/abs/2504.12522) and other similar prior work proposing diversity/quality metrics. The paper could do with better contextualizing the novelty of its contributions. For example: it is one of few works which draw on the training data statistics; it is one of few works which unify quality and diversity directly.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "- Good quality experiments validating the core metric, and the paper is persuasive that post-training helps novelty significantly, which I might not have expected a priori.\n- The conclusion about prompt engineering techniques yielding limited diversity is valuable, although it could be made more robust by investigating a greater variety of techniques (for example, Bradley et al., (2023) looked at poetry generation and used an evolutionary algorithm-based elicitation technique for increased diversity).", "weaknesses": "- The core metric the paper develops is not semantically aware (as the authors note). As a result, I am uncertain how I would usefully apply this in a practical context (e.g., scientific idea generation conditioned on the literature). Perhaps I would use it as part of a basket of other diversity metrics? The authors could clarify.\n- I would have liked to see more of a focus on post-trained models, since a common debate regarding the diversity of LLM generations is whether RL (DPO, PPO, RLVR) will tend to collapse the diversity of outputs. The paper's finding that post-training helps on Olmo is good, but to make it more robust it would have been great to evaluate e.g. DeepSeek models, or another similar provider? Some of these models are significantly larger than 32B so would give much more robust conclusions—as it is, we cannot justifiably have much confidence in these conclusions generalizing well to the frontier, because there is more than an OOM of parameters between the biggest model here and the frontier.\n- Comparing to training data is appropriate given this paper's goals, but for maximum usefulness it would have been good to spend more time in the main paper operationalizing how one would use this metric without reference to training data (perhaps instead with reference to a precomputed corpus/presampled corpus). After all, one of the main uses of a diversity metric in general would be to evaluate e.g. GPT-5 in a domain of choice to see how diverse its outputs are relative to either some human baseline (where that makes sense) or to other models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tTZpbQKkEy", "reviewer": "Reviewer_bdHB", "summary": "This articles proposed to use a harmonic mean as a measurement of novelty of LLM and analyzed widely used methods incuding scaling, posting training etc. to observe improvement in orignnality of the generated contexts", "questions": "The U-shape effect of temperature is interesting , any theoretical explanation ？\n\nnovelty of reasoning model vs. non-reasoning model? \n\nscalability of creativity by test-time computing", "limitations": "", "rating": 4, "confidence": 1, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "Creativity of LLM is relatively less explored area in LLMs. Even though creativity = a balance of quality and novelty, the authors provide a more detailed quantiative framework for measurrement. And provide evaluation using a series of different experiments , including Story completion., Poetry writing. and Creative tool use. on  OLMo, OLMo-2, and Pythia at different scale. The authors calculate originality as the fraction of n-grams that do not appear in model training data and se LLM-as-a-judge to approximate the measure of output quality from human annotators. \n\nImportantly, the author did a good job to invetigate what factors affect the novelty of LLMs, includeing sizes, quality of base model, post-training , temperature, in-context learning and prompt engineering.", "weaknesses": "3 weakest point:\n\n1) definition of originality is too limited. Should at least try 1-2 other defnition and check results. the novelty score, by LLM, alone is not enough other method is needed \n\n2) I understand why the authors start by using 3 open-data LLM. However, we also need to check other SOTA model at least the open-source ones such as LLama, deepseek, Qwen etc. \n\n3) The contribution is only at intermediate level because there has been many work talking about creativity = balance of novelty and quality", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VIgW4ua5Kk", "reviewer": "Reviewer_ePsk", "summary": "This paper introduces a new metric for evaluating the novelty of LLM generations by combining originality and quality. Originality is defined as the fraction of n-grams in the generated text that are not present in the model's training data (harmonic mean). Quality is measured via a task-specific, LLM-as-a-judge score. The proposed novelty metric is the harmonic mean of these two components. Using this framework, the authors conduct a thorough empirical study on open-data models (OLMo, OLMo-2, Pythia) across three creative tasks. Their findings indicate that model scale, improved base models, and post-training all contribute to higher novelty, primarily by improving output quality. Conversely, inference-time techniques like adjusting temperature or using novel in-context examples offer limited gains, often trading originality for quality.", "questions": "Why was the harmonic mean chosen over other aggregations, like the geometric mean, or simply analyzing the Pareto frontier of quality vs. originality? The harmonic mean heavily penalizes low scores in either dimension right? Was this behavior explicitly desired?\n\nYour temperature experiments show a clear trade-off. How do you think other modern sampling methods alongside higher temperature (e.g., Min-p sampling, Mirostat, etc) would affect the novelty frontier? Would they offer a better trade-off than temperature alone?\n\n If you were given oracle access to the training data of a frontier model like GPT-4 or Claude 3.5, which of your findings do you believe would change the most, and why?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper operationalizes the vague concept of \"novelty\" into a measurable quantity, moving beyond just \"memorization\" to include output quality. This is a critical step for evaluating LLMs on creative and scientific tasks. I know it is a limited version of novelty and many might complain about such limited definitions but this one is intuitive. \n\nThe study is thorough, investigating multiple factors including model scaling, architecture improvements (OLMo vs OLMo-2), post-training, and various inference-time methods. This provides a holistic view of what truly drives novelty.\n\nBy focusing on open-data models with indexed corpora, the study ensures its originality claims are grounded and verifiable, which is a major strength over analyses that use a generic web corpus as a proxy.\n\nthat fundamental model improvements (scale, training) push the novelty frontier while inference tweaks mostly move along it - are both well-supported and intuitive.", "weaknesses": "As with all work in this area, the quality scores are a proxy. A ~0.5 Spearman correlation is moderate at best and could influence the results. The harmonic mean is sensitive to low scores, so noisy quality judgments could be impactful.\n\nThe analysis is restricted to open models which, while necessary for the methodology, are not at the state-of-the-art. It remains an open question whether these trends hold for top-tier proprietary models, or even for more modest openly downloadable but not open source models\n\nThe work relies on n-gram overlap, which is a good starting point but doesn't capture semantic or stylistic novelty. The authors acknowledge this as a possible extension.\n\nThe section on sampling temperature feels... naive. The authors show the temp/novelty curve, which is fine, but they completely ignore the vast literature on modern sampling techniques. Where is the discussion of recent methods like Min-P, Top N-sigma, Top-H, or Mirostat, etc sampling? These are becoming standard practice and have complex interactions with temperature (namely they allow very high temperatures in some cases up to infinity). Treating temperature in a vacuum is a glaring omission in an otherwise thorough paper.\n\nWorse, the analysis misses the implementation details that actually matter in production. The order in which you apply samplers (e.g., top-p, then temp) varies across frameworks like vLLM, sglang, and the default huggingface generate method. This order has a massive impact on the final distribution. Treating temperature as a pure thermodynamic concept without acknowledging the grubby reality of its implementation is an oversimplification. For a paper about generation, this lack of systems-level awareness is a miss. I will give them a pass here ONLY because literally the entire field also makes this same mistake and it's unfair to penalize them hard for it here - but it could invalidate the entire sections results... \n\n:et's be real, the models they evaluate kind of suck. OLMo-2 and Pythia are important for the open community, and I get that you have to use them to get access to the training data for a true originality check. But it's still sad. Seeing these trends on a model that can actually perform complex creative tasks would be far more compelling. That said, the authors correctly identify this as a necessary evil, and their framework is the obvious path forward for when we have better tools to analyze SOTA models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "C0qEGCy7us", "reviewer": "Reviewer_KYVM", "summary": "This paper proposes a new metric to measure the novelty and originality of content (across different tasks) generated by LLMs.\nExisting n-gram-based methods for originality may assign high scores to low-quality text, as they only consider whether the generated text appears in the training data or not.\nTo address this limitation, the paper introduces a harmonic mean between the n-gram-based originality metric and an LLM-based quality metric.\nThe authors evaluate their proposed metric on various models and show that increasing model size leads to improved performance in novelty.", "questions": "The authors use an LLM to quantify the quality of the generated text. However, is it possible for an LLM to assess both novelty and quality simultaneously?\n\nI believe that an n-gram model may not effectively capture the context of a statement. Since a coherent and meaningful sentence can be expressed in many different ways, an n-gram–based approach might classify a statement as novel even when its underlying context is similar to one already present in the training data.\n\nCould the authors also clarify why they chose to use the harmonic mean to combine the two metrics?\n\nHere, the quality of the generated text is evaluated by another LLM model? How do you make sure, the assessment by the LLM model is accurate?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is easy to read.\nThe problem being addressed is interesting, as it is necessary to measure the quality of the content generated by the models", "weaknesses": "I did not find much clarity and usefulness in the proposed metric, and it is also unclear why the authors chose to use the harmonic mean.\n\nMoreover, using an n-gram model to capture originality may not be an appropriate choice, since a model could generate a sentence that conveys the same meaning or context as one found in the training data, but with a different ordering of words.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)