Evaluating Text Creativity across Diverse Domains: a Dataset and Large Language Model Evaluator
Authors: Qian Cao
Xiting Wang
Yuzhuo Yuan
Yahui Liu
Fang Luo
Ruihua Song
Venue: ICLR 2026 Poster (2026)
Keywords: creativity evaluation
text evaluation
Primary area: datasets and benchmarks
TL;DR: We propose a dataset for text creativity evaluation and an LLM-based evaluator that can assess text creativity in a pairwise comparison manner, which outperforms existing evaluation methods in alignment with human judges.

Abstract: Creativity evaluation remains a challenging frontier for large language models (LLMs). Current evaluations heavily rely on inefficient and costly human judgments, hindering progress in enhancing machine creativity. While automated methods exist, ranging from psychological testing to heuristic- or prompting-based approaches, they often lack generalizability or alignment with human judgment. To address these issues, in this paper, we propose a novel pairwise-comparison framework for assessing textual creativity, leveraging shared contextual instructions to improve evaluation consistency. We introduce CreataSet, a large-scale dataset with 100K+ human-level and 1M+ synthetic creative instruction-response pairs spanning diverse open-domain tasks. Through training on CreataSet, we develop an LLM-based evaluator named CrEval. CrEval demonstrates remarkable superiority over existing methods in alignment with human judgments. Experimental results underscore the indispensable significance of integrating both human-generated and synthetic data in training highly robust evaluators, and showcase the practical utility of CrEval in boosting the creativity of LLMs. We will release all data, code, and models publicly to support further research.

Published as a conference paper at ICLR 2026
EVALUATING TEXT CREATIVITY ACROSS DIVERSE DOMAINS: A DATASET AND LARGE LANGUAGE
CrEval
MODEL EVALUATOR
Qian Cao', Xiting Wang Yuzhuo Yuan', Yahui Liu', FFang Luo", Ruihua Song!
RRnmmn University of China, *Beijing Normal University, "Kuaishou Technology {caoqian4real, xitingwang. rsong) }
@ruc..du.cn, jo. @ mai..bn.. yahui.cvrs @ gmail.com, l wnw.ewbbu.eu..cc
Project Page: httpgs///zeeaalccccettttteeeeeee
io
ABSTRACT
Creativity evaluation remains challenging frontier for large language models (LLMs). Current evaluations heavily rely on inefficient and costly human judg- ments, hindering progress in enhancing machine creativity. While automated methods exist, ranging from psychological testing to heuristic- or prompting- based approaches, they often lack generalizability or alignment with human judg- ment. To address these issues, we propose novel pairwise-compant comparison frame- work for assessing textual creativity that leverages shared contextual instructions to improve evaluation consistency. We introduce CreataSet, large-scale dataset with 100K+ human-level and IM+ synthetic creative instruction-respone pairs spanning diverse open-domain tasks. Through training CreataSet, we develop an LLM-based evaluator named CrEval. CrEval demonstrates remarkable supe- riority over existing methods in alignment with human judgments. Experimental results underscore the indispensable significance of integrating both human and synthetic data to train highly robust evaluators, and showcase the practical utility of CrEval in boosting the creativity of LLMs.
INTRODUCTION
Creativity, defined as "ideas artifacts that new, surprising and valuable" Boden (2003). has long been defining trait of human intelligence and fueled the progress of modern civilization Guil- Ford (9677). As current large language models (LLMs) exhibit increasingly remarkable capabilities across diverse domains and downstream tasks, they have also shown the ability to perform tasks requiring creativity [Summers-Star Stay et al. 2023); Zhao et al. (202); Zhong et al. (2024); Wu et al. (2025b). Evaluating the creativity of LLMs not only sheds light on their applicability to critical cre- ative domains such as creative writing Chakrabarty et al. (2025); Marco et al. (2024), literature Bena & KKlltaa (019); Cao et al. y He et al. (2023) and other creative domains Naeini et al. (2023); Summers-Ster Stay et al. (2023); ; Tian et al. (2024). but also has the potential to reveal gaps between LLM and human capabilities, offering valuable insights for future improvements.
Although evaluating LLM creativity is increasingly important, current methods face limitations that restrict their broader applicability. First (cross-domain applicability), most current methods assess creativity within a single domain or constrained task, like problem- solving Naeini et al. (2023) ; Tian et al. (2024), humor Zhong et al. (2024), or simile generation He et al.) (2023), where creativity is just one of several assessed aspects. Unlike open-domain tasks, they are often entangled with other concepts, making it hard to isolate and generalize creativity itself to other domains, such as liter- ature. Second (granularity), most methods evaluate creativity at the model or subject level rather than at the level of individual responses Mednick & Halpem| (1968); Torrance (1966). While use- ful for comparing models, they struggle to distinguish which of two responses to the same prompt more creative Chakrabarty al. (2024): Zhao et al.) (2025). We refer to the latter as text-level creativity (or simply text creativity). It is especially valuable it highlights specific responses for improvement, providing more actionable insights than coarse model- or subject-level evaluations.
R Corresponding author.

---

Published as a conference paper at ICLR 2026
(a)
Comporing creativity for text pains
(b)
Comparing creativity for text pairs
without sharing context
with sharing context
Text
Text (Creative Response):
Instruction (context):
Your neighbor's alarm.
human agreement
Your neighbor's alarm.
human agreement
Text 2:
ICC=0.59
What else wakes you up besides an alarm?
Text (Ordinary Response):
1CC5
Invest in the Future.
Hard to judge X
Sunshine, birdsong, or your phone vibrating.
Easy to judge
Figure 1: An example of how formulate the problem of text creativity evaluation to better evaluate.
Third, (effective automation) automating cross-domain creativity evaluation reduces human effort and supports iterative improvement. L.MMs have shown effectiveness automatic evaluators, in areas such as helpfulness and coherence Hu et al. (2024b); Kim et ( (2024) et al. (2024). known as LLM-as-a-judee Gu et al. (2024); Li al. 2 Zheng et al. (202)). However, creativity eval- uation remains underexplored. While early attempls prompt LLMs to assess creativity |Summers-
Stay et al. 2023); Zhao
(2025), leveraging the gross-doman strengths of advanced models like Go Hurst et al. (2024). their judgments often suffer from unreliability Chakrabarty et al. (2024).
inconsistency Wang et al. (2024a), and high cost Chen et al. (2023).
This paper aims to address these issues by proposing a novel evaluation methodology for automated, cross-domain creativity assessment, including a cross-domain benchmark dataset labeled by 30 hu- man judges and effective LL--ased creativity evaluator. Developing this framework presents two key challenges. The first is how to facilitate consistent human labeling. We observe that without clear contextual guidance, human annotators may struggle to reach consistent judgments, since cre- ativity may be understood differently in different contexts. For example, as shown in Figure | (a), when three annotators independently rated 400 decontextualized text pairs, the agreement among them was only moderate (with an Intraclass Correlation Coefficient, i.e,, ICC, of 0.59). The sec- ond challenge is how to train reliable LLM evaluator given the scarcity of creative data. Data scarcity limits the ability of evaluators to generalize across diverse domains and their effectiveness. To address this, it is thus crucial to collect large-scale training data in a weakly supervised manner.
Our work resolves these two challenges by introducing a framework that generates multiple creative responses conditioned on the same context. On the one hand, this setup ensures -uuhttyalltyy human annotations of text creativity pairs. As shown in Figure when shared instruction was provided as a context, the agreement improved significantly (ICC increases to a good level of 0.75). On the other hand, by controlling the response generation process for the same context, we automatically generate large-scale pseudo labels for their creativity levels in a weakly supervised manner, which solves the data scarcity issue. Specifically, our contributions
follows:
We propose a context-aware, pairwise comparison- -bsed evaluation protocol for assessing text creativity. Using this protocol, we manually annotate a test set of over 3,000 samples to benchmark text creativity evaluators. Notably, even state-of-thh-an LLMs perform poorly on it compared to humans, underscoring key performance bottleneck in current evaluators. To support training, we further construct CreataSet, a large-scale dataset incorporating creative tasks in 87 domains, including over 1M instruction-resppnse pairs with varying weakly supervised creativity levels.
Building on CreataSet, we introduce CrEval, an LLM-based creativity evaluator. To the best of our knowledge, our work is among the first to evaluate creativity across multiple domains using pairwise assessments. CrÉval outperforms strong frontier models, e.g., GPT-4o by 18.7% % in agreement with human judges, and demonstrates strong domain generalization capabilities. We further show that CrEval can enhance LLM creativity, offering a practical approach to improve generative AI.
2 RELATED WORK
Creativity Evaluation Evaluating creativity has been a long-standing challenge Kim (2006); Acar
& (2019) Many proposed methods Gray et al. (219 Zhao et (2025 Beketayev & Runco (2066); Sun al. (2025) adopt frameworks targeting particular tasks, such the Remote Associates Test Mednick & Halpern (1968) or the Torrance Test of Creative Thinking (TTCT) Torrance 1966). which measures human divergent thinking through scoring ideas on fluency, originality,
2

---

Published as a conference paper at ICLR 2026
Type A: (I)
Instruction /:
Across-Domain Creative Dataset Initialization
What else wakes you up besides alarm?
Context-Aware Response Augmentation
Type 8: (R)
CreatoSet- Base
Type C: (I,
Response R:
Your neighbor's alarm.
(I.
Label Construction Mixed Strategies
Instruction /
Response
A MiniCPM-ali (Prompte,)
CrEval
weak supervision
human labeling
! Maybe the sounds of the morning.
Evaluator Training
MiniCPM- -al (Prompt,) Sunshine, birdsong, or your phone vibrating.
(IR2y, y
R2) y
Owenz 5-14 (Prompt,)
Passion for life and dreams.
CreatoSet- (train)
CreataSet-Exx (test)
2w1n (Prompte)) Your dreams- they get you up chasing them.
Figure The construction process of CreataSet and training process of CrEval.
flexibility, and elaboration (e.g., listing diverse uses for a paperclip). Subsequent adaptations have applied TTCT principles to creative writing (TTCW) Chakrabarty et al. (2024): Li et al.) (2025c)
evaluate LLMs such tasks Zhao et al. (2025) while other work problem-solve
creativity proxy Naeini et al. (2023); Tian et al.
(2024). However, these approaches are often narrow
in scope, focusing than the creativity of the generated text itself.
limited set of tasks and primarily assessing model's creative ability rather
Existing methods for evaluating textual creativity face significant limitations. Heuristic scoring He ett al. (2023), matching for unique n-grams on reference corpus (Creativity Index) Lu et al. (2025b), and calculating divergent semantic integration using BERT Devlin al. (2019) (DSI)Johnson (2023) are often constrained by their specific designs, reliance on static corpora, and limited gener- alizability. While prompting general-purpose LLMs (e.g., GPT-4) has become common SSmmmers- Stay et al. (2023): Zhao et al. (2025): Chakrabarty et al. (2024 2025), results are often unsatis- factory Olson et al. (2024); Chakrabarty et al. (2024, 2025): Lu et al. (2025a). Another work, LitBench Fein et al. (2025). has trained reward models on specialized preference data |Fein et al.
al.
2025). but their applicability remains confined to creative writing, lacking broader generalization to other domains. Consequently, reliable evaluation of textual creativity still depends heavily on costly and inefficient human judgment, such as the Consensual Assessment Technique (CAT) Baer & Mccool 2 Marco et al. 22024). which cannot provide automated feedback for improving models. To overcome these chal challenges, we propose a novel approach that leverages the LLM-as-a- judge paradigm for more efficient and accurate creativity assessment.
LLM-as-a-Judge In the area of automatic evaluation for text generation, recent advent of large language models (LLMs) has enabled the evaluation paradigms incorporate LLMs be more accurate and flexible Gao et al. (2025), known as LL.M-as-a-jude [Zeng et al. (2023); Gu et al.) (2024): Li et al. (2025a), capable assessing more diverse dimensions of text quality. Prior works focus more the evaluation of text attributes like relevance Liu et al. (2023): Abbasiantaeb et al. or overall excellence Dongfu ct al. (2024); Hu et al. (2024b), etc. Other works also explore how to adapt LLMs to evaluate specific domains such as code generation Tong & Zhang (2024); Wu et al. (025a) and dialogue generation Lin & Chen (2023): ZZhag et al. (2024). However, few of these works investigate how to evaluate text creativity, making it hard to assess and improve the creative aspects of text generation. In our work, we propose assess it in pairwise-comparison manner, and provide comprehensive study on leveraging LLMs to evaluate text creativity.
(2024) Liu al. ( (2024), helpfulness Kim al. (2024); et
(2024), (
3 METHODOLOGY
In this section, we employ a three-step process to construct our large-scale weakly supervised dataset CreataSet to support our evaluation protocol and train CrEval. First, in Across- Domss-Domain Creativity
3

---

Published as a conference paper at ICLR 2026
TypeA:
Type R:
TypeC:
ExistingCreative Creative Data
Creativity Texts
Ordinary Instruction Ressponnneeeeeeeeeeeee Pairs
(Data Source: Oogirl-GO, Ruozhibe)
(Data Source: Short Texts, Lyrics, Ancient Poetry, Modem Poetry, Prose)
(Data Source: Infinity- Instiyct
Instruction: Create a catchy title for the following topic: saving money for the future.
Instructions W hat will be
Response: Sunflowers I cry; even when down, they face the sun.
humanity's 's final question?
Responser hat does this button do?
Constructed Instruction:
Original Response: The Secret to
Please share your thoughts on
Future W wealth: How to Save to A chieve Y Your Dreams.
Synthetic Responses:
maintaining a positive attitude.
Can't answer. 14
Synthetic Responses:
Enhanced Response:
2
Humanity's 's ultimate B4
M Maintaining positive mindset and an optimistic c atti tude helps us live more pmactively. (x)
Time Investment: W Weaving the
question: the meaning of life.
Blueprint for Future Wealth.
What is the meaning of life?
Synthetic Resporeess
Have ever truly
② positive attitude makes you stronger. and persistence leads to success, B4
Savings for the Future: Y our Smart
wnderstood love?
Investment Choice. B
@ positive attitude brings energy and helps overcome chall lnnges y positive attitude is light that illuminates the path ahead: persistence keeps it shining, sp
2
Invest in the Future. Save for Wealth.
MiniCPM-2B
Prompto
8 Smart B Budgeting, Seving for a
Qwena.5-14B
Prompt
Golden Future. 86
GPT-40
Time Bank: Saving for the Future with Wise Investment.
Figure 3:
The examples of three different types of data. The original data are above the dashed line, while our constructed components are below.
Dataset Initialization, we gather initial data in 87 diverse domains with varying lengths, generat- ing corresponding instructions to create initial instructionrrssponee pairs (I, R). Second, Context- Aware Response Augmentation expands these pairs by generating responses of varying creative levels for the same instruction I. Finally, in Label Construction with Mixed Strategy, we pair the responses and assign a label y, yielding training samples of the form (I, Ri R2, y) for the creativity evaluator CrEval. For meta-evaluation, we manually annotate test set benchmark CrEval against other evaluators. The overall data construction pipeline is illustrated in Figure 2
3.1 ACROSS-DOMAIN CREATIVITY DATASET INITIALIZATION
To build a creativity dataset across diverse domains, we gather initial data with varying creativity levels from eight sources. We unified them into a consistent (1, R) format.
Multi-Domain Multi-Source Data Collection We aim collect data from diverse sources and domains to construct a broad distribution in both domain coverage and response length, thereby en- abling the model to generalize across wide range of scenarios. Specifically, we begin by collecting data from existing creativity datasets, such as Dogini-GOZZoong et al. (2224) and Ruozhiba Bai et al. 2025), which naturally contain creative (J. R) pairs in the humor domain (Type in Figure 3). We further incorporate creativity-dense texts (R) from corpora of human creative works, such as po- ctry, lyrics, and prose, sourced from well-known w To enhance length diversity, we curate sub-dataset called Short Texts, comprising inspiring and thought-provokiee from online source?? Most of these entries consist of standalone texts (R) without explicit input prompts (Type B in Figure3). In addition, aiming to capture data with diverse creativity levels and expand domain coverage, we leverage an existing instruction-tuning dataset Infinity-IIssee Instruct et al. (025b), given its high-quality (T, R) pairs spanning wide range of domains (Type C in Figure 3).
sentences collected
Unified Instruction-Response Standardization To standardize the multi-source data into a uni- fied (I, R) format, first enrich standalone texts by generating missing instructions. We train an instruction generator by reversing an instruction-tuning dataset (Infinity-nntrruct). The gener- ator learns produce an instruction given response R. We generate an instruction for each standalone text, thus forming (I, R) pairs. To prevent non-creative data from obscuring creative data, we followed previous work Ritchie (2007) and applied some filters. All data are ultimately
prose http://hhhhhhoommmmsss//ep/////s https:/wwwwjjuiiknn..com
https://music. http:://music1133.oomm 163.com httpps:/mmmsskkk.coom
4

---

Published as a conference paper at ICLR 2026
Datasct
Cross-domain
Granularity
Auto-Evaluator Total Words # Samples Train/Test
Oorigi-GO Zhong et al. 2024
(humor)
Subject Level
894.712
15.797
train & test
MacGyver
(problem-solving]
Subject Level
249,385
1,833
DPT Jr. el al 2025
X (problem- solvingg)
Subject Level
X
12,576
803
test
TTCW Chikrabarty et al
2024
(creative writing)
Subject Level
58,426
48
test
Creative Writin V3 Paccit
(creative writing)
Subject Level
10,176
TTCT+ Zhao et al 2023
/ (7 domains)
Subject Level
X
700
test
LitBench Fell et al 2025
(creative writing)
Individual Text Level
16,309, 661
43,827
train & test
WritingBeneh Wi al 2025b
(100 domains)
Individual Text Level
1,875,146
1,000
CreataSel-B5ae Biise (ours)
/ (87 domains)
Individual Text Level
20,720,179
112,965
train & test
Table I: The statistics of different creative datasets. Auto-Evaluator denotes whether an automatic
evaluator is proposed based on this dataset. TTCT+ and training data for the evaluator in Writing- Bench are not publicly available. We calculate the total word count of the responses of each dataset.
formatted as (T, R) pairs (as shown in Figure 3 forming CreataSet-Base with over 113k creative samples. Due to the deeply contextual nature of creativity, which is highly subject to cultural and linguistic context, the dataset predominantly in Simplified Chinese. However, our framework is language-agnostic and can be easily extended to other languages.
Table compares CreataSet-Base with other creativity- related datasets, highlighting its larger scale. To assess domain diversity (i.e., thematic category), we followed prior works Tian et al. (2024); Wang et al. (2023): Jin al.) (2024) and started from manually curated seed taxonomy. We then adopted GPT-4o-mini to classify each data sample into a fine-grained category, yielding 87 distinct subdomains. They were then aggregated by the model into broader, semantically coherent ones, re- sulting in 17 core domains. The distribution of these domains is shown in Figure 4 Additional details on re- sponse length and semantic distributions provided in Appendix A.3
Perforning Arts
Dwip
Teclnology
Arts 16.2% %
Vauul
Arts
4.2%
Rusanov
Hazkhh &
Social
Witting
Lyncal
Wring Linccan
4.6%
Life Sciences
Leiure
Ancient
Entertait:
Poetry
Daily Life
ment 5.1%
6.0%
Monnstichi
Aesthetics
Modern
Education
Literature Poetry
38.3%
Career 8.2%
Humanibes
9.25%
Figure 4:
Domain distribution of
3.2 CONTEXT-AWARS RESPONSE AUGMENTATION
CreataSet-Bass Secondary domains for the top 5 primary ones are shown in gray.
Before constructing pairwise data (I, R1, R2) for train- ing the evaluator, we first augment the set of responses for each instruction, i.e. (T, Ri...., R₂). This aims to enrich the creative diversity, enabling the construction of pairs with creative differences. To effi- ciently construct such data at scale, we employ open-sourced models with different levels of capabil- ity, e.g., Qwen2.5- 14B-Instruct Yang et al. (2024) and MiniCPM-2D FFFSTT Hu et al. (2024a). to gen- erate responses for instructions in CreataSet- Base, as illustrated in Figure 2 For each model, we use two prompting modes to induce varying creativity levels: (1) P proppt.. general prompt that elicits ordinary responses; and (2) P Prompt,. a creativity-oriented prompt that encourages more imagina- tive outputs. By adopting different models/prompts, original responses in Type C data are direct answers to instructions with weak creativity. To enrich those, we further prompt GPT-4o to generate more creative ones to the same instructions. Finally, we name this dataset in the form (I, Ri...., R&) as Creatašet-BB Exx. The diversity analysis of aug- mented responses and the prompts used are in Appendix
generate multiple synthetic responses. The
(.pondixX.3..110 A.3.5 and Appendix A.6 respectively.
3.3 LABEL CONSTRUCTION WITH MIXED STRATEGIES
We combine responses into pairs R1, R2) and use mixed strategy to assign labels for training and testing separately, since the label requirements differ between them. Reliable -amantadd labels are essential for meta-evaluation to accurately assess model performance, while constructing labels in a large scale is more important for training. We detail them in the following.
To ensure diversity in the test set, we sample 50 instances from each data source in CreataSet, yielding 400 initial samples. These are further augmented using GPT-4o-mini with both prompts to enhance the distribution dif-
High-Quality Human Labeling for Test Benchmark Construction
5

---

Published as a conference paper at ICLR 2026
ference for evaluation. Following prior work Weinstein et al. (2022); ;; Johnson al. (023
recruited 30 qualified annotators from 18 different majors to rate response creativity on a 4 point Likert scale, with responses presented in randomized order (more details are in Appendix A.8] Each response's creativity score is computed as the average of all ratings. The annotations exhibit high inter-rater reliability (Intraclass Correlation Coefficient, ICC((K)=0.92). Finally, we construct a 3K test set in the format (I, R1, R2, y), where pairs with score differences > 0.3 are labeled as distinguishable, and those with differences 0.1 as comparable (tie).
Pseudo Labels for Training Set Construction sign weakly supervised pseudo-labels response pairs CreataSet-Bass enabling large-scale label construction. Our approach is based on two key assumptions: (1) stronger models tend to produce more creative responses than weaker ones, and (2) preativity-focusee prompts elicit more creative oulputs than ordinary prompts.
Weakly-Superrve
For the training set, we as-
To validate these assumptions, we sampled 150 data groups ((I, Ri,.... .. R&)) with 1,050 response pairs for each model/prompt combination and recruited 3 annotators to compare their creativity. The results show that creativity distinctions based on assumption (1) achieve 86.6% accuracy. and assumption (2) achieves 81,4%, confirming the reliability of both heuristics. For creatively com- parable samples (the tie cases), we randomly pair responses produced by the same models using propptt- Using these assumptions, we assign labels to response pairs in CreataSet-Ext, resulting in training data of the form (T, R1, R2,y).
3.4 CREVAL TRAINING
The constructed large-scale CreataSet-EE Extt can enable us to train CrEval. It provides triplets (I, R., R2) E DD as input, and trained to minimize the classification loss:
C=
Σ RO)ED
log PlylT, I, R1, R2).
where P(y]I, R1, R2) represents the probability of the label y given the triplet (L, Rg To mitigate the positional bias, we follow previous works Wang et al. (2024a b): Li al. (2024) by augmenting the data by swapping R and R2 in the input and adjusting the corresponding label. Additionally, we apply negative sampling by randomly selecting a response to serve as the least creative response, further enhancing the model's awareness of the instruction context
During inference, the model predicts whether R1 is more creative than R2, vice versa, or they are creatively comparable. Moreover, reference response R", generated by either a human a model, can be a baseline for comparing the creativity of another response It in such a comparison manner.
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUP
In our experiments, set k 5 in response augmentation. This is shared across all data sources. Based on our human-labeled test set of CreataSet, we adopt F7 score, Kappa score, and Agreement rate evaluate the performance of different methods, following previous work Wang al, (202bb); Li et al. (2024). All metrics are calculated twice by swapping the order of the two responses, and then the average scores are reported. Following Hu et al. (2024b), to eliminate the influence of sampling randomness, we set the temperature T to O for deterministic results, while other methods retain their original settings. We conduct pairwise comparison experiments on CreataSet where CrEval is compared with the following baselines:
Traditional Metrics:
(1) Perplexity (PPL): A simple baseline where we use Qwenn.55-B-IInstiee
to calculate the perplexity of response. Higher perplexity indicates higher novelty and creativity. (2) Divergent semantic integration (DSI) Johnson et al. (2023): adopts BERT Devlin et al. (019) to calculate the average semantic distance between all words in the response. A higher DSI indicates higher creativity. (3) Creativity Index Lu et al. (2025b): A corpus-based metric calculates creativity inversely to n-gram similarity with reference corpus.
Since use L.L backbones, the classification label is treated output conditioned on the prompt.

---

Published as a conference paper at ICLR 2026
Average
Method
S.T.
Lyr.
A.P.
M.P.
Pro.
Oog.
Ruo.
Inf.
F1
Kappa
Agree.
Truditional Metriex
PPI.
0.245
0.316
0.349
0.329
0.374
-0.022
DSI
0,440
0.430
0.354
0.527
0.377
0.578
0,561
0.522
0,480
0,175
0.457
Creativity Index
0.695
0.368
0.338
0.417
0.592
0.585
0.566
0.640
0.531
0.231
Frontier LLMs
03
0,802
0,589
0.596
0.667
0.663
0.774
0.832
0.769
0.721
0,578
0.725
o1
0.807
0.573
0.629
0.670
0.672
0.738
0.790
0.798
0.720
0.563
0.664
GP'T-lo
0.800
0.605
0.641
0.699
0.667
0.749
0.633
0.789
0.703
0,519
0.642
G.5
0.686
0.486
0.425
0.548
0.489
0.667
0.567
0.743
0.585
0.350
0.522
DeepSock-R1
0.743
0,479
0.494
0.578
0.612
0.751
0,745
0.733
0.653
0.457
0.547
DeepSeek V3
0.780
0.584
0.584
0.681
0.684
0.765
0.774
0.784
0.714
0.558
0.668
Claude-3.5 Sonnntt
0.775
0,603
0.634
0.671
0.702
0.762
0,850
0.810
0.727
0.609
0.740
Claude 3.5. Haiku
0.748
0.573
0.509
0,633
0.652
0.724
0.695
0.779
0.669
0.496
0.641
Gemini-2.5.PP Pro
0.764
0,569
0.585
0.639
0.656
0.760
0.866
0.752
0.708
0.557
0.702
Gemini 2.5 Flash
0.785
0.588
0.642
0,670
0.692
0.761
0.858
0.797
0.731
0.582
0.682
G-Eval (GPT-40)
0.772
0,583
0.568
0.665
0.694
0.759
0.803
0.793
0.712
0.558
0.677
G-Eval (GPT-3.5)
0.636
0,494
0,460
0.561
0.493
0.608
0,575
0.774
0.582
0,339
0.500
7B Scale LLMs
Semma-2299B-4
0.795
0.562
0.619
0.654
0.646
0.751
0.779
0.788
0.544
0.654
LLaMA3. 1-8B Instruct
0.713
0,548
0.440
0.618
0.615
0.649
0,573
0.782
0.621
0.418
0.565
PundaLM-7B
0.435
0.346
0.540
0.506
0.129
Prumetheus-7B
0,330
0.365
0.326
0.315
0.342
0.369
0.449
0.498
0.377
0,097
0.352
AUTO-J
0.659
0.526
0.377
0,561
0.541
0.565
0.730
0.567
0.323
WrilingBench Critic
0.715
0,528
0.500
0.626
0,548
0.600
0.641
0.712
0.612
0,362
0.576
Qwen2.5.77h
0.710
0.494
0.426
0.578
0.487
0.647
0.704
0.403
CrEval-7B (ours) A (v:s. base model)
0.779
0,556
0.649
0,681
0.665
0,778
0.873
0.820
0.732
0.601
0,745
7%
+12. 6%
+52. 3%
+36.
+.0.. 2%
+24.
6. 4%
2.
+49.
+29. 8%
138 Soale and Lorger I.IMs
Qwen2.5 S. 72B Instruct
0.751
0,558
0.520
0.655
0.594
0.734
0.833
0.806
0.692
0.535
0.673
1LMMA.1-700-Inmst 3-mma-3-27B-ii
0.736
0.564
0.559
0.642
0.624
0.732
0.764
0.810
0.684
0.535
0.675
0.792
0.572
0.608
0.650
0.666
0.753
0.789
0.783
0.706
0,564
0.702
Gemma 3-
0
0.542
0.575
0.615
0.674
0.729
0.667
0.772
0.672
0.498
0.633
Prometheus-11 13B
0.445
0.377
0.329
0.372
0.410
0.386
0,367
0.641
0.416
0.095
0.400
Qwen2.5- 148-tnstt
0.742
0.568
0.523
0.649
0.629
0.717
0.783
0.797
0.683
0,523
0.661
CrEval-14B (ours)
0.786
0,556
0.650
0.680
0.672
0.797
0,882
0.810
0.735
0.613
0.762
A (v.s. base model)
-5.9%
-2. 1%
+24.
3%
+4. 6%
+6.9%
+12.
6%
+1, 6%
+7.
+17.2%
+15. 3%
Table 2: Results of different methods our CreataSet test set. Best results in the same group are highlighted in bold, and the second-best are underlined. S.T., Lyr., A.P., M.P., Pro., Oog., Ruo., and Inf. represent Short Texts, Lyrics, Ancient Poetry, Modern Poetry, Prose, Oogiri-Go, Ruozhiba, and Infinity-Instruc respectively. We gray out the results of frontier LLMs due to their larger sizes.
Several evaluation-centro models including prompting- based G- Eval Liu et al. (2023) and fine-tuned LLMs PandaLM [Wang et al. (2024b). Prometheus Kim et al. 2024), AUTO-J aieral 2024) and Krittingennhh-ritte Wu et al. (2025b).
Evaluation-Centric Models:
General-purpose LLMs as Evaluators: We compare CrEval against several general-purpose Dubey et al. (2024). Gemma--2-99,3-113
ILMss. including LLa.A3.11 (IaVAA31-8.70)
2-B Rivière et al. 2024); Kamath et al. (2025). Qwen2.5-
B-4,72]B-Inntruul Yang et al.
(2024). GPT-3.5-Turib 1106 OpenAl (2022)
GPT-4o Hurst et al. (2024). OpenAl o1/o3 Jaech
et al. (2024): OpenAl (2025). DeepSeek-1 [V3,R1} DeepSeek-AT et al. (2024 2025), Claude-3.5-
[Haiku, Sonnet Anthropic (2024a)b) b), and Deemini-25-(Flaasll Pro Comanici et al. 2025). We
evaluate all models using the same prompt as for CrEval.
4.2 How WELL CAN CREVAL SIMULATE HUMAN EVALUATION?
As shown in Table 2 CrEval demonstrates consistent and significant improvements over all base- lines across the evaluated metrics. Notably, the 14B variant even surpasses most frontier baselines, improving F1 by 2.9%, Kappa by 9.7%, and agreement rate by 12.6% compared to strong com- petitors like DeepSeek-V3. These results validate the effectiveness of our approach for simulating human creativity assessment. Second, traditional metrics such as PPL and DSI perform poorly; e,g. PPL yields a Kappa score near zero, indicating their weak correlation with human judgment. The Creativity Index metric improves them but remains limited by its reliance n-gram matching and fails to capture the semantic creativity conveyed through conventional lexical choices. Third, while Gemma models achieve the highest F1 scores of the according groups on Short Texts and
7

---

Published as a conference paper at ICLR 2026
Consistency Rate (%)
100
80
85.
778
85.6
82.4
80.8
805
80.7
80
505
02A
87.
73.0
721
70
60
501
LLaMA3. 1--8--nst Inst
LLaMAS. 1-708- fast
Genma-22 Genma-2-99--4 ----
Gemma-3 Gomma-3 Gomma-3-271 B-17 B-i
1-1ma-3-12844 12B-it
Qwenz. 7. 3-7B--B- Bl-sttt
Qwen 5-1 /48 last.
Owen? 5-728 lhst
DavpSeck V3
DeepSesk-t
Clodde33. 1oude-3.5-Huib 5-Huika
Claske-2 Someer
Ganini 2omini-2.5-PPo --Falahhh
Genni-2.3 2.
GPE-3.
7P744
PandaL.M-77
G-Eval (GPT-3.3
AUTO-
G- G-Eval (GPT (GP--40)
Crival- 7B (wol Swap)
CrEyal- -B
CrEval-1 14B
Figure 5: Consistency rate of different methods when swapping the order of responses. We inlcude
an ablation version CrEval-7B (w/o Swap) without explicitly swapping response positions.
0.8
076
Method
F1
Kappa
Agreement
070
0.7
0.7
03
CrEval-7B
0.732
0.601
0.745
Value
0.6
007
056
055
0.80
080
089
0.0
069
w/o Neg.
0.723
0,586
0,745
w/o Syn.
0.665
0.464
0.634
w/ Only Syn.
0.585
0.356
0.589
0.66
Kappa Agreement
Evaluation results of ablation study different data components.
Figure 6:
SK
10K 20K 30K 50K 100K2000300KK000K 1M 1M8
Data Volume
Figure 7: Performance variation with data scales.
Lyries, they struggle to generalize across other creative domains like humor (e.g., Oogiri-Go and Ruozhiba) and ancient genres. Claude-3.5-Sonnee excels in evaluating Prose, indicating stronger capacity for assessing creativity in longer texts. In contrast, CrEval exhibits more balanced and robust performance across all creative domains.
LLMs may favor certain positions of the response, known as positional bias Wang et al. (022a), which may lead to inconsistent evaluation results when swapping the order of responses. We have conducted a consistency analysis to evaluate the stability of different methods, inlcuding comparing with ablation version CrEval (w/o Swap) where CrEval was trained without explicitly balancing the positions. As shown 5] CrEval achieves the highest consistency rate of 94,4, indicating that it more consistent and reliable in evaluating creativity compared other methods. Also, omitting position swapping during training introduces position bias and leads to decreased performance.
4.3 How Do DATA INFLUENCE CREVAL?
Data Composition. In training CrEval, we use (T, R1, R2, v) of different pseudo-creativity levels. To investigate their influence, we conduct an ablation study by training multiple CrEval variants with
different data compositions as follows: (I) CrEval-w/o Neg.: Training CrEval without sampling negative responses. (2) CrEval-w/o Syn.: Training CrEval with only the original responses (highest creativity) in CreataSet without synthetic ones. (3) CrEval-w/ Only Syn.: Training CrEval with only synthetic responses (lower creativity) in CreataSet without original ones. Table o the ablation study results. The results indicate that each type of data makes a positive contribution. The original human-created responses contribute the most, as they provide diverse, high-quality information that better aligns CrEval with human preferences. Synthetic data plays crucial role in helping the model grasp the characteristics of creative responses, particularly those that LLMs can generate. Meanwhile, negative responses offer additional information improve the model's ability to measure the relevance between responses and instructions.
Data Scale. To assess the impact of data volume, train CrEval datasets of varying scales, shown in Figure 7 F1, Kappa, and agreement rates improve with data size but plateau after 100K samples. This suggests that while more data benefits CrEval, the gains diminish at higher scales.
8

---

Published as a conference paper at ICLR 2026
Creative Writing
WritingPromppts
AUT
80
65
ol
75.00
63.89
57
49.30
86.23
82.07
GPT-4o
80
55.56
80
56.57
42.72
75.42
62.29
GPT-3.5
4028
5.56
49.77
27.70
75.22
60.87
DeepSeck-Ri
80.56
DeepSeek-Vo V3
76339
55.56
70
57.28
32.39
81.05
71.35
Clmude-3.--Sonnc
63.89
44.44
6098
52.11
89.83
84.66
Claude-3.S-Haike
6111
44 44
60
56.60
20
81.33
73.94
Gemini-2.5-Pro
Gemini-2.5-Flash
80.56
66
61.03
51.17
79.94
65.62
G-Eval (GPT-4o)
59.22
41.67
50
52.83
41.78
50
76.80
66.54
G-Eval (GPT-3.5)
20.33
2.78
50
27.23
56.01
15.16
Gemmaa2-9B-it
5423
LLaMA3.1 100A331----lan 88s-Inst.
43.06
33.33
51.90
29.11
45
63.32
55.27
65
PandaLM-7B
0.00
0.00
50.90
33.80
78.00
62:66
Prometheus-72
17.24
13.89
29.58
29.58
4.1
488
60
AUTO-J
37.14
40
62.66
WrttinBennc--rriee
44 44
44.44
32.39
32.39
69.13
69:13
Qwen2.5-7B-lnn 5-St-Inst.
43.06
11.11
56.57
38.97
84.17
61.37
CrEval-7B (ours)
79:17
63.89
58.22
45.54
92.46
78.37
10
35
F1
Agreement
F1
Agreement
F1
Agreement
Figure 8: Agreement and Fl scores three O.O.D. datasets.
Win
Tip
Loss
SFT
2.00%
DPO-Negative
037%
DPO
62175
11%%
4667%
61.37%
0.25%
38.38%
56.00%
7.37%
37
DPO-100F
56.1%%
150%
42.3%%
66.25%
025%
33.50%
60.87%
8.0%
51.3%
DPO-70E30H
5883%
187%
40 0Ox
68 75%
0.52 311
62.12%
312%
3478%
60%
80%
100%
20%
40%
80%
100%
60%
8o%
(a) Evaluate by Human
(b) Evaluate by CrEval
(c) Evaluate by GPT-4o
Figure 9: Win rate of different methods over GPT-4o-mini responses. DPO-Negative denotes DPO with negative sampled responses reject samples. DPO-1008 and DPO-70E30H use all easy and 70% casy+30% hard responses as reject samples, respectively.
4.4 DOES CREVAL DEMONSTRATE
GENERALIZATION
Due to the scarcity of human-annotated creative pairs, finding suitable out-of nuttof-diisributionn (O.O.D.) datasets for meta-ever veta-evaluation remains challenging. To address this, we conduct three O.O.D. experi- ments on two creative writing datasets and one classical creativity benchmark, the Alternative Uses Task (AUT). For creative writing, we adopt data from Chakrabarty al. (2024), which contains long responses produced by both humans and models. Following their findings, treat human responses as more creative and construct 36 evaluation pairs. Besides, we curate 213 pairs from another Writ- ingPrompts dataset, selecting samples with like-count difference greater than 10 as proxy for creativity. For the AUT task, we use the dataset from Sun et al. (2023), focusing on the annotated alternative for "bowl". We form pairs from responses whose human-assignes creativity scores differ by more than two points, resulting in 541 test pairs.
As shown in Figure 8 CrEval achieves the best performance among models of similar scale (~7B) and even outperforms much larger frontier models like GPT-4o and DeepSeek-V3. This strong generalization ability be attributed its robust training on diverse and creative text sources, enabling it better to capture subtle qualitative differences in open-ended generation tasks. The con- sistent advantage across both datasets underscores its effectiveness in text creativity evaluation.
4.5 CAN CREVAL ENHANCE MODEL CREATIVITY?
As a creativity evaluator, CrEval can differentiate response creativity, allowing us to leverage it for enhancing model creativity. We randomly sample 10K data from CreataSet train Qwen2.5-7B- Instruct, using the original response as the ground truth, serving as the standard (the SFT baseline). By utilizing synthetic candidate responses (randomly sampled), we apply DPO Rafailov et al. (0023)

---

Published as a conference paper at ICLR 2026
to take low-creativity responses reject samples (the DPO baseline). We also randomly sample negative responses from other instructions as reject samples, denoted as DPO-Negative.
Given instruction and multiple candidate responses, CrEval performs pairwise creativity compar- isons, scoring wins as 3 points, ties as 1, and losses as O. This scoring yields a creativity ranking, with the top-ranked responses hard and the lowest easy samples. We control creativity diff- culty by adjusting the hard/easy ratio in DPO rejections, evaluating methods on the CreataSet test set, with win rates against GPT-4o-mini measured by CrEval, GPT-4o, and human annotators.
Results in Seeureddemoostratt that DPO yields significant gains over SFT in all evaluation settings (CrEval, GPT-4o, human). The inferior performance of DPO-Negative relative DPO demon- strates that contextual conditioning is crucial for accurate creativity assessment. Leveraging CrEval for creativity-aware data selection leads to further improvements, with DPO-70E30H achieving the highest win rate. DPO- 100E, which treats all easy samples as rejections, shows marginal improve- ment, indicating that clearer distinction between chosen and rejected examples is crucial for learn- ing creativity. DPO-70E300 achieves the highest win rate by using 30% hard samples as rejections, underscoring the benefit of balanced mixture creativity difficulty levels.
5 CONCLUSION
In this paper, propose novel pairwise-comparison framework for evaluating textual creativity and present CreataSet, large-scale dataset across diverse domains. Based on it, we develop CrEval, an LLM-based evaluator that significantly outperforms existing methods in alignment with human judgments. Our experiments highlight the essential role of combining both human and synthetic data in training robust creativity evaluators, and demonstrate that CrEval exhibits -if---i-iitiitttionnn generalization. We further find the practical value integrating CrEval into generation pipelines to boost LLM creativity. We believe that CreataSet and CrEval will be valuable assets for the research community, driving progress toward more accurate and scalable creativity evaluation.
ETHICS STATEMENT
This work adheres to the ICLR Code of Ethics. The study involved human participants who were recruited as data annotators, and the annotation tasks were designed to be of minimal risk. The study did not involve any animal experimentation. All datasets used were sourced in compliance with relevant usage guidelines, with appropriate measures taken to protect privacy and prevent any unauthorized use of data. We have strived to mitigate potential biases and avoid discriminatory outcomes throughout the research process. No personally identifiable information was utilized, and no experiments were conducted that would raise ethical, privacy, or security concerns. We are committed to upholding principles of transparency and integrity in all aspects of this research.
REPRODUCIBILITY STATEMENT
We have taken comprehensive steps to ensure the reproducibility of the results presented in this paper. We have made our code, datasets and models publicly available. The experimental setup, including training procedures and model configurations, is described in detail. We believe these steps will enable the community to validate and build upon work effectively.
ACKNOWLEDGMENTS
This work supported by the National Key R&D Program of China (2023YFF0905402) and Kuaishou Technology. We acknowledge the reviewers for their insightful comments.
REFERENCES
Zahra Abbasiantaeb, Chuan Meng, Leif Azzopardi, and Mohammad Aliannejadi. Can we use large language models fill relevance judgment holes? arXiv preprint:2403: (epprint:2245.0555 0550000000 2024.
10

---

Published as a conference paper at ICLR 2026
Selcuk Acar and Mark A Runco. Divergent thinking: New methods, recent research, and extended theory. Psychology of Aesthetics, Creativity, and the Arts, 13(2):153, 2019.
Anthropic. haiku
Claude 3.5 haiku, 2024a.
URL
Anthropic.
Claude 3.5 sonnet, 2024b.
URL http:://www.anttrrgic...oo/////
claude-S-S-sonnet
John Baer and Sharon S McKool. Assessing creativity using the consensual assessment technique. In Handbook of research assessment technologies, methods, and applications in higher edu- cation, pp. 65-77. IGI Global, 2009.
Yuelin Bai, Xeron Du, Yiming Liang, Leo Jin, Junting Zhou, Ziqiang Liu, Feiteng Fang, Mingshan Chang, Tianyu Zheng, Xincheng Zhang, et al. Coig-cqia: Quality is all you need for chinese instruction fine-tuning. In Findings of the Association for Computational Linguisties: NAACL 2025, pp. 8190-8205, 2025.
Baptiste Barbot, Richard W Hass, and Roni Reiter-Palmon. Creativity assessment in psychological research:(re) setting the standards. Psychology of Aesthetics, Creativity, and the Arts, 13(2):233, 2019.
Kenes Beketayev and Mark A Runco. Scoring divergent thinking tests by computer with semantics-basse algorithm. Europe's 's journal of psychology, 12(2):210, 2016.
Brendan Bena and Jugal Kalita, Introducing aspects of creativity in automatic poetry generation. In Proceedings of the 16th International Conference on Natural Language Processing, pp. 26-35, 2019.
Margaret A. Boden. The Creative Mind Myths and Mechanisms (2. ed.). Routledge, 2003.
Qian Cao, Xu Chen, Ruihua Song. Hao Jiang, Guang Yang, and Zhao Cao. Multi-modal experi- ence inspired Al creation. In João Magalhaes, Alberto Del Bimbo, Shin 'ichi Satoh, Nicu Sebe, Xavier Hameda--Pindds Qin Jin, Vincent Oria, and Laura Toni (eds.), MM '22: The 30th ACM International Conference on Multimedia, Lisboa, Portugal, October 10 14, 2022, pp. 1445- 1454. ACM, 2022. doi: 0..1145/350316135555
URL https: /ttp:::://cc...o/11000.
350316 61. 3548189
Tuhin Chakrabarty, Philippe Laban, Divyansh Agarwal, Smaranda Muresan, and Chien-Sheng Wu. Art or artifice? large language models and the false promise of creativity. In Florian 'Floyd' Mueller. Penny Kyburz, Julie R. Williamson, Corina Sas, Max L. Wilson, Phoebe O. Toups Dugas, and Irina Shklovski (eds.), Proceedings of the CHI Conference on Human Factors in Computing Systems, CHI 2024, Honolulu, HI, USA, May 11-16, 2024, pp. 30:1 30:1-30:34. ACM, 2024. doi: 00.1145/6113044364273 URL https::/ http//oi.oggg///// 1111//.111155331990000
3642731
Tuhin Chakrabarty, Philippe Laban, and Chien-Sheng Wu. Ai-slop to ai-polish? aligning lan- guage models through edit-based writing rewards and test-time computation. arXiv preprint arXiv:2504.07532 2025.
Lingjiao Chen, Matei Zaharia, and James Zou. Frugalgpt: How to use large language models while reducing cost and improving performance. CoRR, abs/2305.05176 2023, doi: 10..8555/..XIV. 2305.05176. URL httpt:ttt/.i.ooottee org/0. 48550 http://iivvv..... 05176
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavar- ian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plap- pert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert- Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji. Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan
11

---

Published as a conference paper at ICLR 2026
Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Pe- ter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained code. CoRR, abs/2107.03374. 2021.
URL
abwpaabs..11//1///// 03374
Gheorghe Comanici, Eric Bieber, Mike Schackermann, Ice Pasupat, Noveen Sachdeva, Inder- jit S. Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, Luke Marris, Sam Petulla, Colin Gaffney, Asaf Aharoni, Nathan Lintz, Tiago Cardal Pais, Henrik Jacobsson, Idan Szpek- tor, Nan-Jiang Jiang. Krishna Haridasan, Ahmed Omran, Nikunj Saunshi, Dara Bahri, Gaurav Mishra, Eric Chu, Toby Boyd, Brad Hekman, Aaron Parisi, Chaoyi Zhang, Kornraphop Kaw- intiranon, Tania Bedrax-Weiss, Oliver Wang, Ya Xu, Ollie Purkiss, Uri Mendlovic, Ili Deu- tel, Nam Nguyen, Adam Langley, Flip Korn, Lucia Rossazza, Alexandre Ramé, Sagar Wagh- mare, Helen Miller, Nathan Byrd, Ashrith Sheshan, Raia Hadsell Sangnie Bhardwaj, Pawel Janus, Tero Rissa, Dan Horgan, Sharon Silver, Ayzaan Wahid, Sergey Brin, Yves Raimond, Klemen Kloboves, Cindy Wang, Nitesh Bharadwaj Gundavarapu, Ilia Shumailov, Bo Wang, Mantas Pajarskas, Joe Heyward, Martin Nikoltchev, Maciej Kula, Hao Zhou, Zachary Garrett, Sushant Kafle, Sercan Arik, Ankita Goel, Mingyao Yang, Jiho Park. Koji Kojima, Parsa Mah- moudieh, Koray Kavukcuoglu, Grace Chen, Doug Fritz, Anton Bulyenov, Sudeshna Roy, Dim- itris Paparas, Hadar Shemtov, Bo-Juen Chen, Robin Strudel, David Reitter, Aurko Roy, An- drey Vlasov, Changwan Ryu, Chas Leichner, Haichuan Yang. Zelda Mariet, Denis Vnukov, Tim Sohn, Amy Stuart, Wei Liang, Minmin Chen, Praynaa Rawlani, Christy Koh, JD Co- Reyes, Guangda Lai, Praseem Banzal, Dimitrios Vytiniotis, Jieru Mei, and Mu Cai. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next genera- tion agentic capabilities. CoRR, abs/2507.062611 2025. doi: 1044850/ARXX..250000066 https://doo www.orgggloooo /10. 48540 (w155//arXiv..o 250 2507 06261
URL
DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Cheng- gang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang. Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao. Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J. L. Cai, Jian Liang, Jianzhong Guo, Jiaqi Ni, Jiashi Li. Jiawei Wang. Jin Chen, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, Junxiao Song, Kai Dong, Kai Hu, Kaige Gao. Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Lei Xu, Leyi Xia, Liang Zhao, Litong Wang, Liyue Zhang, Meng Li, Miaojun Wang. Mingchuan Zhang. Minghua Zhang, Minghui Tang. Mingming Li, Ning Tian, Panpan Huang, Peiyi Wang, Peng Zhang, Qiancheng Wang, Qihao Zhu, Qinyu Chen, Qiushi Du, R. Chen, R. L. Jin, Ruiqi Ge. Ruisong Zhang. Ruizhe Pan. Runji Wang, Runxin Xu, Ruoyu Zhang, Ruyi Chen, S. S. Li, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shaoqing Wu, Shengfeng Ye, Shengfeng Ye, Shirong Ma, Shiyu Wang, Shuang Zhou, Shuiping Yu, Shunfeng Zhou, Shuting Pan, T. Wang. Tao Yun, Tian Pei, Tianyu Sun, W. L. Xiao, and Wangding Zeng. Deepseek- v3 technical report. CoRR, abs/2412. 2024. doi: 10.4550/ARXX..2242219999 URL 00t
httpg//1...550/arr//. hwtp:////.xwi.221..... 2412. 19437
DeepSeck-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao. Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang. Jingchang Chen. Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang. Lecong Zhang, Liang Zhao, Litong Wang, Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, and S. S. Li. reepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. CoRR, abs/2501. 12948, 2025. doi: 10. 48550/ARXX ARXIV.2500. 12948. URL //doi wwwg//ll....... 85500 a x 2501 12948
12

---

Published as a conference paper at ICLR 2026
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre- pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), 2019.
Jiang Dongfu, Li Yishan, Ge Zhang, Huang Wenhao, Lin Bill Yuchen, and Chen Wenhu, Tiger- score: Towards building explainable metric for all text generation tasks. Computing Research Repository, 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra. Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurélien Rodriguez. Austen Gregerson, Ava Spataru, Baptiste Rozière, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloc Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song. Danielle Pintz, Danny Livshits, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Ander- son, Graeme Nail, Grégoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Ko- revaar, Hu Xu, Hugo Touvron, lliyan Zarov, Imanol Arrieta Ibarra, Isabel M. Kloumann, Ishan Misra, Ivan Eytimov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Ma- hadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang. Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Al- wala, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, and et al. The llama 3 herd of models. CoRR, htbs/2200.22783.. 2024. doi: 1044550/AXXIV.240.22222 URL http:/1...4.5////// http: 2407..2244... 21783
Daniel Fein, Sebastian Russo, Violet Xiang, Kabir Jolly, Rafael Rafailov, and Nick Haber. Litbench: A benchmark and dataset for reliable evaluation of creative writing. CoRR, abs/2507.007666 2025. doi: 10.48550/ARXIV22557.000
URL sittps://dol: org 485508445002xiii. http:///ob...g.....885/5////./1. 2507.
00769
Mingqi Gao, Xinyu Hu, Xunjian Yin, Jie Ruan, Xiao Pu, and Xiaojun Wan. Llm-based NLG evaluation: Current status and challenges. Computational Linguistics, pp. 1-28, 2025.
Kurt Gray, Stephen Anderson, Eric Evan Chen, John Michael Kelly, Michael S Christian, John Patrick, Laura Huang, Yoed N Kenett, and Kevin Lewis. "forward flow": A new measure to quantify free thought and predict creativity. American Psychologist, 74(5):539, 2019.
Maarten Grootendorst. Bertopic: Neural topic modeling with class-based tf-idf procedure. arXiv preprint arXiv:2203.057999 2022.
Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Ying- han Shen, Shengjie Ma, Honghao Liu, et al. survey on Ilm-as-a-judge. arXiv preprint arXiv:2411.15594, 2024.
Joy Paul Guilford. The nature of human intelligence. 1967.
Qianyu He, Yikai Zhang, Jiaqing Liang, Yuncheng Huang. Yanghua Xiao, and Yunwen Chen. towards holistic and automatic evaluation of simile generation. In Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), 2023.
HAUSER:
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Joaquin Vanschoren and Sai-Kit Yeung (eds.), Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1. NeurIPS Datasets and Benchmarks 2021, December 2021, virtual, 2021. URL
haas://essadceedddd//////cccaaaa -
round2.hter
13

---

Published as a conference paper at ICLR 2026
Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng. Yewei Fang, Yuxiang Huang, Weilin Zhao, Xinrong Zhang, Zhen Leng Thai, Kai Zhang, Chongyi Wang. Yuan Yao. Chenyang Zhao, Jie Zhou, Jie Cai, Zhongwu Zhai, Ning Ding, Chao Jia, Guoyang Zeng, Dahai Li, Zhiyuan Liu, and Maosong Sun. Minicpm: Unveiling the potential of small language models with scalable training strategies. CoRR, abs/2404.06395,, 2024a. doi: 10.48550/ARXIV. 2404.06395. URL http:: http://oooo.rg....//// 48w.1//..4350//x11..222220 06395
Xinyu Hu, Li Lin, Mingqi Gao, Xumjian Yin, and Xiaojun Wan. Themis: A reference free nlg evalu- ation language model with flexibility and interpretabilityy. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 15924-15951, 2024b.
Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander Madry, Alex Baker-Whitcomb, Alex Beutel, Alex Borzunov, Alex Carney, Alex Chow, Alex Kirillov, Alex Nichol, Alex Paino, Alex Renzin, Alex Tachard Passos, Alexander Kirillov, Alexi Christakis, Alexis Conneau, Ali Ka- mali, Allan Jabri, Allison Moyer, Allison Tam, Amadou Crookes, Amin Tootoonchian, Ananya Kumar, Andrea Vallone, Andrej Karpathy, Andrew Braunstein, Andrew Cann, Andrew Codis- poti, Andrew Galu, Andrew Kondrich, Andrew Tulloch, Andrey Mishchenko, Angela Baek, An- gela Jiang, Antoine Pelisse, Antonia Woodford, Anuj Gosalia, Arka Dhar, Ashley Pantuliano, Avi Nayak, Avital Oliver, Barret Zoph, Behrooz Ghorbani, Ben Leimberger, Ben Rossen, Ben Sokolowsky, Ben Wang, Benjamin Zweig, Beth Hoover, Blake Samic, Bob McGrew, Bobby Spero, Bogo Giertler, Bowen Cheng, Brad Lightcap. Brandon Walkin, Brendan Quinn, Brian Guarraci, Brian Hsu, Bright Kellogg, Brydon Eastman, Camillo Lugaresi, Carroll L. Wainwright, Cary Bassin, Cary Hudson, Casey Chu, Chad Nelson, Chak Li, Chan Jun Shern, Channing Con- ger, Charlotte Barette, Chelsea Voss, Chen Ding, Cheng Lu, Chong Zhang, Chris Beaumont, Chris Hallacy, Chris Koch, Christian Gibson, Christina Kim, Christine Choi, Christine McLeavey, Christopher Hesse, Claudia Fischer, Clemens Winter, Coley Czarnecki, Colin Jarvis, Colin Wei, Constantin Koumouzelis, and Dane Sherburn. Gpt-4o system card. CoRR, abs/2410.211666 2024. doi: 1044550/ARXIV22410.2222
URL
2410.
21276
Aaron Jacch, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, Alex Iftimie, Alex Karpenko, Alex Tachard Passos, Alexander Neitz, Alexander Prokofiev, Alexander Wei, Allison Tam, Ally Bennett, Ananya Kumar, Andre Saraiva, Andrea Vallone, Andrew Duberstein, Andrew Kondrich, An- drey Mishchenko, Andy Applebaum, Angela Jiang, Ashvin Nair, Barret Zoph, Behrooz Ghor- bani, Ben Rossen, Benjamin Sokolowsky. Boaz Barak, Bob McGrew, Borys Minaiev, Botao Hao. Bowen Baker, Brandon Houghton, Brandon McKinzie, Brydon Eastman, Camillo Lu- garesi, Cary Bassin. Cary Hudson, Chak Ming Li, Charles de Bourcy, Chelsea Voss, Chen Shen, Chong Zhang. Chris Koch, Chris Orsinger, Christopher Hesse, Claudia Fischer, Clive Chan, Dan Roberts, Daniel Kappler, Daniel Levy, Daniel Selsam, David Dohan, David Farhi, David Mely, David Robinson, Dimitris Tsipras, Doug Li, Dragos Oprica, Eben Freeman, Eddie Zhang, Ed- mund Wong, Elizabeth Prochl, Enoch Cheung, Eric Mitchell, Eric Wallace, Erik Ritter, Evan Mays, Fan Wang, Felipe Petroski Such, Filippo Raso, Florencia Leoni, Foivos Tsimpourlas, Francis Song, Fred von Lohmann, Freddie Sulit, Geoff Salmon, Giambattista Parascandolo, Gildas Chabot, Grace Zhao, Greg Brockman, Guillaume Leclere, Hadi Salman, Haiming Bao, Hao Sheng, Hart Andrin, Hessam Bagherinezhad, Hongyu Ren, Hunter Lightman, Hyung Won Chung, Ian Kivlichan, Ian O'Connell, Ian Osband, Ignasi Clavera Gilaberte, and Ilge Akkaya. URL.
Openai 0l system card. CoRR, abs/2412.167200, 2024. doi: 1044850/ARXXV.2112..66 48550 2412 16720
Chuhao Jin, Kening Ren, Lingzhen Kong, Xiting Wang, Ruihua Song, and Huan Chen. Persuading across diverse domains: dataset and persuasion large language model. In Lun-Wa Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), ACI. 2024, Bangkok, Thailand, August 11-16, 2024. pp. 1678-1706. Association for Computational Linguistics, 2024. doi: 18653/ long.
V1/2024ACLL-LNN..0 URL Intpt:::///// :ttpp:://doi.. org/10....... 1ww.1111111/0000
92
Dan R Johnson, James C Kaufman, Brendan Baker, John D Patterson, Baptiste Barbot, Adam E Green, Janet van Hell, Evan Kennedy, Grace F Sullivan, Christa I. Taylor, et al. Divergent seman-
14

---

Published as a conference paper at ICLR 2026
tic integration (dsi): Extracting creativity from narratives with distributional semantic modeling. Behavior Research Methods, 55(7):3726-377 3759, 2023.
Antonio Laverghetta Jr. Tuhin Chakrabarty, Tom Hope, Jimmy Pronchick, Krupa Bhawsar, and Roger F. Beaty. How do humans and language models reason about creativity? A comparative analysis. CoRR, abs/2500.033253 2025. doi: (0448550/ARXXV.2522.0222 URL https:
//doi. megi.org/100 10. 48550 2502. 03253
Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Ta- tiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geof- frey Cideron, Bean-Bastied Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penchev, Gaäl Liu, Francesco Visin, Kathleen Kenealy, Lucas Beyer, Xiaohai Zhai, Anton Tsitsulin, Róbert Busa-Fekete, Alex Feng, Noveen Sachdeva, Benjamin Coleman, Yi Gao, Basil Mustafa, lain Barr, Emilio Parisotto, David Tian, Matan Eyal, Colin Cherry, Jan-Thorsten Peter, Danila Sinopalnikov, Surya Bhupatiraju, Rishabh Agarwal, Mehran Kazemi, Dan Malkin, Ravin Kumar, David Vilar, Idan Brusilovsky, Jiaming Luo, Andreas Steiner, Abe Friesen, Abhanshu Sharma, Abheesht Sharma, Adi Mayrav Gilady, Adrian Goedeckemeyer, Alaa Saade, Alexan- der Kolesnikov, Alexei Bendebury, Alvin Abdagic, Amit Vadi, András György, André Susano Pinto, Anil Das, Ankur Bapna, Antoine Miech, Antoine Yang, Antonia Paterson, Ashish Shenoy, Ayan Chakrabarti, Bilal Piot, Bo Wu, Bobak Shahriari, Bryce Petrini, Charlie Chen, Charline Le Lan, Christopher A. Choquette-Choo, CJ Carey, Cormac Brick, Daniel Deutsch, Danielle Eisen- bud, Dee Cattle, Derek Cheng, Dimitris Paparas, Divyashree Shivakumar Sreepathiballi, Doug Reid, Dustin Tran, Dustin Zelle, Eric Noland, Erwin Huizenga, Eugene Kharitonov, Frederick Liu, Gagik Amirkhanyan, Glenn Cameron, Hadi Hashemi, Hanna Singh, Harsh Mehta, Harshal Tushar Lehri, Hussein Hazimeh, Ian Ballantyne, Idan Szpektor, and Ivan Nardini. Gemma 3 technical report. CoRR, abs/2503. 19786, 2025. doi: 10,48550/ARXIV. 2503.19786. URL http:::t://// /ttp://ooiiote wrg/ /8550/ arxiv. 2503. 2503.
Harman
Kyung Hee Kim. Can we trust creativity tests? review of the torrance tests of creative thinking (Itct). Creativity research journal, 18(1):3-14, 2006.
Seungone Kim, Jamin Shin, Yejin Choi, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thome, and Minjoon Seo. Prometheus: Inducing fine- grained evaluation capability in language models. In The International Conference on Learning Representations (ICLR), 2024.
Dawei Li, Bohan Jiang, Liangjie Huang, Alimohammad Beigi, Chengshuai Zhao, Zhen Tan, Amrita Bhattacharjee, Yuxuan Jiang, Canyu Chen, Tianhao Wu, et al. From generation to judgment: Op- portunities and challenges of llm-as-a-judge. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp. 2757-2791, 2025a.
Jijie Li, Li Du, Hanyu Zhao, Bowen Zhang. Liangdong Wang, Boyan Gao, Guang Liu, and Yonghua Lin. Infinity instruct: Scaling instruction selection and synthesis to enhance lan- guage models. CoRR, 1lds2500..11166. 2025b. doi: 10.4850/ARXIV:256.1111 URL
https:: http:::/doo..oooo wtt.ooooor......5/r///...../ 2522 11116
Junlong Li, Shichao Sun, Weizhe Yuan, Run-Ze Fan, Hai Zhao, and Pengfei Liu. Generative judge for evaluating alignment. In The International Conference on Learning Representations, (ICLR), 2024.
Ruizhe Li, Chiwei Zhu, Benfeng Xu, Xiaorui Wang, and Zhendong Mao. Automated creativity eval- uation for large language models: A reference-based approach. arXiv preprint arXiv:2504.1578 2025c.
Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic hu- man falsehoods. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Pro- ceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Vol-
ume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pp. 3214-3252. Asso- ciation for Computational Linguistics, 2022. doi: 12L2 https: tttps://doii 0rrg/10. /00. 1653366 acl ong 229
URL
15

---

Published as a conference paper at ICLR 2026
Yen- Ven-Ting Lin and Yun-Nung Chen. LLM-eval: Unified multi- multi-dimensional automatic evaluation for open-domain conversations with large language models. In Yun-Nung Chen and Abhinav Rastogi
(eds.), Proceedings of the 5th Workshop on NLP for Conversational Al (NLP4ConvAI 2023), pp. 47-58. Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/ w1/2023.nl4cconval
URL https://accl http:///allltahhhoaaaao logy wwt. http://hooooyy..o22233..333 p4convai
57
Mingian Liu, Ying Shen, Zhiyang Xu, Yixin Cao, Eunah Cho, Vaibhav Kumar, Reza Ghanadan, and Lifu Huang. X-eval: Generalizable aulli- text evaluation via augmented instruction tuning with auxiliary evaluation aspects. In Kevin Duh, Helena Gómez-Adorno, and Steven Bethard (eds.), Proceedings of the Conference the North American Chapter of the Association for Com- putational Linguistics (NAACL), 2024.
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang. Ruochen Xu, and Chenguang Zhu. G-eval: NLG evaluation using gpt-4 with better human alignment. Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, (EMNLP), 2023.
2023.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In The International Conference on Learning Representations (ICL.), 2019.
Li-Chun Lu, Miri Liu, Pin-Chun Lu, Yufei Tian, Shao-Hua Sun, and Nanyun Peng. Rethinking creativity evaluation: A critical analysis of existing creativity evaluations. CoRR, abs//508./5/577777 05470, 2025a. doi: 1044550/ARXIV25508.000 05470. URL 2508 05470
Ximing Lu, Melanie Sclar, Skyler Hallinan, Niloofar Mireshghallah, Jiacheng Liu, Seungju Han, Allyson Ettinger, Liwei Jiang, Khyathi Raghavi Chandu, Nouha Dziri, and Yejin Choi. Al as humanity's 's salieri: Quantifying linguistic creativity of language models via systematic attribu- tion of machine text against web text. In The Thirteenth International Conference on Learn- ing Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net, 2025b. URL net. forum? werpummid=lllm 1=111=1111111ppp
Guillermo Marco, Julio Gonzalo, María Teresa Mateo Girona, and Ramón Santos. Pron vs prompt: Can large language models already challenge world-class fiction author creative text writing? In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.). Proceedings of the 2024 Con- ference on Empirical Methods in Natural Language Processing, EMNLP 2024, Miami, FL, USA, November 16, 2024, pp. 19654- 19654-19670. Association for Computational Linguistics, 2024. URL
org 2024. emnlp-main 1096
John McCarthy, Marvin I. Minsky, Nathaniel Rochester, and Claude E. Shannon. A proposal for the dartmouth summer research project artificial intelligence, august 31, 1955. Al magazine, 27 (4):12-12, 2006.
Martha Mednick and Sharon Halpern. Remote associates test. Psychological Review, 1968.
Saeid Alavi Naeini, Raeid Saqur, Mozhgan Saeidi, John M. Giorgi, and Babak Taati. Large lan- guage models are fixated by red herrings: Exploring creative problem solving and einstellung effect using the only connect wall dataset. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems (NeurIPS), 2023.
Matthew Lyle Olson, Neale Ratzlaff, Musashi Hinck, Shao-yen Tseng, and Vasudev Lal. Steering large language models to evaluate and amplify creativity. arXiv preprint arXiv:2412.00060, 2024.
OpenAl. Introducing chatgpt, 2022. URL https: httt:p::/ppeeaaa.ccom///ees/s/// wttpc////nne//cattp/t//////////
OpenAl. 03 and 04 mini system card.
2025.
URL https: cdn.
//cdn.
wpwnai.com..e pdf 2ttpf//2222187522222222 4789 8788-8800--77778337222
o3-and- 04 - mini system-ymemmmmmmmm card. pdf
Samuel Paech. Eq-bench: An emotional intelligence benchmark for large language models. arXiv preprint arXiv:23/2.06221, 2023.
16

---

Published as a conference paper at ICLR 2026
Howard B Parkhurst. Confusion, lack of consensus, and the definition of creativity as a construct. The Journal of Creative Behavior, 33(1):1-21, 1999.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems (NeurlPS), 2023.
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: memory optimizations toward training trillion parameter models. In Christine Cuicchi, Irene Qualters, and William T. Kramer (eds.), Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2020, Virtual Event / Atlanta, Georgia, USA, November 9-19, 2020, pp. 20. IEEE/ACM, 2020. doi: 10.1 (0.1109/SC44405 2000000004. URL https: 7/doi 1ttp://111111111111150001 20..00....220.00..000.00... 00024
Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System op- timizations enable training deep learning models with over 100 billion parameters. In Ra- jesh Gupta, Yan Liu, Jiliang Tang, and B. Aditya Prakash (eds.), KDD '20: The 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Virtull Event, CA, USA, Au- gust 23-27, 2020, pp. 3505 3506. ACM, 2020. doi: 00.1145/339448864406700 URL https: 7/doi. org. 0. 45/ 3055533444888 3406703
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11 2019. http:://tappiii
abs/1908. 10084
Graeme Ritchie Some empirical criteria for attributing creativity to a computer program. Minds Mach., 17(1):67-99, 2007. doi: 1007/S 10.1007/511123-00000 1023-00779066 URL org/10. L007/811023-007-9060
Morgane Rivière, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan, Sammy Jerome, Anton Tsitsulin, Nino Vieillard, Piotr Stanczyk, Sertan Girgin, Nikola Momchev, Matt Hoffman, Shantanu Thakoor, Bean-Bastien Grill, Behnam Neyshabur, Olivier Bachem, Alanna Walton, Aliaksei Severyn, Alicia Parrish, Aliya Ahmad, Allen Hutchison, Alvin Abdagic, Amanda Carl, Amy Shen, Andy Brock, Andy Coenen, Anthony Laforge, Antonia Paterson, Ben Bastian, Bilal Piot, Bo Wu, Brandon Royal, Charlie Chen, Chintu Kumar, Chris Perry, Chris Welty, Christopher A. Choquette-Choo, Danila Sinopalnikov. David Weinberger, Dimple Vi- jaykumar, Dominika Rogozinska, Dustin Herbison, Elisa Bandy, Emma Wang, Eric Noland, Er- ica Morcira, Evan Senter, Evgenii Eltyshev, Francesco Visin, Gabriel Rasskin, Gary Wei, Glenn Cameron, Gus Martins, Hadi Hashemi, Hanna Klimezak-Plucinnke Ivan Nardini, Jacinda Mein, Jack Zhou, James Svensson, Jeff Stanway, Jetha Chan, Jin Peng Zhou, Joana Carrasqueira, Joana Iljazi, Jocelyn Becker, Joe Fernandez, Joost van Amersfoort, Josh Gordon, Josh Lipschultz, Josh Newlan, Ju-yeong Ji, Kareem Mohamed, Kartikeya Badola, Kat Black, Katie Millican, Keelin McDonell, Kelvin Nguyen, Kiranbir Sodhia, Kish Greene, Lars Lowe Sjösund. Lauren Usui, Laurent Sifre, Lena Heuermann, Leticia Lago, and Lilly Mc- Nealus. Gemma 2: Improving open language models at a practical size. CoRR, abs/2408.001 2024. doi: 10.48550/ARXIV.24006 2248800011
Harleen Batra, Harsh Dhand,
18. URL https: //o.i http:////oo.oo///....oo.e 48550//r8i5//
Mark A Runco and Garrett J Jaeger. The standard definition of creativity. Creativity research journal, 24(1):92 96, 2012.
Claire E. Stevenson, Iris Smal, Matthijs Baas, Raoul P. P. P. Grasman, and Han L. J. van der Maas. Putting gpt-3's creativity to the (alternative uses) test. In Maria M. Hedblom, Anna Aurora Kantosalo, Roberto Confalonieri, Oliver Kutz, and Tony Veale (eds.). Proceed- ings of the 13th International Conference on Computational Creativity, ICCC 2022, Bozen- Bolzano. Italy, June 27- July 1, 2022, pp. 164-168. Association for Computational Creativity (ACC), 2022. URL /ttp://////ooottaaoooooaaaaaaaaa httpaa/arraa.aaa....ee/////////a (BCC-2022__0 10022_paper____ 11e__110.0 .ddf
17

---

Published as a conference paper at ICLR 2026
Douglas Summers- Stay, Clare R Voss, and Stephanie M Lukin. Brainstorm, then select: generative language model improves its creativity score. In The AAAI Workshop on Creative Al Across Modalities, 2023.
Luning Sun, Hongyi Gu, Rebecca Myers, and Zheng Yuan. A new dataset and method for creativity assessment using the alternate uses task. In BenchCouncil International Symposium on Intelligent Computers, Algorithms, and Applications, pp. 125-138. Springer. 2023.
Luning Sun, Yuzhuo Yuan, Yuan Yao, Yanyan Li, Hao Zhang, Xing Xie, Xiting Wang, Fang Luo, and David Stillwell. Large language models show both individual and collective creativity com- parable to humans. Thinking Skills and Creativity, pp. 101870, 2025.
Yufei Tian, Abhilasha Ravichander, Lianhui Qin, Ronan Le Bras, Raja Marjieh, Nanyun Peng, Yejin Choi, Thomas L. Griffiths, and Faeze Brahman. Macgyver: Are large language models creative problem solvers? In Kevin Duh, Helena Gómez-Adomo, and Steven Bethard (eds.), Proceedings of the Conference of the North American Chapter of the Association for Computational Linguis- tics (NAACL), 2024.
Weixi Tong and Tianyi Zhang, Codejudge: Evaluating code generation with large language models. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2024.
E Paul Torrance. Torrance tests of creative thinking. Educational and psychological measurement, 1966.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 11,,, 2008.
Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Lingpeng Kong, Qi Liu, Tianyu Liu, and Zhifang Sui. Large language models are not fair evaluators. In Lun- Wei Ku, Andre Martins, and Vivek Srikumar (eds.). Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), 2024a.
Yidong Wang. Zhuohao Yu, Wenjin Yao, Zhengran Zeng, Linyi Yang. Cunxiang Wang, Hao Chen, Chaoya Jiang, Rui Xie, Jindong Wang. Xing Xie, Wei Yc, Shikun Zhang, and Yue Zhang. Pan- dalm: An automatic evaluation benchmark for LLM instruction tuning optimization. In The International Conference on Learning Representations (ICLR), 2024b.
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannanch Hajishirzi. Self-instruct: Aligning language models with self gelf-generated instructions. In Proceedings of the 6lst anmual meeting of the association for computational linguistics (volume 1: long papers), pp. 13484-1333 13508, 2023.
Theresa J Weinstein, Simon Majed Ceh, Christoph Meinel, and Mathias Benedek. What's creative about sentences? computational approach assessing creativity sentence generation task. Creativity Research Journal, 34(4):415 1414449940, 2022.
Yang Wu, Yao Wan, Zhaoyang Chu, Wenting Zhao, Ye Liu, Hongyu Zhang, Xuanhua Shi, Hai Jin, and Philip Yu. Can large language models serve as evaluators for code summarization? IEEE Transactions Software Engineering, 2025a.
Yuming Wu, Jiahao Mei, Ming Yan, Chenliang Li. Shaopeng Lai, Yuran Ren, Zijia Wang, Ji Zhang, Mengyue Wu, Qin Jin, et al. Writinghench: A comprehensive henchmark for generative writing. arXiv preprint 2rXiv:2503.05244 2025b.
Yuhui Xu, Lingxi Xie, Xiaotao Gu, Xin Chen, Heng Chang, Hengheng Zhang, Zhengsu Chen, Xi- aopeng Zhang, and Qi Tian. Qa-lora: Quantization-awwe aware low-rank adaptation of large language models. In The International Conference on Learning Representations (ICIR), 2024.
Aiyuan Yang, Bin Xiao, Bingning Wang, Borong Zhang, Ce Bian, Chao Yin, Chenxu Lv, Da Pan, Dian Wang, Dong Yan, Fan Yang. Fei Deng, Feng Wang, Feng Liu, Guangwei Ai. Gu- osheng Dong, Haizhou Zhao, Hang Xu, Haoze Sun, Hongda Zhang, Hui Liu, Jiaming Ji, Jian Xie, Juntao Dai, Kun Fang, Lei Su, Liang Song, Lifeng Liu, Liyun Ru, Luyao Ma, Mang
18

---

Published as a conference paper at ICLR 2026
Wang, Mickel Liu, MingAn Lin, Nuolan Nie, Peidong Guo, Ruiyang Sun, Tao Zhang, Tian- peng Li, Tianyu Li, Wei Cheng, Weipeng Chen, Xiangrong Zeng, Xiaochuan Wang, Xiaoxi Chen, Xin Men, Xin Yu, Xuchai Pan. Yanjun Shen, Yiding Wang, Yiyu Li, Youxin Jiang, Yuchen Gao, Yupeng Zhang, Zenan Zhou, and Zhiying Wu. Baichuan 2: Open large-scale language models. CoRR, abs/2309. 1000230 2023. doi: 10.4850/ARXXV2230000002
URL
https://doi.com 2ttg//0...85//r2111
10305
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng. Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang. Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men. Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yugiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. CoRR, abs/2412.151155, 2024. doi: 10.48550/ARXI..24...5555
URL https: /tccc ottps////o...rr//...45588
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jian Yang. Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang. Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang. Zeyu Cui, Zhenru Zhang. Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report. CoRR, walbb/25550093355 2025. doi: 1044550/ARRXX.2555.0055 URL https: www.orgggo.ooooooooooooo........ 48550/ http:/axivvvv....
Chen Zhang, Luis Fernando D'Haro, Yiming Chen, Malu Zhang, and Haizhou Li. A compre- hensive analysis of the effectiveness of large language models automatic dialogue evaluators. In Michael J. Wooldridge, Jennifer G. Dy, and Sriraam Natarajan (eds.), Thirty-Eighth AAAI Conference on Artificial Intelligence, AAAI 2024, Thirty-Sixth Conference on Innovative Appli- cations of Artificial Intelligence, IAAI 2024, Fourteenth Symposium on Educational Advances in Artificial Intelligence, EAAI 2014, February 20-27, 2024, Vancouver, Canuda, pp. 19515-19524, URL htlihttss: http:///ooioorgg//// 1609/
AAAI Press, 2024. doi: aaai www.17111.... 29923
Yunpu Zhao, Rui Zhang, Wenyi Li, and Ling Li. Assessing and understanding creativity in large language models. Machine Intelligence Research, 22(3)4417-466, 2025.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt- bench and chatbot arena. Advances in Neural Information Processing Systems (NeurIPS), 2023.
Shanshan Zhong, Zhongzhan Huang, Shanghua Gao, Wushao Wen, Liang Lin, Marinka Zitnik, and Pan Zhou. Let's think outside the box: Exploring leap-of-thought in large language models with creative humor generation. In JEEE/CVF Conference on Computer Vision and Pattern Recogni- fion (CVPR), 2024.
19

---

Published as a conference paper at ICLR 2026
A
APPENDIX
A.1 SUBJECTIVITY AND IMPORTANCE OF CREATIVITY
While creativity is inherently subjective, we wish to highlight that:
(1) Creativity is a core Al objective, both historically and practically, whose complexity should be embraced rather than avoided. "Randomness and creativity" was identified one of the seven key problems at the Dartmouth Summer Research Project on Artificial Intelligence and recognized as essential to human-level intelligence and central to the very definition of machine intelligence|Me- Carthy et al. (006). With the rise of L.LMs in domains such as storytelling, ideation, and design, evaluating creative ability has become both timely and necessary. Without creativity, Al models cannot generate truly novel, out-of-domain ideas- a crucial aspect of human-level intelligence that has shaped modern civilization.
(2) The subjectivity of creativity is inevitable but controllable through our rigorous evaluation design. Although creativity involves personal and cultural variation, psychological studies [Barbot et al. (2019); Parkhurst (1999) show that people often converge in recognizing creative content. For instance, more creative ideas, such as the novel Harry Potter, typically receive higher engagement (e.g., more likes). Rather than eliminating subjectivity, our goal is to model shared human judgment in a reproducible manner using pairwise comparisons and consensus-driven aggregation. To this end, we collaborate with annotators from diverse backgrounds to ensure that the resulting evaluation framework captures a robust, collective understanding of creativity.
(3) Our goal is not to equate creativity with conformity, but rather to approximate shared hu- man judgments in reproducible and scalable way. Psychological studies have shown humans can reliably recognize creativity across cultures and domains, especially when it combines novelty with usefulness Runco & Jaeger (2012); BBrbbot et al. (2019). Our use of consensus aims to reflect this shared intuition, not to suppress unconventionally. Importantly, our evaluation framework sup- ports multiple forms of creativity, including surprising, offbeat, or even subversive responses, as long as they are meaningful and novel to the prompt. In this sense, we aim to capture a broad and inclusive view of creativity, grounded in human judgment but not reduced to majority taste.
A.2 COMPARISON WITH THE ABSOLUTE SCALE OF CREATIVITY
Although absolute score-based evaluation has some value, we emphasize that the pairwise compar- ison approach offers several distinct advantages.
Absolute creativity scales are difficult to define and apply. practice, defining universal abso- lute scale for creativity is difficult: annotators find it hard to define what to 5 means across samples and keep consistent standards/trresholds across people. Depending on the context of the instruction, response scoring 3 could be deemed creative, whereas another instruction might require 5 score for its response to constitute a creative answer, making "high score = high creativity" an unreli- able standard. In our pilot experiments, the ratings of 3 annotators 50 data groups (each with 5 responses) demonstrate that they produced similar relative rankings across responses but diverged substantially in absolute scores, with differences up 1.02 points. This level of inconsistency fur- ther reduces the usefulness of absolute scoring for large-scale alignment.
Pairwise comparison aligns directly with how LLMs are trained. We frame CrEval pair- wise task due to its direct applicability to various model training algorithms, including DPO, re- ward model training, etc. These methods require reliable, scalable, and implicit understanding of preference. Given the above challenges, absolute scale methods like compt-bbsed LLM scor- ing "Summers-Stay et al. (2023); Zhao et al. (2025), often struggle to provide fine-grained, relative assessments needed for model alignment. This is partly because the definition of an absolute score can be ambiguous and prone to individual interpretation, leading to inconsistency. As a result, pairs derived from absolute scores tend to be less reliable than those constructed directly through pairwise comparison, which provides clearer and more consistent training signals. If we want to quickly rank model responses using pairwise comparisons, reference. Ranking can then be efficiently achieved based on win rate, incurring minimal computa- tional overhead. These advantages above motivate our design choice for CrEval.
can leverage the response from any model
20

---

Published as a conference paper at ICLR 2026
A.3
ADDITIONAL INFORMATION OF CREATASET
A.3.1 DETAILS OF CREATASET-BANT BASE CONSTRUCTION
Across-Domain Creativity Dataset Initialization We use the Oogiri-GO dataset from CLoT ZZhng al. (2024) contains 15K Chinese humorous responses given questions. The Ruozhiba dataset Bai ct al. (2025), derived from an interest- online community, which demon- strates linguistic creativity through various linguistic features, including puns, wordplay, and humor. Since most of the creativity in this dataset of 1K entries is concentrated in the instructions, we re- formulate the task by generating instructions from responses. The evaluator is to judge whether the generated instruction is creative.
The instruction generator is trained based on the Baichuan2 Yang et al. (2023) model. We sam- pled 600k reversed data pairs from the large-scale instruction tuning dataset Infinity-Instruet. To further enhance instruction diversity, we also employ GPT-4o-mini to generate additional instruc- tions. Each creativity dense text is paired with generated instruction after filtering, forming set pairs from
of creative prruuction-rssoons pairs. We sample 100k ordinary instruction Infinit--Instruct Then, these instructions are used prompt GPT-4o generate creative responses.
Unified Instruction Responss Standardization To verify the quality of generated instructions, conduct several steps for quality control. First, generated instructions through length filtering, eliminating repeated phrases, and removing those containing the response substring. Then, annotate 200 data samples across all sources to assess whether each in- struction aligns with its corresponding response. We finally obtained an accuracy of 96.5%, which indicates that our instructions are of high quality.
carefully refined
After collecting (1, It) pairs, we employ GPT-4o-mini to score the creativity of each (t) pair on scale from 6. This creativity score serves as a quality indicator, enabling us to filter out low-quality data. At last, only pairs with a score exceeding 4 are retained. The prompt used in this step will be presented in the following.
A.3.2 STATISTICS
We present the details of our CreataSet in original and paired samples, shown in Ta- ble 3 The dataset consists of multi-source data, including short texts, lyrics, ancient poetry. modern poetry. prose, Oogiri-GO, worth
# Samples
# Paired Samples
Scenario
Train
Test
Train
Test
Short Texts
36,205
361,090
410
Lyrics
9,186
50
81,566
364
Ruozhiba, and Infinity-Instruet. noting that the infinity-instruct source can provide a large number of data with general instructions, which is beneficial for training creativity evaluators. Besides, prose offers long texts with rich content, enabling CrEval to handle a longer context. We will release the dataset along with the CrEval to facilitate future research on creativity evaluation.
Ancient Poctry
11,222
50
111,590
369
Modern Poetry
17,359
50
159,973
368
Prose
806
5,786
380
Oorigi-Go
10,008
50
99,409
430
Ruozhiba
1,135
50
11,315
451
Infinity Instruct
27,044
225,876
424
Total
112,965
400
1,056,605
3,196
Table 3: The statistics of the CreataSet dataset.
A.3.3 LENGTH DISTRIBUTIONS
We present the length distributions of CreataSet-Base and other creative-related datasets in Fig- ure II As shown, CreataSet- Base has a broader response length distribution.
Additionally, randomly sample 800 samples from each source in the dataset and present the distribution the response lengths in Figure 10 For better visualization of their KDE curves, we applied log transformation to the length as the x-axis.
From the figure, we observe that: (1) The Short Texts, Ruozhiba, and Ancient Poetry sources primar- ily consist of shorter responses. (2) The Lyrics, Modern Poetry, and Lnfinity-Instrue mostly contain medium-length responses. (3) The Prose source exhibits longer responses than other sources, while CLoT shows uniform distribution, covering both short and medium-length responses.
21

---

Published as a conference paper at ICLR 2026
ShortText
Ruozhiba
CLoT
Ancient-Poetry
Lyrics
Modern Modemn-Poctry Poctry
Insinit
Prose
3
Lug-Irand Log-Iramsformee formed Length
Figure 10: Length Distribution of Different Sources.
12
Dataset Length Distribution (Area Emphasized)
10
MacGyver
CretaSet (ours)
Number of Samples
Oorigi-GO
TTCW
800
Vac
WritingBench
CreataSot (ours)
TSNE dim2
8
600
Oorigi-Go
400
2
200
100
200
300
400
500
Sample Length
-2
0
2
4
10
12
Figure 11: The length distributions of MacGyver, Oorigi-GO, DPT, WritingBench and CreataSet- Baase. For better visualization, have omitted TTCW and Creative Writing v3 due to their small dataset sizes.
TSNE diml
Figure 12: The t-SNE visualization of se- mantic distributions of DPT, TTCW, Mac- Gyver, Oorigi and CreataSet-Bas Base.
Overall, our dataset encompasses diverse range of response lengths from short to long. This di- versity ensures that the evaluators trained by this can capture a broad spectrum of linguistic patterns and structural characteristics.
A.3.4 SEMANTIC DISTRIBUTION OF DIFFERENT DATASETS
To verify the diversity our data semanties, we use Sentence- BERT Reimers & Gurevych (2019) and HERRTopicGrrootendee
(2022) to extract the semantic embeddings of each sample in CreataSet- Base, and adopt t-SNE Van der Maaten Hinton (2008) to visualize the semantic distribution of these samples, as shown in Figure 12] Our dataset covers wide range of domains, which can effectively support the generalization the model's evaluation ability across diverse contexts.
A.3.5 DIVERSITY ANALYSIS OF AUGMENTED RESPONSES
To validate whether the responses in section 3.2 exhibit meaningful diversity, we use the Qwen3- Embedding-8B Yang al. (2025) model to compute semantic distances (cosine similarities) for the training pairs. The distances span 0. 0.19-0.94 with median of 0.64, showing that the responses vary across both fine-graing and coarse semantic differences. We also conducted human diversity assessment: 100 randomly sampled groups (k 5 responses each) of responses were rated by annotators on 1-5 scale. The diversity scores fall within 1-5, with an average score of 3.84. This confirms the responses not clustered around narrow semantic band. These results indicate data exhibit substantial and meaningful diversity, which supports learning both subtle distinctions (when responses semantically close) and broader conceptual differences (when they diverge).
22

---

Published as a conference paper at ICLR 2026
A.4 ADDITIONAL INFORMATION OF CREVAL
A.4.1 IMPLEMENTATION DETAILS
The backbone of CrEval is
Yang et al. (2024) and Low-Rank Adaptation (LoRA) Xu et al. (2024) with a=16 and r=8 applied to enhance efficiency. It is trained with DpepSpeed Rasley et al. 2020) Zero Redundancy Optimizer (ZeRO) Rajbhandari et al. (2020) Stage 2 and 6 (6) mix computation precision, using the AdamW Oppimizer|Loshhhiloo & Hutter (019) with 31 = 0.9, 8g = 0.999. The learning rate is Ie 5 with 0.1 warmup ratio, followed by cosine decay schedule. CrEval is trained for 2 epochs with a batch size of 2 and gradient accumulation steps of 8 on 8 NVIDIA H100 GPUs, while the max sequence length is set to 3072.
A.4.2 COMPARING TO THE STANDARD BRADLEY TERRY LOsS
We chose the loss formula in because is simple, efficient, and naturally compatible with the training paradigm of LLMs, where labels are treated as text outputs conditioned prompts. More- over, this formulation can be easily extended to multi-class preference settings. Fundamentally, we view both our loss and the Bradley-Terry (BT) loss approaches to the same underlying goal: mod- cling the probability of preference between responses. Both can be adopted for pairwise preference learning by maximizing the likelihood of the preferred sample.
To compare the results of different loss functions, we compare our method with BT loss. The results in Table 4 show that the two approaches perform comparably, with our method achieving better consistency. We consider that the specific form of preference modeling may not be the pri- mary bottleneck for creativity evaluation perfor- mance at this stage.
Method
F1
Kappa
Agree.
Consis.
BT Loss
0.722
0.593
0.703
0.846
CrEval-7B
0.732
0.601
0.745
0.920
Table The results of different loss functions.
A.4.3 RESULTS ON DIFFERENT BASE MODELS
To identify the most effective base model, evaluate several candidates, with the results presented in Table 5 Among them, Qwen2.5- Instruct-145 4B consistently delivers superior per- formance across all evaluation metries. Its ad- vantage may stem from its larger model capac- ity and instruction tuning, which allow to bet- ter capture the nuances of creativity in texts. Accordingly, we adopt the Qwen2.5 series as the base model for all experiments.
Method
F1
Kappa
Agree.
Consis.
Baichuan2-78 -Bhhhhtt
0.721
0.588
0.741
0.927
Llama3.1-885 1-8B-Chat
0.729
0.590
0.738
0.922
5-
0.732
0.601
0.745
0.920
Qwen2,5-Instruch
0.735
0.613
0.762
0.944
Table 5: The results of performance differ- ent base models. Agree. and Consis. represents Agreement and Consistency, respectively.
A.4.4 AN ANALYSIS OF CREVAL'S DECISION
Our dataset provides further value for deeply analyzing the latent factors and what specific features and seman- tic patterns CrEval learn to recognize as creative. For every pairwise com- parison in the test set (3K+ pairs), use an LLM (i.e,, DeepSeck-V3. V3.2) to identify which creative attributes were associated with the response CrEval judged more creative. We then aggregated these attributes and showed the most frequent attributes in Table 6 The distribu- tion reveals that CrEval not relying on superficial artifacts but consistently attends to semantic, stylistic, and structural patterns that align with widely accepted dimensions of creativity.
Category
Ratio
Category
Ratio
Unique Imagery
~21
Concrete Details
55%
Vivid Metaphor
199
Rich Visuals
5%
Unconventional Expression
122
Imaginative claboration
4%
Sincere Emotion
~10%
Distinctive Layers
~ 3%
Profound Symbolism
-6%
Precise word choice
29%
The top 10 most frequent attributes that CrEval judged as more creative.
Table 6:
23

---

Published as a conference paper at ICLR 2026
A.4.5 FURTHER ANALYSIS OF CREVAL-ENHANCEL MODELS
(1) How CrEval Enhance Model Creativity by Selecting Data Difficulty?
We further examine how the win rate varies with different ratios of hard reject samples in DPO training. As shown in Figure 13, the win rate increases slightly until the ratio reaches 30%, where peaks. Beyond this point, it declines rapidly, with the worst performance observed when all reject samples are hard responses. These findings indicate that incorporating an optimal proportion of hard samples can enhance learning creativity; however, careful balance is crucial for effective training.
75
70
87
68
66:25
66
Win Rate
65
60
60.87
6125
61
6100
6212
5912
50550
63.7
63.87
69.63
55
64 37
52
50
62.37
5162
12
50
45
CrEval
44337
40
GPT-4o
37
37
35
0
10
20
30
40
50
60
70
80
90
100
Hard Samples Ratio (%)
Figure 13:
Win rate curves incorporating different ratios of hard reject samples in DPO training, evaluated by CrEval and GPT-4o.
(2) Are CrEval-Enhanen Models Compromised in Reasoning Prone More Hallucination?
While Section 4.5 demonstrates a promising direction for enhancing creativity, we further investigate whether -enannee models are compromised in core capabilities by evaluating their reasoning ability and tendency to hallucinate.
Creativity and reasoning not contradictory and can be multi-dimensionally optimized. We evaluated the reasoning ability of the DPO- 70E30H model (introduced in Section 4.5) on the MATH Hendrycks et al.] (2021) and HumanEval Chen (2021) benchmarks, comparing it with the origi- nal base model under identical settings. As shown in Ta- ble 7 enhancing creativity did not compromise reasoning or factual accuracy. Instead, the CrEval-enhancee model exhibited slight improvement (though not statistically significant), possibly due to increased exploration during training, which may have led to more robust solution patterns.
Model
MATH
HvmanEval
Qwen2.5-7HB 5----ssttttt
59.76
72.97
DPO-70E30H
60.28
73.98
Table 7: The results of reasoning abili- ties on MATH and HvmanEval.
b. Enhancing model creativity did not increase hallucination.
We evaluated DPO-70E30H and its base model -0wen.55-BB-Inste Instruct on the TruthfulQA benchmark Lin et al. 20022) to assess poten- tial hallucination tendencies. Results in Table show comparable performance between the two models, indicating that the creativity enhance- ment in DPO-70E30H maintains factual in- tegrity without introducing hallucinations.
Model
ROUGE-L
BLEU
MC1
MC2
Owen2.5-7B-Instmen
48.5
50.06
47.61
64.76
DPO-70E30H
48.96
51.41
47.12
64.83
Table 8: The results of hallucination analysis.
A.5 QUALITATIVE RESULTS
To provide valuable intuition about what CrEval has learned and how it makes judgments, we include some test pairs along with CrEval's predictions and human preferences in Table Both success and
24

---

Published as a conference paper at ICLR 2026
failure cases included to offer more balanced and in-depth examination of the discrepancies between CrEval and human judgments.
Success examples highlight that CrEval can favor responses that are not only novel but also rele- vant-hhallmrrks of mcaningful creativity. Other failure cases suggest that when the creativity levels between two responses very close, CrEval may encounter challenges in accurately identifying subtle semantic preferences. We will further investigate these failure patterns in future work to better understand the nuances in creative evaluation.
Success cases
Instruction: Even though three generations in our family haven't 't understood music or played any instruments, can a child still learn one well?
Response I: I'm the only programmer in my entire ancestral line - going back 300 generations. Response 2: As long as the child has an interest and puts in the effort, they can definitely learn instrument well.
Ground Truth: Response
CrEval: After careful consideration, think Response 1 is more creative.
Instruction: Express the reason for giving up in one sentence.
Response 1: I'm tired - not physically, but because all my hard work hasn't 't paid off.
Response 2: gave up, it's because I lost it's because I understood.
Ground Truth: Response 2
CrEval: After careful consideration, think Response 2 is more creative.
Failure cases
Instruction: Please describe the distance between people in one sentence. Response 1: wall; bridge; an embrace.
Response 2: Amid the bustling warmth of human life, only a gentle word lies between heart and heart.
Ground Truth: Response 1
CrEval: After careful consideration, think Response 2 is more creative.
Instruction: Why do people always seem to lose one sock?
Response 1: In the sock world's ballroom, solo dancers always lose their way.
Response 2: If both went missing, you wouldn't i even notice.
Ground Truth: Response 2
CrEval: After careful consideration, think Response 1 is more creative.
Table Qualitative examples from the test data.
A.6 PROMPTS
We present the prompts used in this section. Table and which we adopt to synthesize responses with different creative levels (Section 3.3). Table [2] shows the prompt used in Section A.3.I where we employ it to score the creativity of innstruction-responss pairs and filter out those with low creativity scores. We adopt the prompt in Table [3] to generate creative responses for Ordinary Instruction-responss pairs (i.e., Infinity-Instruct) using GPTT-4o.
and II are ordinary and creative prompts,
25

---

Published as a conference paper at ICLR 2026
Ordinary Prompt (Prompt,)
Please reply the following instruction. The length of the answer should be about {[len (eiiininallesssssssss } words. Only give reply, do not output anything else.
Instruction: {[Instruction]]
Your reply:
# A
+ & [[llen/oringinalleepossssss } * A 15
T t
#
[Instruction)]
# 69 X
Table 10: The ordinary prompt (Prompt,) used to synthesize ordinary responses.
Creative Prompt (Prompt.)
You talented creative expert. Use your imagination respond to the instructions atively as possible. Creativity standard: novel, clever, and meaningful. Only give reply, do not output anything else. Please respond creatively the following instructions, and the length of the answer should be about (lenninilliial _eesponse))] ([len(oringinalleessosss)) words. Instruction: ([Instruction))
Your reply:
#
A 44 #
28
B @ (lennoringinnal_esponnn
$
#
[Instruction]]
#
Table 11: The creative prompt (Prompt,) used to synthesize creative responses.
26

---

Published as a conference paper at ICLR 2026
Creative Data Filtering Prompt
### Task Description:
You are a keen and rigorous literary critic responsible for evaluating the quality and creativity of {category)})
### Specific Requirements:
1. Assess whether the core creative elements are novel and meaningful by considering aspects such as word choice, word order, syntax, symbolism, rhetorical devices, and overall imagery. a text contains many creative elements, such as novel syntactic structures and expressions, it should receive a high score. Conversely, if it is merely a simple statement or lacks creative potential, should receive a low score.
3. Provide concise critical analysis of the text, followed by creativity score ranging from I to
4. Your response must be in JSON format, containing only two fields: "analysis" and "score", with no additional output.
5. Novel expressions and original meanings should be awarded high scores, while excessive repetition and commonplace expressions should be assigned low scores. If the creativity level is deemed moderate, the score should not exceed
Adhere strictly to all requirements; otherwise, the overseeing critic will impose severe penalties. #*# Given Text: {[Text]} ### Your reply:
### i #
T &
# * * { de
### $
1. A E M 4) #
X 1 2. 000-2444000110133333334441133313
*
3. 355
4. "
$
5.
4078
## [[Text)}
### #
Table 12: The prompt used to score and filter creative responses.
27

---

Published as a conference paper at ICLR 2026
Prompt for Creative Response Generation
You exceptionally talented expert in creativity. Utilize your imagination to respond to the given instructions the most inventive manner possible. Creativity Criteria: Your responses should be novel, ingenious, and meaningful. Reference Features Creativity:
1. Uncommon novel word choices and combinations;
2. Unique syntactic structures, including unconventional word order and sentence arrangements;
Rhythmic or phonetic elements, such as rhyme alliteration;
4. Clever rhetorical devices, literary allusions, quotations, or basor-based wordplay.
Specific Requirements:
1. The creative response must align with the given instructions.
There is no restriction on response length both longer responses (fluid, intricately structured,
etc.) and shorter ones (concise, witty, etc.) can exhibit creativity.
Provide only the response to the instructions without any additional commentary.
Instruction: {[Instruction]]
Your reply:
- K # A
1 ; # 69
1. 1
#
1. 01 8
2 #
A
$ -
# : (Instruction))
Table 13: The prompt used to generate creative response for Ordinary Instruction-response Pairs.
28

---

Published as a conference paper at ICLR 2026
A.7 DATASET EXAMPLES
For each source from our dataset, we present an example from Figure 14 the original form of the data from its source and our synthetic contents for training creativity evalu- ator CrEval (divided by the dashed line). We have omitted some texts for a clearer presentation. worth noting that what we provide a synthesis method. If necessary, one can use our method to synthesize more similar data for training.
Each example contains
Type: Existing Creative Data
Source:
Oogiri-GO
Instruction:
What will be humanity's final question?
A. % St 9 & f 6)
17 # 2?
Response:
What does this button do? i 1 th 2 f #5?
Synthetic Responses:
answer. (By MiniCPM--B--rxmppt))
E A t x
Humanity's ss ultimate question: the meaning of life. (By MiniCPM-BB-Prompt.)
A $ ž 48 65 & & 16 # 4 th $ x
What is the meaning of life? (By
4 i0 a x * + 22?
Have we ever truly understood love? (By Owen2.5-14B-Prompe
# MM E T $ ?
An example from type Existing Creative Data and source Oogiri-GO. We present texts in English and Chinese for better understanding. The original data listed in the upper part of the dashed line, and our constructed components are in the lower part.
Figure 14:
29

---

Published as a conference paper at ICLR 2026
Type: Existing Creative Data
Source:
Ruozhiba
Instruction:
Can person survive with only one heart left?
R 1É 3 M 75 ?
Response:
Yes, humans naturally have only one heart.
* 1
Constructed Instruction:
Based on the answer below, guess the possible question. Provide only the question, no extra
content. nn Answer: Yes, humans naturally have only one heart. in Possible question:
* 2 F 1
n oे tee é6 A8E
[o] AR dt M
7 *
in 1
$: # * 1 in HE 13
Synthetic Responses:
Do humans have only one heart? (By MiniCPM-2B-Prommt..
S $ PL कं - +'s R 23?
Why do humans have only one heart? (By MiniCPM-2B-Prompt.)
3 ft 2 A श DÉ?
How many hearts do humans have? (By Qwen2.55114--rromptt)
t IL !E?
How many hearts do humans have? (By Qwen2.5-14B-Proomt
A $ RÉ?
An example from type Existing Creative Data and source Ruozhiba, We present texts in English and Chinese for better understanding. The original data are listed in the upper part of the dashed line, and our constructed components are in the lower part.
Figure 15:
30

---

Published as a conference paper at ICLR 2026
Type: Creativity-Dense Texts
Source:
Short Texts
Response:
Sunflowers don't 't cry; even when down, they face the sun. ( A & % $ IR a pp lt i & t $4 # /
Constructed Instruction:
Please share your thoughts on maintaining positive attitude. i & $. 4/. 2 i # ta t t A 64 #
Synthetic Responses:
Maintaining positive mindset and an optimistic attitude helps us live more proactively.
(By MiniCPM-2B-Promaas
# & to de (9 #6 # E t ix A 66 & A for f aI ih A
@ A positive attitude makes you stronger, and persistence leads to success, (By MiniCPM-
2B-P rompt.)
* tæ & A 1t
# sh
A positive attitude brings energy and helps overcome challenges. (By Qwen2.5-14B-
Prompt,) $ AE
# # #
A
A positive attitude light that illuminates the path ahead; persistence keeps it shining.
(By Owenn.5514B.Proomtt
2 tl $ A A
S 1
#
An example from type Creativily-Demt Texts and source Short Texts. We present texts in English and Chinese for better understanding. The original data listed in the upper part of the dashed line, and our constructed components are in the lower part.
Figure 16:
31

---

Published as a conference paper at ICLR 2026
Type: Creativity-Denss Texts
Source: Lyrics
Response:
Millions of eyes are focused on miracle I begin awaken. Millions of sweat droplets are for purification, the courage before the eruption. don't 't mind the thorns ahead because giving up is not an option. ...... I will overcome because dare to fly, unafraid of tears, I won't 't stop, I will surpass because I dare to chase, even if it means bleeding, I won't retreat. #8 R 7
+ 7j RR
R 2
ME E 4 - 3. + 75
&
2e *, T K & 4
$, T
# i6, 7
Constructed Instruction:
Write lyrics for inspiring song that conveys determination and courage in the face of challenges, encouraging perseverance and the spirit of never giving up. it fJ: 3 मं t 5h Ais # i # 1 #J. & it di 7
$.
Synthetic Responses:
Move forward fearlessly, undaunted by storms or rainbows. No matter the challenges, we never give up. We must seize life, constantly striving and pursuing our dreams. Regardless of the hardships ahead, we face them with confidence and dignity. With more courage and stronger faith, we chase our dreams, making life more vibrant. Because life is worth fighting for. (By MiniCPM-2B-Promae
$ th i $
M. fr ;
ビ # 2 T
Mb
4pp
#
T
4
A.
b Through storms and struggles, we keep moving forward. \n Despite the hardships, we never back down. \n Setbacks and pain can't break us, \n because we have dreams in our hearts. in Let's fight together, (n and chase our dreams. \n Only with persistence can we achieve our own victory. (By MiniCPM 2B 2--Pr Promp,.)
#
11 #
M7 + # fl - 28 मं fi
\n
H
E & #
J ve in fi
'n #5
-
\n i8 C 15
\n B
t R
# #
\n S 26
The lyrics are as follows: \n \n Chorus: \n We have dreams, undeterred by hardships. The wind against only makes us soar higher. \n Let's charge forward together, bravely, Let dreams light our way, and courage guide We fear difficulty, and will never give up. \n Let's move forward together, chasing the light even in the dark. (By SiniCPM-2B-Prompg * io| F in In & t
chasing the light even through the darkest night. in
\n #1 शे # #
7.
- # P
a 18
&
$ T M.R.AATTKE T #
1 -
(continued on the next page)
An example from type Freativir-Densse Texts and source Lyrics. We present texts in English and Chinese for better understanding. The original data are listed in the upper part of the dashed line, and our constructed components are in the lower part. Owing to length constraints, the middle part of each response is omitted.
Figure 17:
32

---

Published as a conference paper at ICLR 2026
Type: Creativity-Dense Texts
Synthetic Responses (continued);
! [Song of the Wind Against Us] \n An (Prelude) \n In the dark of night, we see the first light dawn. An adversity, we hear the song of our dreams. \ With every step, we walk in unwavering faith, \n Letting the flame in our hearts guide our way. ..... (Epiilguue) \n As the dawn breaks, the world will shine brighter, in For we have each other, writing something extraordinary together. \n Let's 's join hands, moving toward the dream's 's shore, \n And in the headwind, shine our brightest light. [ 22 of u 66
n
& In #1
6
# L # #
L
in 1 $,
Figure 17: An example from type Creativity Dense Texts and source Lyrics (continued).
Type:
Creativity-Dena DDenee Texts
Source:
Ancient Poetry
Response:
Lament on Youzhou Tower
No sages remain to guide the past, voices echo from the future. \n Alone beneath the boundless sky, mourn-lost in time, lost in tears.
ビイ S 1 #
Constructed Instruction:
Please write four-line ancient poem about the passage of time and loneliness, with a heavy and emotional tone.
i 5 - $ & it o ot fo (4
$ # I
Synthetic Responses:
Rivers never cease, time flows on. Tears of solitude through the years, unknown in the endless night. (By MiniCPM--B-2roopt.,
I ग. iu T BJj [9]
F Mf i # A #6 i
Time flies, nights grow long, lost in thought alone. (By MiniCPM-2B-Prompt)) # A 2 * R. - +
$ A t , B #
@ lone shadow with the setting sun, as time flows like water. \n In the silent night, listening wind and rain, letting worries drift away. (By Owen2.5-14B-Prompph Ra. z i * M. - \n A ve of A df) is $ H $ M as $ $ 4% A4.
Time flies, shadows linger. Laughter fades-where has gone? \n Midnight bells chime in solitude, leaving only tears to drench my robe. (By Qwen2.5. 14B Prmot,.) B A 36 * # E # Ao in 2 # t $ 4
Figure 18: An example from type Creativity-Demi -Dese Texts and source Ancient Poetry. We present texts in English and Chinese for better understanding. The original data listed in the upper part of the dashed line, and our constructed components are in the lower part.
33

---

Published as a conference paper at ICLR 2026
Type: Preativity-Dense Texts
Source:
Modern Poetry
Response:
Distance \n universe in pile of rubble. 'n Cracks magnified infinitely, are \n bacteria under a microscope, seeing our sorrow (n in a giant's 's hand. Two people, i a grain of rice apart- yet lifetime away. Jf 45 L A T 1 - # BE
-
Constructed Instruction:
Write modern poem exploring the theme of distance and the relationship between the vast universe and human insignificance.
i # - ml fR # 3 #
&
Synthetic Responses:
In the vast cosmos, humanity's flecting existence drifts like tiny plancts among the stars explorinn the universe's depths while pondering our own insignificance. (By MiniCPM-2--Pres t #
1 i $ $
The universe is vast, humanity is small, distance is great, but hearts connect. (By MiniCPM-2B-Promppt
$ * R Distance-ss gentle leap through the cosmos, where stars are far yet near. \n Humanity speck of dust, gazing at the unreachable sky, yet somehow connected. (By Qwen2.5- 14B- Prompt.))
A ## Ap
2 & T N 3Z
O The universe is vast and boundless, drifting like dust among the stars. n A lonely journey across light-years you and apart. \n In endless darkness, we seek a glimmer of light. \ Distance is not just space it's bridge between hearts. (By Qwen2.5- 14B-Promot.)
$ $ R A #
\n D # ta Ms \n E
$
|n A T
Figure 19: example from type Creativity Dense Texts and source Modern Poetry. We present texts in English and Chinese for better understanding. The original data are listed in the upper part of the dashed line, and our constructed components are the lower part.
34

---

Published as a conference paper at ICLR 2026
Type: Creativity-Dens Texts
Source: Prose
Response:
Time gently lifts the veil of youth- eyes filled with innocence deepen with wisdom. Time carves its soft foreheads once untouched by years now glow with maturity. ..... A gentle breeze, soft moonlight, tranquil waters, and graceful melodies-wwid who wouldn't long for such a life? But there is also beauty in bearing the weight of mountains, enduring ruthless trials, and emerging from life's fiercest storms with unshaken strength. That, too, is desire of the soul.
y A tt # 42
T i T
/ -
2
Constructed Instruction:
Write prose piece the passage of time and its impact on life, exploring the insights gained through experience and growth.
i S 3 & /
Synthetic Responses:
Time is ruthless blade, leaving its mark all living things plants, animals, and hu- mans. Its passage erases moments, fades memories, and inevitably takes life itself. Yet, every fleeting joy and irreplaceable imprint it leaves is a treasure unique to existence.
Time is relentless yet beautiful. It pushes us forward, filling our lives with experiences and memories. Let's cherish every moment, engraving it into the story of our lives as a vital part of our growth and journey. (By MiniCPM-2B-P omot.)
S A -e
- lE A 1E
# A
i 4 + - 9
Time passes like a silent feast, quietly taking away our youth while leaving behind endless memories. Life is journey, with each stage offering different views and emotions. We should cherish every moment, face each challenge with gratitude, and let the passing of time become our greatest treasure. (By MiniCPP--2B-Prompp. t p.
# イl i
$ : |
T
i
K $ *
(continued on the next page)
An example from type Creativity-Denss Texts and source Prose. We present texts in English and Chinese for better understanding. The original data are listed in the upper part of the dashed line, and our constructed components are in the lower part. Owing to length constraints, the middle part of each response is omitted.
Figure 20:
35

---

Published as a conference paper at ICLR 2026
Type: Creativity-Dense Texts
Synthetic Responses (continued):
@ Time flies, pushing us forward with unstoppable force. As it passes, we transform from innocent children to wise adults. Along the way, we've faced countless storms and moments of sunshine, all of which have nourished our growth and strengthened our spirits, making us more resilient and mature.
Time may take away our youth, but also brings the wisdom understand the true meaning of life and cherish what have now.
of growth, helping (By MiniCPM-2B- proptt.))
$ A t
B a in lg
1 AA & È
T htw.::/hh.......................
M. mf
*
M ................................ 3 #2T
2. # 1 3 サク
Time flows silently, like water, never speaking yet slowly unfolding the story our lives. We grow from innocent children to inexperienced youths, then into mature adults, and fi- nally, elderly ones with gray hair. Each stage is a gift from time. every embrace, every farewell, is an essential part of life. Time reminds us that life is like a journey, where the value lies not in the destination, but in the scenery along the way and the mindset with which appreciate the beauty of life.
It teaches us that
experience it. Only through the passage of time can truly
A * #2 H A E HELL セ A $ # -
1 AA
T. 3F,-::088~~.1111110,333300013333
-
Figure 20: An example from type Creativity- -Desse Texts and source Prose (continued).
36

---

Published as a conference paper at ICLR 2026
Type:
Ordinary Instruction InstructionRResponse Pairs
Source:
Infinity-Instrus
Instruction:
Create a catchy title for the following topic: saving money for the future.
A 12 7 i. - A 19 fi
&
* * #4
Original Response:
"The Secret Future Wealth: How Save Achieve Your Dreams" & 9 m "
* M 3 81 f to
Enhanced Response:
Time Investment: Weaving the Blueprint for Future Wealth B 2 f T 2 * ih o 1E R
Synthetic Responses:
Savings for the Future: Your Smart Investment Choice. (By MiniCPM-2B-Prompt,,
f v *
2 6 5] 2 th oे #
Invest in the Future. Save for Wealth. (By MiniCPM--BB-Prompt)) fit & ui $
X म R
Smart Budgeting, Saving for Golden Future. (By Qwen2.5-14B-Proompt
5 xe fit & * * tb # $
Time Bank: Saving for the Future with Wise Investment (By Owen2.5-144-Prron
B fT
filt * 4 E 2 # o
Figure 21: An example from type Existing Creative Data and source Infinity-Instret We present texts English and Chinese for better understanding. The original data are listed in the upper part of the dashed line, and our constructed components are in the lower part.
37

---

Published as a conference paper at ICLR 2026
Labeling
Query: Describe blooming flower poelic language.
rragrance Iepertal, graceful Figure is sty. Even the auturn cold drives the leaves to old, It still holes the starens and andles the branchos.
retatiedd
Greathety
petals gently spread out, splendor and tranquility of 11fe.
brocade, delicately deplcting
Creatiity O4 C208a
gentle norning Light, the Like fragrance, gentle and firm.
Creakity
dancing drese, colorfat, like the widspen sprinn,,
Creatityy Ot Ca
bloceiing dreaws, fragrance spreads quietly, singing and dancingg with the heartbeat of spring.
Like
brightest
Cz Ca
Confidence O1 C4
Submit
Page 19
Figure 22: Human annotation screenshot.
A.8 DETAILS OF HUMAN ANNOTATION
The goal of human annotation in a reproducible and structured way using pairwise comparisons and consensus- aggrega- tion. The human annotation was conducted through a professional data annotation company. The recruited 30 qualified annotators are from 18 different majors, aged from 21 to 29. The group in- cluded 12 male and 18 female annotators. Though drawn from diverse backgrounds, they achieved strong inter-annotator agreement (ICC 0.75), which is typically considered highly consistent (>= 0.75). While it is difficult to empirically prove that their views represent the global majority, such high consensus among 30 diverse annotators marks notable advance prior work that relied on fewer (2 Stevenson et al. 2022) or 10 Chakrabarty et al. (2024) annotators.
eliminate subjectivity but model shared human judgment
Figure 22 llusttates the interface used during the human annotation process. Annotators presented with group of responses, along with the corresponding instructions. They were instructed carefully read both the instructions and the responses, and then give creativity score. The interface was designed to be minimal and intuitive, allowing annotators to focus on the content rather than the mechanies of annotation. Each annotator compensated at rate of 50 RMB/hour.
38

---

Published as a conference paper at ICLR 2026
A.9 LIMITATIONS
Although we have established a viable evaluation method for textual creativity, understanding and analyzing the core of text creativity remains challenge that has not yet been fully addressed by machines or even humans. Our dataset also cannot cover all possible creative scenarios that may appear in texts, which requires collective efforts from the community in the future. We hope that through improved creativity evaluation, we can ultimately enhance the model's ability to understand and generate creativity, but the true mechanisms behind this process remain unknown and will be our future focus. In addition, there are more ways of using CrEval as a plug-in to improve any LLMs' creativity, egg. using it as a reward model and refining LLMs via PPO. We leave this for future work.
A.10 POTENTIAL SOCIETAL IMPACTS
CreataSet and CrEval could advance the development of Al systems that better generate creative content, benefiting education, entertainment, and content creation. However, improved automated creativity assessment may also lead to over-optimization for maciile-friendly metrics, potentially stifling genuine human creativity or reinforcing biases in machine-genn -eeeernee content.
A.11
LLM USAGE
In this study, Large Language Models (LLMs) were utilized as tools to assist with two specific tasks: 1) the construction and cleaning of datasets. The specific methodologies employed for these tasks described in detail within the main text, and every effort was made mitigate potential risks. 2) the polishing of the manuscript's language to improve clarity, including tasks such as rephrasing sentences, checking grammar, and improving textual flow. It is important emphasize that the LLM did not author the manuscript or generate any of the core scientific content, analysis, or conclusions. All intellectual contributions remain solely with the authors.
39


Reviews:
[{"review_id": "tE7i5yxurx", "reviewer": "Reviewer_Re9M", "summary": "This paper tackles the problem of evaluating text creativity by building a large-scale dataset (CreataSet and its extended version, CreataSet-Ext) and training a pairwise creativity evaluator CrEval. Each instruction is paired with multiple responses generated by different models under both ordinary and creativity-oriented prompts, with human-annotated labels serving as ground truth. The paper finds that CrEval aligns well with human judgments, generalizes to unseen domains, and can further help enhance model creativity. The results show that creativity-oriented prompts and stronger models tend to produce more creative responses, and data composition meaningfully affects CrEval’s performance.", "questions": "NA", "limitations": "", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The experiments are fine. CrEval consistently outperforms strong baselines including large proprietary models across proposed metrics. The paper includes appropriate ablations, along with OOD tests on external datasets. The authors further show CrEval can be used to improve model creativity", "weaknesses": "-  The paper mentions constructing tuples (I, R1, ..., Rk) but does not specify the exact value of k used in experiments. How many responses are generated and used per instruction? Does this vary across data sources?\n- In constructing CreataSet-Ext, they prompt two models to generate more responses for augmenting each instruction. But there is no testing of whether these k responses actually exhibit meaningful diversity. If these responses are similar to each other, it could limit the model’s ability to learn fine-grained distinctions.\n- At the beginning of the paper, they define creativity as “ideas or artifacts that are new, surprising and valuable”, which I personally appreciate. While novelty and surprise are well-measured, the “valuable” aspect is not explicitly evaluated. Aside from gpt-4o-mini filtering, there is no systematic check for helpfulness. A response can be novel but incoherent or unhelpful, which weakens the notion of genuine creativity.\n-  The two core assumptions are validated with only 50 samples each with three annotators. This seems insufficient given the scale of the dataset (1M+ samples).\n- The paper treats creativity evaluation as a pairwise comparison task without deeply analyzing what constitutes creativity or providing any meaningful understanding. What specific features, semantic patterns, or structural elements does CrEval learn to recognize as creative?\n- Personally the task are a bit strange and adhoc. I would not treat them as something that requires creativity \n-Data contamination could inflate the performance of models being evaluated, especially for proprietary models whose training data composition is not fully disclosed.\n- I am not convinced LLM eval is a surrogate for human eval. Are your 18 humans experts ?? Without knowing much we cant conclude anything", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gtO3p4Vcxm", "reviewer": "Reviewer_AWwk", "summary": "This paper introduces CreateSet, a dataset with evaluation pipeline including a performant evaluator—CrEval. The author enriched the responses to existing prompts by generating more responses (and formed CreateaSet-Ext). The work annotated over 3,000 samples, with more using weak labels. The author extensively studied if the dataset helped create a more power evaluator.", "questions": "- Line 43, How is problem-solving a single domain?\n- It is important to break down the language used in the dataset. Is this dataset 100% (Simplified) Chinese?\n- Line 47 \"most methods evaluate creativity at the model or subject level rather than at the level of individual responses\" Please better support this claim.\n- How are deepseek models \"Proprietary LLMs\"?\n- How are annotators compensated?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- A reasonably well-curated dataset with good mix of topics for coverage.\n- Good amount of human labels, and put into good use in training evaluator models with good comparison with other metrics and very extensive model lineup. I appreciate the authors showing many proprietary results. \n- Evaluation of the trained evaluator is complete and convincing.", "weaknesses": "- The definition of creativity, which is subjective, should be detailed better in this work. This is a key bottleneck of this work's quality and rigor.\n- CrEval are comparison pairs of creativity. However, there might be some value to an absolute scale of creativity, especially if we want to rank model responses quickly. Motivation here is less clear.\n- This work suffers from a few overstatements:\n    - The paper prides itself over context awareness (i.e., showing a prompt when evaluating responses for creativity), including using an entire Fig 1 to emphasize. But the authors fail to explicitly demonstrate if this is common.\n    - \"87 domains\" is a stretch. \"Domain\" is defined too loosely across the paper.\n- While I believe performance parity with o3 is not so important given the model scale difference, the authors could use more motivation on why a smaller model as an evaluator is important when a large model can do better. \n\nI found these weaknesses not fundamental and am happy to update my views during the coming discussion period.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cXaHooiGIC", "reviewer": "Reviewer_s6ik", "summary": "This paper focuses on evaluating the creativity of LLM-generated text. Specifically, it identifies two key challenges in evaluating creativity: 1) ensuring consistency in creative annotation for human experts; and 2) training a large-scale model for evaluating creativity given the scarcity of evaluation corpora. To address these challenges, this paper proposes CreataSet and CrEval, a large-scale pairwise comparison framework for cross-domain text creativity evaluation and a logic learning model (LLM)-based evaluator, respectively. Testing on various baseline models validates the effectiveness of CreataSet in evaluating creativity.", "questions": "Please refer to Weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) Large-scale and multi-domain dataset. CreataSet includes 100K+ human-level and 1M+ synthetic creative instruction-response pairs across 87 domains, which is promising in providing a scalable fundation for studying creative generation and evaluation.\n2) Improved human label protocol. The proposed context-aware pairwise comparison protocol improves inter-annotator consistency (evaluated by ICC).\n3) Comprehensive experiments. Multiple metics are applied for providing an through evaluation, such as F1score, Kappa score and Agreement rate.", "weaknesses": "1. The rules for quantifying creativity are not differentiated across different domains.For example, creativity is manifested differently in poetry and scientific writing. Future work should further differentiate the measurement of creativity for each domain.\n2. Insufficient example/failure case analysis. The paper presents overall statistics but does not systematically list typical examples of discrepancies between CrEval and human behavior.\n3. Insufficient generalization analysis. The paper lacks an assessment of the enhancement effect of CreataSet on diverse baseline models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ea0rRsMwYW", "reviewer": "Reviewer_RwCK", "summary": "The paper tackles the challenge of automatically evaluating textual creativity in large language models. Current creativity assessment rely heavily on human judgement, which is subjective, inconsistent, and expensive.\nThe author propose a new evaluation framework which is a context-aware, pairwise comparison method that judges creativity between two responses to the same prompt. The author propose a new dataset containing over 1 million instruction-response pairs across 87 domains, mixing human and synthetic data. Over this, the author train a new LLM-based creativity evaluation on the dataset.", "questions": "1. The CreataSet-Base domain distribution seems quite unbalanced. Is creativity really isotropic across categories? Some subdomains (like business writing vs. poetry) might not be equally suited for creativity evaluation. A more fine-grained per-domain analysis would help clarify this.\n2. Were the choices of Qwen2.5-14B-Instruct and MiniCPM-2B-SFT arbitrary? Why were these two particular models selected for response generation and augmentation?\n3. The pairwise comparison framework works well for controlled benchmarking, but its practical usability might be limited. Do you think it could be extended to continuous or scale-based scoring while preserving consistency?\n4. It would be interesting to test CrEval on more classical creativity benchmarks—like Alternative Uses or other divergent-thinking tasks—where the “good” and “bad” responses are clearer.\n5. In real-world creative contexts, can CrEval replace or complement the Consensual Assessment Technique (CAT)?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The overall idea is clear and well-motivated. Focusing on fine-grained text-level creativity rather than broad model-level creativity makes the work much more practically valuable.\n2. The authors explicitly explore the challenge of human annotation inconsistency through empirical analysis, which shows solid motivation and thoughtful experimental design.\n3. The two key assumptions used for weak supervision are both empirically validated, and the paper carefully considers position bias in pairwise evaluation that is easily to be ignored.\n4. The experimental setup is comprehensive and convincing. The baselines cover a wide range of categories—traditional metrics, general LLMs, and fine-tuned evaluators. The study includes analyses of consistency, ablation on data composition, OOD generalization, and even the ability to enhance creativity through DPO training. Altogether, this gives the results strong credibility.", "weaknesses": "1. There’s no discussion of whether the model maintains interpretative correctness—in other words, does it still “understand” why a response is creative under its defined criteria?\n2. The OOD evaluation is relatively small in scale, which limits how strongly the generalization claim can be made.\n3. Some references, such as Zhao (2024), have already been officially published, but the citations still point to arXiv versions. The bibliography needs to be updated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)