POEMetric: The Last Stanza of Humanity
Authors: Bingru Li
Han Wang
Hazel Wilkinson
Venue: ICLR 2026 Poster (2026)
Keywords: Poetry evaluation metrics
poetry generation
large language models
dataset and benchmark
Primary area: datasets and benchmarks
TL;DR: The first comprehensive framework for LLM poetry evaluation

Abstract: Large Language Models (LLMs) can compose poetry, but how far are they from human poets? In this paper, we introduce POEMetric, the first comprehensive framework for poetry evaluation, examining 1) basic instruction-following abilities in generating poems according to a certain form and theme, 2) advanced abilities of showing creativity, lexical diversity, and idiosyncrasy, evoking emotional resonance, and using imagery and literary devices, and 3) general appraisal of the overall poem quality and estimation of authorship. We curated a human poem dataset - 203 English poems of 7 fixed forms annotated with meter, rhyme patterns and themes - and experimented with 30 LLMs for poetry generation based on the same forms and themes of the human data, totaling 6,090 LLM poems. Based on POEMetric, we assessed the performance of both human poets and LLMs through rule-based evaluation and LLM-as-a-judge, whose results were validated by human experts. Results show that, though the top model achieved high form accuracy (4.26 out of 5.00, with Gemini-2.5-Pro as a judge; same below) and theme alignment (4.99), all models failed to reach the same level of advanced abilities as human poets, who achieved unparalleled creativity (4.02), idiosyncrasy (3.95), emotional resonance (4.06), and skillful use of imagery (4.49) and literary devices (4.67). Humans also defeated the best-performing LLM in overall poem quality (4.22 vs. 3.20). As such, poetry generation remains a formidable challenge for LLMs.

Published as a conference paper at ICLR 2026
POEMETRIC: THE LAST STANZA OF HUMANITY
Bingru Li
Han Wang"
Department of Linguistics and Communication University of Birmingham
Department of Information Engineering
and Computer Science
bx1 12142 60 student www.. ac. uk
University of Trento han, wang@unitn. it
Hazel Wilkinson
Department of English I Literature Institute of Data and Al (IDAI)
University of Birmingham
h.j. wilkkknso@bhamm.c
ABSTRACT
Large Language Models (LLMs) can compose poetry, but how far are they from human poets? In this paper, we introduce POEMetric, the first comprehensive abilities
framework for poetry evaluation, examining 1) basic in generating poems according to certain form and theme, 2) advanced abilities of showing creativity, lexical diversity, and idiosyncrasy, evoking emotional resonance, and using imagery and literary devices, and 3) general appraisal of the overall poem quality and estimation of authorship. We curated a human poem dataset 203 English poems of 7 fixed forms annotated with meter, rhyme patterns and themes and experimented with 30 L.MSs for poetry generation based on the same forms and themes of the human data, totaling 6,090 LLM poems. Based POEMetric, we assessed the performance of both human poets and LLMs through rule-based evaluation and LLM-as-a-judge, whose results were validated by human experts. Results show that, though the top model achieved high form accuracy (4.26 out of 5.00, with Gemini-2.5- models failed to reach the same level of advanced abilities as human poets, who achieved unparalleled creativity (4.02), idiosyncrasy (3.95), emotional resonance (4.06), and skillful use of imagery (4.49) and literary devices (4.67). Humans also defeated the best- perforrffming LLM in overall poem quality (4.22 vs. 3.20). As such, poetry generation remains formidable challenge for LLMs.
judge; same below) and theme alignment (4.99), all
INTRODUCTION
Large Language Models (LLMs) Hurst et al. (2024); Grattafior et (2024); Team (2025b); Anthropic (2024): Guo et al. (205)): Qwen et al. (2025) have demonstrated outstanding capabilities in reasoning and logic tasks, ranging from solving math problems coding. Nevertheless, less attention has been allocated to I.LMs' abilities in terms of arts and humanities, let alone advanced tasks such as literary writing. Among the various literary forms, poetry, has long stood the ultimate testament to linguistic artistry, demanding the perfect synthesis of verbal precision, emotional resonance, and cultural literacy within constrained forms. As such, compared with other forms such as essays and fictions, the compact and formulaic style of poetry makes it valuable lens through which we are able to examine the generative abilities of LLMs.
While LLMs have excelled in text generation across numerous domains, the generation of authentic poetry remains challenge. Though extant literature Belouadi Eger (2023); Ling & Zhang (2022)) Yu et al. (2024)) has demonstrated the high formal accuracy in the meter and rhyme patterns of LLM generated poems, there is still a lack of creativity and diversity Walsh al. (2024b); CCen et al. (0024). Moreover, little has been explored in terms of evaluating the artistic beauty well
*Corresponding author.

---

Published as a conference paper at ICLR 2026
intrnction
LUN ao jurgge
Form Accuratty
Creatilty
Id egyecrsty
Ovara Iise
Neter
Ledial Steenity
Emetional
ratherih Eiteration
Thems Atgrentt
Figure 1: POEMetric. It comprises 10 metrics, including 1) basic instruction-following abilities (form accuracy and theme alignment), 2) advanced creative abilities (creativity, lexical diversity, idiosyncrasy, emotional resonance. and use of literary devices and imagery), and 3) general appraisal (overall poem quality and authorship estimation). Human- and LLM-authored poems are compared through rule-based evaluation and L.-M-as-- auud-d, whose results are validated by human experts.
as author intentions and emotions in the poems generated by LLMs, which are in fact the essence of poetry writing Greene et al. (2012). Therefore, the central inquiry of this paper lies in whether state-of-the-ar LLMs can transcend mere syntactic competence to achieve what T.S. Eliot termed "the auditory imagination" " [Eliot (1986)- the fusion of sound, sense, and cultural memory that distinguishes enduring poetry from mere grammatical arrangement.
To address these issues, we propose POEMetric, the first comprehensive metrics for the evaluation of poetry, which examines 1) basic instruction-following abilities (form accuracy and theme alignment), 2) advanced creative abilitics (creativity, lexical diversity, idiosyncrasy, emotional resonance, and use of literary devices and imagery), and 3) general appraisal (overall poem quality and authorship estimation). To the best of our knowledge, POEMetric is so far the most comprehensive evaluation framework for poetry, making up for what has been lacking in previous metrics in terms of poetic beauty, personal characteristics, and emotional effects. Based on POEMetric, we compared human- written and LLM-ggeerated poems through rule-based evaluation, with self-written algorithm for automated form detection, and L-a---a-ae (Gemini-2.5-PPyg whose results were validated by human experts. An illustration of POEMetric shown in Figure !
We also curated human poem dataset, comprising 203 human English poems of fixed forms, which spans the past 200 years and ranges from canonical works to less-known recent creations. According to the same form and themes in the human data, we prompted 30 LLMs for poetry generation. Evaluation results showed that, though top LLMs were able to achieve high scores in terms of form accuracy and theme alignment- for example, Gemini-2.5-PP topped at 4.26 and 4.99 out of 5.00 (with Gemini-2.5-Pre as judge; same below) they still struggled to attain the same level of advanced creative abilities as human poets, where the latter excelled in creativity (4.02), idiosyncrasy (3.95), emotional resonance (4.06), and skillful use of imagery (4.49) and literary devices (4.67). Human poets also defeated the best-performing LLM, i.c., DeepSeek-RI. 1, in terms of overall poem quality, at 4.22 vs. 3.20. While both evaluators could recognize some original poems, human poems remained markedly distinguishable from LLM verse, with distinct patterns emerging in areas such as emotional resonance and idiosyncratic use of language. The agreement among based evaluation, LLM-as-a-judge and human experts validates the effectiveness of POEMetric.
To sum up, our contributions can be summarized as three-fold:
We propose POEMetric, the first comprehensive framework for poetry evaluation, covering basic instraction-followinn abilities, advanced creative abilities, and general appraisal;
We curated a human poem dataset, carefully annotated with the forms (including meter and rhyme patterns), themes, and imagery:
We designed an algorithm to automatically detect the formal patterns of poems. We have provided the code and the public-domain human poem dataset as supplementary materials ensure reproducibility.
2

---

Published as a conference paper at ICLR 2026
2 RELATED WORKS
Poetry generation with Language Models Some attempts have been made to train Language Models (LMs) to generate poetry that adheres to formal constraints such patterns of meter, rhyme, and style. For example. ByGPTS (Belouadi & Eger 2023). PoeLM (Ormazabal et al. 2022). GPoet (Popescu-Bai Belis al. 2023), and a GPT-22-baed model (Possi et al. 2024) integrated structural metrics such as rhyme and meter into generation. (Bena & Kalita 2020) fine-tuned GPT-2 to express
and elicit emotions in poems.
Language-specific adaptations have yielded high-quality poetry in low-resource languages (e.g..
Pashto Ullah al. (2024). Arabic (Alyafeai et al. | Beheitt HajHmidal 2023). Vietnamese
Huynh & Bao 2024). Czech (Chudoba & Rosa (024) and culturally nuanced styles (c.g.. classical
Chinese poetry (Ling & Zhang [022] Yu et al. 2024 Wang et al. [0166] Yi et al. 2017 Zhang
et al. 2017| Liu et 2018| Yi et al. 2018 Liao et al. 2019 Liu et al., 2019 Yang et al. 2023
Fang 2024). limericks (Lo et al., 2022), and Homeric poetry (Lamar & Chambers, (019)). However,
models struggle with stylistic variation and creativity (Walsh et al. 2024b Chen et al. 2024 Cao &
Cheng 2024
Evaluation of poetic quality Combining objective metrics (e.g., meter and rhyme accuracy. BLEU (Beheitt & HajHaida 2023 Liu et al. 2019), perplexity (Ormazabal et al. 2022 Liu et al. 2019) and human judgments have provided robust evaluation. More recent metries include ProFTAP (Deng jet al. 2024) which adopted Turing- Besing-test-ing test-eessssppp inspired frameworks to evaluate poetic indistinguishabilit
from human works. Others et al., (022)) applied LLM--s-a-judge in evaluating LL.M-genrrtte poems, examining fluency, meaning, coherence, relevance, and aesthetics. Still others fine-tuned LMS for evaluation, as in (Sawicki et al. 2023) who fine-tuned GPT-3 to classify if an L.LM-generatee poem was written in the style of Whitman. In addition, diversity evaluations revealed gaps in semantic and formal variance and artistic creativity compared human-written poetry (Walsh et al. 2024b
Chen et al. 2024).
To sum up. extant poem evaluation metries are limited to meter and rhyme accuracy and formal diversity, or overly general aspects of text generation such as fluency and coherence, whereas more advanced and nuanced abilities are at the essence of poetry composition. As L.LMs have proven competitive in writing in certain poetic forms, metrics that look at more advanced, poem-specific abilitics in such areas as creativity, author intentions and emotions, and poetic aesthetics such as use of imagery and literary devices (Greene al.) [012)) which are particular poetic features, are urgently required; this is where our POEMetric comes into play.
THE HUMAN POEM DATASET
In this section, we report on how we collected the human poem dataset, covering 7 poetry forms, An elaboration on the features of these forms can be found in Appendix A Our focus on fixed-form poetry was designed to address a fundamental challenge in creative evaluation: ensuring the benchmark is rigorous and diagnostic. By first evaluating poetry within these constrained forms, we establish a quantifiable baseline that crucial for systematically developing and validating the more subjective metrics needed for the ambiguous challenge of free-verse poetry.
Following previous research (Walsh et al., 2024b a). we collected the poems from two online databases, the Poetry Foundation and the Academy of American Poets]] totaling 1,309 poems. Due to the fact that not all human poems were strictly written according to a typical meter and rhyme pattern, we designed an algorithm to detect the meter and rhyme patterns for each poem, and only kept those that followed a certain prosodic pattern. In the end, the human dataset comprises 203 poems, which includes 95 ballads, 9 ghazals, limericks, 3 pantoums, sestinas, 71 sonnets, and 12 villanelles, as shown in Table [ The time frame of this dataset spans from as early as the 1800s to the present time, including both well-known artworks written by famous poets well as knowee or newly released poems. We also annotated the themes of the poems by drawing on public analysis (e.g., Poem Analysis (web) a) and Poem Hunter (web) b)). and the imagery used by designing a list of common imagery in English poems. An example of the human poem data is illustrated in Figure 2
hectps://www.pot ry foundationnnn .o..cr https::/poets..
3

---

Published as a conference paper at ICLR 2026
Table The distribution human poems by form and source
Source Form
Ballad
Ghazal
Limerick
Pantoum
Sestina
Sonnet
Villanelle
Poetry Foundation
86
6
/
6
61
10
Academy of American Poets
3
10
2
Title
Auther Stidine
Form
Source Partry Foundatien
Please o
do rhin tegether Htemonisua harvest, Andloreed liglight anslwwhttc combinee the the weeath to male tround and
according followiing
Theme The preet commerroocter bnddrape close to powd's hearl, andic ponseffll
Form: (form)
Meter Pattern bembic pentammnter
Meter Meterr (Meter Pustern) Rhymee (Rhyyne Pattent
To me my troubled life dath now appeer Like aesacaooe distingiilll
Themee-Themme
summita hume Apoundiine the biue hor horizun places where even io miveler puracdert to steer Whenwot migrant birs pasa sung. get closed sendovenst
Onl the poern withut engritles. 20 not indlude any other contents.
Imagery brrd wint, sky. p918s 903
ABBMEBAACDECEE
Human Poem Data
LLM Generation Prompt
Figure 2: An example of the human poem data and the generation prompt for LLMs. On the left are the related data annotated about poem, including author, title, poem content, source, form, meter pattern, rhyme pattern, theme, and imagery. Based these data, the prompt for LLMs to generate poems is curated in the template on the right.
4 POEMETRIC
The paradigm of POEMetric shown in Figure The main part of the figure illustrates the 10 dimensions proposed, ranging from basic instruction-followingg abilities to advanced creative abilities and general appraisal, which will be discussed in 4.1 These dimensions are deeply rooted in literary theories and have been important in literary critique of poems (Greene al. 2012) the dimensions in advanced creative abilities in particular and yet overlooked by previous studies, as reviewed in Section 2 We also apply both objective and subjective evaluation techniques to triangulate our methodology, including a handcrafted algorithm quantitative metric, and LLM-aa-a-juddee and human experts for the more nuanced evaluation. Details are presented in Section 4.2
4.1 THE DIMENSIONS OF POEM EVALUATION
Basic cnntruuctionfolllwwiii abilities These include the examination of how well a poem written in response to the given prompt, specifically in terms of the extent to which they follow the instructions on the form, including meter and rhyme where applicable, and the theme of poem.
Advanced creative abilities POEMetric systematically and quantitatively applies the core, often qualitative, elements from traditional literary criticism to the more sophisticated evaluation of poetry generated by LLMs. To evaluate the more advanced abilities in poem creation, we assess the creativity, lexical diversity, idiosyncrasy, emotional resonance, and the use of literary devices and imagery. Creativity looks at whether the poem written in a novel and creative way. Lexical diversity measures if the poem uses varied vocabulary. Whether poem demonstrates personal characteristics of the author measured by idiosynerasy, and whether a poem evokes emotional resonance also examined. Literary devices commonly used in human poems, and here we evaluate four typical techniques, i.e., simile, metaphor, personification, and allusion. The use of imagery shows a poem trigger vivid image and engage the readers' senses. These 6 dimensions for advanced creative abilities that we have chosen are intended as a distillation the features on which literary critics typically focus in the analysis of poetry, often known 'Practical Criticism' (Richards) 2014).
4

---

Published as a conference paper at ICLR 2026
General appraisal Apart from the above fine-grained metrics, also ask two general questions. The first is to ask if a poem is good or not to evaluate its overall quality, and the second is to estimate the authorship of the poem, i.c., by a human an LLM, which aims to explore to what extent the evaluators can distinguish between human-written and L.LM-generated poems.
4.2 THE METHODS OF POEM EVALUATION
To make the framework more robust, we triangulate LLM-as-a-judge with rule-based quantitative evaluation and human expert judgments, detailed below.
Rule-based automated evaluation We apply handcrafted, rule-based algorithm to automatically detect the meter and rhyme patterns in each poem in order to gauge the overall accuracy of each author. A flowchart of the algorithm can be found in Appendix [B] For both human and L.MM poems, lexical diversity is calculated with Moving Average Type-Token Ratio (MATTR) averaged across poems for each author, and creativity is quantified the ratio of repetition of words in an LLM poem compared to the original human work, which is also averaged across poems for each author.
LLM-as-a-judde automated evaluation and human validation To balance the need for large- scale evaluation with the practical constraints of high-quality literary analysis, we did not perform human validation on the entire dataset, which would otherwise be resource-intensive. The required evaluators domain experts such as poets and literary academics - are scarce resource. Furthermore, the annotation of a single poem is a highly demanding and time-consuming task, far exceeding the complexity of standard data-labeling. Therefore, our methodology leverages L.L.Mas-a-jude for broad coverage, complemented by the validation from panel of human experts smaller, representative sample to ensure the reliability of the automated results.
We first provided all the anonymized LLM poems and human poems for LLM-as-a-judge for evaluation based on the dimensions discussed in 4.1 In order to validate the results, with Institutional Review Board (IRB) approval, recruited 7 expert human judges to evaluate subset of anonymized poems (58 in total) by humans and 7 representative L.LMs. These human experts have backgrounds in poetry studies English literature, including professional poets, doctoral students, post-doc researchers, professors, and other researchers. We designed a prompting template (Li & Wang 2024) for LLM-as-a-judge and a survey for human judges based on POEMetric, asking them to answer questions after reading the generation prompt and the poem written in response to the prompt. The questions comprised 10 multiple-choice questions (in line with the 10 metrics in POEMetric) using a 5-point Likert scale, asking the evaluators to score from 1 (Strongly Disagree) to 5 (Strongly Agree), and open-ended questions where the evaluators could comment on why they gave that score the previous question. The template of the survey for human experts and the evaluation prompt for the LLM judge can be found in Appendix C
5 EXPERIMENTS
5.1 EXPERIMENT SET-UP
To better compare different LLMs, we adopted default sampling parameters for each LLM. For open-source models. we applied vLLM (Kwon et al., 2023) deploy them on local GPUs. We guaranteed that each LLM received the same text prompt. As for system prompt, we adopted the default setting. In choosing as--aaeee our pilot study (see Appendix (D) showed that, Gemini- 2.5-Pro, compared with DeepSeek-Ri and GPT-4o, yielded higher agreement with human experts (PAo=0.662 vs. 0.548/0.433, and superior discriminative ability in evaluating Overall Poem Quality (Std. Dev. 0.63 vs. 0.20/0.22) which were crucial for ensuring evaluation validity. At the same time, averaging with other LLMs would have introduced noise and bias. Thus, we chose Gemini-2.5 (Team, (025b) the single LLM judge, which is also one of the strongest generalist LLMs across different benchmarks (Phan et al. 20255 Rein et al., 2024 AIM Jain et al., (2024] Wei et al. 2024 Yue et al. 2024 Chiang al. 2024) with free access for the research community.
5

---

Published as a conference paper at ICLR 2026
5.2 LLM SELECTION AND POEM GENERATION
We prompted 30 models of 7 leading Al companies for poem generation: an overview of the selected models shown in Appendix B We employed a simple prompting template (see Figure to include the form, rhyme, meter and theme of each human poem. Each LLM responded to 203 prompts generated based on the human poem dataset, totaling 6,090 LLM poems. A general description the linguistic features of the human-LLMy poem dateset, such as most frequent words, top opening words, and most common imagery, can be found in Appendix
RESULTS
In this section, with focus on the best- perttrreffomnggggg LLMs representative of the Al companies, we will first present a case study, discuss the results produced by rule-based evaluation, then turn to the evaluation by Gemini-2.5-PP and finally explain similarity to human evaluation results.
6.1 CASE STUDY: AN ILLUSTRATIVE COMPARISON
To provide concrete illustration of the aggregate findings, begin with direct comparison. This case study showcases a poem generated by a nigh-performing LLM, i.e., DeepSeek-RI. alongside human-written poem, based on the same prompt. In Figure 3 before revealing the authors, can the reader discern which poem was written by a human? By examining the works side-by-side, along with their POEMetric scores, the nuanced differences between machine- generrtte text and human artistry become tangibly clear. Our LLM judge, Gemini-2.5-PH decided that DeepSeek-RI output was technically flawless, adhering strictly to the prompt's formal constraints, and employed evocative imagery and literary devices, leading to higher scores across various dimensions than the human-written work- do you agree with our judge? More showcases can be found in Appendix
Fearr file ne pocae manii the aabe me anjoo ling
Whhe the sa
E dowari mosee plaraars..
Gralley
sase de
shoattoe days
Sierrty
so mat Paskaed
for aeetare levnt
the soaliaall wane of
Aip be silara Thene
ane pertiry
det vo trod allo ta "ecol
Inggrr
thone guod
Authssiig
The henett
Poim
Peen
Figure 3: A showcase of the poems by DeepSeck-RI (Poem A) and human poet (Poem B) in response the same prompt. The bar charts show their POEMetric scores judged by Gemini-2.5-Pro.
Furthermore, to illuminate the generative process of advanced models, present the Chain-of- Thought (CoT) output from DeepSeck-RI when generating its poem in Figure 4 This internal monologue reveals structured, intentional process of creative reasoning, which demonstrates that the model's 's process is not a black box. methodically deconstructs tasks, plans its structure, and even critiques its own word choices, which very similar to the thinking process of human poet.
6.2 RULE-BASED EVALUATION
Figure shows the rule-based form accuracy, MATTR, and repetition rate of each author. First, our automated form detection algorithm, including the meter and rhyme patterns, examined the representative LLMs, where -emini-2.5-Pro and Claude-3.77 3.7de-3.--Soneet showed relatively high form accuracy (0.50 and 0.47). Second, LLMs demonstrated higher lexical diversity than humans did according to MATTR. Last but not the least, LLM LLM-generatee generated poems exhibited high repetition rates the word level when compared with the human poems, suggesting pronounced imitation human works.

---

Published as a conference paper at ICLR 2026
-thinle
Alnggt, the user wands hulladiw with ABCis rhyme scheme on the theme of whime cal dreams about money whils being broke but happy, Let me break this down.
First, ballad structuro usualy has quatrairs (four tine stanzas] with the second and fourth lines thymingg ASCB means Incs 2 and 4 thyms each time. Need to make sure each follows that.
Theme whimaical dreams about money, humoreus scknowietgment
funds, and finding heppiness dreamingg the poem
have imaginativve
what
money, contrast with reality, and on peaitire note shout the joy of dreaminng
Let me brainstom some whimmical ideas. Maybe buying a castle made of clouds trading coins torthe moon, a goldon boxt, a forest of candy trees. These are playful and show extrovaga noe without real money.
New, contrast with the
Lines lee "but my pockets bare" or 'my walle''s swatteland" shour they can it afford these things Wixn humor,, like debrs
squiral piggy bankjat
standas emphasiting that oven without money, dreaming brings joy they in the tunes" their dreamns. Check the meter. Ballads often have a frythm, may be altemoting between 8 and 6 syflables Let me count syliables for each line ensure a flow. Not too strict, buta consistent best.
Finally, happiness from dreamiing
Make each stanza fallows ASCB, For soample:
1. dream
action
(ends rhyme B)
Line It reality check
Line & humorous inn (thymes with line 2)
Let me draft few stanaas, check the rhymes and flow. Revise where necessary to keep the whiimicall and humorous tono. Ensure the ending ties back the theme of happiness from dreaming. +/btinc
Figure 4: Chain-of- Chauuhhf-Thouhht (CoT) process from DeepSeek-FR R1 for the poem generation. The model explicitly breaks down the prompt, plans the thematic progression stanza by stanza, brainstorms mter and rhymes, and attempts to strategically insert literary devices.
Rufo bared Evaluation
0,0
Oower daneet
Home 3 rochat
deming
Figure 5: Rule-based evaluation results. L.MSs were able to achieve high form accuracy and MATTR. However, their poems were highly repetitive compared with the original human poems.
6.3 LLM-AS-A-JUIGEE EVALUATION
Basic instructioo-following abilities In Figure Gemini-2.5-Par scored the highest in terms of form accuracy (4.26) and theme alignment (4.99), suggesting outstanding instruction-folloiing abilities compared with the other LLMs while
ranked low in both metrics (2.29, 4.91). We found that some of the poorly performing LLMS would stick default form for certain type of poem; for instance. they would use the common rhyme pattern ABAB when writing ballads, instead of following the specific ABCB instruction in the prompt given, thus resulting in unsatisfying performance in form accuracy.
Parn Accurary
Theme Afgement
35-
3.0
35
221
2.5
1.0
Chosle 3.7 Somml
dembouz Shhe
Denens- 270
Uses 1h8 bosrut
Choote 17 Soodd
Bemauu25rt Soh
Deaino 2728
Figure 6: Form accuracy and theme alignment scores. Gemini-2.. 2.5-Pro achieved the highest scores in both dimensions, whereas Llama-3.3 a..ma-..-700---- 7-B--Instttcct ranked the lowest among the 7 LLMs.
7

---

Published as a conference paper at ICLR 2026
Advanced creative abilities As shown in Figure 7 compared with LLM-generated poems, the poems written by humans excelled in terms of creativity (4.02), idiosynerasy (3.95), emotional resonance (4.06), and the use of imagery (4.49) and literary devices (4.67). Among the 7 represen- tative LLMs, DeepSeek-R1 1 yielded the best performance while 1aama-33-770--Inttme the lowest scores. Meanwhile, somewhat expected, LLMSs showed significantly less idiosyncrasy in their poems, indicating a lack of personal distinctiveness or experience. However, DeepSeck-RI (3.85) outperformed humans (3.82) in terms of lexical diversity.
achieved
Oratde
(a) Gemini- Semini-2.5-PP Prr as a judge
(b) Human as a judge
Figure 7: Advanced creative abilities. Compared with LLMs, human poets excelled in creativity,
idiosyncrasy, emotional resonance, and use of imagery and literary devices.
General appraisal Figure 8 demonstrates the overall poem quality and human authorship estimation of the poems. For one thing, poems written by humans achieved a higher mean score (4.22) than those generated by I.LMs in terms of the overall quality due to the effective and idiosyncratic use of language by humans, according to the comments given by Gemini-2.5-Prr in the open-ended questions. Following humans was Re,, which was only slightly better than the other LLM authors. For another, although authorship was not revealed to Gemini-2.5-PP as a judge, it was generally able to distinguish between a human poem and an LLM poem. Of all the 203 human poems, Gemini-2.5-Pro was able to recognize 80 poems (39.4%), either by reciting the original poem or by recognizing the distinctive style of a poet. Figure 2 shows the overall performance of humans and all 30 L.MMs, and the scores were averaged across basic instruction-following abilities, advanced creative abilities, and poem quality. There was a general tendency that models with more parameters within the same family series performed better in poem generation. Thinking models were not necessarily better than their non- non-thinking family members; for instance, GPT-4o and GPT-4 ranked higher than ol and 03-mini. Besides, DeepSeek-RR --Distllll models were generally worse than the original models, except that 30 LLMs each specific dimension can be found in Appendix
3.3-70B performed better than its original. More results of humans and all
Authorship
5.0
45-
3.
2.5
2.0
2.0
132
15
15
Homan
Choale Sootel
Deeptook d1
dleest Sutee
demate 75
Unoo mol mettuu
Homan
Conaled Seotel
Deepswer-
domiit2 Sitte
5278
Usos autuud
Figure& 8: Overall poem quality and human authorship estimation scores. Humans ranked first in terms of overall poem quality, and human poems remained largely distinguishable from LLM poems.
8

---

Published as a conference paper at ICLR 2026
Dicption dlmes
4.
Dec
De
JPA
Cenves
Paromators (B)
Figure The mean scores of POEMetric human poets and 30 LLMs, evaluated by Gemini-2.5- Models with more parameters within the same family generally performed better. Thinking models were not necessarily better than their non-thinking family members, as GPT-4o and GPT-4 ranked higher than ol and 03-mini. DeepSeek-R R1-Distill models were generally worse than the original models, except Distil-Lllama-33-700 performed better than its original.
6.4 HUMAN VALIDATION
In order to validate the evaluation results given by Gemini-2.5-Pro we calculated its Proportion Agreement, Observed (PAo) (Neuendorf) (007)) with the expert human evaluators in order test inter-rater reliability. The Proportion Agreement, Observed (PAo) calculated using the following
formula:
2A1 nA + nB
PAo
where A is the number of agreements between the raters, nA the total number of ratings by Rater A, and ng is the total number of ratings by Rater B. This formula quantifies the degree of agreement between two raters, providing a measure of how often their ratings coincide. The PAo test between the scores given by Gemini-2..5Prrs as judge and the human evaluators across the 10 multiple-choice questions found strong agreement (0.662). In addition, we also calculated Cohen's Kappa and Spearman's 's rank correlation coefficient, where strong correlation was found in human- LLM agreement (Quadratic Weighted Kappa x 0.361, Spearman Correlation p = 0.378). This performance echoes existing studies involving LLLM-uuman inter-rater agreement; for example. in evaluating an agentic reviewer Jiang & Ng 2026), between one human reviewer and another is 0.41, whereas between Al and human reviewer is 0.42; in 20 NLP Evaluation Tasks Bavaresco et al. (2025), the agreement between the best-performing L.M and humans falls between ş = 0.28+0.32. Hence the results our study were robust. what follows, we will discuss the similarities and discrepancies between the results given by the two groups of judges.
As shown in Figure 6 there terms of the form accuracy and theme alignment of the LLM--genrrted poems, with Gemini-2.5- Pro and Slaude-3.7-Somaee ranking top. However, when judging theme alignment, human evaluators tended to give higher scores than Gemini-2.5-Par did. As for advanced creative abilities, in Figure both Deminii2..5PPa and the human judges decided that, compared with L.LM poems, human poems excelled in terms of creativity, idiosyncrasy, emotional resonance, and the use of imagery and literary devices. By comparison, some LLMs were thought to be able to use more varied vocabulary than humans did, though there was disagreement in which LLM was more lexically diverse. In Figure 8 as for overall poem quality, it is shown that the human judges were more restrained in the first-ranking human poems achieved only mean score of 3.46. meaning the human evaluators were prone to agree that these were good poems, but not so certain. As for estimating authorship, human evaluators also generally able tell if poem was written
high similarity between human evaluators and Gemini-2.5-Pro in
giving high scores:

---

Published as a conference paper at ICLR 2026
human LLM. Compared with Gemini-2.5-Proi relatively high ratio of recognizing the original human poems (39.4%), within the 13 human poems evaluated by human experts, only 1 poem was recognized as a famous poem, and yet all 13 poems were scored 3 (neutral) or higher (agreeing or strongly agreeing this poem was written by human). This implies that, though the human judges could not recognize many original poems as Gemini-2.5-PPr could, they still likely to find out the authorship of a poem. Nevertheless, in the face of LLM-generated poems, the human evaluators were less confident about their authorship compared with Gemini-2.5-Pre judge.
CONCLUSION AND LIMITATIONS
We introduce POEMetric, the most comprehensive evaluation framework for poetry generation far. We also curated human poem dataset, covering 203 poems of different poetic forms and themes, and experimented with 30 state-of-the-an LLMs. Although the top models have the capabilities of writing poems of certain styles and themes, they still fall short of attaining advanced creative abilities such as creativity, idiosyncrasy, evoking emotional resonance, and skillful use of imagery and literary devices. Moreover, our findings have demonstrated the effectiveness and efficiency of automated poetry evaluation with POEMetric. More explorations are encouraged to adjust POEMetric to evaluate frre--tyle poems. Besides, this paper only examines the English language, while POEMetric is applicable to other low-resource languages as well. We leave to future work.
ETHICS STATEMENT
The research presented in this paper adheres to the ICLR Code Ethics. In our commitment to scientific excellence and transparency, introduce POEMetric comprehensive framework and provide our code and dataset in the supplementary materials to foster reproducible research, with our methods, model selection, and limitations detailed throughout the paper. All work involving human participants was conducted under Institutional Review Board (IRB) approval, with informed consent and full data anonymization to protect privacy and honor confidentiality. Furthermore, respect intellectual property by sourcing our dataset from properly credited, publicly accessible archives and building upon prior research as detailed Section 2
REPRODUCIBILITY STATEMENT
To facilitate the full reproducibility of our findings, we have made all key components of our research publicly available. The evaluation framework and methodology are clearly presented in Section 4
The code for our rule-based evaluation algorithm and the curated human poem dataset are provided anonymously in the supplementary materials. Comprehensive details regarding our experimental setup, including the complete list of the 30 LLMs evaluated (Appendix E the model configurations (Section 5). and the precise generation prompt template (Figure 2 are provided to enable the replication of our poem generation process. Furthermore, the exact evaluation prompt used for the LLM-as-a- a-judge and the full survey administered to our human experts included in Appendix ensuring that our multi-faceted evaluation can be independently verified and extended.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Aime
Poem analysis, URL htt.ittttss/ppemmaaaysssss
Poem hunter, b. URLhttpss//ww.ppehhunnee
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et Gpt-4 technical report. arXiv preprint arXhv:2303.08774, 2023.
Zaid Alyafeai, Maged S Al-Shaibani, and Moataz Ahmed. Ashaar: automatic analysis and generation of arabic poetry using deep learning approaches. arXiv preprint wwxivv2307.06211 2023.
AI Anthropic. The claude model family: Opus, sonnet, haiku. Claude-3 Model Card, 1:1, 2024.
Anna Bavaresco, Raffaella Bernardi, Leonardo Bertolazzi, Desmond Elliott, Raquel Fernández, Albert Gatt, Esam Ghaleb, Mario Giulianelli, Michael Hanna, Alexander Koller, al. Llms instead of human judges? a large scale empirical study across 20 nlp evaluation tasks. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguisties (Volume 2: Short Papers), pp. 238-255 2025.
Mohamed El Ghaly Beheitt and Moez Ben HajHmida. Effectiveness of zero-shot models in automatic arabic poem generation. Jordanian Journal of Computers and Information Technology, 9(1), 2023.
Jonas Belouadi and Steffen Eger. ByGPT5:
End-to-end style-conditioned poetry generation with token-free language models. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the sl Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 7364-7381, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10. (0.18653/11/0023 aal-233.cllooooo hww.:/22....l-loo.40......00 406. URL https: aattps//acaaattooooo.ooooo 2023.acl-1on 4067]
Brendan Bena and Jugal Kalita. Introducing aspects of creativity in automatic poetry generation. arXiv preprint 1rXiv:2002.02511, 2020.
Danyang Cao and Cheng Cheng. Survey deep learning applications in automated chinese poetry composition. In 2024 5th International Conference on Artificial Intelligence and Computer Engineering (ICAICE), pp. 662 666. IEFE, 2024.
Yanran Chen, Hannes Gröner, Sina Zarrieß, and Steffen Eger. Evaluating diversity in automatic poetry generation. arXiv preprint arXiv:2406.15267 2024.
Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li. Banghua Zhu, Hao Zhang, Michael Jordan, Joseph E Gonzalez, al. Chatbot arena: An open platform for evaluating Ilms by human preference. In Forty-first International Conference on Machine Learning, 2024.
Michal Chudoba and Rudolf Rosa. Gpt ezech poet: Generation of ezech poetic strophes with language models. arXiv preprint trXiv:2407.127900 2024.
Zekun Deng, Hao Yang, and Jun Wang. Can write classical chinese poetry like humans? an empirical study inspired by turing test. arXiv preprint MxXiv:2401.04952 2024.
Thomas Stearns Eliot. The use of poetry and the use of criticism: studies in the relation of criticism to poetry in England, volume 39. Harvard University Press, 1986.
Haosen Fang. Ancient poetry generation based on bidirectional Istm model neural network. Science and Technology of Engineering, Chemistry and Environmental Protection, 1(6), 2024.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinay Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The la herd of models. arXiv preprint arXiv:2407.22 27783 2024.

---

Published as a conference paper at ICLR 2026
Roland Greene, Stephen Cushman, Clare Cavanagh, Jahan Ramazani, and Paul Rouzer. The Princeton encyclopedia of poetry and poetics. Princeton University Press, 2012.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprini arXiv:2501.12945 2025.
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, al. Gpt-4o system card. arXiv preprini arXiv:2410.21276, 2024.
Triet Minh Huynh and Quan Le Bao. Vietnamese poem generation & the prospect of cross-language poem-to-poce toooooopommmmooooooooooooooooo translation. arXiv preprint arXiv:2401.01076 2024.
Aaron Jacch, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Alcksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint arXiv:2412.16720. 2024.
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and lon Stoica. Livecodehench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07977, 2024.
Yixing Jiang and Andrew Ng. Tech overview, 2026. URL httpps://paperrrvveee.u. tech-overview
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Annie Lamar and America Chambers. Generating homeric poetry with deep neural networks. In 2019 First International Conference Transdisciplinary Al (TransAI), pp. 68-75. IEEE, 2019.
Bingru Li and Han Wang. Tacomore: Leveraging the potential of Ilms in corpus- bbsedd discourse analysis with prompt engineering. arXiv preprint arXiv22412.10139, 2024.
Yi Liao, Yasheng Wang, Qun Liu, and Xin Jiang. Gpt-based generation for classical chinese poetry, arXiv preprint arXiv:1907 0007000151 2019.
Zhangmin Ling and Lin Zhang. Chinese poetry generation model with unilm. In 2022 2nd Interna- tional Conference on Consumer Electronics and Computer Engineering (ICCECE), pp. 925-930. IEEE, 2022.
Aixin Liu, Bei Feng. Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arX1v:2412.19437, 2024.
Dayiheng Liu, Quan Guo, Wubo Li, and Jiancheng Lv. A multi-modal chinese poetry generation model. In 2018 International Join Conference on Neural Networks (IJCNN), pp. 1-8. IEEE, 2018
Zhiqiang Liu, Zuohui Fu, Jie Cao, Gerard De Melo, Yik-Cheung Tam, Cheng Niu, and Jie Zhou. Rhetorically controlled encoder-decoder for modern chinese poetry generation. In Proceedings of the 57th annual meeting of the Association for Computational Linguistics, pp. 1992-2001 2019.
Kai-Ling Lo, Rami Ariss, and Philipp Kurz. Gpeet-2: A gpt-2 based poem generator. arXiv preprint 12Xi:22005.08447 2022.
Kimberly A Neuendorf. The content analysis guidebook. sage, 2017.
Aitor Ormazabal, Mikel Artetxe, Manex Agirrezabal, Aitor Soroa, and Eneko Agine. Poelm: A meter-and contelcont language model for unsupervised poetry generation. arXiv preprint arXiv:2205. / /2006, 2022.
12

---

Published as a conference paper at ICLR 2026
Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, Chen Bo Calvin Zhang, Mohamed Shaaban, John Ling, Sean Shi, et al. Humanity's s last exam. arXiv preprint 1rXf1:2501.14249 2025.
Andrei Popescu-Belis, Alex R Atrio, Bastien Bernath, Étienne Boisson, Teo Ferrari, Xavier Theimer- Lienhardt, and Giorgos Vernikos. Gpoet: language model trained for rhyme generation on synthetic data. In Proceedings of the 7th Joint SIGHUM Workshop on Computational Linguisties for Cultural Heritage, Social Sciences, Humanities and Literature. Association for Computational Linguisties, 2023.
Maurilio De Araujo Possi, Alcione De Paiva Oliveira, Alexandra Moreira, and Lucas Mucida Costa. A neural network-based language model for automatic poem generation. In 2024 IEEE 20th International Conference on Intelligent Computer Communication and Processing (ICCP), pp. 1-8. IEEE, 2024.
Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang. Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu,, Keqin Bao, Kexin Yang. Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025. URL htttp:://arxies /ttp://aaxvvvv/om wttplvv..rrr/////2/222222
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-pre poogll q&a benchmark. In First Conference on Language Modeling, 2024.
Ivor Armstrong Richards. Practical criticism V4. Routledge, 2014.
Piotr Sawicki, Marek Grzes, Fabricio Goes, Dan Brown, Max Peeperkom, and Aisha Khatun. Bits of grass: Does gpt already know how write like whitman? arXiv preprint arXiv:2305.11064 2023.
Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rívière, et al. Gemma 3 technical report. arXiv preprint arXiv:2503. 19786, 2025.
Google DeepMind Team. Gemini-2.0-pro, 2025a. (technologies// gttpe///emminiipp..o..
URL https://deepmind..ggg google/
Google DeepMind Team. Gemini-2.5-pro, 2025b. URL https::/deepmidd..ggg. techno 1 eeseee//ee http///mminnnipm n /
Mistral AI Team. Mistral large, 2024. URL htt.l///////aaaaaaaaaaa////aaaaa
OpenAl Team. Gpt-3.5-turbo, 2023.
URL httes:/ppaafoom..gnnaa.ccoooo//
models/ https//ttt.....bbbbb.
OpenAl Team. Gpt-4.5, introducing gpt 4-57]
2025c.
URL
hatpar//ggenal.com//nnee
OpenAl Team. 03-mini, 2025d. htttp////oopraa..ooomnaae///////
Qwen Team. Qwg-32b: Embracing the power of reinforcement learning, March 2025e. URL hhtpps://qwen www.mm.............o..oooeo ho/bllog//gg qwpp////////////////
Imran Ullah, Khalil Ullah, Hamad Khan, Khursheed Aurangzeb, Muhammad Shahid Anwar, and Ikram Syed. Pashto poetry generation: deep learning with pre-trained transformers for low-resource languages. PeerJ Computer Science, 10:e2163, 2024.
Melanie Walsh, Anna Preus, and Maria Antoniak. Sonnet or not, bot? poetry evaluation for large models and datasets. arXiv preprint arXiv:2406. 18906, 2024a.
Melanie Walsh, Anna Preus, and Elizabeth Gronski. Does chatgpt have a poetic style? arXiv preprint arXiv:2410. 15299, 2024b.
13

---

Published as a conference paper at ICLR 2026
Qixin Wang, Tianyi Luo, Dong Wang, and Chao Xing. Chinese song iambics generation with neural attention- based model. arXiv preprint arXiv:664..06274, 2016.
Jason Wei, Nguyen Karina, Hyung Won Chung, Yunxin Joy Jiao, Spencer Papay, Amelia Glaese,
John Schulman, and William Fedus. Measuring short-form factuality in large language models. arXiv preprint arXiv:2411.04580 2024.
Liang Yang, Zhexu Shen, Fengqing Zhou, Hongfei Lin, and Junpeng Li. Tpoet: Topic-enhanced chinese poetry generation. ACM Transactions on Asian and now-Resource Language Information Processing, 22(6):1-15, 2023.
Xiaoyuan Yi, Ruoyu Li, and Maosong Sun. Generating chinese classical poems with encoder- decoder. In Chinese Computational Linguistics and Natural Language Processing Based on Naturally Annotated Big Data: 16th China National Conference, CCL 2017, and 5th International Symposium, NLP-NABD 2017, Nanjing, China, October 13-15, 2017, Proceedings 16, pp. 211-223. Springer, 2017.
Xiaoyuan Yi, Ruoyu Li, and Maosong Sun. Chinese poetry generation with a salient-clue mechanism. arXiv preprint 1809.v8800.000 01199.0433,,,.. 2018.
Chengyue Yu, Lei Zang, Jiaotuan Wang, Chenyi Zhuang, and Jinjie Gu. Charpoet: A chinese classical poetry generation system based on token-free Ilm. arXiv preprint arXiv:2401.03511 2024.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, al. Mmmu: A massive multi-discipline multimodal under- standing and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9556-9567, 2024.
Jiyuan Zhang, Yang Feng, Dong Wang, Yang Wang, Andrew Abel, Shiyue Zhang, and Andi Zhang. Flexible and creative chinese poetry generation using neural memory. arXiv preprint arXiv:1705.03773.20 2017.
14

---

Published as a conference paper at ICLR 2026
A FIXED FORMS OF ENGLISH POETRY
Ballad Ballads are usually long poems consisting of quatrains (4-line stanzas, where a stanza means section of poem), following the rhyme pattern of ABCB or ABAB for each stanza. The two main types of ballads, the traditional folk ballads and the literary ballads, adopt varied meter patterns, and sometimes creative forms such as 6-line or 8-line stanzas with new rhyme patterns.
Ghazal Originating in the Arabic poetry, ghazals are set of couplets (2-line stanzas), Each couplet ends on the same word or phrase (the radif), and is preceded by the couplet's rhyming word (the qafia, which appears twice in the first couplet).
Limerick traditional limerick a stanza of 5 lines, with fixed rhyme pattern of AABBA and varying meter patterns for each line. Later limericks were popularized by the poet Edward Lear, which consist of 4-line stanza rhyming AABA, with the third line comprising two sentences split by comma and both rhyming B.
Pantoum The pantoum is a Malay verse form, a series of quatrains with the second and fourth lines of each quatrain repeated as the first and third lines the next. The second and fourth lines of the final stanza repeat the first and third lines of the first stanza.
Sestina The sestinas are a complex French verse form, usually unrhymed, consisting of six stanzas of six lines each and three-line envoi. The end words of the first stanza are repeated in different order as end words in each of the subsequent five stanzas; the closing envoi contains all six words, two per line, placed in the middle and at the end of the three lines.
Sonnet Sonnets usually consist of 14 lines following the meter pattern of iambic pentameter, which a line of verse composed of ten syllables arranged in five metrical feet (iambs), each of which consists of an unstressed syllable followed by stressed syllable. There different types of sonnets. The Petrarchan subcategory usually consists of one octave (8-line stanza) and one sestet (6-line stanza), and adopts typical rhyme pattern of ABBAABBA CDCDCDICDECDE An English variation of the Petrarchan sonnets, i.e., the Italian sonnets, rhyme with ABBAABBA CDDCEE. The Shakespearean and Spenserian types usually comprise three quatrains followed by couplet, each with different rhyme patterns. Apart from these, poets have also created new patterns such as 16-line sonnets and reversed sonnets.
Villanelle As a French verse form, a villanelle consists of five three-line stanzas and a final quatrain, with the first and third lines of the first stanza repeating alternately in the following stanzas and forming the final couplet in the quatrain.
15

---

Published as a conference paper at ICLR 2026
B THE ALGORITHM OF RULE-BASED FORM ACCURACY
In detecting the form accuracy via a rule-based algorithm, we tested on a subset of the human poems to optimize the trade-off between precision and recall for form detection. higher threshold would sort out almost perfect poems, but would incorrectly reject many poems that contain minor stylistic variations, thus unfairly penalizing creativity. A lower threshold would be too lenient; would incorrectly accept many poorly-formed poems. Therefore, we opted for a 0.7 threshold. The algorithm is as follows.
Algorithm Rule-Based Based Poetry Form Evaluation (POEMetric)
Input: Poem text T' Target Form F. Target Meter M, Target Rhyme R Output: Boolean indicating 'f satisfies the constraints
1: Phase I: Linguistic Feature Extraction
2: W +- Tokenize(())
Split into lines and words, remove punctuation
3: Ameter t- 9, Arhyme + 0
4: for each line E W do
5:
Pstress f- ExtractStress(I, CMUdict)
Map to 'u', 'S', '*1
6:
Aineter-appende (P strees)
7:
Wend + (LstWord()))
8: Pphoneme Extracthyymeoooquuaa CMUdict)
9:
append Phommeme)
end for
11: Srhyme <
D e.g., 'A', 'B', 'A', 'B'
Phase 2: Sprc-Specific Structural Validation
13: Ffized +- -LLimerick, Villanelle, Sestina, Pantoum, Ghazal)
14: if F E Ffized then
15:
is alid <- False
16:
if F = Ghazal then
17:
isV <- CheckCouplets (W) A CheckRadifQafia(V) )
18:
else F ===Semm Sestina then
19:
isV alid <- Check(ermutatiinss(())) CheckFnvoi(W)
20:
else F Villanelle then
is alid + CheckRefrrins (W) A CheckGlobalRhymeeSSSooo
22:
else if F = Pantoum then
isV alid <- Cheekkiineeppttitions(\W//
24:
else if F = Limerick then
25:
is SaValla alid <- CheckPattemnS (Srkmme 'AABBA' 'AABA')
26:
end if
27:
if not asValid then return False
28: end if
29: R < None
Bypass general rhyme check for fixed forms
30: end if
31: Phase 3: Meter and Rhyme Validation (with Tolerance)
32: if M j None then
33:
Emeter &- GeeEExpeteePPttem(MM)
MatchRatio + [Angter EpeAmeter I(p Emeter)
35: if MatchRatio 0.7 then return False
36:
end if
P 70% threshold
37: end if
38: if R +
None then Match Ratio &- ( (atteeSstmttaeeees R)
39:
40: if Match Ratio 0.7 then return False
41:
end if
b 70% threshold
end if
43:
return True
16

---

Published as a conference paper at ICLR 2026
C
THE POEMETRIC-BASED LLM PROMPT AND HUMAN SURVEY
Below are the prompting template for LLM-as-a-juue as-judeee and the survey template for human expert judges, which share the same set of POEMetric-basse questions.
POEMetric-based Human Expert Survey
Please read prompt (i.e. instructions that were given for writing each specific poems reproduced below), and then read the poem that was written in response this prompt. Then answer the questions that follow. Please do not check any online or offline resourees while completing this survey; we are interested your own direct and personal response!
will give you poems in total; pleuse as many as you can. Do not worry you could not finish all of them.
Poem
The prompt
fgeneration_ p
The poem
[the_poem]
Questions
Below are required multiple-choice questions and 3 optional open-ended questions. For each multipe-choicee question, please score the poem from I to based on how strongly you agree disagree with question.
The poem follows the given prompt in terms of form, including meter and rhyme where applicable. (Required)
1.
Strongly disagree
Disugree
3 Neutral
4 Agreo
5 Strongly agree
The poem follows the given prompt terms of theme. (Required)
1 Strongly disagree
2. Disagree
3 Neutrul
17

---

Published as a conference paper at ICLR 2026
4 Agree
5 SSrongly agree
3. The poem uses varied vocabulary. (Required)
1 Strongly disagree
2 Disagree
3 Neutrul
4 Agree
5 Strongly agree
The poem a creative work. (Required)
1 Strongly disagree
2 Disagree
3 Neutral
4 Agrec
5 Strongly agree
5. This poem shows idiosyncrasy. (Required)
1 Strongly disagree
2 Disagree
3 Neutral
4 Agree
5 Strongly agree
This poem evokes emotional resonance. (Required)
1 Strongly disagree
2 Disagree
3 Neutral
4 Agrec
5 Strongly agree
7. The imagery in this poem is used well. (Required)
18

---

Published as a conference paper at ICLR 2026
0 N/A (No imagery is used)
1 Strongly disagree
2 Disagree
3 Neutral
4 Agree
5 Stronglyy agroc
least one of literary devices listed below is used well poem. (Required)
Simile
Metaphor
Personification
Allusion
0 A-NA (No literary devices used)
1 Strongly disagree
2 Disagree
3 Neutral
4 Agrec
5 Strongly agree
9. Please comment on why you gave the answer that you did for question 8 above.
10. This is good poem. (Required)
Strongly disagree
2 Disagree
3 Neutral
4 Agrec
5 S-Srrongly agree
11. Please comment why you gave the that you did for question 10 above.
19

---

Published as a conference paper at ICLR 2026
12. The poem is written by human. (Required)
1 Strongly disagree
2 Disagree
3 Neutral
4 Agree
5 Strongly agroc
Please give comments on why you gave the answer that you did question above.
20

---

Published as a conference paper at ICLR 2026
POEMetric-based LLM Evaluation prompt
Role Description
You are professional poetry critic and analyst. Your job is to evaluate English poctry written by human beings and English poctry generated by large language models.
Task Definition
will ask you evaluate one poem by answering 10 multiple-choice questions and open-ended questions, without telling you poem written by a human an LLM. For each multiple-choice question, please score the poem from to based on how strongly you agrce disagrce with the question.
Task Procedures
Step I, prompt (i.e. instructions were given for writing each of
specific poems reproduced below).
Step read the poem that was written in response to this prompt.
Step answer questions that follow.
Context
## The prompt
cceneration_promppll
#Wl The poem
(the poem]
#ll Questions
1. The poem follows the given prompt in terms of form, including meter and thyme where applicable.
Strongly disagree
Disagree
3 Neutral
Agree
5 Strongly agree
The poem follows the given prompt terms theme.
- Strongly disagree
Disagree
Neutral
4- Agree
5 Strongly agrec
21

---

Published as a conference paper at ICLR 2026
The poem uses varied vocabulary.
- Strongly disagrec
Disagree
Neutral
4 Agree
5 Strongly agree
4. The poem is a creative work.
1- Strongly disagrec
Disagree
3 Neutral
4 Agree
- Strongly agree
This poem shows idiosynerasy.
- Strongly disagree
2 Disagree
Neutral
4 Agree
5 Strongly agree
6. This poem evokes emotional resonance.
1 Strongly disagree
- Disagree
3 Neutral
Agree
Strongly agree
The imagery in this poem is used well.
0 N/A (No imagery is used)
Strongly disagree
Disagree
- Neutral
4 Agree
Strongly agree
8. At least of the literary devices listed below is used well in the poem.
Simile
Mctaphor
Personification
Allusion
0- N/A (No literary devices are used)
Strongly disagrec
22

---

Published as a conference paper at ICLR 2026
2- Disagree
Neutral
4 Agree
5 Strongly agree
9. Please comment on why you gave the answer that you did for question 8 above.
This good poem.
- Strongly disagree
Disagree
Neutral
4 Agree
5 S-rronngly agrec
11. Please comment on why you gave the answer that did for question 10 above.
12. The poem is written by human.
- Strongly disagree
Disagree
3 Noutral
4 Agree
Strongly agrec
13. Please give comments why you gave the answer that you did for question 12 above.
## Output Format
For each multiple-choice question, please give your score directly, without any explanation. Your oulput should be in the json formal as follows:
("I": <insert your score here?, "2"; sinsert your score here", .... "9"; "cinsert your comments here"" ....
23

---

Published as a conference paper at ICLR 2026
D LLM-AS-A-JUDGE JUSTIFICATION
We agree that cross-model validation the ideal, and we performed this analysis our pilot studies. Our results revealed that averaging scores from multiple LLMs would degrade the evaluation quality, as other leading models proved to be flawed evaluators in two key ways:
Low Agreement with Human Experts We tested DeepSeek-RR R1 and GPT-4o, and they demon- strated substantially lower inter-rater reliability with our human experts. The Observed Proportion Agreement (PAo) (Neuendorf, 2017) was low for GPT-4o (0.548) and DeepSeek-R (0.438), but strong for Gemini-2.5-Pru (0.662). This divergence from human consensus would introduce signifi- cant noise and undermine the validity of our findings.
Lack of Discrimination Ability Other models failed distinguish between high- and low-quality poems in the "overall poem quality" dimension. As shown in Table 2 the extremely low standard deviations for DeepSeck-RI (0.20) and GPT-4o (0.22) confirm that their scores were tightly clustered at the high end of the scale (as shown by their mean scores of 4.26 and 3.69). Including them would inevitably introduce bias and noise. In contrast, the standard deviation of Gemini-2.5- 2.55min- scores (0.63) was much closer to that of our human experts (1.09), indicating it was a far more reliable and discerning instrument for measurement.
Table 2: Human vs LLM-as-a-judge evaluation results on the "overall poem quality" dimension.
Judge
Human
DeepSeek-RI
Gemini-2.5-Pat
GPT-4o
Mcan
2.43
4.26
3.00
3.69
Standard Deviation
1.09
0.20
0.63
0.22
In conclusion, our selection of Gemini-2.5-Pro was a rigorous decision to ensure the quality and validity of our evaluation.
24

---

Published as a conference paper at ICLR 2026
E AN OVERVIEW OF THE 30 SELECTED LLMS
Table 3: The features of 30 selected LLMs.
Thinking
Thinking
Open-source
DeepSeek-V (Liu 2024
DeepSeek-BB RJ (Guo 2025 QwQ-32B
DeepSeek-2 R -istill- Laama- Jama--B/70B
Guo
TTam
2025e
ctal.
2025
DeepSeek-RI ----beee--R---isilll M-ii------l--eeere
1.5B/7B/L4B/32B (Guo al 2025
Gemma-3-27B (Team al. 2025)
Grattafiori
al.
2024
Mistral-L 1- 123B (Team 2024
0 32B/7
Instruct (Qwen et al.
2025)
Closed-source
(Anthropic
2024
Claude-3.7-Somee (Anthropic
2024
Gemini-2.0-Pra Team 2025a
Gemini-2.5-PP 2-5-rro (Team 2025b
GPT-3.5-Turbo Team 2023
ol (Jacch et al.
2024)
GPT-4 (Achiam ct al. 2025
03-mini Team
2025d
GPT-40 Hurst al.
GPT-4.5
Team 2025c
25

---

Published as a conference paper at ICLR 2026
F LINGUISTIC FEATURES OF THE HUMAN-LLM POEM DATASET
Figure T0 demonstrates the top 20 case-insensitive words in the human poems and the poems generated by state-of-the-art LLMs representative of the AI companies, with stop words removed. Among them, Claude 3.7 Sonnet resembles humans the most, with cosine similarity of 0.602. Figure 11 illustrates the most frequent opening words and imagery used by LLMs and human poets. For the choice of first words, each LLM has a distinctive taste. For example. Llama-3.3-706 --sttunttt uses "In" significantly more than the other authors. Similarly, "The" appears more in poems generated by Gemma-3-27B and Gemini-2.5-Pry while GPT-4o uses "Beneath" and Claude-3.7 3.n---nn-n-n adopts "In" as the most common opening word. In comparison, human poets show more balanced preference for choosing opening words. As for imagery, both LLMs and human poets tend to use the imagery "eyes", "sun" and "face", but each author also shows different preferences. While human poets frequently write about "water" and "god", DeepSeek RI prefers "threads" and "bloom", QwQ-32B loves depicting "thread".
by Aorter
Humra
Saare
Dreptonic
02mri225 5
Germa
Lemed di 725 tntrus
Owo
Figure 10: The top 20 words across the human and LLM poem datasets.
Amoortie the ee Artivr
64
at
70-
Tho
Figure 11: The top opening words and top imagery cross the human and LLM poem datasets..
26

---

Published as a conference paper at ICLR 2026
G
MORE SHOWCASES OF LLM AND HUMAN POEMS
leats gre new file Thouugh denned seeetooron
This smeapee Tauue welerse www.eegeeeeiieiinei pire
Sharry
Asd cecootiir 1O97) io Coggham
came petureet holl
Sqae al aie quiau
you
Fallion ligial
Inepyy
Phin
Cit de-2 Fo
Figure 12: A showcase of the poems by Claude-3, Boaudd-..7-oomii 7-Sonnet and a human poet in response to the same prompt. The bar charts show their POEMetric scores judged by Gemini-2.5-Pro.
purtan prowar welt genn ai
Thamsrae oou tertens Then oder tumpur nat
Wettorn oat
Todps" and pank oftad aeconolatiss
The polntet uffic de ond surdeplitann
Thurs sboveic jun sorther dioee
vige wiigg
aheeee contaets
Eave the adt der
cncertiinng nurt porbiggion
surio proaar sall. SrateL3PP
Themrgan resuel gied lachen
Figure 13: A showcase of the poems by Gemini-2.5-PPe and human poet in response to the same prompt. The bar charts show their POEMetric scores judged by Gemini-2.5-PP
27

---

Published as a conference paper at ICLR 2026
H POEMETRIC SCORES OF HUMAN POETS AND ALL 30 LLMS
The average scores basic cnrnucciio-followwiii abilities all 30 LLMs are shown in Figure 14 the average scores of advanced creative abilities of both human poets and LLMs Figure 15 those of overall poem quality in Figure 16 and those of human authorship estimation in Figure T7 Overall, models with more parameters within the same family series performed better in poem generation. Thinking models were not necessarily better than their non-thinking family members; for instance, GPT-4o and GPT-4 ranked higher than and o3-mini. Besides, DeepSeek-RR R--issll models were generally worse than the original models, except that Distill-L Jaaa-...m.am0000 performed better than its original.
4.-
Lamr Cemee
Gaee de
6RE
Genmande
Baeis tratruatti Folowing Aovenge Btrwe
Clordoo Ceeadises
Ceniee
3.500
Paromator: (B)
Figure 14: Basic InstrrctionnFoollowingg Abilities, Average Scores
Desoleee Duti Over Sales
Lina Serie
Adteented Rotteen Auvenge Stpons
Gamon Teme
GPT Canie
Genma
OPF2
Deedeeedt000
Paromzters (B)
Figure 15: Advanced Creative Abilities, Average Scores
28

---

Published as a conference paper at ICLR 2026
DeeoCeee Distl Over Sates
lurpo
Deo 220
Oveen Prowe Awengg Soceas
3,943
18
Figure 16: Overall Poem Quality, Average Scores
Deopfook Dis Oper Sotes
Q4O 120
Homee Asthoonip Eximates Averge Scrres
Disadion
Gamon
95
Gervel
12
12
Figure 17: Human Authorship Estimation, Average Scores
29

---

Published as a conference paper at ICLR 2026
1 LLM USAGE STATEMENT
We have used LLMs only to aid or polish writing when drafting this paper.
30


Reviews:
[{"review_id": "e9kFN15AmB", "reviewer": "Reviewer_YAYx", "summary": "This paper presents a framework for the evaluation of poems. A new dataset of 203 English poems is created and experiments with 30 LLMs are presented, assessing human and LLM-generated poems. Results showed that LLMs fail to produce creative poems.", "questions": "A. In Line 467, the task is mentioned as authorship, but it is not really authorship attribution but authorship type (human vs. machine). Could the authors elaborate?\n\nB. In Figure 9, models are called \"thinking\". Do the authors mean models employing \"chain of thought\"?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "* A new resource is created, which can be useful for the community.\n* Extensive benchmark with 30 LLMs, providing useful insights.\n* Interesting insights are drawn from the experimental analysis.", "weaknesses": "* **Agreement not chance-corrected** is not measured by Cohen's Kappa, which is an established chance-corrected measure for such tasks. Percentage agreement of 0.66, with this in mind, is low and puts the findings of this work at stake. \n* **TTR is biased** to short texts (by design, the denominator increases linearly with the text length). Combined with the limited data statistics shared, the findings are at stake. For instance, in Figure 6, Llama could be producing shorter texts that would explain the higher TTR values. \n* **Definitions are missing** including what is creative and novel (Line 204), authentic (Line 040), famous (Line 470).\n* **Evaluation** is not well motivated: results with precision, recall and F1 are missing, though the metrics are mentioned in Appendix B.  \n* **Related work** is not properly studied and comparisons hinder the evaluation of this work. Existing evaluation frameworks (e.g., https://link.springer.com/chapter/10.1007/978-3-031-49011-8_1) could have been used to show case their limitations, and the same applies for the existing datasets. The need for another dataset/framework should be backed with empirical evidence. The same applies for the algorithm introduced (Line 145); if there is a lack of existing algorithms, this should be stated explicitly. A simple search, though, leads to one: https://arxiv.org/pdf/2406.18906.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "L0J1ngISFN", "reviewer": "Reviewer_YnfA", "summary": "This paper proposes a framework for evaluating poetry quality based on aspects of instruction following, creative ability, and general appraisal. The authors also construct a dataset, including 203 poems across 7 fixed forms, all collected and annotated. The paper examines multiple LLMs and uses LLM-as-a-judge to evaluate the poems.", "questions": "1. Would form difference affect the credibility of metric? \n\n2. How reliable are LLMs in each evaluation aspects? \n\n3. How are the human annotated samples selected? \n\n4. Other questions see weakness", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "Evaluating creative domains such as poetry remains an underexplored yet meaningful challenge. This paper explores this issue by proposing a comprehensive evaluation framework encompassing three key aspects and ten specific subdimensions.", "weaknesses": "- Given the general scarcity of poetry data, it is essential to clarify the rationale for selecting this particular dataset. In this paper, the dataset comprises 203 human-authored English poems covering 7 fixed poetic forms, accompanied by an equal number of 203 corresponding prompts. Each prompt specifies a combination of form, meter, rhyme, and theme. However, it remains unclear whether all 203 poems represent unique combinations of these attributes. Notably, the dataset appears imbalanced in terms of poetic form, being dominated by Ballads and Sonnets. Such imbalance could potentially affect the reliability of the overall evaluation. As reported in (Sonnet or not, bot? poetry evaluation for large models and datasets), large language models (LLMs) used as evaluators exhibit considerable variation in their judgments across different poetic forms, suggesting that the reported performance may also suffer from this concern. Also, LLMs tends to overestimate their own output. It is not clear whether Gemini-2.5-Pro performance is due to this effect or not. \n\n- Another major concern lies in the variation of agreement (PAo) between LLM-based evaluations and human judgments across different aspects. As illustrated in Figure 7, the overall mean scores appear consistent for human evaluators but vary substantially across different models and evaluation aspects. For example, poems generated by DeepSeek-R1 shows large discrepancies in lexical diversity and imagery between human and LLM evaluation, while GPT-4o exhibits inconsistencies across nearly all aspects between human and LLM evaluation. It would be necessary to examine the percentage agreement (PAo) for evaluations of poems generated by different models across various aspects. This could help determine whether LLMs themselves exhibit systematic biases toward certain models, which may further affect the reliability of the evaluation.\n\n- Details of the human evaluation are needed. Similar to Question 1, human evaluators assessed poems generated by both human authors and seven models, resulting in annotations for a total of 58 poems. However, additional clarification is necessary: do these 58 poems cover all 7 fixed poetic forms, or were they drawn from a random subset of the dataset? This distinction is important, as it directly relates to the concerns raised in Question 1 regarding form imbalance and its potential influence on evaluation outcomes.\n\n- The discussion of the theoretical background and motivation underlying the framework’s development is relatively limited. While the proposed metric places particular emphasis on creativity measurement, the related literature addressing this aspect of evaluation is largely absent. A more thorough engagement with prior work on the conceptualization and assessment of creativity would help strengthen the framework’s theoretical foundation (for example, Rethinking Creativity Evaluation: A Critical Analysis of Existing Creativity Evaluations). In this context, the evaluation of general creative tasks is also highly relevant. Findings from other creative domains suggest that LLMs used as evaluators tend to exhibit bias toward their own outputs (e.g., How Good Are LLMs for Literary Translation, Really? Literary Translation Evaluation with Humans and LLMs). This underscores the importance of human verification to ensure the credibility of the evaluation results in question 3.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xx6oXcex2O", "reviewer": "Reviewer_cK2s", "summary": "This paper presents a method for poetry generation and evaluation using modern LLMs. The authors curate a new human dataset with rich metadata (title, topic, style, meter, etc) and use this to show that machine generated poetry is high quality in terms of the structural elements of poetry, but falls short on creativity, style and metaphor etc. \n\nOverall this is well motivated and executed, however I feel that the main contributions of the work are the very detailed user study, as the method itself is quite simplistic, and as such this isn't a good fit for ICLR.", "questions": "1. There’s a lot of algorithmic detail in the appendix, around how the dataset is created that isn’t fleshed out and reproducible (appendix B, page 16, most steps are loaded: Levenshtein distance with walk? how to check for repetition and consistency? what is align?). (Note code is provided separately, according to the statement, so probably ok.)\n1. The use of prompting seems a bit naive, as this is formatted as a user survey which may not be well handled by many LLMs. I expect that tailoring the prompt may improve the quality and robustness of model outputs. Consider k-shot prompting, providing examples of good vs bad outputs for each criteria, asking for ratings and rationales, versus ratings alone, etc.\n1. Is thinking on/off a factor? There was a brief comparison of pairs of models with and without thinking, but I think this was confounded by model size. For some models thinking can be turned on and off. The same holds true for search, which may be relevant to evaluation. \n1. Are the eval metrics closely correlated to one another? Perhaps you might distill down to a handful of metrics instead.\n1. Might the model be directed to generate more creative poetry with more detailed prompting, or an iterative approach where feedback is given to make the eval more human-like (e.g., RL train over key evaluation metric)? This would be beyond the scope of one paper, but would be worthy of discussion. \n1. The human expert survey for eval includes “Please do not check any online or offline resources while completing this survey; we are interested in your own direct and personal response!” – what about plagiarism of text and ideas from other sources? I note that there was no such directive given to the LLM as judge eval, is search thought to be useful here?\n1. Agreement between human experts - the PoA analysis just uses LLM vs human setting, not human vs human. It's common to hold-out one human and compare them against the remaining humans as an upper bound.\n1. Regarding confidence in predictions: LLMs are often overconfident, see the extensive work on calibration of LLMs. The discussion appears to rely on what is a rather unreliable signal.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "This paper’s main contribution was a detailed analysis of poetry outputs, and a careful and detailed comparison between various LLMs versus humans. \n\nThe dataset and evaluation should be useful to others working in the area, the wide range of poem types and rich metadata would make it a valuable resource.\n\nThe method used for generating and evaluating poetry was well motivated, clearly presented and sound (although many details were pushed to appendices and attached code, the latter which I haven't inspected.)", "weaknesses": "As stated above, I don’t think ICLR is the right venue to properly assess the human elements of the work. \n\nThe language generation and evaluation aspects were solid, but this is more a data + prompting exercise than a scientific contribution. The real contributions of the paper were the detailed evaluation, which is reflected in the way the paper in written, and the split of materials between the main text and the appendices.\n\nAccordingly, I think this work would be better suited to other venues, e.g., computational linguistics, even CHI or more user-centric fields, where the reviewing process would be better able to judge the quality of the human study side of this work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XnxGPNmrb1", "reviewer": "Reviewer_xKLP", "summary": "The authors introduce a novel evaluation dataset and framework for formal verse poetry. They first collect 203 human-created poems which they annotate with meter, rhyme pattern, topic, and other styles. These annotations are then used to condition a wide range of large language models to generate style-conditioned poetry. To evaluate generated (and to a lesser degree also human) poems, the authors propose various evaluation schemes. E.g., rule-based evaluation of form, meter, and rhyme scheme; and LLM/human-as-a-judge evaluation of basic instruction-following abilities, advanced creative abilities, and general appraisal. The authors show that LLMs-as-a-judge correlate well with human assessments and that generated poetry generally follows stylistic constraints well but fails to reach the same mastery of language (use of literary devices, creativty, emotional resonance, ...) that humans achieve.", "questions": "* l.259 why would averaging with other LLMs introduce noise and bias? Wouldn't that just be model ensembling which is a common technique that usually works well? ByGPT5 (one of your cited papers) showed that general purpose LLMs (as of 2023) are bad at following stylistic constraints such as rhyme and meter. Do the authors have an intuition what changed with current LLMs which apparently now work much better?\n\nComments\n* The poems in Figure 3 are too small to read comfortably.\n* l480: ALthough -> Although", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* Poetry evaluation is an challenging research problem, especially for general-purpose large language models whose poetry generation performance is not well understood. The authors successfully investigate a gap and their findings will be useful for future research.\n* The authors promise to release their dataset artifacts which will be useful for future work.\n* The authors evaluate many different dimensions of poetry which give a broad overview of performance.", "weaknesses": "* The scope of the paper is a bit limited. It would have been interesting to see how some of the uncovered gaps between humans and generated poems could be closed either by model training or more advanced prompting. At least some suggestions for future work would have been insightful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)