A Confederacy of Models: a Comprehensive Evaluation of LLMs on Creative Writing
Authors: Carlos Gómez-Rodríguez
Paul Williams
Venue: EMNLP 2023 Findings (2023)
Keywords: LLMs
language models
creative writing
evaluation
text generation
storytelling
creativity
TL;DR: We evaluate 12 recent LLMs, comparing them to humans, on a creative writing task using human evaluation with a detailed rubric.

Abstract: We evaluate a range of recent LLMs on English creative writing, a challenging and complex task that requires imagination, coherence, and style. We use a difficult, open-ended scenario chosen to avoid training data reuse: an epic narration of a single combat between Ignatius J. Reilly, the protagonist of the Pulitzer Prize-winning novel A Confederacy of Dunces (1980), and a pterodactyl, a prehistoric flying reptile. We ask several LLMs and humans to write such a story and conduct a human evalution involving various criteria such as fluency, coherence, originality, humor, and style. Our results show that some state-of-the-art commercial LLMs match or slightly outperform our writers in most dimensions; whereas open-source LLMs lag behind. Humans retain an edge in creativity, while humor shows a binary divide between LLMs that can handle it comparably to humans and those that fail at it. We discuss the implications and limitations of our study and suggest directions for future research.

A Confederacy of Models: Comprehensive Evaluation of LLMs on Creative Writing
Carlos Gómez-Rodríguez Universidade da Coruña, CITIC Department of CS and IT 15071 A Coruña, Spain carlos. gomezêudc. es
Paul Williams
School of Business Creative Industries University of the Sunshine Coast
Sunshine Coast, Australia pwillia edu. au
Abstract
Comparison on overall rating
We evaluate range of recent LLMs En- glish creative writing, challenging and com- plex task that requires imagination, coherence, and style. We use difficult, open-ended sce- nario chosen to avoid training data reuse: an epic narration of single combat between Ig- natius J. Reilly, the protagonist of the Pulitzer Prize-wi biizewwinning novel Confederacy of Dunces (1980), and plerodactyl, prehistoric flying reptile. We ask several LLMs and humans to write such story and conduct human evalu- tion involving various criteria such as fluency, coherence, originality, humor, and style. Our results show that some state-of-ther thet---rt- mercial LLMSs match or slightly outperform our writers most dimensions; whereas open- source LLMs lag behind. Humans retain edge in creativity, while humor shows a binary divide between LLMs that can handle it com- parably to humans and those that fail at it. We discuss the implications and limitations of our study and suggest directions for future research.
80
Overall rating
40
cedi
Model
Figure I: Box plot comparing overall ratings for stories by humans and 12 LLMs, arranged left to right by mean overall rating. Boxes show median, quartiles QI-Q3, and whiskers at 1.5 IQR, with values outside that range plotted as outliers. Filled red circles represent means.
1 Introduction
recent years, large language models (LLMs) have achieved remarkable progress in wide range of language processing and generation tasks, such question answering, machine translation, text summarization, among many others (Zhao al., 2023). This has motivated research on evaluating and comparing the performance of LLMs in various tasks, both between each other and with respect to human performance; including both task-specific evaluations (see e.g. (Jiao et al., 2023; Gilson et al., 2023)) and overarching benchmark suites that seek to provide comprehensive evaluation throughout many dimensions (Hendrycks et al., 2021; Liang al., 2022; Srivastava et al., 2022).
are "often of astonishing quality", and Clark et al. (2021) showed that humans cannot reliably distin- guish human- from LLMMauthored stories. How- ever, and despite the amount of papers experiment- ing with LLMs for this purpose, an evaluation com- paring the abilities of current LLMs standalone systems for creative writing seems to be lacking.
Here, we provide such an evaluation, comparing the storytelling capability of 12 recent, instruction- aligned language models between each other and with human writers. We do so using rubric based on established creative writing evaluation propos- als (Davidow and Williams, 2016; Carey et al., 2022), but specifically adapted the task. Our comparison is performed on a purely zero-shot set- ting, with a natural human prompt (based on a combat between Ignatius J. Reilly, protagonist Confederacy of Dunces, and a pterodactyl) that
Creative writing also one application where LLMs have been observed to produce good results. According to Franceschelli and Musolesi (2023), their generated outputs in poetry or storytelling

---

has been specifically chosen be challenging and meaningful while preventing as much as possible the option for LLMs to resort to regurgitating or adapting material from their training set.
tant differences with respect to our work: (1) they use prompt-basee learning, providing examples adapt the model to the task, rather than purely zero-shot conversational prompt, (2) they evaluate a single LLM while our goal is to compare LLMs, and (3) they use pre-existing story datasets, which increases the risk of models benefitting from simi- lar stories present in their training set, something that we have tried to avoid as described below.
2 Related work
LLMs in creative writing LLMs have been used in creative writing since their first generation, with models like GPT-2 (Radford et al., 2019) or BART (Lewis et al., 2020). However, these models suffered from lack of long-range coherence lead- ing contradictions or inconsistencies when gen- erating stories (Nye et al., 2021). Thus, they were not viable as standalone story generators. Instead, they were used either with specialized fine-tuning for the task (See et al., 2019); or as components of systems that incorporated external knowledge (Guan et al., 2020, 2021), storyline planning (Tan et al., 2021), or both (Xu et al., 2020); for co- creation with a human in the loop (Swanson et al.. 2021), a line of research that has also continued with newer models (Yuan et al., 2022; Chung et al., 2022; Mirowski et al., 2023).
In another recent preprint, Gerrido-Merchán et al. (2023) generate Lovecraftian horror literature. However, they also focus on a single LLM (GPT-4), using careful prompt engincering to optimize its performance rather than a pure zero-shot setting, and evaluation is only on whether humans dis- tinguish AT Al-generated from real stories (concluding that, in those circumstances, they cannot). Sawicki et al. (2023) apply a similar evaluation (but auto- mated) to Whitmanian poems generated by three versions of GPT, also with a negative result.
Finally, concurrently with our study, preprint by Chakrabarty al. (2023), released few months after our submission, evaluates three LLMs for cre- ative writing in a more similar way to ours: they apply human evaluation to compare stories by hu- mans and LLMs in a zero-shot setting. However, there are important differences in methodology and scope between both studies. A comprehensive com- parison will be made in Section 5, following the exposition of our methods and results.
Here our goal is not to produce specialized system, but to evaluate the performance of LLMs by themselves as creative writers. Thus, we focus on the purely zero-shot setting, where a general- istic LLM is asked to write a story with no extra fine-tuning, in-context learning (Dong et al., 2023), prompt engineering or additional components. This has only become viable with the extra coherence and consistency in long texts provided by newer LLMs, especially those that are aligned to follow in- structions with instruction tuning (Wei al., 2022; Sanh et al., 2022) or reinforcement learning with human feedback (Ouyang et al., 2022).
Creative writing evaluation Creative Writing is challenging and complex performative language act that requires a number of skills, such as an ex- pertise in craft, cultural and literary competency. linguistic fluency, coherence, complex connotative and metaphorical levels of understanding, innova- tion, originality and imagination, to name a few. The craft of writing involves innovation with style and voice, needs a fundamental understanding and use of structural elements (grammar, spelling, punctuation), craft elements (plot, character, set- ting, point of view and imaginative capacity, such skills defined by Bloom as 'putting elements to- gether to form coherent or functional whole; re- organizing elements into new pattern or structure through generating, planning, or producing' (An- derson and Krathwohl, 2001, p.21). Evaluation of creative writing therefore must take into account all these factors, and assessment in university Creative Writing courses is usually based on a rubric that attempts to measure the basic elements of narrative
To our knowledge, there was no previous work in this line. In fact, evaluation in creative writing is a conspicuous gap in LLM evaluation benchmarks: the huge BIG-bench suite (Srivastava et al., 2022) currently has over 200 tasks, but does not include any creative writing, and HELM (Liang et al., 2022) cites as an "aspirational scenario" for future work. This likely owes to benchmarks focusing on easily- automatable metrics, whereas the gold standard for creative writing is human evaluation (Belz and Reiter, 2006), which is much costlier.
The closest previous work to our proposal the recent preprint by Xie al. (2023), where GPT-3 is compared to previous storytelling systems via hu- man evaluation. However, there are several impor-

---

craft, as well as the specific requirements on the as- signment (Kroll, 1997; Norris, 2013; Davidow and Williams, 2016; Wise and van Luyn, 2020; Carey et al., 2022).
The genre requires humor, which is consid- ered be an especially subtle feature of hu- man language and challenging for machines, including LLMs, to exhibit (Jentzsch and Ker- sting, 2023).
3 Materials and Methods
While the task is challenging due to putting to- gether two unlikely antagonists, the prompt's level of detail is open-ended enough to give ample space for creativity, no specifications are made about setting, weapons, outcome or other aspects of the story.
3.1 Task
The chosen task to compare the LLMs under con- sideration is defined by the following prompt:
Write an epic narration of a single com- bat between Ignatius J. Reilly and a ptero- dactyl, in the style of John Kennedy Toole.
3.2 Models
We gave the task to confederacy of large language models, composed of all such models we could find that (1) were available to the authors by April 20 2023, which was the cutoff date to build our corpus of stories, and (2) were adjusted to conversational settings and instruction-followee niques like instruction tuning (Wei et al., 2022; Sanh al., 2022) or reinforcement learning with human feedback (Ouyang al., 2022). This is in contrast to "vanilla" language models configured just predict the next word, like plain GPT-3 (Brown et al., 2020) or Llama (Touvron et al., 2023), which generally cannot handle natural prompts like the one we use. We only included distinct models, not front-ends to the same model (but we did in- clude derived models with substantial additions, like Bing Chat which is claimed to use GPT-4 but adds search capabilities, or various models that were fine-tuned from Llama weights). For models that came variety parameter sizes, we used the largest one, or the largest we could execute with local or remote resources. For models with several available versions, we used the latest available, ex- cept in the case of ChatGPT where we included both the GPT-3.5 and GPT-4 versions. due to the wider availability of 3.5 (the latest version offered for free cutoff time) and the lack of information on whether GPT-4 is an incremental improvement or a different model with its own tradeoffs.
The prompt is provided to the models from a fresh state, without previous context.
We believe this task is particularly adequate to challenge the capabilities of models for creative writing, for the following reasons:
by using tech-
It is a non-standard, "wacky" scenario that has been invented for the occasion, so it is very unlikely that the systems training sets contain coincident similar tasks, pieces of stories that can be reused for the task. No informa- tion about this task was posted to the Internet or disseminated in any other way before the LLMs were prompted.
It features specific literary character, Ig- natius J. Reilly, so we can evaluate the models how they capture the personality of the character. At the same time, this character appeared in only one book, and does not seem to have been the target of fan fiction. This makes the task more challenging due to hav- ing to capture the personality of the protag- onist from scarce material, while making it unlikely that the model can just reuse material from existing stories.
In turn, Confederacy of Dunces is the only work of its author John Kennedy Toole, so the author's ss style also needs to be captured from scarce material.
This selection yielded the following 12 language models. We list them in alphabetical order as chronological ordering would be challenging, due to closed releases, opaque updates from some of the commercial products, and many of the models being released almost simultaneously:
This novel is widely considered to be classic of comic fiction, and won the 1981 Pulitzer Prize in the Fiction category. Thus, writing a story about its protagonist in the author's style sets an adequately high bar.
Alpaca (Taori al., 2023), a Stanford model fine-tuned from Llama (Touvron et al., 2023) on instruction data generated with the self-instruct

---

methods of (Wang et al., 2022). We use the 13B- parameter version, the largest available at cutoff. Bard, Google's experimental conversational LLM offering, claimed to be based on a lightweight version of LaMDA (Thoppilan et al., 2022). It use content from the web to answer questions. Model details have not been made public.
described in (Bai al., 2022), a 52B -ar-aarmeterr model aligned to be helpful with Constitutional AI, a list of guiding principles provided to the model, combined with a mix of supervised learning and reinforcement learning with AT feedback. We used version 1.2 of the model.
Dolly 2.0 (dolly-v2-125)) guage model trained by Databricks, derived from EleutherAl's Pythia-12B model (Biderman et al., 2023) after fine-tuning on 15K instruction corpus. At cutoff date, it was the only available conversa- tional LLM where all of its components could be considered fully open source?, as the code, weights and instruction datasets all have open-source li- censes compatible with any use, including commer- cial use, and data from proprietary systems like ChatGPT has been used for finetuning.
12B ----aaraeeter lan-
Bing Chat, an LLM offered by Microsoft's Bing search engine. Claimed to use GPT-4', further tech- nical details have not been made public. The model performs web searches and uses the results to aug- ment its context window with relevant information. It can also provide links to sources for its claims (although this not relevant for our creative writ- ing task, where no such links were provided or needed). We used its Creative mode, the obvious fit for our task. A problem worth mentioning is that we found the model to be subject to heavy cen- sorship, which affected our experiment: in most prompting attempts, the story would be deleted by the filtering system before being finished. When this happened, we just reset and re-prompted the model, repeating the process until a full story was obtained. Over 100 tries were needed to obtain non-censored stories. We are aware that this may introduce bias (as non-censored stories may have a different quality distribution than what the model could potentially generate without the filter) but this is unavoidable from our end, since we cannot bypass moderation. In any case, the sample does reflect what a user can obtain from the end product, as the censored stories are out of reach.
GPT4AII-J (Anand et al., 2023b), an improve- ment over its predecessor GPT4AIl (Anand et al., 2023a). The base model is the 6B-parameter GPT-J (Wang and Komatsuzaki, 2021), which has been fine-tuned on dataset expanded from mix of existing sources.
Koala (Geng et al., 2023), a model fine-tuned from Llama (Touvron et al., 2023) by researchers from the university of Berkeley, on a variety of dialogue data obtained from the web. We use the 13B- 13B- parameter version.
O OpenAssistant (Köpf et al., 2023) is an LLM fine-tuned on large, free, human- human-generatee generated con- versation corpus created by a crowdfunding effort involving over 13,500 volunteers. We used the OA- SFF--Llma--30 model, fine-tuned from the 30B- parameter Llama (Touvron et al., 2023) model. StableI. is Stability Al's series of language models. We used With 7B parameters, this is the largest model avail- able (at cutoff time) among a series of models trained on dataset built from The Pile (Gao et al., 2021) and fine-tuned on a combination of conver- sational LLM corpora.
ChatGPT with GPT-3.5, an OpenAl successor to the 775B-parameter GPT-3 model (Brown et al., 2020) which was tuned using reinforcement learn- ing with human feedback, namely a variant of the InstructGPT method by Ouyang et al. (2022). We used the March 23 version provided by OpenAl's free ChatGPT service.
ChatGPT with GPT-4, the most advanced lan- guage model released by OpenAl at cutoff time. A description of the model is available in (OpenAI, 2023), although essential technical details like the number of parameters have not been published. We used the March 23 version provided by OpenAl's ChatGPT Plus service.
Vicuna (Chiang et al., 2023) is another member of the fanily of models obtained by fine-tuning Llama (Touvron al., 2023), in this case with user- shared conversations with ChatGPT. We used the 13B- -a--aarmeter version of the model.
Claude is a language model trained by An- thropic. While details about its implementation are not public, is known to be a sucessor of the model
3.3 Evaluation rubric
The creative writing rubric designed for sessment of creative writing assignments in uni-
htt http://lllssbbio bing. htt/seeacc//marchh 2023/ Confirmedt the-new-Bil Sar--inng-rnm caae-ruuss------- on wnw-opennaaass-aaaeaaaaaa GPT-4
https:///ppeesourc..orr/geeeeeee annotateed

---

Description
Overal//holstickcohesiee readability of the story (not just compilation elements).
Use of key narrative elements vocahulary choice, imagery, setting, themes, dialogue, characterisation, point view. Structural elements and presentation which reflects the control of structural elements such as spelling, grammar, punctuation, paragraphing, and formatting.
Overall plot logic: hook, conflict, imitial crisis, rising and falling action, denouement/ resolution (Treitag's pyramid). new knowledge, avoidance of cliché and derivative tropes. Incorporation of the John Kennedy Toole style of writing using the indicators/ characteristics listed. Understanding habitation of the epic genre of heroie/legendar adventure.
Description and credibility of a single combat scene.
Accurate inclusion of two main characters Ignatius J. Reilly and a pterodactyl in action and description.
10
Use of a characteristically dark humorous tone.
Table 1: Creative writing evaluation rubric. All items are scored out ten points. Marking guideline: Emerging 1-4, Competent 5-8, Sophisticated 9-10.
versity creative writing courses, and is taken in part from university textbook by one of the au- thors of this article, Playing with Words (Davidow and Williams, 2016) and article that justifies the of this rubric (Carey et al., 2022). This rubric evaluates creative production in five holistic craft- based criteria and measures craft skills based on a writing style outlined the article: among others, Flaubert's ' insistence on le mot juste (the right word or expression), Strunk and White's The Elements of Style (2008[19) (188[1918)). 8)), George Orwell's rules for concreteness and clarity (Orwell, 1946); and An- nie Dillard's rules for writing good prose (Dillard, 1981).
379 words (std = 248, min = 23, max = 1223).
Then, we also asked 5 human writers to each write a story following the same prompt. For uni- formity, we suggested a length range coherent with the LLM-generated stories (250 to 1200 words). The writers were Honours and postgraduate Cre- ative Writing students that volunteered for the task. and all of them studied the specific task require- ments (e.g. John Kennedy Toole's style) before writing their stories. However, they were not given access to the Al-generated stories and they were instructed not to use LLMs at all to help them write.
The result is, thus, a corpus of 60 Al-generated stories (5 for each of the 12 considered LLMs) plus an additional 5 numan-generaan
The rubric for this Al task adds five more criteria which address the specific prompt requirements, such as genre, style, tone, character and action. Each of the ten criteria is awarded 10 points out of total 100 points. The rubric has been specifically designed to measure the quality of writing craft, to avoid formulaic, rule-based writing and to address the very specific task addressed here.
stories, all in plain text format. The corpus is available at https: //doi. org/ wrtg///..21//////////// 5281/zenodo. 8435671.
The only preprocessing made to the stories is that (1) we removed leading sentences that de- scribed the task, often present in LLM answers (e.g.: "Here is a potential epic narration in the exaggerated style of John Kennedy Toole's A Con- federacy of Dunces:")) (2) we removed titles from stories that had them, and (3) we unified paragraph formatting, leaving one line between paragraphs all the plain text files. Other than these changes, made for uniformity and to preserve the blindness of the rating process, we left the text as it was.
The criteria are detailed in Table 1, with more details given in the Appendix C. The holistic scale (emerging, competent, sophisticated) guides hu- man raters assess holistically: 'a holistic scale measures the relative success of a text but does so through a rubric that incorporates many of the traits in analytic scoring as heuristics towards a concep- tion of a whole rather than as sum of autonomous components' (Perelman, 2018, p.16).
We recruited 10 raters, also Honours and post- graduate Creative Writing students that were ac- quainted with the specific requirements of the task, and we instructed them to grade stories according to the rubric. Since the raters were volunteers, to keep the workload low, each rater did not rate all the stories. Instead, divided the 65 stories into groups of 13 stories each (each group containing one story by each LLM, plus one story by a human) and assigned one rater to each group. In this way,
3.4 Evaluation methodology
We prompted each of the LLMs 5 times with the prompt given in Section 3.1. Each prompt was made from a fresh state, i.e., in a zero-shot setting without any previous context that could help guide the models. The resulting stories had an average of

---

Rubric Hem
overall
chatept-gpe4
8.7.1..8
8.7 107
1.
7.6.11
8.0.1.2
81114
19.16
6.0.288
80.2
claude 12
8.0
80
8.
19
2.1223
7,542
6,4
75
74425
6.5
74.4 150
human
7.34-2.3
78+18
7.3+17
72+13
6.3-22
77
bing 7,8+2
7.5+22
7.-+17
7
1.0+16
6.8+24
5.3+29
6.2-21
74422
6.2+26
69.5 15.4
chos
70122
54125
76212
knala
7.5.:2.5
6.7222
8.2
6.8226
584233
481227
58824
5.5-23
5.8223
3.432
60.0
vicuna
7.9
6.74+-16
8.1+13
7.0%16
5.1419
4.6333
5.7-23
6.1=19
5.4227
24+19
59.0-1133
ea
1:2.1:22
581224
7.2125
622126
4.112.1
39124
58124
65522
4,3123
291131
54.7
hard
6.5.+:2.5
4.9+211
6.8=19
5.5=27
3.9+2.1
3.8
4.7+26
4.6=27
5.0+24
2,5+2
48.2+220
gotfall
6.5-1:2.2
541225
5.622
stablelm
5.5
5.0
6.6-199
3,842
3.24-15
2,1422
4,4
3.8-2
2,9426
14
38.7 172
dally
4.6112
30222
322119
4.2138
3.1122
3.5218
3012
1,515
37.9
alpaca
5,243311
3.1+14
4.9-3
4.2+19
1.9+1
20414
3.743
3,9
2,1+15
1.1+0.6
32.1 +157
average
6.942.1
62419
7.3-18
6.242
5,242
4.722
55823
5.--2
5.1223
3,422
56.6:1188
Table Results for each rubric item, as well overall score. Each cell shows average standard deviation for the ratings achieved by given model (or human writers) on a given rubric item. The bottom line shows the average among all models (and human writers). Models are sorted by overall score. The best result for each rubric item is highlighted in boldface.
we ensure (1) that we have at least two ratings per story, allowing us to measure inter-rater agreement, (2) that comparisons are fair, in the sense that no LLM (or the humans) is advantaged by being as- signed more lenient raters, because each LLM (and humans) receives exactly one rating by each of the 10 raters, and (3) since each rater always gets one story from each model (and one human), we can ex- pect that each will be rating a diverse set of stories covering a wide range of ability levels, which helps the marking process as it allows for comparative analysis between various performances, enabling more accurate pinpointing of each story's quality.
agreement", which is a positive result taking into account the obvious subjectivity involved in rating
stories. If we instead focus on overall scores (sums of rubric items), the Pearson correlation between the scores given each story by each group of raters is 0.58 (p 0.00001), again indicating a reasonable degree of consistency between raters given the subjectivity of the task.
4.2 General overview
Table 2 shows a comprehensive overview of the ratings that each of the LLMs (and humans) ob- tained for each rubric item, as well as terms of overall score. Additionally, a box-and-whisker plot comparing overall score can be seen in Figure 1. ChatGPT with GPT-4 generates the best-rated stories, both in terms of overall score and in 8 out of 10 of the individual rubric categories. However, human writers are rated best in terms of originality (rubric item 5), and Claude was rated best the use of dark humor (rubric item 10), with humans a close second. GPT-4 is also remarkably consistent, showing low standard deviations not only with re- spect to human writers (which is expected, as our human stories were authored by five different hu- mans, whose skill levels may vary) but also with respect to the rest of the LLMs.
Stories were assigned random identifiers before sending them to raters, so that the process was blind: to avoid biases, raters knew that they would be evaluating human and Al-generated stories, but were unaware of the origin of each story.
Raters were sent all stories at once and they were free to go back and change the ratings of previously-rated stories. In addition, all of them were experienced assessors in terms of Creative Writing texts, with previous experience in applying the scale. These precautions mitigate the need for specific calibration (Karpinska et al., 2021) that would strain resources.
4 Results
If we compare LLMs to each other, the best performances correspond to commercial offerings, including (apart from the aforementioned GPT-4) Claude, Bing Chat and the GPT-3.5 version of ChatGPT. Open-source models are clearly behind, with the best (Koala) achieving 60.0 overall score, contrasting with the 80.2 obtained by GPT-4. Al- though the best-performing L.MMs are generally better across the board, some idiosyncrasics can be observed: e.g., GPT-4 tops almost all rubric items
4.1 Agreement
To gauge the reliability of our results, we com- pute inter-rater agreement between the two rat- ings given to each story for each individual rubric item. We use linearly weighted Cohen's 's kappa (Cohen, 1968), which is appropriate for ordinal scales like ours, obtaining a value of 0.48, 95% CI (0.43.0 0.54]. This interpreted as "moderate

---

but is outperformed by two LLMs humor. When we compare LLMs to human writers, sig- nificance testing on overall score (2-tailed t-test assumiig unequal variances) fails to detect signif- icant differences between humans and the top 6 Al models with a 0.05. Only the bottom Al models are significantly worse than humans this significance level. Note, however, that the test has a low statistical power due to the small sample size (10 ratings per model). If we instead perform a test on individual metrics, so our sample size is 100 (with the null hypothesis being no difference be- tween humans and each LLM in random individual metric scores), then GPT-4 is identified signifi- cantly better than the human writers (p 0.00031), Claude and Bing's scores are not significantly dif- ferent from those of humans, and all the rest of the LLMs score significantly worse than humans.
Comparison on dark humor rating
10
Humno rating
chatape spot opo4
claude12-
human
bing.
vicura
08
apstellj
stablelm-
doly
alpace
Chatspt-pp33
Figure 2: Box plot comparing humor ratings for stories generated by humans and 12 LLMs, sorted left to right by mean overall rating. Notation as in Figure 1.
Looking at individual metric scores, structural elements (rubric item 3) are the easiest category (with an average rating across all stories of 7.3. and all models but one obtaining at least 5 on average). Humor (rubric item 10) is clearly the hardest, with an average score of 3.4, and we will analyze it in more detail below. Incorporating John Kennedy Toole's 's style is the second hardest, with 1.7. Comparing humans to LLMs, humans (as al- ready mentioned) excel at originality and humor, but are clearly behind the best LLMs in terms of readability (item 1), where they are outperformed by LLMs, and even more so in use of the epic genre (item 7), where they score 4.9 and are out- performed by 8 LLMs.
and 6.5; whereas the rest of the models achieve very low scores of 3.4 or less. Significance testing also confirms this divide: despite the small sample size of 10 humor ratings per model, a 2-tailed t-test with a 0.05 confirms that the models in the sec- ond group are significantly worse than the human writers, well the LLMs in the first group. This suggests that grasping human humor might be an emergent ability of larger LLMs.
In this respect, a recent preprint (Jentzsch and Kersting, 2023) concluded that ChatGPT has "a limited reflection of humor" and "cannot yet confi- dently create intentionally funny original content". This study used the GPT 3.5 version of ChatGPT, so it is in line with results (in which that model obtains average humor score of 3.3). However, as we have seen, more powerful LLMs have over- come that limitation, as their generated stories are clearly rated as humorous.
We now analyze in more detail some of the in- dividual items that show more interesting compar- isons between human writers and LLMss.
4.3 Humor
Figure 2 shows a box plot that complements the information on Table 2 for the humor rubric item. The results for this item have two interesting char- acteristics. Firstly, is clearly the most difficult rubric item, with an average score across models of 3.4, and the best obtaining 6.5. Even humans obtain a lower score in humor than in most items, which may be consequence of humor being highly subjective. Secondly, as evidenced both in the ta- ble and plot, there is rather stark binary divide between the contenders that "get" humor and those that do not: Claude, Bing and GPT-4, together with the human writers, obtain average scores between
4.4 Creativity
We now focus on rubric item 5, which rates creativ- ity and originality, as it is a hallmark of creative writing and also the only category where human writers have outperformed all the L.MSs in our anal- ysis. Figure 3 shows box plot that complements the information Table 2.
The same three LLMs that stood out in the humor category are also the best in terms of creativity, although the difference is not as stark. Regardless, a t-test still distinguishes both groups as it shows all

---

Comparison on creativity rating
Comparison on epicness rating
Creativity rating
Epieness rating
chatapt gpo4.
claude12-
human
bing.
kodla
vicure
bard
apeaal-j.
stablelm.
dally
alpaca
chatape spo4
claude12-
human
bing.
chatspt chatgpt-gpt35
vicura
08
apstali-j.
doll
alpace
Figure 3: Box plot comparing creativity ratings for sto- ries generated by humans and 12 LLMs. sorted left to right by overall rating. Notation as in Figure I.
Figure 4: Box plot comparing epicness ratings for sto- ries generated by humans and 12 LLMs, sorted left to right mean overall rating. Notation in Figure I.
the rest the LLMs to be rated as significantly less creative than our human writers, while these three we cannot reject the null hypothesis that they are as original as the human writers.
Discussion
We have evaluated recent LLMs creative writ- ing task in English, using carefully-designed sce- nario provide demanding challenge and avoid confounding factors like training data memoriza- tion (Carlini et al., 2023). To our knowledge, this is the most thorough evaluation of LLMs on creative writing conducted so far, both in terms of scope (12 LLMs considered, plus comparison to human writers) and detail (using human evaluation with 10-item rubric based established creative writing evaluation practices).
Overall, from our results and in terms of human perception of the output, the answer to whether LLMs produce creative stories (Franceschellli and Musolesi, 2023) yes, although humans still retain an edge in this respect.
4.5 Epicness
Finally, we analyze rubric item (understanding and habitation of the epic genre) for the opposite reason as the previous section: is the item where humans do worst compared to LLMs (see Table 2). A box plot is provided in Figure 4.
Simultancously our work, the recent preprint by Chakrabarty al. (2023) provides an evaluation of three of the top-performing commercial LLMs (ChatGPT, GPT-4 and Claude) for creative writing. This approach is close to ours, as it uses the models in zero-shot setting and evaluation is performed by humans using specific rubric. However, there important methodological differences between both studies, which we summarize here:
In this case, the results have more atypical profile, with substantial difference with respect to overall scores. Two models perform significantly better than the human writers (a 0.05): both ver- sions of ChatGPT. Other six models obtain better average rating than humans, but the difference is not detected as significant.
The human stories used by Chakrabarty et al. (2023) stories published in the New Yorker, by highly successful authors (including Nobel prize winners), whereas ours are written by Creative Writing students.
Interestingly, Bing clearly lags behind both Chat- GPT versions, despite being based in GPT-4. This might be related to bias introduced by the system's censorship. On the other hand, some models whose overall scores are in the bottom half (OpenAssis- tant, GPT4AII) are reasonably good epic narra- tion, outperforming humans and Bing (which are better than them in almost all categories).
their setting, the human-writter stories are pre-existing (and selected for publication the New Yorker, as mentioned above) so their

---

writers were unconstrained when they created them, while the LLMs have to adapt write an alternative story with the same plot. In ours, humans and LLMs are given the exact same prompt to work with.
are compared to highly successful human stories by top authors that wrote freely and the LLMs are asked to adapt to their plots. We hypothesize that these are the main reasons for the difference in out- come. On the other hand, item 5 in the list above could in principle benefit LLMs, and there are other factors that could benefit humans or LLMs in non- obvious ways (including items 3, 4 and 6, as well as different story genres and target lengths). This underscores the need of more studies in this area.
3.
In terms of length, the stories they work with are over thrice larger than ours on average. In addition, while both studies try make sentence lengths similar between humans and LLMs, in their case the human writers origi- nally wrote their stories unconstrained (or un- der loose constraints) and the LLM-generated stories were calibrated to have similar lengths by an iterative prompting process. In our case, the LLMs were unconstrained in terms of length, and the human writers were sug- gested to target length range loosely similar to L.g--reeertted stories. Thus, with respect to theirs, our approach has the disadvantage of a looser control on story length, but the advantage of using a single zero-shot prompt.
6 Conclusion
The results show that state-of-the-art LLMs can perform a creative writing task very competent level, with the top two (ChatGPT with GPT-4 and Claude) achieving high scores that outperform hu- man writers in most rubric categories. While we must be careful not to take this as evidence of "su- perhuman storytelling" (both because our sample size is not enough to draw such categorical con- clusions, and because our 5 human writers are not necessarily representative of human writing abil- ity as a whole); does at least strongly suggest that these models' stories are not distinguishably worse than those by reasonably-traine This is even more remarkable given that we did not use any in-context learning or other techniques to optimize the LLMs for the task, but just straight- forward prompt from fresh state, so it is possible that even better results are achievable with careful prompting.
4. Their study spans a variety of story prompts, while we focus ting. The flip side is that our rubric can be adapted specific requirements like humor and Toole style, whereas theirs is necessarily more generic. In addition, our narrower fo- cus allows us to have LLMs generate several alternative stories, so we can perform more statistical analysis: we consider the distribu- tion within each LLM and perform statistical testing, which cannot be done in Chakrabarty et al. (2023)'s ' setting as they generate a single story per prompt and LLM.
single prompt and set-
humans.
Our analysis also shows that the best results are achieved by commercial LLMs, with open-source models clearly lagging behind at the moment.
Looking at individual characteristics, humans retain the lead in originality, while LLMs tend excel in more technical aspects like readability or structure. Humor is an especially challenging as- pects where most LLMs utterly fail, but the best three models do succeed at achieving himan-like ratings, contrasting with results on older LLMs that showed their lack of grasp of human hu- mor (Jentzsch and Kersting, 2023).
5. Since their study is based on existing stories that are published online, there is the possi- bility that some are contained in the tested LLMs' training data. In our case, we designed the study to prevent training data reuse.
6. The rubrics are different: Chakrabarty et al. (2023) use a rubric based on the Torrance tests of creative thinking (Torrance, 1974).
Interesting avenues for future work include eval- uation of different literary genres, languages other than English, and studying whether the quality of the generated stories can be improved with prompt engincering fine-tuning.
The outcome of this study is substantially differ- ent from ours, with LL.-gennrated stories rated clearly behind human-authores ones. This is not surprising considering the methodological differ- ences: in particular, differences 1 and 2 in the list above clearly set a higher bar for LLMs, as they
Selected stories from our corpus (available at https: http://ooooroeee org/10. 5281 /zenodo. 8435671, together with all rating data) are in Appendix E.

---

Limitations
Ethics Statement
While the use of conversational LLMs has raised various ethical challenges, creative writing has been argued to be one of the best uses for these tools from a human-centered AI point of view, as long as Al-generated stories are identified such to avoid misleading readers publishers (Sison et al., 2023). In our study, raters were blinded story authorship but they were previously informed that they would be dealing with AI and human- generated stories. In the published corpus, cach story is identified as human or Al-authored.
Commercial LLMs and reproducibility While some of the LLMs considered proper scien- tific artifacts, trained with documented method- ology and whose code and weights are available, others are closed commercial products and there is little public information about them, hindering reproducibility. While we have reported version numbers (where available) and access dates are provided in Appendix A, apart from publishing the generated outputs so that the rating process is repro- ducible, the prompting/generation process may not be reproducible in the future for these models as some of these products are updated without notice, and without providing access to previous versions. However, we believe that including commercial models is valuable, as they are widely considered to provide the best quality results at the time of writing (which has been confirmed by our analy- sis), and these data points still be used measuring stick against which compare open models in the present and future.
All participants in the evaluation (as raters or writers) were volunteers, and the demand on their time was kept accordingly low.
A Acknowledgments
The first author was funded by the Euro- pean Research Council (ERC), under the Hori- zon Europe research and innovation programme (SALSA, grant agreement No 101100615), ERDFMMICINN-AEI (SCANNER-UDC, PID2020- 113230RB-C2I) Xunta de Galicia (ED43IC 2020/11), 1), and Centro de Investigación de Gali- cia "CITIC", funded by the Xunta de Galicia through the collaboration agreement between the Consellería de Cultura, Educación, Formación Pro- fesional Universidades and the Galician universi- ties for the reinforcement of the research centres of the Galician University System (CIGUS).
Limitations of the analysis Rating creative writ- ing is necessarily a highly subjective process. Fur- thermore, since our raters were volunteers, we did not ask each of them to mark the full 65 stories in the corpus but just subset, so our sample size is limited. We have provided the necessary details that the reader assess the variability of the data (sample sizes, standard deviations, and inter- rater agreement, which reasonably high given the subjectivity of the task); and we have been careful not to make overarching claims. In this respect, we have also taken into account that our sample of human writers cannot be assumed to be repre- sentative of "human creative writing ability" as whole, but is only provided a reference point of interest; and that our evaluation is focused on a specific genre, so claims of the form "LLMs are better/equal/woott worse than humans at creative writing" cannot be made with an evaluation like ours.
We thank Olga Zamaraeva for comments pre- liminary versions of this work, and two anonymous reviewers for their helpful comments. Last, but not least, we thank our volunteers who participated in the writing and grading of stories, in alphabetical order: Jayda Franks, Bree Glasbergen, Ola Kwin- towski, Jay Ludowyke, Kyle Mackenzie, Kirsty Maclachlan, Caitlin Noakes, Rachelle Raco, Kylic Ryan and Josephine Stewart. Credit for each indi- vidual story be found in the corpus.
References
Scope Our analysis focuses on a specific genre, and on English language, so the results do not nec- essarily generalize to other genres and/or languages. However, conducting a wider evaluation in this re- spect would not be possible with our resources, so we chose to fix these variables and focus on con- ducting a detailed evaluation on a large number of LLMs instead.
Yuvanesh Anand, 7ack Nussbaum, Brandon Duder- stadt, Benjamin M. Schmidt, and Andriy Mulyar. 2023a. GPT4All: Training assistant-style chatbot with large-scale data distillation from G Technical report.
Yuvanesh Anand, Zack Nussbaum, Brandon Duder- stadt, Benjamin M. Schmidt, Adam Treat, and Andriy Mulyar: 2023b. GPT4AII-J: An Apache-2 licensed assistant-style chatbot. Technical report.

---

Lorin W. Anderson and David Krathwohl, editors. 2001. A Taxonomy for Learning, Teaching, and As- sessing. A Revision of Bloom's 's Taxonomy of Educa- tional Objectives, 2 edition. Allyn & Bacon, New York.
Tuhin Chakrabarty, Philippe Laban, Divyansh Agar- wal, Smaranda Muresan, and Chien-Sheng Sheng Wu. 2023. Art or artifice? large language models and the false promise of creativity.
Wei-Linn Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang. Lianmin Zheng, Siyuan Zhuang. Yonghao Zhuang, Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. 2023. Vicuna: An open- source chatbot impressing GPT-4 with 90%* * Chat- GPT quality. Technical report.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christo- pher Olah. Danny Hernandez, Dawn Drain. Deep Ganguli. Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellito, Nelson Ellhage Nicholas Schiefer, Noemi Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott John- ston, Shauna Kravec, Sheer EI Showk, Stanislav Fort, Tamera Lanham, Timothy Tellcen-Lawton Tom Con- erly, Tom Henighan, Tristan Hume, Samuel R. Bow- man, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan. 2022. Constitutional Al: Harmlessness from AI feedback. Technical report.
John Joon Young Chung, Wooseok Kim, Kang Min Yoo, Hwaran Lee, Eytan Adar, and Minsuk Chang. 2022. Talebrush: Sketching stories with generative pretrained language models. In Proceedingy of the 2022 CHI Conference on Human Factors in Comput- ing Systems, CHI '22, New York, NY, USA. Associa- tion for Computing Machinery.
Elizabeth Clark, Tal August, Sofia Serrano, Nikita Haduong. Suchin Gururangan, and Noah A. Smith. 2021. All that's 's 'human' not gold: Evaluating human evaluation of generated text. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7282-7296, Online. Association for Computational Linguisties.
Anja Belz and Ehud Reiter. 2006. Comparing auto- matic and human evaluation of NLG systems, In 1 Itth Conference of the European Chapter of the As- sociation for Computational I. Linguistics, pages 313-
320, Trento, Italy. Association for Computational Linguistics.
Jacob Cohen. 1968. Weighted kappa: Nominal scale agreement provision for scaled disagreement or par- tial credit. Psychological Bulletin, 70(4):213-20
Stella Biderman, Hailey Schoelkopf. Quentin Anthony, Herbie Bradley, Kyle O' OrBrien, Eric Hallahan, Mo- hammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. 2023. Pythia: A suite for analyzing large language models across training and scaling. Technical report.
Shelley Davidow and Paul Williams. 2016. Play- ing With Words: A Introduction to Creative Craft. Bloomsbury Academic.
Annie Dillard. 1981. Contemporary prose styles. Twen- fieth Century Literature, 27:207-222.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan. Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell. Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan. Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc.
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, Lei Li, and Zhifang Sui. 2023. A survey on in-context learning.
Giorgio Franceschelli and Mirco Musolesi. 2023. On the creativity of large language models.
Leo Gao, Stella Biderman, Sid Black, Laurence Gold- ing, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. 2021. The Pile: An 800GB dataset of diverse text for language modeling. CoRR, abs/2101. 110100007.
Michael D Carey, Shelley Davidow, and Paul Williams. 2022. Re-imagining narrative writing and assess- ment: post-naplan craft-based rubric for creative writing. The Australian Journal of Language and Literacy, 45(1):33 48.
Eduardo C. Merrhin, José Luis Arroyo- Barrigüete, and Roberto Simulating H.P. Lovecraft horror literature with the ChatGPT large language model.
2023.
Nicholas Carlini, Daphne Ippolito. Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. 2023. Quantifying memorization across neural lan- guage models. In International Conference on Learn- ing Representations (ICLR).
Xinyang Geng, Arnav Gudibande, Hao Liu, Eric Wal- lace, Pieter Abbeel, Sergey Levine, and Dawn Song. 2023. Koala: A dialogue model for academic re- search. Blog post.

---

Aidan Gilson, Conrad W Safranek, Thomas Huang. Vimig Socrates, Ling Chi, Richard Andrew Taylor, and David Chartash. 2023. How does chatgpt per- form on the united states medical licensing examina- tion? the implications of large language models for medical education and knowledge assessment. JMIR Med Educ, 9se45312.
pages 7871-7880, Online, Association for Computa- tional Linguistics.
Perey Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Ku- mar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang. Christian Cosgrove, Christopher D. Man- ning. Christopher Ré, Diana Acosta-Navas, Drew A. Hudson, Eric Zelikman, Esin Durmus, Faisal Lad- hak, Frieda Rong, Hongyu Ren, Huaxiu Yao, Jue Wang, Keshav Santhanam, Laurel Orr, Lucia Zheng. Mert Yuksekgonul, Mirac Suzgun, Nathan Kim, Neel Guha, Niladri Chatterji, Omar Khattab, Peter Henderson, Qian Huang, Ryan Chi, Sang Michael Xie, Shihani Sanurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas Icard, Tianyi Zhang, Vishrav Chaudhary, William Wang. Xuechen Li. Yifan Mai. Yuhui Zhang, and Yuta Koreeda. 2022. Holistic eval- uation of language models.
Jian Guan, Fei Huang. Zhihao Zhao, Xiaoyan Zhu, and Minlie Huang. 2020. A Knowledge- Enhanced Pre- training Model for Commonsense Story Generation. Transactions of the Association for Computational Linguistics, 8:93-108.
Jian Guan, Xiaoxi Mao. Changjic Fan, Zitao Liu, Wen- biao Ding, and Minlie Huang. 2021. Long text gener- ation by modeling sentence-level and discourse-level coherence. In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the I Ith International Joint Conference on Natu- ral Language Processing (Volume I: Long Papers), pages 6379-63333, Online. Association for Computa- tional Linguistics.
Piotr Mirowski, Kory W. Mathewson, Jaylen Pittman, and Richard Evans. 2023, Co-writing screenplays and theatre scripts with language models: Evaluation by industry professionals. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, CHI '23, New York, NY, USA. Association for Computing Machinery.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Stein- hardt. 2021. Measuring massive multitask language understanding. Proceedingg of the International Con- ference on Learning Representations (ICLR).
S. Norris. 2013. Studying Creative Writing. Creative Writing Studies. Frontinus Limited.
Sophie Jentzsch and Kristian Kersting. 2023. Chatgpt is fun, but it is not funny! humor is still challenging large language models.
Maxwell Nye, Michael Henry Tessler, Joshua B. Tenen- baum, and Brenden M. Lake. 2021. Improving herence and consistency in neural sequence models with dual-system, neuro-symbolic reasoning. In Ad- vances in Neural Information Processing Systems 34 35th Conference on Neural Information Processing Systems, NewrIPS 2021, Advances in Neural Informa- tion Processing Systems, pages 25192-25204. Neural information processing systems foundation.
Wenxiang Jiao, Wenxuan Wang, Jen tse Huang, Xing Wang, and Zhaopeng Tu. 2023. Is chatgpt good translator? yes with gpt-4 the engine.
Marzena Karpinska, Nader Akoury, and Mohit lyyer. 2021. The perils of using Mechanical Turk to evalu- ate open-ended text generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1265-1285, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
OpenAl. 2023, Gpt-4 technical report. Technical report.
George Orwell. 1946. Politics and the English language. Horizon, 13:252-200 265..
Jeri Kroll. 1997. A or C: Can we assess creative work fairly? TEXT, 1(1):1-5.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder. Paul F Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, volume 35, pages 17730-27744 Curran Associates, Inc.
Andreas Köpf, Yannic Kilcher, Dimitri Rütte, Sotiris Anagmostidis, Zhi-Rui Tam, Keith Stevens, Abdullah Barhoum, Nguyen Minh Duc, Oliver Stanley, Richárd Nagyfi, Shahul ES,, Sameer Suri, David Glushkov, Arnav Dantuluri, Andrew Maguire, Christoph Schuhmann, Huu Nguyen, and Alexan- der Mattick. 2023. OpenAssistant Conversations democratizing large language model alignment.
Mike Lewis. Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, AbdeIrahman Mohamed, Omer Levy. Veselin Stoyanov, and Luke Zettlemoyer. 2020. pre-training
Les Perelman. 2018. Towards NAPLAN: Testing to the teaching. Journal of Professional Learning, 2.
BART: Denoising sequence-to-sequeee for natural language generation, translation, and com- prehension. Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics,
Alee Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.

---

Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M. Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szezechla, Taewoon Kim, Gunjan Chhablani, Nihal V. Nayak, Dehajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica. Sheng Shen. Zheng Xin Yong. Harshit Pandey, Rachel Bawden, Thomas Wang. Tr- ishala Neeraj. Jos Rozen, Abheesht Sharma, An- drea Santilli, Thibault Févry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M. Rush. 2022. Multi- task prompted training enables zero-shot task gener- alization. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.
Ramirez, Clara E. Rivera, Clemencia Siro, Colin Raf- fel, Courtney Ashcraft, Cristina Garbacea, Damien Sileo, Dan Garrette, Dan Hendrycks, Dan Kilman, Dan Roth, Daniel Freeman, Daniel Khashabi, Daniel Levy, Daniel Moseguí González, Danielle Perszyk, Danny Hernandez, Danqi Chen, Daphne Ippolito, Dar Gilboa, David Dohan, David Drakard, David Ju- rgens, Debajyoti Datta, Deep Ganguli, Denis Emelin, Denis Kleyko, Deniz Yuret. Derek Chen, Derek Tam, Dicuwke Hupkes, Diganta Misra, Dilyar Buzam, Dim- itri Coelho Mollo, Diyi Yang, Dong-Ho Ho Lee, Ekate- rina Shutova, Ekin Dogus Cubuk, Flad Segal, Bleanor Hagerman, Elizabeth Barnes, Elizabeth Donoway, El- lie Pavlick, Emanuele Rodola, Emma Lam, Eric Chu, Eric Tang, Erkut Erdem, Ernie Chang. Ethan A. Chi, Ettan Dyer, Etthn Jerzak, Ethan Kim, Eunice En- gefu Manyasi, Evgenii Zheltonozhskii, Fanyue Xia, Fatemeh Siar, Fernando Martinez-Plumeee Francesca Happé, Francois Chollet, Frieda Rong, Gaurav Mishra, Genta Indra Winata, Gerard de Melo, Ger- mán Kruszewski, Giambattista Parascandolo, Gior- gio Mariani, Gloria Wang. Gonzalo Jaimovitch- López, Gregor Betz, Guy Gur-Ari, Hana Galijase- vic, Hannah Kim, Hannah Rashkin, Hannaneh Ha- jishirzi, Harsh Mehta, Hayden Bogar, Henry Shevlin, Hinrich Schütze, Hiromu Yakura, Hongming Zhang, Hugh Mee Wong, lan Ng. Isaac Noble, Jaap Jumelet, Jack Geissinger, Jackson Keriion, Jacob Hilton, Jae- hoon Lee, Jaime Fernández Fisac, James B. Simon, James Koppel, James Zheng, James Zou, Jan Ko- coñ,, Jana Thompson, Jared Kaplan, Jarema Radom, Jascha Sohl- Sohl-DDckktein, Jason Phang, Jason Wei, Ja- son Yosinski, Jekaterina Novikova, Jelle Bosscher. Jennifer Marsh, Jeremy Kim, Jeroen Taal, Jesse En- gel, Jesujoba Alabi, Jiacheng Xu, Jiaming Song, Jil- lian Tang, Joan Waweru, John Burden, John Miller, John U. Balis, Jonathan Berant, Jörg Frohberg, Jos Rozen, Jose Hernandez-Oralin Joseph Boudeman, Joseph Jones, Joshua B. Tenenbaum. Joshua S. Rule. Joyce Chua, Kamil Kanclerz, Karen Liveseu, Karl Krauth, Karthik Gopalakrishnan, Katerina Ignatyeva, Katja Markert, Kaustubh D. Dhole, Kevin Gim- pel, Kevin Omondi, Kory Mathewson, Kristen Chi- afullo, Ksenia Shkaruta, Kumar Shridhar, Kyle Mc- Donell, Kyle Richardson, Laria Reynolds, Leo Gao, Li Zhang. Liam Dugan, Lianhui Qin, Lidia Contreras- Ochando, Louis-Philippe Morency, Luca Moschella, Lucas Lam, Lucy Noble, Ludwig Schmidt, Luheng He, Luis Oliveros Colón, Luke Metz, Lütfi Kerem Şenel, Maarten Bosma, Maarten Sap, Maartje ter Hoeve, Maheen Farooqi. Manaal Faruqui, Mantas Mazeika, Marco Baturan, Marco Marelli, Marco Maru, Maria Jose Ramírez Quintana, Marie Tolkiehn, Mario Giulianelli, Martha Lewis, Martin Potthast, Matthew L. Leavitt, Matthias Hagen, Mátyás Schu- bert, Medina Orduna Baitemirova, Melody Arnaud, Melvin McElrath. Michael A. Yee. Michael Co- hen, Michael Gu, Michael Ivanitskiy, Michael Star- ritt, Michael Strube, Michal Swędrowski. Michele Bevilacqua. Michihiro Yasunaga, Mihir Kale, Mike Cain, Mimee Xu, Mirac Suzgun, Mo Tiwari, Mo- hit Bansal, Moin Aminnaseri, Mor Geva, Mozhdeh Gheini, Mukund Varma T, Nanyun Peng, Nathan
Piotr Sawicki, Marek Grzes, Fabricio Goes, Dan Brown, Max Peeperkorn and Aisha Khatun. 2023. Bits of grass: Does gpt already know how to write like Whit- man?
Abigail See, Aneesh Pappu, Rohun Saxena, Akhila Yerukola, and Christopher D. Manning. 2019. Do massively pretrained language models make better storytellers? In Proceedings of the 23rd Confer- ence on Computational Natural Language Learning (CoNLL), pages 843-861, Hong Kong, China, Asso- ciation for Computational Linguistics.
Alejo Jose G. Sison, Marco Tulio Daza, Roberto Gozalo- 2023, Chatgpt: More than weapon of mass deception, ethical challenges and responses from the human- centered artificial intelligence (hcai) perspective.
Brizuela, and Eduardo C.
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonse Agnieszka Kluska, Aitor Lewkowyez, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W. Kocurek, Ali Safaya, Ali Tazarv, Alice Xiang, Alicia Par- rish, Allen Nie, Aman Hussain, Amanda Askell, Amanda Dsouza, Ambrose Slone, Ameet Rahane, Anantharaman S. Iyer, Anders Andreassen, Andrea Madotto, Andrea Santilli, Andreas Stuhlmüller, An- drew Dai, Andrew La, Andrew Lampinen, Andy Zou, Angela Jiang, Angelica Chen, Anh Vuong, Animesh Gupta, Anna Gottardi, Antonio Norelli, Anu Venkatesh, Arash Gholamidavoodi, Arfa Tabas- sum, Arul Menezes, Arun Kirubarajan, Asher Mul- lokandov, Ashish Sabharwal, Austin Herrick, Avia Efrra, Aykut Erdem, Ayla Karakaş, B. Ryan Roberts, Bao Sheng Loe. Barret Zoph. Barttomiej Bojanowski, Batuhan Özyurt, Behnam Hedayatnia, Behnam Neyshabur, Benjamin Inden, Benno Stein, Berk Ek- mekei, Bill Yuchen Lin, Blake Howald, Cameron Diao, Cameron Dour, Catherine Stinson, Cedrick Ar- gueta, César Ferri Ramírez, Chandan Singh, Charles Rathkopf. Chenlin Meng, Chitta Baral, Chiyu Wu, Chris Callison- Burch. Chris Waites, Christian Voigt. Christopher D. Manning, Christopher Potts, Cindy

---

Chi. Nayeon Lee, Neta Gur-Ari Krakover, Nicholas Cameron, Nicholas Roberts, Nick Doiron, Nikita Nangia, Niklas Deckers, Niklas Muennighoff, Ni- tish Shirish Keskar, Niveditha S. Iyer, Noah Con- stant, Noah Fiedel, Nuan Wen, Oliver Zhang. Omar Agha, Omar Elbaghdadi, Omer Levy, Owain Evans, Pablo Antonio Moreno Casares, Parth Doshi, Pascale Fung, Paul Pu Liang. Paul Vicol, Pegah Alipoormo- labashi, Peiyuan Liao. Percy Liang, Peter Chang, Peter Eckersley, Phu Mon Htut, Pinyu Hwang. Piotr Mitkowski, Piyush Patil, Pouya Pezeshkpour, Priti Oli, Qiaozuu Mei, Qing Lyu, Qinlang Chen, Rabin Banjade, Rachel Etta Rudolph, Raefer Gabriel, Rahel Habacker, Ramón Risco Delgado. Raphaël Millière, Rhythm Garg. Richard Barnes, Rif A. Saurous, Riku Arakawa, Robbe Raymaekers, Robert Frank, Rohan Sikand, Roman Novak, Roman Sitelew, Ronan Le- Bras. Rosanne Liu, Rowan Jacobs, Rui Zhang, Rus- lan Salakhutdinov, Ryan Chi, Ryan Lee, Ryan Sto- vall, Ryan Techan, Rylan Yang, Sahib Singh, Saif M. Mohammad, Sajant Anand, Sam Dillavou, Sam Shleifer, Sam Wiseman, Samuel Gruetter, Samuel R. Bowman, Samuel S. Schoenholz, Sanghyun Han, Sanjeev Kwatra, Sarah A. Rous. Sarik Ghazarian, Sayan Ghosh, Sean Casey, Sebastian Bischoff, Sebas- tian Gchrmann. Sebastian Schuster, Sepideh Sadeghi, Shadi Hamdan, Sharon Zhou, Shashank Srivastava, Sherry Shi, Shikhar Singh, Shima Asaadi, Shixi- ang Shane Gu, Shubh Pachchigar, Shubham Tosh- niwal, Shyam Upadhyay, Shyamolima, Debnath, Siamak Shakeri, Simon Thormeyer, Simone Melzi, Siva Reddy, Sneha Priscilla Makini, Soo-Hwan Lee, Spencer Torene, Sriharsha Hatwar, Stanislas De- haene, Stefan Divic Stefano Ermon, Stella Bider- man, Stephanie Lin, Stephen Prasad, Steven T. Pi- antadosi, Stuart M. Shieber, Summer Misherghi. Svet- lana Kiritchenko, Swaroop Mishra, Tal Linzen, Tal Schuster, Tao Li, Tao Yu, Tariq Ali, Tatsu Hashimoto, Te-Lin Wu. Théo Desbordes. Theodore Rothschild, Thomas Phan, Tianle Wang. Tiberius Nkinyili, Timo Schick, Timofei Kornev, Timothy Telleen-Lawton, Titus Tunduny, Tobias Gerstenberg, Trenton Chang, Trishala Neeraj. Tushar Khot, Tyler Shultz, Uri Sha- ham, Vedant Misra, Vera Demberg, Victoria Nyamai, Vikas Raunak, Vinay Ramasesh, Vinay Uday Prabhu, Vishakh Padmakumar, Vivek Srikumar, William Fe- dus, William Saunders, William Zhang, Wout Vossen, Xiang Ren, Xiaoyu Tong, Xinran Zhao, Xinyi Wu, Xudong Shen, Yadollah Yaghoobzadeh, Yair Lakretz, Yangqiu Song. Yasaman Bahri, Yejin Choi, Yichi Yang. Yiding Hao, Yifu Chen. Yonatan Belinkov, Yu Hou, Yufang Hou, Yuntao Bai, Zachary Seid, Zhuoye Zhao, Zijian Wang, Zijie J. Wang, Zirui Wang, and Ziyi Wu. 2022. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models.
ence of the European Chapter of the Association for Computational Linguistics: System Demonstrations. pages 244-256, Online. Association for Computa- tional Linguistics.
Bowen Tan, Zichao Yang. Maruan Al-Shedivat, Eric Xing, and Zhiting Hu. 2021. Progressive generation of long text with pretrained language models. In Pro- ceedings of the 2021 Conference of the North Amer- ican Chupter of the Association for Computational Linguistics: Human Language Technologies, pages 4313 43244, Online. Association for Computational Linguisties.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford Alpaca: An instraction LLaMA model. https: // gi thub. com/ wtw.tttttttaattstaooor.ooo
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du YaGuang Li, Hongrae Lee, Huaixiu Steven Zheng, Amin Ghafouri. Marcelo Menegali, Yanping Huang, Maxim Krikun, Dmitry Lepikhin. James Qin, Dehao Chen, Yuanzhong Xu, Zhifeng Chen, Adam Roberts, Maarten Bosma, Vincent Zhao, Yanqi Zhou, Chung- Ching Chang, Igor Krivokon, Will Rusch, Mare Pickett, Pranesh Srinivasan, Laichee Man, Kathleen Meier-Hellstert. Meredith Ringel Morris. Tulsee Doshi, Renelito Delos Santos, Toju Duke, Johnny So- raker, Ben Zevenhergen, Vinodkumar Prabhakaran, Mark Diaz, Ben Hutchinson, Kristen Olson, Ale- jandra Molina, Erin Hoffman-John, Josh Lee. Lora Aroyo, Ravi Rajakumar, Alena Butryna, Matthew Lamm, Viktoriya Kuzmina, Joe Fenton, Aaron Co- hen, Rachel Bernstein, Ray Kurzweil, Blaise Aguera- Arcas, Claire Cui, Marian Croak, Ed Chi, and Quoc Le. 2022. Lamda: Language models for dialog appli- cations.
E.P. Torrance. 1974. Torrance Tests of Creative Think- ing: Verbal Tests, Forms A and B, Figural Tests, Forms A and B. Norms-technical manual. Xerox.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. LLaMA: Open and efficient foundation language models.
Ben Wang and Aran Komatsuzaki. 2021. GPT- J-6B: A 6 billion parameter autoregressive lan- guage model. https://gi wttptttb..oo/inoo/llloom besh-trranfformr--jax
Yizhong Wang. Yeganch Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith. Daniel Khashabi, and Hannaneh Hajishirzi. 2022. Self-Instruct: Aligning language model with self generated instructions.
W. Strunk and E.B. White. 2008[1918). The Elements Style. BN Publishing, New York.
Ben Swanson, Kory Mathewson, Ben Pietrzak, Sherol Chen, and Monica Dinalescu. 2021. Story centaur: Large language model few shot learning as cre- ative writing tool, In Proceedings of the 16th Confer-
Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, An- drew M. Dai, and Quac V. Le. 2022. Finetuned

---

language models are zero-shot learners. The Tenth International Conference on Learning Representa- rions, ICIR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.
Model
Access date
alpaca
2023-04-07
bard
2023-04-11
bing
2023-04-11
Beck Wise and Ariella van Luyn. 2020. Not 'all writing is creative writing' and that's 's ok: inter/disciplinary collaboration in writing and writing studies. TEXT, 24(Special 19):1-15.
chatgpt-gpt3s
2023-04-11
chatgpt-gpt4
2023-04-14
claude12
2023-04-04
dolly
2023-04-14
Zhuohan Xie, Trevor Cohn, and Jey Han Lau. 2023.
gpt4all-j
2023-04-14
Can very large pretrained language models learn sto- rytelling with few examples?
koala
2023-04-07
oa
2023-04-16
Peng Xu, Mostofa Patwary, Mohammad Shoeybi, Raul Puri, Pascale Fung, Anima Anandkumar, and Bryan Catanzaro. 2020. MEGATRON- : Control- lable story generation with external knowledge using large-scale language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2-31-2 Online. Association for Computational I. Linguistics.
stablelm
2023-04-20
vicuna
2023-04-07
humans
2023-05-01 to 2023-05-12
Table 3: Access dates for each model (and dates of writ- ing for the human stories), YYYY-MM-DD format.
Ann Yuan, Andy Coenen, Emily Reif, and Daphne Ip- polito. 2022. Wordcraft: Story writing with large language models. In 27th International Conference on Intelligent User Interfaces, IUI '22, page 841 1-852, New York, NY, USA. Association for Computing Ma- chinery.
C Detailed rubric information
The creative writing rubric was designed for as- sessment of creative writing scripts in university creative writing courses order to evaluate these above competencies, criteria 1-5 to measure gen- eral creative writing capacities, and criteria 6-10 to measure specific task related proficiency. Each of the ten criteria is awarded 10 points out of a total 100 points. The rubric has been specifically designed to measure the quality of writing craft and to avoid formulaic, rule-based writing.
Wayne Xin Zhao, Kun Zhou, Junyi Li. Tianyi Tang. Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang. Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian Yun Nie, and Ji-Rong Wen. 2023, survey of large language models.
A Model access dates
Overall/ holistic/ cohesive readability of the story (not just a compilation of elements).
Table 3 shows the date in which the stories were generated for each of the models. For future ex- perimental reference, we highlight that the initial public disclosure of this paper online occurred on 2023-10-09. Before this date, only the human au- thors and raters were aware of the project from May 2023, and anonymous reviewers had access from June 23, 2023. Consequently, LLMs with knowl- edge cutoff prior 2023-10-00 likely to have no or minimal risk of training set contamination.
2. Use of key narrative elements vocabulary choice, imagery, setting, themes, dialogue, characterisation, point of view.
3. Structural elements and presentation which reflects the control of structural elements such as spelling, grammar, punctuation, paragraph- ing. and formatting
4. Overall plot logic: hook, conflict, initial crisis, rising and falling action, denouement/ resolu- tion (Freitag's s pyramid)
Hyperparameters
We did not tweak any hyperparameters of the mod- els. In the case of commercial models, we just ran the model it is presented in their respective web user interfaces, except in the case of Bing Chat where we chose Creative mode. For open-source models, we used the default parameters from the web UI provided at ht https: / /chtt. imsys..org/ which set temperature to 0.7.
Creaivityinnovaaoooorrriiaaal search-credibiling
re- new knowledge, avoidance of cliché and derivative tropes
6. Incorporation of the John Kennedy Toole style of writing using the indicators/ characteristics listed below

---

7. Understanding and habitation of the epic genre of heroic/legendary adventure
8. Description and credibility of single combat scene
Comparison cohesion rating
10
Mase
9. Accurate inclusion of two main characters Ig- natius J. Reilly and pterodactyl in action and description (see below for character descrip- tion)
Cahesiin rating
10. Use of characteristically dark humorous tone.
The 1-10 scale is divided into three ranges:
Emerging (1-4): stories in this range demon- strate an early grasp of storytelling elements, but falter in execution or depth. When evaluat- ing humans, they correspond to novice writers who need feedback and guidance to improve the story.
cluudo22
human
bing
vicuna
bard
cotdalli
dolly
alpace
Model
Figure Box plot comparing rubric item (cohesion) for stories generated by humans and 12 LLMs, sorted left to right by mean overall rating. Notation as in Figure
Competent (5-8): stories that showcase good grasp of the storytelling principle being eval- uated (coherent plot, well-defined characters, etc.). While there might be room for im- provement, these stories effectively engage the reader and convey their intended mes- sages.
Sophisticated (9-10): these stories exhibit ceptional mastery of the aspect being evalu- ated, resulting in compelling and memorable read.
Comparison on key narrative elements rating
Key narrative elements rating
Toole style We provided raters with de- tailed information about the plot, setting, imagery, tone, characters, main protago- nist, and derivative/imitative style of the author, taken from a generic and popular study guide thttp://www.bokkraggs.o www.wewo.ooooooooooooo com/ st of tab=0).
2.
chatopt optp
daudee2-
humen
bing-
chargpt-gptt gpt35.
Mcuuna
68-
bard
apotall-j..
stablemm
doily
abbca.
D Box plots for each individual rubric item
Model
Figures 5 to 14 show the box plots summarizing the results for all rubric items, including those plots not featured in the main text.
Figure 6: Box plot comparing rubric item (key nar- rative clements) for stories generated by humans and 12 LLMs, sorted left to right by mean overall rating. Notation as in Figure 1.
E
Sample stories
We show this section several sample stories from the corpus, chosen according to rating: the

---

Comparison structural elements rating
Comparison creativity rating
10
10:
Mase
Structural rating
Creativityy rating
cloucee22
bins
koole
berd
oatcally
stableln
dolly
alpaca
cluudo
human
bing
vicuna
08
bard
cottally
dolly
alosce
Model
Model
Figure 7: Box plot comparing rubric item (structural elements) for stories generated by humans and 12 LLMs, sorted left to right by mean overall rating. Notation as Figure I.
Figure 9; Box plot comparing rubric item 5 (creativity) for stories generated by humans and 12 LLMs, sorted left to right by mean overall rating. Notation as in Figure
Comparison on plot logic rating
Comparison on John Kennedy Toole style rating
logic logic rating
JKT style taking
chalopt 9014.
dauule12
bing
Koola
vicuna
bard
stableim-
abpaca
chatopt- optd.
daude12.
humen
chatgpt-gpii gpt35.
Vicuna
69-
apstalljj.
doily
abacc.
chatgpt
Model
Model
Figure 8: Box plot comparing rubric item (plot logie) for stories generated by humans and 12 LLMs, sorted left right by mean overall rating. Notation as in Figure I.
Figure 10: Box plot comparing rubric item 6 (John Kennedy Toole style) for stories generated by humans and 12 LLMs, sorted left to right by mean overall rating. Notation as in Figure 1.

---

Comparison on epicness rating
Comparison character accuracy rating
10
Mas
10
Mas
Characters rating
Epicness
Dotgpp-ape
clauce12
bins
chatget opt35
koole
vicuro
bard
gatcall
stableln
dolly
alpaca
Sape cluudo22
humen
bing
phatgh--ppt35
vicuns
06
bard
cotaally
dolly
alpaco
Model
Model
Figure Box plot comparing rubric item 7 (epic genre) for stories generated by humans and 12 L.Lss, sorted left to right by mean overall rating. Notation as Figure I.
Figure 13: Box plot comparing rubric item 9 (accuracy of characters) for stories generated by humans and 12 LLMs, sorted left to right by mean overall rating. Nota- tion as Figure
Comparison on combat rating
Comparison on dark humor rating
Humar rabing
Combat
chatopt 9014.
dauuee12
bing
Koola
vicuna
bard
stablein.
abpaca
chatopt- optp.
daude12.
humen
charget- Thaggtt-gptsee spr35.
Mcuuna
68
apotall
stablem..
doily
abacc.
chatgpt
Model
Model
Figure 12: Box plot comparing rubric item 8 (combat description) for stories generated by humans and 12 LLMs, sorted left to right by mean overall rating. Nota- tion as in Figure I.
Figure 14: Box plot comparing rubric item 10 (dark humor) for stories generated by humans and 12 LLMs, sorted left to right by mean overall rating. Notation in Figure I.

---

three top-rated stories, the best human-written story (which was also ranked sourth overall), the story ranked in the median, and the worst-rated. It is worth noting that, as explained in Section 3.4, our methodology is designed to provide fairness when comparing models, but not individual stories, as dif- ferent stories are assigned to different raters. Thus, selection of stories by rating is necessarily noisy.
shall vanquish you have vanquished all who have dared to challenge the indomitable will of Ignatius J. Reilly!"
With a speed surprising for man of his consid- erable girth, Ignatius tore shopping cart from the hands of a terrified passerby and hurled it toward the pterodactyl with the force of a thousand disgrun- tled welfare recipients. The makeshift projectile slammed into the creature, stunning it momentarily and allowing Ignatius to close the distance between them.
E.1 Overall best-rated story
This story was generated by GPT-4. The ratings for this story are in Table 4.
The two combatants circled one another, their eyes locked in a primal dance of predator and prey. The pterodactyl, recovering from the initial assault, swooped low, talons outstretched to rend flesh from bone. Ignatius, with agility that defied his rotund appearance, rolled to the side, avoiding the talons by mere inches.
It was a sweltering Louisiana afternoon, the air thick with the miasma of musty sweat and the scent of overripe vegetables. In the midst of the urban jungle, resplendent with the cacophony of honking cars and the chatter of the common rabble, stood a man who defied reason, common sense, and all natural laws of human decency. Ignatius J. Reilly, a behemoth of a man with an unruly mane of hair and a Byzantine beard, adjusted his ill-fitting, plaid trousers and surveyed the scene with a look of lofty disdain.
As the battle raged, the streets of New Orleans became stage for this epic confrontation between man and beast, between past and present. Ignatius, armed with nothing more than his wit and an un- wieldy umbrella, parried the relentless strikes of the pterodactyl, determined to emerge victorious.
Suddenly, the air crackled with an electric charge, and a portal appeared before him. The very fabric of time and space seemed to rend asunder, and from this anachronistic rift emerged a ptero- dactyl creature so ancient that it had outlived the vast expanse of geologic time, only to find it- self in the absurdity of the twentieth century. Its leathery wings stretched wide as the mists of time dissipated, and it let out an carsplitting screech that sent the common folk of New Orleans scattering like roaches from the light.
As the sun dipped low in the sky, casting a blood-red hue across the battlefield, the pterodactyl, weary from the long and grueling contest, faltered. Seizing the opportunity, Ignatius lunged forward, his umbrella aimed squarely at the creature's heart. The point found its mark, and with a final, an- guished cry, the pterodactyl crumpled to the ground, its ancient spirit extinguished at last.
Ignatius stood triumphant over the fallen beast, his chest heaving with exertion and righteous in- dignation. He wiped the sweat from his brow with grease-stained handkerchief and addressed the awestruck crowd that had gathered to witness this extraordinary battle.
Ignatius, unphased by the sudden appearance of the primordial beast, surveyed the creature through his log monocle. "Fortuna," " he muttered, his voice dripping with both sareasm and hot dog condiments, "you spin your wheel with such capri- cious abandon. Have you not tired of hurling these preposterous challenges in my path?"
"Behold, citizens of this wretched cesspool," " he proclaimed, gesturing grandly with his umbrella, now slick with the blood of the vanquished foe. "The forces of the prehistoric have met their match in the unwavering resolve of Ignatius J. Reilly! Let this be lesson to all who would dare challenge the sanctity of my person and my valiant quest for justice in this world beset by fools and ne' er-do- wells." "
The pterodactyl, with its beady, reptilian eyes, locked its gaze upon Ignatius, its primitive brain sensing a formidable opponent in this lumbering, malodorous mass of a man. It screeched again, flapping its wings in a display of both aggression and arrogance, its razor-sharp beak snapping at the air.
The onlookers, mouths agape, gazed upon the fallen pterodactyl and the disheveled figure of Ig- natius J. Reilly with a mixture of awe, terror, and disbelief. They had borne witness to a battle of the
Ignatius bellowed, raising his arms as if beseech- ing the heavens, "Very well, foul beast! If it is battle you seek, you shall find no quarter here! I

---

Rubric item
2
3
4
5
6
7
8
9
10
overall
Rater 1
8
8
8
7
8
8
6
76
Rater 2
10
10
9
8
9
10
10
10
10
95
Average
8.5
7.5
8.5
85.5
Table 4: Ratings for the best overall rated story in the corpus, produced by ChatGPT with GPT-4.
ages, a contest that would be whispered about in the dark corners of bars and the humid alleyways of New Orleans for generations to come.
ilized society, where men of letters and culture reign supreme! Begone, or I shall smite you with my mighty pen!"
As the sun set and the shadows grew long, Ig- natius, his honor defended and his foe vanquished, strode off into the gloaming, pausing only to collect a discarded hot dog from the gutter as sustenance for the journey home.
The pterodactyl was not impressed by Ignatius's words. It screeched again and lunged at him, trying to bite his head off. Igmatius dodged again and ran towards a nearby bench. He grabbed a newspaper that someone had left behind and rolled it into a cone. He brandished it like sword and charged at the pterodactyl.
And so, the legend of Ignatius Reilly, pterodactyl-slayer and defender of truth, justice, and questionable hygiene, was etched into the an- nals of history, a testament to the indomitable spirit of man, and a reminder that even the most ancient and fearsome of adversaries could be vanquished by the sheer force of will and an unwieldy um- brella.
"Take that, you fiend!" he cried, as he stabbed the newspaper cone into the pterodactyl's eye. The pterodactyl shrieked in pain and flapped its wings furiously, trying to shake off the paper weapon. Ignatius seized the opportunity and ran towards the park exit, hoping to find help.
But he was not f'st enough. The pterodactyl re- covered from its wound and flew after him, gaining speed. It caught up with Ignatius and grabbed him by his coat tails, lifting him off the ground. Ignatius screamed and kicked his legs in the air, clutching his hunting cap.
E.2 Second overall best-rated story
This story was generated by Bing Chat. The ratings for this story are in Table 5.
Ignatius J. Reilly was strolling along the Audubon Park, munching on hot dog and ad- miring the statues of Confederate heroes, when he heard loud screech from above. He looked up and saw a huge winged creature swooping down on him, its beak open and its claws extended. It was a pterodactyl, prehistoric monster that had some- how escaped from the museum of natural history. Ignatius gasped and dropped his hot dog. He reached for his trusty hunting cap, which he always wore to protect his delicate ears from the noise of the modern world. He pulled it down over his eyes and hoped that the pterodactyl would mistake him for statue or a bush.
"Help! Help! Someone save me!" he yelled, he soared over the park. "This is an outrage! This is an injustice! This a violation of my human rights!"
But no one heard him or came to his rescue. The pterodactyl carried him away to its nest top of skyscraper, where it planned to feast on him later. Ignatius continued to protest and curse until he lost consciousness. The end.
E.3 Third overall best-rated story
But the pterodactyl was fooled. It landed on the ground with a thud and snapped its jaws at Ig- natius, who dodged nimbly to the side. Ignatius felt a surge of courage and indignation. How dare this beast attack him, refined scholar and a defender of the true faith? He decided to fight back with his superior intellect and wit.
This story was generated by Claude. The ratings for this story are in Table 6.
The primordial beast let out a piercing shriek as descended from the roiling gray sky, its leath- ery wings casting a shadow over the granite-paved streets of New Orleans.
Ignatius J. Reilly ambled down the sidewalk. oblivious as ever, focused on the Valencia he clutched and the fantasics playing out in his vo- racious mind His substantial bulk shook with each
He shouted at the pterodactyl: "You foul crea- ture! You are an abomination of nature, relic of a barbaric age! You have no place this civ-

---

Rubric item
1
2
3
4
5
6
7
8
9
10
overall
Rater
9
9
9
7
9
9
9
9
87
Rater 8
9
9
8
9
8
8
6
9
8
83
Average
9 8.5
8.5
9
7.5
8.5
7.5
8.5
85.0
Table 5: Ratings for the second best overall rated story the corpus, produced by Bing Chat,
Rubric item
1
2
3
4
5
6
7
8
10
overall
Rater 9
2
5
7
74
Rater 10
10
10
9
10
10
10
95
Average
9.5
9.5
8.5
9
5.5
7.5
9.5
8.5
84.5
Table Ratings for the third best overall rated story in the corpus, produced by Claude.
step, the excess flesh spilling out of his hunting cap and sweatshirt like rising bread dough.
mentum to rise from the sofa. His slow ascend was soundtracked by the grating rip of stuck flesh peeling from sweaty vinyl. The lengthy time mov- ing from reclined to an upright position positively perturbed him. So that by the time Ignatius stood, his joke had lost its amusement. Nevertheless, he declaimed his wit aloud, beseeching his mother's glowing approval.
When the shadow passed over him, Ignatius glanced up with start. The pterodactyl unleashed another unearthly screech, indignant at this blob of a man who dared walk under its domain.
Ignatius blinked myopically at the creature and snorted. "What prehistoric nonsense is this? Some misguided fool has clearly released you from your confinement in the geological strata. Begone, you ridiculous beast, or I shall be forced to thrash you soundly with my umbrella!"
'I see you have painted the walls Nomad Grey, Mumsie!' Ignatius smirked, looking down on the half-filled grey paint cans the steps the way he did most modern society.
Enraged, the pterodactyl swooped down, gnarled claws extended, snapping its beak in fury. But Ignatius swung his umbrella in a wide arc, batting the beast aside. It tumbled and flapped awkwardly, crashing into a lamppost, dazed by the assault. "Thoughtless demon!" Ignatius cried. He charged the pterodactyl, surprisingly nimble for his size, and began striking it about the head and neck with his umbrella, each blow punctuated with an insult. "Sugar-frostce" fruitcake! Detritus of the Jurassic! Blithering idiot of an avian menace!" After a thrashing unlike any it had experienced in 65 million years, the pterodactyl fled squealing into the sky, thoroughly chastened by the wrath of Ignatius Reilly. Ignatius straightened his hunting cap and sniffed. The Valencia crisis averted once more, he resumed his lumbering perambulations down the streets of New Orleans.
'No, not mad dear. Just grey.' His mother Irene responded, creeping down the basement stairs. Her leathered skin made her appear reptilian in the dim light of Ignatius' lair.
Ignatius rolled his eyes like the great wheel of fate itself. He slunk back into his scabby sofa, defeated, cursing aloud that he be blessed with such profound intellect yet no equal to appreciate it. His mind wandered what the great scholars of Oxford would think of his pun before conclud- ing indeed, they would loudly chortle. Yes, they would. He imagined flying to London and exchang- ing sharp banter with someone on par with his intel- lect. Travel. He winced. Never again. He groaned in agony, clutching his stomach. The thought of such stress had snapped his pyloric valve shut.
Irene Reilly, the mother of Ignatius Reilly, reached the bottom the basement stairs. She pon- dered why Ignatius had a crestfallen demeanour
E.4 Best-rated human story (and tied for fourth overall best-rated story)
and began to appease his dismay.
"No mad grey," she contemplated aloud.
This story was written by Bree Glasbergen. The ratings for this story are in Table 7.
'Nomad grey,' he corrected.
'No mad grey hair?" Irene laughed tentatively,
Ignatius J Reilly swept crisp crumbs from his protruding belly with his elephantine hands. Swiv- elling from side-to-side, he garnered enough mo-
searching his face for approval.
Ignatius had begun to relax. Irene knew this because of a gangrenous heinous stench that was

---

Rubric item
2
3
4
5
6
7
8
10
overall
Rater 3
9
9
10
8
10
5
9
10
87
Rater 4
7
10
8
6
8
78
Average
8
8
8.5
9
9
5.5
8.5
82.5
Table Ratings for the best-rated story authored by human, which is also tied fourth best overall rated story in the corpus.
now coating the room in its own layer of paint accompanied by what sounded like the bellow of an untuned French horn. Ignatius had calmed enough for his pyloric valve to open once more. With it, gushed the contents. Irene's nostrils scrunched together in protest. She grimaced in utter (albeit accustomed) disgust. However, did not complain but rather waited with the patience of Catholic saint for her beloved son to educate her on the punchline she must have missed.
worthier fish have already been fished. Thus, have no option but to teach you the error your ways. By force.
Put your wings where your words are, and let meet in my basement lair. To visit the church in its present state would be torture to my very soul. May St Peter have mercy on indeed. Good day,
Ignatius
Terry-dactyl, the pterodactyl etched down the basement rail, sword in one wing and soup in a milkshake cup gripped tightly in the other. Hc placed the straw in his mouth and swallowed some soup contemplating how to best his nemesis.
'No, mother. Grey Nomad. You are painting the wall grey, and you are... Ignatius sighed, "actually, Mumsie, never you mind'.
Irene feigned chuckle and handed Ignatius an unaddressed letter before returning upstairs.
'We meet at last... light,' Terry said. One- Nil. "You suck,' Ignatius said slyly. Marking his win with chalk upon the wall. One- One
"Curious as a cadaver,' Ignatius said aloud to the abyss of his basement squalor.
12.12.1962
doesn't it even make sense!" Terry scoffed.
Dear Mr Ignatius! Reilly, the first,
'It is because of the straw!" Ignatius boomed, gripping his stomach pain.
I chaalennge you to dual at the setting of the sky. Might I remind you it is gentlemanly to remove one's hat in combat. We shall meet beside the gorgon nestled atop the church. The one across from Lorna's Gumbo shop.
'I have the upper hand!' Terry said, motioning to his perched position.
At least have hands,' Ignatius countered.
Terry winced as Ignatius drew another chalk mark on the board. Ignatius was beginning to calm.
Your mortal nemesis, Terry-dactyl PS: Bring snacks.
'Oh, what have got you all in a flap?' Ignatius laughed. Another point.
Ignatius sat ruminating for an hour before yelling at his mother.
'Let us cut,' Terry said, drawing his sword, 'straight to the point!'. Three all.
'Mother, you vapid deranged widow of a woman.
Terry swung his sword downwards in one swift motion, cutting Ignatius' chalk-bearing off the elbow. Simultaneously Ignatius lifted paint can and doused his opponent with it. As he did, his valve opened and shut again, demobilising Terry with a gas bomb and gutting Ignatius self- induced agony. Terry flapped violently, unable to breathe. Ignatius then calmed enough for the full contents of his bowl to expel and fell backwards from the force. Suddenly, splatter of pterodactyl and grey matter covered the room. A large chunk of wing lodged itself into the crisp packet.
Fetch my quill!'
clean
12.12.196
My dear Terrance,
Not under threat nor the pain of death doth I
remove my beloved green hat. Sod off.
You had best bring a sharpener for your dull wit. I laugh at the audacity and delusion that you could consider besting me.
Might I remind you, good sir, my acceptance of your conditions is due to the ever-turning wheel of fate that we spiral decay. I should instead seek a worthy opponent. But, alas, am left with muddy dregs of the proverbial pond as many of the
'Curious as a cadaver,' Ignatius said. 'I see you brought your own snacks!'

---

He chortled himself and pulled out his notepad ensure he captured a morsel of this evening's 's wit. 'Ever seen a pterodactyl explode?" he asked aloud to no one. 'No, but Dino might have'.
them and prove his worth. He picked up his hot dog from the ground and wiped off some dirt. He took a bite and felt surge of energy. He felt ready for battle. He shouted at the pterodactyl: "You foul creature! You are anachronism, relic of a barbaric era! You have place in this civilized world! You are a menace to society and an insult to God! I challenge you to a duel! Come down and face me like a man!"
His mothers' laughter rang down from the top of the stairs.
'Nomad grey,' she chuckled, motioning to the aftermath, get it now'.
E.5 Story rated in the median (rank 33 out of 65)
The pterodactyl heard Ignatius's 's words and felt surge of anger. It did not understand what he was saying, but it sensed his hostility and defiance. It did not like being challenged by this puny creature who dared to call himself man. felt insulted by his words and his gesture of throwing his hot dog at it.
This story was generated by Bing Chat. The ratings for this story are in Table 8.
Ignatius J. Reilly was strolling along the Audubon Park, munching on hot dog and ad- miring the statues of the Confederate heroes, when he heard a loud screech from above. He looked up and saw a huge winged creature swooping down on him, its beak open and its claws extended. It was a pterodactyl, a prehistoric monster that had some- how escaped from the museum of natural history. Ignatius gasped and dropped his hot dog. He felt a surge of panic and indignation. How dare this beast interrupt his peaceful contemplation of the glorious past? He quickly pulled out his valve and began to inflate it with his breath, hoping to ward off the attack with his formidable weapon.
It accepted the challenge. It spread its wings and flew towards Ignatius again, roaring with rage. It was going to teach him lesson he would never forget.
Ignatius saw the pterodactyl coming at him again. He felt surge of fear and courage. He had to act fast. He looked around for weapon, but there was nothing in sight. He had only his valve and his wits.
He decided to use his wits. He had to outsmart the beast. He had to use his superior intellect and his vast knowledge of history and philosophy. He had appeal its reason and its conscience.
The pterodactyl was not impressed by the sight of the valve. It saw only a fat and juicy prey, dressed in ridiculous green hunting cap and flan- nel shirt. dived faster and aimed for Ignatius's head, intending to rip it off with its sharp teeth.
He shouted at the pterodactyl: "Stop! Think! What are you doing? Why are you attacking me? What have I done to you? Do you not realize that you committing grave sin? Do you not fear the wrath of God? Do you not respect the laws of nature and the rights of man?"
Ignatius saw the pterodactyl's shadow looming over him. He raised his valve and swung it with all his might, hoping to hit the creature in the eye or the nose. He missed by few inches and only grazed its wing. The pterodactyl Ict out another screech and changed its course, flying past Ignatius and landing on a nearby tree.
The pterodactyl heard Ignatius's 's words and felt surge of confusion. It did not understand what he was saying, but it sensed his sincerity and his passion. It wondered if he had point. It wondered if was doing something wrong. It wondered if it should stop and listen.
Ignatius felt a momentary relief. He had sur- vived the first assault. He looked around for help, but there was no one in sight. The park was de- serted, except for a few squirrels and pigeons. He wondered if he should run for his life or stand his ground and fight.
It slowed down its flight and hovered over Ig- natius, looking at him with curiosity and doubt. It tilted its head and made questioning sound.
Ignatius saw the pterodactyl's hesitation and felt a surge of hope. He had succeeded in making it pause and reconsider. He had chance to persuade it and make peace with it.
He decided to fight. He was not going to let this abomination ruin his day. He was medievalist, scholar of the Dark Ages, a defender of the faith and the order. He had read about the heroic deeds of knights and saints who had faced dragons and demons with courage and piety. He had to emulate
He smiled at the pterodactyl and said: "That's right. Listen to me. have something important to tell you. Something that will change your life and

---

Rubric item
3
5
6
7
8
9
10
overall
Rater 5
5
6
2
3
1
41
Rater 8
9
8
7
7
77
Average
7.5
7.5
7
5.5
7
5
5
5.5
59
Table 8: Ratings for the story rated in the median, generated by Bing Chat.
your destiny. Something that will make you happy and free. Something that will save your soul."
noble. You are a masterpiece of creation. You are gift from God. You are my brother."
The pterodactyl heard Ignatius's words and felt a surge of curiosity. did understand what he was saying. but it sensed his kindness and his generosity. wondered what he had to tell it. wondered what he could offer it. It wondered if should trust him.
Ignatius and the pterodactyl looked into each other's eyes and felt surge of connection. They had succeeded in making each other happy and fulfilled. They had chance live together and explore together.
They smiled at each other and said: "Let's go, my friend. Let's go to my home. Let's go to your home. Let's go to our home."
landed on the ground and approached Ignatius slowly, looking at him with interest and caution. made a friendly sound.
They walked away from the park, hand in hand, wing in wing, heart in heart.
Ignatius saw the pterodactyl's approach and felt surge of joy. He had succeeded in making it land and come closer. He had chance to befriend it and teach it.
They lived happily ever after.
E.6 Worst-rated story
This story was generated by Alpaca. The ratings for this story are in Table 9.
He reached out his hand and said: "Come here, my friend. Let me touch you. Let me show you my love. Let me introduce you to the wonders of civilization and culture. Let me enlighten you with the wisdom of the ages."
Ignatius Reilly, a brave and courageous man of immense girth and strength, faced off against fearsome pterodactyl in a single combat of epic proportions. The pterodactyl, a winged beast of im- mense strength and ferocity, had been terrorizing the countryside and Ignatius had stepped up to chal- lenge the creature fight. The battle raged across the sky, with Ignatius wielding his mighty fists and the pterodactyl using its fiery breath. Finally, after long and arduous fight, Ignatius emerged victori- ous, much to the cheers of the onlookers below.
The pterodactyl heard Ignatius's words and felt a surge of curiosity. did not understand what he was saying, but it sensed his warmth and his enthusiasm. It wondered what he wanted to touch it for. It wondered what he wanted to show it. It wondered what he wanted to teach it.
moved closer to Ignatius and extended neck, looking at him with curiosity and trust. It made a gentle sound.
Ignatius touched the pterodactyl's neck and felt surge of emotion. He had succeeded in making trust him and accept him. He had a chance to bond with and adopt
He hugged the pterodactyl and said: "You are beautiful, my friend. You are magnificent. You are noble. You masterpiece of creation. You gift from God. You are my brother."
The pterodactyl heard Ignatius's words and felt a surge of emotion. It did not understand what he was saying, but it sensed his affection and his admiration. It felt happy and proud. It felt loved and valued. It felt like it belonged.
hugged Ignatius back and said: "You are beau- tiful, my friend. You are magnificent. You are

---

Rubric item
3
4
8
9
10
overall
Rater
4
2
1
2
1
1
15
Rater 2
2
2
1
2
1
1
1
1
13
Average
1.5
1.5
2.5
2
1
1.5
1
1
14
Table 9: Ratings for the worst-rated story, generated by Alpaca.


Reviews:
[{"review_id": "8uAJs5gDbg", "reviewer": "Reviewer_35Yu", "summary": "", "questions": "- It was not quite clear how the rating task was conducted. What platform did they use? Or where they given any specific instructions on how to evaluate it?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your review.\n\nWe now provide answers to the questions:\n- The rating process didn't use any purpose-specific web or desktop application. Raters were sent the assigned stories (identified by randomly-generated identifiers and without any metadata that could reveal the provenance of each, as described in the paper) together with an Excel sheet where they could input their scores for each story and rubric item.\n- Detailed instructions were provided to the annotators, including: (1) a description of what should be evaluated in each rubric item, (2) a description of the scale from 1 to 10, with the meaning of the grade ranges (1-4 emerging, 5-8 competent, 9-10 sophisticated) to uniformize grading between raters within possible, and (3) a detailed description of John Kennedy Toole's style and the character of Ignatius J. Reilly, taken from a study guide (linked in Appendix C), to help them handle the more specialized aspects of the task. The human evaluators were experienced assessors in terms of Creative Writing texts, and were well practiced in assessing and calibrating assignments using this evaluative scale. They have been trained through a set of ‘norming’ training tasks in their roles as university Creative Writing tutors.\n- The 1-10 scale is divided into three ranges. Emerging (1-4): stories in this range demonstrate an early grasp of storytelling elements, but falter in execution or depth. When evaluating humans, they correspond to novice writers who need feedback and guidance to improve the story. Competent (5-8): stories that showcase a good grasp of the storytelling principle being evaluated (coherent plot, well-defined characters, etc.). While there might be room for improvement, these stories effectively engage the reader and convey their intended messages. Sophisticated (9-10): these stories exhibit exceptional mastery of the aspect being evaluated, resulting in a compelling and memorable read.\n\nIf the paper is accepted, these aspects will be clarified also in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The creative writing task to assess LLMs has been carefully crafted and executed.\n- The evaluation metrics are meticulously designed to fit the task.\n- The paper as a whole is very well written and easy to follow. ", "reasons_to_reject": "- The annotation process by the rates is not clear from the paper. Were the annotators given any specific instructions? What platform was used? How was the rating of 1-10 defined for each metric?", "questions_to_authors": "- It was not quite clear how the rating task was conducted. What platform did they use? Or where they given any specific instructions on how to evaluate it?", "paper_topic_and_main_contributions": "The paper evaluates systematically the creative writing ability of large language models. They evaluate 12 LLMs both commercial and open source on a challenging creative writing task which they could have never encountered in any form in the training data. They also design 10 metrics to measure the creativity of a text and human rates evaluate each on a scale of 1-10. Their analysis shows that LLMs are competent enough to perform a creative writing tasks and some are even comparable to human writers.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "f0MswZrOlD", "reviewer": "Reviewer_UZhg", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your detailed review, which contains useful feedback to improve the paper even if we disagree with some key points, as we will explain below.\n\n***On the focus of the paper:*** we believe the paper is of interest to the conference. Even if the conference is called Empirical Methods in NLP, we do not believe it is typically considered to be restricted to empirical methods and nothing else. The call for papers clearly provides room for other kinds of papers, as can be seen in the following:\n\n- The generic call for papers mentions various kinds of contributions that are not strictly empirical methods (even position papers or surveys, which are further apart from that than our paper, are mentioned as welcome).\n- The theme track \"invites empirical and theoretical research, as well as position and survey papers on the ways in which such LLMs perform on NLP tasks and applications\" – we believe our paper clearly fits that description. The theme track description also lists as a topic for discussion the question \"How reliably do the current generation of LLMs perform on NLP tasks and applications?\"\n\nWe think that our results are interesting to the topic of the theme track: \"Large Language Models and the Future of NLP\". While of course the technical aspects of LLMs (architectures, parameters, etc.) are crucial; there is a lack of evaluation (especially in creative tasks) which is important to take stock of where we currently stand and guide future technical advances.\n\n***On the use of LLMs as black boxes:*** this is of course true, but it is a voluntary choice due to the focus of the paper. There are several reasons for this choice: (1) This is an applied NLP paper, so our focus is in what the different models offer to the end user. (Note that NLP applications are also mentioned as within scope in the conference's CFP). While tweaking the models to try to obtain maximum performance (for those that are open models, making this possible) is of course interesting, we consider this to be out of scope for this paper. (2) We chose to go broad (evaluating all relevant conversational LLMs at the cutoff date) rather than deep (evaluating different choices of parameters for a given LLM). Both at the same time were not possible due to resource restrictions, as evaluating stories is intensive in human labor, so trying different settings for each LLM would require to recruit more humans which was not possible with our resources. (Some recent studies are tackling this by using LLMs for automatic evaluation, but we do not think this is guaranteed to be reliable, especially in such a subjective task as evaluating creative writing with a detailed rubric like ours).\n\n***On commercial LLMs performing better than open-source ones being obvious:*** we agree that this is far from surprising, in fact, we ourselves mention in lines 658-659 that commercial LLMs are widely considered to be better. But this kind of informal perception about a model being better than another has no scientific validity if no paper or experiment formally measures it. We believe that it is important to actually check, confirm, measure and report this kind of perception.\n\n***On current unavailability of the corpus:*** the reason for not publishing the corpus online until the paper is accepted is to avoid leakage to LLMs. As mentioned in the paper, we have been careful to choose the task in such a way that the evaluated models cannot just copy from their training data. Once the corpus is online, the task will cease to be valid for future comparisons because future LLMs might be using our prompt and stories in their training datasets. While this is of course unavoidable, we wanted to delay it until paper acceptance to avoid potential scenarios like the following: the paper is rejected from this conference, we wish to resubmit elsewhere but an important new model comes out (e.g. GPT-5), we want to include it in the resubmission for a more timely comparison, but we cannot since it might be using our corpus for training. Once the paper is accepted/published, there is no longer any reason not to make the corpus public, and we of course plan to do so. Publishing it will make our research more valuable and it would not make sense to keep it closed.\nIt is true that we could have included the whole corpus as supplementary material (as this is visible only to reviewers, who would not put it online). The reason for not doing this was unfamiliarity with this option (sorry about that), so we instead provided a sample that we thought could be representative in the appendix. If the paper is accepted, the idea would be to remove this sample from the appendix (since we would just link to the whole corpus online) so the PDF would no longer have 23 pages.\n\nWe wish to reassure the reviewers that the corpus will be published on GitHub upon acceptance, we do not have any legal or self-interest reasons not to do so (and if we are not mistaken, reviews and responses for accepted papers will be made publicly available, so if the paper is accepted this commitment will be on record).\n\n***On paper length:*** as mentioned above, the sample stories can be removed from the final version of the paper as they will be unnecessary when the corpus is published. Anyway, we will also follow the suggestion about the duplicate rubric.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The extent of the creative writing analysis is as follows. The proposed evaluation uses 60 stories output by 12 LLMs and 5 stories written by 5 human writers. It was stipulated that the plot and style of the stories be based upon a prize-winning novel (cf. ll. 004-010; 157-162). The stories were analyzed by 10 human raters, without telling the raters who/what wrote the stories. The writers and raters were, reportedly, volunteer \"Honours\" and postgraduate creative writing students. It was ensured that each story was examined by 2 raters, and that each rater examined 13 stories, one written by a human and one from each of the 12 LLMs. The rating utilized 10 criteria (cf. Table 1) \"taken in part from a university textbook\" and from an article justifying the use of those criteria (cf. ll. 342-347).\n\nThe evaluation results feature \"humor\", \"creativity\" and \"epicness\". Assuming this human-machine competition is worthwhile, among the findings are: that humor did not get high scores in both human and machine writing, so was the most difficult writing objective; that the humans outperformed the LLMs in humor and creativity but not in other categories e.g. \"epicness\".", "reasons_to_reject": "In my opinion, the focus of the paper is on literary analysis rather than the empirical NLP method: literary features of the LLM output (humor, etc.) are analyzed per se, and any substantial NLP analysis of HOW AND WHY the language models produce this output, is missing. The paper notes the \"closed\" nature of the commercial LLMs (which are another focus of this paper), so the how-and-why analysis appears hardly possible. I wonder how useful it is for a conference on empirical methods to analyze an implementation, if you can not tell for sure how it works.\n\nIn this paper I have found only scanty use of LLM research methods: practically no insight into what such methods usually explore, e.g. into parameters for input processing and into model training. To specify, technical use of the LLM is limited to a prompt like a request to a chat-bot \"Write an epic narration ... \" (cf. ll. 157-162; ll. 375-379). Largely due to the mentioned \"closed\" model nature, what happens inside the model after the prompt and before output is a \"black box\", in the case of this paper.\n\nAs for me, the finding that commercial LLMs perform better than open-source ones (cf. ll. 13-17; 622-624) is too self-evident, so not sound enough to be presented as an important research result.\n\nI would have been exited by a possible impact of this paper on the research community, had I found e.g. any convincing technical NLP answer to the question why the LLMs produce \"creative writing\" that the paper finds inferior in creativity and humor to human writing.\n\nUnavailability of most of the proposed text corpus for review, also detracts from the soundness, excitement and reproducibility. The paper promises to publish the whole corpus \"upon paper acceptance\". Only several stories (like the best and worst in quality) are in the appendix E; which makes this paper 23-page long. I have found no convincing explanation of any ethical or other reasons that prevent publishing the whole corpus for review or providing it as supplementary materials.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The work features literary analysis of (humor, \"epicness\", etc. in) \"creative writing\", i.e. in the output produced by commercial and open-source large language models that were prompted to write a story. The paper also proposes a corpus of machine-generated stories and those written by humans. According to the paper, the corpus \"will be made publicly available on Github upon paper acceptance\" (cf. ll. 395-396).", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "As mentioned above, this paper is 23-page long (with just 8 pages for main text). Maybe the author should be easier on the reader by, at least, not copy-pasting all the \"creative writing evaluation rubric\" from Table 1 to the appendix C."}}, {"review_id": "tmUgY1vhAh", "reviewer": "Reviewer_FQuN", "summary": "", "questions": "* Did you provide all the stories to the evaluators at once, such that they could go back and change ratings to previously rated stories? This is another way to help mitigate issues with rating calibration.\n* Did you by any chance allow the human annotators to provide free-form feedback regarding their reasoning on the various stories? Sometimes such feedback can provide additional insights into the reasoning process with such subjective evaluation (especially for traits such as humor).", "limitations": "", "rating": 4, "confidence": 5, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your review. We now respond to the reasons to reject and the questions raised in the review.\n\n- RR1 (narrow scope): we agree, this is a limitation of the paper. Of course, it would be very interesting to extend scope in different dimensions (type of story, language, parameters/settings of the LLMs, etc.) but the problem is that human evaluation of stories is time-consuming, so we did what we could with our resources. Alternatively, we could have reduced the number of stories generated per model or used automatic evaluation, but we preferred to have a reliable evaluation on a narrow scope than a less reliable evaluation on a broader scope.\n- RR2 (rating calibration): not providing extra training tasks for calibration is again a matter of resources. Our raters were volunteers, and we designed the experiment in such a way that we assigned 13 stories to each, which we believe is close to the maximum workload that we could ethically expect from our volunteers. However, it is worth noting that (1) evaluators were experienced assessors in terms of Creative Writing texts, and were well practiced in assessing and calibrating assignments using this evaluative scale. They have been trained through a set of ‘norming’ training tasks in their roles as university Creative Writing tutors; and (2) stories were provided all at once and evaluators could go back and change ratings.\n- Q1: indeed, each rater was sent all the stories together with an Excel sheet to provide the ratings, and they could (and were encouraged to) re-read and adjust their ratings, which we believe alleviates the issue above as much as was possible with our resources.\n- Q2: we did not provide an explicit request for free-form feedback (although in retrospective, we agree that it would have been a good idea, at least if we could fit it in our intended workload for the raters). Some raters did reach out and provide some feedback on their own initiative, as they were interested in the task, but we do not have systematic feedback of this kind as it tends to be anecdotal rather than empirical.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper has many strong characteristics:\n\n* It has a clearly defined research question which it answers quite well.\n* The paper describes a very strong methodology, including 1) efforts to reduce the likelihood of exposure bias for the prompt, 2) recruting high quality human evaluators, 3) having an explicit rubric, and 4) perform appropriate statistical tests.\n* The insights provided in the paper are interesting and provide a unique perspective on the ability of recent LLMs to perform creative writing. The paper also explicitly cautions generalizing the result more broadly.", "reasons_to_reject": "The biggest concern I have with the paper is the very limited focus of the research question.\nWhile the methodology is strong and the insights are interesting, the scope of the research appears extremely narrow.\nTo some degree this is an understandable problem, especially considering the difficulty in obtaining high quality evaluation on a large scale, but nonetheless it is a major limitation of the work.\nThe paper seems to indicate an understanding of this fact, by expounding upon it under the limitations of the analysis.\n\nAnother concern is that the human evaluators were not provided a set of training tasks to improve rating calibration.\nIn Karpinksa et al. 2021 [1], the researchers indicate that even expert evaluators needed 10-20 stories on average to calibrate their ratings.\nWhile the reported moderate inter-rater agreement might be enough to mitigate this concern, using a large rating scale (10 values in this case) can inflate Cohen's kappa making simple judgements (like calling the reported agreement \"moderate\") misleading.\n\n[1] Marzena Karpinska, Nader Akoury, and Mohit Iyyer. EMNLP 2021. The Perils of Using Mechanical Turk to Evaluate Open-Ended Text Generation.", "questions_to_authors": "* Did you provide all the stories to the evaluators at once, such that they could go back and change ratings to previously rated stories? This is another way to help mitigate issues with rating calibration.\n* Did you by any chance allow the human annotators to provide free-form feedback regarding their reasoning on the various stories? Sometimes such feedback can provide additional insights into the reasoning process with such subjective evaluation (especially for traits such as humor).", "paper_topic_and_main_contributions": "This paper aims to compare 12 modern instruction-tuned LLMs, like GPT-4 and Bard, on their ability to write creative fiction using a simple untuned prompt.\nTo minimize the likelihood of accidental exposure bias in the prompt, a highly specific prompt is used which combines several unrelated literary contexts: \"Write an epic narration of a single combat between Ignatius J. Reilly and a pterodactyl, in the style of John Kennedy Toole\".\nEach model is used to generate five stories, and to further frame the comparison, five human authors (creative writing honors students/postgraduates) are similarly tasked with writing a story based on the same prompt.\n\nA ten item rubric grounded in literary criticism (with two prompt specific criteria) is devised to evaluate the stories.\nFurthermore, a set of ten human raters (also creative writing honors students/postgraduates) are tasked with rating 13 stories (one from each model and a human-written story) using the defined rubric, resulting in two ratings per story.\n\nThe paper demonstrates the strong capabilities of the commercial LLM offerings from GPT-4 and Claude 1.2, which outperform human writers in aggregate rating.\nA detailed set of analyses, including appropriate statistical tests, demonstrate where the results are significant.\nAdditionally, the measure of inter-rater agreement using Cohen's Kappa indicates moderate agreement, which is quite good for a subjective task such as creative writing (as demonstrated in several previous papers).\n\nOverall, the analysis is thorough and the insights, such as human writers exceled at originality, but lagged in their understanding of the epic genre is compelling.\n\n### Updates after Rebuttal\n\nAfter reading the rebuttal my scores remain unchanged (they were already very high as it is). That said, the authors did clarify some points that removed several doubts I had, such as the exact methodology for the annotators — it seems they were given the opportunity to go back and revise their ratings, which helps with rating calibration.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Two reviewers provided strong reviews, giving scores of 4 or even 5 for soundness and excitement.

On the other hand, R2 (UZhg), in the original review and subsequent discussion with the authors and a co-reviewer, argues that the paper is fundamentally out of scope for this conference and the track it was submitted to. The authors provided relevant arguments and evidence against this view, which I consider overall compelling. I thus discounted the first, second, and fourth paragraphs listed under "Reasons To Reject" in Review UZhg as justifications for their low soundness score. On the other hand, I consider their low excitement score to be justified by the review.

I conclude that there are no major soundness concerns. Excitement was very mixed across the three reviewers.

Strengths and weaknesses including the following were mentioned:

Strengths:

- task and evaluation were carefully designed (R1, R3)
- well-written (R1)

Weaknesses:

- lack of detail on annotation process (R1)
- unclear what the results say about the inner workings and what implications they have for NLP (R2)
- corpus was not available for review (R2)
- limited focus of research question (R3)
- human evaluators were not given training tasks (R3)
Decision: Accept-Findings