# Evaluating Creativity: Can LLMs Be Good Evaluators in Creative Writing Tasks?
## 1 Introduction  
The rapid advancement of Artificial Intelligence (AI) has catalyzed transformative advancements across numerous domains, including the generation and evaluation of texts. This technological paradigm shift includes Large Language Models (LLMs) such as ChatGPT, Gemini, and Llama, which have shown remarkable capabilities in generating human-like texts by utilizing extensive datasets and advanced algorithms [1, 2, 3, 4]. Closed-source models like GPT-4 and Gemini consistently outperform their open-source models, such as Llama, particularly in terms of linguistic accuracy, complexity of generated content, and consistency in evaluation [5]. Closed-source models benefit from greater resources and larger training datasets, enabling them to generate sophisticated, contextually accurate, and nuanced text [5, 6]. For example, GPT-4 consistently outperforms GPT-3.5 in producing high-quality, cohesive text across various tasks [7, 8]. This paper focuses on the potential of LLMs, particularly GPT-4, to evaluate creative writing, a task traditionally handled by humans. Human evaluations are often inconsistent and subjective, posing challenges for reliable assessments [9]. Traditional automatic metrics like BLEU, ROUGE, and METEOR have been widely used to evaluate language tasks. However, these metrics perform poorly in assessing creative and open-ended writing because they cannot capture subjective and nuancedelements [10; 11; 12]. To overcome these limitations, new evaluation frameworks, such as MT-Bench, PandaLM, AlpacaEval, and DeepEval, have been introduced [13]. These methods aim to evaluate subjective and creative aspects of LLM performance in more detail. Previous studies primarily utilized Large language Models (LLMs) to evaluate the linguistic quality and logical coherence of AI-generated content [14]. However, recent developments suggest that AI is now capable of evaluating the intricate details of creativity that were previously considered exclusive to human judgment [15; 16]. This advancement raises an important question: can LLMs function as effective evaluators in creative writing tasks? To address this, we compared texts written by humans and those generated by GPT-4 by using six creativity criteria: fluency, flexibility, originality, elaboration, usefulness, and specific creative strategies [17]. A total of 200 human-written texts and 200 GPT-4-generated texts were evaluated by eight human experts and LLMs configured with varying temperature settings and personas, as seen in Figure 1. This approach allowed for a comprehensive comparison of the consistency and performance of human and AI evaluations. However, evaluating creative writing presents significant challenges, particularly due to the inherent ambiguity of creativity as a concept. Unlike objective linguistic measures such as grammaticality, coherence, and relevance, creativity lacks a universal standard, and evaluators may have different subjective preferences and interpretative frameworks. This variability is further influenced by human evaluators' background and personal values, leading to inconsistencies even among experienced professionals in literary assessment. Differences in assessment criteria, biases, and varying levels of expertise further complicate reliability, making it difficult to establish standardized evaluation methods. These limitations raise concerns about the reproducibility of human-based assessments and highlight the need for alternative or complementary approaches. Given these challenges in creative writing evaluation, we examine the differences between human and AI assessments and investigate whether LLMs can provide a more consistent and systematic approach. By systematically comparing human-written and AI-generated texts, this work explores whether LLMs can address the challenges of creative writing assessment.