# Evaluating Creativity: Can LLMs Be Good Evaluators in Creative Writing Tasks?
## 5 Conclusions and Future Works  
This paper investigates the ability of Large Language Models (LLMs) to evaluate creative writing, emphasizing their potential for providing automated, reliable, and time-efficient assessments. By automating the evaluation process, models like GPT-4o and GPT-3.5 Turbo significantly reduce evaluation time, making them valuable tools for large-scale creative writing assessments. Our findings indicate that these models demonstrate high consistency and performance across criteria such as fluency, flexibility, originality, elaboration, usefulness, and specificity. However, notable limitations were identified. The models often failed to detect deeper narrative flaws, such as incomplete compositions, redundancy, and logical inconsistencies, yet still assigned high scores to these outputs. Additionally, AI-generated assessments may exhibit biases toward certain stylistic patterns, potentially overlooking unconventional or culturally diverse writing styles. Furthermore, unlike human evaluators, AI models lack the ability to provide qualitative feedback or assess the underlying intent of a creative piece, which may limit their applicability in real-world educational or professional settings. This underscores a significant challenge in using AI models for comprehensive assessments, despite their efficiency and consistency. A key limitation of this study is the lack of precise definitions for some evaluation criteria, such as fluency and flexibility. The overlap and ambiguity between these concepts may have contributed to inconsistent evaluations, as both human and AI evaluators interpreted these criteria differently. To improve future assessments, it is crucial to refine these definitions, ensuring they capture distinct aspects of creativity more accurately. Additionally, variability in Inter-Annotator Agreement (IAA) was observed, particularly among human evaluators. While AI models maintained high consistency, human evaluators struggled to reach agreement due to the subjective nature of creative writing assessments. This variability highlights a key limitation in practical applications, where AI-based evaluations may require human oversight to ensure fairness and adaptability. Future research should focus on integrating AI and human evaluations to address these limitations. An effective approach is developing hybrid evaluation frameworks that allow AI models to provide initial assessments, which are then refined by human evaluators to incorporate qualitative insights. Additionally, advancing LLM training on more diverse creative datasets, including culturally varied narratives and unconventional writing styles, could enhance AI's ability to evaluate a broader range of content. Moreover, ethical considerations must be addressed, particularly in terms of ensuring authenticity and originality in creative evaluations. Establishing clear guidelines on how AI-generated assessments should be interpreted in practical applications, such as education, publishing, or automated grading, will be essential to maximizing their effectiveness while mitigating potential biases. \begin{table}
\begin{tabular}{p{142.3 pt} p{227.6 pt}} \hline \hline  & [WP] Rewrite your favorite song as a story. \\ \hline GPT-4 & Once upon a time, in a small town nestled between the mountains and the sea, lived a young man named [PERSON]. He was a dreamer, always lost in his thoughts, yearning for something more than the mundane life he was living.... The star had not only guided him to his dreams but also taught him the value of perseverance and belief. And so, [PERSON]’s story became a legend, inspiring others to follow their dreams **and** \\ \hline \hline \end{tabular}
\end{table} 
Table 5: The table provides an excerpt from GPT-4’s output for the prompt “Rewrite your favorite song as a story”. This excerpt illustrates the incomplete narrative generated by GPT-4 in response to this creative writing. **Author Contributions:** Conceptualization, S.K. and D.O. ; methodology, S.K. ; software, S.K. ; validation, S.K. and D.O.