# Evaluating Creativity: Can LLMs Be Good Evaluators in Creative Writing Tasks?
## 4 Experiments
### Main Results  
The quantitative analysis of the evaluation metrics for each criterion provides valuable insights into the comparative effectiveness of evaluations by human experts and LLMs (GPT-4o and GPT-3.5), as shown in Tables 1 and 2. Also, statistical analyses and score distribution are provided in Appendices A, B, and C.  
In fluency, LLMs consistently achieved higher IAA and mean scores compared with human evaluators. Among the models, GPT-3.5 with persona-based evaluation demonstrated the highest consistency, achieving an IAA of 96% for both human-written and GPT-4-generated texts. These results emphasize the reliability of LLMs in assessing the smoothness and readability of creative writing. In the evaluation of flexibility and originality, human evaluators exhibited significant variability, with extremely low agreement levels. For flexibility, human evaluators achieved an IAA of only 2%, while for originality, there was no agreement among human evaluators for human-written texts. In contrast, LLMs, particularly GPT-3.5, displayed consistent performance with significantly higher agreement levels and mean scores. This demonstrates the ability of these models to objectively evaluate adaptability and uniqueness in creative outputs. Elaboration emerged as the strongest criterion for LLMs. GPT-4o with persona-based evaluation assigned perfect scores, with an average score of 3.00, to all GPT-4-generated texts. It also achieved complete agreement among evaluators, with an IAA of 100%. This highlights the ability of LLMs to effectively recognize detailed and well-structured ideas, surpassing human evaluators in consistency and agreement levels. \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline \multirow{2}{*}{Human Writings} & \multicolumn{2}{c}{Fluency} & \multicolumn{2}{c}{Flexibility} & \multicolumn{2}{c}{Originality} \\ \cline{2-7}  & Mean/SD & IAA/\% & Mean/SD & IAA/\% & Mean/SD & IAA/\% \\ \hline Human & 2.14/0.66 & 0.13 (1.00\%) & 2.11/0.73 & 0.20 (0.00\%) & 2.11/0.74 & 0.19 (0.00\%) \\ \hline GPT-4o (T) & 2.15/0.36 & 0.79 (63.50\%) & 2.06/0.30 & 0.80 (66.00\%) & 2.07/0.33 & 0.80 (64.50\%) \\ \hline GPT-4o (P) & 2.14/0.35 & 0.79 (84.50\%) & 2.08/0.28 & 0.80 (88.50\%) & 2.08/0.31 & 0.80 (86.00\%) \\ \hline GPT-3.5 (T) & 2.87/0.33 & 0.79 (75.00\%) & 2.37/0.48 & 0.75 (43.00\%) & 2.70/0.46 & 0.76 (59.50\%) \\ \hline GPT-3.5 (P) & 2.92/0.28 & 0.80 (90.50\%) & 2.31/0.46 & 0.76 (71.00\%) & 2.74/0.44 & 0.76 (80.50\%) \\ \hline \multirow{2}{*}{Human Writings} & \multicolumn{2}{c}{Elaboration} & \multicolumn{2}{c}{Usefulness} & \multicolumn{2}{c}{Specific Creativity} \\ \cline{2-7}  & Mean/SD & IAA/\% & Mean/SD & IAA/\% & Mean/SD & IAA/\% \\ \hline Human & 2.14/0.72 & 0.21 (0.50\%) & 2.08/0.68 & 0.13 (0.00\%) & 1.92/0.64 & 0.08 (0.50\%) \\ \hline GPT-4o (T) & 2.89/0.31 & 0.79 (73.50\%) & 1.97/0.2  
For the usefulness criterion, human evaluators tended to assign higher scores compared with LLMs, particularly for human-written texts. This difference is likely influenced by cognitive biases among human evaluators, such as attributing greater intent and practicality to human-created content. While LLMs displayed more consistent scoring patterns, their average scores in this criterion were generally lower than those assigned by human evaluators. In the evaluation of specific creativity strategies, LLMs outperformed human evaluators in agreement levels. GPT-3.5 with persona-based evaluation demonstrated notable consistency, achieving an IAA of 60% for GPT-4-generated texts. However, human evaluators' subjectivity often resulted in lower agreement levels, particularly for this nuanced and context-dependent criterion. Overall, the results indicate that LLM evaluators consistently outperform human evaluators in all criteria, particularly in evaluation consistency. Elaboration emerged as the strongest criterion, with GPT-4o with persona-based configurations achieving perfect scores for all 200 GPT-4-generated stories, highlighting its ability to assess detailed and well-structured content effectively.