# Evaluating Creativity: Can LLMs Be Good Evaluators in Creative Writing Tasks?
## 4 Experiments
### Case Study  
#### 4.3.1 Strengths of LLMs in Evaluation  
LLMs such as GPT-3.5 and GPT-4o demonstrated remarkable consistency in their evaluations of both human-written and AI-generated content. As seen in Table 3, in a case study where both human-written and AI-generated texts explored a story about [PERSON] intentionally sabotaging his career by choosing bad movies, only to unexpectedly win an Oscar for his role in \"Paul Blart: Mall Cop 3\", both the AI and human texts presented a consistent storyline. The narratives emphasized [PERSON]'s dramatic and paradoxical decisions, using similar structures and vocabulary to highlight the irony of his actions.  
In Table 4, GPT-3.5 models, regardless of temperature or persona settings, consistently assigned a score of 3.00 for fluency and originality to the human-written texts. In contrast,  
\begin{table}
\begin{tabular}{c c} \hline \hline \multicolumn{2}{c}{[WP] [PERSON] in a fit of rage begins to torpedo his own career by deliberately acting poorly and taking on bad films. He finally wins an Oscar for starring in _Paul Blart: Mall Cop 3_. \\ \hline \multirow{4}{*}{GPT-4} & In a shocking turn of events, **Hollywood’s golden boy, [PERSON]**, has embarked on a self-destructive journey to torpedo his own illustrious career. The actor, known for his impeccable performances and choice of critically acclaimed films, has suddenly started delivering subpar performances and choosing questionable scripts. It’s as if he’s deliberately trying to tarnish his own reputation. The first sign of this bizarre behavior was when he signed on for the third installment of the widely panned **’Paul Blart: Mall Cop’ series**. \\ \hline \multirow{4}{*}{Human} & **A famous actor, [PERSON]** intentionally damages his career while trying to win an Oscar. Interestingly, he chooses to star in terrible films and fails in them. The film **”Paul Blart: Mall Cop 3”**, which no one thought would help his reputation, was his choice to star in. However, his role as a mall cop going through a difficult time in this film manages to touch the hearts of all viewers. Award season is approaching, and rumors exist that [PERSON] might take home an award for this performance. \\ \hline \hline \end{tabular}
\end{table}
Table 3: The table shows excerpts from both human-written stories and GPT-4-generated stories about [PERSON]’s narrative, specifically focusing on a creative prompt where he sabotages his career by choosing bad films. The selected excerpts illustrate the differences in narrative style and coherence between human-written and AI-generated content, allowing for a comparative analysis of writing quality across different evaluators and models.  
human evaluators gave significantly lower mean scores of 1.38 for the same texts, highlighting the discrepancy between human and AI evaluations. Similarly, GPT-4o models (both temperature and persona variants) showed uniformity, assigning a score of 2.00 in the fluency and originality criteria for human-written content. This pattern of consistency also extended to AI-generated texts. For the specific creativity strategies criterion, GPT-3.5 models consistently gave a score of 2.00, while GPT-4o models assigned a slightly higher mean score of 2.13, demonstrating a minor variation. These results confirm that LLMs maintain a high level of consistency in their evaluations, regardless of the criteria or content origin.  
LLMs have advantages, particularly in terms of evaluation speed and consistency. Since LLMs operate automatically, they can assess large volumes of creative writing much faster than human evaluators, who may require more time. The automated nature of LLMs ensures that each evaluation is carried out with consistency, applying the same criteria across different texts without being influenced by personal biases or other factors that can be inconsistent.