Comparing Large Language Models verbal creativity to human verbal creativity
Authors: Anca Dinu, Andra Maria Florescu (University of Bucharest, 2024)
Topic: Language Models

Abstract
This study investigates the verbal creativity differences and similarities between Large Language Models and humans, based on their answers given to the integrated verbal creativity test in AUTHOR. Since this article reported a very small difference of scores in favour of the machines, the aim of the present work is to thoroughly analyse the data through four methods: scoring the uniqueness of the answers of one human or one machine compared to all the others, semantic similarity clustering, binary classification and manual inspection of the data. The results showed that humans and machines are on a par in terms of uniqueness scores, that humans and machines group in two well defined clusters based on semantics similarities between documents comprising all the answers of an individual (human or machine), per tasks and overall, and that the separate answers can be automatically classified in human answers and LLM answers with traditional machine learning methods, with F1 scores ranging from 68 to 74. The manual analysis supported the insight gained from the automated methods in that LLMs behave human-like while performing creativity tasks, but there are still some important distinctive features to tell them apart.

Introduction
Creativity has made it possible for humanity to survive and develop since prehistoric times. Despite the perception that some people are more creative than others, many psychologists argue that everyone has the capacity for creativity or that creativity is innate and encoded in human nature AUTHOR. Creativity is inherently interdisciplinary, involving domains like psychology, cognitive sciences, philosophy, arts, engineering, mathematics, or computer science. Recently, it has become a field of interest in GenerativeAI (GenAI) AUTHOR in general, and in particular, in Large Language Models (LLMs) AUTHOR. However, much of the current research in generative models AUTHOR is concerned with constraining them so they do not harm people, so they are well-behaved, factual, non-hallucinating, non-biased, non-negative, non-misleading, non-toxic, etc., and for a good reason. In contrast, fewer studies (see section previousWork) focus on encouraging them to be original, unconstrained, or creative, although computational creativity, as a research field, dates back to the late '90s AUTHOR, AUTHOR with various disciplines including creative writing, music, or graphics, utilizing artificial intelligence, particularly neural networks, heuristics, and so on. A good survey on LLMs' verbal creativity is AUTHOR. Since work on LLMs creativity is just at the beginning, there is a need for methods, resources, and evaluation to better understand LLMs' creative abilities and their differences and similarities with human creative traits. In a recent article, AUTHOR designed a verbal creativity test, integrating a wide range of tasks and criteria inspired from psychological creativity testing, and administrating it to both humans and LLMs. The scope of this paper is to analyze the answers given by LLMs and human respondents to this previous study, for a direct comparison of human and machine verbal creativity. To this end, we will compute uniqueness scores, cluster the individual answers per task and overall, perform supervised binary classification with classic machine learning methods on all answers and manually analyze some of the data particularities.

Conclusion
This study showed that there are some differences between human and machine answers given to a verbal creativity test, but also plenty of similarities. The LLMs' answers vary much like the humans answers. Individual, unique answers, w.r.t. to the set of all answers are produced by both humans and machines alike, with no noticeable difference. Still, at a semantic level, humans and machines generally group together as individuals. The performance of automatic classification between human and machine answers is moderate and leaves room for improvement. The general findings of this study indicate that LLMs' creative capabilities are comparable with human abilities and, as such, they could be put to good use in the creative domain. Humans "just" need to adapt to their usage, mind the ethics and safety issues, and discern the information at every step, instead of blindly using them. In future work, we will focus on expanding the dataset, by adding more LLMs' and humans' answers to the test, for a better statistical coverage. Also, we aim to manually investigate more in-depth the database, to look for more systematic patterns for both humans and machines. As creativity remains a domain with endless possibilities, we also plan to investigate other aspects of LLMs' creativity, such as language or image. Another future approach worthy of pursuing is using Deep Learning approaches instead of traditional Machine Learning approaches for the binary classification task, or using metrics specific to LLM-generated tasks.