On impact and evaluation in Computational Creativity : A discussion of the Turing Test and an alternative proposal

Computational Creativity is the AI subfield in which we study how to build computational models of creative thought in science and the arts. From an engineering perspective, it is des irable to have concrete measures for assessing the progress made from one version of a program to another, or for comparing and contras ting different software systems for the same creative task. We de scribe the Turing Test and versions of it which have been used in orde r to measure progress in Computational Creativity. We show th at the versions proposed thus far lack the important aspect of inte rac ion, without which much of the power of the Turing Test is lost. We a rgue that the Turing Test is largely inappropriate for the purpos es of evaluation in Computational Creativity, since it attempts to ho mogenise creativity into a single (human) style, does not take into ac count the importance of background and contextual information for a c eative act, encourages superficial, uninteresting advances in fro nt-ends, and rewards creativity which adheres to a certain style over tha t which creates something which is genuinely novel. We further argu e that although there may be some place for Turing-style tests for C omputational Creativity at some point in the future, it is curren tly untenable to apply any defensible version of the Turing Test. As an alternative to Turing-style tests, we introduce two de scriptive models for evaluating creative software, the FACE mode l which describes creative acts performed by software in terms of tu ples of generative acts, and the IDEA model which describes how such creative acts can have an impact upon an ideal audience, given id eal information about background knowledge and the software de v lopment process. While these models require further study and e l boration, we believe that they can be usefully applied to current sys ems as well as guiding further development of creative systems. 1 The Turing Test and Computational Creativity The Turing Test (TT), in which a computer and human are interr ogated, with the computer considered intelligent if the huma n interrogator is unable to distinguish between them, is principal ly a philosophical construct proposed by Alan Turing as a way of determ ining whether AI has achieved its goal of simulating intelligence . The TT has provoked much discussion, both historical and contem porary, however this has principally been within the philosophy of A I: most AI researchers see it as a distraction from their goals, enco uraging a mere trickery of intelligence and ever more sophisticated n atural language front ends, as opposed to focussing on real problems. D espite the appeal of the (as yet unawarded) Loebner Prize, most subfi elds of AI have developed and follow their own evaluation criteri a and methodologies, which have little to do with the TT. 1 School of Informatics, University of Edinburgh, UK 2 Department of Computing, Imperial College, London, UK Computational Creativity (CC) is a subfield of AI, in which re searchers aim to model creative thought by building program s which can produce ideas and artefacts which are novel, surprising and valuable, either autonomously or in conjunction with humans. Th ere are three main motivations for the study of Computational Creat ivity: • to provide a computational perspective on human creativity , in order to help us to understand it (cognitive science); • to enable machines to be creative, in order to enhance our liv es in some way (engineering); and • to produce tools which enhance human creativity (aids for cr eative individuals). Creativity can be subdivided into everyday problem-solvin g, and the sort of creativity reserved for the truly great, in which a problem is solved or an object created that has a major impact on other people. These are respectively known as “little-c” (mundane) a nd “bigC” (eminent) creativity . Boden [3] draws a similar disti nction in her view of creativity as search within a conceptual space, w h re “exploratory creativity” searches within the space, and “tran sformational creativity” involves expanding the space by breaking one or m e of the defining characteristics and creating a new conceptua l space. Boden sees transformational creativity as more surprising , i ce, according to the defining rules of the conceptual space, ideas w ithin this space could not have been found before. There are two notions of evaluation in CC: ( i) judgements which determine whether an idea or artefact is valuable or not (an e ssential criterion for creativity) – these judgements may be made int rnally by whoever produced the idea, or externally, by someone else and (ii ) judgements to determine whether a system is acting creativ ely or not. In the following discussion, by evaluation, we mean the latter judgement. Finding measures of evaluation of CC is an active area of research, both influenced by, and influencing, practical a nd theoretical aspects of CC. It is a particularly important area, s ince such measures suggest ways of defining progress in the field, 3 as well as strongly guiding program design. While tests of creativity in humans are important for our understanding of creativity, they do n ot usually causehumans to be creative (creativity training programs, which train people to do well at such tests, notwithstanding). Way s in which CC is evaluated, on the other hand, will have a deep influence o future development of potentially creative programs. Clearl y, different modes of evaluation will be appropriate for the different mo tivations listed above. 3 The necessity for good measures of evaluation in CC is somewh at paralleled in the psychology of creativity: “Creativity is becoming a p opular topic in educational, economic and political circles throughout th e world – whether this popularity is just a passing fad or a lasting change in in terest in creativity and innovation will probably depend, in large part, on wh ether creativity assessment keeps pace with the rest of the field.” [4, p. 64] The Turing Test is of particular interest to CC for two reason s. Firstly, unlike the general situation in AI, the TT, or varia tions of it, arecurrently being used to evaluate candidate programs in CC. T hus, the TT is having a major influence on the development of CC. Thi s influence is usually neither noted nor questioned. Secondly , there are huge philosophical problems with using a test based on imita tion to evaluate competence in an area of thought which is based on or iginality. While there are varying definitions of creativity, t he majority consider some interpretation of novelty and utility to be es sential criteria. For instance, one of the commonalities found by Rothe nberg in a collection of international perspectives on creativit y is that “creativity involves thinking that is aimed at producing ideas o r products that are relatively novel” [5, p.2], and in CC the combin ation of novelty and usefulness is accepted as key (for instance, s ee [6] or ). In , Plucker and Makel list “similar, overlapping a nd possibly synonymous terms for creativity: imagination, ingenuity, innovation, inspiration, inventiveness, muse, novelty, originality, serendipity, talent and unique”. The term ‘imitation’ is simply antipodal to many of these terms. In the following sections, we firstly describe and discuss so me attempts to evaluate Computational Creativity using the Turi ng Test or versions of it ( §2), concluding that these attempts all omit the important aspect of interaction, and suggest the sort of directio n that a TT for a creative computer art system might follow. We then pres ent a series of arguments that the TT is inappropriate for measuring creativity in computers (or humans) in §3, and suggest that although there may be some place for Turing-style tests for Computational C reativity at some point in the future, it is currently untenable and impractical. As an alternative to Turing-style tests, in §4, we introduce two descriptive models for evaluating creative software, the F ACE model which describes creative acts performed by software in term s of tuples of generative acts, and the IDEA model which describes h ow such creative acts can have an impact upon an ideal audience, given ideal information about background knowledge and the softw are development process. We conclude our discussion in §5. 2 Attempts to evaluate Computational Creativity using the Turing Test or versions of it There have been several attempts to evaluate Computational Cre tivity using the Turing Test or versions of it. While these are us f l in terms of advancing our understanding of CC, they do not go f ar enough. In this section we discuss two such advances ( §2.1 and§2.2), and two further suggestions on using human creative behavio ur as a guide for evaluating Computational Creativity ( §2.3). We highlight the importance of interaction in §2.4. 2.1 Discrimination tests Pearce and Wiggins [7] assert for the need for objective, fal sifi ble measures of evaluation in cognitive musicology. They propo se the ‘discrimination test’, which is analogous to the TT, in whic subjects are played segments of both machine and human-generated mus ic and asked to distinguish between them. This might be in a part icular style, such as Bach’s music, or might be more general. The y also present one of the most considered analyses of whether Turin g-style tests such as the framework they propose might be appropriat e for evaluating Computational Creativity [7, §7]. While they do not directly refer to Boden’s exploratory creativity , instea d referring to Boden’s distinction between psychological (P-creativity , concerning ideas which are novel with resepct to a particular mind) and h istorical creativity (H-creativity, concerning ideas which are novel with respect to the whole of human history ), they do argue that much creative work is carried out within a particular style. They cite Garnham’s response