The Multimodal Nature of Embodied Conversational Agents

The Multimodal Nature of Embodied Conversational Agents Max Louwerse (mlouwerse@memphis.edu) Nick Benesh (nbenesh@memphis.edu) Shinobu Watanabe (swatanab@memphis.edu) Bin Zhang (bzhang@memphis.edu) Patrick Jeuniaux (pjeuniau@memphis.edu) Divya Vargheese (dvarghes@memphis.edu) Department of Psychology / Institute for Intelligent Systems University of Memphis Memphis, TN 38152 USA Abstract Embodied conversational agents (ECA’s) have become ubiquitous in human-computer interaction applications. Implementing humanlike multimodal behavior in these agents is difficult, because so little is known about the alignment of facial expression, eye gaze, gesture, speech and dialogue act. The current study used the data from an extensive study of human face-to-face multimodal communication for the development of a multimodal ECA, and tested to what extent multimodal behavior influenced the human-computer interaction. Results from a persona assessment questionnaire showed the presence of facial expressions, gesture and intonation had a positive effect on five assessment scales. Eye tracking results showed facial expressions played a primarily pragmatic role, whereas intonation played a primarily semantic role. Gestures played a pragmatic or semantic role, dependent on their level of specificity. These findings shed light on multimodal behavior within and between human and digital dialogue partners. Keywords: embodied conversational agents, multimodal communication, avatars. Introduction Embodied conversational agents (ECA’s) are animated characters that emulate human multimodal communication. Such communication involves both linguistic (e.g., speech intonation, discourse structure) and paralinguistic (e.g., facial expression, hand gestures, eye gaze) signals. There has been a baby boom of these agents both in the virtual world as well as in the literature. Offspring are produced in departments of psychology, artificial intelligence, computer science and education: human-like (Graesser, et al., 2004) and cartoon-like (Cassell et al., 2008); with anticipated careers in the military (Johnson et al., 2004), education (Graesser et al., 2004; McNamara, Levinstein, & Boonthum, 2004) or speech pathology (Massaro, 2006). In all these domains, the role of the agent is to improve the communication of a given message. The genealogy of these agents goes back many centuries. One of the first proposals for embodied interfaces came from Heron of Alexandria’s (62 AD) who described an ‘automatic’ puppet-theatre operated by weights. In the 18 th century Friedrich von Knaus developed the first talking heads, while C. G. Kratzenstein synthesized vowel sounds using a set of acoustic resonators and vibrating reeds, and Von Kemplen developed the first speaking machine that produced sound combinations. In the first part of the 20 th century Jacques Vaucanson developed mechanical animated objects like a flute-playing boy and a duck that could flap its wings, eat, and digest grain. Obviously, lots of progress has been made, with today’s ECA’s being far more human-like than the older systems. At the same time, today’s advanced embodied interfaces, like their predecessors, have a limited use of the multimodal aspects of communication. Even though some of today’s systems have excellent speech interfaces (Pellom, Ward & Pradhan, 2000), conversational skills (Graesser et al., 2004), gestural movements (Cassell, Kopp, Tepper, Ferriman, & Striegnitz, 2007), or mouth movements (Massaro, 2006), they typically excel on just one aspect of multimodal communication. And even when human-like facial expressions and gestures are integrated in ECA’s, they are carefully guided by literature but otherwise intuitive (Baylor & Kim, 2005), or come from actors acting out different modalities which are then transmitted to the agent, for instance by using body suits. The reason a full implementation of linguistic and paralinguistic channels of communication naturally used by humans has not been realized so far is that relatively little is known about how these channels combine within and across speakers in human-human communication. Although evidence has been collected on the alignment of pairs of modalities (e.g. Gullberg & Homqvist, 2006; Thompson & Massaro, 1996), few studies have investigated the associations between more than two modalities at a time. In addition, it is questionable whether the development of, and research in, human-like ECA’s is valuable in the first place. After all, the argument can be made that humans project human characteristics on objects that do not even slightly resemble humans (Reeves & Nass, 2003). Moreover, aiming for humanlike ECA’s increases the chances of entering the uncanny valley (Mori, 2005), with users liking the humanlike avatar less than cartoonish avatars. At the same time, there is evidence that humanlike characteristics like stereotypes are applied to humanlike but not to cartoonlike agents (Louwerse, Graesser, Lu, & Mitchell, 2005).