Multimodal Communication in Face-to-Face Computer-Mediated Conversations

Multimodal Communication in Face-to-Face Computer-Mediated Conversations Max M. Louwerse (mlouwerse@ memphis.edu) a Nick Benesh (nbenesh@memphis.edu) a Mohammed E. Hoque (mhoque@memphis.edu) b Patrick Jeuniaux (pjeuniau@memphis.edu) a Gwyneth Lewis (glewis1@ memphis.edu) a Jie Wu (jwu4@ memphis.edu) c Megan Zirnstein (mzirnstn@ memphis.edu) a Department of Psychology / Institute for Intelligent Systems a Department of Electrical and Computer Engineering / Institute for Intelligent Systems b Department of Computer Science / Institute for Intelligent Systems c Memphis, TN 38152 USA Abstract Multimodal communication involves multiple communicative channels including speech, facial movement, and gesture. Relatively few studies on how various communicative modalities are aligned in natural, face-to-face communication exist. As part of a larger project, the current study investigates how discourse structure, speech features, eye gaze, and facial movements interrelate during a map coordination task. The study thereby sheds light on multimodal communication in humans and gives guidelines for the development of embodied conversational agents. Keywords: multimodal communication; discourse structure; dialogue; embodied conversational agents Introduction Multimodal communication is comprised of various modalities such as speech, facial movement, and gesture. Avoiding gestures while talking on the phone is difficult, not looking at someone in a face-to-face setting is challenging, , and listening to someone in a face-to-face setting without acknowledging information with the occasional mhmm’s and head nods, again, is a challenge. Despite the deceptively simple appearance of these communicative tools, little is understood regarding their interaction and alignment. Modalities such as speech, discourse, facial movements, eye gaze, and gesture seem to be intrinsically related. At the same time, little is known about how they interrelate and how they are aligned. Knowing the nature of the interaction of modalities and their alignment can shed light on various areas of cognitive science. From a psychological perspective, an understanding of the interplay of modalities can help us understand language and communication (Clark, 1996). Limited experimental research is available that can help determine whether modalities can be substituted or whether they are complementary (cf. Doherty-Sneddon, et al., 1997). From an educational perspective, an understanding of modalities can help answer questions regarding student motivation, interest, and confusion, as well as how instructors and tutors can monitor and respond to these cognitive states (Kort, Reilly & Picard, 2001). But with little information available concerning the conditions under which students use facial movements or eye gaze, tapping into students’ cognitive states is difficult (cf. Graesser, et al., in press). From a computational perspective, an understanding of the interplay between modalities can help in the development of animated conversational agents (Louwerse, Graesser, Lu & Mitchell, 2005). These agents maximize the availability of both linguistic (semantics, syntax) and paralinguistic (pragmatic, sociological) features (Cassell & Thorisson, 1999; Massaro & Cohen, 1994; Picard, 1997). But without experimental data on multimodal communication, the guidelines for implementing human-like multimodal behavior in agents are missing (cf. Cassell, et al., 1994). The current paper presents some initial results of an extensive data collection study of 256 conversations from 64 participants, all native speakers of English, and totaling 35 hours. Conversations were monitored for dialogue acts, speech, facial movements, gesture, eye gaze, and route drawing accuracy. The data from this study will shed light on human multimodal communication and will provide guidelines for the development of natural, embodied conversational agents. Map Task Dialogues Though there is considerable amount of research on multimodal communication (Argyle & Cook, 1976; Doherty-Sneddon, et al. 1997; Ekman, 1979; Goldin- Meadow, 2003; Louwerse & Bangerter, 2005; McNeill, 1992), this research can be characterized by the fact that 1) pairs of modalities are considered, so that it remains unclear whether multiple modalities are mutually substitutable; 2) language situations are highly diverse, making it difficult to interpret why certain multimodal behavior occurs; and 3) dialogue is unpredictable, making it hard to model when modalities behave in certain ways. In the current research project on multimodal communication in humans and agents (Louwerse, et al., 2004), we are investigating the interaction between dialogue act, speech, eye gaze, facial movements, gesture, and map drawing. The project aims to determine how these