Eyebrow Raising, Discourse Structure, and Utterance Function in Face-to-face Dialogue

Eyebrow Raising, Discourse Structure, and Utterance Function in Face-to-face Dialogue Mar´ia L Flecha-Garc´ia (marisa@ling.ed.ac.uk) Linguistics and English Language, University of Edinburgh 14 Buccleuch Place, Edinburgh, EH8 9NL, UK found that, indeed, recurring gesture features revealed a dis- course organisation that correlated 100% with the hierarchi- cal structure of their segmentation. McClave (2000) looked at head movements in dialogue and, among other functions, she associated changes in head position with a switch in dis- course from indirect to direct speech and also with listing or presenting alternatives. In a study by Cassell, Nakano, Bick- more, Sidner, and Rich (2001), body posture shifts were more frequent when starting a new discourse segment which was often at the boundary between turns. From this, Cassell et al. (2001) concluded that posture shifts can signal boundaries of units. As for brow raises, Chovil (1991) observed, with an inductive approach, that in her recorded dialogues these sometimes seemed to mark the organizational structure of a dialogue, by marking the beginning, end, or continuation of a topic. The findings above are for English. Cav´e, Guaitella, and Santi (2002) found that rapid eyebrow raises could have a role in turn-taking by marking the start of a new speaking turn in French. In relation to utterance function, eyebrow raising has been traditionally associated with questioning. This func- tion was mentioned by Ekman (1979) in a publication that has been very influential in later research even though it was presented as a preliminary observation without empirical ev- idence. Chovil (1991) also observed this questioning func- tion in her recorded dialogues. More recently, Srinivasan and Massaro (2003) found that both eyebrow raising and head tilt- ing could be used, together with auditory cues, to distinguish echoic questions from statements in perception studies with synthetic stimuli. They reported, however, that participants relied most strongly on the auditory cues, even when the vi- sual cues were enhanced. In summary, observations in previous research suggest that body movement, including eyebrow raising, could be related to the organization of a discourse into segments. Also, eye- brow raises have been associated with the function of utter- ances, particularly with questioning. Thus eyebrow raises would appear to have conversational functions, but there is not strong empirical evidence. Research into when eyebrow raises occur in dialogue could have important implications for models of speech production and could provide crucial information for the design of embodied animated agents in multimodal dialogue systems. The current study investigated eyebrow raising as it oc- curred spontaneously in task-oriented dialogues in English. Considering the question of when we raise our eyebrows when we are engaged in conversation, brow raises were pre- dicted to occur more frequently at the start of new segments in the conversation. In particular, and using the structure coding scheme by Carletta et al. (1997), this hypothesis was stated as: Abstract Some studies have suggested a relationship between eyebrow raising and different aspects of the verbal message, but our knowledge about this link is still very limited. If we could es- tablish and characterise a relation between eyebrow raises and the linguistic signal we could better understand human multi- modal communication behaviour. This study investigated eye- brow raising in a corpus of task-oriented English dialogues in relation to discourse structure and utterance function. Based on previous observations of body movement, eyebrow raises were predicted to occur more frequently in utterances starting a new segment in the structure of the dialogue. It was also pre- dicted that they would be more frequent in instructions and in queries than in other types of utterance. Analyses using gener- alised linear models showed that eyebrow raises occurred more frequently in the initial utterance of high-level discourse seg- ments than in other parts of the dialogue, and more frequently also in instructions than in other types of utterance. The start of a lower level in the discourse structure was not associated with eyebrow raising, nor were utterances with a questioning function. These findings provide tentative evidence of a re- lation between eyebrow raising and aspects of the linguistic message which is important not only from a psycholinguistic point of view but also for practical applications in the design of multimodal dialogue systems. Keywords: Psycholinguistics; Multimodal communication. Introduction Like many of the subtleties of human communication, the use of the face is something we believe we understand but cannot yet describe. Human facial movements have attracted a great deal of research, but the information we have about the use of the face in multimodal communication is still very limited. Research on facial movements has been largely dominated by the study of the expression of emotion. By contrast, and leav- ing aside movements that are necessary for the articulation of speech, studies on facial movements in connection with spoken language have been scarce. Observations have, how- ever, been made which suggest possible conversational func- tions of eyebrow raises in particular, and in recent years a few studies have taken an empirical approach to this issue. The sketchy nature of the evidence for coordination is particularly surprising because other body movements have been shown to have a non-random relation with the speech they accom- pany. The study reported here investigated possible linguistic functions of eyebrow raising associated with discourse struc- ture and utterance function in face-to-face dialogue. There have been several studies suggesting a relationship between body movements and discourse structure. Kendon (1972, 1980) suggested that common discourse themes will produce gestures with recurring features. Based on this, Mc- Neill et al. (2001) studied possible cues for discourse struc- ture from manual gestures in videotaped conversations. Seg- menting the conversations in terms of discourse goals, they