Grounding Dialogue: Eye Movements Reveal the Coordination of Attention During Conversation and the Effects of Common Ground

Grounding Dialogue: Eye Movements Reveal the Coordination of Attention During Conversation and the Effects of Common Ground Daniel C. Richardson (dcr@ucsc.edu) Department of Psychology, University of Caifornia, Santa Cruz 273 Social Sciences 2, Santa Cruz, CA 95064 Rick Dale (rad28@cornell.edu) Department of Psychology, Cornell University, 211 Uris Hall, Ithaca, NY 14853 Abstract Monologues and visual common ground Richardson and Dale (2005) focused on cases in which conversational partners are looking at a visual scene that is the topic of the discussion. The situation is analogous to two people discussing a diagram on a whiteboard, figuring out a route on a map, or talking during a movie. In the first study, the speech and eye movements of one set of participants were recorded as they looked at pictures of six cast members of a TV sitcom (either ‘Friends’ or ‘The Simpsons’). They spoke spontaneously about their favourite episode and characters. One-minute segments were chosen and then played back unedited to a separate set of participants. The listeners looked at the same visual display of the cast members, and their eye movements were recorded as they listened to the segments of speech. They then answered a series of comprehension questions. Listener and speaker eye movements were coded as to which of the six cast members was being fixated during every 33ms time slice. Cross-recurrence analysis (Zbilut, Giuliani, & Webber, 1998) quantified the degree to which speaker and listener eye positions overlapped at successive time lags (see below for a brief explanation). This speaker X listener distribution of fixations was compared to a speaker X randomized-listener distribution, produced by shuffling the temporal order of each listener’s eye movement sequence and then calculating the cross recurrence with the speakers they had heard. This randomized series serves as a baseline of looking ‘at chance’ at any given point in time, but with the same overall distribution of looks to each picture as the real listeners (see Figure 1). From the moment a speaker looks at a picture, and for the following six seconds, a listener was more likely than chance to be looking at that same picture. The breadth of this timeframe suggests that speakers and listeners may keep track of a subset of the depicted people who are relevant moment-by-moment (Brown-Schmidt et al., 2004). The overlap between speaker and listener eye movements peaked at about 2000ms. In other words, two seconds after the speaker looked at a cast member, the listener was most likely to be looking at the same cast member. The timing of this peak roughly corresponds to results in the speech production and comprehension literatures. Speakers will fixate objects 800-1000ms (Griffin & Bock, 2000) before naming them, and listeners will typically take 500-1000ms to fixate an object from the word onset (Allopenna et al., 1998). The coupling between speaker and listener eye movements was pervasive, suggesting that planning diverse When two people discuss something in front of them, what is the relationship between their eye movements? In Richardson and Dale’s (2005) study, participants talked extemporaneously about a TV show while viewing pictures of its cast members. Later, other participants listened to these monologues while viewing the same screen. Cross-recurrence analysis revealed that the coupling between speaker and listener eye- movements predicted how well the listener understood what was said. In our current research, we extended these findings by studying the eye movements of two conversants engaged in a live, spontaneous dialog. The participants talked to each other over the telephone while viewing identical visual displays, and we tracked the eye movements of both conversants simultaneously. In our first study, we found the conversants’ eye movements were coupled across several seconds. In the second study we showed that this coupling increases if participants both heard the same background information prior to their conversation. Our results highlight the central role of grounding utterances in the visual context. Introduction Coordinating attention across a visual common ground is essential for successful communication (Clark, 1996; Clark & Brennan, 1991; Schober, 1993). In collaborative tasks, conversants readily use gestures, actions and pointing to manipulate each other’s attention (Bangerter, 2004; Clark, 2003; Clark & Krych, 2004), and the ability to manipulate joint attention is thought to emerge prelinguistically (Baldwin, 1995). A burgeoning research area has demonstrated that eye movements are tightly linked to the time course of language comprehension (e.g., Allopenna, Magnuson, & Tanenhaus, 1998; Brown-Schmidt, Campana, & Tanenhaus, 2004; Hanna & Tanenhaus, 2004; Hanna, Tanenhaus, & Trueswell, 2003; Henderson & Ferreira, 2004; Kamide, Altmann, & Haywood, 2003; Matlock & Richardson, in press; Tanenhaus, Spivey Knowlton, Eberhard, & Sedivy, 1995) and language production (Griffin & Bock, 2000; Meyer, Sleiderink, & Levelt, 1998). In the current studies, we used eye movements as a fine-grained index of how two conversants deployed their attention within a visual ‘common ground’. This allowed us to investigate the temporal coupling between conversants’ eye movements and to examine how this coupling relates to communication.