Speaker gaze affects utterance comprehension beyond visual attention shifts

Speaker gaze affects utterance comprehension beyond visual attention shifts Maria Staudte & Matthew W. Crocker Department of Computational Linguistics, Saarland University Saarbr¨ucken, Germany Alexis Heloir & Michael Kipp DFKI, Embodied Agents Research Group Saarbr¨ucken, Germany Abstract Previous research has shown that listeners follow speaker gaze to mentioned objects in a shared visual environment to ground referring expressions, both for human and robot speakers. What is less clear is whether listeners exploit speaker gaze to infer referential intentions (Staudte & Crocker, 2010), or whether the benefits of gaze can be more simply explained by (reflexive) gaze following (Friesen & Kingstone, 1998). To investigate this issue, we conducted two eye-tracking studies which directly contrast speech-aligned speaker gaze of a vir- tual agent with a non-gaze visual cue (arrow). Our findings show that speaker gaze is beneficial to listeners only when the order of gaze cues matched the order of mentioned objects in the utterance. Similarly timed arrow cues, however, benefit listeners regardless of the order in which they occur. These findings are consistent with the view that gaze is interpreted as reflecting the speaker’s referential intentions, while other vi- sual cues regarding mentioned objects can be exploited more flexibly and strategically. Keywords: gaze; comprehension; visual attention shift; ar- rows; referring expressions Introduction In face-to-face communication, the speaker’s gaze to objects in a shared scene provides the listener with a visual cue to the speaker’s focus of (visual) attention (Emery, 2000; Flom, Lee, & Muir, 2007). This potentially offers the listener valu- able information to ground and disambiguate referring ex- pressions, to hypothesize about the speaker’s communicative intentions and goals and, thus, to facilitate comprehension (Hanna & Brennan, 2007). It is an open question, however, whether this functionality of speaker gaze results simply from its established ability to drive listeners’ visual attention, as do other cues as well, or whether gaze uniquely expresses (ref- erential) intentions. More precisely, there are two levels on which a visual at- tention shift in response to a speaker’s gaze may affect utter- ance processing (Staudte & Crocker, 2010). On a perceptual level, gaze-following may be considered as (reflexive) visuo- spatial orienting which increases the visual saliency of the particular target object and/or location in focus (Driver et al., 1999; Friesen & Kingstone, 1998; Langton & Bruce, 1999). On a cognitive level, gaze may additionally be understood as a cue to the speaker’s referential intentions which elicits expectations about which referent would be mentioned next (Hanna & Brennan, 2007). Previously, these two levels have been identified as the Visual and the Intentional Account, re- spectively (Staudte & Crocker, 2010). Crucially, the issue whether gaze is processed on both levels – that is, whether the Intentional Account but not the Visual Account alone – of- fers a satisfying explanation of gaze effects on utterance com- prehension, is still under debate. However, recent evidence seems to converge in support of such an assumption (Bec- chio, Bertone, & Castiello, 2008; Meltzoff, Brooks, Shon, & Rao, 2010; Staudte & Crocker, 2010). Staudte and Crocker (2010), for instance, synchronized gaze movements of a robot (as one instance of an artificial agent) with its speech in a human-like manner. This was shown to be similarly useful for grounding and resolving spo- ken references as human gaze (Hanna & Brennan, 2007). Fur- ther, Staudte and Crocker (2010) have shown that the order of respective gaze and speech cues is important for efficient comprehension whereas the temporal alignment of those cues is not. That is, when referential gaze cues and the correspond- ing referring expressions occurred in a coherent linear order, utterance comprehension was facilitated. When this order was reversed, however, gaze did not only not help but instead even slowed comprehension. In contrast, whether the respec- tive gaze cues occurred one second or five seconds prior to the corresponding referring expression onsets, did not affect the facilitative or disruptive influence of gaze order on com- prehension time. Previous studies have typically manipulated only the va- lidity or credibility of such gaze cues and neglected a direct assessment of the question whether effects on utterance pro- cessing are due to shifts in visual attention per se, or whether speaker gaze specifically (as opposed to other exogenous or even endogenous visual cues, Posner, 1980) elicited those at- tention shifts. To further explore the hypothesis that gaze is indeed interpreted with respect to referential intentions, we adopt an improved experimental design from (Staudte & Crocker, 2010) with a virtual character replacing the robot. We then contrast the influence of gaze and arrows by replac- ing the gaze cue with an arrow cue, directly comparing the effects of two, possibly different types of visual cues. Specif- ically, we report supporting evidence from two studies that, firstly, replicate the results on the relevance of gaze cue or- der for comprehension (Experiment 1) and, secondly, show that other, purely visual cues such as arrows (Experiment 2), induce similar attention shifts as gaze but crucially lack an ef- fect of (inconsistent) order. This supports the hypothesis that gaze does, but arrows do not, elicit inferences about referen- tial intentions.