Effects of speaker gaze versus depicted actions on visual attention during sentence comprehension

Effects of speaker gaze versus depicted actions on visual attention during sentence comprehension Helene Kreysa (helene.kreysa@uni-jena.de) Department of General Psychology, Friedrich Schiller University of Jena, Germany Pia Knoeferle (knoeferl@cit-ec.uni-bielefeld.de) Eva M. Nunnemann (eva@nunnemann.com) Cognitive Interaction Technology Excellence Cluster Department of Linguistics Bielefeld University, Inspiration I, 33615, Bielefeld, Germany Abstract An eye-tracking study compared the effects of actions (depicted as tools between on-screen characters) with those of a speaker’s gaze and head shift between the same two characters. In previous research, each of these cues has rapidly influenced language comprehension on its own, but few studies have directly compared these two cues or, more generally, distinct non-linguistic cues in their effects on real- time sentence comprehension. We investigated how participants used action tools and speaker gaze separately and in combination for visually anticipating the upcoming mention of a sentence referent. We discuss implications for accounts of visually situated language comprehension. Keywords: eye tracking, spoken language comprehension, speaker gaze, depicted actions Introduction Recent years have seen numerous studies on how individual contextual cues affect the unfolding interpretation of spoken sentences (for recent reviews see Altmann, 2011; Huettig, Rommers, & Meyer, 2011; for theoretical accounts and computational models see Altmann & Kamide, 2009; Crocker, Knoeferle & Mayberry, 2010; Mayberry, Crocker, & Knoeferle, 2009; Knoeferle & Crocker, 2006, 2007). Among these cues are sentence-based ones such as case marking, verb meaning or temporal adverbs, but also extralinguistic cues such as contrast between objects, or information from action depictions. For instance, when a verb refers to an action, participants rapidly integrate the action (and its associated thematic role relations between two characters) and use it as a cue for anticipating upcoming role fillers when the sentence context is otherwise ambiguous regarding the referents’ thematic role relations (Knoeferle, Crocker, Scheepers, & Pickering, 2005). As already mentioned, actions are by no means the only cue that can rapidly affect real-time spoken sentence comprehension. A speaker's emotional expression can for example enhance a listener’s visual attention to valence- matching event photographs as they are identified by a spoken sentence (Carminati & Knoeferle, 2013). However, seeing an action does appear to be of substantial importance in informing comprehension, and comprehenders may prioritize actions even when they are very infrequent (e.g., Abashidze, Knoeferle, & Carminati, 2013). In that context, Abashidze and colleagues have argued that recently-seen actions may be prioritized because they can be verified and have actually happened, while the absence of a visible action leads to uncertainty as to whether that action is going to be performed (even if such a “future” action occurs overall very frequently in the experimental context). On the other hand, our everyday conversations do include communication about absent actions and events; even concrete verbs do not invariantly reference actions and their associated patients in the immediate environment. Especially when compared to other contextual cues, the precise importance of visible and/or depicted actions for situated language comprehension thus remains an open issue. Consider for instance, a speaker’s eye gaze as another contextual cue for comprehenders. We know that when speakers talk about nearby objects, they tend to inspect them just before mentioning them (e.g., Bock, Irwin, Davidson, & Levelt, 2003; Griffin, 2004; Griffin & Bock, 2000; Kuchinsky, Bock, & Irwin, 2011; Meyer & Lethaus, 2004; Meyer, Roelofs & Levelt, 2003). This close link between speech-related eye movements and reference has important implications for language comprehension. Crucially, a speaker's gaze can help listeners to visually anticipate the next-mentioned referent (Hanna & Brennan, 2008; Knoeferle & Kreysa, 2012; MacDonald & Tatler, 2013; Staudte & Crocker, 2011). In summary, numerous studies have shown that individual cues – such as actions or a speaker’s eye gaze – can permit comprehenders to rapidly anticipate relevant referents. By contrast, only few studies have asked how the type of contextual cue affects visual anticipation. Neider, Chen, Dickinson, Brennan, and Zelinsky (2010), for instance, examined how two partners locate a randomly-appearing sniper target in a semi-realistic city environment. The two partners communicated either by shared voice, by shared gaze, or they could exploit both of these information sources in their joint search. Both partners had to locate the sniper target and make a joint decision. In the shared gaze condition, one partner would see the other’s eye gaze in the form of a gaze cursor which was superimposed on the city scene. Partners took less time to find the target in the shared