Eye can't ignore what you're saying: Varying the reliability of gaze and language

Eye can’t ignore what you’re saying: Varying the reliability of gaze and language Ross G. Macdonald (rgmacdonald@dundee.ac.uk) School of Psychology, University of Dundee, Nethergate, Dundee, DD1 4HN, UK Benjamin W. Tatler (b.w.tatler@activevisionlab.org) School of Psychology, University of Dundee, Nethergate, Dundee, DD1 4HN, UK detect, localise and identify the target when the cue was directed to the invalid location. A later study (Ricciardelli, Bricolo, Aglioti, & Chelazzi, 2002) found that these invalid gaze cues also lead to eye movements in the incorrect direction. The above experiments provide strong evidence for a reflexive shift in attention caused by gaze cues. The automatic capture of attention, however, cannot fully account for the way we respond to gaze cue stimuli, as these responses have been shown to vary with perceived cue- reliability. When gaze cue validity is reduced to 20%, the reflexive effect is still present, but only at short SOAs (~100ms) (Kuhn & Kingstone, 2009). Hill et al. (2005) directly compared responses to different gaze cue reliabilities. Half of their participants took part in a version of the gaze cueing paradigm in which all gaze cues were either valid or invalid. The remaining participants took part in a task in which 50% of gaze cues were valid and the rest invalid. The invalid trials had a detrimental effect on response time for participants in the former task at SOAs up to 150ms only, however in the latter paradigm the detrimental effect was apparent up to 750ms. The authors argue that this is evidence for two streams of attentional control when viewing a gaze cue. Initially there is an automatic orienting effect (present in both tasks) and then a slower, top-down, selective effect (present when cues were 100% invalid). Whether both, one or neither of these attentional effects is unique to social cues is unclear (for discussion see Birmingham & Kingstone, 2009), however it is clear that the perceived reliability of a gaze cue modulates our response. Language has also been shown to modulate the utilisation of gaze cues. Linguistic research using the visual-world paradigm has shown an intimate link between the words people hear and where they look; people will not only look to areas referenced in language, but also make anticipatory looks to items relating to sentences they are hearing (Altmann & Kamide, 1999). Research looking specifically at the use of gaze cues in a real-world interaction (Hanna & Brennan, 2007) found that gaze cues facilitate communication; listeners in a target selection task used gaze cues to identify targets before linguistic disambiguation. In a more controlled experiment, Staudte and Crocker (2011) showed participants videos of a robot providing gaze cues to visible items while they heard incorrect sentences about these items. The sentences could always be corrected in two different and equally plausible ways. Participants mostly corrected the sentences in the way that made the gazed-at item the object of the sentence. These studies clearly show Abstract Gaze cues quickly orient attention, but language can affect the extent to which we follow these cues (Macdonald & Tatler, 2013). We investigated how reliability of language and gaze cues affect attention. Participants, provided with gaze and verbal cues, selected one of two potential targets and received immediate feedback. Different combinations of gaze and language reliabilities (50%, 80%, 100%) were used across nine sessions. The most reliable cue available informed participants’ decisions. Language was favoured when reliability was equal and cues incongruent. When language cues were 100% reliable, incongruent gaze cues had a larger detrimental effect on performance when they were 80% reliable compared to 50%. When gaze cues were 100% reliable, there was an overall detrimental effect of unreliable language, with performance slower when language was 50% reliable compared to 80%. We conclude that language cues are favoured and cause disruption when unreliable, even when superfluous to the task. Keywords: social attention; eye movements; joint attention; language; gaze cueing. Introduction During an everyday social interaction, we provide verbal and non-verbal cues to express our intentions, thoughts and emotions. In the research areas of social cognition and attention, cues provided by the gaze of another have been the focus of much empirical research. Gaze cues have been shown to orient attention and viewer gaze automatically, leading some to conclude that humans have evolved to reflexively orient attention in the direction of a gaze cue (Emery, 2000). However, there is evidence that the effects of gaze cues on attention and eye movements can be modulated by the perceived reliability of the cue (Hill et al., 2005). Furthermore, the verbal cues that often accompany gaze cues in the real world have been shown to affect the extent to which people seek and follow the gaze of another (Macdonald & Tatler, 2013). The present study aims to bring together these two factors to investigate how varying the reliability of both gaze cues and language cues interact to affect attention and behaviour. Friesen and Kingstone (1998) developed a Posner-type (1980) gaze-cueing paradigm to investigate the effect of distracting gaze cues on attention. The paradigm involved detecting, localising and identifying a target (a letter – “F” or “T”) on the left or right of a display screen. In the centre of the screen there was a simple drawing of a face looking left, right or centrally, which participants were informed did not predict target location. Participants were slower to