How speech processing affects our attention to visually similar objects: Shape competitor effects and the visual world paradigm

How speech processing affects our attention to visually similar objects: Shape competitor effects and the visual world paradigm Falk Huettig (f.huettig@psych.york.ac.uk) M. Gareth Gaskell (g.gaskell@psych.york.ac.uk) Philip T. Quinlan (p.quinlan@psych.york.ac.uk) Department of Psychology, University of York York, YO10 5DD, United Kingdom Abstract the word refers to. Second, his study highlighted the influence of semantic relationships on language-mediated fixation behavior: participants are more likely to fixate a visual referent that has some semantic relationship with the word heard than a semantically unrelated visual referent (see Huettig & Altmann, 2004; Yee & Sedivy, 2001; for follow-up studies). The primary goal of research in the visual world paradigm following Cooper’s (1974) pioneering study has been to use eye movements as a tool to shed light on linguistic processing. For example, Allopenna, et al. (1998) asked participants to ‘Pick up the candy. Now put it …’ in the context of a visual display of objects including (among other things) a candy and a candle. They found evidence for a phonological competitor effect: eye movements to both the candy and the candle increased as the word ‘candy’ acoustically unfolded but that soon after its acoustic offset, looks to the candle decreased while looks to the candy continued to rise. The Allopenna et al. (1998) study provided evidence for a standard competitor effect as predicted by theories of auditory word recognition such as TRACE (e.g. McCleland & Elman, 1986). Importantly the study demonstrated this effect in real-time as the speech stream was unfolding acoustically (see also Dahan et al., 2001; for more evidence that the visual world paradigm provides fine-grained measures of lexical processing). However, far less attention has focused on examining the interaction of spoken language with directed attention and the visual properties of the presented objects. In this regard, Cooper (1974) also found that participants tended to fixate a picture of a snake when hearing the word ‘wormed’ (in the context ‘just as I had wormed my way on my stomach’). This finding (although not discussed by Cooper) suggests that there may also be a strong link between lexical processing and the visual properties of an object such as an object’s shape (although it cannot be ruled out that in Cooper’s experiment participants mistook the snake for a worm and therefore directed their attention to the picture of the snake when hearing ‘wormed’). The visual world paradigm and the access of lexical representations Importantly, Cooper’s study (1974) is indicative that similar processes to semantic priming are taking place when people map spoken words onto related visual objects. The semantic priming paradigm (Meyer & Schvanefeldt, 1971) has proven to be particular useful for It was investigated how spoken language is mapped onto the mental representations of objects in the visual field. Specifically, the visual world paradigm was used to test the hypothesis that during ‘passive’ listening tasks attention is directed more towards objects in the visual field that match the physical shape of the concept of the word concurrently heard than towards objects that do not match on physical shape. Participants listened to sentences containing certain critical target words of concepts with a typical shape (e.g. ‘snake’) while concurrently viewing a visual display of four objects. We found that participants tended to fixate conceptually unrelated objects with a similar physical shape (e.g. cable) as soon as information from the target word (e.g. ‘snake’) started to acoustically unfold. The results indicate that (contrary to some priming studies, e.g. Moss et al., 1997) shape information is accessed long before the offset of the spoken word. We discuss the findings with respect to the applicability of the visual world paradigm for the investigation of the access of lexical representations and theories of active vision. Humans ‘translate’ between spoken language and concurrent visual input in such a natural way that we are hardly ever consciously aware of the processes involved. Surprisingly, there has been little research that has attempted to explore explicitly the interaction of these processes. A notable exception has been the research within the ‘visual world paradigm’ (the measuring of eye movements around a visual scene or display of objects in response to concurrent speech: Cooper, 1974; Tanenhaus et al. 1995) using both linguistic and visual contexts. Cooper (1974) established that when participants were presented simultaneously with spoken language and a visual field containing referents of the spoken words, participants tended to spontaneously fixate the visual referents of the words currently being heard. For example, participants were more likely to fixate the picture of a snake when hearing part or the entire word ‘snake’ than to fixate pictures of unrelated control words. Cooper (1974) also found that participants were more likely to fixate pictures showing a lion, a zebra, or a snake when hearing the semantically related word ‘Africa’ than to fixate semantically unrelated control words. Cooper’s (1974) early study thus established two main findings: first, during the acoustic duration of a spoken word participants show a strong tendency to fixate objects that