Language Mediated Visual Search: The Role of Display Preview

Language Mediated Visual Search: The Role of Display Preview John L. Jones (jones@psy.fsu.edu) Michael P. Kaschak (kaschak@psy.fsu.edu) Walter R. Boot (boot@psy.fsu.edu) Department of Psychology, Florida State University 1107 W. Call St, Tallahassee FL 32306 USA Abstract Visual search efficiency is increased when the search target’s identity is revealed incrementally via language while the display is in view. One view posits that search efficiency is increased because language enhances perceptual processing. We examined an alternative view that increased efficiency is due to delaying the onset of target-seeking eye movements, allowing a preview of the search array. Two eye-tracking experiments tested these alternatives. Observed patterns of eye movements indicated that increased efficiency with concurrent language was not likely due to linguistic enhancement of perceptual processes. Keywords: efficiency. language comprehension; visual search; Introduction In visual search for a target defined by one feature (feature search) reaction times are independent of the number of distractors. In contrast, when the target is defined by two features (conjunction search), reaction times increase with the number of distractors. This effect is mitigated when different kinds of information are made available before search begins. For example, knowledge of what color subset the target will appear in (e.g., Egeth, Virzi, & Garbart, 1984, Friedman-Hill & Wolfe, 1995), a preview of non-target locations (e.g., Watson & Humphreys, 1997, 2002), and a preview of locations that potentially do contain the target (e.g., Hannus et al., 2006; Olds, Cowan, & Jolicœur, 2000a, 2000b; Rutishauser & Koch, 2007) all reduce the effect of an increasing number of distractors (set size). When search time is independent of set size, the search process is said to be efficient. Spivey, Tyler, Eberhard, & Tanenhaus (2001) provided evidence that visual search efficiency can be mediated linguistically. Participants were presented a classic conjunction search task in which they had to indicate the presence or absence of a target defined by both color and orientation. When the target’s identity was revealed incrementally through speech (e.g., ‘red horizontal’) while the search display was in view, search was as efficient as in feature search (as indexed by near zero response time by set size functions). This benefit to search is termed linguistic assistance. To account for this increase in efficiency, it was proposed that, due to the incremental nature of language, the perception of the auditory stimulus (e.g., the word ‘red’) together with perception of features of the visual stimulus (the color red) enhanced the salience of the subset of items matched to this feature in the cognitive representation of the search display (i.e., the salience map; see Reali et al., 2006 for an elaboration of this point). Spivey et al.’s interpretation of this interaction between language and perceptual processing was that participants executed two nested feature searches instead of a single conjunction search. The concept of nested feature search is not well specified in the visual search literature. To our knowledge the concept was introduced by Spivey et al. (2001) to explain their results. Based on their description, two nested feature searches can be thought of as a sequence of pop-out searches. First, the (spatially noncontiguous) subset of items matching the color named in the linguistic cue is isolated attentionally due to the increased salience of these items. Then, the oddly oriented target pops-out from within this set. For example, as soon as participants hear the word ‘red’ (or enough of it to distinguish it from the word from ’green’), the salience of the red subset of items in the display is boosted relative to the salience of the green items. This effectively reduces the number of items to be searched by half and makes the presence of an item of differing orientation easily detectable in the set of red items. The critical idea here is that hearing a word that describes the target’s features automatically biases the perceptual system towards items that match that feature by means of enhancement in the salience map. Gibson, Eberhard, and Bryant (2005), however, demonstrated that the benefit of linguistic assistance reported by Spivey et al. (2001) is limited to slow speech rate (3.0 syllables/second) or to smaller set sizes with faster speech rate (4.8 syllables/second). Slower speech would provide participants with more viewing time of the search display before hearing the identity of the target. That is, more viewing of the search display provides additional information about potential target locations independently of the information conveyed by the speech cue. Evidence of linguistic assistance suggests that top-down processing can bias perceptual processes towards certain environmental features to enhance bottom-up information. Therefore, evidence that these effects are due to other (bottom-up) information acquired before search begins, would place important constraints on what influence top- down processing can have on bottom-up processes. The present study tests these accounts by examining eye movement data while participants performed conjunction visual searches. If the benefit of linguistic assistance reported by Spivey et al. (2001) is the result of nested feature searches it should be reflected in eye movements. The pattern of reaction time data and eye movement data