VISUAL SALIENCY DOES NOT ACCOUNT FOR EYE MOVEMENTS DURING VISUAL SEARCH IN REAL-WORLD SCENES

We tested the hypothesis that fixation locations during scene viewing are primarily determined by visual salience. Eye movements were collected from participants who viewed photographs of real-world scenes during an active search task. Visual salience as determined by a popular computational model did not predict region-to-region saccades or saccade sequences any better than did a random model. Consistent with other reports in the literature, intensity, contrast, and edge density differed at fixated scene regions compared to regions that were not fixated, but these fixated regions also differ in rated semantic informativeness. Therefore, any observed correlations between fixation locations and image statistics cannot be unambiguously attributed to these image statistics. We conclude that visual saliency does not account for eye movements during active search. The existing evidence is consistent with the hypothesis that cognitive factors play the dominant role in active gaze control. Elsevier AMS Ch25-I044980 Job code: EMAW 14-2-2007 1:11p.m. Page:539 Trimsize:165×240MM Basal Fonts:Times Margins:Top:4.6pc Gutter:4.6pc Font Size:10/12 Text Width:30pc Depth:43 Lines Ch. 25: Visual Saliency Does Not Account for Eye Movements During Visual Search 539 During real-world scene perception, we move our eyes about three times each second via very rapid eye movements (saccades) to reorient the high-resolving power of the fovea. Pattern information is acquired only during periods of relative gaze stability (fixations) due to a combination of central suppression and visual masking (Matin, 1974; Thiele, Henning, Buishik, & Hoffman, 2002; Volkman, 1986). Gaze control is the process of directing the eyes through a scene in real time in the service of ongoing perceptual, cognitive, and behavioral activity (Henderson, 2003; Henderson & Hollingworth, 1998, 1999). There are at least three reasons that the study of gaze control is important in real-world scene perception (Henderson, 2003; Henderson & Ferreira, 2004a). First, human vision is active, in the sense that fixation is directed toward task-relevant information as it is needed for ongoing visual and cognitive computations. Although this point seems obvious to eye movement researchers, it is often overlooked in the visual perception and visual cognition literatures. For example, much of the research on real-world scene perception has used tachistoscopic display methods in which eye movements are not possible (though see Underwood, this part; Gareze & Findlay, this part). While understanding what is initially apprehended from a scene is an important theoretical topic, it is not the whole story; vision naturally unfolds over time and multiple fixations. Any complete theory of visual cognition, therefore, requires understanding how ongoing visual and cognitive processes control the direction of the eyes in real time, and how vision and cognition are affected by where the eyes are pointed at any given moment in time. Second, eye movements provide a window into the operation of selective attention. Indeed, although internal (covert) attention and overt eye movements can be dissociated (Posner & Cohen, 1984), the strong natural relationship between covert and overt attention has recently led some investigators to suggest that studying covert visual attention independently of overt attention is misguided (Findlay, 2004; Findlay & Gilchrist, 2003). For example, as Findlay and Gilchrist (2003) have noted, much of the research in the visual search literature has proceeded as though viewers steadfastly maintain fixation during search, allocating attention only via an internal mechanism. However, visual search is virtually always accompanied by saccadic eye movements (e.g., see the chapters by Hooge, Vlaskamp, & Over, this part; Shen & Reingold, this part). In fact, studies of visual search that employ eye tracking often result in different conclusions than do studies that assume the eyes remain still. As a case in point, eye movement records reveal a much richer role for memory in the selection of information for viewing (e.g. McCarley, Wang, Kramer, Irwin, & Peterson, 2003; Peterson, Kramer, Wang, Irwin, & McCarley, 2001) than research that uses more traditional measures such as reaction time (e.g. Horowitz & Wolfe, 1998). To obtain a complete understanding of the role of memory and attention in visual cognition, it is necessary to understand eye movements. Third, because gaze is typically directed at the current focus of analysis (see Irwin, 2004, for some caveats), eye movements provide an unobtrusive, sensitive, real-time behavioral index of ongoing visual and cognitive processing. This fact has led to enormous insights into perceptual and linguistic processing in reading (Liversedge & Findlay, 2000; Rayner, 1998; Sereno & Rayner, 2003), but eye movements are only now becoming Elsevier AMS Ch25-I044980 Job code: EMAW 14-2-2007 1:11p.m. Page:540 Trimsize:165×240MM Basal Fonts:Times Margins:Top:4.6pc Gutter:4.6pc Font Size:10/12 Text Width:30pc Depth:43 Lines 540 J. M. Henderson et al. a similarly important tool in the study of visual cognition generally and scene perception in particular. 1. Fixation placement during scene viewing A fundamental goal in the study of gaze control during scene viewing is to understand the factors that determine where fixation will be placed. Two general hypotheses have been advanced to explain fixation locations in scenes. According to what we will call the visual saliency hypothesis, fixation sites are selected based on image properties generated in a bottom-up manner from the current scene. On this hypothesis, gaze control is, to a large degree, a reaction to the visual properties of the stimulus confronting the viewer. In contrast, according to what we will call the cognitive control hypothesis, fixation sites are selected based on the needs of the cognitive system in relation to the current task. On this hypothesis, eye movements are primarily controlled by task goals interacting with a semantic interpretation of the scene and memory for similar viewing episodes (Hayhoe & Ballard, 2005; Henderson & Ferreira, 2004a). On the cognitive control hypothesis, the visual stimulus is, of course, still relevant: The eyes are typically directed to objects and features rather than to uniform scene areas (Henderson & Hollingworth, 1999); however, the relevance of a particular object or feature in the stimulus is determined by cognitive information-gathering needs rather than inherent visual salience. The visual saliency hypothesis has generated a good deal of interest over the past several years, and in many ways has become the dominant view in the computational vision literature. This hypothesis has received primary support from two lines of investigation. First, computational models have been developed that use known properties of the visual system to generate a saliency map or landscape of visual salience across an image (Itti & Koch, 2000, 2001; Koch & Ullman, 1985). In these models, the visual properties present in an image give rise to a 2D map that explicitly marks regions that are different from their surround on image dimensions such as color, intensity, contrast, and edge orientation (Itti & Koch, 2000; Koch & Ullman, 1985; Parkhurst, Law, & Niebur, 2002; Torralba, 2003), contour junctions, termination of edges, stereo disparity, and shading (Koch & Ullman, 1985), and dynamic factors such as motion (Koch & Ullman, 1985; Rosenholtz, 1999). The maps are generated for each image dimension over multiple spatial scales and are then combined to create a single saliency map. Regions that are uniform along some image dimension are considered uninformative, whereas those that differ from neighboring regions across spatial scales are taken to be potentially informative and worthy of fixation. The visual saliency map approach serves an important heuristic function in the study of gaze control because it provides an explicit model that generates precise quantitative predictions about fixation locations and their sequences, and these predictions have been found to correlate with observed human fixations under some conditions (e.g., Parkhurst et al., 2002). Second, using a scene statistics approach, local scene patches surrounding fixation points have been analyzed to determine whether fixated regions differ in some image Elsevier AMS Ch25-I044980 Job code: EMAW 14-2-2007 1:11p.m. Page:541 Trimsize:165×240MM Basal Fonts:Times Margins:Top:4.6pc Gutter:4.6pc Font Size:10/12 Text Width:30pc Depth:43 Lines Ch. 25: Visual Saliency Does Not Account for Eye Movements During Visual Search 541 properties from regions that are not fixated. For example, high spatial frequency content and edge density have been found to be somewhat greater at fixated than non-fixated locations (Mannan, Ruddock, & Wooding, 1996, 1997b). Furthermore, local contrast (the standard deviation of intensity in a patch) is higher and two-point intensity correlation (intensity of the fixated point and nearby points) is lower for fixated scene patches than control patches (Krieger, Rentschler, Hauske, Schill, & Zetzsche, 2000; Parkhurst & Neibur, 2003; Reinagel & Zador, 1999). Modulating the evidence supporting the visual saliency hypothesis, recent evidence suggests that fixation sites are tied less strongly to saliency when meaningful scenes are viewed during active viewing tasks (Land & Hayhoe, 2001; Turano, Geruschat, & Baker 2003). According to one hypothesis, the modulation of visual salience by knowledge-driven control may increase over time within a scene-viewing episode as more knowledge is acquired about the identities and meanings of previously fixated objects and their relationships to each other and to the scene (Henderson, Weeks, & Hollingworth, 1999). However, even the very first saccade in a scene can often take the eyes in the likely direction of a search target, whether or not the targ