The Role of Scene Gist and Spatial Dependency among Objects in the Semantic Guidance of Attention

The Role of Scene Gist and Spatial Dependency among Objects in the Semantic Guidance of Attention Chia-Chien Wu (chiachie@cs.umb.edu) Hsueh-Cheng Wang (hchengwang@gmail.com) Marc Pomplun (marc@cs.umb.edu) Department of Computer Science, University of Massachusetts at Boston 100 Morrissey Boulevard, Boston, MA, 02125-3393, USA Abstract A previous study (Hwang et al., 2011) found evidence for semantic guidance of visual attention during the inspection of real-world scenes, i.e., an influence of semantic relationships among scene objects on overt shifts of attention. In particular, the results revealed an observer bias toward gaze transitions between semantically similar objects. However, these results are not necessarily indicative of semantic processing of individual objects but may be confounded by knowledge of the scene gist, which does not require object recognition (Torralba et al., 2006), or by known spatial dependency among objects (Oliva & Torralba, 2007). To examine the mechanisms underlying semantic guidance, in the present study, subjects were asked to view a series of displays with the scene gist removed and spatial dependency varied. Our results confirm the previous finding of semantic guidance and show that it is not entirely due to either the effect of scene gist or the spatial dependency among objects. Even without scene gist or spatial dependency, subjects still retrieved semantic information to guide their attention. This strategy may facilitate scene understanding and object memorization. Keywords: Attention, semantics, eye movements, visual guidance, real-world scenes. Introduction Real-world scenes contain rich information, which usually is not thoroughly processed during natural viewing. Therefore, the way in which the visual system deploys the limited attention resources is crucial for effective vision and has drawn huge interest over the last two decades. The guidance of attention based on the features of stimuli in the visual environment has been well investigated in both its bottom-up (Itti & Koch, 2001; Koch & Ullman, 1985) and top-down aspects (Hayhoe et al, 2003; Hwang, Higgins & Pomplun, 2009; Pomplun, 2006). Visual attention is not only affected by factors based on the overt visual appearance, but also by inherent factors, such as meaning and semantic relations among objects. Hwang, Wang and Pomplun (2011) found that during natural scene viewing, humans tend to bring their gaze to the objects that are semantically similar either to the currently fixated one or to the specified search target. This result, however, may have been confounded by the observers’ knowledge of the global scene context. That is, instead of considering the semantic relation between the currently fixated object and the objects located in the extrafoveal visual field, observers may simply use their knowledge about the scene type to decide where to look next. For example, if observers are aware that the viewed image is a kitchen, they may only attend the regions nearby the counter or sink, where most of the kitchenware is likely located. The ways in which people acquire such global contextual information is not well understood. Torralba, Oliva, Castelhano and Henderson (2006) found that observers could extract some global scene properties - referred to as scene gist - without recognizing individual objects and use this information to guide their attention and eye movements. Even when the global context, which usually comes from visual background information, is missing, it is still possible to learn some context of the scene. Chun (2000) showed that some contextual information can be learned merely by the typical arrangement of elements and affect the deployment of attention. Oliva and Torralba (2007) also found that spatial dependency among objects could provide different contextual information about a scene. For example, a chair may be expected to be located behind a table, or a fork may be expected to be next to a spoon. In summary, both the scene gist and the spatial dependency among scene objects may have caused a bias in observers’ gaze patterns that could explain the results of Hwang et al. (2011) without the need for semantic analysis of extrafoveal scene objects. If gist, object dependency, or both were entirely responsible for the effect observed in that study, the concept of semantic guidance would not be a new phenomenon but rather a bias introduced by already known factors. The aim of the present study was to discern the contributions of scene gist, object dependency, and semantic object analysis to semantic guidance in order to address this problem. To study the influence of spatial dependency among scene objects, we employed the LabelMe object annotated image data base (Russell, Torralba, Murphy, & Freeman, 2008) in which scene images were manually segmented into annotated objects by volunteers. In addition, the locations of objects are provided as coordinates of polygon corner and all objects are labeled with English words or phrases. It provides an excellent opportunity for not only segregating each object from its scene, but also shifting the object’s coordinates to any desired location in the image. One way to eliminate potential influence of scene gist on attentional guidance is to remove all background information and only keep segregated objects in the scene. We used the resulting images in an experimental condition