Evaluating Visual Saliency Algorithms: Past, Present and Future

A salient image region is defined as an image part that is clearly different from its surround. This difference is measured in terms of a number of attributes, namely, contrast, brightness and orientation. By measuring these attributes, visual saliency algorithms aim to predict the regions in an image that would attract our attention under free viewing conditions. As the number of saliency models has increased significantly in the past two decades, one is faced with the challenge of finding a metric that can be used to objectively quantify the performance of different saliency algorithms. To address this issue in this article, first, the state of the art of saliency models is revisited. Second, the major challenges associated with the evaluation of saliency models are discussed. Third, ten frequently used evaluation metrics are examined and their results are discussed for ten latest state-of-the-art saliency models. For the analysis, a comprehensive open source fixations database has been quantitatively examined. c © 2015 Society for Imaging Science and Technology. [DOI: 10.2352/J.ImagingSci. Technol.2015. 59.5. 050501] INTRODUCTION Our visual system is selective, i.e., we concentrate on certain aspects of a scene while neglecting other things. This is evident from studies on change blindness,1–3 which show that large changes can be made in a visual scene that can remain unnoticed. The reason why our visual system is selective is because our brains do not process all the visual information in a scene. In fact, while the optic nerve receives information at the rate of approximately 3× 106 bits/s, the brain processes less than 104 bits/s of this information.4 In other words, the brain uses a tiny fraction (<1%) of the collected information to build a representation of the scene, a representation that is good enough to perform a number of complex activities in the environment such as walking, aiming at objects and detecting objects. Based on this, we can ask what mechanisms are responsible for building this representation of the scene. In the literature, two main attention mechanisms are discussed: top-down and bottom-up.5–11 Top-down is voluntary, goal-driven and slow, i.e., typically in the range between 100 ms and several seconds.9 It is assumed that the top-down attention is closely linked with cognitive aspects such as memory, thought and reasoning. For example, by employing top-down mechanisms, we can attend to a Received Oct. 23, 2013; accepted for publication June 1, 2015; published online Aug. 4, 2015. Associate Editor: Susan Farnand. 1062-3701/2015/59(5)/050501/17/$25.00 person sitting next to us in a busy restaurant and neglect other people and visual information in the background. In contrast, bottom-up attention (also known as visual saliency) is associatedwith attributes of a scene that draw our attention to a particular location. These attributes include motion, contrast, orientation, brightness and color.12 Bottom-up mechanisms are involuntary, and faster than top-down.9 For instance, flickering lights, a yellow target among green objects, and a horizontal target among vertical objects are some stimuli that would automatically capture our attention in the environment. Studies13,14 show that in search tasks, such as looking for a target object among distractors, both bottom-up and top-down mechanisms work together to guide our attention. While bottom-up attention is based on elementary attributes of a scene, top-down is quite complex and strongly influenced by task demands.15 For example, the studies (Refs. 16, 17) suggest that for tasks such as picking up and placing objects by hand in the environment, attention is mainly driven by top-down mechanisms. In the past two decades, modeling of visual saliency has generated a lot of interest in the research community. In addition to contributing towards the understanding of human vision, it has also paved the way for a number of computer vision applications. These applications include target detection,18 image and video compression,19–22 image segmentation,23 context aware image resizing,24 robot localization,25,26 image retrieval,27 image and video quality assessment,28,29 dynamic lighting,30 advertisement,31 artistic image rendering32 and human–robot interaction.33,34 To evaluate the performance of visual saliency algorithms, the two-dimensional saliency maps are compared with the image regions that attract observers’ attention.35–37 This is done by displaying to the observers a set of images and using an eye tracker to record their eye fixations. Further, it is thought that a higher number of fixations correspond to salient image regions. The recorded fixations are thus compared with the associated visual saliency maps in a pairwise manner.38–40 Unfortunately, studies32,41,42 have shown that while viewing images, observers tend to fixate on the center of the image more than the peripheral regions. This effect is known as center bias and is well documented in vision studies.41,43 The presence of center bias in fixations makes it difficult to evaluate the correspondence between the fixated regions and the salient image regions. This can be explained by the fact that in a comprehensive eye tracking study by Judd et al.,32 it was shown that a dummy J. Imaging Sci. Technol. 050501-1 Sept. -Oct. 2015 Sharma: Evaluating visual saliency algorithms: past, present and future classifier defined by a Gaussian blob at the center of the image was better at predicting the eye fixations than any of the visual saliency models.35,44,45 In light of these results, one can ask what factors influence the performance of saliency algorithms, and which metric is a good candidate for comparing the different saliency models. This article is an attempt to address these issues. The rest of the article is organized as follows. In the second section, the literature on visual saliency algorithms is reviewed. Next, in the third section, the different metrics used for judging the performance of saliency models are discussed. Finally, in the fourth section, the results obtained for the evaluation metrics are examined. STATE OF THE ART INMODELING VISUAL ATTENTION In this section, the computer models for predicting eye fixations in still images are discussed. In the study by Borji et al.,46 the authors state that the visual attention models in the literature can be divided into eight classes: Bayesian, cognitive, decision theoretic, graphical, information theoretic, pattern classification, spectral analysis, and others. The classification made by Borji et al.46 has been updated with the latest saliency models. The different classes and the attention models associated with these classes are shown in Table I. In Bayesian models, prior knowledge about the scene, and sensory information such as target features are employed to calculate salient image regions. The objective is to learn from past search experiences in similar scenes, and use those strategies that lead to a successful search (of the target). For instance, models such as those of Torralba,47 Olivia et al.39 and Zhang et al.48 fall in this category. Cognitive models are the ones that are strongly based on psychological and neuro-physiological findings from experiments. For instance, experiments have shown that difference of Gaussians (DOG) is a good approximation of how the receptive fields (i.e., basic units of our visual system) extract information from a scene before sending it to the brain.49 This finding is the basis for many cognitive saliency models such as those of Itti et al.,44Walther,50Walther et al.,51 Frintrop52 and Borji and Itti.53 Othermodels in this category include those ofMeur et al.,54 Rajashekar et al.,55 Cerf et al.,36 Erdem and Erdem24 and Alsam et al.56,57 We can see that a majority of saliency models belong to this category. Decision theoretic models are based on the concept of identifying the optimal factors based on how people make decisions. Saliency is defined in terms of discrimination analysis, where salient features are those that best distinguish the target objects from all other visual classes of interest. For instance, models such as those of Gao and Vasconcelos,58 Gao et al.,59 Li et al.60 and Wang et al.61 are classified under this category. A graphical model is a probabilistic model which represents a set of calculated image features as pairs connected by links, where the interconnected features are vertexes, and the links connecting some pairs of vertexes Table I. The various visual attention models and their categories according to the study by Borji et al.46 The classification made by Borji et al.46 has been updated with the latest