OOSTING SALIENCY PREDICTION WITH FEATURE MAPS TRAINED ON I MAGE N ET

Recent results suggest that state-of-the-art saliency models perform far from optimal in predicting fixations. This lack in performance has been attributed to an inability to model the influence of high-level image features such as objects. Recent seminal advances in applying deep neural networks to tasks like object recognition suggests that they are able to capture this kind of structure. However, the enormous amount of training data necessary to train these networks makes them difficult to apply directly to saliency prediction. We present a novel way of reusing existing neural networks that have been pretrained on the task of object recognition in models of fixation prediction. Using the well-known network of Krizhevsky et al. (2012), we come up with a new saliency model that significantly outperforms all state-of-the-art models on the MIT Saliency Benchmark. The structure of this network allows new insights in the psychophysics of fixation selection and potentially their neural implementation. To train our network, we build on recent work on the modeling of saliency as point processes. By understanding how humans choose eye fixations, we can hope to understand and explain human behaviour in a number of vision-related tasks. For this reason human eye movements have been studied for more than 80 years (e. g. Buswell, 1935). During the last 20 years, many models have been developed trying to explain fixations in terms of so called “saliency maps”. Recently, it has been suggested to model saliency maps probabilistically using point processes (Barthelmé et al., 2013) and to evaluate them using log-likelihood (Kümmerer et al., 2014). This evaluation revealed that state-of-the-art models of saliency explain only one third of the explainable information in the spatial fixation structure (Kümmerer et al., 2014). Most of the existing models use low-level cues like edge-detectors and color filters (Itti et al., 1998) or local image statistics (Zhang et al., 2008; Bruce & Tsotsos, 2009). However, human fixations are largely clustered around objects (see Figure 1 for examples). This has led to some models trying to incorporate more high level features: Cerf et al. (2008) combined existing saliency map models with a face detector, while Judd et al. (2009) included detectors for faces, people, cars and horizon. Nevertheless, current saliency models mostly fail to capture these high-level influences which might be the main reason for the poor overall performance of state-of-the-art models. This analysis raises the question whether there are any computational systems capable of capturing such high-level influences. Independent of these developments, the last two years have seen the rise of deep neural networks to solve multifarious tasks like object detection, speech recognition or automatic translation. Provided with enough training data, deep neural networks show impressive results, often outperforming all competing methods. It has also been shown that deep convolutional networks that have been optimized for object classification can be used to predict neuron responses in higher brain areas of the visual system (Yamins et al., 2014; Razavian et al., 2014). Deep neural networks have also proven to generalize well over tasks (Donahue et al., 2013): a network trained for some task like object detection can often be easily retrained to achieve state-of-the-art performance in some other only loosely related task like scene recognition. Motivated by these developments, we here try to use pretrained deep neural networks to model fixation selection. The results of Yamins et al. (2014) connect neural network representations with IT 1 ar X iv :1 41 1. 10 45 v4 [ cs . C V ] 9 A pr 2 01 5 Accepted as a workshop contribution at ICLR 2015 Im ag e w ith fix at io ns