# Deep Global-Local Gazing: Including Global Scene Properties in Local Saliency Computation

Authors: Samad Zabihi; Mehran Yazdi; Eghbal Mansoori
Journal: Mobile Information Systems
 (2022)

---

## Abstract

Visual saliency models imitate the attentive mechanism of the human visual system (HVS) to detect the objects that stand out from their neighbors in the scene. Some biological phenomena in HVS, such as contextual cueing effects, suggest that the contextual information of the whole scene does guide the attentive mechanism. The saliency value of each image patch is influenced by its visual (local) features as well as the contextual information of the whole scene. Modern saliency models are based on deep convolutional neural networks. Because the convolutional operators operate locally and use weight sharing, such networks inherently have difficulty capturing global and location-dependent features. In addition, these models calculate the saliency value pixel-wise using local features. Therefore, it is necessary to provide global features along with local features. In this regard, we propose two approaches for capturing the contextual information from the scene. In our first method, we introduce a shift-variant fully connected component to capture global and location-dependent information. Instead of using the native CNN of our base model, in our second method, we use a VGGNet to capture the global and context information of the scene. To show the effectiveness of our methods, we use them to extend the SAM-ResNet saliency model. To evaluate our proposed approaches, four challenging saliency benchmark datasets were used. The experimental results showed that our methods could outperform the existing state-of-the-art saliency prediction models.


---

## Body

## 1. Introduction

In looking at a scene, HVS tends to gaze at the salient regions of the scene and ignores less salient parts [1]. The perceptual and cognitive resources of humans are limited. This attentive mechanism leads humans to rapidly process visual information and assign these limited resources to the salient subsets or objects of scenes [2]. This ability of HVS has been studied by neuroscientists and computer vision researchers to develop models that emulate this attention mechanism. Saliency prediction models are helpful to figure out human attention mechanisms, and also predict where people focus when they look at images or watch videos [2, 3]. Visual saliency models are useful across domains such as advertising, robotics, auto-driving [4], defense, game, assistive systems, and human–computer interaction.In general terms, early saliency prediction models employed biologically motivated low-level features [5–8] driven from low-level stimuli, for example, color, intensity, orientation, and texture. Subsequent models incorporated semantic concepts such as face [9], text [10], and gaze direction [11]. However, these techniques are not able to generally incorporate high-level features (e.g., contextual information, center prior, and complex objects) and inherent correlation of various visual subsets in a scene (e.g., correlation of eyes, nose, ears, and mouth).Since 2014, new sort of visual saliency models based on deep neural networks (DNNs) has emerged. They achieved strong improvements over classic saliency models. The hierarchical deep structure of convolutional neural networks (CNNs) enables the salience models to capture some complex cues whereas pioneering saliency models were not able to learn from data. However, studies revealed that they continue to fall short in capturing some high-level features of the scene such as center prior and global context [2]. Some studies have tried to compensate for such deficiencies in the CNN structure and capture the global properties by incorporating center prior [12–15] or by the means of convolutional long short-term memory (LSTM) [14, 16].Here, we propose two methods that incorporate contextual cues, global properties, and location-dependent features into pixel-wise saliency prediction to compensate for some deficiencies in CNN-based saliency models. Our first approach employs the VGGNet structure to capture the global and contextual information of the scene and then incorporates this information into the pixel-wise saliency prediction. This model predicts the saliency value of each image patch, by taking into account not only the locally extracted features of that patch but also the global scene properties. In our second approach, we introduce a shift-variant fully connected component to combine locally extracted features and learn the location-dependent information that simple convolution layers are incapable of capturing.The remainder of the study is organized as follows: the next section discusses related saliency prediction models. Section3 discusses the facts and arguments that motivated us and supported our saliency modeling. Section 4 presents our proposed saliency models. Section 5 describes some popular evaluation metrics, evaluation baselines, and saliency datasets, and provides our implementation details. Section 6 reports the evaluation results of our proposed models over several saliency benchmark datasets. Finally, Section 7 presents evaluation results and in Section 8 we conclude.


## 2. Related Work

The saliency prediction spreads in several research areas: free-viewing gaze prediction [4, 9–12] which tries to model the human eye fixations under free-viewing conditions, egocentric gaze prediction [13–15] which aims to predict the human eye fixations during a specific task, salient object detection [17–20] that detects and segments the salient objects in scenes and co-saliency detection [21, 22] that detects the common salient objects from a group of images. These visual saliency models are useful in vision tasks such as object recognition [17], image/video retargeting [18], segmentation [19–22], visual tracking [23], and image compression [24]. As an instance, Wang et al. [21] proposed a method that uses object-level cues for unsupervised video object segmentation.


### 2.1. Classic Saliency Prediction Models

Pioneer saliency prediction methods were mostly inspired by psychological and psychophysical models of attention as studied in HVS and they mainly focused on extracting better-handcrafted features and using better learning methods. Many of these bottom-up saliency models were based on Treisman’s “the feature integration theory” [25], which proposed strategies for combining various kinds of visual features without any bias to find the salient subsets of the scene. In 1985, Koch and Ullman [26] were one of the first to use the feature integration theory to propose a feed-forward model for combing a set of maps of elementary cues like contrast, color, and motion to produce a map of saliency.In 1998, Itti et al. [5] proposed an approach based on the Koch and Ullman feed-forward model [26]. In their model, they computed multi-scale center-surround contrast maps of preattentive features and then integrated these contrast maps to predict the saliency map. Their work triggered a lot of interest in the visual salience community. Many saliency models such as adapted this center-surround structure in the spatial domain [27, 28]. Itti and Baldi [29] proposed a model based on Bayesian approaches. Some methods adopted an information-theoretic justification for attentive selection [30–32]. Harel et al. [8] proposed a saliency model based on graph theory. Hou and Zhang [33] calculated saliency from frequency analysis. Some traditional saliency models used machine learning algorithms [34–36]. Some of these models incorporate high-level features such as face or text to steer the top-down process, thus they may not be purely bottom-up [37]. While many models fall into the bottom-up saliency model category, these models fail to capture the factors that contribute to attentional selection.


### 2.2. Deep Saliency Prediction Models

Employing deep convolutional neural networks (CNNs) in the saliency prediction model has made some drastic improvements over well-established saliency benchmark datasets [2]. Since 2014, using DNNs for saliency prediction gained much attention. To compensate for the lack of sufficiently large fixation data, most of these DNN-based models use transfer learning by employing a pretrained model that was trained for similar/different visual tasks on large image datasets.One of the first saliency models that used DNN was proposed by Vig et al. [38]. Their model, ensemble of deep networks (eDN), generates a large number of richly-parameterized neuromorphic networks for the feature extraction phase. Then, extracted features are applied to a linear support vector machine (SVM) to predict the saliency value. In [13, 39], Kümmerer et al. introduced a deeper structure for the encoder. DeepGaze I [39] uses pretrained AlexNet and DeepGaze II [13] uses pretrained VGG-19 for extracting features from the input image. Huang et al. [40] proposed a deep CNN structure that integrates information at different image scales. They showed that adding multi-scale information improves the saliency prediction results. Kruthiventi et al. [12] introduced a fully CNN model with a new “location biased convolutional (LBC) layer” to learn “location specific patterns” such as the center bias. Jetley et al. [41] formulated saliency map prediction as a probability distribution prediction task and trained a model to learn this distribution. Liu and Han [16] introduced a saliency model with a convolutional long short-term model (LSTM) to learn the global context. Cornia et al. [14] proposed a new saliency prediction architecture that incorporates a convolutional LSTM network and a spatial attentive mechanism. In [42], a saliency model based on image segmentation was introduced that exploits the object information for the saliency prediction task. Wang et al. [3] proposed a video saliency model, called ACLNet that uses the CNN-LSTM network to predict visual attention over dynamic scenes. Wang et al. [43] proposed a model that incorporates multi-level saliency predictions within a single network to decrease redundancy. Some researches focus on decreasing the model complexity and inference time for real-time application [44].In summary, Table1 compares the main properties of some prominent saliency prediction models and our proposed models. In Table 1, NSS, KL-D, CC, MSE, and SIM stand for normalized scanpath saliency, Kullback–Leibler divergence, linear correlation coefficient, mean square error, and similarity respectively.Table 1
A comparison between the main properties of some prominent saliency models.
Model nameCenter biasTransfer learningCNNLoss functionLSTM-based SAM [14]Multiple-learned priors✓VGG, ResNetNSS, KL-D, CCDSCLRCN [16]—✓VGG, ResNetNSSDeepGaze II [13]Gaussian prior✓VGGLog-likelihoodML-Net [15]Single multiplicative map✓VGGNormalized MSEPDP [41]—✓VGGProbability distancesSalNet [45]—✓VGGEuclidean lossSALICON [40]—✓AlexNet, VGG, GoogLeNetKL-D, NSS, CC, SIMDeepFix [12]Handcrafted priors✓VGGEuclidean losseDN [38]—-1 to 3 layer networksEuclidean lossGLG-I (proposed model)Multiple-learned priors, fully connected component✓ResNetNSS, KL-D, CCGLG-II (proposed model)Multiple-learned priors✓ResNet, VGGNSS, KL-D, CC


### 2.3. Salient Object Detection

The goal of salient object detection is to detect the most salient objects of a scene. Zhang et al. [46] used the multistage refinement mechanism to propose augmenting feedforward neural networks for addressing feature resolution reduction in CNNs. Zhao et al. [47] proposed a CNNs-based architecture that uses contrast prior to enhance the depth of information for salient object detection. Zhang et al. [48] proposed a probabilistic RGB-D saliency detection model based on conditional variational autoencoders. Li et al. [49] proposed a model that uses a pixel-level fully convolutional stream and a segment-wise spatial pooling stream to overcome the problem of blurry saliency maps, especially near the boundary of salient objects. To better segment salient and preserve the salient edges, Wang et al. [50] also proposed a model with a salient edge detection module. In most of these methods, incorporation of global and local information is missing and the need to use an appropriate model of jointly considering this information is still a challenge.


## 2.1. Classic Saliency Prediction Models

Pioneer saliency prediction methods were mostly inspired by psychological and psychophysical models of attention as studied in HVS and they mainly focused on extracting better-handcrafted features and using better learning methods. Many of these bottom-up saliency models were based on Treisman’s “the feature integration theory” [25], which proposed strategies for combining various kinds of visual features without any bias to find the salient subsets of the scene. In 1985, Koch and Ullman [26] were one of the first to use the feature integration theory to propose a feed-forward model for combing a set of maps of elementary cues like contrast, color, and motion to produce a map of saliency.In 1998, Itti et al. [5] proposed an approach based on the Koch and Ullman feed-forward model [26]. In their model, they computed multi-scale center-surround contrast maps of preattentive features and then integrated these contrast maps to predict the saliency map. Their work triggered a lot of interest in the visual salience community. Many saliency models such as adapted this center-surround structure in the spatial domain [27, 28]. Itti and Baldi [29] proposed a model based on Bayesian approaches. Some methods adopted an information-theoretic justification for attentive selection [30–32]. Harel et al. [8] proposed a saliency model based on graph theory. Hou and Zhang [33] calculated saliency from frequency analysis. Some traditional saliency models used machine learning algorithms [34–36]. Some of these models incorporate high-level features such as face or text to steer the top-down process, thus they may not be purely bottom-up [37]. While many models fall into the bottom-up saliency model category, these models fail to capture the factors that contribute to attentional selection.


## 2.2. Deep Saliency Prediction Models

Employing deep convolutional neural networks (CNNs) in the saliency prediction model has made some drastic improvements over well-established saliency benchmark datasets [2]. Since 2014, using DNNs for saliency prediction gained much attention. To compensate for the lack of sufficiently large fixation data, most of these DNN-based models use transfer learning by employing a pretrained model that was trained for similar/different visual tasks on large image datasets.One of the first saliency models that used DNN was proposed by Vig et al. [38]. Their model, ensemble of deep networks (eDN), generates a large number of richly-parameterized neuromorphic networks for the feature extraction phase. Then, extracted features are applied to a linear support vector machine (SVM) to predict the saliency value. In [13, 39], Kümmerer et al. introduced a deeper structure for the encoder. DeepGaze I [39] uses pretrained AlexNet and DeepGaze II [13] uses pretrained VGG-19 for extracting features from the input image. Huang et al. [40] proposed a deep CNN structure that integrates information at different image scales. They showed that adding multi-scale information improves the saliency prediction results. Kruthiventi et al. [12] introduced a fully CNN model with a new “location biased convolutional (LBC) layer” to learn “location specific patterns” such as the center bias. Jetley et al. [41] formulated saliency map prediction as a probability distribution prediction task and trained a model to learn this distribution. Liu and Han [16] introduced a saliency model with a convolutional long short-term model (LSTM) to learn the global context. Cornia et al. [14] proposed a new saliency prediction architecture that incorporates a convolutional LSTM network and a spatial attentive mechanism. In [42], a saliency model based on image segmentation was introduced that exploits the object information for the saliency prediction task. Wang et al. [3] proposed a video saliency model, called ACLNet that uses the CNN-LSTM network to predict visual attention over dynamic scenes. Wang et al. [43] proposed a model that incorporates multi-level saliency predictions within a single network to decrease redundancy. Some researches focus on decreasing the model complexity and inference time for real-time application [44].In summary, Table1 compares the main properties of some prominent saliency prediction models and our proposed models. In Table 1, NSS, KL-D, CC, MSE, and SIM stand for normalized scanpath saliency, Kullback–Leibler divergence, linear correlation coefficient, mean square error, and similarity respectively.Table 1
A comparison between the main properties of some prominent saliency models.
Model nameCenter biasTransfer learningCNNLoss functionLSTM-based SAM [14]Multiple-learned priors✓VGG, ResNetNSS, KL-D, CCDSCLRCN [16]—✓VGG, ResNetNSSDeepGaze II [13]Gaussian prior✓VGGLog-likelihoodML-Net [15]Single multiplicative map✓VGGNormalized MSEPDP [41]—✓VGGProbability distancesSalNet [45]—✓VGGEuclidean lossSALICON [40]—✓AlexNet, VGG, GoogLeNetKL-D, NSS, CC, SIMDeepFix [12]Handcrafted priors✓VGGEuclidean losseDN [38]—-1 to 3 layer networksEuclidean lossGLG-I (proposed model)Multiple-learned priors, fully connected component✓ResNetNSS, KL-D, CCGLG-II (proposed model)Multiple-learned priors✓ResNet, VGGNSS, KL-D, CC


## 2.3. Salient Object Detection

The goal of salient object detection is to detect the most salient objects of a scene. Zhang et al. [46] used the multistage refinement mechanism to propose augmenting feedforward neural networks for addressing feature resolution reduction in CNNs. Zhao et al. [47] proposed a CNNs-based architecture that uses contrast prior to enhance the depth of information for salient object detection. Zhang et al. [48] proposed a probabilistic RGB-D saliency detection model based on conditional variational autoencoders. Li et al. [49] proposed a model that uses a pixel-level fully convolutional stream and a segment-wise spatial pooling stream to overcome the problem of blurry saliency maps, especially near the boundary of salient objects. To better segment salient and preserve the salient edges, Wang et al. [50] also proposed a model with a salient edge detection module. In most of these methods, incorporation of global and local information is missing and the need to use an appropriate model of jointly considering this information is still a challenge.


## 3. Motivation

Psychological and neurobiological experiments have discovered the role of contextual information in guiding the attentive mechanism of HSV. To understand the influence of contextual information on local saliency prediction, assume a red apple among green apples. In this scene, this red apple is certainly a salient object because of its distinct color, but among some apples with similar shape and color, it may not be a salient object. Hence, an ideal saliency model that aims to mutate this attentive mechanism is supposed to incorporate the contextual information of the scene in saliency prediction. Despite the state-of-the-art performance of deep saliency models, some experimental results have approved that CNN-based saliency models fail to capture global information and location-dependent features of the scene. In this regard, we proposed two approaches to incorporate the global scene properties and remedy some deficiencies in CNN-based saliency models. In this section, we peruse the importance of the global scene properties in saliency prediction and deficiencies of CNN structures.


### 3.1. Contextual Cues and HVS

In order to understand the importance of global properties in saliency prediction, we explain how the HVS computes the visual saliency of a scene. To capture the global features of the scene which describe the context of the scene, the brain encodes the consistent properties of the scene [51]. Experimental results reported by [39] show that the neurons belonging to the visual part of the brain demonstrate tuning characteristics that can be optimized to respond to recurring features in the scenes with comparable contents [52], thus the scenes with similar global characteristics will get similar processes in the human brain.The global visual context guides the attentive mechanism of the HVS, i.e., what to expect in the scene and where is the most salient region in familiar scenes. Indeed, HVS uses an unsupervised learning mechanism to determine the optimal features from input scenes and localize the salient regions in these scenes. When it confronts with unfamiliar scenes having comparable global properties, it uses its past experiences to efficiently process these scenes and optimally allocate the perceptual and cognitive resources [51]. In neurobiology, it is called a contextual cueing effect [53].


### 3.2. Deficiencies in CNN Structures

In a single convolutional layer, every neuron observes the input through an aperture called the convolution window. Prevalently, the size of this window is much smaller than the spatial size of the input, hence a convolutional layer is capable of extracting local features from images but it fails to efficiently extract the high-grade contextual features instead. A CNN typically consists of a series of convolutional layers. Every hidden layer in this structure uses the output of its previous layer as input. In the applications where contextual features are needed (e.g., image classification tasks), they employ some fully connected layers as the later stage to combine these mostly local features and to generate more effective global features.Statistically, it has been observed that the human eye fixations are strongly biased toward the center of an image [54] which is often explained through the photographer’s bias [12] or through an uninterested observer’s viewing strategy [55]. This phenomenon can be observed in many saliency benchmark datasets. For instance, Figure 1 shows the average of all ground truth saliency maps in the SALICON 2017 train set. This property can be considered as a global feature of the fixations of any saliency benchmark dataset. One of the most important drawbacks of using CNN structures for saliency prediction is that fully convolutional networks (FCNs) are unable to extract the center bias of the eye fixations because of the global nature of this bias. In addition, convolutional layers use weight sharing, and hence they are location-invariant (or shift-invariant). Hence, they are incapable of learning the location-dependent patterns too [12].Figure 1
The average of all ground truth saliency maps in the SALICON 2017 train set.To compensate for some of these aforementioned deficiencies, several methods have been proposed since 2014. It has been shown that cues like center bias may improve model performance [41]. To account for the center bias, some approaches linearly combined the saliency prediction with a fixed Gaussian blob (an estimate of the prior distribution) [13, 39]. Kruthiventi et al. [12] introduced an LBC filter for capturing location-dependent patterns. Instead of using predefined priors, Jia et al. [1] used a prior image to capture center bias and then pixel-wise multiplied this prior image by the predicted saliency map.


## 3.1. Contextual Cues and HVS

In order to understand the importance of global properties in saliency prediction, we explain how the HVS computes the visual saliency of a scene. To capture the global features of the scene which describe the context of the scene, the brain encodes the consistent properties of the scene [51]. Experimental results reported by [39] show that the neurons belonging to the visual part of the brain demonstrate tuning characteristics that can be optimized to respond to recurring features in the scenes with comparable contents [52], thus the scenes with similar global characteristics will get similar processes in the human brain.The global visual context guides the attentive mechanism of the HVS, i.e., what to expect in the scene and where is the most salient region in familiar scenes. Indeed, HVS uses an unsupervised learning mechanism to determine the optimal features from input scenes and localize the salient regions in these scenes. When it confronts with unfamiliar scenes having comparable global properties, it uses its past experiences to efficiently process these scenes and optimally allocate the perceptual and cognitive resources [51]. In neurobiology, it is called a contextual cueing effect [53].


## 3.2. Deficiencies in CNN Structures

In a single convolutional layer, every neuron observes the input through an aperture called the convolution window. Prevalently, the size of this window is much smaller than the spatial size of the input, hence a convolutional layer is capable of extracting local features from images but it fails to efficiently extract the high-grade contextual features instead. A CNN typically consists of a series of convolutional layers. Every hidden layer in this structure uses the output of its previous layer as input. In the applications where contextual features are needed (e.g., image classification tasks), they employ some fully connected layers as the later stage to combine these mostly local features and to generate more effective global features.Statistically, it has been observed that the human eye fixations are strongly biased toward the center of an image [54] which is often explained through the photographer’s bias [12] or through an uninterested observer’s viewing strategy [55]. This phenomenon can be observed in many saliency benchmark datasets. For instance, Figure 1 shows the average of all ground truth saliency maps in the SALICON 2017 train set. This property can be considered as a global feature of the fixations of any saliency benchmark dataset. One of the most important drawbacks of using CNN structures for saliency prediction is that fully convolutional networks (FCNs) are unable to extract the center bias of the eye fixations because of the global nature of this bias. In addition, convolutional layers use weight sharing, and hence they are location-invariant (or shift-invariant). Hence, they are incapable of learning the location-dependent patterns too [12].Figure 1
The average of all ground truth saliency maps in the SALICON 2017 train set.To compensate for some of these aforementioned deficiencies, several methods have been proposed since 2014. It has been shown that cues like center bias may improve model performance [41]. To account for the center bias, some approaches linearly combined the saliency prediction with a fixed Gaussian blob (an estimate of the prior distribution) [13, 39]. Kruthiventi et al. [12] introduced an LBC filter for capturing location-dependent patterns. Instead of using predefined priors, Jia et al. [1] used a prior image to capture center bias and then pixel-wise multiplied this prior image by the predicted saliency map.


## 4. Proposed Methods

When predicting the saliency map of an input image, the saliency value of each image patch is influenced not only by the visual features of that patch (local features) but also by the global properties of the whole scene, contextual information, and the location of the patch in that image. In this section, we propose two approaches that incorporate both locally extracted features and global scene properties into local saliency prediction. In pixel-wise saliency prediction, these methods enable the saliency model to take into account not only the locally extracted features of each pixel location but also the global scene properties. Accordingly, we call these methods the global-local gazing (GLG) based method. To evaluate the effects of employing the global-local gazing concept in saliency prediction, we use SAM-ResNet [18] as the base model and extend this model using our proposed methods.


### 4.1. Base Model

The saliency attentive model (SAM) is among the best saliency models and was proposed by Cornia et al. [14]. Figure 2 presents the architecture of this deep saliency model. It is consisting of a dilated convolutional network and a ConvLSTM network. The dilated convolutional network is an extended version of a deep convolutional neural network that has higher resolution feature maps. Cornia et al. introduced two versions of the saliency model. One of them uses VGG-16 [56] and the other version uses ResNet-50 [57] as the backbone. This dilated neural network extracts some local feature maps from the input image. The role of attentive ConvLSTM is to focus iteratively on related spatial locations to enhance extracted features. The number of timesteps for this Attentive ConvLSTM has been set to 4. An explicit prior component has been introduced in order to learn the center prior. At the final stage, a convolutional layer predicts the saliency map of the input image. To train and evaluate the model, a loss function has been defined as [14]:(1)Ly˜,yden,yfix=αNSSy˜,yfix+βCCy˜,yden+γKLy˜,yden,where y˜, yden, and yfix are the predicted saliency map, the ground truth density distribution, and the ground truth binary fixation map respectively. NSS( ), CC( ), and KL( ) are the normalized scanpath saliency, the linear correlation coefficient, and the Kullback–Leibler divergence respectively which are among the most popular saliency measures. Loss parameters [14]: in this work, we use SAM-ResNet as the base model to evaluate the effectiveness of our proposed approaches. We extended the SAM-ResNet [14] using our GLG methods, to inject the global scene properties into local saliency prediction.Figure 2
The saliency attentive model [14].


### 4.2. The GLG-I Saliency Model

As aforementioned, the convolutional layers use weight sharing to reduce the number of model parameters. Namely, all the neurons in a convolutional layer use the same weights. These weights do not depend on the location of neurons and are being used for all the spatial locations of the input. This property makes the convolutional layers location-invariant. Hence, the convolutional layers are unable to use different weights for different locations and to extract location-specific features. For CNN-based saliency models that predict the output saliency map pixel-wise, it is necessary to employ a component that compensates for such shortcomings, because the saliency value of a pixel or an image patch is very dependent on the context information of the whole scene and other global properties such as center prior.In this subsection, we introduce a component called the fully connected component. We use this component to extend and modify the base model to create our GLG-I saliency model. This extension is able to extract location-dependent and global properties of the scene to reinforce the global information for pixel-wise saliency prediction. Figure3 presents the architecture of our GLG-I saliency model.Figure 3
The GLG-I saliency model.To compute the location-dependent features and global scene properties, the locally extracted feature maps that are extracted by the dilated ResNet are applied to the fully connected component. The architecture of this proposed component is presented in Figure4. This component is composed of three convolutional layers and a fully connected layer. Two convolutional layers with a core size of 3 × 3 are employed at the primary stage to reduce the number of input channels. These layers help the component to reduce the number of parameters. Afterward, a 2D array of 1200 fully connected neurons with a size of 30 × 40, called the fully connected layer, is employed to compute the location-dependent features and global scene properties. The fully connected neurons of this layer are connected to all neurons of the second convolutional layer. Unlike the convolutional layers, the fully connected layer is location-variant because every fully connected neuron in this layer has its own weights and is able to capture location-dependent patterns/features. Finally, a convolutional layer is used to smooth the output of the fully connected layer.Figure 4
The fully connected component.Through the use of fully connected neurons, compared to the base model, these neurons increase the number of parameters only by 1.6 percent. The fully connected component has a limited number of parameters too and compared to the base model, it increases the number of the parameters only by 15 percent. However, this number of parameters can still be reduced by selecting the appropriate number of cores for the first convolutional layer in the fully connected component. For example, if we set the number of cores for the first convolutional layer to 16, compared to the base model, the number of the model parameters increases only by 2 percent without any noticeable performance reduction. Table2 presents the architectural details of our fully connected component.Table 2
The architecture of our fully connected component.
Core sizeNumber of coresOutput sizeNumber of parametersConv2d3 × 351230 × 40 × 5129,437,696Conv2d3 × 3130 × 40 × 14,6082D fully connected layer——30 × 40 × 11,124,400Conv2d3 × 3130 × 40 × 110Total number of parameters:10,566,715The resulting feature map is concatenated with the output of the learning prior module, and then these feature maps are applied to a convolutional layer for predicting the saliency map of the input image. In the training phase, this predicted saliency map is evaluated using the ground truth. Table2 compares the number of parameters in our GLG-I model with the base model.


### 4.3. The GLG-II Saliency Model

As aforementioned, GLG-I uses a fully connected component to compute the location-dependent features and global scene properties. Instead of using a fully connected component, here we introduce another approach called GLG-II that uses the output fully connected layer at the final stages of a deep neural network for extracting the contextual features of the scene. Most deep models predict the saliency value pixel-wise, and hence we use a new approach to make the contextual information available pixel-wise. To do so, we repeat this global feature vector to make it available at any spatial location of the image. Here, we use the VGG neural network (VGGNet) [56] to extend and modify our base model and to create our GLG-II saliency model, but in general, the fully connected layers of the backend neural network can be used instead to avoid using an additional deep model. Figure 5 presents the architecture of our GLG-II saliency model. The weights of this VGGNet are initialized with that of the VGG-16 trained on ImageNet [58]. The output of the second fully connected layer of this VGG structure was considered as contextual features because this neural network has been trained to classify input images based on their context, and thus the features that are extracted at later layers are expected to describe the contextual information of the input image.Figure 5
The GLG-II saliency model.For every input image, the locally extracted features and contextual information are computed using dilated ResNet and VGGNet respectively. The VGG neural network generates a vector of 4096 features at its second fully connected layer. We use this feature vector as the contextual information of the scene. CNN-based saliency models predict the saliency value pixel-wise. To incorporate the contextual information in the saliency prediction of every pixel, we embed the contextual information in every spatial location of each pixel. To do this, we repeat this feature vector along two spatial dimensions (width and height) to generate a 3D global feature array. The globally and locally extracted feature maps are concatenated to enable the model to predict the saliency value of each pixel by using both of this information. A 3 × 3 convolutional layer is employed to reduce the number of channels in concatenated feature maps and as a result, the number of model parameters reduces. However, the increase in the number of model parameters compared to the base model is due to this layer. Then, a convolutional LSTM fine-tunes the resulting features. After the prior module, at the final stage, a convolutional layer predicts the saliency map of the input image. For the training phase, this predicted saliency map is evaluated using the ground truths.We initialize the weights of the VGGNet with that of the VGG-16 trained on ImageNet [58]. As we want to use VGGNet to extract contextual information, the trained weights on ImageNet would be enough, and no training phase is required for our VGGNet. That is, the weights of VGGNet would stay frozen and the number of the model parameter would not increase by employing VGGNet. Table 3 compares the number of parameters in our GLG-II model with the base model.Table 3
The number of parameters in GLG models compared to the base model.
Model nameNumber of parametersSAM-ResNet70,093,441GLG-I80,660,157GLG-II88,967,809


## 4.1. Base Model

The saliency attentive model (SAM) is among the best saliency models and was proposed by Cornia et al. [14]. Figure 2 presents the architecture of this deep saliency model. It is consisting of a dilated convolutional network and a ConvLSTM network. The dilated convolutional network is an extended version of a deep convolutional neural network that has higher resolution feature maps. Cornia et al. introduced two versions of the saliency model. One of them uses VGG-16 [56] and the other version uses ResNet-50 [57] as the backbone. This dilated neural network extracts some local feature maps from the input image. The role of attentive ConvLSTM is to focus iteratively on related spatial locations to enhance extracted features. The number of timesteps for this Attentive ConvLSTM has been set to 4. An explicit prior component has been introduced in order to learn the center prior. At the final stage, a convolutional layer predicts the saliency map of the input image. To train and evaluate the model, a loss function has been defined as [14]:(1)Ly˜,yden,yfix=αNSSy˜,yfix+βCCy˜,yden+γKLy˜,yden,where y˜, yden, and yfix are the predicted saliency map, the ground truth density distribution, and the ground truth binary fixation map respectively. NSS( ), CC( ), and KL( ) are the normalized scanpath saliency, the linear correlation coefficient, and the Kullback–Leibler divergence respectively which are among the most popular saliency measures. Loss parameters [14]: in this work, we use SAM-ResNet as the base model to evaluate the effectiveness of our proposed approaches. We extended the SAM-ResNet [14] using our GLG methods, to inject the global scene properties into local saliency prediction.Figure 2
The saliency attentive model [14].


## 4.2. The GLG-I Saliency Model

As aforementioned, the convolutional layers use weight sharing to reduce the number of model parameters. Namely, all the neurons in a convolutional layer use the same weights. These weights do not depend on the location of neurons and are being used for all the spatial locations of the input. This property makes the convolutional layers location-invariant. Hence, the convolutional layers are unable to use different weights for different locations and to extract location-specific features. For CNN-based saliency models that predict the output saliency map pixel-wise, it is necessary to employ a component that compensates for such shortcomings, because the saliency value of a pixel or an image patch is very dependent on the context information of the whole scene and other global properties such as center prior.In this subsection, we introduce a component called the fully connected component. We use this component to extend and modify the base model to create our GLG-I saliency model. This extension is able to extract location-dependent and global properties of the scene to reinforce the global information for pixel-wise saliency prediction. Figure3 presents the architecture of our GLG-I saliency model.Figure 3
The GLG-I saliency model.To compute the location-dependent features and global scene properties, the locally extracted feature maps that are extracted by the dilated ResNet are applied to the fully connected component. The architecture of this proposed component is presented in Figure4. This component is composed of three convolutional layers and a fully connected layer. Two convolutional layers with a core size of 3 × 3 are employed at the primary stage to reduce the number of input channels. These layers help the component to reduce the number of parameters. Afterward, a 2D array of 1200 fully connected neurons with a size of 30 × 40, called the fully connected layer, is employed to compute the location-dependent features and global scene properties. The fully connected neurons of this layer are connected to all neurons of the second convolutional layer. Unlike the convolutional layers, the fully connected layer is location-variant because every fully connected neuron in this layer has its own weights and is able to capture location-dependent patterns/features. Finally, a convolutional layer is used to smooth the output of the fully connected layer.Figure 4
The fully connected component.Through the use of fully connected neurons, compared to the base model, these neurons increase the number of parameters only by 1.6 percent. The fully connected component has a limited number of parameters too and compared to the base model, it increases the number of the parameters only by 15 percent. However, this number of parameters can still be reduced by selecting the appropriate number of cores for the first convolutional layer in the fully connected component. For example, if we set the number of cores for the first convolutional layer to 16, compared to the base model, the number of the model parameters increases only by 2 percent without any noticeable performance reduction. Table2 presents the architectural details of our fully connected component.Table 2
The architecture of our fully connected component.
Core sizeNumber of coresOutput sizeNumber of parametersConv2d3 × 351230 × 40 × 5129,437,696Conv2d3 × 3130 × 40 × 14,6082D fully connected layer——30 × 40 × 11,124,400Conv2d3 × 3130 × 40 × 110Total number of parameters:10,566,715The resulting feature map is concatenated with the output of the learning prior module, and then these feature maps are applied to a convolutional layer for predicting the saliency map of the input image. In the training phase, this predicted saliency map is evaluated using the ground truth. Table2 compares the number of parameters in our GLG-I model with the base model.


## 4.3. The GLG-II Saliency Model

As aforementioned, GLG-I uses a fully connected component to compute the location-dependent features and global scene properties. Instead of using a fully connected component, here we introduce another approach called GLG-II that uses the output fully connected layer at the final stages of a deep neural network for extracting the contextual features of the scene. Most deep models predict the saliency value pixel-wise, and hence we use a new approach to make the contextual information available pixel-wise. To do so, we repeat this global feature vector to make it available at any spatial location of the image. Here, we use the VGG neural network (VGGNet) [56] to extend and modify our base model and to create our GLG-II saliency model, but in general, the fully connected layers of the backend neural network can be used instead to avoid using an additional deep model. Figure 5 presents the architecture of our GLG-II saliency model. The weights of this VGGNet are initialized with that of the VGG-16 trained on ImageNet [58]. The output of the second fully connected layer of this VGG structure was considered as contextual features because this neural network has been trained to classify input images based on their context, and thus the features that are extracted at later layers are expected to describe the contextual information of the input image.Figure 5
The GLG-II saliency model.For every input image, the locally extracted features and contextual information are computed using dilated ResNet and VGGNet respectively. The VGG neural network generates a vector of 4096 features at its second fully connected layer. We use this feature vector as the contextual information of the scene. CNN-based saliency models predict the saliency value pixel-wise. To incorporate the contextual information in the saliency prediction of every pixel, we embed the contextual information in every spatial location of each pixel. To do this, we repeat this feature vector along two spatial dimensions (width and height) to generate a 3D global feature array. The globally and locally extracted feature maps are concatenated to enable the model to predict the saliency value of each pixel by using both of this information. A 3 × 3 convolutional layer is employed to reduce the number of channels in concatenated feature maps and as a result, the number of model parameters reduces. However, the increase in the number of model parameters compared to the base model is due to this layer. Then, a convolutional LSTM fine-tunes the resulting features. After the prior module, at the final stage, a convolutional layer predicts the saliency map of the input image. For the training phase, this predicted saliency map is evaluated using the ground truths.We initialize the weights of the VGGNet with that of the VGG-16 trained on ImageNet [58]. As we want to use VGGNet to extract contextual information, the trained weights on ImageNet would be enough, and no training phase is required for our VGGNet. That is, the weights of VGGNet would stay frozen and the number of the model parameter would not increase by employing VGGNet. Table 3 compares the number of parameters in our GLG-II model with the base model.Table 3
The number of parameters in GLG models compared to the base model.
Model nameNumber of parametersSAM-ResNet70,093,441GLG-I80,660,157GLG-II88,967,809


## 5. Experimental Setup

In this section, some popular evaluation metrics, evaluation baselines, and saliency datasets are described, and then implementation details are provided.


### 5.1. Evaluation Metrics

For measuring the saliency model performance, several measures are being used. Some of these evaluation measures are distribution based and they compare predicted saliency maps and fixation maps. Other metrics are location based and compute some statistics at fixated locations. In this section, these metrics are concisely described.


#### 5.1.1. Pearson’s Correlation Coefficient

The correlation coefficient measure (CC), calculates the correlation between the ground truth mapG and the predicted map P. It can be measured as [59]:(2)CCP,G=covP,GstdP×stdG,where std( ) and cov( ) compute the standard deviation and covariance, respectively. The CC ranges between −1 and 1. A value of 1 shows a complete positive correlation between P and G. A value of 0 shows no relationship between these two maps.


#### 5.1.2. Kullback–Leibler Divergence

Kullback–Leibler divergence (KL-D) can be used to calculate the difference between two probability distributions. If we interpret the predicted mapP and ground truth map G, it can be computed as [59]:(3)KLP,G=∑iGilogε+Giε+Pi,where ε is a constant is used for regularization and i indexes the ith pixel. As can be seen, the KL score is asymmetric. A larger KL value shows a larger difference between the predicted saliency map and fixation map while a KL score of zero indicates that the model is predicting the saliency values perfectly.


#### 5.1.3. Earth Mover’s Distance

The Earth mover’s distance (EMD) measures the spatial distance between the predicted map and the ground truth map over a region. EMD computes how much transformation the predicted saliency map would need to match the fixation map [59].A larger difference between the predicted map and fixation maps results in a larger EMD value while a zero value shows that the predicted and fixation maps are the same.


#### 5.1.4. Similarity or Histogram Intersection

The similarity metric (SIM) measures the similarity between the predicted saliency mapP and ground truth fixation map G. SIM is computed as the sum of the minimum values of the normalized P and G at each pixel. It can be computed as [59]:(4)SIMP,G=∑iminPi,Gi,where∑iPi=∑iGi=1.The SIM ranges between zero and one. A value of 1 showsP and G are the same. A value of 0 shows no overlap between P and G.


#### 5.1.5. Normalized Scanpath Saliency

The normalized scanpath saliency (NSS) calculates the correspondence of predicted saliency mapsP and the binary fixation map of GB. It measures the average of the predicted saliency values in fixated points after normalization and can be computed as [59]:(5)NSSP,GB=1N∑iP¯i×GiB,whereN=∑iGiBandP¯=P−meanPstdP,where std( ) and mean( ) compute the standard deviation and average, respectively, i indicates the ith pixel, and N is the number of fixation points.A larger NSS indicates higher saliency values in fixated points and better performance of the model. An NSS of zero shows that the saliency model does not work better than a random number generator and a negative NSS shows that the saliency model performs worse than a random number generator.


#### 5.1.6. Area under ROC Curve

By interpreting a saliency model as a binary classifier that classifies each pixel into fixated and nonfixated. The ROC curve can be created by plotting the true positive rate (TPR) versus the false positive rate (FPR) at various discrimination thresholds. The area under the ROC curve (AUC) is used to evaluate the model’s performance. Various kinds of AUC have been proposed which differ in how TPR and FPR are calculated. The AUC values range between 0.0 and 1.0. A larger AUC indicates that the model performs better.The AUC-Judd [36] (or AUC in this study) uses the samples from saliency map values as the thresholds. For a given threshold, the percentage of saliency values smaller than the threshold at fixation locations is TPR and the percentage of saliency values higher than the threshold at unfixated pixels is FPR.To bypass the effects of the center bias on FPR calculation, The AUC-Borji [60] calculates the FPR at random pixels that are sampled uniformly from all image pixels and the shuffled AUC (sAUC) [61, 62] calculates the FPR at random pixels that are sampled uniformly from fixations on other images. Despite the difference in the definition of TPR, the AUC-Judd, the AUC-Borji, and the shuffled AUC calculate the TPR similarly.


#### 5.1.7. Information Gain

Information gain (IG) [63] is an information-theoretic metric that computes the average information gain of the saliency map P for the center-prior baseline B at fixated locations GB [59]. Information gain is computed as:(6)IGP,GB=1N∑iGiBlog2ε+Pi−log2ε+Bi,where ε is a constant for regularization, i indicates the ith pixel, and N is the number of fixation points. An IG score above zero indicates the model outperforms the center prior to baseline in the prediction of ground truth fixations.


### 5.2. Evaluation Baselines

(i)
Infinite: this baseline uses the fixation points of an infinite number of observers to predict the fixation points of another infinite number of observers.(ii)
One human: this baseline uses the fixation points of an observer to predict the fixation points of the other observers.(iii)
Center: this baseline uses a symmetric 2D Gaussian map as the predicted fixation map of the input image.(iv)
Permutation: this baseline uses fixation points of a randomly selected image as the predicted fixation points of the input image.(v)
Chance: this baseline uses a randomly generated saliency map as the predicted fixation map of the input image.


### 5.3. Saliency Datasets

In this work, we train and evaluate our models over four datasets: the dataset of SALICON Challenge 2015, the dataset of SALICON Saliency Prediction Challenge (LSUN 2017), MIT300, and MIT1003 that are among the most popular image-based saliency datasets.


#### 5.3.1. SALICON 2015 and SALICON 2017

The dataset of SALICON Challenge 2015 [64] and the dataset of the SALICON Saliency Prediction Challenge (LSUN 2017) are among the richest saliency datasets based on the MS COCO image dataset [65]. They consist of 10,000 images for training, 5,000 images for validation data, and 5,000 images for the test. We call these datasets SALICON 2015 and SALICON 2017 respectively. Presently, the model evaluation over SALICON 2015 test set is not available because it has been closed by the provider.Deep neural networks need abundant data for the training phase. Currently, many studies train their deep saliency models on the SALICON dataset and then fine-tune on other saliency datasets for predicting fixations of small datasets. Considering the evaluation result of state-of-the-art saliency models over the SALICON 2015 test set that is available in [2], our base model, SAM-ResNet [14], is among the best models over SALICON 2015 test set.


#### 5.3.2. MIT300

The MIT300 [66] consists of 300 color images of natural indoor and outdoor scenes in JPG format that is used as a benchmark test set. The ground truth (fixation points and saliency map) of this dataset is not provided and the MIT/Tuebingen Saliency Benchmark [67, 68] uses it for evaluation of the saliency models according to multiple metrics.


#### 5.3.3. MIT1003

The MIT1003 [36] consists of 1003 color images of natural indoor and outdoor scenes in JPG format. The ground truth (fixation points and saliency map) of this dataset is provided and it is available as the training data for MIT/Tuebingen Saliency Benchmark [67, 68].


### 5.4. Implementation Details

As mentioned before, our models are evaluated on SALICON 2015, SALICON 2017, and MIT300. For SALICON 2015 and SALICON 2017, we train our model on the training data and are validated on the validation set of these datasets using the loss function in (1). For SALICON datasets, a batch size of 10 samples is chosen for the training and validation phase. As instructed by the MIT Saliency Benchmark [67], for MIT300, we pretrain our models on the SALICON and then fine-tune them on MIT1003. To find the appropriate version of the SALICON dataset that leads to better performance on MIT300, we tested both SALICON 2015 and SALICON 2017 for the pretraining phase separately. To fine-tune the models on MIT1003, this dataset is split randomly into 904 images of the training set and 99 images of the validation set. In the pretraining phase on SALICON and fine-tuning phase on MIT1003, batch sizes of 10 and 9 samples are chosen respectively.For the pretraining and finetuning stages, the learning rate is initialized to 10−4 and after every two epochs it is decreased by a factor of 10. Finally, the models with the best validation loss are chosen for evaluation on the test set.We use a computer with 16 GB RAM and NVIDIA Tesla K80 GPU. The number of rows and columns of the input images is 240 and 320 pixels, respectively. The inference time of the base model using the aforementioned GPU is about 200 ms. The inference times of our models are about 250 ms which shows only a 25 percent increase. This indicates that our methods do not increase the overall inference time of the model. The reason for this is that the base model uses a recursive component that requires a lot of time to calculate its output.


## 5.1. Evaluation Metrics

For measuring the saliency model performance, several measures are being used. Some of these evaluation measures are distribution based and they compare predicted saliency maps and fixation maps. Other metrics are location based and compute some statistics at fixated locations. In this section, these metrics are concisely described.


### 5.1.1. Pearson’s Correlation Coefficient

The correlation coefficient measure (CC), calculates the correlation between the ground truth mapG and the predicted map P. It can be measured as [59]:(2)CCP,G=covP,GstdP×stdG,where std( ) and cov( ) compute the standard deviation and covariance, respectively. The CC ranges between −1 and 1. A value of 1 shows a complete positive correlation between P and G. A value of 0 shows no relationship between these two maps.


### 5.1.2. Kullback–Leibler Divergence

Kullback–Leibler divergence (KL-D) can be used to calculate the difference between two probability distributions. If we interpret the predicted mapP and ground truth map G, it can be computed as [59]:(3)KLP,G=∑iGilogε+Giε+Pi,where ε is a constant is used for regularization and i indexes the ith pixel. As can be seen, the KL score is asymmetric. A larger KL value shows a larger difference between the predicted saliency map and fixation map while a KL score of zero indicates that the model is predicting the saliency values perfectly.


### 5.1.3. Earth Mover’s Distance

The Earth mover’s distance (EMD) measures the spatial distance between the predicted map and the ground truth map over a region. EMD computes how much transformation the predicted saliency map would need to match the fixation map [59].A larger difference between the predicted map and fixation maps results in a larger EMD value while a zero value shows that the predicted and fixation maps are the same.


### 5.1.4. Similarity or Histogram Intersection

The similarity metric (SIM) measures the similarity between the predicted saliency mapP and ground truth fixation map G. SIM is computed as the sum of the minimum values of the normalized P and G at each pixel. It can be computed as [59]:(4)SIMP,G=∑iminPi,Gi,where∑iPi=∑iGi=1.The SIM ranges between zero and one. A value of 1 showsP and G are the same. A value of 0 shows no overlap between P and G.


### 5.1.5. Normalized Scanpath Saliency

The normalized scanpath saliency (NSS) calculates the correspondence of predicted saliency mapsP and the binary fixation map of GB. It measures the average of the predicted saliency values in fixated points after normalization and can be computed as [59]:(5)NSSP,GB=1N∑iP¯i×GiB,whereN=∑iGiBandP¯=P−meanPstdP,where std( ) and mean( ) compute the standard deviation and average, respectively, i indicates the ith pixel, and N is the number of fixation points.A larger NSS indicates higher saliency values in fixated points and better performance of the model. An NSS of zero shows that the saliency model does not work better than a random number generator and a negative NSS shows that the saliency model performs worse than a random number generator.


### 5.1.6. Area under ROC Curve

By interpreting a saliency model as a binary classifier that classifies each pixel into fixated and nonfixated. The ROC curve can be created by plotting the true positive rate (TPR) versus the false positive rate (FPR) at various discrimination thresholds. The area under the ROC curve (AUC) is used to evaluate the model’s performance. Various kinds of AUC have been proposed which differ in how TPR and FPR are calculated. The AUC values range between 0.0 and 1.0. A larger AUC indicates that the model performs better.The AUC-Judd [36] (or AUC in this study) uses the samples from saliency map values as the thresholds. For a given threshold, the percentage of saliency values smaller than the threshold at fixation locations is TPR and the percentage of saliency values higher than the threshold at unfixated pixels is FPR.To bypass the effects of the center bias on FPR calculation, The AUC-Borji [60] calculates the FPR at random pixels that are sampled uniformly from all image pixels and the shuffled AUC (sAUC) [61, 62] calculates the FPR at random pixels that are sampled uniformly from fixations on other images. Despite the difference in the definition of TPR, the AUC-Judd, the AUC-Borji, and the shuffled AUC calculate the TPR similarly.


### 5.1.7. Information Gain

Information gain (IG) [63] is an information-theoretic metric that computes the average information gain of the saliency map P for the center-prior baseline B at fixated locations GB [59]. Information gain is computed as:(6)IGP,GB=1N∑iGiBlog2ε+Pi−log2ε+Bi,where ε is a constant for regularization, i indicates the ith pixel, and N is the number of fixation points. An IG score above zero indicates the model outperforms the center prior to baseline in the prediction of ground truth fixations.


## 5.1.1. Pearson’s Correlation Coefficient

The correlation coefficient measure (CC), calculates the correlation between the ground truth mapG and the predicted map P. It can be measured as [59]:(2)CCP,G=covP,GstdP×stdG,where std( ) and cov( ) compute the standard deviation and covariance, respectively. The CC ranges between −1 and 1. A value of 1 shows a complete positive correlation between P and G. A value of 0 shows no relationship between these two maps.


## 5.1.2. Kullback–Leibler Divergence

Kullback–Leibler divergence (KL-D) can be used to calculate the difference between two probability distributions. If we interpret the predicted mapP and ground truth map G, it can be computed as [59]:(3)KLP,G=∑iGilogε+Giε+Pi,where ε is a constant is used for regularization and i indexes the ith pixel. As can be seen, the KL score is asymmetric. A larger KL value shows a larger difference between the predicted saliency map and fixation map while a KL score of zero indicates that the model is predicting the saliency values perfectly.


## 5.1.3. Earth Mover’s Distance

The Earth mover’s distance (EMD) measures the spatial distance between the predicted map and the ground truth map over a region. EMD computes how much transformation the predicted saliency map would need to match the fixation map [59].A larger difference between the predicted map and fixation maps results in a larger EMD value while a zero value shows that the predicted and fixation maps are the same.


## 5.1.4. Similarity or Histogram Intersection

The similarity metric (SIM) measures the similarity between the predicted saliency mapP and ground truth fixation map G. SIM is computed as the sum of the minimum values of the normalized P and G at each pixel. It can be computed as [59]:(4)SIMP,G=∑iminPi,Gi,where∑iPi=∑iGi=1.The SIM ranges between zero and one. A value of 1 showsP and G are the same. A value of 0 shows no overlap between P and G.


## 5.1.5. Normalized Scanpath Saliency

The normalized scanpath saliency (NSS) calculates the correspondence of predicted saliency mapsP and the binary fixation map of GB. It measures the average of the predicted saliency values in fixated points after normalization and can be computed as [59]:(5)NSSP,GB=1N∑iP¯i×GiB,whereN=∑iGiBandP¯=P−meanPstdP,where std( ) and mean( ) compute the standard deviation and average, respectively, i indicates the ith pixel, and N is the number of fixation points.A larger NSS indicates higher saliency values in fixated points and better performance of the model. An NSS of zero shows that the saliency model does not work better than a random number generator and a negative NSS shows that the saliency model performs worse than a random number generator.


## 5.1.6. Area under ROC Curve

By interpreting a saliency model as a binary classifier that classifies each pixel into fixated and nonfixated. The ROC curve can be created by plotting the true positive rate (TPR) versus the false positive rate (FPR) at various discrimination thresholds. The area under the ROC curve (AUC) is used to evaluate the model’s performance. Various kinds of AUC have been proposed which differ in how TPR and FPR are calculated. The AUC values range between 0.0 and 1.0. A larger AUC indicates that the model performs better.The AUC-Judd [36] (or AUC in this study) uses the samples from saliency map values as the thresholds. For a given threshold, the percentage of saliency values smaller than the threshold at fixation locations is TPR and the percentage of saliency values higher than the threshold at unfixated pixels is FPR.To bypass the effects of the center bias on FPR calculation, The AUC-Borji [60] calculates the FPR at random pixels that are sampled uniformly from all image pixels and the shuffled AUC (sAUC) [61, 62] calculates the FPR at random pixels that are sampled uniformly from fixations on other images. Despite the difference in the definition of TPR, the AUC-Judd, the AUC-Borji, and the shuffled AUC calculate the TPR similarly.


## 5.1.7. Information Gain

Information gain (IG) [63] is an information-theoretic metric that computes the average information gain of the saliency map P for the center-prior baseline B at fixated locations GB [59]. Information gain is computed as:(6)IGP,GB=1N∑iGiBlog2ε+Pi−log2ε+Bi,where ε is a constant for regularization, i indicates the ith pixel, and N is the number of fixation points. An IG score above zero indicates the model outperforms the center prior to baseline in the prediction of ground truth fixations.


## 5.2. Evaluation Baselines

(i)
Infinite: this baseline uses the fixation points of an infinite number of observers to predict the fixation points of another infinite number of observers.(ii)
One human: this baseline uses the fixation points of an observer to predict the fixation points of the other observers.(iii)
Center: this baseline uses a symmetric 2D Gaussian map as the predicted fixation map of the input image.(iv)
Permutation: this baseline uses fixation points of a randomly selected image as the predicted fixation points of the input image.(v)
Chance: this baseline uses a randomly generated saliency map as the predicted fixation map of the input image.


## 5.3. Saliency Datasets

In this work, we train and evaluate our models over four datasets: the dataset of SALICON Challenge 2015, the dataset of SALICON Saliency Prediction Challenge (LSUN 2017), MIT300, and MIT1003 that are among the most popular image-based saliency datasets.


### 5.3.1. SALICON 2015 and SALICON 2017

The dataset of SALICON Challenge 2015 [64] and the dataset of the SALICON Saliency Prediction Challenge (LSUN 2017) are among the richest saliency datasets based on the MS COCO image dataset [65]. They consist of 10,000 images for training, 5,000 images for validation data, and 5,000 images for the test. We call these datasets SALICON 2015 and SALICON 2017 respectively. Presently, the model evaluation over SALICON 2015 test set is not available because it has been closed by the provider.Deep neural networks need abundant data for the training phase. Currently, many studies train their deep saliency models on the SALICON dataset and then fine-tune on other saliency datasets for predicting fixations of small datasets. Considering the evaluation result of state-of-the-art saliency models over the SALICON 2015 test set that is available in [2], our base model, SAM-ResNet [14], is among the best models over SALICON 2015 test set.


### 5.3.2. MIT300

The MIT300 [66] consists of 300 color images of natural indoor and outdoor scenes in JPG format that is used as a benchmark test set. The ground truth (fixation points and saliency map) of this dataset is not provided and the MIT/Tuebingen Saliency Benchmark [67, 68] uses it for evaluation of the saliency models according to multiple metrics.


### 5.3.3. MIT1003

The MIT1003 [36] consists of 1003 color images of natural indoor and outdoor scenes in JPG format. The ground truth (fixation points and saliency map) of this dataset is provided and it is available as the training data for MIT/Tuebingen Saliency Benchmark [67, 68].


## 5.3.1. SALICON 2015 and SALICON 2017

The dataset of SALICON Challenge 2015 [64] and the dataset of the SALICON Saliency Prediction Challenge (LSUN 2017) are among the richest saliency datasets based on the MS COCO image dataset [65]. They consist of 10,000 images for training, 5,000 images for validation data, and 5,000 images for the test. We call these datasets SALICON 2015 and SALICON 2017 respectively. Presently, the model evaluation over SALICON 2015 test set is not available because it has been closed by the provider.Deep neural networks need abundant data for the training phase. Currently, many studies train their deep saliency models on the SALICON dataset and then fine-tune on other saliency datasets for predicting fixations of small datasets. Considering the evaluation result of state-of-the-art saliency models over the SALICON 2015 test set that is available in [2], our base model, SAM-ResNet [14], is among the best models over SALICON 2015 test set.


## 5.3.2. MIT300

The MIT300 [66] consists of 300 color images of natural indoor and outdoor scenes in JPG format that is used as a benchmark test set. The ground truth (fixation points and saliency map) of this dataset is not provided and the MIT/Tuebingen Saliency Benchmark [67, 68] uses it for evaluation of the saliency models according to multiple metrics.


## 5.3.3. MIT1003

The MIT1003 [36] consists of 1003 color images of natural indoor and outdoor scenes in JPG format. The ground truth (fixation points and saliency map) of this dataset is provided and it is available as the training data for MIT/Tuebingen Saliency Benchmark [67, 68].


## 5.4. Implementation Details

As mentioned before, our models are evaluated on SALICON 2015, SALICON 2017, and MIT300. For SALICON 2015 and SALICON 2017, we train our model on the training data and are validated on the validation set of these datasets using the loss function in (1). For SALICON datasets, a batch size of 10 samples is chosen for the training and validation phase. As instructed by the MIT Saliency Benchmark [67], for MIT300, we pretrain our models on the SALICON and then fine-tune them on MIT1003. To find the appropriate version of the SALICON dataset that leads to better performance on MIT300, we tested both SALICON 2015 and SALICON 2017 for the pretraining phase separately. To fine-tune the models on MIT1003, this dataset is split randomly into 904 images of the training set and 99 images of the validation set. In the pretraining phase on SALICON and fine-tuning phase on MIT1003, batch sizes of 10 and 9 samples are chosen respectively.For the pretraining and finetuning stages, the learning rate is initialized to 10−4 and after every two epochs it is decreased by a factor of 10. Finally, the models with the best validation loss are chosen for evaluation on the test set.We use a computer with 16 GB RAM and NVIDIA Tesla K80 GPU. The number of rows and columns of the input images is 240 and 320 pixels, respectively. The inference time of the base model using the aforementioned GPU is about 200 ms. The inference times of our models are about 250 ms which shows only a 25 percent increase. This indicates that our methods do not increase the overall inference time of the model. The reason for this is that the base model uses a recursive component that requires a lot of time to calculate its output.


## 6. Experimental Results

The evaluation results of our GLG models over the SALICON 2015 validation set, the SALICON 2017 test set, and MIT300 are reported and compared with the state-of-the-art saliency models. Currently, evaluations on the CAT2000 test set and the SALICON 2015 test set are closed and are not available anymore.Considering Table4, over the SALICON 2015 validation set, the GLG-I model outperforms the base model according to AUC, CC, and sAUC and outperforms all other existing state-of-the-art saliency models according to AUC. The GLG-II model outperforms the base model (SAM-ResNet) according to AUC, sAUC, and NSS and outperforms almost other existing state-of-the-art saliency models according to sAUC and NSS.Table 4
Performance of GLG models, compared to state-of-the-art saliency models over the SALICON 2015 validation set, compiled from SALICON challenge 2015 website.
Model nameAUCCCSAUCNSSEOF-MODEL [42]0.8860.8510.7913.026GLG-I0.890.8460.7883.243GLG-II0.8870.8430.7913.262DSCLSTM [16]0.8870.8350.7883.221DSCLRCN [16]0.8870.8350.7853.221SAM-ResNet [14]0.8860.8440.7873.26DeepGaze II [13]0.8860.5050.7671.34FSM [44]0.8840.8030.7752.756SAM-VGG [14]0.8830.830.7823.219ML-Net [15]0.8690.7440.7762.829SalNet: deep convnet [45]0.8580.6090.7271.822SalNet: shallow convnet [45]0.8170.5480.6581.625Considering Table5, our models outperform the base model according to AUC, CC, KL, IG, and SIM. Our models also outperform other state-of-the-art models according to CC, AUC, and SIM, and over the SALICON 2017 test set.Table 5
Performance of GLG models compared to SAM-ResNet over the SALICON 2017 test set.
Model nameCCAUCSIMKLIGNSSGLG-I0.9030.8670.7980.370.7641.99GLG-II0.9030.8670.7990.430.7081.987EOF-MODEL [42]0.9000.8660.7940.3920.7231.954SAM-ResNet [14]0.8990.8650.7930.610.5381.99MSI-Net [69]0.8890.8650.7840.3070.7931.931EML-NET [1]0.8860.8660.780.520.7362.05GazeGAN [70]0.8790.8640.7730.3760.721.899FSM [44]0.8750.8620.7720.3650.7161.863MD-SEM [71]0.868——0.568—2.058SalNet [45]0.622————1.859Considering Table6, over the MIT300, the GLG-I model outperforms the base model according to EMD, AUC-B, sAUC, CC, and KL, and the GLG-II model outperforms the base model according to EMD, AUC-B, sAUC, CC, NSS, and KL. In Table 6, the evaluation results were sorted based on SIM, CC, and AUC-B. Overall, our proposed models also outperform as well as the best state-of-the-art models.Table 6
Performance of GLG models compared to state-of-the-art saliency models over the MIT300 dataset, compiled from [2].
Model nameSIMCCAUC-BKLEMDsAUCNSSAUC-JBaseline: infinite110.88000.813.290.92EOF-MODEL [42]0.680.790.801.052.100.722.310.87GLG-I_salicon 20170.680.790.801.132.000.712.340.87GLG-II_salicon 20170.680.790.801.101.990.712.340.87GLG-I_salicon 20150.680.790.800.992.030.712.330.87GLG-II_salicon 20150.680.790.791.242.050.712.350.87EML-NET [1]0.680.790.770.841.840.702.470.88SAM-ResNet [12]0.680.780.781.272.150.702.340.87SAM-VGG [12]0.670.770.781.132.140.712.300.87FSM [44]0.650.740.800.802.320.712.100.86SalGAN [60]0.630.730.811.072.290.722.040.86PDP [37]0.60.700.800.922.580.732.050.85ML-Net [13]0.590.670.751.102.630.702.050.85DVI [43]0.580.680.78-3.050.711.980.85SalNet [38]0.520.580.820.813.310.691.510.83GBVS [6]0.480.480.800.873.510.631.240.81Deep gaze 2 [11]0.460.520.860.963.980.721.290.88Baseline: center0.450.380.771.243.720.510.920.78IttiKoch2 [3]0.440.370.741.034.260.630.970.75eDN [34]0.410.450.811.144.560.621.140.82Deep Gaze 1 [35]0.390.480.831.234.970.661.220.84Baseline: 1 human0.380.520.666.193.480.631.650.80SUN saliency [45]0.380.250.661.275.100.610.680.67Baseline: Perm.0.340.200.596.124.590.500.490.68Baseline: Chance0.3300.502.096.350.5000.50IttiKoch [61]0.20.140.542.305.170.530.430.60It also shows that pretraining on SALICON 2017 and SALICON 2015 does not affect noticeably on model performance over MIT300.It can be concluded from the evaluation results over SALICON 2015, SALICON 2017, and MIT300 that our methods improved the performance of the base model. These extensions on the base model enable the saliency model to capture global information better and improve the accuracy of the saliency prediction task. In Figure6, we compare the output of our model with EML-NET and SAM-ResNet. Figure 6 demonstrates that by using our proposed methods for including the contextual information and location-dependent patterns, the focus of attention gets corrected in most cases and the model performance improves according to several evaluation metrics.Figure 6
Qualitative results and comparison to the state of the art.


## 7. Discussion

As aforementioned, convolutional layers use weight sharing and as a result, they are location-invariant. Hence, the fully convolutional neural networks [44] make them incapable of learning the location-dependent patterns [12], and global scene properties. In our GLG-I model, we propose a novel fully connected component to incorporate these properties into the local saliency prediction. Unlike the convolutional layers, the fully connected layer is location-variant because every fully connected neuron in this layer has its own weights and is able to capture location-dependent patterns/features. Considering the performance of the GLG-I model on different datasets, it can be concluded that by employing some location-variant structures in the model, the performance of saliency prediction improves considerably.Experimental results demonstrate that the neurons of the visual part of the brain show tuning properties that can be optimized to better react to recurring features in the scenes with comparable contents [52]. HVS provides a good platform to learn the best features and locations of the salient region of a scene and extend this for similar scenes [51]. Our GLG-II model imitates this mechanism in the human brain and employs an additional VGGNet to extract and incorporate the contextual information of the scene. Considering the performance of the GLG-II model on different datasets, it can be concluded that as expected from the contextual cueing effect [53], by incorporating the contextual features of the scene into the local saliency prediction, the performance of saliency prediction improves.Despite the fact that the deep state-of-the-art saliency models have shown tremendous improvements over the classic saliency models, these models mainly suffer from a high number of parameters. Although deep saliency models are suitable for applications that require high accuracy, they are not recommended for real-time applications due to their high number of parameters. The models with high complexity require more calculation and powerful and expensive hardware for training and test phases. The new studies need to focus not only on higher performance but on the lower model complexity. Some domains with the real-time application demand light models with mediocre performance. For instance, in [44] a compact and light saliency prediction model with acceptable performance has been proposed for real-time applications on CPU.As can be seen in the second row of Figure6, based on the given ground truth image, an observer finds the man’s face and the plastic bag as the salient objects of the input scene, but saliency models including our GLG models were not able to detect the bag as a salient object. It is mainly due to the partial occlusion of the plastic bag. None of the saliency models in Figure 6 perceived the connection between the man and the bag in his hand. As a result, we can conclude that complex backgrounds and partially occluded objects are two big challenges for saliency models. Another example of the partially occluded salient object is the third cow in the first input image in Figure 6. The head of the cow is occluded and as a result, none of the saliency models in Figure 6 (including our GLG models) could find it as a salient object. On the other hand, the human brain can easily identify the brown spot behind the second cow as the third cow by semantically completing missing parts in partially occluded objects.


## 8. Conclusion

In this study, we proposed two novel saliency models to predict human attention during scene free-viewing of natural scenes. To investigate the effectiveness of our methods, we used the SAM-ResNet [14] as the base model. We extended the base model using our proposed methods to inject contextual cues and capture location-dependent patterns/features in order to overcome the deficiencies of CNN structures in the base model. In our first approach, a novel fully connected component is used to incorporate the location-dependent and global scene properties. In the second approach, a VGGNet is employed to extract the contextual information of the scene.Experimental results showed that our GLG models outperform not only the base model but also most previous saliency models over SALICON 2015, SALICON 2017, and MIT300 datasets. Our effort to incorporate the contextual information and global scene properties may supply new inspirations for future works on saliency models to apply such an amendment to the computational saliency models.



---

*Source: 3279325-2022-08-25.xml*