# Inferring Visual Perceptual Object by Adaptive Fusion of Image Salient Features

Authors: Xin Xu; Nan Mu; Hong Zhang
Journal: Mathematical Problems in Engineering
 (2015)

---

## Abstract

Saliency computational model with active environment perception can be useful for many applications including image retrieval, object recognition, and image segmentation. Previous work on bottom-up saliency computation typically relies on hand-crafted low-level image features. However, the adaptation of saliency computational model towards different kinds of scenes remains a challenge. For a low-level image feature, it can contribute greatly on some images but may be detrimental for saliency computation on other images. In this work, a novel data driven approach is proposed to adaptively select proper features for different kinds of images. This method exploits low-level features containing the most distinguishable salient information per image. Then the image saliency can be calculated based on the adaptive weight selection scheme. A large number of experiments are conducted on the MSRA database to compare the performance of the proposed method with the state-of-the-art saliency computational models.


---

## Body

## 1. Introduction

Saliency computational model with active environment perception can be useful for many applications including image retrieval, object recognition, and image segmentation. Generally, visual saliency can be defined as what captures human perceptual attention. Saliency detection plays an important role in image analysis and processing, which is capable of allocating the limited resources effectively. For example, the detecting of visual saliency can be effectively used to automatically zoom the “interesting” areas [1] or automatically crop the “important” areas in an image [2]. Object recognition algorithms can use the results of saliency detection to quickly locate the position of visual salient objects. Salient object detection can also reduce the interference of cluttered background to further improve the performance of image segmentation algorithm and image retrieval system [3].Most of the existing saliency computational models are based on the bottom-up mechanism because the visual attention is generally driven by the low-level stimulus such as edge [4], color [5, 6], orientation [7], and symmetry [8]. These models typically contain two main procedures in saliency computation. The first step extracts low-level features from input image. Then the saliency map can be computed by fusing the extracted low-level features. In the past, low-level image features have been extensively studied. However, the selecting of proper features containing saliency information per-image is still complex and difficult to determine. The reason is mainly due to the lack of a well-defined feature which can exhaustively interpret saliency information in different images. Most of the existing saliency computational models face great difficulties in adaptively selecting low-level image features towards different images.Aiming to address this problem, this paper puts forward an adaptive fusion scheme towards low-level image features for saliency detection. This method firstly extracts various low-level features to largely reflect saliency information of image. Then, the visual perceptual object can be detected by the adaptive weight selection towards these low-level image features. This method can retain the most significant low-level image features for saliency computation. The flowchart of the proposed method is illustrated in Figure1.Figure 1
Flowchart of the proposed saliency computational model.


## 2. Related Works

Visual saliency reflects how much an image region or object stands out from its surrounding. Saliency computational model aims to provide a numerical measure indicating the degree of visual saliency, and is useful for a wide range of applications. Consequently, a number of computational approaches for salient object detection have been developed in recent years. Based on the biologically plausible visual attention architecture [9] and the feature integration theory [10], Itti et al. [11] proposed the bottom-up saliency model which depends only on low-level image features. The salient object in image can be determined using dynamic neural networks for feature fusion. Following this model, many state-of-the-art saliency computational models focus on low-level features, such as color and contrast [12, 13].For example, Achanta et al. [14, 15] used the luminance and color features to detect salient object in image. This method calculated the contrast between local image region and its surrounding region and utilized the average color vector difference to obtain the saliency value of image. Aziz and Mertsching [8], Liu et al. [16], and Cheng et al. [17] proposed a contrast computational model based on region segmentation. These methods took use of the image segmentation algorithm to divide the image into different regions according to the homogeneity of different low-level image features such as luminance, texture, and color.Most of the existing saliency computational models are based on low-level image features. Consequently, there are various formulations for exploiting well-defined salient features. Gao and Vasconcelos [18] studied the statistics of nature images and constructed an optimal detector based on the discriminate saliency. This method can effectively integrate the saliency features. Klein and Frintrop [19] detected the salient object by reconstituting the cognitive visual attention model. Their approach computed the saliency of all feature channels in an information-theoretic way. The optimal features can be determined by using Kullback-Leibler divergence (KLD) to fuse different features. Lu et al. [20] put forward the diffusion-based salient object detection model. This model can learn the optimal saliency seeds by combining two kinds of features: the bottom-up saliency maps and midlevel vision cues.However, most of the existing saliency computational models fail to consider the adaptability of different low-level features towards different images. Some low-level features which contribute greatly on some images may actually be detrimental to the detection of saliency on other images. In this paper, a novel data driven approach is proposed to adaptively select proper features for different images. This method exploits the most distinguishable low-level features containing salient information per image. Then the image saliency can be calculated based on the adaptive weight selection scheme.The rest of this paper is organized as follows. Section3 details the extraction of different low-level image features. Section 4 puts forward the adaptive feature fusion scheme. The experimental results are given in Section 5. And finally, the conclusions are drawn in Section 6.


## 3. Low-Level Image Feature Extraction

According to physiological experiments, it can be found that human attention towards image is mainly driven by low-level image features [21]. In order to fully and accurately describe the saliency information in image, ten low-level image features are chosen according to their visual properties. These ten low-level features are lightness, color, contrast, intensity, edge, orientation, shape, gradient, coarseness, and sharpness. In this section, we describe the extraction of these features and analyze the contributions of each feature in saliency computation.The low-level features are extracted using the block-based method from input image (denoted byI), which is divided into 8
×
8 blocks (denoted by b) with 50% overlap. Let F
n
(
I
), n
=
1
,
…
,
10, denote the saliency map corresponding to each low-level feature; then F
n
(
b
) can be used to denote the saliency map of block b.


### 3.1. Lightness Feature Extraction

Lightness is the perception attribute of human vision system for visible objects radiation or glow amount. The lightness feature may affect the performance of other features to a certain extent.The lightness feature measures how the brightness of each block is different from the average brightness of the image. Therefore, the object having a high brightness can be treated as a salient object by the human eye gaze. LetF
1
(
b
) denote the Euclidean distance of the average lightness value between each image block b and the image I. The larger distance stands for the greater lightness value difference. Specific methods are shown as follows.Firstly, the input image is converted from the RGB color space to a uniform LAB color space. Then, the lightness featureF
1
(
b
) can be obtained by(1)
F
1
b
=
L
¯
b
-
L
¯
I
,where L
¯ represents the average lightness component L in LAB color space. The lightness feature can well distinguish the illumination differences of object in image.


### 3.2. Color Feature Extraction

As the salient object has a strong contrast or a strong variation, the color value of the image object is relatively far from the average color value of the whole image. The background region can be seen as the smooth area in which the color value is closed to the average color value. Therefore, the color feature can be extracted by calculating the Euclidean distance of the average color value between each image block and the background region. The color featureF
2
(
b
) of the image block b can be computed via(2)
F
2
b
=
A
-
b
-
A
-
I
2
+
B
-
b
-
B
-
I
2
,where A
- and B
- represent the average A and B color components in LAB color space, respectively.Most of the existing saliency computational models are conducted in the RGB or LAB color feature space. RGB is mainly used for the color representation of the image, while CIELab provides a representation of color that corresponds to how observers perceive chromatic differences. Thus, this method extracts the color feature in the LAB color space.Color feature can simply describe the global distribution of colors in an image. Thus, it can measure the proportion of different colors accounting for the whole image. The color feature is especially suitable for the image which does not take the spatial position of the object into account.


### 3.3. Contrast Feature Extraction

The saliency of the image blockb in image I depends on the difference between the image block and its surrounding, while the visual characteristic of the image block itself does not determine its saliency. Thus, the greater the difference between the image block and its surrounding, the more likely it is a salient region. The calculation method to extract the contrast feature F
3
(
b
) of the image block b is shown as follows.First, convert each block to luminance image block via(3)
l
b
=
i
+
j
b
ε
,where i
=
0.7656, j
=
0.0364, and ε
=
2.2 denote the display conditions of the Adobe RGB color space. The contrast feature of the image block b can be expressed as(4)
F
3
b
=
σ
b
μ
b
,
μ
b
>
0
,
0
,
μ
b
=
0
,where σ
(
b
) and μ
(
b
) represent the standard deviation and the average value of l
(
b
), respectively. The contrast is critically important on the visual attention. Generally, the greater the contrast is, the sharper the image can be.


### 3.4. Intensity Feature Extraction

The intensity feature (denoted byS
(
I
)) can also be treated as a lightness feature and can be obtained by averaging the three RGB color channels (denoted by R
(
I
), G
(
I
), and B
(
I
)). Besides, the intensity feature can also reflect the brightness information and the color variations of image, which can represent human subjective feeling. The intensity feature F
4
(
b
) of image block b can be generated by calculating the Euclidean distance between the intensity value of the image block S
(
b
) and the whole image S
(
I
): (5)
F
4
b
=
S
¯
b
-
S
¯
I
,where S
¯
(
b
) and S
¯
(
I
) represent the average intensity values of S
(
b
) and S
(
I
), respectively. The human eye perception towards the brightness is mainly associated with the luminous intensity of observed objects.


### 3.5. Edge Feature Extraction

The edge refers to a collection of image pixel in which the grayscale intensity has a strong contrast change. Thus, we use the edge feature to capture the image region with dramatic brightness variations. LetE
(
b
) denote the binary map obtained through the Roberts edge detection. The edge feature F
5
(
b
) of the image block b is then achieved by computing the average value of E
(
b
) as(6)
F
5
b
=
μ
E
b
.The edge in image is often associated with the discontinuity of image grayscale.


### 3.6. Orientation Feature Extraction

Orientation feature is similar to the color feature, which is also an integrity feature. Based on the Itti model, the proposed method calculates the orientation feature by running the Gabor filter on grayscale image (denoted byg
(
x
,
y
)) of input image I. The Gabor wavelet can extract the direction feature effectively and can eliminate the redundant information at the same time. The two-dimensional Gabor function G
(
x
,
y
) is shown as follows:(7)
G
λ
,
θ
,
φ
,
σ
,
γ
x
,
y
=
e
x
p
-
x
′
2
+
γ
2
y
′
2
2
σ
2
c
o
s
2
π
x
′
λ
+
φ
,
x
′
=
x
c
o
s
θ
+
y
s
i
n
θ
,
y
′
=
-
x
s
i
n
θ
+
y
c
o
s
θ
,where γ
=
0.5 represents the spatial aspect ratio and σ
=
0.56
λ represents the standard deviation of the Gaussian factor. The parameters λ, φ, and θ denote the wavelength, the phase offset, and the angle, respectively.The 2D Gabor operator based orientation image feature extraction (denoted byO
θ
(
I
),  θ
∈
{
0
°
,
4
5
°
,
9
0
°
,
13
5
°
}) can be obtained using the following form of convolution:(8)
O
θ
I
=
I
x
,
y
∗
G
θ
x
,
y
.LetF
6
(
b
) denote the orientation feature of image block b, which can be obtained by computing the Euclidean distance between the orientation image block O
θ
(
b
) and the orientation image O
θ
(
I
) via(9)
F
6
b
=
∑
θ
=
0
°
,
4
5
°
,
9
0
°
,
13
5
°
O
¯
θ
b
-
O
¯
θ
I
,where O
¯
(
b
) and O
¯
(
I
) represent the average orientation values of O
(
b
) and O
(
I
), respectively. The orientation feature has global properties and thus can enable the saliency information with good feasibility and stability even in complex scenes.


### 3.7. Shape Feature Extraction

The shape feature can be extracted using Hu’s moment invariants, which has the invariant properties of rotation, translation, and scale. The Hu moment invariants are commonly used to identify the large objects in an image and can better describe the shape feature of an object. The proposed method calculates the 7 moments (denoted byM
i
,
i
=
1
,
…
,
7) by Hu’s method, which is obtained by normalizing the central moments through orders two and three.LetM
i
(
b
) and M
i
(
I
),  i
=
1
,
…
,
7, denote the 7 moments of the image block b and the input image I, respectively. Thus, the shape feature (denoted by F
7
(
b
)) of b can be expressed as the Euclidean distance between M
i
(
b
) and M
i
(
I
): (10)
F
7
b
=
∑
i
=
1,2
,
…
,
7
M
i
b
-
M
i
I
.The shape feature is not sensitive to the lightness and the contrast changes and thus can effectively reduce the influence of lightness.


### 3.8. Gradient Feature Extraction

The gradient feature is sensitive to the gradient variation; however, it is not sensitive to the grayscale of the image. The image gradient can not only be able to capture the contour, silhouette, and some texture information, but also further weaken the influence of illumination. Letg
(
x
,
y
) be the grayscale at pixel (
x
,
y
) in the image region, with the size M
×
N; the gradient feature (denoted by F
8
(
b
)) of image block b can be calculated by averaging the abscissa squared gradient (denoted by g
x
(
b
)) and the ordinate squared gradient (denoted by g
y
(
b
)) through(11)
g
x
b
=
∑
x
=
0
M
-
2
∑
y
=
0
N
-
1
g
x
+
1
,
y
-
g
x
,
y
2
,
g
y
b
=
∑
x
=
0
M
-
1
∑
y
=
0
N
-
2
g
x
,
y
+
1
-
g
x
,
y
2
,
F
8
b
=
μ
g
x
b
+
g
y
b
2
.Gradient value can describe the magnitude of the dramatic changes of the pixel values. Thus, the gradient map constituted by the pixel gradient values can reflect the local grayscale changes in image.


### 3.9. Coarseness Feature Extraction

Coarseness is the fundamental perceptual texture feature. It can measure the particle size of the texture pattern. The larger particle size means the coarser image texture. The coarseness feature can be calculated as follows.Firstly, the average gray value (denoted byA
k
(
x
,
y
)
,
i
=
1,2
,
…
,
5) of the neighborhood with size 2
k
×
2
k in image I is calculated as(12)
A
k
x
,
y
=
∑
i
=
x
-
2
k
-
1
x
+
2
k
-
1
-
1
∑
j
=
y
-
2
k
-
1
y
-
2
k
-
1
-
1
g
x
,
y
2
2
k
,where g
(
x
,
y
) is the gray value at pixel (
x
,
y
) in the active window.Then, for each pixel, the average intensity difference (denoted byE
k
,
h
(
x
,
y
) and E
k
,
v
(
x
,
y
)) can be calculated between the nonoverlapping neighborhoods in the horizontal and vertical directions, respectively:(13)
E
k
,
h
x
,
y
=
A
k
x
+
2
k
-
1
,
y
-
A
k
x
-
2
k
-
1
,
y
,
E
k
,
v
x
,
y
=
A
k
x
,
y
+
2
k
-
1
-
A
k
x
,
y
-
2
k
-
1
.Finally, the optimal size can be set byk which gives the highest value of E. The coarseness (denoted by C) is the average of S
b
e
s
t
(
x
,
y
)
=
2
k:(14)
C
=
1
m
×
n
∑
x
=
1
m
∑
y
=
1
n
S
b
e
s
t
x
,
y
.LetC
(
b
) and C
(
I
) denote the coarseness of the image block b and image I, respectively. Thus, the coarseness feature (denoted by F
9
(
I
)) of b can be expressed as the Euclidean distance between C
(
b
) and C
(
I
): (15)
F
9
b
=
C
b
-
C
I
.Coarseness feature represents the surface properties of the whole image and can well describe the integrity of the salient object. Meanwhile, the coarseness feature has good rotation invariance; it can effectively resist the interference of noise.


### 3.10. Sharpness Feature Extraction

The sharpness feature measures how the acutance of each region is different from its surrounding, which can indicate the contrast of the adjacent region. The proposed method extracts the sharpness valueϕ
(
p
) at position p by computing the convolution between the input image I and the first-order derivatives of the Gaussian via(16)
ϕ
p
=
∑
x
∑
y
g
x
,
y
∗
G
x
σ
x
,
y
2
+
g
x
,
y
∗
G
y
σ
x
,
y
2
,where g
(
x
,
y
) is the grayscale at pixel (
x
,
y
) in the image region. G
x
σ
(
x
,
y
) and G
y
σ
(
x
,
y
) represent the first-order derivatives of the Gaussian in the vertical and the horizontal directions, respectively. σ is the scale of the Gaussian filter.The Gaussian derivative method can measure the acutance variation and suppress the influence of noise and illumination. Letϕ
¯
(
b
) and ϕ
¯
(
I
) denote the average sharpness value of the image block b and the input image I, respectively. Thus, the sharpness feature (denoted by F
10
(
b
)) of b can be expressed as the Euclidean distance between ϕ
¯
(
b
) and ϕ
¯
(
I
): (17)
F
10
b
=
ϕ
¯
b
-
ϕ
¯
I
.Sharpness represents the image definition and the edge acuteness. The higher the sharpness is, the higher the image contrast can be. The sharpness feature can be less susceptible to local variations.


## 3.1. Lightness Feature Extraction

Lightness is the perception attribute of human vision system for visible objects radiation or glow amount. The lightness feature may affect the performance of other features to a certain extent.The lightness feature measures how the brightness of each block is different from the average brightness of the image. Therefore, the object having a high brightness can be treated as a salient object by the human eye gaze. LetF
1
(
b
) denote the Euclidean distance of the average lightness value between each image block b and the image I. The larger distance stands for the greater lightness value difference. Specific methods are shown as follows.Firstly, the input image is converted from the RGB color space to a uniform LAB color space. Then, the lightness featureF
1
(
b
) can be obtained by(1)
F
1
b
=
L
¯
b
-
L
¯
I
,where L
¯ represents the average lightness component L in LAB color space. The lightness feature can well distinguish the illumination differences of object in image.


## 3.2. Color Feature Extraction

As the salient object has a strong contrast or a strong variation, the color value of the image object is relatively far from the average color value of the whole image. The background region can be seen as the smooth area in which the color value is closed to the average color value. Therefore, the color feature can be extracted by calculating the Euclidean distance of the average color value between each image block and the background region. The color featureF
2
(
b
) of the image block b can be computed via(2)
F
2
b
=
A
-
b
-
A
-
I
2
+
B
-
b
-
B
-
I
2
,where A
- and B
- represent the average A and B color components in LAB color space, respectively.Most of the existing saliency computational models are conducted in the RGB or LAB color feature space. RGB is mainly used for the color representation of the image, while CIELab provides a representation of color that corresponds to how observers perceive chromatic differences. Thus, this method extracts the color feature in the LAB color space.Color feature can simply describe the global distribution of colors in an image. Thus, it can measure the proportion of different colors accounting for the whole image. The color feature is especially suitable for the image which does not take the spatial position of the object into account.


## 3.3. Contrast Feature Extraction

The saliency of the image blockb in image I depends on the difference between the image block and its surrounding, while the visual characteristic of the image block itself does not determine its saliency. Thus, the greater the difference between the image block and its surrounding, the more likely it is a salient region. The calculation method to extract the contrast feature F
3
(
b
) of the image block b is shown as follows.First, convert each block to luminance image block via(3)
l
b
=
i
+
j
b
ε
,where i
=
0.7656, j
=
0.0364, and ε
=
2.2 denote the display conditions of the Adobe RGB color space. The contrast feature of the image block b can be expressed as(4)
F
3
b
=
σ
b
μ
b
,
μ
b
>
0
,
0
,
μ
b
=
0
,where σ
(
b
) and μ
(
b
) represent the standard deviation and the average value of l
(
b
), respectively. The contrast is critically important on the visual attention. Generally, the greater the contrast is, the sharper the image can be.


## 3.4. Intensity Feature Extraction

The intensity feature (denoted byS
(
I
)) can also be treated as a lightness feature and can be obtained by averaging the three RGB color channels (denoted by R
(
I
), G
(
I
), and B
(
I
)). Besides, the intensity feature can also reflect the brightness information and the color variations of image, which can represent human subjective feeling. The intensity feature F
4
(
b
) of image block b can be generated by calculating the Euclidean distance between the intensity value of the image block S
(
b
) and the whole image S
(
I
): (5)
F
4
b
=
S
¯
b
-
S
¯
I
,where S
¯
(
b
) and S
¯
(
I
) represent the average intensity values of S
(
b
) and S
(
I
), respectively. The human eye perception towards the brightness is mainly associated with the luminous intensity of observed objects.


## 3.5. Edge Feature Extraction

The edge refers to a collection of image pixel in which the grayscale intensity has a strong contrast change. Thus, we use the edge feature to capture the image region with dramatic brightness variations. LetE
(
b
) denote the binary map obtained through the Roberts edge detection. The edge feature F
5
(
b
) of the image block b is then achieved by computing the average value of E
(
b
) as(6)
F
5
b
=
μ
E
b
.The edge in image is often associated with the discontinuity of image grayscale.


## 3.6. Orientation Feature Extraction

Orientation feature is similar to the color feature, which is also an integrity feature. Based on the Itti model, the proposed method calculates the orientation feature by running the Gabor filter on grayscale image (denoted byg
(
x
,
y
)) of input image I. The Gabor wavelet can extract the direction feature effectively and can eliminate the redundant information at the same time. The two-dimensional Gabor function G
(
x
,
y
) is shown as follows:(7)
G
λ
,
θ
,
φ
,
σ
,
γ
x
,
y
=
e
x
p
-
x
′
2
+
γ
2
y
′
2
2
σ
2
c
o
s
2
π
x
′
λ
+
φ
,
x
′
=
x
c
o
s
θ
+
y
s
i
n
θ
,
y
′
=
-
x
s
i
n
θ
+
y
c
o
s
θ
,where γ
=
0.5 represents the spatial aspect ratio and σ
=
0.56
λ represents the standard deviation of the Gaussian factor. The parameters λ, φ, and θ denote the wavelength, the phase offset, and the angle, respectively.The 2D Gabor operator based orientation image feature extraction (denoted byO
θ
(
I
),  θ
∈
{
0
°
,
4
5
°
,
9
0
°
,
13
5
°
}) can be obtained using the following form of convolution:(8)
O
θ
I
=
I
x
,
y
∗
G
θ
x
,
y
.LetF
6
(
b
) denote the orientation feature of image block b, which can be obtained by computing the Euclidean distance between the orientation image block O
θ
(
b
) and the orientation image O
θ
(
I
) via(9)
F
6
b
=
∑
θ
=
0
°
,
4
5
°
,
9
0
°
,
13
5
°
O
¯
θ
b
-
O
¯
θ
I
,where O
¯
(
b
) and O
¯
(
I
) represent the average orientation values of O
(
b
) and O
(
I
), respectively. The orientation feature has global properties and thus can enable the saliency information with good feasibility and stability even in complex scenes.


## 3.7. Shape Feature Extraction

The shape feature can be extracted using Hu’s moment invariants, which has the invariant properties of rotation, translation, and scale. The Hu moment invariants are commonly used to identify the large objects in an image and can better describe the shape feature of an object. The proposed method calculates the 7 moments (denoted byM
i
,
i
=
1
,
…
,
7) by Hu’s method, which is obtained by normalizing the central moments through orders two and three.LetM
i
(
b
) and M
i
(
I
),  i
=
1
,
…
,
7, denote the 7 moments of the image block b and the input image I, respectively. Thus, the shape feature (denoted by F
7
(
b
)) of b can be expressed as the Euclidean distance between M
i
(
b
) and M
i
(
I
): (10)
F
7
b
=
∑
i
=
1,2
,
…
,
7
M
i
b
-
M
i
I
.The shape feature is not sensitive to the lightness and the contrast changes and thus can effectively reduce the influence of lightness.


## 3.8. Gradient Feature Extraction

The gradient feature is sensitive to the gradient variation; however, it is not sensitive to the grayscale of the image. The image gradient can not only be able to capture the contour, silhouette, and some texture information, but also further weaken the influence of illumination. Letg
(
x
,
y
) be the grayscale at pixel (
x
,
y
) in the image region, with the size M
×
N; the gradient feature (denoted by F
8
(
b
)) of image block b can be calculated by averaging the abscissa squared gradient (denoted by g
x
(
b
)) and the ordinate squared gradient (denoted by g
y
(
b
)) through(11)
g
x
b
=
∑
x
=
0
M
-
2
∑
y
=
0
N
-
1
g
x
+
1
,
y
-
g
x
,
y
2
,
g
y
b
=
∑
x
=
0
M
-
1
∑
y
=
0
N
-
2
g
x
,
y
+
1
-
g
x
,
y
2
,
F
8
b
=
μ
g
x
b
+
g
y
b
2
.Gradient value can describe the magnitude of the dramatic changes of the pixel values. Thus, the gradient map constituted by the pixel gradient values can reflect the local grayscale changes in image.


## 3.9. Coarseness Feature Extraction

Coarseness is the fundamental perceptual texture feature. It can measure the particle size of the texture pattern. The larger particle size means the coarser image texture. The coarseness feature can be calculated as follows.Firstly, the average gray value (denoted byA
k
(
x
,
y
)
,
i
=
1,2
,
…
,
5) of the neighborhood with size 2
k
×
2
k in image I is calculated as(12)
A
k
x
,
y
=
∑
i
=
x
-
2
k
-
1
x
+
2
k
-
1
-
1
∑
j
=
y
-
2
k
-
1
y
-
2
k
-
1
-
1
g
x
,
y
2
2
k
,where g
(
x
,
y
) is the gray value at pixel (
x
,
y
) in the active window.Then, for each pixel, the average intensity difference (denoted byE
k
,
h
(
x
,
y
) and E
k
,
v
(
x
,
y
)) can be calculated between the nonoverlapping neighborhoods in the horizontal and vertical directions, respectively:(13)
E
k
,
h
x
,
y
=
A
k
x
+
2
k
-
1
,
y
-
A
k
x
-
2
k
-
1
,
y
,
E
k
,
v
x
,
y
=
A
k
x
,
y
+
2
k
-
1
-
A
k
x
,
y
-
2
k
-
1
.Finally, the optimal size can be set byk which gives the highest value of E. The coarseness (denoted by C) is the average of S
b
e
s
t
(
x
,
y
)
=
2
k:(14)
C
=
1
m
×
n
∑
x
=
1
m
∑
y
=
1
n
S
b
e
s
t
x
,
y
.LetC
(
b
) and C
(
I
) denote the coarseness of the image block b and image I, respectively. Thus, the coarseness feature (denoted by F
9
(
I
)) of b can be expressed as the Euclidean distance between C
(
b
) and C
(
I
): (15)
F
9
b
=
C
b
-
C
I
.Coarseness feature represents the surface properties of the whole image and can well describe the integrity of the salient object. Meanwhile, the coarseness feature has good rotation invariance; it can effectively resist the interference of noise.


## 3.10. Sharpness Feature Extraction

The sharpness feature measures how the acutance of each region is different from its surrounding, which can indicate the contrast of the adjacent region. The proposed method extracts the sharpness valueϕ
(
p
) at position p by computing the convolution between the input image I and the first-order derivatives of the Gaussian via(16)
ϕ
p
=
∑
x
∑
y
g
x
,
y
∗
G
x
σ
x
,
y
2
+
g
x
,
y
∗
G
y
σ
x
,
y
2
,where g
(
x
,
y
) is the grayscale at pixel (
x
,
y
) in the image region. G
x
σ
(
x
,
y
) and G
y
σ
(
x
,
y
) represent the first-order derivatives of the Gaussian in the vertical and the horizontal directions, respectively. σ is the scale of the Gaussian filter.The Gaussian derivative method can measure the acutance variation and suppress the influence of noise and illumination. Letϕ
¯
(
b
) and ϕ
¯
(
I
) denote the average sharpness value of the image block b and the input image I, respectively. Thus, the sharpness feature (denoted by F
10
(
b
)) of b can be expressed as the Euclidean distance between ϕ
¯
(
b
) and ϕ
¯
(
I
): (17)
F
10
b
=
ϕ
¯
b
-
ϕ
¯
I
.Sharpness represents the image definition and the edge acuteness. The higher the sharpness is, the higher the image contrast can be. The sharpness feature can be less susceptible to local variations.


## 4. Adaptive Fusion of Low-Level Image Features

According to the discrete degree and clarity of the saliency map of each low-level feature, different weights are assigned to different features. Thus, low-level features that contribute greatly on some images will assign a large weight; other low-level features that may be detrimental to the detection of saliency will assign a low weight or completely ignore during the saliency computation. Letν
n denote the statistical validity and ω
n denote the weights of different feature maps F
n
(
I
),  n
=
1,2
,
…
,
10. The ν
n can be defined as(18)
ν
n
=
σ
n
2
+
κ
n
,where σ
n
2 and κ
n represent the variance and the kurtosis of F
n
(
I
), respectively.The weightsω
n of different feature maps F
n
(
I
) are determined by the numerical magnitudes of the statistical validity ν
n via(19)
ω
n
=
1
,
i
f
ν
n
=
ν
1
∗
4
5
,
i
f
ν
n
=
ν
2
∗
3
5
,
i
f
ν
n
=
ν
3
∗
2
5
,
i
f
ν
n
=
ν
4
∗
1
5
,
i
f
ν
n
=
ν
5
∗
0
,
o
t
h
e
r
w
i
s
e
,where ν
i
∗
=
s
o
r
t
{
ν
i
} using the descending order. The proposed method assigns different weights according to numerical sort of ν
n.The final fusion map (denoted byF
I) is calculated by the weighted sum of the ten feature maps F
n
(
I
),  n
=
1,2
,
…
,
10:(20)
F
I
=
∑
n
w
n
F
n
I
∑
n
w
n
.To enhance the robustness of detection process and achieve a preferable visual effect, the proposed method is performed in three scales100
%
,
50
%
,
25
% to better restrain the background information.The obtained saliency map is then refined using center prior principle to enhance visual effect. When humans watch a picture, they will naturally gaze on the objects next to the center of image [22]. Thus, in order to obtain the saliency objects closer to the human visual fixations, more weight is needed to add to the center of image. Therefore, a feature (denoted by f
c
(
b
)) is included to indicate the distance between each image block and the center of image. For each image block feature F
n
(
b
), n
=
1
,
…
,
10, it can be recalculated via(21)
F
n
∗
b
=
F
n
b
f
c
b
,
f
c
b
=
1
-
r
b
-
C
∗
2
+
c
b
-
R
∗
2
M
/
2
2
+
N
/
2
2
,where (
r
(
b
)
,
c
(
b
)
) denote the upper-left coordinate of image block b, (
C
∗
,
R
∗
) represent the center coordinate of the estimated image region, and M and N denote the width and height of the image region, respectively.The feature mapF
n
(
I
) of the whole image I can be generated by combining the image feature F
n
∗
(
b
) of all the blocks. The normalized feature map is calculated by (22)
F
n
I
=
F
n
I
-
m
i
n
F
n
I
m
a
x
F
n
I
-
m
i
n
F
n
I
.Finally, the generated saliency map is smoothed by a Gaussian filter (the template size is10
×
10, and σ is 2.5). The saliency maps of each low-level feature and the final saliency maps are shown in Figure 2.Figure 2
Saliency maps of each low-level features. (a) Input image, ((b)–(k)) corresponding saliency maps of low-level image features respectively, (l) the final saliency map.
(a)
Input
(b)
Lightness
(c)
Color
(d)
Contrast
(e)
Intensity
(f)
Edge
(g)
Orientation
(h)
Shape
(i)
Gradient
(j)
Coarseness
(k)
Sharpness
(l)
Final mapAs illustrated in Figure2, the ten low-level image features have their own advantages and disadvantages towards different images. On the contrary, the final saliency map can adaptively fuse the optimal features to achieve better performance.In addition, we also provide the statistical of the ten low-level image features in different images. The result from different images is shown in Figure3. As illustrated in Figure 3, the sharpness feature and the shape feature have good stability in our testing. Their weights have little change due to the deviation caused by local contrast transformation. On the contrary, the gradient feature can only reflect the local differences.Figure 3
The average weight distribution of the ten low-level image features.


## 5. Experimental Results

The performance evaluation is conducted using the MSRA salient object database [16]. This database contains over 20 000 images, which includes two parts: (i) image set A, containing 20,000 images, and the principle salient objects are labeled by three users, and (ii) image set B, containing 5,000 images, and the principle salient objects are labeled by nine users. The proposed multifeature fusion (MF) approach is compared with the other seven state-of-the-art methods:Itti’s (IT) method [11],spectral residual (SR) method [23],saliency using natural statistics (SUN) method [24],frequency-tuned (FT) method [25],S3 method [26],nonparametric (NP) method [27], andcontext-aware (CA) method [28].Figure4 illustrates the performance comparison of these various salient region detection methods. As can be seen from Figure 4, the saliency maps extracted using our proposed method are more consistent with the ground-truth rectangle, and the detected saliency objects are more similar to the ground-truth binary masks. The approaches developed in [23, 24, 26] fail to detect and clearly identify the location of salient object from complex background. The saliency maps generated using methods [11, 27] look rather blurry and are difficult to clearly distinguish the salient region. The saliency map of method [25] retains a lot of background information. The CA method [28] can achieve good detection performance on some images; however this method is unable to highlight the entire salient object itself.Figure 4
Saliency maps obtained from the proposed method (MF) and state-of-the-art saliency computational models.
(a)
Input
(b)
Labeled
(c)
Binary
(d)
IT [11]
(e)
SR [23]
(f)
SUN [24]
(g)
FT [25]
(h)
S3 [26]
(i)
NP [27]
(j)
CA [28]
(k)
MFThe objective assessment is implemented by computing thetrue positive rate (TPR) and thefalse positive rate (FPR). Given the ground-truth binary masks G
(
x
,
y
) and the obtained saliency map F
I
(
x
,
y
)
(
0
≤
F
I
(
x
,
y
)
≤
1
), a threshold t
(
0
≤
t
≤
1
) is used to obtain the binary masks B
t
(
x
,
y
), in which 0 denotes the background and 1 denotes the salient objects. The TPR and FPR can be computed via(23)
T
P
R
=
E
∏
t
B
t
x
,
y
·
G
x
,
y
,
F
P
R
=
E
∏
t
1
-
B
t
x
,
y
·
G
x
,
y
.Figure5 shows the TPR and FPR results of the seven methods and the proposed method. As seen in Figure 5, the overall performance of the proposed method is more excellent than the other seven methods.Figure 5
ROC curves obtained from the proposed method and state-of-the-art saliency computational models.Given the generated saliency mapF
I
(
x
,
y
), we set a threshold T (computed by Otsu’s method) to segment the saliency objects. The binary mask is denoted by B
I
(
x
,
y
).LetG and B
I denote the ground-truth binary masks and binary masks of the proposed approach, respectively. The P
r
e
c
i
s
i
o
n
=
R
(
B
I
∩
G
)
/
R
(
B
I
) and R
e
c
a
l
l
=
R
(
B
I
∩
G
)
/
R
(
G
), where R
(
·
) represent the salient region. The evaluation criterion F-measure can be computed via:(24)
F
β
=
1
+
β
2
P
r
e
c
i
s
i
o
n
×
R
e
c
a
l
l
β
2
×
P
r
e
c
i
s
i
o
n
+
R
e
c
a
l
l
.The proposed method usesβ
2
=
0.5 to weigh the precision and recall. The precision, recall, and F-measure of these methods are shown in Table 1.Table 1
The precision, recall, andF-measure of the saliency computational models.
IT [11]
SR [23]
SUN [24]
FT [25]
S3 [26]
NP [27]
CA [28]
MF
Precision
0.3548
0.5662
0.2769
0.6845
0.2159
0.2132
0.4894
0.7155
Recall
0.9477
0.4177
0.7491
0.6519
0.6991
0.9522
0.8770
0.8150
F-measure
0.4391
0.4951
0.3291
0.6443
0.2715
0.2781
0.5580
0.7370Finally, we compare the computational complexity of the different saliency computational models discussed. These models are implemented using the MATLAB programming language and run on a PC with a Pentium G2020 CPU and a 4 GB RAM. Table2 shows the results of the proposed method and the other methods. The proposed method yields slightly higher computational load than the conventional approach; however, the proposed method can achieve more accurate saliency detection in various images.Table 2
Average execution time (in seconds) for the saliency computational models.
IT [11]
SR [23]
SUN [24]
FT [25]
S3 [26]
NP [27]
CA [28]
MF
Run time
1.0047
0.1810
1.7783
0.1710
8.4211
1.4895
68.3333
10.3720


## 6. Conclusion

In this paper, a novel feature selection scheme is proposed to adaptively select the proper features for different images. This method exploits the most distinguishable salient information in ten low-level features per image. The generated saliency map can highlight the salient object in different images even containing complex background. A large number of experiments are conducted on the MSRA database to compare the performance of the proposed method with the state-of-the-art saliency computational models. And the experimental results indicate that the proposed method outperforms the state-of-the-art saliency computational models to achieve better performance.



---

*Source: 973241-2015-08-27.xml*