1896

IEEE TRANSACTIONS ON MULTIMEDIA, VOL. 15, NO. 8, DECEMBER 2013

Co-Salient Object Detection From Multiple Images
Hongliang Li, Senior Member, IEEE, Fanman Meng, and King Ngi Ngan, Fellow, IEEE

Abstract—In this paper, we propose a novel method to discoverco-salient objects from a group of images, which is modeled asa linear fusion of an intra-image saliency (IaIS) map and aninter-image saliency (IrIS) map. The first term is to measure thesalient objects from each image using multiscale segmentation
voting. The second term is designed to detect the co-salient objects
from a group of images. To compute the IrIS map, we performthe pairwise similarity ranking based on an image pyramid
representation. A minimum spanning tree is then constructed todetermine the image matching order. For each region in an image,
we design three types of visual descriptors, which are extracted
from the local appearance, e.g., color, color co-occurrence andshape properties. The final region matching problem between
the images is formulated as an assignment problem that can be
optimized by linear programming. Experimental evaluation ona number of images demonstrates the good performance of the
proposed method on co-salient object detection.
Index Terms—Attention model, co-saliency, minimum spanningtree, similarity.

I. INTRODUCTION

H

UMANS have an extraordinary ability to rapidly scan aset of images and fixate their attention on the most valuable information (e.g., similar entity). This fixation ability canbe viewed as visual co-attention which can be carried out bothin a fast, saliency-driven and bottom-up manner, as well as in a
top-down processing and memory-dependent manner [1], [2].
Visual co-saliency is a subjective perceptual quality thatmakes similar objects in a group of images stand out from theirneighbors and grab our attention by visually co-salient stimuli[1]. A co-salient region usually exhibits the following properties, i.e., 1) a salient region in an image should be prominent or
noticeable with respect to its surroundings. 2) high similarity
can be observed for such regions with respect to certain features(e.g., intensity, color, texture or shape). Co-saliency detection
is a key attentional mechanism by allocating the perceptualand cognitive resources to the most relative common data
while ignoring other dissimilar contents. Two examples canManuscript received October 10, 2012; revised January 27, 2013 and April
15, 2013; accepted April 15, 2013. Date of publication June 27, 2013; date ofcurrent version November 13, 2013. This work was supported in part by NSFC
(No. 61271289), National High Technology Research and Development Program of China (863 Program, No. 2012AA011503), and The Ph.D. Programs
Foundation of Ministry of Education of China (No. 20110185110002). The associate editor coordinating the review of this manuscript and approving it for
publication was Dr. Cees Snoek.
H. Li and F. Meng are with the School of Electronic Engineering, University of Electronic Science and Technology of China, Chengdu 610073, China(e-mail: hlli@uestc.edu.cn; fanmanmeng@yahoo.com).
K. N. Ngan is with the Department of Electronic Engineering, The Chinese
University of Hong Kong, Shatin, Hong Kong, and also with the School of Electronic Engineering, University of Electronic Science and Technology of China,
Chengdu 610073, China (e-mail: knngan@ee.cuhk.edu.hk).
Color versions of one or more of the figures in this paper are available online
at http://ieeexplore.ieee.org.
Digital Object Identifier 10.1109/TMM.2013.2271476

Fig. 1. Examples of visual co-saliency detection. First and Third rows: Twosets of images that contain co-salient objects footballer and cat, respectively.
Second and Fourth rows: Our co-saliency maps.

be found in Fig. 1, where co-salient objects footballer and cat
attract more attention than others in the image groups. The
corresponding co-saliency maps by our method can be foundin the second and the last rows of Fig. 1.
In this paper, we focus on a co-saliency driven attention
model which is to simulate the attention search process ina group of images. Two saliency maps namely intra-image
saliency (IaIS) and inter-image saliency (IrIS) are defined tomodel the proposed visual co-saliency via a linear combination.
The first term utilizes multiscale segmentation to detect thesalient objects from each image, while the second term is tomeasure the co-salient objects from a set of images. In our work,
we build an image pyramid representation to perform pairwise
similarity ranking, which is then employed to construct a minimum spanning tree for image matching. Three types of visualdescriptors (i.e., color, color co-occurrence and shape descriptors) are defined to represent the region aspects in an image.
The final region matching problem between the images canbe solved by linear programming as an assignment problem.
Experimental evaluation on many public image datasets showsthat the proposed method can detect co-saliency effectively,
and can be easily extended to the image co-segmentation task.
This paper is organized as follows. The related work is briefly
described in Section II. Section III introduces our proposed algorithm for co-saliency detection. Experimental results are provided in Section IV to demonstrate the effectiveness of our approach. Finally, Section V concludes this paper.
II. RELATED WORKIn the last decade, a number of methods have been presented
to identify the visual saliency from an image, which can be applied in many fields, such as image search [3], image retargeting[4] and segmentation [5]. Generally, visual saliency can be classified into two categories, i.e., local and global models. The local
saliency model focuses on the extraction of local contrast features, which aims to find the local salient region that stands out

1520-9210 © 2013 IEEE

LI et al.: CO-SALIENT OBJECT DETECTION FROM MULTIPLE IMAGES

from its neighborhood. Based on a biologically-plausible model
proposed by Koch and Ullman [6], Itti et al. [2] first presented
a saliency-based visual attention model for rapid scene analysis, which combined multiscale image features into a single
topographical saliency map. This model was successfully extended to segment video objects of interest such as the facialsaliency model [7] and the focused saliency model [8]. Inspiredby the biologically plausible and the center-surround mechanisms, various local saliency models have been proposed tomeasure the local saliency from different aspects, such as graphbased visual saliency [9], site entropy rate saliency [10], featurelearning based saliency [11] and local region contrast methods[12], [13]. Unlike the local model, global saliency model is tomeasure a pixel’s contrast to all pixels within an image. The
global contrast feature can be computed from the frequency or
spatial domain, such as log-spectrum (SR) [14], frequency tuned(FT) [15], symmetric surrounds [16], context-aware (CA) [17]and histogram/region based contrast [18], [19]. Recently, someworks aim to incorporate the high level knowledge to detect thesaliency from a single image [20]–[22], which show good performance compared with low-level feature based methods.
Unlike the single image saliency models, visual co-saliency is
to discover co-salient objects from a set of images. It is the subjective perceptual quality that implies a selection and/or ranking
by importance and makes similar or common objects in a set ofimages stand out from their neighbors. It is known that common
object detection from a set of images has become one of themost important tasks in computer vision, such as common pattern discovery [23], [24], image matching [25] and co-recognition [26]. Generally, the visual co-saliency map can be simply
obtained by performing the single image saliency detection for
all images. However, the inherent similarity cue between the images will be ignored, which may result in poor performance afterthe ‘blind’ co-saliency detection. Recently, a co-saliency model
has been proposed to simulate the attention search process for animage pair [1]. This model first combines three existing saliency
techniques to generate a local saliency map. Then a co-multilayer graph and the normalized single-pair SimRank algorithm
are employed to find the co-salient objects from the image pair.
Good performance can be observed for the co-saliency detection of a pair of images. However, high computational load isneeded, which is difficult to apply to a number of images. Although the similar goal of detecting the co-saliency can be observed, we can see that the proposed method is significantly
different from the previous work [1] in the entire framework.
Firstly, the proposed method mainly focuses on co-saliency detection from a group of images instead of an image pair [1]. Secondly, the proposed method proposes a new method to measure
the intra-saliency using multiscale segmentation voting, while
the work [1] simply combines the existing saliency detection
techniques. Thirdly, unlike the work [1], the proposed method
designs a different framework to compute the inter-saliency map
via a minimum spanning tree. Finally, the proposed method defines three powerful types of visual descriptors, i.e., color, color
co-occurrence and shape properties, which are distinctly different from the previous work [1].
In addition, a similar work with our approach is called ‘cosegmentation’ that aims to segment the common regions from images [27]–[34]. This method can be traced back to the work of

1897

TABLE ICOMPARISON OF A NUMBER OF DIFFERENTSALIENT OBJECT EXTRACTION MODELS

Rother [27] which cosegmented the common parts of an image
pair by histogram matching. Mukherjee et al. [28] extended thiswork by adding the histogram constraint as the regularized termin a MRF energy function for the simultaneous segmentation.
Instead of penalizing the difference of the two foreground histograms, Hochbaum and Singh [29] turned to reward their similarity using a maximum flow procedure on an appropriately
constructed graph, which leaded to a polynomial time algorithm for cosegmentation. Joulin et al. [30] proposed a discriminative clustering framework for image cosegmentation, which
combines the existing tools for bottom up image segmentationsuch as the spectral clustering technique and positive definite
kernels. Recently, a number of cosegmentation methods havebeen proposed to extract the common regions from multiple images based on different optimization models [33]–[37]. Compared with the co-saliency mode, distinct differences can beobserved between co-saliency and cosegmentation tasks, i.e.,
cosegmentation usually performs in supervised or weakly supervised manner, where several object-like proposals should beobtained before object cosegmentation. In other words, cosegmentation can extract a specific common object by combining
the object classifier even the object is not salient in the images.
In addition, the co-salient result can be easily extended to thecosegmentation task if we take the co-saliency map as the object-like proposal. Table I summarizes the differences between
these related works and the co-salient object detection.
III. PROPOSED METHOD
The co-saliency defined in our paper is obtained by computing the intra-image saliency and inter-image saliency maps.
The first is used to identify the salient regions within each image.
The second aims to measure the saliency of a set of images. The
framework of our proposed method is illustrated in Fig. 2.
A. Intra-Image Saliency (IaIS)
In the current literature, most saliency models focus on thelocal or global contrast mechanism for the salient region detection. Generally, good results can be achieved when the salient
object is surrounded by a uniform background. However, it isstill a challenging task to identify the salient object from complex scenes. In order to achieve robust saliency detection, a newsaliency detection method is proposed in our work, which aims
to improve detection performance based on the multiscale segmentation voting.
1) Multiscale Object Segmentation: Multiple or over segmentation is widely used in saliency detection [38], image categorization [27] and object recognition [39] as an important pre-

1898

IEEE TRANSACTIONS ON MULTIMEDIA, VOL. 15, NO. 8, DECEMBER 2013

Fig. 2. The framework of our proposed method.

Fig. 4. The predefined segmentation windows. Left: Basic partition modes.
Right: Generated segmentation windows (yellow region).
Fig. 3. First row: Superpixel results by the method [40]. Second row: Fourcandidate segmentation windows with red color. Third row: Corresponding segmentation results by Grab-Cut.

cessing step. After over segmentation, an image can be divided
into a lot of homogeneous regions called “superpixels”, which
can reduce the computational cost and avoid undersegmentation [40]. An example can be found in the first row of Fig. 3,
where 30, 10 and 5 superpixels are created from the original
image. Since there is no semantic information linked to thesesuperpixels, it is unknown that which superpixel belongs to thesalient object. In order to discover the possible salient object, we
propose a new method to produce the object superpixels from animage. The idea is motivated by an observation that the salient
object or its part can be successfully segmented from its surroundings using the interactive segmentation algorithms suchas graph-cut [41], lazy snapping [42] or grab-cut [43]. As illustrated in the second row of Fig. 3, we select four rectangular
windows to perform image segmentation based on the Grab-Cut
algorithm [43]. The first window includes most image regions
except for the image boundary, while the others enclose small
parts of the image. The results are presented in the third rowof Fig. 3 which show that only the object regions are extracted
from the predefined windows. It means that most objects can besegmented when we properly choose the candidate segmentation windows.
In order to extract most objects, we define multiscale segmentation windows which are shown in Fig. 4. Note that the leftpart presents the basic partition modes which are used to create
the candidate windows shown in the right part. Here denotes
the scale number. It is seen that different positions and different
sizes of windows are taken into account to perform the object
segmentation. For example, the first window (i.e.,) selects
the whole image as the segmentation window, while the second
window (i.e.,) only takes the center region as the candidate window. To deal with multiple objects, we define several
mixed windows that are listed in the last four cases. For eachwindow, we run the Grab-Cut [43] to perform the binary segmentation, which assigns the foreground pixels with label one
and background with zero. Assume
denote the segmentation
label at the th scale, we have
if
if

(1)

2) Intra-Image Saliency Map: After the multiscale image
segmentation, we extract the objects from an image at various
scales. In other words, an object or its parts may appear in manyLI et al.: CO-SALIENT OBJECT DETECTION FROM MULTIPLE IMAGES

1899

Fig. 5. Examples of the intra image saliency map. First row: Original images.
Second row: Multiscale image segmentation. Third row: Saliency maps.

Fig. 6. Saliency map by the existing methods. First row: Original images. Rows
2–4: Saliency maps by SR [14], FT [15] and RC [19], respectively.

scales due to the overlapped candidate windows. Here, we define that a pixel is said to be salient if this pixel is voted as an object pixel with many times. Assume
denotes the intra
image saliency value at pixel . We have(2)
is defined as the voting number by the multscale
where
segmentations,denotes the total number of defined scales,andis a normalized constant to ensure the IaIS value in therange of [0, 1]. From (2), we can see that if a pixel is voted as
the object label for all scales, it will have the maximum salient
value.
As shown in Fig. 5, four test images are given in the firstrow. The second row presents multiscale segmentation results.
The intra image saliency maps by (2) are shown in the last row,where most objects are highlighted with large salient values.
In addition, we compute the saliency map using the existing
methods, i.e., spectral residual (SR) [14], frequency tuned (FT)[15] and global contrast (RC) [19], which are shown in Fig. 6.
For the first two images with simple backgrounds, most methods
achieve good performance and extract those football players
successfully. However, for the last two images with complex
backgrounds, our method tends to produce more robust results
than the existing methods.
B. Inter-Image Saliency (IrIS)
Visual system relies on several heuristics to direct attention
to important locations and objects [44]. The subject is searching

Fig. 7. The image descriptor based on a pyramid representation.

for a favorite object, and the attention is geared to react when
it appears [45]. Therefore, if a group of images includes a
co-salient object (e.g., cat), the object region in each image
should be highlighted with distinct saliency value. In thiswork, we define such saliency as inter-image saliency (IrIS)
which aims to describe the co-salient object from a group ofimages. Unlike the intra-image saliency that measures object
saliency within an image, inter-image saliency is to discover
the co-attention object that simultaneously stands out in theimage group. As illustrated in Fig. 2, there are three stages
included in our IrIS detection, i.e., pairwise similarity ranking,pyramid feature extraction and pairwise image matching.
1) Pairwise Similarity Ranking: In order to identify theco-salient object from multiple images, we should solve theimage correspondence problem to find out which parts ofan image correspond to which parts of another image. If theco-salient objects share the similar color information, somemethods, e.g., image pair saliency detection [1], can be employed to solve the correspondence problem between theimages. However, co-salient objects usually exhibit the colorand shape diversities in images, which can be described asfollows.
(i) The co-salient objects, especially for those man-made objects, are allowed to exhibit different colors.
(ii) The co-salient objects may appear with different shapes
when they are captured in different views or positions.
It is known that object matching highly depends on the featuredescription, while the feature description is usually sensitive tothe object diversity. Thus, it is still a challenging task to extract
co-salient objects with various color or shape features from multiple images. Some work simplifies this diversity problem byassuming the objects are with similar color [1][34].
However, it is interesting to see that co-salient objects appear
to exhibit local feature consistence for some instances although
they are allowed to have various color and shape features. As
shown in Fig. 2, some cat instances appear with black color,while some others are calico cats. Based on this phenomenon,
we first compute a pairwise similarity for the image group. Thenthe images ranking is performed to generate the local consistentimages, i.e., the similar pair of images.
Assume there areimages that need to perform theco-salient object detection, which are denoted by,, respectively. Firstly, we compute the RGB
color histogram for each image based on a pyramid structure,
which is shown in the first column of Fig. 7. We use threelevels to perform the pyramid decomposition. The whole image

1900

IEEE TRANSACTIONS ON MULTIMEDIA, VOL. 15, NO. 8, DECEMBER 2013is set to the first level, while the center-surround partitionis the second level. The third level is created by a quadtree
partition of an image. Prior to the histogram computation, wequantize all the pixels in ,into (=100 in ourwork) color words using the k-means clustering algorithm. For
each level, we count the occurrence number for each word to
produce the histogram. Let
denote the set of pixels in theth pyramid level. The histogram descriptor at the th levelcan be expressed by
Fig. 8. The construction progress of our co-occurrence descriptor.

(3)
denotes the clustering label of pixel
after thewherek-means clustering, andis the total number of pixels at
the th pyramid level and is used to normalize the sum of
the histogram into one. As shown in Fig. 7, the final image
histogram descriptor is built by concatenating the histogramsof all levels together, which can be expressed by(4)Here, the histogram dimension is set to 600 (i.e.,) in our work.
Secondly, we measure the similarity between images basedon the obtained histograms. The chi-square distanceisused to evaluate the pairwise similarity, which is described asfollows.

(5)
where denotes the total number of histogram bins (=600). Finally, we rank the image pairs according to the similarity values
in a descending order. From (5), we can see that the first ranked
image pairs exhibit the similar color content.
2) Image Feature Extraction: After the pairwise similarity
ranking, we can perform the image matching to discover whichparts are co-salient between images. Here, we implement theimage matching at the region level. In order to perform image
region matching, three types of visual descriptors are built to
describe the region properties, i.e., color word, color co-occurrence word and shape word. The first two visual descriptors aredesigned to capture the image appearance from the color distribution, while the third is used to describe the image appearance
in terms of the shape property.
To create a color descriptor, we use RGB color space to represent the color feature. All the pixels in an image pair are
quantized into 50 clusters using the k-means clustering algorithm. Each cluster center is called a codeword. As mentioned
in the previous section, each image has been partitioned into
a number of regions after multiscale image segmentation. For
each region, we compute the histogram by counting the numberof codewords at each bin (i.e., cluster) according to (3). The
color descriptorof a region is represented by the bins of thehistogram.
To create a co-occurrence descriptor, we first extract theimage contour based on the globalized probability of boundary(gPb) [46], which can detect and localize the candidate contour

using a combination of local and global cues. Inspired by thefact that color variation usually occurs at image boundaries, we
can describe the color co-occurrence around boundary pixels.
In this work, we compute the co-occurrence descriptors fromthe salient boundary points (i.e.,). As shown in
Fig. 8, we first design a polar location grid with the radius of
8 pixels, which centers at the boundary point. Then we divide
this polar grid into 4, 8, 16 subregions in an angular direction,
which yield a four-level pyramid structure. For each subregion
at a pyramid level, we compute the mean RGB color value ofall the pixels in this subregion, which are then set to the bins ofa co-occurrence descriptor. This produces a 87 bins histogram
descriptor. In order to achieve the rotation-invariant property,
we determine the start subregion based on the maximum intensity change. Assume two points
andequally divide
the polar grid into two parts which are denoted byandin a clockwise direction. There can be 8 possible
divisions of the polar grid into 2 equal parts, which result in
16 possible endpoints forselection. The start subregion isdefined as(6)denotes the average intensity value
where
for the part
. All the descriptors in an image pair are also
quantized into 50 clusters (e.g., codeword) using the k-meansclustering algorithm. The color co-occurrence descriptor
fora region is represented by the bins of the histogram which is
generated by counting the number of codewords at each bin (i.e.,cluster) according to (3).
The third descriptor used in our work is the shape descriptor.
Unlike the above color related descriptors, the shape descriptor
focuses on measuring intensity variations of boundary points.
We also employ the salient boundary points to build the shapedescriptors. The construction process is illustrated in Fig. 9,
which consists of two parts, i.e., gPb coefficients and normalized AC coefficients. For each subregion at a pyramid level, we
compute the mean gPb values of all boundary pixels in this subregion, which are then set to the bins of a shape descriptor. In addition, we compute the mean intensity value for each subregion
in the last level, and perform a discrete cosine transform (DCT).
The obtained 15 AC coefficients are normalized by the DC coefficient, and then combined with the gPb coefficients to produce the final shape descriptor. To describe a region, we quantize all shape descriptors in an image pair into 50 clusters using
the k-means clustering algorithm. The shape descriptorof aLI et al.: CO-SALIENT OBJECT DETECTION FROM MULTIPLE IMAGES

1901

Fig. 9. The construction progress of our defined shape descriptor.

region is generated by measuring the frequency of codewords
at each bin according to (3).
3) Minimum Spanning Tree: After multiscale segmentation
for a set of images, each image has been divided
intosegments. Heredenotesthe segment set of the image , where
corresponds to theth segment. For simplicity, we use
to denote a segmentin the image , i.e.,. Letdenote a group ofregions, i.e.,, which are taken from the image
set. In other words, we can build a region groupby simply
choosing one region from each image. Thus, the co-salient map
for a region
can be defined as
(7)
with

From (9), we can see that the region matching is only performedon the image pairs with edge links in MST graph.
4) Image Pairwise Matching: After the MST construction,
we are ready to measure the pairwise similarity so as to infer
the co-salient region
with the high matching score froma group of images. Given a pair of images and that havean edge in the MST graph (i.e.,), we first computethe correspondence matrix between regions, where each element
denotes the distance between regions
and
.
Here,
and
denote two regions in the image and ,respectively. Based on the feature extraction, each region can be
represented by three types of descriptors, i.e., color descriptor, color co-occurrence descriptor
and shape descriptor .
Here we further combine
andinto a color descriptor
based on the scheme used in (4). For a pair of regions
and, the distance
between regions is defined as:
(10)
where
is used to evaluate the histogram distance between
tow regions. It is observed that a weighted
distance is used
to generate the matrix . For a pair of images with high similarity, large weight will be imposed to the first term (i.e., color
distance). Otherwise, the image pair will be evaluated mainly
based on the shape feature due to the distinct color difference.
Let
be an indicator variable. If region
corresponds
to region, we have, otherwise. The
one-to-one constraint matching between this image pair can be
expressed as the following optimization problem(8)
Here the similarity of a group of regions is obtained by summating the pairwise similarity among these regions. It is seenthat each region in a region groupwill have the sameco-salient value, i.e., the summation of all pairwise similarities. However, the total number of region groups will reach to, which is an impossible task forcurrent computation capability especially for large and .
It is known that the more similar a pair of image, the more accurate the image matching. Therefore, we identify the co-salient
objects by performing the pairwise similarity computation for
those images with high similarities. To achieve this goal, we
first construct a minimum spanning tree (MST) based on theranked similarities. Given an undirected graphwith nodes
and edges, where the nodes
denote a set of images. Two nodes andare connected by an undirected edge
which has the weight. It is known that a minimum spanning
tree is a subgraph that connects all the vertices together, which
has the weight less than the weight of every other spanning tree.
Here, this tree can be easily obtained by cutting the image pairs
with high ranking scores to build a spanning tree [48].
Let
denote the edge set for the MST graph. Based on(7), we measure the co-saliency of a region in the region group
by
(9)

(11)
From (11), we can see that this problem is actually an assignment problem, which can be solved via linear programming.
groups ofUsing the image matching (11), we can obtainregions. We compute the total matching similarities for all thegroups, and then sort them in a descending order. Generally, we
can select the region group with the maximum matching score
to yield a final inter-image saliency map. However it is difficult
to deal with multiple co-salient object detection. To achieve arobust performance, we consider the firstgroups to generate
our inter-image saliency map. Let
denote the sorted groups according to the matching scores. Assume denotes a pixel in an image , the inter-image saliency
value of pixel
can be computed by
(12)
with
(13)
if

(14)

1902

IEEE TRANSACTIONS ON MULTIMEDIA, VOL. 15, NO. 8, DECEMBER 2013

A. Parameter Settings and Evaluation Metrics
We first introduce the parameter settings in our experiments.
For the multiscale object segmentation, we design 28 candidate
windows to run the Grab-Cut segmentation. In order to avoid
the effects of possible false segmentations, we sort the segmentation results according to the number of foreground pixels, andremove the first two and last two segmentation results, i.e., thosesegmentations with seldom or numerous foreground pixels inthe segmentation window. For the final object co-saliency map,
we choseas the weight in (15). Here, we set
tocompute the control parameter in (13), which shows good performance from our empirical study.
In order to evaluate the quality of our proposed method,
we perform an objective comparison based on the extracted
co-saliency map and the hand-annotated ground-truth mask.
The comparison between an algorithm’s output and the groundtruth is performed on three evaluation metrics, i.e., Precision(Pre), Recall (Rec), and F-measure (F) [1]. Given a groupof images, the Precision is defined as the ratio of correctly
segmented object regions to all the segmented regions, while
the Recall is computed by the ratio of correctly extracted
object regions to the ground-truth masks. A weighted mean of
precision and recall [15] is employed to calculate F-measure,which can be expressed as(16)Here, we set
work[15].
where is a weighting coefficient, and the parameter
the range (i.e., distance) similarity.

adjusts

C. Object Co-Saliency Map
Based on intra-image and inter-image saliency maps, we areready to extract the co-salient object from a group of images. Letdenote the co-salient value of a
pixel in the image . By combining the two saliency maps (2)and (12), we have
(15)
where is a constant that is used to control the impact of the
IaIS and IrIS on the image co-saliency. From (15), we can seethat the co-saliency map is built by a linear combination of the
IaIS and IrIS [1], which means that a pixel with high co-saliency
value will not only exhibit strong intra-image saliency but alsointer-image saliency. The contributions of the IaIS and IrIS arecontrolled by the weight . The detailed steps of our proposed
method are presented in Algorithm 1.
IV. EXPERIMENTS
In this section, we evaluate the performance of our proposed
method on a number of image groups. Four public image
datasets are used for the extensive experiments and comparisons with the state-of-the-art methods. Some subjective andobjective assessments of detection results are reported.

that was also recommended in the

B. Experiments on ICoseg DatabaseWe first evaluate our proposed method on the public image
dataset ICoseg,1 which consists of 38 groups (643 images) alongwith pixel ground-truth hand annotations [32]. Each group contains a common object with the similar color, e.g., bear, pandas,kite.
We compare our result with four state-of-the-art methods forsaliency detection, i.e., frequency model SR [14], FT [15], computational model SER [10] and global contrast model RC [19].
To perform a fair comparison, most methods are implemented
based on the source codes or executable codes by authors. Notethat all the results are computed using the default parameters
given by the source codes. We implement our method with the
Matlab code.
Some experimental results are illustrated in Fig. 10, whichcontains two image groups, i.e., Red Sox and Cheetah. For
the first image group Red Sox, we can see that most existing
methods can provide good performance when the common
object (i.e., players) appears in the simple backgrounds, suchas the first three images in the upper part of Fig. 10. But for
those images that not only include the players but also thecomplex backgrounds, many false detections can be observed
for the existing saliency models. In addition, for the second
image group Cheetah, the common object usually exhibits
more complex texture with respect to the first group, which1http://chenlab.ece.cornell.edu/projects/touch-coseg/

LI et al.: CO-SALIENT OBJECT DETECTION FROM MULTIPLE IMAGES

1903

Fig. 10. Evaluation results for two ICoseg image groups. Top and Bottom: Some results for image groups Red Sox and Cheetah, respectively. Row 1: Someoriginal images. Rows 2–6: Results for FT, SR, SER, RC, and Our method, respectively.

leads to a poor performance for the existing methods. However,as shown in the last row of Fig. 10, it is seen that our method
is able to discover co-salient objects successfully from each
image group.
To provide a fair comparison, we follow the evaluation
strategies used in [1], [15]. Firstly, for each image, we employ
an adaptive threshold in [14], [15] to obtain the binary saliency
map, which is computed by two times the mean saliency of agiven image. We compute the average value of the evaluation
metrics for each image group. Table II gives the comparison
results on all ICoseg image groups which show that our methodmeasures the co-saliency more accurately with the high Precision, Recall and F-measure. Compared with the method
SER [10], our method achieves about 41.03%, 50.77% and53.76% improvements of Recall, Precision and F-measure,respectively. Compared with the method RC [19], our method
yields about 32.85%, 5.74% and 25.83% gains of Recall,
Precision and F-measure, respectively. Secondly, we vary thisthreshold from 0 to 255, and calculate the precision and recall
at each value of the threshold. It provides a reliable comparisonof how well various saliency maps highlight salient regionsin images. We compute the area under the precision versus
recall curve (PRC-Area) using the method [9]. The last termin Table II presents the mean area result for each image group,
which shows that our method ourperforms the state-of-the-art
methods. About 1.44%, 25.21%, and 33.38% gains can beachieved by our proposed method compared with RC [19], FT[15] and SER [10], respectively.

C. Experiments on MSRC Database
We next evaluate our proposed method on the public MSRC
object class recognition dataset2 which consists of 20 groups(591 images) along with pixel-wise labelled annotations.
Different objects, such as cow, aeroplane, car and flowers,
are included in the dataset. Compared with the ICoseg image
dataset, different colors are allowed for the common objects
within the image groups. Here, we further compare our proposed method with the similar work FCO [36] which computed
the co-saliency map by combining the single-view saliency
and repeatedness together. Since the authors have not released
their code for open evaluation, we implement the co-saliency
algorithm according to the original paper [36]. Note that in our
experiment the single-view saliency is replaced by the more
effective saliency model RC [19] instead of SER [17] usedin [36].
Fig. 11 shows the comparison results of two MSRC imagegroups, i.e., sign and flower, where the common objects exhibit distinct diversities in color or shape property. Note thatsince most existing methods identify the salient object from asingle image, such diversity will produce more effect on theco-saliency model. Compared with existing methods, our proposed method still achieves good performance on salient object detection, which highlights the common objects with large
saliency values.
2http://research.microsoft.com/en-us/projects/objectclassrecognition/

1904

IEEE TRANSACTIONS ON MULTIMEDIA, VOL. 15, NO. 8, DECEMBER 2013

TABLE IIEVALUATION RESULTS FOR ICOSEG IMAGE DATASETThe objective performance on all MSRC image groups areillustrated in Table III, where the results by adaptive thresholds
are shown at the first three terms in Table III, respectively.
Compared with the method SER [10], It is seen that our method
provides about 50.52%, 22.10%, and 32.35% improvementsof Recall, Precision and F-measure, respectively. It is notedthat similar precision on average can be observed between
our method and the global contrast model RC [19]. But our
method achieves about 74.22% and 31.69% gains of Recalland F-measure, respectively. Compared with FCO model [36],our method also provides distinct improvements in the Recall,
Precision and F-measure with 71.30%, 4.12% and 31.23%
gains on average, respectively. It means that our method isable to discover most salient object regions from image groups
under the similar precision levels. The last term in Table III
shows the mean area result for each image group, which showsthat our method still outperforms the state-of-the-art methods,
which yields 3.60%, 40.10%, 35.83%, 15.16% and 5.70% gains
compared with RC [19], FT [15], SR [14] SER [10], and FCO[36], respectively.
To further investigate the performance of our proposed
method, we combine the previous saliency methods, i.e., SR[14], FT [15], SER [10] and RC [19], with our inter-saliency
to generate a final co-saliency map. The results are given in
Table IV which shows that all the previous saliency methods
can achieve improvements with respect to Recall, Precision
and F-measure. For example, about 0.1249, 0.1633, 0.2732
and 0.1926 gains of F-measure on ICoseg dataset can beachieved for RC [19], FT [15], SR [14], and SER [10],respectively. For MSRC dataset, the corresponding gains ofF-measure are also about 0.1046, 0.1642, 0.1509 and 0.1042,
respectively. Compared with the previous saliency methods
with the IrIS saliency, our proposed method still achieves
more accurate saliency detection with the high Precision,
Recall and F-measure. In addition, we only consider our
IaIS map to fairly compare with the other methods, which
can be found in Table IV. Compared with the RC saliency
model, our IaIS achieves about 10.9%, 0.71% and 11.83%
improvements of Recall, Precision and F-measure on the
ICoseg dataset, respectively. The corresponding improvements
will reach to 59.9%, 1.83% and 28.86% for the MSRC dataset.
The experimental results further demonstrate that our IaIS cancompete with the state-of-the-art methods for the single image
saliency detection.

LI et al.: CO-SALIENT OBJECT DETECTION FROM MULTIPLE IMAGES

1905

Fig. 11. Evaluation results for two MSRC image groups. Top and Bottom: Some results for the image group sign and flower, respectively. Row 1: Some original
images. Rows 2–7: Results for FT, SR, SER, RC, FCO, and Our method, respectively.

TABLE IIIEVALUATION RESULTS FOR MSRC IMAGE DATASET

D. Experiments on Images With Larger Variations
We collect six image groups with large variations from thepublic image datasets (e.g., PASCAL VOC 2008, UIUC Butterfly, Stanford-40 and Flickr). Some image examples can befound in Fig. 12(a), which includes Running, Baby, Butterfly,
Dog2, Riding-horse, and Toy-baby image groups. It is seen thatthe first two image groups are taken from the road, room orcountryside. The butterfly images contain a similar object (i.e.,admiral butterfly) in a cluttered background. The objective comparison results are illustrated in Fig. 12(b), which shows that our
proposed method also achieves good performance comparedwith the existing methods. Compared with the RC method, ourmethod yields about 19.14% and 8.27% gains of F-measure andmean area on average for all images. Furthermore, two imagegroups with multiple objects are evaluated, which are shown in1906

IEEE TRANSACTIONS ON MULTIMEDIA, VOL. 15, NO. 8, DECEMBER 2013

TABLE IV
COMPARISON RESULTS BY COMBINING THE EXISTING METHODS WITH OUR IRIS

Fig. 12. Some examples taken from six image groups. (a) From left to right: Running, Baby, Butterfly, Dog2, Riding-horse, and Toy-baby. (b) Evaluation results
for five image groups for Recall, Precision and F-measure, respectively.

evaluation metrics in Fig. 12(b), we can see that good performance can be achieved by our proposed method. Compared with
RC method [19], about 78.36% and 20.74% gains of F-measure
on average can be obtained for our method, respectively.
E. Experiments on Image Pair Database

Fig. 13. Experimental results for image pairs. (a): Original image pairs, i.e.,llama, elephant, hawksbill. (b)-(f): Results by CA [17], SER [10], RC [19],IPCO [1], and our method.

the last two columns of Fig. 12(a). The first contains two different objects (i.e., toy baby and toy dog), where some occlusions and lighting variations can be observed. From objective

In this experiment, we compare our result with the recent
image pair co-saliency (IPCS) detection method [1] which measures visual saliency for a pair of images. Three saliency models
used in [1], i.e., SER [10], CA [17] and RC [19], are also considered for the evaluation. The comparisons are performed on the
public image pair dataset3 given in [1], which consists of 105
pairs of images, such as human objects, flowers, buses, cars,
boats and various animals.
Some examples are shown in Fig. 13, where the original
image pairs are presented in the first column. The corresponding
results are presented in Figs. 13(b)–(f), respectively. It is shownthat good performance for co-salient object detection can beachieved by our proposed method. For the image pairs elephantand hawksbill, the object appears with different sizes or views
making it a challenging image to detect. However, our method
provides more accurate results to identify the co-salient regions.
3http://ivipc.uestc.edu.cn/hlli/projects/cosaliency.html

LI et al.: CO-SALIENT OBJECT DETECTION FROM MULTIPLE IMAGES

Fig. 14. Evaluation results for 105 image pairs given in [1]. (a) Precision-recall
curves for varying thresholds. (b) Precision-recall bars for adaptive thresholds.

1907

Fig. 16. The relative performance of the evaluation metric verse the selected
windows.

Fig. 15. The curve of the evaluation metric verse the parameter .

We follow the evaluation strategies used in [1]. The precision versus recall curve is plotted in Fig. 14(a), which showsthat our method achieves high precision for most recall values
especially for the recall in the range of [0.2 0.9]. Then, we employ an adaptive threshold in [15] to obtain the binary saliency
map. As shown in Fig. 14(b), the comparison results show that
our method measures the co-saliency more accurately with thehighest Precision, Recall and F-measure. Compared with themethod [1], our method achieves about 23.14%, 3.73% and8.84% improvements of Recall, Precision and F-measure, respectively.
F. Discussion
In this work, we use the linear fusion to generate the final
co-saliency map, which is controlled by the weight parameter. From (15), we can see that the large weight will be imposedto the inter-image saliency with the decreasing . To investigate
the effect of on the co-saliency detection, we also compute
Precision, Recall and F-measure metrics based on the adaptive
threshold by changing from 0 to 1. Fig. 15 plots the curveof the evaluation metrics for all the images (namely ICoseg+ MSRC datasets), which shows that good performance canbe achieved when the weight is in the range of [0.4, 0.6].
The result also demonstrates the effectiveness of our proposedco-saliency model.
In order to measure the co-saliency map, the multiscale segmentation windows (see Fig. 4) are defined based on different
positions and different sizes of the windows. There are total
28 segmentation windows used in our work. To investigate
influence of the number of windows on the final accuracy,
we compute Precision, Recall and F-measure metrics basedon the adaptive threshold for different windows (i.e., 5, 15,
20, 28). Each window scale was randomly selected from thetotal window set for many times. Fig. 16 plots the curve of theFig. 17. Evaluation results for girl image group with significantly clutteredimages. Rows 1 and 3: Some original images. Rows 2 and 4: Co-saliency results.

relative improvement on evaluation metrics for all the images(namely ICoseg + MSRC datasets). Here, the 5-window scale
was used as a reference and the relative differences for theremaining window scales were calculated. We can see that theworst performance can be found when there are only 5 windows used. The main reason is that a small number of windows
cannot provide enough object candidates for co-salient object
detection. Furthermore, a slight improvement can be observed
when we increase candidate windows.
In addition, we investigate the effect of three types of visual descriptors (i.e., color-based, color co-occurrence-based,
and shape-based visual descriptors) on the final performance.
Table V shows the relative performance values for different
visual descriptors. For each evaluation metric, the worst descriptor was used as a reference and the relative differences forthe remaining descriptors were calculated. It is seen that thecolor word and shape word respectively provide large gains for
the ICoseg and MSRC datasets if only one visual descriptor isconsidered. For the ICoseg dataset, the combined descriptors
achieve the highest gains compared with other descriptors, i.e.,
4.79%, 5.97% and 4.79% for Recall, Precision and F-measure,respectively. For the MSRC dataset, the combined descriptors
also provide higher gains of Recall and F-measure except for
the Precision.
Furthermore, it should be noted that the false detection will
occur when the objects have no clear shape or color within
the image group, especially for the significantly cluttered images. An example can be found in Fig. 17, where some false
co-saliency detections can be observed (i.e., marked with red
windows) due to the cluttered background. The main reason is1908IEEE TRANSACTIONS ON MULTIMEDIA, VOL. 15, NO. 8, DECEMBER 2013

TABLE VRELATIVE PERFORMANCE ON THREE VISUAL WORDS FOR ALL ICOSEG AND MSRC DATASETS

that our method is based on the assumption that a co-salient region should exhibit high similarity with respect to certain features (e.g., intensity, color, texture or shape).
V. CONCLUSION
In this paper, we have presented a new method to discoverco-salient objects from a set of images. This method aims tosimulate the attention search process and predict the human
fixed objects within an image group. The proposed co-saliency
model is built based on two saliency maps, i.e., the intra-image
saliency (IaIS) and the inter-image saliency (IrIS) maps. The
first term is designed to identify the salient objects from a single
image according to the multiscale image segmentation voting,while the second term is to discover the co-salient objects froma set of images. To compute the IrIS map, we first construct animage pyramid representation to perform a pairwise similarity
ranking. The ranked results are then used to build a minimum
spanning tree for image pairwise matching. To describe the region aspects of local appearance in an image, we design three
types of visual descriptors in terms of color, color co-occurrence
and shape properties. Finally, we solve the final region matchingproblem between images by linear programming. Experimental
evaluation on a number of image groups demonstrates the good
performance of the proposed method on the co-salient object
detection.
REFERENCES[1] H. Li and K. N. Ngan, “A co-saliency model of image pairs,” IEEE
Trans. Image Process., vol. 20, no. 12, pp. 3365–3375, Dec. 2011.
[2] L. Itti, C. Koch, and E. Niebur, “A model of saliency-based visual attention for rapid scene analysis,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 20, no. 11, pp. 1254–1259, Nov. 1998.
[3] J. Huang, X. Yang, X. Fang, W. Lin, and R. Zhang, “Integrating visualsaliency and consistency for re-ranking image search results,” IEEE
Trans. Multimedia, vol. 13, no. 4, pp. 653–661, 2011.
[4] Y. Fang, W. Lin, B.-S. Lee, C.-T. Lau, Z. Chen, and C.-W. Lin,
“Bottom-up saliency detection model based on human visual sensitivity and amplitude spectrum,” IEEE Trans. Multimedia, vol. 14, no.
1, pp. 187–198, 2012.
[5] Z. Liu, R. Shi, L. Shen, Y. Xue, K. N. Ngan, and Z. Zhang, “Unsupervised salient object segmentation based on kernel density estimationand two-phase Graph Cut,” IEEE Trans. Multimedia, vol. 14, no. 4,
pp. 1275–1289, 2012.
[6] C. Koch and S. Ullman, “Shifts in selective visual attention: Towards the underlying neural circuitry,” Human Neurobiol., vol. 4, pp.
219–227, 1985.
[7] H. Li and K. N. Ngan, “Saliency model based face segmentation inhead-and-shoulder video sequences,” J. Visual Commun. Image Represent. (Elsevier Science), vol. 19, no. 5, pp. 320–333, 2008.
[8] H. Li and K. N. Ngan, “Unsupervised video segmentation with low
depth of field,” IEEE Trans. Circuits Syst. Video Technol., vol. 17, no.
12, pp. 1742–1751, Nov. 2007.
[9] J. Harel, C. Koch, and P. Perona, “Graph-based visual saliency,” Adv.
Neural Inf. Process. Syst., vol. 19, pp. 545–552, 2007.

[10] W. Wang, Y. Wang, Q. Huang, and W. Gao, “Measuring visual saliency
by site entropy rate,” in Proc. IEEE Computer Society Conf. Computer Vision and Pattern Recognition (CVPR), Jun. 13–18, 2010, pp.
2368–2375.
[11] T. Liu, Z. Yuan, J. Sun, J. Wang, N. Zheng, X. Tang, and H. Shum,“Learning to detect a salient object,” in Proc. IEEE Computer Society
Conf. Computer Vision and Pattern Recognition (CVPR), Minneapolis,
MN, 2007, pp. 1–8.
[12] R. Achanta, F. Estrada, P. Wils, and Süsstrunk, “Salient region detection and segmentation,” in Proc. Int. Conf. Computer Vision Systems,
2008, pp. 66–75.
[13] Y.-F. Ma and H.-J. Zhang, “Contrast-based image attention analysis byusing fuzzy growing,” in Proc. ACM Int. Conf. Multimedia, New York,NY, USA, 2003, pp. 374–381.
[14] X. Hou and L. Zhang, “Saliency detection: A spectral residual approach,” in Proc. IEEE Computer Society Conf. Computer Vision and
Pattern Recognition (CVPR), 2007, pp. 1–8.
[15] R. Achanta, S. S. Hemami, F. J. Estrada, and S. Ssstrunk, “Frequency-tuned salient region detection,” in Proc. IEEE Computer
Society Conf. Computer Vision and Pattern Recognition (CVPR),
2009, pp. 1597–1604.
[16] R. Achanta and S. Süsstrunk, “Saliency detection using maximum
symmetric surround,” in Proc. 17th IEEE Int. Conf. Image Processing(ICIP), Hong Kong, Sep. 2010, pp. 2653–2656.
[17] S. Goferman and L. Zelnik-Manor, “Context-aware saliency detection,” in Proc. IEEE Computer Society Conf. Computer Vision and
Pattern Recognition (CVPR), 2010, pp. 2376–2383.
[18] Y. Zhai and M. Shah, “Visual attention detection in video sequencesusing spatiotemporal cues,” in Proc. ACM Multimedia, 2006, pp.
815–824.
[19] M. M. Cheng, G. Zhang, N. J. Mitra, X. Huang, and S. Hu, “Global
contrast based salient region detection,” in Proc. IEEE Computer Society Conf. Computer Vision and Pattern Recognition (CVPR), Colorado Springs, CO, USA, 2011, pp. 409–416.
[20] A. Borji and L. Itti, “Exploiting local and global patch rarities forsaliency detection,” in Proc. IEEE Computer Society Conf. Computer Vision and Pattern Recognition (CVPR), Jun. 16–21, 2012, pp.
478–485.
[21] M. Wang, J. Konrad, P. Ishwar, K. Jing, and H. Rowley, “Imagesaliency: From intrinsic to extrinsic context,” in Proc. IEEE Computer
Society Conf. Computer Vision and Pattern Recognition (CVPR),
2011, pp. 417–424.
[22] X. Shen and Y. Wu, “A unified approach to salient object detection
via low rank matrix recovery,” in Proc. IEEE Computer Society Conf.
Computer Vision and Pattern Recognition (CVPR), Jun. 16–21, 2012,
pp. 853–860.
[23] H.-K. Tan and C.-W. Ngo, “Common pattern discovery using earth
mover’s distance and local flow maximization,” in Proc. IEEE Int.
Conf. Computer Vision (ICCV), 2005, pp. 1222–1229.
[24] J. Yuan and Y. Wu, “Spatial random partition for common visual pattern discovery,” in Proc. IEEE Int. Conf. Computer Vision (ICCV),
2007, pp. 1–8.
[25] A. Toshev, J. Shi, and K. Daniilidis, “Image matching via saliency region correspondences,” in Proc. IEEE Computer Society Conf. Computer Vision and Pattern Recognition (CVPR), 2007, pp. 1–8.
[26] M. Cho, Y. M. Shin, and K. M. Lee, “Co-recognition of image pairs
by data-driven Monte Carlo image exploration,” in Proc. Eur. Conf.
Computer Vision (ECCV), 2008, pp. 144–157.
[27] C. Rother, T. Minka, A. Blake, and V. Kolmogorov, “Cosegmentationof image pairs by histogram matching-incorporating a global constraint
into MRFs,” in Proc. IEEE Computer Society Conf. Computer Vision
and Pattern Recognition (CVPR), 2006, vol. 1, pp. 993–1000.
[28] L. Mukherjee, V. Singh, and C. R. Dyer, “Half-integrality based algorithms for cosegmentation of images,” in Proc. IEEE Computer Society Conf. Computer Vision and Pattern Recognition (CVPR), 2009,
pp. 2028–2035.

LI et al.: CO-SALIENT OBJECT DETECTION FROM MULTIPLE IMAGES[29] D. S. Hochbaum and V. Singh, “An efficient algorithm for co-segmentation,” in Proc. IEEE Int. Conf. Computer Vision (ICCV), 2009, pp.
269–276.
[30] A. Joulin, F. Bach, and J. Ponce, “Discriminative clustering for image
co-segmentation,” in Proc. IEEE Conf. Computer Vision and Pattern
Recognition, Jun. 2010, pp. 1943–1950.
[31] S. Vicente, V. Kolmogorov, and C. Rother, “Cosegmentation revisited: Models and optimization,” in Proc. Eur. Conf. Computer Vision(ECCV), 2010, pp. 465–479.
[32] D. Batra, A. Kowdle, D. Parikh, J. Luo, and T. Chen, “iCoseg: Interactive Co-segmentation with Intelligent Scribble Guidance,” in Proc.
IEEE Computer Society Conf. Computer Vision and Pattern Recognition (CVPR), 2010, pp. 3169–3176.
[33] G. Kim, E. P. Xing, L. Fei-Fei, and T. Kanade, “Distributed cosegmentation via submodular optimization on anisotropic diffusion,” in Proc.
Int. Conf. Computer Vision, Nov. 2011, pp. 169–176.
[34] F. Meng, H. Li, G. Liu, and K. N. Ngan, “Object co-segmentation basedon shortest path algorithm and saliency model,” IEEE Trans. Multimedia, vol. 14, no. 5, pp. 1429–1441, 2012.
[35] S. Vicente, C. Rother, and V. Kolmogorov, “Object cosegmentation,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition(CVPR), Jun. 2011, pp. 2217–2224.
[36] K. Chang, T. Liu, and S. Lai, “From co-saliency to co-segmentation:
An efficient and fully unsupervised energy minimization model,” inProc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR),
2011, pp. 2129–2136.
[37] L. Mukherjee, V. Singh, and J. Peng, “Scale invariant cosegmentation
for image groups,” in Proc. IEEE Conf. Computer Vision and Pattern
Recognition (CVPR), Jun. 2011, pp. 1881–1888.
[38] Y. Lu, W. Zhang, H. Lu, and X. Xue, “Salient object detection using
concavity context,” in Proc. IEEE Int. Conf. Computer Vision (ICCV),
Nov. 6–13, 2011, pp. 233–240.
[39] B. Alexe, T. Deselaers, and V. Ferrari, “What is an object?,” inProc. IEEE Computer Society Conf. Computer Vision and Pattern
Recognition (CVPR), San Francisco, CA, USA, Jun. 13–18, 2010, pp.
73–80.
[40] A. Levinshtein, A. Stere, K. N. Kutulakos, D. J. Fleet, and S. J. Dickinson, “Turbopixels: Fast superpixels using geometric flows,” IEEE
Trans. Pattern Anal. Mach. Intell., vol. 31, no. 12, pp. 2290–2297, Dec.
2009.
[41] Y. Boykov and M.-P. Jolly, “Interactive graph cuts for optimal
boundary & region segmentation of objects in n-d images,” in Proc.
IEEE Int. Conf.Computer Vision (ICCV), Vancouver, BC, Canada, Jul.
7–14, 2001, vol. 1, pp. 105–112.
[42] Y. Li, J. Sun, C.-K. Tang, and H.-Y. Shum, “Lazy snapping,” in Proc.
SIGGRAPH, 2004, pp. 303–308.
[43] C. Rother, V. Kolmogorov, and A. Blake, “GrabCut-interactive
foreground extraction using iterated graph cuts,” in Proc. SIGGRAPH
2004, Los Angeles, CA, USA, Aug. 10–12, 2004.
[44] S. L. Franconeri, A. Hollingworth, and D. J. Simons, “Do new objects
capture attention?,” Psychol. Sci., vol. 16, no. 4, pp. 275–281, 2005.
[45] D. Ballard, M. Hayhoe, P. Pook, and R. Rao, “Deictic codes for the embodiment of cognition,” Behav. Brain Sci., vol. 20, no. 4, pp. 723–767,
1997.
[46] M. Maire, P. Arbelaez, C. Fowlkes, and J. Malik, “Using contours todetect and localize junctions in natural images,” in Proc. IEEE Computer Society Conf. Computer Vision and Pattern Recognition (CVPR),
2008, pp. 1–8.
[47] A. Joulin, F. Bach, and J. Ponce, “Multi-class cosegmentation,” inProc. IEEE Conf. Computer Vision and Pattern Recognition, Jun. 2012,
pp. 542–549.
[48] J. B. Kruskal, “On the Shortest Spanning Subtree of a Graph and theTraveling Salesman Problem,” Proc. Amer. Math. Soc., vol. 7, no. 1,
pp. 48–50, 1956.

1909

Hongliang Li (SM’12) received his Ph.D. degree inElectronics and Information Engineering from Xi’an
Jiaotong University, China, in 2005. From 2006 to
2008, he joined the visual signal processing andcommunication laboratory (VSPC) of the Chinese
University of Hong Kong (CUHK) as a Postdoctoral
Fellow. He is currently a Professor in the Schoolof Electronic Engineering, University of Electronic
Science and Technology of China. His research interests include image segmentation, object detection,
image and video coding, and visual attention.
Dr. Li has authored or co-authored numerous technical articles in well-known
international journals and conferences. He is a co-editor of a Springer book titled “Video segmentation and its applications”. Dr. Li was involved in many
professional activities. He is a member of the Editorial Board of the Journal on
Visual Communications and Image Representation. He served as TPC members
in a number of international conferences, e.g., ICME 2012–2013, ISCAS 2013,
PCM2009, and VCIP2010, and General co-chair of the 2010 ISPACS. He serves
as a local chair of the 2014 IEEE International Conference on Multimedia andExpo (ICME).

Fanman Meng received the B.Sc. degree in computer science and technology and the M.Sc. degree
in computer software and theory in 2006 and 2009
respectively. Since September 2009, he has beenworking toward the Ph.D. degree in the Intelligent
Visual Information Processing and Communication
Laboratory (IVIPC) at University of Electronic
Science and Technology of China (UESTC). His
research interests include image segmentation,
object detection and visual attention.

King Ngi Ngan (F’00) received the Ph.D. degree inElectrical Engineering from the Loughborough University in U.K. He is currently a chair professor at
the Department of Electronic Engineering, Chinese
University of Hong Kong. He was previously a full
professor at the Nanyang Technological University,
Singapore, and the University of Western Australia,
Australia. He holds honorary and visiting professorships of numerous universities in China, Australia
and South East Asia.
Prof. Ngan served as associate editor of IEEE
TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, Journal
on Visual Communications and Image Representation, EURASIP Journalof Signal Processing: Image Communication, and Journal of Applied Signal
Processing. He chaired and co-chaired a number of prestigious international
conferences on image and video processing including the 2010 IEEE International Conference on Image Processing, and served on the advisory andtechnical committees of numerous professional organizations. He has publishedextensively including 3 authored books, 6 edited volumes, over 300 refereed
technical papers, and edited 9 special issues in journals. In addition, he holds
10 patents in the areas of image/video coding and communications.
Prof. Ngan is a Fellow IET (U.K.) and IEAust (Australia), and an IEEE Distinguished Lecturer in 2006–2007.