IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 24, NO. 12, DECEMBER 2015

5033

Visual Quality Evaluation of Image Object
Segmentation: Subjective Assessment
and Objective Measure
Ran Shi, Student Member, IEEE, King Ngi Ngan, Fellow, IEEE, Songnan Li, Member, IEEE,
Raveendran Paramesran, Senior Member, IEEE, and Hongliang Li, Senior Member, IEEE

Abstract— A visual quality evaluation of image object
segmentation as one member of the visual quality evaluation
family has been studied over the years. Researchers aim atdeveloping the objective measures that can evaluate the visualquality of object segmentation results in agreement with humanquality judgments. It is also significant to construct a platform forevaluating the performance of the objective measures in order toanalyze their pros and cons. In this paper, first, we present a novel
subjective object segmentation visual quality database, in which
a total of 255 segmentation results were evaluated by more than
thirty human subjects. Then, we propose a novel full-reference
objective measure for an object segmentation visual quality
evaluation, which involves four human visual properties. Finally,
our measure is compared with some state-of-the-art objective
measures on our database. The experiment demonstrates thatthe proposed measure performs better in matching subjective
judgments. Moreover, the database is available publicly for otherresearchers in the field to evaluate their measures.
Index Terms— Object segmentation, visual quality, subjective
evaluation, objective measure.

I. I NTRODUCTIONOBJECT segmentation as a pre-processing step plays
various important roles for different applications. It aims
at assigning a unique label (“object” or “background”) toeach pixel. In this way, the object-level semantic information can be obtained instead of just the pixel-level information, which is more meaningful and easier to analyze andoperate on. For some image synthesis applications, such asimage editing, image retargeting and 2D to 3D conversion,
the subjective quality of the segmentation result is importantbecause the final output is to be evaluated by the end user,
Manuscript received January 12, 2015; revised June 1, 2015; accepted
August 8, 2015. Date of publication August 25, 2015; date of current version
September 23, 2015. This work was supported in part by the Universityof Malaya, Malaysia, under Project UM.C/625/1/HIR/MOHE/ENG/42 andin part by the Research Grants Council, Hong Kong, under Project CUHK
14201115. The associate editor coordinating the review of this manuscript
and approving it for publication was Prof. Lei Zhang.
R. Shi, K. N. Ngan, and S. Li are with the Department of Electronic
Engineering, The Chinese University of Hong Kong, Hong Kong (e-mail:rshi@ee.cuhk.edu.hk; knngan@ee.cuhk.edu.hk; snli@ee.cuhk.edu.hk).
R. Paramesran is with the University of Malaya, Kuala Lumpur 50603,
Malaysia (e-mail: ravee58@gmail.com).
H. Li is with the School of Electronic Engineering, University of Electronic Science and Technology of China, Chengdu 611731, China (e-mail:hlli@uestc.edu.cn).
Color versions of one or more of the figures in this paper are available
online at http://ieeexplore.ieee.org.
Digital Object Identifier 10.1109/TIP.2015.2473099

i.e., the human viewer. The performance of these applications
is directly influenced by the visual quality of the object
segmentation process. They require the object segmentation
visual quality to be as close as possible to the ground truth(usually generated manually) in order to generate high quality
synthesized images. Therefore, it is necessary to evaluate
whether an object segmentation algorithm can satisfy thisrequirement. Although object segmentation algorithms have
evolved over the past decades, object segmentation quality
assessment is less studied, especially in terms of human visual
perception.
Object segmentation visual quality evaluation sharemany similarities with traditional image quality assessment (IQA) [1], [2]. Firstly, the most reliable evaluation
method for object segmentation visual quality is subjective
evaluation by human observers. However, subjective evaluation is impractical because it is time consuming involving highlabor cost. Therefore, there is a great demand for developing
the objective measures which can automatically evaluate thesegmentation visual quality that correlates well with humanjudgment. Secondly, according to the availability of manually generated ground truth, the objective measures in IQA
can be classified into three types: full-reference, reducedreference and no-reference [1]. Moreover, full-reference andno-reference segmentation quality assessment can also bereferred to as supervised and unsupervised, respectively [3].
Thirdly, we need subjective segmentation quality databases
similar to LIVE [2] and TID2008 [4] to evaluate how well
the objective measures correlate with the subjective evaluation. It should contain the source images, the segmentation results and their corresponding ground truths and theassociated subjective scores. To the best of our knowledge,
there is no such object segmentation visual quality evaluation database available in the public domain to satisfy theserequirements.
In this paper, we focus on the full-reference objective measure which can be used to evaluate segmentation algorithms’
performance. For single object segmentation, the objective
full-reference measures can be classified into three categories:
region-based measures, boundary-based measures and hybridmeasures [5]. Region-based measures distinguish the pixels
as matching pixels or mismatching pixels by overlapping the
ground truth with the segmentation result. Matching pixels
are labeled as “object” in both the ground truth and the1057-7149 © 2015 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

5034

IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 24, NO. 12, DECEMBER 2015

segmentation result, while the mismatching pixels are labeled
differently. Jaccard Index [6] and F1measure [7] are two
typical region-based measures. Jaccard Index is a ratio of
the number of matching pixels to the total number of pixels.
In F1measure, the mismatching pixels are further divided
into false positive and false negative according to their labelsin the ground truth. Then, these two types of mismatchingpixels are measured by their ratios to the matching pixels,
respectively. These two ratios are named as precision andrecall. Finally, the F1measure combines the precision andrecall with equal weight to evaluate the overall segmentation
quality. Since the region-based measures are not sensitive tovariation of the object’s shape, they can only provide roughevaluation of segmentation quality, but cannot reflect humanvisual perception.
Different from the region-based measures, boundary-based
measures need to extract the ground truth’s and the segment’s
boundaries, then evaluate the segmentation quality by measuring the boundary similarity. In [8], Hausdorff distance wasadopted to measure the maximum distortion of the segment’s
boundary. It can only reflect local distortion rather than the
global one. Boundary displacement error as a global measure
was proposed in [9], which used the mean of the shortest
distances between the pixels on the boundaries of the groundtruth and the segment to measure the similarity. It ignored
the fact that the shortest distance is just a relative measure regardless of the image size. Some measures [10], [11]take human perception into account. In [10], fuzzy set theory
was used to assign each pixel with two likelihood probabilities associated with two boundary sets. Then, the difference
between these probabilities indicates the boundary distortion.
Csurka and Perronnin [11] and Kohli et al [12] both introduced
a concept of tolerance band, so the boundary can be adjusted
since they assumed that errors can be tolerated within the band.
The difference is that the bandwidth in [11] is adaptive tothe image size rather than being fixed as in [12]. Althoughthese boundary-based measures can measure shape variation
and reflect the human visual property, they are still sensitive to
distortions compared with region-based measures and cannot
describe the different influence induced by different error
positions.
Some measures can be treated as combinations ofregion-based and boundary-based measures. For example,
Movahedi and Elder [5] proposed a mixed measure basedon the average distance from mismatching pixels to theircorresponding boundary. However, average value is not a goodindex for segmentation quality. In [13] and [14], mismatchingpixels were assigned with different weights according totheir shortest distances to the ground truth boundary andtheir positions. This strategy merely penalized some errors
according to the human visual properties, but did not consider
the compensation induced by the human visual properties. So,a better measure should not just combine the region-based andboundary-based measures together as discussed in [11], butalso comprehensively integrate the human visual properties
into it.
For multiple objects segmentation, Villegas andMarichal [13] proposed two different strategies.

If correspondences can be established between objects
in the ground truth and the segmentation result, the overall
quality can be evaluated by pooling the segmentation qualityof each individual object. Otherwise, multiple objects can betreated as a whole. For the first strategy, the pooling method
is a linearly weighted addition of each individual object’squality. The weight of the individual object is determined by
the area proportion of its corresponding object in the groundtruth in order to reduce an excessively negative influence
induced by badly evaluated small objects. However, thisfactor alone is not enough to describe human perception.
For the second strategy, all the aforementioned measures for
single object evaluation are valid. It is a simple approximation
to the first one [13].
In this paper, we present our work from two aspects:
Firstly, we construct a novel subjective object segmentation
visual quality database [15] which consists of both single
object and multiple objects segmentations, with a total of
255 segmentation results, and there are more than thirty
viewers involved in each survey session. In addition to abrief introduction of the single object database in our previous
conference paper [16], more technical details and analysis
about our database are described in this paper. Secondly, wedevelop a novel objective measure based on four human visualproperties for object segmentation visual quality evaluation.
It includes a single object segmentation quality assessment
method based on [16] and a pooling method for multiple
separated objects segmentation.
This paper is organized as follows. Section II gives thedetails of our subjective database. Section III presents our proposed objective measure for object segmentation visual quality
evaluation. The validity of our database and the performanceof our measure are discussed in Section IV. We conclude ourpaper in Section V.
II. O BJECT S EGMENTATION V ISUALQ UALITY E VALUATION DATABASE
In our database, as mentioned above, each segmentation
result is assigned a corresponding ground truth and a subjectivescore. The main database construction stages include source
images collection, segmentation results generation, subjective
survey and data processing.
A. Source Images Collection
Subjects are sensitive to the distortions in the regions
which bear semantic information. However, different objects
have different semantic meaning. A comprehensive object
segmentation quality evaluation database should include adequate diversity of object categories. The source images withcorresponding ground truth in our database are selectedfrom four well known public object segmentation database:Weizmann [17], VOC2012 [18], MSRA (Image B) [19]and Microsoft Research Cambridge’s grabcut database [20](it selects some images from Berkeley Segmentation
Database [21]). The single object subset includes 85 source
images covering diverse object categories, such as human,
animal, car, boat, plant, everyday goods and building. TheseSHI et al.: VISUAL QUALITY EVALUATION OF IMAGE OBJECT SEGMENTATION

Fig. 1.

Samples of the source images in the single object subset.

Fig. 2.

Samples of the source images in the multiple objects subset.

objects present different sizes and rich texture characteristics.
Fig. 1 shows a collection of the source images in this subset.
In the multiple objects subset, there are 35 source images
with three possible object locations configurations, such as
separateness, overlap and a mixture of both (some objects are
separated and the rest are overlapping). Samples of the source
images in this subset are shown in Fig. 2.
B. Segmentation Results Generation
A good quality evaluation database should satisfy
two requirements: high coverage of possible distortion types
and low redundancy of samples. In the single object subset,
we consider Gelasca’s four basic error types (added region,
added background, border hole and inside hole) [14] and theireleven combinations (C42 + C43 + C44 = 11), which can cover
all possible errors. Examples of the four error types and oneof their combinations are shown in Fig. 3. These four error
types are also summarized in Table. I. Added region and added
background are false positive, and border holes and insideholes are false negative. Added background and border holes
are adjacent to the ground truth boundary, while added region
and inside holes are not. One real segmentation example ofthe four error types is shown in Fig. 4. The segmentation
quality is subjectively divided into five levels (Excellent,
Good, Fair, Poor and Bad) by the authors. In the multiple

5035

objects subset, since each individual object has five quality
levels as mentioned above, there are 31 possible types ofquality level combinations (C51 + C52 + C53 + C54 + C55 = 31) in
one segmentation result. Note that the subscript “5” represents
the five quality levels, and the superscript number indicates
how many different levels appear in the result. For example,
C51 means that there exist five cases that the segmentation
quality of all objects in the result is the same.
We adopt four interactive segmentation methods (namely
seeded region growing [22], interactive graph cut [23],simple interactive object extraction [24] and interactive
segmentation using binary partition trees [25]) provided by
McGuinness’ interactive segmentation tool [10], twosemi-automatic segmentation methods (i.e., Li’s Distance
Regularized Level Set Evolution (DRLSE) method [26], andMohit Gupta and Krishnan Ramnath’s grabcut tool-box [27])and two automatic object segmentation methods (i.e.,Achanta’s [28] and Rathu’s [29]) to generate the segmentation
results. The segmentation results generated by different
segmentation methods have their own characteristics. For
example, Rathu’s method may generate large inside holes,
while seed region growing will not produce any insideholes and added regions. Since the interactive methods use
manual input, they can generate better segmentation quality
compared with semi-automatic and automatic methods. This
database focuses on evaluating the perceptual quality ofthe segmentation results rather than the performance of thesegmentation methods. Therefore, the above methods satisfied
our requirements to generate different types and qualities ofsegmentation errors.
In real practice, we find there are always some errors
perceived on the object boundary more or less. It means thateach segmentation result always has the added background or
the border hole, and it is unimportant to pursue the results
merely having added region, inside hole or their combinations
as shown in Fig. 3(a), (d) and (e). So, we exclude these threeerrors types in the single object part. Although different segmentation methods have their own characteristics, they can still
generate similar results in terms of error types and severity.
Therefore, it is not necessary to collect all segmentation results
into our database. In order to reduce the redundancy, we
select 15 segmentation results with diverse object categories
for each error type and cover the five perceptual quality levels
mentioned above. Finally, there are totally 180 segmentation
results in the single object part.
As mentioned above, five quality levels can generate
31 types of quality level combinations. But two cases(four different quality levels and five different quality levels
in one result) are quite rare in the real segmentation results,
which correspond to six possible combinations. Therefore,
the multiple objects subset focuses on the other 25 quality
level combinations. We select three segmentation results withdifferent object content for each quality level combination.
C. Subjective Surveys
Compared with traditional subjective image/video quality
evaluation, object segmentation subjective quality evaluation

5036

IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 24, NO. 12, DECEMBER 2015

Fig. 3. Illustrations of Gelasca’s four basic error types and one combination. The red rectangle and the blue rectangle represent the ground truth and thesegmentation error respectively. (a) Added region, (b) Added background, (c) Border hole, (d) Inside hole and (e) The combination of the added region andthe inside hole.
TABLE IS UMMARIZATION OF G ELASCA’s F OUR BASIC E RROR T YPES

Fig. 4. (a) Original image [19], (b) ground truth [28], (c) segmentation result [29] and (d) segmentation errors [14]. In (d), the purple regions are added
regions, the blue regions are added background, the orange regions are border holes and the red region is an inside hole.

lacks standard methodologies. In order to overcome the ambiguity in the segmentation results and make the object being
evaluated more apparent, we adopt the simultaneous double
stimulus for discrete evaluation (SDSDE) [30] method to conduct our subjective survey. This method is based on the simultaneous double stimulus for continuous evaluation (SDSCE)
method, which is a standard subjective evaluation method
specified by ITU-R BT.500-13 [31]. The only difference in
SDSDE is the use of the absolute category rating (ACR) scale
which employs a five-grade discrete segmentation quality scale(5: Excellent, 4: Good, 3: Fair, 2: Poor, 1: Bad) [32], which
conforms to the segmentation quality levels in our database.
In [33], the experimental data has demonstrated that there
are no obvious overall statistical differences between the
different rating scales. Therefore, the five-grade discrete scaleis employed to reduce the viewer’s fatigue and make thesubjective rating more distinguishable [30]. The subjective
survey interface is shown in Fig. 5. The source image plays
an auxiliary role to help the viewers understand the image
content. By comparing the segmentation result with the corresponding ground truth, viewers can select the subjective
ratings. The subjective ratings are recorded in numerical values
according to the 5-grade quality scale during the subjective
survey.

In order to analyze the relationship between the individual
object quality and the overall quality, an additional rating
step is designed for the multiple separated objects case. The
procedure is illustrated in Fig. 6. The individual objects are
extracted and viewers are asked to rate their quality one byone. Therefore, there are totally 352 segmentation results inthe subjective survey. Considering the constraint on survey
duration to reduce the effect of the viewers’ fatigue, thesurvey is divided into two sessions (176 images in eachsession). Since human visual comparison does not satisfy
the axiom of symmetry [34], the order of observation ofdifferent results could influence human judgment. In order
to avoid the contextual and memory effects on the subjective
quality ratings, the image triplet (the source image, the groundtruth and the segmentation result) are randomly presented
to each viewer [31]. Furthermore, the segmentation results
generated from the same source image except the multiple
separated objects case will not be presented consecutively.
Note that multiple separated objects case is treated as one
rating group, and the extracted individual object quality rating
follows consecutively the overall quality rating.
Before the formal subjective survey, three pre-sessions arearranged. Firstly, a brief introduction of the objective of thissurvey and how to do the quality evaluation is presented

SHI et al.: VISUAL QUALITY EVALUATION OF IMAGE OBJECT SEGMENTATION

Fig. 5.

Fig. 6.

5037

The interface of our subjective survey [16]. The images from left to right are: the source image, the ground truth and the segmentation result.

An example of the rating group of one multiple separated objects segmentation result. The number in each ground truth indicates the rating order.

to the viewers. Then, a training session is conducted. There
are a total of 10 segmentation results in this session, which
includes single object and multiple objects cases with the different perceptual qualities ranging from “Excellent” to “Bad”.
A quality scale is suggested to the viewers for each training
result. It is emphasized that the viewers should rate the segmentation result independently of the suggested quality scale.
At the end of this session, viewers should pass a “Groundtruth/Ground truth” pair test [31] in order to ensure that theviewers fully understand the survey methodology. Otherwise,
the viewers will be trained again until they pass the test. In thenext session, a mock evaluation session using 6 segmentation
results is conducted to consolidate the viewers’ training.
All the viewers in the subjective test are the students
from the Chinese University of Hong Kong and Shanghai
University. There are 33 and 31 viewers who participated in
the Session 1 and Session 2 respectively. Most of them arenaive viewers who have little experience on image processing.
D. Data Processing
In order to obtain the final mean opinion score (MOS)
and standard deviation value for each segmentation result,
the raw data processing as suggested by [30] is adopted.
Firstly, the raw score ri j k is converted into Z -scor ei j k in orderto reduce the negative influence introduced by the viewer’s
rating habit [35], where i j k indicates the i th viewer rating
the j th segmentation result in the session k = {1, 2}. Then, astandard screening procedure [31] is conducted to reject theunreliable viewers. After this procedure, 3 out of 33 viewers
and 4 out of 31 viewers are rejected in Session 1 andSession 2, respectively. Assuming that the Z -scor es assigned
by a viewer follow the Gaussian distribution, then 99% of thescores will lie in the range [−3, +3] [36]. The Z -scor es arethen linearly mapping to the range of [0, 100] by:100(Z -scor ei j k + 3)
(1)
Zijk =
6
Finally, the MOS value and the standard deviation of eachsegmentation result are computed as follows:
k
1 
Zijk
Nk
i=1


Nk
 1 

std j k =(Z i j k − M O S j k )2
Nk − 1N

M O Sjk =

(2)

(3)

i=1

where Nk is the number of remaining viewers of session kafter the screening procedure. The MOS value and the standard
deviation are treated as the ground truth representing theperceptual quality of the segmentation result. The histogramof the MOS values is shown in Fig. 7. We can see that the
MOS values occupy a wild range of perceptual quality from
low to high.
E. Analysis of the Subjective Survey
In the single object subset, we find that the subjective
evaluation of single object segmentation visual quality can bedivided into three levels:
1) In the low level, viewers use low features, such as areaand boundary, to measure the similarity between the
ground truth and the segmentation result.

5038

IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 24, NO. 12, DECEMBER 2015

Fig. 7.

Fig. 8.
An example of human visual tolerance. (a) Ground Truth,(b) Segmentation Result. The blue-circled part is far worse than the red-circled
part in terms of visual quality.

The histogram of the MOS values.

2) In the middle level, error position is an important factor
that affects the visual quality. Viewers pay more attention to errors inside the object. When viewing similarareas, the errors which make the object incomplete rather
than add redundant background can lead to worse visualquality.
3) In the high level is related to the object semantics. For
example, even though the errors all happen at the interiorof the object, the visual quality can be diverse since different regions may have different semantic information.
Some regions with rich semantic information, such as
face and sign, should not be distorted. Otherwise, thevisual quality is unacceptable.
For multiple objects part, especially the separateness cases, theoverall quality mainly depends on the worst individual object
quality, but it is also influenced by area proportion among theindividual objects. These observations inspire us to design theobjective measure accordingly.
III. P ROPOSED O BJECTIVE M EASURE FOR O BJECTS EGMENTATION V ISUAL Q UALITY E VALUATION
Human visual properties are important cues for designingobjective measures. In [6], [11], [12], and [37], four humanvisual properties for object segmentation quality evaluation are
discussed. Based on their conclusions, these four properties
can be interpreted as:1) Human can put up with added background andborder holes to some extent, i.e., human visualtolerance [10], [11].
2) It is easier for human to quantify small errors, but moredifficult for larger ones, i.e. human visual saturation [10].
3) The perceptual importance of false negative and false
positive pixels are different [5].
4) The overall quality is mostly determined by the stronger
distortions [37].
Our objective measure comprises two parts. One part is single
object quality evaluation based on the first three human visualproperties. The other part is the pooling method for multiple
separated objects based on the fourth property.

A. Single Object Quality Evaluation
The problem formulation is introduced below. Assuming
there are n separated objects (overlapped objects are treatedas one object in our measure) in the ground truth. It can be
represented as G = {G O1 , G O2 , . . . , G On }. Each object in
the ground truth can be assigned a corresponding segment in
the segmentation result, so S = {SO1 , S O2 , . . . , S On }. These
correspondences are determined by the shortest distances ofpixels in the segmentation result to the objects in the groundtruth. Our goal is to measure the similarity Si m(G, S) between
the ground truth G and the segmentation result S. There are
only G O1 and S O1 in the single object case, so we use G and S
for short. Our measure for single object is defined as Eq. (4), as
shown at the bottom of this page, where A(·) is the operationof calculating area. Our measure is based on Jaccard Index [5]with two additional terms: compensation term C O M(·) andpenalty term P E N(·).
1) Compensation Term: The compensation term reflects
human visual tolerance. As shown in Fig. 8, we take two
local regions as an example. The red-circled added backgroundis along the ground truth boundary. It roughly maintains
the shape of the true boundary. However, the blue-circled
added background is protruding which destroys the shapeof the true boundary. Although these two parts have similarerror area, the blue part is far worse in terms of perceptual
quality [5]. In other words, we can tolerate the red-circled
errors to some extent, thus compensating it. For pixel i in
the added background (AB), if it could be tolerated, we
can treat it like a part of the true positive rather than anerror. Since the numerator of the Jaccard Index A(G ∩ S) isthe area of the true positive, we add a compensation value
C O M(i ) to the numerator. Meanwhile, if we can tolerate
pixel j in border holes (B H ), it can be regarded as the true
background rather than a missing part. So its compensation
value C O M( j ) is subtracted from the denominator of theJaccard Index A(G ∪ S) which includes the area of missing
parts. Since added regions (A R) and inside holes (I H ) do
not distort object’s boundary, we do not assign compensation
values to pixels in these regions.

A(G ∩ S) +Si m(G, S) =A(G ∪ S) −j ∈B HC O M(i )i∈AB

C O M( j ) +j ∈B HP E N( j ) +


k∈I H

P E N(k)(4)

SHI et al.: VISUAL QUALITY EVALUATION OF IMAGE OBJECT SEGMENTATION

5039

Fig. 11.
Fig. 9.
Zoom in of the segmentation result. “r” and “b” represent thered pixel and blue pixel respectively.

as follow:
C O M(i ) = 2 −Fig. 10. The comparison of two curves with σ = 2. (a) Logistic function
curve, (b) Compensation function curve.

y=1
1 + exp(−x/σ )(5)where σ is the bandwidth of the exponential function. x and y
can be treated as the physical stimulus and the human
response, respectively. From Fig. 10(a), we can see that the
curve becomes approximately horizontal when the physicalstimulus is large, which is consistent with the human visualsaturation effect mentioned above. According to the above
analysis, we adjust the logistic function to define C O M(i )2
1 + exp(−(DG (i ) + D S (i ))/σ )(6)where DG (i ) and D S (i ) are the shortest distances of the i th
pixel to the ground truth boundary and the segmentation result
boundary, respectively. C O M( j ) can be similarly defined.
Thus, the sum of these two shortest distances is treated as thephysical stimulus, and the output corresponds to the compensation value. If the sum is too large, this pixel severely distorts
the boundary’s shape, and human can tolerate it no more.
So no compensation value should be assigned to this pixel.
We control compensation value by setting a bandwidth [16].
2−

The shortest distances of pixel i to the ground truth boundary and the segmentation result boundary are two importantfactors to measure the tolerance degree [10]. We select one
region of the segmentation result and its zoom-in, which isshown in Fig. 9. The red and blue pixels have similar shortest
distances to the ground truth boundary. However, the blue pixel
has much longer distance to the segmentation result boundary.
Since the blue pixel belongs to the protruding error region,
it should have less compensation value than the red one.
In terms of the sum of the two distances, a smaller sum
indicates that this pixel contribute more to maintaining theobject’s shape well, and vice versa. According to this design,
the compensation values of the red pixel and the blue pixelin Fig. 9 can be distinguished. Furthermore, we take the humanvisual saturation effect into account, i.e., when the object shapeis heavily distorted, viewers find it hard to quantify the severityof errors [10]. In other words, when the sum of the two shortest
distances is larger, the compensation value should be lower andits variation should become smaller. Logistic function [38] asshown in Fig. 10(a) is one type of psychometric functions
which establish a relationship between physical stimulus andthe human response. The following formula is the definitionof the logistic function.

Compensation value map.

Or equivalently,2=τ
1 + exp(−R/σ )
σ = −R ln

τ
2−τ

(7)

−1
(8)

where R is a reference length and τ is the corresponding
compensation value of R. In [11], R is adaptive to the image
size. We follow this approach making R = α · Dlengt h where
α is a constant and Dlengt h is the diagonal length of theimage. τ and α are a pair of parameters. By setting α and τ
appropriately, the compensation value could approach 0 when
the sum is too large. The compensation function is drawn
in Fig. 10(b) with σ = 2. From this figure, we can seethat the compensation value is large when the sum is small.
Conversely, when the sum becomes larger, the compensation
value is lower and its variation also becomes smaller. Theseobservations demonstrate that our compensation function properly reflects human visual tolerance and saturation.
We generate a compensation value map as shown in Fig. 11.
The lighter blue pixels indicate larger compensation values.
From this map, we can see that some added backgroundregions along the ground truth boundary possess larger compensation values while the severely protruding region havecompensation values closely approach to 0 (black), which
conforms to the viewers’ perception.
2) Penalty Term: The penalty terms mainly describe how
the human visual system assigns different weighting to visualquality. Fig. 12 shows a typical example about this property inour database. Although the right image has larger error area,
it still obtains a higher MOS. The reason is that false negative
pixels (border holes and inside holes) degrade the object
itself, which makes it incomplete in the left image; hence,viewers tend to assign higher weights to false negative pixels,

5040

IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 24, NO. 12, DECEMBER 2015

on the left in the ground truth is wrongly segmented to belarger in the segmentation result. This area variation makes
the left earthenware more visually prominent, thus should
be assigned with a higher weight. In addition, we need toconsider the human visual property that overall quality ismostly determined by the stronger distortions.
Our pooling method is thus defined as:
Fig. 12.
An example of different visual quality weights assignment.
(a) MOS = 43.551, (b) MOS = 52.301.

Si m M =

1
n

p=1

w( p) =

(11)(w( p) × sim(G O1 ,S O ) )
p

p

A( p)
n

A(q)

(12)

q=1

A( p) = A(G O p )Fig. 13. An example of multiple objects segmentation result. (a) Groundtruth, (b) Segmentation result.

and give those results which maintain the completeness of theobject better quality ratings. The penalty terms are defined asfollows:
P E N(k) = β k ∈ I H
P E N( j ) = β(1 − C O M( j )) j ∈ B H

(9)
(10)

where β is a constant. For pixels in the inside holes, weassign higher weights. Meanwhile, since pixels in the borderholes can be tolerated to some extent, they are given lower
weighting.
The formulation of Eq. (4) can also be explained from theperspective of the attributes. Each pixel in the segmentation
result is assigned with a basic region attribute by Jaccard
Index, i.e., whether it belongs to the true object or not. On the
one hand, since added background and border holes are along
the boundary, part of their pixels may carry boundary attribute.
However, pixels in border holes and inside holes still carry theobject completeness attribute. Therefore the boundary attribute
may compensate for the loss of region attribute. On the other
hand, completeness attribute can aggravate the loss of region
attribute. We achieve the combination of region cue, boundarycue and human visual properties by integrating the pixels’
multiple attributes.
B. Multiple Separated Objects Quality Evaluation
In this part, we assume that each object’s quality has been
evaluated by its own single object measure described above.
Here, we focus on developing the pooling method to evaluate
the overall quality. As mentioned in [13], the perceptual
relevance of larger objects is usually higher. Therefore area
proportion is an important cue for each object’s weight.
However, we should not only consider the area proportionof objects in the ground truth but also take into accountcorresponding segmented objects in the result. One example
in our database is shown in Fig. 13. The smaller earthenware

A(SO p )(13)

where A(·) represents the area. We use the union area of G O p
and SO p to measure SO p ’s weight w( p), which covers bothcases mentioned above. Our pooling method is a weighted
harmonic mean, whose output is mainly determined by thesmallest one. We take advantage of this characteristic to
describe the human visual property. So, after weighting theoverall quality evaluated by our measure strongly depends on
the worst case.
IV. E XPERIMENTAL R ESULTS
A. Subjective Agreement
Since all segmentation results in the database are subjectively selected by us, it is necessary to firstly test whether
viewers can achieve similar opinions on these results before
any processing on viewers’ ratings. If the ratings are quite
different, the MOS as mean value is meaningless and ourselection of these results fails the consistency requirement.
In [30], each viewer’s ratings compose of a vector, andthe normalized cross correlation (NCC) and the Euclidean
distance (EU D) are used to measure the correlation between
two vectors. NCC and EU D are defined as:
v1 t · v2(14)
 v 1  v 2  v1 − v2 
EU D =(15)d
where v 1 and v 2 are the two vectors corresponding to
two viewers’ ratings and d denotes the dimension of thevector. The higher value of NCC and the lower value of
EU D indicate high correlation between two rating vectors.
We follow this method to test the subjective agreement of ourdatabase. Since there are 33 and 31 viewers in Session 1 and
2 ) and 465(C 2 ) values of NCC and EU D
Session 2, 528(C3331
are obtained from each session, respectively. The average
NCC values are 0.95 and 0.94 for Sessions 1 and 2, and theaverage EU D values are roughly 0.09 for both sessions. The
large NCC values indicate that angular difference between
every two rating vectors is very small. Meanwhile, the small
EU D values reflect the small magnitude difference between
the two rating vectors. These observations demonstrate that theviewers have very good agreement on the segmentation results’
NCC =SHI et al.: VISUAL QUALITY EVALUATION OF IMAGE OBJECT SEGMENTATION

5041

TABLE IIP ERFORMANCES OF S EVEN Q UALITY M EASURESFOR S INGLE O BJECT S EGMENTATION

Fig. 14. From left to right, their MOSs are 36.453, 46.548 and 45.633,
respectively; the Jaccard Index are 0.656, 0.658 and 0.838; the quality scores
given by our measure are 0.660, 0.721 and 0.708.
TABLE IIIP ERFORMANCES OF F OUR P OOLING M ETHODS FORC OMPONENT P ERFORMANCE A NALYSIS

visual quality in our database It means that our selections arereasonable and MOS is a reliable index to measure the visualquality.
B. Overall Performance
Since our objective measure consists of the single object
measure and the pooling method, we test them respectively
using our object segmentation visual quality database.
Following the work of Video Quality Expert Group [39], eachobjective score x is mapped to Q(x) by fitting the following
function in order to obtain a linear relationship with MOS:
Q(x) = β1 ×(0.5 −1)+β4 × x + β5
1 + exp(β2 × (x − β3 ))(16)

For the single object measure, we evaluate its performanceon the single object subset of our database. We use fourcommon performance evaluation criteria, i.e., the Spearman
Rank-Order Correlation Coefficients (SROCC), theLinear Correlation Coefficient (LCC), the root meansquared error (RMSE), and the outlier ratio (OR), which use
Q(x) and MOS as their inputs [40]. Higher values of thefirst two criteria indicate better performance, while the last
two are on the reverse. Our measure is compared against six
object segmentation evaluation measures which are Jaccard
Index (JI) [6], F1measure (F1) [7], Fuzzy Contour (FC) [10],Boundary F1measure (BF) [11], Mixed Measure (MM) [5] andnormalized spatial Weighted Quality Measure (WQM) [13].
In this experiment, we empirically set α = 0.02, τ = 0.1 andβ = 2. The comparison results are shown in Table II. From
the table, we can see that Mixed Measure’s performanceis the worst. It demonstrates that the average distance has
the lowest correlation with the human perception. It also
lacks a suitable normalization step which causes mapping
failure, leading to much lower LCC and higher RMSE.
Two region-based measures, Jaccard Index and F1measure
have similar performance. Without involving human visualproperties, they cannot achieve higher predictive accuracy.
Compared with Fuzzy Contour, Boundary F1measure not
only integrates human visual perception, but also adjusts
the tolerance band according to the image size. So its
performance is better than that of Fuzzy Contour whoseparameters are fixed, and even better than those of the tworegion-based measures. Since Weighted Quality Measure has
merits from both region-based and boundary-based measures,its performance is the second best. Our measure achieves thebest performance in terms of all four criteria. It indicates our“compensation” and “penalty” strategy not only reasonablyorganizes region and boundary information, but also suitably
takes the human visual properties into account. It allows our
objective measure to more closely approximate the humanjudgment. Examples shown in Fig. 14 exhibit the adjustmentof our measure based on Jaccard Index. The Jaccard Index
values of the left and middle figures are nearly the same.
However, MOS of the middle figure is obviously higher thanthat of the left one. Since a part of the added background isalong the star’s boundary which roughly maintains the shapeof the star, our measure increases the Jaccard Index value by
assigning the compensation value to this region. Meanwhile,
although the Jaccard Index value of the right figure is muchhigher than that of the middle one, their MOSs are similar.
Our measure penalizes the border holes which destroy theleaf’s shape significantly. These two adjustments make ourmeasure more consistent with the MOS.
We use 42 groups of multiple separated objects to test
the performance of the proposed pooling method. Since wehave already derived each individual object’s MOS in eachgroup by the subject survey, the pooling method becomes
the only factor which determines the predictive accuracy ofthe overall quality. We directly evaluate the overall quality
by balancing the individual objects’ MOSs. We also compare
our pooling method with the typical method adopted in [13].
The typical method can be generalized as “GA (object areain the ground truth) + LS (linear summation)”. Correspondingly, our method can be generalized as “UA (union area) +HM (harmonic mean)”. In order to evaluate the effectivenessof the processing components, we develop another two combinations “UA + LS” and “GA + HM”. The comparison results
are shown in Table III. This table clearly shows that eachcomponent of our pooling method contributes to the overall
performance. Compared with the union area for calculating
the weight, the harmonic mean which properly describes
the human visual perception plays a more important role.

5042

IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 24, NO. 12, DECEMBER 2015

TABLE IVP ERFORMANCE OF F OUR O BJECT S EGMENTATIONV ISUAL Q UALITY M EASURES

Fig. 16. SROCC of our measure using different α and β values when τ = 0.1.

Fig. 15.
Scatter plots of our measure on the entire database (after the
nonlinear mapping).

As shown in the Table III, the gains provided by HM over LS
are much larger than UA over GA. The experimental results
demonstrate that our pooling method is better at balancing
individual object quality.
Finally, we combine our single object measure and the pooling method to form the complete objective object segmentation
visual quality measure. We test it on our database including all
255 segmentation results (180 single object and 75 multiple
objects results). Table. IV lists the overall performance. The
scatter plot of our measure on the entire database is shownin Fig. 15, where each circle represents one result. The verticalaxis denotes the MOS and the horizontal axis denotes thenonlinearly mapped measure output Q(x).
Since Jaccard Index, Boundary F1measure and Weighted
Quality Measure have better performance in the previous
test, we combine them with the typical pooling method [13]and compare with our measure. The experimental results
demonstrate that our measure outperforms the others for object
segmentation visual quality evaluation on all possible cases inour database, as shown in Table IV.
C. Parameterization
In the proposed measure, there are three parameters to be
determined, i.e., compensation value τ , length proportion α
and additional weight β. As we have mentioned, τ and α
are a pair of parameters to control the bandwidth in ourcompensation function. So, it is not necessary to tune thesetwo parameters simultaneously. Thus, we fix τ = 0.1 and seek
for the best α.
We simply select SROCC as the index to show ourmeasure’s performance under different α and β values on
the single object subset. The result is shown in Fig. 16.
From this figure, we can see that our measure maintains

Fig. 17. Two compensation value maps using different α. (a) α = 0.02,(b) α = 0.04.

good performance under different parameter values. The performance of our measure is not very sensitive to these twoparameters. It means the design of our measure itself isreasonable which can tolerate certain variation of parameters.
According to the SROCC value, α ∈ {0.02, 0.03, 0.04} andβ ∈ [1.0, 3.0] are the better choices. However, if we set
α = 0.04, the compensation range is a little too wide to meet
our intuition. One example is shown in Fig. 17. Comparedwith the compensation value map on the right, the left one
better conforms to our intuition that the arms and left legsshould not have very high compensation values. Therefore,
we set α = 0.02, τ = 0.1 and β = 2 in our measures.
D. Statistical SignificanceIn order to verify the statistical significance of the predictive
accuracy improvement induced by our measure, F-test [41]is conducted on the residuals between the measure outputs(after nonlinear mapping) and MOS. Since the assumption of
F-test is that the residual distribution is Gaussian, we need totest whether this assumption is satisfied before we conduct
the F-test. A simple criterion proposed in [2] is used tomeasure the Gaussianity of the residuals: if the residuals have akurtosis between 2 and 4, they are taken to be Gaussian. After
that, we calculate the ratio between the residual variance ofour measure and that of compared measure (with the larger
variance as the numerator). If the ratio is larger than Fcrit ical
which is a threshold based on the number of residuals and agiven confidence level, then the gain induced by our measure
is considered to be significant at the specified confidence
level. The F-test results on single object subset and the entire
database are given in Table V, where “1” indicates Gaussian,(0) indicating non-Gaussian and the confidence level ofSHI et al.: VISUAL QUALITY EVALUATION OF IMAGE OBJECT SEGMENTATION

5043

TABLE VF-T EST R ESULTS ON O UR D ATABASE

Fig. 18. One example of our failure case. In each pair, the left image isthe ground truth and the right image is the segmentation result. The visualquality is indicated by MOS, and evaluated by Jaccard Index and our measure.
(a) MOS=32.849, JI=0.744 and OUR=0.575, (b) MOS=46.683, JI=0.744
and OUR=0.512.

Fcrit ical is 0.95. As shown in Table IV, all residuals are
Gaussian and all ratios are larger than the corresponding
Fcrit ical . It means that the proposed measure outperforms allcompetitors statistically.
E. Discussion
From the experimental results, we can see that our measurecorrelates better with the subjective segmentation quality.
It can be treated as a candidate benchmark to evaluate an
object segmentation method’s performance in terms of humanperceptual quality. But, there is still about 10% gap between
the predictive value and MOS. One reason is that we treat
overlapping objects as a whole, which is not so precise.
Another reason is about the semantic information. As wehave analyzed, subjective evaluation can be divided into threelevels from low to high. Our measure which currently stays at
low and middle levels which measure the area, the boundary
and the region weights. However, it does not describe thesemantic information which belongs to the high level. Thisis the main reason for the performance gap. One failurecase of our measure is shown in Fig. 18. Since the facecontains important semantic information, distortions on theface should lead to much worse visual quality when comparedwith lossing a part of the petals. However, our measurepredicts that the segmentation result in Fig. 18(a) is better
which is not consistent with MOS. Note that our measureprovides a good interface to involve the semantic information.
The penalty terms can be improved to evaluate the loss ofsemantic information. More specifically, the penalty value canbe varied according to the semantic importance rather thanbeing constant as in the current measure.

V. C ONCLUSION
A subjective segmentation visual quality database isconstructed and introduced in this paper. There are totally
255 segmentation results involving single object and multiple
objects in the database, and more than thirty viewers participated in each survey session. By analyzing the database, subjective evaluation for single object is divided into three levels
and a subjective pooling method for multiple objects is also
verified. In the proposed objective measure, the compensation
term describes the human visual tolerance and saturation; thepenalty term mainly reflects perceptual importance of different
error positions; and the harmonic mean is used to approximate
the subjective pooling method. Moreover, the compensationand penalty terms are also used in the design of the objective
measure in terms of the pixel attribute. In future work, wewill integrate semantic information into our measure, which
is treated as the high level factor in subjective evaluation of
object segmentation visual quality.
R EFERENCES[1] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image
quality assessment: From error visibility to structural similarity,” IEEE
Trans. Image Process., vol. 13, no. 4, pp. 600–612, Apr. 2004.
[2] H. R. Sheikh, M. F. Sabir, and A. C. Bovik, “A statistical evaluation ofrecent full reference image quality assessment algorithms,” IEEE Trans.
Image Process., vol. 15, no. 11, pp. 3440–3451, Nov. 2006.
[3] H. Zhang, J. E. Fritts, and S. A. Goldman, “Image segmentation
evaluation: A survey of unsupervised methods,” Comput. Vis. Image
Understand., vol. 110, no. 2, pp. 260–280, 2008.
[4] N. Ponomarenko, V. V. Lukin, A. Zelensky, K. Egiazarian, M. Carli,and F. Battisti, “TID2008—A database for evaluation of full-reference
visual quality assessment metrics,” Adv. Modern Radioelectron., vol. 10,
no. 4, pp. 30–45, 2009.
[5] V. Movahedi and J. H. Elder, “Design and perceptual validationof performance measures for salient object segmentation,” inProc. IEEE Comput. Soc. Conf. Workshops Comput. Vis. Pattern
Recognit. (CVPRW), Jun. 2010, pp. 49–56.
[6] G. Feng, S. Wang, and T. Liu, “New benchmark for image segmentation
evaluation,” J. Electron. Imag., vol. 16, no. 3, p. 033011, Jul. 2007.
[7] D. M. Powers, “Evaluation: From precision, recall and F-measure
to ROC, informedness, markedness and correlation,” J. Mach. Learn.
Technol., vol. 2, no. 1, pp. 37–63, 2011.
[8] D. P. Huttenlocher, G. A. Klanderman, and W. J. Rucklidge, “Comparing
images using the Hausdorff distance,” IEEE Trans. Pattern Anal. Mach.
Intell., vol. 15, no. 9, pp. 850–863, Sep. 1993.
[9] J. Freixenet, X. Muñoz, D. Raba, J. Martí, and X. Cufí, “Yet another survey on image segmentation: Region and boundary information integration,” in Proc. 7th Eur. Conf. Comput. Vis. (ECCV), 2002, pp. 408–422.
[10] K. McGuinness and N. E. O’Connor, “A comparative evaluation ofinteractive segmentation algorithms,” Pattern Recognit., vol. 43, no. 2,
pp. 434–444, Feb. 2010.
[11] G. Csurka, D. Larlus, and F. Perronnin, “What is a good evaluation
measure for semantic segmentation?” in Proc. 24th Brit. Mach. Vis.
Conf. (BMVC), 2013, pp. 1–11.
[12] P. Kohli, L. Ladický, and P. H. S. Torr, “Robust higher order potentials
for enforcing label consistency,” Int. J. Comput. Vis., vol. 82, no. 3,
pp. 302–324, May 2009.
[13] P. Villegas and X. Marichal, “Perceptually-weighted evaluation criteria for segmentation masks in video sequences,” IEEE Trans. Image
Process., vol. 13, no. 8, pp. 1092–1103, Aug. 2004.

5044

IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 24, NO. 12, DECEMBER 2015[14] E. D. Gelasca and T. Ebrahimi, “On evaluating video object segmentation quality: A perceptually driven objective metric,” IEEE J. Sel. Topics
Signal Process., vol. 3, no. 2, pp. 319–335, Apr. 2009.
[15] R. Shi, K. N. Ngan, and S. Li. Image Object Segmentation Visual Quality
Evaluation Database. [Online]. Available: http://www.ee.cuhk.edu.
hk/~rshi/, accessed Jan. 2015.
[16] R. Shi, K. N. Ngan, and S. Li, “Jaccard index compensation for
object segmentation evaluation,” in Proc. IEEE Int. Conf. Image
Process. (ICIP), Oct. 2014, pp. 4457–4461.
[17] S. Alpert, M. Galun, R. Basri, and A. Brandt, “Image segmentation by
probabilistic bottom-up aggregation and cue integration,” in Proc. IEEE
Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2007, pp. 1–8.
[18] M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, andA. Zisserman. The PASCAL Visual Object Classes Challenge
2012 (VOC2012) Results. [Online]. Available: http://www.pascal-network.org/challenges/VOC/voc2012/workshop/index.html, accessed
May 2013.
[19] T. Liu et al., “Learning to detect a salient object,” IEEE Trans. PatternAnal. Mach. Intell., vol. 33, no. 2, pp. 353–367, Feb. 2011.
[20] C. Rother, V. Kolmogorov, and A. Blake, “‘GrabCut’: Interactive
foreground extraction using iterated graph cuts,” ACM Trans. Graph.,vol. 23, no. 3, pp. 309–314, Aug. 2004.
[21] D. Martin, C. Fowlkes, D. Tal, and J. Malik, “A database of humansegmented natural images and its application to evaluating segmentation
algorithms and measuring ecological statistics,” in Proc. 8th IEEE Int.
Conf. Comput. Vis. (ICCV), vol. 2. Jul. 2001, pp. 416–423.
[22] R. Adams and L. Bischof, “Seeded region growing,” IEEE Trans. PatternAnal. Mach. Intell., vol. 16, no. 6, pp. 641–647, Jun. 1994.
[23] Y. Y. Boykov and M.-P. Jolly, “Interactive graph cuts for optimal
boundary & region segmentation of objects in N-D images,” in Proc.
8th IEEE Int. Conf. Comput. Vis. (ICCV), vol. 1. Jul. 2001, pp. 105–112.
[24] G. Friedland, K. Jantz, and R. Rojas, “SIOX: Simple interactive object
extraction in still images,” in Proc. IEEE Int. Symp. Multimedia (ISM),
Dec. 2005, pp. 1–7.
[25] P. Salembier and L. Garrido, “Binary partition tree as an efficient
representation for image processing, segmentation, and information
retrieval,” IEEE Trans. Image Process., vol. 9, no. 4, pp. 561–576,
Apr. 2000.
[26] C. Li, C. Xu, C. Gui, and M. D. Fox, “Distance regularized level set
evolution and its application to image segmentation,” IEEE Trans. Image
Process., vol. 19, no. 12, pp. 3243–3254, Dec. 2010.
[27] M. Gupta and K. Ramnath. Interactive Segmentation Tool-Box. [Online].
Available: http://www.cs.cmu.edu/~mohitg/segmentation.htm, accessed
Jun. 2013.
[28] R. Achanta, S. Hemami, F. Estrada, and S. Susstrunk, “Frequency-tuned
salient region detection,” in Proc. IEEE Conf. Comput. Vis. Pattern
Recognit. (CVPR), Jun. 2009, pp. 1597–1604.
[29] E. Rahtu, J. Kannala, M. Salo, and J. Heikkilä, “Segmenting salient
objects from images and videos,” in Proc. 11th Eur. Conf. Comput.
Vis. (ECCV), 2010, pp. 366–379.
[30] L. Ma, W. Lin, C. Deng, and K. N. Ngan, “Image retargeting quality
assessment: A study of subjective scores and objective metrics,” IEEE
J. Sel. Topics Signal Process., vol. 6, no. 6, pp. 626–639, Oct. 2012.
[31] Methodology for the Subjective Assessment of the Quality of Television
Pictures, document Rec. ITU-R BT.500-13, 2012.
[32] Subjective Video Quality Assessment Methods for Multimedia
Applications, document Rec. ITU-T P.910, 2012.
[33] Q. Huynh-Thu, M.-N. Garcia, F. Speranza, P. Corriveau, and A. Raake,“Study of rating scales for subjective quality assessment ofhigh-definition video,” IEEE Trans. Broadcast., vol. 57, no. 1, pp. 1–14,
Mar. 2011.
[34] Y. Gavet, M. Fernandes, J. Debayle, and J.-C. Pinoli, “Dissimilarity
criteria and their comparison for quantitative evaluation of image
segmentation: Application to human retina vessels,” Mach. Vis. Appl.,
vol. 25, no. 8, pp. 1953–1966, 2014.
[35] A. M. van Dijk, J.-B. Martens, and A. B. Watson, “Quality asessment ofcoded images using numerical category scaling,” Proc. SPIE, Adv. Image
Video Commun. Storage Technol., vol. 2451, pp. 90–101, Feb. 1995.
[36] K. Seshadrinathan, R. Soundararajan, A. C. Bovik, and L. K. Cormack,“Study of subjective and objective quality assessment of video,” IEEE
Trans. Image Process., vol. 19, no. 6, pp. 1427–1441, Jun. 2010.
[37] S. Li, L. C.-M. Mak, and K. N. Ngan, “Visual quality evaluation
for images and videos,” in Multimedia Analysis, Processing andCommunications. Berlin, Germany: Springer-Verlag, 2011, pp. 497–544.
[38] F. A. Wichmann and N. J. Hill, “The psychometric function: I. Fitting,sampling, and goodness of fit,” Perception Psychophys., vol. 63, no. 8,
pp. 1293–1313, Nov. 2001.

[39] Video Quality Expert Group (VQEG). (2003). Final Report From theVideo Quality Experts Group on the Validation of Objective Models ofVideo Quality Assessment II. [Online]. Available: http://www.vqeg.org[40] S. Li, F. Zhang, M. Lin, and K. N. Ngan, “Image quality assessment
by separately evaluating detail losses and additive impairments,” IEEE
Trans. Multimedia, vol. 13, no. 5, pp. 935–949, Oct. 2011.
[41] D. C. Montgomery and G. C. Runger, Applied Statistics and Probability
for Engineers. New York, NY, USA: Wiley, 1999.

Ran Shi (S’14) received the B.S. degree inelectronic science and technology from the
Changshu Institute of Technology, and the
M.S. degree in signal and information processing
from Shanghai University, in 2009 and 2012,
respectively. He is currently pursuing the
Ph.D. degree with the Department of Electronic
Engineering, The Chinese University of Hong Kong.
His research interests include object segmentation
visual quality evaluation, interactive segmentation,and salient object detection.

King Ngi Ngan (F’00) received the Ph.D. degree
in electrical engineering from Loughborough University, U.K. He was a Full Professor with Nanyang
Technological University, Singapore, and the University of Western Australia, Australia. He has been
appointed as a Chair Professor with the University
of Electronic Science and Technology, Chengdu,
China, since 2012, under the National Thousand
Talents Program. He is currently a Chair Professor
with the Department of Electronic Engineering,
The Chinese University of Hong Kong. He holds
honorary and visiting professorships of numerous universities in China,
Australia, and South East Asia.
Prof. Ngan has published extensively, including three authored books,
seven edited volumes, over 370-refereed technical papers, and edited ninespecial issues in journals. In addition, he holds 15 patents in the areasof image/video coding and communications. He served as an Associate
Editor of the IEEE T RANSACTIONS ON C IRCUITS AND S YSTEMS FORV IDEO T ECHNOLOGY, the Journal on Visual Communications and Image
Representation, the EURASIP Journal of Signal Processing: Image Communication, and the Journal of Applied Signal Processing. He was the Chair and
Co-Chair of a number of prestigious international conferences on image andvideo processing, including the 2010 IEEE International Conference on Image
Processing, and served on the advisory and technical committees of numerous
professional organizations.
Prof. Ngan is a fellow of IET (U.K.), and IEAust (Australia), and an IEEE
Distinguished Lecturer in 2006-2007.

Songnan Li (M’13) received the B.Sc. and
M.Phil. degrees in computer science and technology
from the Harbin Institute of Technology, China,
in 2004 and 2006, respectively, and the Ph.D. degree
in electronic engineering from The Chinese University of Hong Kong (CUHK), in 2012. He joined
CUHK in 2007, as a Research Assistant. From2012 to 2014, he was appointed as a Post-Doctoral
Fellow with the Department of Electronic Engineering, CUHK, where he is currently a Research
Assistant Professor with the Department of Electronic Engineering. His research interests include image and video processing,
RGB-D computer vision, and visual quality assessment.

SHI et al.: VISUAL QUALITY EVALUATION OF IMAGE OBJECT SEGMENTATION

Raveendran Paramesran (SM’01) received the
B.Sc. and M.Sc. degrees in electrical engineeringfrom South Dakota State University, Brookings,
SD, USA, in 1984 and 1985, respectively, and the
Ph.D. degree in engineering from the University ofTokushima, in 1994. He was a Systems Designerwith Daktronics, USA. He joined the Department
of Electrical Engineering, University of Malaya,
Kuala Lumpur, in 1986, as a Lecturer. In 1992,
he received a Ronpaku Scholarship from Japan topursue Doctorate degree. He was promoted to Associate Professor and Professor, in 1995 and 2003, respectively. His researchareas include image and video analysis, formulation of new image descriptors
for image analysis, fast computation of orthogonal moments, analysis of
EEG signals, and data modeling of substance concentration acquired fromnoninvasive methods. His contributions can be seen in the form of journal
publications, conference proceedings, authored books, chapters in books, andan international patent to predict blood glucose levels using nonparametricmodel. He has successfully supervised to completion 14 Ph.D. students and12 M.EngSc. students (master’s by research). He is the President of Malaysia
Image Analysis and Machine Intelligence. He is a member of the Signal
Processing Society.

5045

Hongliang Li (SM’12) received the Ph.D. degree
in electronics and information engineering fromXi’an Jiaotong University, China, in 2005. From2005 to 2006, he joined the Visual Signal Processing and Communication Laboratory, The Chinese
University of Hong Kong (CUHK), as a Research
Associate. From 2006 to 2008, he was a PostDoctoral Fellow with the Visual Signal Processing
and Communication Laboratory, CUHK. He is currently a Professor with the School of Electronic
Engineering, University of Electronic Science andTechnology of China. His research interests include image segmentation,
object detection, image and video coding, visual attention, and multimedia
communication system.
Dr. Li has authored or co-authored numerous technical articles in wellknown international journals and conferences. He is a co-editor of a Springer
book titled Video Segmentation and its Applications. He was involved inmany professional activities. He is an Editorial Board Member of the Journal
on Visual Communications and Image Representation (Elsevier Science),
and an Area Editor of Signal Processing: Image Communication (Elsevier
Science). He was a Technical Program Co-Chair in ISPACS2009, a General
Co-Chair of the 2010 International Symposium on Intelligent Signal Processing and Communications Systems, a Publicity Chair of the IEEE VCIP 2013,
and the Local Co-Chair of the 2014 IEEE International Conference on
Multimedia and Expo (ICME). He serves a Technical Program Co-Chair of
the IEEE Visual Communications and Image Processing Conference in 2016.
He served as the Technical Program Committee Member in a numberof international conferences, such as ISCAS2014, ISCAS2013, ICME2014,
ICME2013, ICME2012, ISPACS2005, PCM2007, PCM2009, and VCIP2010.
He was selected as the New Century Excellent Talents in University from the
Ministry of Education, China, in 2008.