2228

IEEE TRANSACTIONS ON MULTIMEDIA, VOL. 18, NO. 11, NOVEMBER 2016

No-Reference Retargeted Image Quality Assessment
Based on Pairwise Rank Learning
Lin Ma, Member, IEEE, Long Xu, Member, IEEE, Yichi Zhang, Yihua Yan, and King Ngi Ngan, Fellow, IEEE

Abstract—In this paper, we propose a novel no-reference image
quality assessment method for the retargeted image based on
the pairwise rank learning approach. Each retargeted image
needs to be first represented as a feature vector, which not
only captures the image characteristics but also is sensitive to
distortions during the retargeting process. As such, we investigate
and examine different image representations for their abilities
depicting the perceptual quality of retargeted image. Based on
the image representations, we resort to the pairwise rank learning
approach to discriminate the perceptual quality between theretargeted image pairs. Experimental results demonstrate that the
proposed method can effectively depict the perceptual quality of
the retargeted image, which can even perform comparably withthe full-reference quality assessment methods.
Index Terms—Image quality assessment, no-reference, rank
learning, retargeted image.

I. INTRODUCTIONOWADAYS, with the rapid development of mobile devices, new applications of image/video have appeared indifferent terminal devices to improve the viewers’ visual qualityof experience (QoE) [1]–[5]. It is greatly demanded to display
the same image/video content on all kinds of terminals, suchas the mobile phone, tablets, and so on, to provide more conveniences and better QoEs for the viewers. However, different
terminals are of different resolutions, which require a better
displaying technique to better meet the viewers’ perception of
the visual contents. Traditionally, simple scaling and cropping
methods are adopted to alter the image resolution arbitrarily.
However, the salient content cannot be well preserved, whichfurther introduces some unpleasant viewing experiences. In order to handle the drawbacks of scaling and cropping processes,
several content-aware retargeting methods [1], [3], [6]–[12] areN

Manuscript received April 29, 2016; revised August 30, 2016; accepted
September 21, 2016. Date of publication September 27, 2016; date of current version October 19, 2016. This work was supported in part by a grant fromthe Research Grants Council of the Hong Kong SAR, China (Project CUHK
415913), and by the National Natural Science Foundation of China under Grant
61572461 and Grant 11433006. The work of L. Xu was supported by CAS
100-Talents. The guest editor coordinating the review of this manuscript andapproving it for publication was Prof. Daniel Keim. (Corresponding author:
Long Xu.)
L. Ma is with Tencent AI Lab, Shenzhen 518057, China (e-mail: forest.
linma@gmail.com).
L. Xu and Y. Yan are with the Key Laboratory of Solar Activity, National
Astronomical Observatories, Chinese Academy of Sciences, Beijing 100049,
China (e-mail: lxu@nao.cas.cn; yyh@nao.cas.cn).
Y. Zhang and K. N. Ngan are with the Department of Electronic Engineering,
The Chinese University of Hong Kong, Hong Kong, China (e-mail: yczhang@ee.cuhk.edu.hk; knngan@ee.cuhk.edu.hk).
Color versions of one or more of the figures in this paper are available online
at http://ieeexplore.ieee.org.
Digital Object Identifier 10.1109/TMM.2016.2614187

developed to adapt the image to different resolutions, which
discard the unimportant information and meanwhile preserve
the salient and important content information of the image. In order to demonstrate the superiorities of the retargeting methods,
the authors usually performs the sample visual comparisons.
However, such comparison cannot be performed at large scaledue to a huge labor. More importantly, it cannot be employed forthe online optimization and guidance of the retargeting process.
Therefore, developing one metric to automatically evaluate theretargeted image quality is demanded, not only for evaluationand comparison but also for guiding the optimization process ofretargeting.
Mean squared error (MSE) and the related peak signal-tonoise ratio (PSNR) are widely adopted to evaluate the fidelity
between one signal and its corrupted version, because of its simplicity, easy optimization, and clear physical meaning. However,
MSE and PSNR have been criticized that they are not able tosimulate the perception of human visual system (HVS) [13],[14]. As such, many quality metrics have been developed tohandle the drawbacks of MSE and PSNR and overcome theirshortcomings. The most famous and representative one is the
structural similarity (SSIM) [15], which is proposed to evaluate
the image perceptual quality from three perspectives, specifically, the luminance, contrast, and structure, other than theimage pixel difference. SSIM has been demonstrated to be effective to evaluate both natural image and video signals. Recently,
more effective quality metrics have been proposed, such as feature similarity (FSIM) [16] for natural image, and motion-based
video integrity evaluation (MOVIE) [17] for natural video. Theses quality metrics have achieved great success on characterizing
the traditional distortions, such as JPEG compression, Gaussian
noise, Gaussian blur, and so on. However, for the quality assessment of retargeted images, the most straightforward observationis that the resolutions between the original and retargeted images have been altered. The developed quality metrics, such as
PSNR, SSIM, FSIM, and so on, cannot evaluate two images
with different resolutions.
More specifically, the distortions in retargeted images are introduced from the following twofold perspectives [18]. Firstly,
the retargeting process will inevitably discard partial information of the image content, even though they are not as importantas other salient image information. Such discarding process will
introduce the information loss to the retargeted image. Therefore, how to measure the effect of the discarded content on theperceptual quality of the retargeted image is very important anddifferent from the traditional distorted image. Secondly, rather
than information loss, one new distortion, namely the shape
distortion, is introduced during the retargeting process. The

1520-9210 © 2016 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See http://www.ieee.org/publications standards/publications/rights/index.html for more information.

MA et al.: NO-REFERENCE RETARGETED IMAGE QUALITY ASSESSMENT BASED ON PAIRWISE RANK LEARNING

retargeting methods [1], [3], [6]–[12] remove the image pixels,
seams, and regions to meet the requirements of the final resolution. The shapes, such as the object boundaries and contours,
will be inevitably distorted, which will severely degrade the perceptual quality of the image and significantly affect the viewing
experiences of subjects. Overall, the retargeted image differswith the original image in resolution and distortion types, making its perceptual quality much more challenging for evaluation.
In this paper, we propose to evaluate the retargeted image
quality in a no-reference (NR) manner. We firstly formalize the
NR quality assessment as a pairwise ranking problem. Afterwards, a rank learning method is proposed to discriminate theperceptual quality of the retargeted image. Based on the learned
ranking model, one can measure retargeted images without references. Our main contributions are listed in the following.
1) We first propose an NR quality assessment method forretargeted images, which achieves comparable performances with the full reference (FR) quality metrics on
the public retargeted image quality databases.
2) We formalize the NR quality assessment as one pairwise ranking problem, where the image perceptual quality
can be effective depicted by our proposed rank learning
approach.
3) We examine and investigate the effects of different image
representations on the retargeted image quality assessment (IQA).
The paper is organized as follows. Section II discusses the related work on retargeting and NR IQAs. We formulate the problem of NR IQA and introduce the pairwise rank learning framework in Section III. Detailed information about our proposed
pairwise rank learning metric for retargeted images is introduced in Section IV. And the experimental results are illustrated
in Section V. Finally, conclusions are provided in Section VI.
II. RELATED WORK
A. Retargeted Image Quality Assessment
Nowadays, there are several research works discussing theretargeted IQA. Based on the approaches, the research work
can be roughly categorized into the subjective [18], [19] andobjective [4], [20]–[22] approaches. The subjective approachis the most reliable way for assessing the perceptual qualityof the retargeted image. However, it requires many subjects toparticipate in the subjective testing process, which is very time
consuming and of a huge labor work. Thus, it cannot be used
for online manipulation and optimization. The subjective testing process is usually employed to construct the database, which
provides the ground truth value of the image perceptual quality.
In [19], the authors constructed the RetargetMe database, which
employed one pairwise comparison method to indicate which retargeted image possesses a better perceptual quality, comparedwith the other retargeted images. The RetargeteMe databaseconsists of the retargeted image and the number of times thatthe image is favored over other images. In [18], [23], the authors
employ the simultaneous double stimulus for continuous evaluation (SDSCE) [24] to perform the subjective quality evaluation.
The subjects provide their personal opinions on the perceptual

2229

quality of the retargeted image with the original image as reference. The database is composed by the retargeted image aswell as its corresponding mean opinion score (MOS). Moreover, the impact of the retargeting process on human fixations
is examined, by gathering eye-tracking data for a representative
benchmark of retargeted images in [25].
For the objective quality metrics for retargeted images, a metric named as bidirectional similarity (BDS) is developed in [20],[22]. Two visual signals, namely the original and retargeted images, are considered to be visually similar where as many as possible patches of one visual signal are shared by the other visualsignal in a bidirectional manner. BDS tries to depict the similarity between two visual signals from the completeness and coherence perspectives, which can be further employed to generate
a retargeted image by minimizing the similarity measurement.
The scale-invariant feature transform (SIFT) [26] is proposed tocharacterize the view-invariant and brightness-independent image structures. Therefore, matching SIFT descriptors can help
to establish meaningful correspondences across images withsignificantly different image contents. Moreover, the pixel displacements obtained from the SIFT flow [27] are spatially coherent. The matching cost can thus indicate the difference andpixel displacement between the original and retargeted images.
In [4], Fang et al. proposed one metric, namely IR-SSIM, to create an SSIM quality map that indicates at each spatial location of
the reference image how the structural information is preservedin the retargeted image. For each pixel in the original image,the best matching pixel in retargeted image is firstly located.
The SSIM measurement is calculated between the local regionsof the original and retargeted image around specific matchingpixels. After obtaining the SSIM quality map for the reference
image, a saliency map is developed to pool the SSIM quality
map into a final quality score. Zhang and Ngan [28] proposed
one metric named as RB-RIQA by employing a region-based
framework to evaluate shape distortion and information lossof the retargeted images. Liang et al. [29] proposed a simple
yet efficient objective assessment method based on similarity,
satisfying aesthetic requirements, and maintaining symmetry
features.
B. No-Reference Image Quality Assessment
The aforementioned metrics for retargeted images can be regarded as the FR metrics, where the original reference image
needs to be present for quality evaluation. However, in practicalscenarios, the reference image is always unavailable. Therefore,
NR quality metrics are highly demanded for quality assessment.
There are also several research works discussing NR IQA, which
can be roughly classified into three categories. The first category
approach takes the behavior of specific distortions into consideration. For example, in [30], Sheikh et al. employed the wavelet
statistical model to capture the distortion introduced by JPEG
2000. Brandão and Queluz [31] proposed an NR quality metricbased on the discrete cosine transform (DCT) domain statistics
to evaluate the quality of JPEG coded image. The second category approach relies on the quality aware clustering. The image
patches of training set are grouped into the given number of2230

IEEE TRANSACTIONS ON MULTIMEDIA, VOL. 18, NO. 11, NOVEMBER 2016classes based on local image features [32], [33]. Each cluster
center has a quality score which is derived from the qualities
of image patches falling into this cluster. Associating cluster
centers with their qualities, a codebook is established. For thepatches of a test image, the codebook is looked up to search
the most similar codeword and retrieve the associated quality
values. In [32], a visual codebook associated with the Gaborfilter based local appearance descriptors as well as the MOS is
proposed. The authors of [33] used FSIM [16] instead of MOS
as image patch quality to establish the codebook. The third category approach is to utilize machine learning method [34] to
map image features into image qualities. In [35], Moorthy and
Bovik proposed to use support vector machine (SVM) [36], [37]and support vector regression (SVR) [38], [39] to learn a classifier and an ensemble of regressors for a distortion-aware quality
metric. It deploys summary statistics called natural scene statistics (NSS) which is derived from wavelet decomposition of animage. In [40], Tang et al. proposed an approach similar to [35]but with more elaborate features, including texture statistics,
blur/noise statistics, and histogram of each subband of image
decomposition.
Current NR quality metrics are developed for natural images
with traditional distortions, such as JPEG, JPEG 2000, Gaussian blur, and so on. However, for retargeted images, to our best
knowledge, there is no literature discussing the NR quality assessment methods. In this paper, we make the first attempt toaddress the NR quality metrics for retargeted images based on
the pairwise rank learning approach.
III. NO-REFERENCE RETARGETED IMAGESQUALITY ASSESSMENT
In this section, we first reformulate the NR IQAs as one
pairwise comparison problem, based on which our proposed
pairwise rank learning metric for retargeted images is thereby
introduced.
A. Problem Formulation
Nowadays, there is a thread of work on the learning based NR
IQAs [16], [32], [33], [35], [40]–[42]. The authors represent theimage or image patch as one feature vector. Afterwards, the clustering or regression learning methods are employed to generate
the perceptual quality index for the provided image or image
patch. Such NR metrics are developed to measure the consistency between the generated quality index and the ground truth
value obtained during the subjective testing process [24], [43].
However, as discussed in [2], such subjective testing process
suffers from some problems, such as dissimilar interpretationsof the quality scales, and so on. Therefore, regressing the image
or image patch into a quality index without reference will bealso very difficult. Recent work turns to an alternative approachwith pairwise comparison for subjective evaluation [19], [44].
For the pairwise subjective comparison, a participant is simply
asked to compare two visual signals simultaneously, and determine which one is of a better perceptual quality. As such,the decision in pairwise comparison is simpler than the traditional scale rating [24], [43]. Same as the pairwise subjective

comparison, we also formulate the NR quality assessment asa pairwise comparison problem, with a fundamental departurefrom the family of existing machine learning method for quality
assessment [16], [32], [33], [35], [40].
Inspired by the development of rank learning [45] in information retrieval, we do not regress the image to a specific quality
index but learn to rank a pair of images. As such, we would
like to develop a quality assessment model targeting at ranking
images instead of assigning a quality score to each image. For
general settings in information retrieval, it usually ranks the retrieved items by their relevance with the query. However, to ourend concerning the quality evaluation of retargeted image, wemeasure the retargeted image qualities by their orders insteadof quality score⎧⎨1, if yi < yj
(1)
fr (xi , xj ) =⎩0, if y ≥ yijwhere xi and xj are the image representations of Ii and Ij ,respectively. yi and yj are the corresponding ground truth quality scores annotated by the human subjects. fr is the ranking
function that has the binary outputs of 0 and 1, representing theinconsistency and consistency between predicted order of image
quality and ground truth subjective values respectively. Our target is to learn the ranking function fr based on the training data.
Afterwards, the learned ranking function fr can help evaluating
all the image pairs. We can further generate the image quality
scores by referring to a exponential curve fitting function basedon the rankings. As such, the image quality ranking information
can be converted to its quality index. The traditional statistical measurements [46] can thus be employed for quality metricperformance evaluation.
B. Pairwise Rank Learning Framework
The framework of the proposed rank learning based NR
quality assessment of retargeted image is illustrated in Fig. 1,
consisting of two phases, specifically the training and testing
phases. During the training process, we firstly prepare the image
pairs for the pairwise rank learning approaches. The retargeted
image pairs are prepared and sampled by referring to their subjective quality scores. Specifically, the image with better subjective quality is admitted to rank higher than the one with poorer
subjective quality. Afterwards, the generated retargeted image
pairs are further represented as feature vectors, which are expected to be not only able to capture the characteristics of theimages, but also sensitive to the distortions introduced during
the retargeting process. Based on the extracted image featurepairs, we resort to the pairwise rank learning to learn a ranking
model, which can discriminate and distinguish the perceptual
quality of the two provided retargeted images. Based on the
ranking results from the learned model as well as their subjective quality scores, an exponential function can be further fitted
to map the rankings to the subjective scores.
For the testing phase, all the possible retargeted image pairs
are sampled to measure their perceptual quality rankings. Sameas the training process, the image pairs are first represented

MA et al.: NO-REFERENCE RETARGETED IMAGE QUALITY ASSESSMENT BASED ON PAIRWISE RANK LEARNING

2231

Fig. 1. Pairwise rank learning framework for NR quality assessment of retargeted images, consisting of the (a) training and (b) testing phases. During the training
phase, the image representations of the sampled image pairs as well as their subjective quality scores are employed to train a ranking model. During the testing
phase, the obtained training model is employed to discriminate perceptual qualities of the test image pairs. Based on the obtained pairwise rankings, the perceptual
quality index of each test image is generated via a curve fitting function.

as feature vectors. The ranking model then discriminates theirrelevant rankings based on the extracted features. With all thepossible ranking pairs on the testing images, the fitted exponential function further maps the rankings to the image perceptual
quality indexes.
IV. PAIRWISE RANK LEARNING BASED QUALITY ASSESSMENT
FOR RETARGETED IMAGESIn this section, each process during the training and testing
phases is introduced in details.
A. Image Representation
For retargeted image quality assessment, the image representation is very important for quality analysis. As discussed in[18], the features should be sensitive to the distortions introduced during the image retargeting process. More specifically,
the features should not only capture the shape information, butalso depict the image content information loss. The shape information is more important to the quality assessment of theretargeted image. In this paper, we examine different image
representations for the purpose of retargeted IQA. We compare
different image representations for retargeted images, specifically the BDS [20], [22], SIFT flow [27], earth mover’s distance(EMD) [47], MPEG-7 [48], pyramid histogram of visual words(PHOW) [49], GIST [50], and the image representation fromthe deep neural networks, specifically the VGG [51]. As demonstrated in Section V, GIST outperforms other image representations on retargeted image quality measurement in terms ofcorrelation with HVS perception. The reason is mainly attributed to that GIST is demonstrated to capture the global information. Therefore, it can effectively capture the shape distortions which are introduced during the retargeting process [52]and severely degrade the retargeted image perceptual quality.
Detailed information about the different image representations
can be found in Appendix.
B. Training Data Preparation
As illustrated in Fig. 1, we need to generate image pairs forboth the training and testing processes. We carry out our workon the existing subjective retargeting image database, specifically the CUHK [18] and RetargetMe [19] databases. The two
databases provide the subjective score for each retargeted image. Detailed information about this database can be found in
Section V-A.
We denote the GIST feature vectors as {xi , i = 1, 2, . . . , n}and the corresponding subjective scores as {yi , i = 1, 2, . . . , n}
for the retargeted images {Ii , i = 1, 2, . . . , n}. y may be the
MOS value in CUHK database or the favored times in RetargetMe database. In both cases, the larger value yi means thebetter perceptual quality of the retargeted image Ii . To establish the pairwise rank learning task for quality assessment ofretargeted images, two images (Ii , Ij ) are simultaneously sampled as a pair to form the training set. The GIST feature vectors(xi , xj ) are used to represent the retargeted images. The subjective scores yi and yj are compared to generate their ranking
label {+1, −1} for the feature vector pair (xi , xj ). If yi < yj ,

2232

IEEE TRANSACTIONS ON MULTIMEDIA, VOL. 18, NO. 11, NOVEMBER 2016

the ranking label is set as +1 for the training pair (xi , xj ).
Otherwise, we will assign the ranking label −1 to (xi , xj ) if
y i > yj .
C. Pairwise Rank Learning
Based on the prepared training data (retargeted image pairs
as well as their ranking labels), we introduce the pairwise rank
learning approach to learn the ranking model for NR retargeted
IQA in this subsection.
Conventionally, the intrinsic principle of machine learning for
IQA is explored to optimize the numerical distance between thesubjective value yi (e.g. MOS value) and the predicted image
quality index ψω (xi ) nω̂ = arg min ψω (xi ) − yi p
(2)
ω

i=1

where ψω denotes the learned function based on the training
data and used to compute the image quality index for an input
image. ω denotes the parameters of the learned function ψω .
xi represents the feature vector of the i-th image Ii . yi is the
subjective quality score of Ii .  · p represents p-norm distance.
For simplicity, we usually assume that the learned function ψωto be a linear formationψω (x) = wx

(3)

where w is one matrix to be learned for mapping the image
feature vector x to the predicted perceptual quality ψω (x). Also
nonlinear mapping functions can be employed, which explore
the nonlinear and complicated relationships between image features and subjective values. By using kernel functions, nonlinear
problems can be converted to linear problems. Observing the
optimization objective of (2), the p-norm distance is optimized.
However, for our pairwise image ranking function, a new optimization objective based on the binary comparison of image
quality is established asyi < yj I ψω (xi ) ≥ ψω (xj ) I(4)arg mini= jwhere []I = 1 if the logic decision  holds; otherwise []I = 0.
Instead of the p-norm distance, (4) is established on the ranksof image qualities other than the numerical values. From (4), a
false rank prediction, i.e., the order of two images in violation of
the ground-truth, would result in the increase of its cost. In order to make a comprehensive utilization of the training data, (4)concerns all possible pairwise comparisons of image qualities
among all training images. Obviously, an image which has thedistinct quality different from others would contribute more to
the defined objective function. Intuitively, if the rank is wrongly
predicted, i.e., contradictory to the ground-truth given by the
subjective preference, the penalty should be emphasized to refrain from such an occurrence. On the contrary, the images withsimilar image qualities tend to contribute less to the optimization
objective. In practice, we set a threshold T to realize the task of
training data selection by excluding the cases of |yi − yj | ≤ T

in (4) for compressing noise and reducing computational complexity.
For simplicity, we also employ the linear function as in(3) to optimize the objective function defined in (4). As such,the optimization objective is to seek a matrix w, which resultsin the minimum value of (4) on the generated training set. Witha linear function ψω (x), (4) is reformulated asyi < yj I wxi ≥ wxj I .
arg min
(5)
w

i= j

We further define an empirical loss L(w) asyi < yj I wxi ≥ wxj I .
L(w) =(6)i= j

We aim to minimize L(w) based on the training data. Since[]I is non-convex, we encounter a non-convex optimization
problem. In order to handle such problem, the Boolean termin (5) is replaced by its upper bound to relax the non-convex
optimization to the convex one(7)
wxi ≤ wxj I ≤ ew (x i −x j ) .
The exponential upper bound ew (x i −x j ) is employed as it isconvex and can efficiently facilitate the optimization process.
The derivative of the exponential upper bound ew (x i −x j ) is obtained by∂ w (x i −x j )e= (xi − xj )ew (x i −x j ) .
(8)∂w
After replacing the term containing the variable w in (5),the empirical loss function L(w) would turn out to be convex.
Afterwards, the gradient decedent method can be employed tosolve (5). The gradient decent direction of L(w) can be writtenas
yi < yj I (xi − xj )ew (x i −x j ) .
(9)
Δw =
i= j

λ acting as an iteration step is further employed to control theconvergence speed during the training process, with the multiplication of Δw.
For (5), given xi , yi , and an initial w, the empirical loss
L(w) can be thus initialized. Replacing w by w + Δw, L(w)can be updated. By iteratively updating w and L(w), the global
minimum objective can be reached, which results in the final
learned ranking function for NR retargeted IQA.
D. Image Quality Generation From Ranking Results
It should be pointed out that the optimization objective in(5) is established intrinsically on image quality ranking insteadof image quality score. However, it cannot directly output image
qualities. For CUHK database, all the MOS values are available
during training. The relation between MOS values and theirrankings can be fitted by a polynomial function, which helps to
output image qualities in the form of quality scores.
In [53], a term “gain” is defined as the number of times of thatan image is preferable against the others. For IQA, the “gain” of aretargeted image is proportional to its perceived quality becauseMA et al.: NO-REFERENCE RETARGETED IMAGE QUALITY ASSESSMENT BASED ON PAIRWISE RANK LEARNING

2233

score can be easily deduced based on the given image quality
rankings. As a accumulative summation is proposed in (10) for
generating d(i), the fitting process of (11) is expected to berobust to the noise interference.
V. EXPERIMENTAL RESULTS
In this section, we present the performances of our proposed
method and other quality metrics for retargeted images. Firstly,
we briefly introduce the image retargeting quality databases.
Afterwards, different image representations as well as their effectiveness on retargeted IQA are presented. Finally, the performances of our proposed NR metric in terms of different
statistical measurements are illustrated.
A. Image Retargeting Quality Database
Fig. 2. Scatter plot of the MOS value and accumulative image quality
difference.

the preference (better or worse image quality) essentially reflects
the relative quality (ranking instead of numerical value) of thisimage relative to others. Moreover, a linear mapping between
the “gain” and the quality score is assumed and can be fitted bytraining data. However, the differences between the test image
and the training image needs to be computed [53], which makeit impractical in real applications as the training images are notavailable. In this paper, our proposed rank function generates arank list of all the images instead of only binary preference ofeach image pair. Afterwards, a mapping function between therank list and image quality scores can thus be fitted. During thetraining process, a nonlinear fitting function is deduced frommapping the predicted rank list to the subjective score rank
list. In the test stage, the nonlinear fitting function can thus
tell quality score of each test image without the need of any
information from the training set.
The i-th image is compared with other images in the training
set. We compute the accumulative image quality difference asnj = i wxi − wxj +.
(10)d(i) =nj = i wxi > wxj I
The numerator nj= i [wxi − wxj ] + summarizes the perceptual quality differences between xi and all the other images,
which are of lower perceptual quality than xi . The denominatornj = i [wxi > wxj ]I captures the number of images, which areof lower perceptual quality than xi . Therefore, (10) generates
the accumulative image quality difference d(i) for xi by representing the relative quality of the i-th image against others. We
further illustrate the scatter plot of the MOS value and d(i) of
the training images in the CUHK dataset, which is illustrated inFig. 2.
We can further fit a mapping function between d(i) and image
subjective score y(i) by an exponential function asy(i) = α1 + α2 × eα 3 × d(i) .

(11)

The parameters α1 , α2 , and α3 can be easily obtained by thenonlinear least square regression. As such, the image quality

As mentioned before, there are two public subjective quality
databases for retargeted images, specifically the RetargetMe and
CUHK databases.
1) CUHK database [18] is composed by 171 retargeted images in total generated from 57 original images. Eachretargeted image is accompanied with its MOS value provided by the subjective viewers, which is denoted as its
ground truth perceptual quality.
2) RetargetMe database [19] consists of 37 images. Each
image is used to generate the retargeted results using 8
different retargeting algorithms. The ground truth rank ofeach image is obtained by sorting the number of times
that it is preferred over other retargeted images by the
subjective viewers.
B. Experimental Results
1) Performance Comparisons Between Different Image Representations: For the CUHK database, each image is accompanied with the MOS value to indicate its perceptual quality. The predicted image quality scores are obtained from the
pairwise rankings as introduced in Section IV-D. Afterwards,
three measurements, specifically the Spearman’s rank correlation coefficient (SROCC), Pearson linear correlation coefficient(PLCC), and outlier ratio (OR) are employed to evaluate therelationships between the ground truth subjective score and thepredicted quality score from each quality metric [46]. Table I
illustrates the performance comparisons between different image representations. For EMD, the histogram is constructed torepresent the feature distribution of the image, which is not able
to capture enough information for the retargeted IQA. PHOW
can somewhat extract the shape information. However, we needto maintain a visual vocabulary to compose the corresponding
histogram at each pyramid scale. Consequently, the shape information is mostly extracted from the local perspective, although apyramid structure is employed. As illustrated in [18], the global
shape information is very important for retargeted image. The
local features to evaluate the quality of retargeted image cannot
yield a good performance. For the descriptors of MPEG-7, theEHD performs the best. The reason is that the local shape information is depicted by the edge histogram in local regions. The

2234

IEEE TRANSACTIONS ON MULTIMEDIA, VOL. 18, NO. 11, NOVEMBER 2016

TABLE IPERFORMANCE COMPARISONS OF DIFFERENT IMAGE
REPRESENTATIONS ON THE CUHK DATABASE

SCD
CSD
CLD
HTD
EHD
BDS
EMD
PHOW
SIFT flow
VGG
GIST

SROCC

LCC

OR

0.17920.16880.08500.08900.27290.28870.29040.23080.28990.37840.51140.15080.15200.10330.08290.30310.2896
0.27600.37060.31410.17280.54430.21640.53220.23980.56730.20470.21610.16960.15790.14620.23390.1576

global shape information is somewhat captured by concatenating the local edge histograms. But the other descriptors, such as
CSD, SCD, and CLD mostly focus on the color component. And
HTD concatenates the energy of each frequency channel, which
does not pay much attention on the shape information of theimage. These are the main reasons why the MPEG-7 descriptors cannot well evaluate the perceptual quality of the retargeted
image. SIFT flow depicts the matching relations between theoriginal and retargeted images, which can help depicting thescene difference. VGG uses the convolutional neural network
to represent the whole image as one feature vector for classification. As such, VGG and SIFT flow can somewhat depict theimage global information, which present good correlations withsubjective rating values than MPEG-7 and EMD.
GIST significantly outperforms the other image representations, which may be attributed to two reasons. Firstly, GIST
tries to depict each image from different perspectives, suchas the horizontal and vertical information, fractal dimensionsof the scene, the convergence of parallel lines, oblique contours,
and so on. As such, GIST image feature is able to capture themost information which is sensitive to the retargeted image perceptual quality assessment. That is also the main reason why weemploy GIST as the image feature to develop our NR quality
metric for retargeted image. Secondly, the metric performs in
an FR manner. By computing the difference of GIST featuresfrom original and retargeted images, the quality can be moreaccurately captured.
Moreover, we further examine the generality of GIST overdifferent groups of images, according to the image attributes
as described in RetargetMe [19]. Besides BDS, EH, SIFT flow,
EMD, CLD, two recently developed objective quality metrics
CSim [21] and Q/Q [29] are included for performance comparison. The experimental results are listed in Table II. The best
objective metric for each group is highlighted in boldface. It canbe observed that GIST can yield the best performances for mostof the image groups. The reason is that GIST depicts the global
shape information, which can easily capture the distortions introduced to some image attributes, such as the “lines/edges”,“faces/people”, “geometric structures”, and so on. Q/Q is
designed by dealing with salient content preservation andmaintaining the symmetry features. Thus it provides the best

TABLE IIMEAN KENDALL CORRELATION COEFFICIENTS OF THE OBJECTIVE QUALITY
METRICS FOR DIFFERENT IMAGE TYPES

BDS
EH
SIFT
flow
EMD
CLD
CSim
Q /Q 
GIST

Lines/
Edges

Faces/
PeopleTextureForeground Geometric Symmetry
Objects
Structures

0.040
0.043
0.097

0.190
− 0.076
0.252

0.060
− 0.060
0.119

0.167
− 0.079
0.218− 0.004
0.103
0.085

− 0.012
0.298
0.071

0.220
− 0.023
0.005
0.159
0.229

0.262
− 0.181− 0.143
0.057
0.273

0.107
− 0.071− 0.125
0.054
0.218

0.226
− 0.183− 0.134
0.250
0.182

0.237
− 0.009
0.000
0.250
0.252

0.500
0.214
0.071
0.571
0.484

TABLE IIIPERFORMANCE COMPARISON OF DIFFERENT QUALITY
METRICS ON THE CUHK DATABASEGISTGLS (linear regression)
GLS (logistic regression with L1 penalty)
GLS (support vector regression with linear kernel)
GLS (support vector regression with polynomial kernel)
GLS (support vector regression with RBF kernel)
GLS (logistic regression)
Proposed NR metric

SROCC

PLCC

OR

0.5114
0.49390.40020.40380.38210.39610.47600.49260.54430.4402
0.39610.3656
0.37110.36580.46220.53710.15760.2046
0.21630.23390.20220.22670.13450.1928

performances on the image groups with “foreground objects”
and “symmetry”. Such experimental results also demonstrate
the generality of GIST, which is also the main reason that we
employ GIST as the image representation to design our NR IQA
for retargeted images.
2) Performances of Our Proposed NR Quality Metric: For
our proposed NR quality metric, as we need to obtain the parameters during the training process, we employ the standard
split for the evaluation. Specifically, the images in the subjective
database are randomly divided into training and testing sets. A
training set consists of 80% of the reference images and theirassociated distorted versions, and a testing set consists of theremaining 20% of the reference images and their associated distorted versions. In order to ensure that the proposed method isrobust across content and is not biased by the specific train-test
split, random 80% train 20% test split is repeated 1000 times.
Table III provides the performance comparisons between our
proposed NR quality metric with GIST and GLS [54]. GLS
extracts and fuses different features of the retargeted image
from the perspectives of the global structural distortion, local
region distortion, and loss of salient information. Several regression models are used to fuse these features together as shown in
Table III. Comparing the performances in Tables I and III, it canbe observed that the proposed NR quality metric outperforms
all the other FR metrics except GIST. As we employ GIST as
the image feature for quality assessment, the performance of our
proposed NR IQA is slightly inferior to that of GIST. For GLS,
although the image is depicted from global, local, and salient
perspectives, the experimental results demonstrated that GLS

MA et al.: NO-REFERENCE RETARGETED IMAGE QUALITY ASSESSMENT BASED ON PAIRWISE RANK LEARNINGTABLE IVPERFORMANCE COMPARISONS OF DIFFERENT QUALITY
METRICS ON THE RETARGETME DATABASE

Kendall τ
distance

does not have the original image for comparison, which issues
great challenges for perceptual quality assessment.

BSD

EHD

EMD

SIFT
flow

IR-SSIMGLSRBRIQA

Proposed
NR metric

0.083

0.004

0.145

0.251

0.363

0.382

0.503

0.4770TABLE VCROSS DATASET PERFORMANCES OF THE PROPOSEDPAIRWISE RANK LEARNING METRIC

Training on CUHK and Testing on RetargetMe
Training on RetargetMe and Testing on CUHK

2235

SROCC

LCC

0.31210.29730.32640.3107cannot well capture the meaningful features for the retargeted
IQA.
For RetargetMe database, the image perceptual quality is indicated by the favored times over the other images. Therefore,same as [19], the Kendall τ distance [55] is employed to evaluate the ground-truth and predicted rankings. The experimental
results are illustrated in Table IV. In addition, we also compare the performances with the newly developed quality metrics
IR-SSIM [4], GLS (logistic regression with L1 penalty), and
RB-RIQA [28]. It can be observed that our proposed metric outperforms other competitor models, except the RB-RIQA metric.
RB-RIQA and IR-SSIM try to capture the information loss andthe shape distortion in local regions, compared with other metrics, such as EMD, EHD, and so on. GLS considers both theglobal and local information, yielding better performance than
IR-SSIM. However, these quality metrics are all FR, which require the original images for quality analysis. On the contrary,
our proposed metric works in a NR fashion, where the original image does not need to be present for quality analysis. As
such, the proposed NR metric can be easily adopted for practical applications, such as monitoring the visual QoE during theretargeting process.
C. Cross Dataset Performance
In order to further demonstrate the effectiveness of our proposed NR metric, we perform the cross dataset performancesby training on one retargeted image dataset and testing on theother. The results are illustrated in Table V. It can be observedthat the performances are not as good as that tested solely oneach database. However, compared with the performances inTable I, it can be observed that the proposed NR IQA basedon pairwise rank learning outperforms some FR metrics developed by the image representations, such as SIFT-flow, PHOW,
EMD, and so on, but performs inferiorly to VGG and GIST.
The superior performance of the proposed metric to SIFT-flow,
EMD, and so on, is due to the effectiveness of GIST for capturing the shape distortions of retargeted image. The inferior
performance to VGG and GIST is because that the NR metric

D. Limitations
In this paper, we consider pariwise rank learning for NR IQAof retargeted images. The proposed method has several limitations. Firstly, the training phase needs to know all the distortions
to make a more robust training result, which can provide a morereliable result during testing phase. That is the main reason
that we cannot provide a good performance when training on
one retargeted image database and testing on the other. The retargeted images generated by the retargeted images have notbeen encountered during the training process. Therefore, thepairwise learning strategy cannot learn such behaviors over theother retargeted images. Secondly, for pairwise rank learning,
the pairwise data needs to be performed during the training andtesting phase. Suppose that there are m samples, the total num2
, which is a huge number,ber of the prepared pair data is Cm
especially when m is extreme large. How to handle such large
number of training samples and make a more reliable training
will be of great challenge. In the future, we will consider how to
select the most representative pairs from all the possible pairs,
which can not only speed up the training process but also makea reliable training.
VI. CONCLUSION AND FUTURE WORKIn this paper, we made the first attempt on the NR quality
assessment for retargeted images. The GIST feature extracted
for each image as well as its accompanied subjective quality
value are employed for the pairwise rank learning approach.
The experimental results demonstrate the effectiveness of our
proposed NR quality metrics for the retargeted image, which canachieve comparable performances with GIST and significantly
outperform other FR metrics.
GIST is employed as the image feature for the pairwise rank
learning of retargeted images. In the future, we will consider
new image features to more accurately represent the retargeted
image. Also we are considering the deep learning methods tolearn the features from image pixels for retargeted IQA. Withbetter image features, a better NR IQA for the retargeted image
is expected. Moreover, besides the linear relationship in the pairwise rank learning, in the future we will consider to employ amore complicated model, such as deep neural network, to modeland learn the nonlinear relationships between image representation and its subjective quality score. Also we will consider
how to handle the large scale pairwise training by selecting themost representative pairs from all the possible pairs, in order tonot only speed up the training process but also make a reliable
training.
APPENDIX
IMAGE REPRESENTATIONS FOR RETARGETED IMAGE
1) MPEG-7 [48] considers different descriptors from thecolor and texture perspectives, specifically the scalable
color descriptor (SCD), color layout descriptor (CLD),

2236

IEEE TRANSACTIONS ON MULTIMEDIA, VOL. 18, NO. 11, NOVEMBER 2016

color structure descriptor (CSD), homogeneous texture
descriptor (HTD), and edge histogram descriptor (EHD).
2) EMD [47] is based on the minimal cost that must be paid
to transform one distribution into the other. The originaland retargeted images are represented as histograms. The
EMD between these two histograms indicates the retargeted image quality.
3) PHOW [49] is obtained based on the SIFT descriptor andimage spatial layout. Multiple descriptors are computed
to allow for scale variations between images.
4) BDS [20], [22] captures how much information of theoriginal image is covered and preserved by the retargeted
image in a bidirectional manner and measures the newly
introduced artifacts.
5) SIFT flow [27] characterizes view-invariant andbrightness-independent image structures, which establishes the meaningful correspondences across the originaland retargeted images and evaluates the the spatial coherence of the pixel displacement (indicated by the SIFT
correspondence matching).
6) GIST [50] is extracted based on a very low dimensional
representation of the scene. A set of perceptual dimensions, such as naturalness, openness, roughness, expansion, ruggedness, is employed to represent the dominantspatial structure of a scene, which can accurately depict
the global information of the image.
7) VGG [51] employs a deep convolutional neural network
to represent each image as a fixed length image feature
vector, where convolution and max-pooling processes are
employed to compose and summarize the image pixels
values to semantic representation for further classification,
detection, localization, and so on.
REFERENCES[1] Y. Fang, Z. Chen, W. Lin, and C. Lin, “Saliency detection in the compressed domain for adaptive image retargeting,” IEEE Trans. Image
Process., vol. 21, no. 9, pp. 3888–3901, Sep. 2012.
[2] Q. Xu et al., “Hodgerank on random graphs for subjective video quality assessment,” IEEE Trans. Multimedia, vol. 14, no. 3, pp. 844–857,
Jun. 2012.
[3] B. Li et al., “Spatiotemporal grid flow for video retargeting,” IEEE Trans.
Image Process., vol. 23, no. 4, pp. 1615–1628, Apr. 2014.
[4] Y. Fang et al., “Objective quality assessment for image retargeting basedon structural similarity,” IEEE J. Emerg. Sel. Topics Circuits Syst., vol. 4,
no. 1, pp. 95–105, Mar. 2014.
[5] W. Lin and C. J. Kuo, “Perceptual visual quality metrics: A survey,”
J. Vis. Commun. Image Representation, vol. 22, no. 4, pp. 297–312, 2011.
[6] S. Avidan and A. Shamir, “Seam carving for content-aware image resizing,” in Proc. ACM SIGGRAPH 2007 Papers, 2007, Art. no. 10.
[7] W. Dong, N. Zhou, J.-C. Paul, and X. Zhang, “Optimized image resizing
using seam carving and scaling,” ACM Trans. Graph., vol. 28, no. 5,
pp. 125:1–125:10, Dec. 2009.
[8] M. Rubinstein, A. Shamir, and S. Avidan, “Multi-operator media retargeting,” ACM Trans. Graph., vol. 28, no. 3, pp. 23:1–23:11, Jul. 2009.
[9] A. Shamir and O. Sorkine, “Visual media retargeting,” in Proc. ACM
SIGGRAPH ASIA 2009 Courses, 2009, Art. no. 11.
[10] F. Banterle et al., “Multidimensional image retargeting,” in Proc.
SIGGRAPH Asia 2011 Courses, 2011, Art. no. 15.
[11] Y. Pritch, E. Kav-Venaki, and S. Peleg, “Shift-map image editing,” inProc. Int. Conf. Comput. Vis., 2009, pp. 151–158.
[12] Y.-S. Wang, C.-L. Tai, O. Sorkine, and T.-Y. Lee, “Optimized scale-andstretch for image resizing,” ACM Trans. Graph., vol. 27, no. 5, Dec. 2008,
Art. no. 118.

[13] L. Ma, S. Li, and K. N. Ngan, “Visual horizontal effect for image quality
assessment,” IEEE Signal Process. Lett., vol. 17, no. 7, pp. 627–630,
Jul. 2010.
[14] S. Li, F. Zhang, L. Ma, and K. N. Ngan, “Image quality assessment byseparately evaluating detail losses and additive impairments,” IEEE Trans.
Multimedia, vol. 13, no. 5, pp. 935–949, Oct. 2011.
[15] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality
assessment: From error visibility to structural similarity,” IEEE Trans.
Image Process., vol. 13, no. 4, pp. 600–612, Apr. 2004.
[16] L. Zhang, L. Zhang, X. Mou, and D. Zhang, “Fsim: A feature similarity
index for image quality assessment,” IEEE Trans. Image Process., vol. 20,
no. 8, pp. 2378–2386, Aug. 2011.
[17] K. Seshadrinathan and A. C. Bovik, “Motion tuned spatio-temporal quality
assessment of natural videos,” IEEE Trans. Image Process., vol. 19, no. 2,
pp. 335–350, Feb. 2010.
[18] L. Ma, W. Lin, C. Deng, and K. N. Ngan, “Image retargeting quality
assessment: A study of subjective scores and objective metrics,” IEEE J.
Sel. Topics Signal Process., vol. 6, no. 6, pp. 626–639, Oct. 2012.
[19] M. Rubinstein, D. Gutierrez, O. Sorkine, and A. Shamir, “A comparative
study of image retargeting,” ACM Trans. Graph., vol. 29, no. 6, Dec. 2010,
Art. no. 160.
[20] C. Barnes, E. Shechtman, A. Finkelstein, and D. B. Goldman, “Patchmatch: A randomized correspondence algorithm for structural image editing,” ACM Trans. Graph., vol. 28, no. 3, Jul. 2009, Art. no. 24.
[21] Y. Liu, X. Luo, Y. Xuan, W. Chen, and X. Fu, “Image retargeting quality assessment,” Comput. Graph. Forum, vol. 30, no. 2, pp. 583–592,
2011.
[22] D. Simakov, Y. Caspi, E. Shechtman, and M. Irani, “Summarizing visual
data using bidirectional similarity,” in Proc. IEEE Conf. Comput. Vis.
Pattern Recog., Jun. 2008, pp. 1–8.
[23] L. Ma, W. Lin, C. Deng, and K. N. Ngan, “Study of subjective andobjective quality assessment of retargeted images,” in Proc. IEEE Int.
Symp. Circuits Syst., May 2012, pp. 2677–2680.
[24] “BT.500.11: Methodology for the subjective assessment of the quality of
television pictures,” 2012. [Online]. Available: https://www.itu.int/rec/RREC-BT.500/en[25] S. Castillo, T. Judd, and D. Gutierrez, “Using eye-tracking to assess different image retargeting methods,” in Proc. ACM SIGGRAPH Symp. Appl.
Perception Graph. Vis., 2011, pp. 7–14.
[26] D. G. Lowe, “Object recognition from local scale-invariant features,” inProc. 7th IEEE Int. Conf. Comput. Vis., Sep. 1999, vol. 2, pp. 1150–1157.
[27] C. Liu, J. Yuen, A. Torralba, J. Sivic, and W. T. Freeman, “Sift flow: Dense
correspondence across different scenes,” in Proc.10th Eur. Conf. Comput.
Vis., 2008, pp. 28–42.
[28] Y. Zhang and K. N. Ngan, “Region-based image retargeting quality
assessment,” in Proc. IEEE Int. Conf. Image Process., Sep. 2015,
pp. 1757–1761.
[29] Y. Liang, Y.-J. Liu, and D. Gutierrez, “Objective quality prediction ofimage retargeting algorithms,” IEEE Trans. Vis. Comput. Graph., 2016,
to be published. [Online]. Available: doi:10.1109/TVCG.2016.2517641[30] H. R. Sheikh, A. C. Bovik, and L. K. Cormack, “No-reference quality
assessment using natural scene statistics: JPEG2000,” IEEE Trans. Image
Process., vol. 14, no. 11, pp. 1918–1927, Nov. 2005.
[31] T. Brandão and M. P. Queluz, “No-reference image quality assessment based on DCT domain statistics,” Signal Process., vol. 88, no. 4,
pp. 822–833, 2008.
[32] P. Ye and D. S. Doermann, “No-reference image quality assessment
using visual codebooks,” IEEE Trans. Image Process., vol. 21, no. 7,
pp. 3129–3138, Jul. 2012.
[33] W. Xue, L. Zhang, and X. Mou, “Learning without human scores for blind
image quality assessment,” in Proc. IEEE Conf. Comput. Vis. Pattern
Recog., Jun. 2013, pp. 995–1002.
[34] X. Wen, L. Shao, Y. Xue, and W. Fang, “A rapid learning algorithm forvehicle classification,” Inf. Sci., vol. 295, pp. 395–406, 2015. [Online].
Available: http://dx.doi.org/10.1016/j.ins.2014.10.040[35] A. K. Moorthy and A. C. Bovik, “Blind image quality assessment: From
natural scene statistics to perceptual quality,” IEEE Trans. Image Process.,
vol. 20, no. 12, pp. 3350–3364, Dec. 2011.
[36] B. Gu and V. S. Sheng, “A robust regularization path algorithm for ν-support vector classification,” IEEE Trans. Neural Netw.
Learn. Syst., 2016, to be published. [Online]. Available: 10.1109/
TNNLS.2016.2527796[37] B. Gu and V. S. Sheng, “Structural minimax probability machine,” IEEE
Trans. Neural Netw. Learn. Syst., 2016, to be published. [Online]. Available: 10.1109/TNNLS.2016.2544779

MA et al.: NO-REFERENCE RETARGETED IMAGE QUALITY ASSESSMENT BASED ON PAIRWISE RANK LEARNING[38] B. Gu, V. S. Sheng, K. Tay, W. Romano, and S. Li, “Incremental support vector learning for ordinal regression,” IEEE Trans. Neural Netw.
Learn. Syst., vol. 26, no. 7, pp. 1403–1416, Jul. 2015. [Online]. Available:
http://dx.doi.org/10.1109/TNNLS.2014.2342533[39] B. Gu, V. S. Sheng, Z. Wang, D. Ho, S. Osman, and S. Li, “Incremental learning for ν-support vector regression,” Neural Netw.,
vol. 67, pp. 140–150, 2015. [Online]. Available: http://dx.doi.org/
10.1016/j.neunet.2015.03.013[40] H. Tang, N. Joshi, and A. Kapoor, “Blind image quality assessment usingsemi-supervised rectifier networks,” in Proc. IEEE Conf. Comput. Vis.
Pattern Recog., Jun. 2014, pp. 2877–2884.
[41] L. Xu et al., “Multi-task rank learning for image quality assessment,”
IEEE Trans. Circuits Syst. Video Technol., 2016, to be published. [Online].
Available: 10.1109/TCSVT.2016.2543099[42] L. Xu et al., “Pairwise comparison and rank learning for image quality
assessment,” Displays, vol. 44, pp. 21–26, 2006.
[43] Methods for Subjective Determination of Transmission Quality, ITU-TP.800, 1996.
[44] K. Chen, C. Wu, Y. Chang, and C. Lei, “A crowdsourceable QoE evaluation
framework for multimedia content,” in Proc. 17th Int. Conf. Multimedia,
2009, pp. 491–500.
[45] T.-Y. Liu, “Learning to rank for information retrieval,” Found. Trends Inf.
Retrieval, vol. 3, no. 3, pp. 225–331, Mar. 2009.
[46] H. R. Sheikh, M. F. Sabir, and A. C. Bovik, “A statistical evaluation ofrecent full reference image quality assessment algorithms,” IEEE Trans.
Image Process., vol. 15, no. 11, pp. 3440–3451, Nov. 2006.
[47] O. Pele and M. Werman, “Fast and robust earth mover’s distances,” inProc. IEEE 12th Int. Conf. Comput. Vis., Sep.-Oct. 2009, pp. 460–467.
[48] B. S. Manjunath, J. Ohm, V. V. Vasudevan, and A. Yamada, “Color andtexture descriptors,” IEEE Trans. Circuits Syst. Video Technol., vol. 11,
no. 6, pp. 703–715, Jun. 2001.
[49] A. Bosch, A. Zisserman, and X. Muñoz, “Image classification using random forests and ferns,” in Proc. IEEE 11th Int. Conf. Comput. Vis.,
Oct. 2007, pp. 1–8.
[50] A. Oliva and A. Torralba, “Modeling the shape of the scene: A holistic
representation of the spatial envelope,” Int. J. Comput. Vis., vol. 42, no. 3,
pp. 145–175, May 2001.
[51] K. Simonyan and A. Zisserman, “Very deep convolutional networks
for large-scale image recognition,” CoRR, 2014. [Online]. Available:
http://arxiv.org/abs/1409.1556[52] L. Ma, L. Xu, H. Zeng, K. N. Ngan, and C. Deng, “How does the shapedescriptor measure the perceptual quality of the retargeting image?” in
Proc. IEEE Int. Conf. Multimedia Expo. Workshops, Jul. 2014, pp. 1–6.
[53] F. Xia, T. Liu, J. Wang, W. Zhang, and H. Li, “Listwise approach tolearning to rank: Theory and algorithm,” in Proc. Int. Conf. Mach. Learn.,
2008, pp. 1192–1199.
[54] J. Zhang and C.-C. J. Kuo, “An objective quality of experience (QoE)
assessment index for retargeted images,” in Proc. ACM Int. Conf. Multimedia, 2014, pp. 257–266.
[55] M. G. Kendall, “A new measure of rank correlation,” Biometrika, vol. 30,
pp. 81–93, 1938.

Lin Ma (M’13) received the B.E. and M.E. degrees
from Harbin Institute of Technology, Harbin, China,
in 2006 and 2008, respectively, both in computerscience, and the Ph.D. degree from the Department
of Electronic Engineering, the Chinese University ofHong Kong (CUHK), Hong Kong, in 2013.
He is currently a Senior Researcher with Tencent
AI Lab, Shenzhen, China. He was a Research Internwith Microsoft Research Asia, Beijing, China, from
October 2007 to March 2008. He was a Research Assistant with the Department of Electronic Engineering, CUHK, from November 2008 to July 2009. He was a Visiting Student with
the School of Computer Engineering, Nanyang Technological University, Singapore, from July 2011 to September 2011. He was a Researcher with Huawei
Noah’s Ark Laboratory, Hong Kong, China. His research interests include deeplearning and multimodal learning, specifically image and language, image/video
processing, and quality assessment.
Dr. Ma was the recipient of the Best Paper Award at the Pacific Rim Conference on Multimedia 2008. He was awarded the Microsoft Research Asia
fellowship in 2011. He was a finalist to HKIS Young Scientist Award in engineering science in 2012.

2237

Long Xu (M’12) received the M.S. degree in appliedmathematics from Xidian University, Xi’an, China,
in 2002, and the Ph.D. degree from the Institute ofComputing Technology, Chinese Academy of Sciences, Beijing, China.
He was a Postdoctoral Researcher with the Department of Computer Science, City University of Hong
Kong, Hong Kong, China, and the Department ofElectronic Engineering, Chinese University of Hong
Kong, Hong Kong, China, from July 2009 to December 2012. From January 2013 to March 2014, he wasa Postdoctoral Researcher with the School of Computer Engineering, Nanyang
Technological University, Singapore. He is currently with the Key Laboratoryof Solar Activity, National Astronomical Observatories, Chinese Academy of
Sciences. His research interests include image/video processing, solar radio astronomy, wavelet, machine learning, and computer vision.
Dr. Xu was selected in the 100-Talents Plan, Chinese Academy of Sciences,in 2014.
Yichi Zhang received the B.S. degree in information engineering from Shanghai Jiao Tong University,
Shanghai, China, in 2013, and is currently working
toward the Ph.D. degree at the Department of Electronic Engineering, The Chinese University of Hong
Kong, Hong Kong, China.
His research interests include image/video processing, image quality assessment, and visualsaliency.

Yihua Yan received the B.E. and M.E. degrees
from Northwestern Polytechnical University, Xi’an,
China, in 1982 and 1985, respectively, and the Ph.D.
degree from Dalian University of Technology, Dalian,
China, in 1990.
He is currently a Professor and Chief Scientist of
the Solar Radio Research and the Director of the Key
Laboratory of Solar Activity and the Solar Physics
Division, National Astronomical Observatories, Chinese Academy of Sciences, Beijing, China. He wasa Foreign Research Fellow with the National Astronomical Observatory of Japan, Tokyo, Japan, from 1995 to 1996 and an Alexander von Humboldt Fellow at Astronomical Institute, Wurzburg University, Germany, from 1996 to 1997. He is currently the President of the International
Astronomical Union Division E: Sun & Heliosphere for 2015–2018. His research interests include solar magnetic fields, solar radio astronomy, space solar
physics, and radio astronomical methods.

King Ngi Ngan (M’79–SM’91–F’00) received the
Ph.D. degree in electrical engineering from Loughborough University, Loughborough, U.K.
He was a Full Professor with Nanyang Technological University, Singapore, and the Universityof Western Australia, Perth, WA, Australia. He iscurrently a Chair Professor with the Department ofElectronic Engineering, Chinese University of Hong
Kong, Hong Kong, China. He holds honorary andvisiting professorships with numerous universities in
China, Australia, and South East Asia. He has published extensively, including three authored books, six edited volumes, and morethan 300 refereed technical papers and has edited nine special issues in journals.
He holds ten patents in image or video coding and communications.
Dr. Ngan is a Fellow of the IET, U.K., and IEAust, Australia. He was anIEEE Distinguished Lecturer from 2006 to 2007. He has served as an Associate
Editor of the IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEOTECHNOLOGY, the Journal on Visual Communications and Image Representation, the EURASIP Journal of Signal Processing: Image Communication, andthe Journal of Applied Signal Processing. He has chaired a number of prestigious
international conferences on video signal processing and communications, andhas served on the advisory and technical committees of numerous professionalorganizations. He co-chaired the IEEE International Conference on Image Processing, Hong Kong, China, in 2010.