FUTURE VIDEO TECHNOLOGY

Recent Advances and Challenges of Visual Signal
Quality Assessment
MA Lin1, 4, DENG Chenwei2*, King Ngi Ngan1, LIN Weisi3
1

Department of Electronic Engineering, Chinese University of Hong Kong, Hong Kong, China
School of Information and Electronics, Beijing Institute of Technology, Beijing 100081, China
3
School of Computer Engineering, Nanyang Technological University, Singapore 639798, Singapore
4
Lenovo Corporate Research Hong Kong Branch, Hong Kong, China
2

Abstract: While quality assessment is essential for testing, optimizing, benchmarking, monitoring, and inspecting related systems and services, it also plays an essential role in the design of virtually all visual signal processing
and communication algorithms, as well as various related decision-making processes. In thispaper, we first provide an overview of recently
derived quality assessment approaches for traditional visual signals (i.e., 2D images/videos),
with highlights for new trends (such as machine learning approaches). On the other hand,with the ongoing development of devices andmultimedia services, newly emerged visualsignals (e.g., mobile/3D videos) are becoming
more and more popular. This work focuses on
recent progresses of quality metrics, whichhave been reviewed for the newly emerged
forms of visual signals, which include scalable
and mobile videos, High Dynamic Range (HDR)
images, image segmentation results, 3D images/videos, and retargeted images.
Key words: objective quality assessment; 2D
images and videos; human perception; newly
emerged visual signals; Human Visual System

I. INTRODUCTION

Received: 2013-02-19
Revised: 2013-03-28
Editor: Ming-Ting SUN

62

The explosion in the number of computers anddigital systems connected by networks such as
the Internet has brought a flow of instant information into an increasingly large number ofhomes and businesses. Most of the information is in the form of digital visual signals (including image, video, etc.), as the most intuitive and faithful depiction of things in life andwork. As a result, products (e.g., phone cameras) and services (e.g., YouTube) based upon
visual signals have grown at an exponential rate.
Since human eyes are the ultimate receiversof the visual signals, how the quality of visualsignals is gauged can be employed to dictate
the formation of most of the corresponding
technologies and applications. Quality measurement can be used as the benchmark of nextgeneration visual signal acquisition, compression, processing, transmission, and so on, toshape the full spectrum of technology development and enable new applications.
Perceptual quality of visual signals can be
judged by human subjects in the most reliable
way [1-2]. However, subjective assessment
methods suffer from drawbacks as follows: 1) it
is time-consuming, laborious and expensive,and requires many human subjects and repeated viewing sessions; 2) it is not feasible
for on-line or real-time signal manipulations(such as encoding and transmission); 3) evenin the cases where human assessment is possible (such as manufacturing assembly lines)
and cost is not a problem, it depends upon theassessor’s physical conditions, emotion states,
personal experience, etc. Therefore, many researchers have carried out subjective tests to build
China Communications • May 2013

databases, which can help to efficiently evaluate the performances of the developed quality
metrics. During the database construction, manyobservers are recruited into the subjective assessment. The subjective rating scores after
processing are believed to be reliable to represent the true perceptual quality of the visualsignal. The built databases include the representative image databases - LIVE, CSIQ, IVC,
Toyama, TID, and A57, and the video databases - LIVE, EPFL, and VQEG FR-TV Phase
I. Detailed information about these databases
can be referred to Refs. [3-7].
Nowadays, automatic objective quality assessment has become an active research area,
which aims at building computational models
to predict perceptual signal quality. In otherwords, mathematical/engineering models areto be developed to take visual signals as inputs,
while the output is a number to denote theperceived quality. Numerous objective metrics
have been proposed for the prediction of multimedia quality. In Refs. [3-4], perceptual visual quality metrics are surveyed, which mostly
focus on the quality assessment of traditional
visual signals (i.e., 2D image/video). However,more and more visual signals are recently
presented to the consumers in different formats, such as 3D image/video, mobile video,High Dynamic Range (HDR) images, and soon. Therefore, a systematic and up-to-datereview of the visual signal quality assessment
will be of great interest to the research community. In this paper, we aim at a comprehensive overview of the state-of-the-art researchin the area of quality assessment of visualsignals, especially for the newly emerged visual signals.
The rest of the paper is organized as follows. In Section II, some representative quality metrics for traditional visual signals areclassified and reviewed, respectively, withhighlights for the new trends (like machinelearning approaches). In Section III, quality
assessment of newly emerged visual signals
will be reviewed in a more detailed way. Finally, Section IV will conclude this paper.
China Communications • May 2013

II. QUALITY ASSESSMENT FOR
TRADITIONAL VISUAL SIGNALS

A comprehensive overview of recently der-Quality assessment methods can be categorized into different classes based on different
criteria [3-6]. With regard to the availability ofreference information, visual quality metrics
for traditional visual signals can be generallyclassified into three board categories: Full-Reference (FR) metrics that need the complete reference signal, Reduced-Reference (RR) ones
that need partial information of the reference
signal, and Non-Reference (NR) ones that theprocessed signal is evaluated without any priorknowledge of the reference signal.
FR quality metric: Mean Square Error(MSE), Signal to Noise Ratio (SNR), andPeak SNR (PSNR) are the simplest and widelyadopted FR metrics, where the pixel-wise signal fidelity is measured. Although they areappealing for optimization, they are demonstrated to be inconsistent with Human Visual
System (HVS) perception, especially for the
non-additive distortions, such as JPEG, and
H.264/AVC compression. In order to handle
the drawbacks of MSE and PSNR, in the past
years, Perceptual Visual Quality Metrics (PVQMs) have been extensively investigated [8-21].
In Ref. [3], W. Lin and C.-C. Jay Kuo reviewed the progresses of the PVQMs during thepast decades. The quality metrics can be roughly divided into model-based PVQMS, such
as Refs. [11-12, 14-19] and signal-driven PVQMs, such as Refs. [8-10, 13]. After the review, two main trends of quality assessment
are proposed for FR visual signal quality assessment, which are impairment decoupling[11, 22-27] and machine learning approaches[28-34], respectively.
For impairment decoupling approach, instead of treating the distortions indiscriminately, the distortions introduced into the visual signal are separated into different components. And different distortions are believed tocorrelate with HVS perception in different
ways. In Refs. [11, 22-24], the authors separated the distortions into detail losses and additive impairments and treated them inde-

ived quality metrics arereviewed for traditional visual signals andnewly emerged visualsignals, which include
scalable and mobile
videos, High Dynamic
Range (HDR) images,
image

segmentation

results, 3D images/videos,andretargeted

images. The subjective
evaluation and objective quality metrics arereviewed. It also discusses how to develop
effective quality metrics for newly emerged
visual signals.

63pendently. In Ref. [25], the authors proposed
to separate distortions into linear frequency
distortions and additive noise degradations,which in essence can be interpreted as the twoterms, i.e., detail losses and additive impairments. However, the decoupling algorithm in
Ref. [25] was specifically designed for halftoning artifacts, while the works in Refs. [11,22-24] are designed for general distortions. In
Ref. [11], the HVS perception of detail losses
is evaluated by calculating the Minkowski
summations by referring to the restored andoriginal image. For the additive impairments,
the HVS responses can be further simulatedby the Minkowski summation with the normalization by the number of pixels. In thisway, the HVS responses to different impairments can be captured and finally combined
together to indicate the corresponding perceptual quality.
The other new trend of PVQM design is
machine learning approaches, the generalframework of which is illustrated in Figure 1.
The subjective databases provided the original
and tested visual signals, as well as the corresponding subjective scores. Therefore, basedon the provided information, the perceptual
quality analysis process can be treated as apattern recognition problem. Several usefulfeatures that are sensitive to visual signal perceptual quality are extracted from the original
and tested visual signal, respectively. Based on
the extracted features, machine learning algorithms will generate a trained model. Thereafter, the trained model will be employed to
predict the perceptual quality. The machine

Fig.1 General framework of machine learning approaches

64

learning approaches are believed to have theabilities to learn complex data patterns andovercome the difficulties for modeling complex HVS properties.
In Ref. [30], the features of perceptual
quality analysis are the singular vectors out ofSingular Value Decomposition (SVD) to quantify major structural information. And the Support Vector Regression (SVR) is employed as
the machine learning algorithm for automatic
prediction of image quality. The use of SVR
exploits the advantages of machine learningwith the ability to learn complex data patterns
for an effective and generalized mapping offeatures into a desired score, in contrast withthe oft-utilized feature pooling process in theexisting quality estimators. Also SVR canovercome the difficulty of model parameter
determination for such a system to emulate therelated, complex HVS modeling.
In Ref. [29], the image features are extracted based on Two-Dimensional (2D) mel-cepstrum, which is demonstrated that these features are effective since they can represent thestructural information. In Ref. [33], the machine learning algorithm is employed to developa Content-Dependent Multi-Metric Fusion(CD-MMF) for evaluating the image perceptual quality. A large number of image samples
have been collected, which are associated withsubjective scores and objective quality scoresfrom different metrics. CD-MMF is set to be
the nonlinear combination of multiple metrics
with suitable weights obtained by a machinelearning process. Moreover, the machinelearning approach is employed to determine
the context automatically for CD-MMF.
RR quality metric: RR quality metrics areroughly classified into three different but related categories. The first approach is based onimage distortion modeling. The quality metrics [35-36] are mostly developed for the videos degraded by specific distortions, such as
MPEG-2 compression. The second approachis developed on modeling HVS. The quality
metrics [37-39] are developed by considering
China Communications • May 2013

the HVS properties. For example, in Ref. [38],several HVS related features are extracted toindicate the spatial information losses, edge
information changes, contrast information, andcolor impairments. Therefore, an effective RR
quality metric named as VQM is developed,
which has been adopted as a North American
Standard by the American National Standards
Institute (ANSI). The third type of approaches
is based on modeling natural image/video statistics. The underline essential of these metrics[40-44] is that most real-world distortions will
disturb the image statistics. The variations of
the image statistics can be used to quantify thedegradation level of the image/video. For example, in Ref. [41], Generalized Gaussian
Density (GGD) is employed to depict the
wavelet coefficient distribution. In Refs. [42,44], GGD is employed to depict the coefficient distribution in Reorganized DCT (RDCT)
domain, which can ensure better performances.
As aforementioned, these three types of approaches are different but related. The developed RR metrics have either explicitly or implicitly considered these approaches together
but to a limited extent. However, in order todevelop a more effective RR quality metric, amore comprehensive consideration of theseapproaches is needed.
NR quality metric: NR quality metric design is an extremely difficult task [45]. Therefore, most of NR metrics focused on the image/video degraded by the specific distortions[46-49]. As JPEG 2000 employs the wavelet
transform to compress the image, the wavelet
statistical model is utilized to capture thecompression distortion [50]. Liang et al. [46]combined the sharpness, blurring, and ringing
measurements together to depict the perceptual quality of the JPEG 2000 coded image.
The distribution of the DCT coefficient after
quantization is modeled in Ref. [47] to predict
the PSNR value of the JPEG coded image.
Furthermore, Ferzli et al. [49] did the psychophysical experiment to test the blurring tolerance ability of the HVS, based on which theJust-Noticeable Blur (JNB) model is developed. These methods employ the behaviors ofChina Communications • May 2013

specific distortions to predict the degradationlevel. Therefore, if a new distortion is introduced, these methods can hardly evaluate theperceptual quality of the distorted image. In
order to handle the drawbacks, generic NR
metrics for images are researched [51-53],
where different NR quality metrics designed
for specific distortions are fused together
without considering the reference visual signal.
In Ref. [54], a comprehensive overview on the
NR quality metric has been provided, which
can be referred to for detailed information.
Tables I and II demonstrate the quality evaluation performances of the corresponding
quality metrics (in LIVE image/video database). Three statistical parameters, specificallyLinear Coefficient Correlation (LCC), Spearman-Rank Order Correlation Coefficient (SROCC), and Root Mean Square Error (RMSE)
are employed to evaluate the metric’s performance. It can be observed that PVQMs
outperforms the simple PSNR, and FR PVQMs
are generally more consistent with humanperception, compared with those of RR and
NR ones. FSIM [9] and MOVIE [19] perform
Table I Comparison of different image quality metrics
Type

LCC

SROCC

RMSE

PSNR

FR

0.872

0.876

13.37

SSIM [8]

FR

0.904

0.910

11.68

VIF [13]

FR

0.960

0.963

7.673

FSIM [9]

FR

0.960

0.963

7.678

IGM [12]

FR

0.958

0.958

7.924

Machine learning approach [30]

FR

0.924

0.925

10.40

Impairment decoupling approach [11]

FR

0.936

0.946

9.627

WNISM [41]

RR

0.738

0.779

18.43

RR metric [42]

RR

0.883

0.879

12.84

NR metric [51]

NR

0.823

0.800

-

BLIINDS [52]

NR

0.680

0.663

20.01

DIIVINE [53]

NR

0.917

0.916

10.90

Table II Comparison of different video quality metrics
Type

LCC

SROCC

RMSE

PSNR

FR

0.539 80.523 4

9.241

SSIM [8]FR

0.499 90.524 7

9.507

MOVIE [19]FR

0.811 6

0.789 0-

VQM [38]RR

0.716 00.702 97.664

65

the best for image and video quality assessment, respectively; but the machine learningand impairment decoupling approaches arequite promising and effective for evaluating
visual signal perceptual quality.

III. QUALITY ASSESSMENT FOR NEWLY
EMERGED VISUAL SIGNALS
Nowadays, newly emerged visual signals haveissued new challenges for perceptual quality
assessment, with the development of devices
and multimedia services. In this section, weare to review recent research works on quality
assessment of these visual signals, specificallyscalable and mobile videos, HDR images,
image segmentation results, 3D images/videos,
and retargeted images.

3.1 Scalable and mobile video
quality assessment
Nowadays, with development of devices, suchas smartphones and tablets, High Definition(HD) video sequences need to be displayed indifferent resolutions. Therefore, scalable videocoding is needed to meet the requirements.
Therefore, Video Coding Experts Group(VCEG) released a call for proposals on Scalable Video Coding (SVC) extensions of High
Efficient Video Coding (HEVC) [55]. Also theperceptual quality assessment of SVC is very
important, as the human eyes are the final receivers of the distributed video sequences.
In Ref. [56], the authors have performed alarge-scale study on the subjective quality of
the SVC for content distribution. The influence of the combination of scalable parameters in SVC was studied. The subjective results of SVC are analyzed with respect to fivedimensions, namely, video codec, content, spatial resolution, temporal resolution, and frame
quality. Based on the study, guidelines areprovided for an adaptation strategy of SVC
that can select the optimal scalability options
for resource-constrained networks. The adaptation strategy can be summarized as follows.
First, when the bit-rate is small and only layers having small spatial resolutions are avail-66

able, a larger spatial resolution is preferable inorder to obtain the lowest acceptable frame
quality without strong blurring. In the case of
H.264/SVC, this observation was valid evenwhen the frame rate decreases along with increase of the spatial resolution. Second, forlarge bit-rate conditions, acceptable frame
quality is achieved and thus a high frame rate,
which is obtained at the cost of the decreased
pixel bit-rate, becomes important for better
subjective quality. The built database has been
made publicly available to the research community working on objective quality design tomeasure the video sequences generated by
SVC [56]. The database was created by apair-wise comparison between two stimuli, i.e.,
SVC video sequences. The participated subject was asked to indicate which one had better quality. In total, sixteen subjects (11 men
and 5 women) participated in the subjective
evaluation procedure. Also in Ref. [57], video
perceptual quality was analyzed in the senseof considering multiple dimensions, specifically the encoder type, video content, bit rate,
frame size, and frame rate. It was reported thatthe video perceptual quality is affected by theencoder type, video content, bit rate, frame
rate, and frame size in a descending order ofsignificance. These findings are very useful forcross-dimensional video assessment and video
adaptation for video distribution.
Moreover, with development of the widely
spread and publicly adopted smartphones andtablets, mobile videos are more and more useful and important for mobile communications
and information sharing. Therefore, the perceptual qualities of mobile videos are very
important. In Ref. [58], the authors built a new
subjective mobile video database, where thehuman study was performed on mobile phones
and tablets in order to gauge the human perception of qualities on mobile devices. The
LIVE mobile database includes distortions thathave been studied such as compression andwireless packet-loss, which also incorporates
dynamically varying distortions that change asa function of time, such as frame-freezes andtemporally varying compression rates. The
China Communications • May 2013

subjective study portion of the database includes both the DMOS computed from theratings that the subjects provided at the end ofeach video clip, as well as the continuous
temporal scores that the subjects recorded asthey viewed the video. The study involved
over 50 subjects and resulted in 5 300 summary subjective scores and time-sampled subjective traces of quality. The human opinion isanalyzed using statistical techniques, and alsostudied by a variety of models of temporal
pooling that may reflect strategies that the
subjects used to make the final decision onvideo quality. Also quality ratings obtained
from the tablet and the mobile phone studies
were to study the impact of these different
display modes on quality.
Several objective image and video quality
assessment algorithms with regards to theirefficacy in predicting visual quality are evaluated over the database, which is illustrated inTable III to provide the benchmark results.
And the LIVE Mobile VQA database, alongwith the subject DMOS and the continuous
temporal scores is being made available toresearchers in the field of VQA at no cost inorder to further research in the area of video
quality assessment. It can be observed thatmost of the work of scalable and mobile video
quality assessment focuses on subjective testing method to build subjective database. The
traditional quality metrics, such as PSNR and
SSIM are employed to evaluate the perceptual
quality. However, they are demonstrated to beinefficient. Therefore, in order to develop an
effective quality metric for scalable and mobile videos, the video content, bit rate, frame
rate, frame rate, etc. should be considered.

3.2 HDR images quality assessment
HDR images are becoming more widely available, due to recent advances in imaging andcomputer graphics technologies. Tone Mapping Operators (TMOs) are employed to visualize HDR images on standard display devices.
How to convert HDR images by tone mapping
operators and how to evaluate the tone mapped
images are now researched in Refs. [59-61].
China Communications • May 2013

Table III Quality metric comparisons on the mobile database
TabletMobile

SROCC

LCC

RMSE

SROCC

LCC

RMSE

PSNR

0.588 6

0.634 80.663 00.678 00.690 90.667 0SSIM [8]0.430 00.489 30.748 30.649 80.663 70.690 1

VIF [13]0.726 1

0.763 50.554 10.743 90.787 00.569 2

VQM [38]

0.555 20.615 00.698 00.694 50.702 30.666 3

MOVIE [19]0.679 2

0.782 80.534 20.642 00.715 70.644 4Due to the reduction in dynamic range, TMOs
cannot preserve all information in HDR images, and human observers of the LDR versions of these images may not be aware of this.
It is demonstrated that structural fidelity plays
an important role in Refs. [59-60]. However,structural fidelity alone does not suffice toprovide an overall quality evaluation. A goodquality tone mapped image should achieve agood tradeoff between structural fidelity preservation and statistical naturalness. The overall structural fidelity is defined according to:L

S = ∏Slβl

(1)

l =1

where L is the total number of scales and βlis the weight assigned to the l-th scale. And
the single score Sl is calculated by:
N

Sl =

1 l
∑Slocal ( xi , yi )Nl i =1(2)

where xi and yi are the i-th patches in the HDR
and LDR images being compared, respectively,
and Nl is the number of patches in the l-th
scale. The statistical naturalness model is built
upon statistics conducted on about 3 000
8-bits/pixel gray-scale images, which can represent many different types of natural scenes.
The means and standard deviations of theseimages can reflect the global intensity andcontrast of images. And the histograms ofmean and standard deviation can be well fittedusing a Gaussian Pm and a Beta probability
density function Pd. Recent studies suggested
that brightness and contrast are largely independent quantities in terms of natural image
statistics and biological computation. As aresult, their joint probability density function
would be the product of two. The statistical67

naturalness measure can be defined as:
N=

1
Pm Pd
k

(3)

where k is the normalization factor given byk = max{Pm, Pd}. This constrains the statisticalnaturalness measure to be bounded between 0
and 1.Finally, the quality model can be determined by:
Q = α S α + (1 − α ) N β(4)

In Ref. [62], an overview about the effect of
the basic image attributes in the HDR tone
mapping is presented. Results of subjective
psychophysical experiments that the authors
performed are publicly available to researchers. Moreover, the evaluations of existing
TMOs are provided based on the HDR database. With validation on the subject-rated image database [62], the metric is demonstrated
to be effective as shown in Table IV. The
structural fidelity and image naturalness areboth correlated to HVS perception of HDR
images. Furthermore, the quality metric can befurther applied to tone mapping operators byparameter tuning and adaptive fusion of multiple tone mapped images.
In Ref. [61], a novel image quality metric is
proposed to be capable of operating on an image pair with arbitrary dynamic ranges, which
defines a new HVS model based on the detection and classification of visible changes in theimage structure. The metric can be considered
as a hybrid of contrast detection and structuralsimilarity metrics. However, instead of detecting contrast changes, the proposed metricis sensitive to three types of structural changes.
1) Loss of visible contrast happens when acontrast that was visible in the reference image becomes invisible in the test image. Thistypically happens when a TMO compresses
details to the level that they become invisible.
2) Amplification of invisible contrast happens when a contrast that was invisible in the
Table IV Performance of HDR quality metric [59-60]
Structural fidelity

68

Naturalness Overall

KROCC

0.692 30.384 60.717 9SROCC

0.791 2

0.538 50.818 7

reference image becomes visible in the test
image. For instance, this can happen when
contouring artifacts starts to appear due tocontrast stretching in the inverse TMO application. 3) Reversal of visible contrast happens
when a contrast is visible in both reference
and test images, but has different polarity. Thiscan be observed at image locations with strong
distortions, such as clipping or salient compression artifacts.
It should be noted that the TMOs should try
their best to preserve the structural information in order to visualize HDR images on
standard display devices. Therefore, the quality metrics [59-61] try to evaluate the structural distortion between high dynamic and low
dynamic images. Moreover, after TMO processing, naturalness of the images is also considered in Refs. [59-60], which further affects
perceptual quality.

3.3 Image segmentation quality
assessment
Image segmentation is a fundamental problem
in computer vision. Over the past decades, alarge number of segmentation algorithms havebeen proposed, which makes the evaluation ofperceptual correctness on the segmentation
output becomes a demanding task. The existing objective evaluation methods for segmentation can be classified as ground-truth based
ones and none-ground-truth based ones.
In non-ground-truth based methods, theempirical goodness measures are proposed tomeet the heuristic criteria in the desirable
segmentations. The quality score is calculated
based on these criteria to predict the segmentation quality. There have been examples of
empirical measures on the uniformity of colors[63-64] or luminance [65] and even the shapeof object regions [66]. Since the criteria aremainly summarized from the common characteristics or semantic information of the objects(e.g., homogeneous regions, smooth boundaries, etc.), they are not accurate enough to describe the complex objects in the images.
The ground-truth based methods measure
the difference between the segmentation result
China Communications • May 2013

and the human-labeled ground truths. They are
more intuitive than the empirical based measures, since the ground truths can well represent the human-level interpretation of an image. Some measures in this category aim tocount the degree of overlapping between regions with strategies of being intolerant [67]or tolerant [68] to region refinement. In contrast to working on regions, there are alsomeasures [69-70] matching the boundaries
between segmentations. Considering only theregion boundaries, these measures are moresensitive to the dissimilarity between the segmentation and the ground truths than the region
based measures. Some other measures use nonparametric tests to count the pairs of pixels
that belong to the same region in different segmentations. The well-known Rand index [71]and its variants [72-73] are of this kind.
In Ref. [74], the authors provided a framework to quantitatively evaluate the quality of agiven segmentation with multiple ground truth
segmentations. It is assumed that if a segmentation is “good”, it can be constructed bypieces of the ground truth segmentations. And
a new ground truth is adaptively constructed,
which can be locally matched to the segmentation as much as possible and preserve thestructural consistency. The quality of the segmentation can then be evaluated by measuring
its distance to the adaptively composite groundtruth. There is a public segmentation databasenamed as the Berkeley Segmentation Dataset
and Benchmark. The goal of this database is toprovide an empirical basis for research on
image segmentation and boundary detection.
To this end, 12 000 hand-labeled segmentations of 1 000 Corel dataset images from 30
human subjects are collected. Half of the segmentations were obtained from presenting the
subject with a color image; the other half frompresenting a grayscale image. The public benchmark based on this data consists of all of thegrayscale and color segmentations for 300 images. The images are divided into a training
set of 200 images, and a test set of 100 images.
With evaluation on the benchmark Berkeley
database, the method can faithfully reflect the
China Communications • May 2013

perceptual quality of the segmentation. During
the evaluation process, the commonly used
F-test [75] and the probabilistic rand index [76]are computed on a large amount of segmentation results.
Table V shows the comparison results ofthe three measures. Each of them gives 500
results on the pair of segmentations, where ourmeasure has 379 results which are consistent
to the human judgment, while the F-measure
and the probabilistic rand index have only 333
and 266, respectively. The number of results
which are only correctly produced by one
measure is named as the “winning” case. The
proposed method [74] obtains 114 winningcases, while F-measure and probabilistic rand
index only obtain 4 and 19, respectively. And
there are 16 results which are wrongly classified by all of the three measures. The proposed measure outperforms the other two inboth of the “consistent” and the “winning” cases.
However, so far there is no standard procedure
for segmentation quality evaluation due to theill-defined nature of image segmentation, i.e.,there might be multiple acceptable segmentations which are consistent to the human interpretation of an image. In addition, there exists
a large diversity in the perceptually meaningful segmentations for different images. The above factors make the evaluation very complex.

3.4 3D images and videos
Stereoscopic/3D Image and Video Quality Assessment (IQA/VQA) has become increasing
relevant in today’s world, owing to the amountof attention that has recently been focused on
3D/stereoscopic cinema, television, gaming,and mobile video. With the development of thedevices, more and more 3D visual signals areprovided for the consumers, which can generatebetter Quality of Experience (QoE). However,Table V The number of results which are consistent to the human judgment, as
well as the numbers of “winning” cases and failure cases by competing methods
Consistent

Winning

F-test [75]3334Index in Ref. [76]

266

19

Method in Ref. [74]

379

114

Failure16

69understanding the quality of experience ofhuman viewers as they watch 3D videos is acomplex and multi-disciplinary problem. Therefore, many researchers have devoted the efforts for creating benchmark dataset and quality

metrics of 3D visual signals [77-80].
In Ref. [77], LIVE 3D IQA database incorporates “true” depth information along with stereoscopic pairs and human opinion scores.
The LIVE 3D IQA database consists of 20

Table VI Performance of 2D IQA algorithms in predicting perceived 3D image quality

PSNR

SSIM

VIF

BIQI[81]

JP2K

JPEG

White Gaussian noise

Blur

Fast-fading

All

SROCC

0.796 70.131 10.931 80.901 60.595 70.837 0LCC

0.788 90.231 10.934 70.893 70.706 20.825 1RMSE

7.958 76.362 45.914 5

6.527 1

8.797 1

9.267 8SROCC

0.857 20.434 6

0.939 50.882 20.584 90.877 2LCC

0.865 00.484 90.937 40.919 70.721 20.872 7RMSE

6.498 45.719 15.794 75.681 48.606 98.005 9SROCC

0.901 80.582 80.932 50.931 20.803 70.920 4LCC

0.936 10.673 80.927 30.957 00.854 20.918 3RMSE

4.557 04.831 96.229 1

4.198 6

6.461 56.490 3

SROCC

0.772 70.488 70.927 70.859 60.706 70.865 2LCC

0.820 30.613 6

0.932 30.899 50.776 20.879 2RMSE

7.410 05.163 66.016 66.323 87.867 97.811 9

Table VII Performance of 3D IQA algorithms in predicting perceived 3D image quality

Benoit [82]Hewage[83]You [84]

Gorley [85]Shen [86]Yang [87]Zhu [88]

Akhter [89]

70

JP2K

JPEG

White Gaussian noise

Blur

Fast-fading

All

SROCC

0.910 30.602 80.929 20.930 80.698 90.899 2LCC

0.939 80.640 50.925 30.948 80.747 20.902 5RMSE

4.426 65.022 06.307 6

4.571 48.257 87.061 7

SROCC

0.855 80.500 10.896 30.690 00.544 70.814 0LCC

0.904 30.530 50.895 50.798 40.669 80.830 3RMSE

5.530 05.543 17.405 68.748 09.226 3

9.139 3

SROCC

0.859 80.438 80.939 50.882 2

0.588 30.878 9LCC

0.877 80.487 4

0.941 20.919 80.730 00.881 4RMSE

6.206 65.709 75.621 65.679 88.492 37.746 3

SROCC

0.420 30.015 20.740 80.749 80.366 30.141 9LCC

0.485 30.312 4

0.796 10.852 70.364 80.451 1

RMSE

11.323 7

6.211 910.197 9

7.562 2

11.569 1

14.635 0

SROCC0.213 30.244 00.891 70.658 6

0.266 50.067 9LCC

0.503 90.389 90.898 80.684 6

0.483 00.574 3

RMSE

12.275 4

6.021 67.293 910.554 7

10.882 0

13.547 3

SROCC

0.150 10.132 80.847 1

0.326 60.142 60.078 5LCC

0.201 20.273 80.870 10.626 10.282 40.390 9RMSE

12.697 96.289 48.200 2

12.129 1

11.946 2

15.248 1

SROCC0.770 80.292 90.465 1

0.793 50.475 2

0.638 8LCC

0.807 30.379 00.517 80.777 00.503 80.626 3RMSE

7.681 3

6.068 414.720 1

9.127 010.736 2

12.782 8

SROCC

0.865 70.675 40.913 70.554 90.639 30.382 7LCC

0.905 90.729 40.904 70.617 70.660 30.427 0RMSE

5.483 64.473 6

7.092 911.387 2

9.332 114.827 4

China Communications • May 2013

reference images, 5 distortion categories and a
total of 365 distorted images along with theassociated DMOS. The authors described thecreation of the database and analyzed the performance of a variety of 2D and 3D quality
models using the new database. The database
as well as the algorithms evaluated is available
for researchers in the field to use in order toenable objective comparisons of future algorithms. Also the authors broadly summarized
the field of 3D QA focusing on key unresolved
problems including stereoscopic distortions,
3D masking, and algorithm development. The
traditional 2D image quality metrics are employed to evaluate the 3D stereoscopic image
qualities, which is illustrated in Table VI. Also
several 3D image quality metrics are employed to provide the benchmark performances shown in Table VII, such as: Benoit [82],Hewage [83], You [84], Gorley [85], Shen[86], Yang [87], Zhu [88], and Akhter [89].
Based on the results in Table VI, it is clearthat for the set of distortions considered, the2D IQA algorithms perform well in terms ofcorrelation with human subjectivity, while theaddition of disparity/depth in the 3D algorithms cannot materially improve the performance. However, disparity activity (e.g., caused by rapid changes in depth) may affect distortion visibility, which can be interpreted
from the human experiences. Also it can beobserved from Table VII that the 3D metric
Benoit [82] can generate the best performanceover the built database. However, the performance is still not good enough. The reason
is that almost of the 3D QA algorithms are
simple extensions of 2D QA algorithms withsome additional “features” extracted from depth(generally disparity differences). The way inwhich this disparity information is incorporated
into these 3D QA algorithms is not based on
any perceptual principles.
As we all know, the depth information isvery important for the 3D video application,such as displaying, rendering, and so on. In
Ref. [78], the authors created both a subjective
experiment and a new objective quality metricto evaluate the depth perception of 3D stereoChina Communications • May 2013

scopic videos. Moreover, the authors in Ref.
[79] investigated the Depth Image-Based Rendering (DIBR) based synthesized view evaluation
problem. Different view synthesis algorithms
are evaluated through subjective and objective
measurements. The hints for developing new
objective quality metric for 3D synthesizedview are discussed. The authors in Ref. [80]derived the RR quality metric for 3-D videos.
The information of depth edges and color images in the areas in the proximity of edges are
extracted for the RR quality metric, which canbe utilized for 3-D video compression andtransmission.
Overall, the field of 3D QA remains an extremely interesting one. There is tremendous
scope for research in this area. The large gaps
in our understanding of the perception ofstereoscopic distortions and of appropriate
statistical models for 3D natural scenes arestill need to be researched. A multi-pronged
approach combining research and concepts
from the visual sciences and image processing
will hopefully lead to models that predict stereoscopic quality with high accuracy.

3.5 Retargeted images
The image retargeting methods are proposed
to adjust the source images into arbitrary sizesfor displaying. Therefore, there issues a newchallenge of objectively evaluating the retargeted image perceptual quality, where the resolution has been changed, the objective shape
may be distorted, and some content information may be discarded. Till now, there are twomain public images retargeting quality databases [90-92]. In Ref. [90], the database isbuilt by concentrating on a comparative studyof existing retargeting methods. The authors
compared which retargeting method generates
the retargeted image with the highest perceptual quality. The subjective test is performed
in a pair comparison way, where the participants are shown two retargeted images at atime, side by side, and are asked to simply
choose the one they like better. The resulting
database comprises the retargeted image andthe corresponding number of times that the71retargeted image is favored over another one.
In Refs. [91-92], a subjective study is conducted to assess the perceptual quality of theretargeted image to build a publicly available
database. Totally 171 retargeted images (intwo different scales) are generated by different
retargeting methods from 57 source images.
With the source image as the reference, theperceptual quality of each retargeted image
has been subjectively rated by at least 30 human viewers on a pre-defined scale. After processing the subjective ratings, the MOS value
and the corresponding standard deviation are
obtained for each image. Based on the MOS
values, the built image retargeting database
was analyzed from the perspectives of theretargeting scale, the retargeting method, andthe source image content. Moreover, some publicly available quality metrics for retargeted
images are evaluated on the built database.
There are several existing quality metrics
on retargeted images, such as Edge Histogram(EH) [93], SIFT-flow [94], Bidirectional Similarity (BDS) [95-96], and Earth’s Mover Distance (EMD) [97-98]. EH captures the spatial
distribution of edges in the image. In order todepict the local edge distribution, the image isdivided into 4×4 sub-images, each of which is
examined by 5 different orientations: vertical,
horizontal, two diagonals, and isotropic (nondirectional). For each sub-image, a normalized
5-bin histogram is obtained by classifying
apparent edges to these five categories. The
feature is defined to be the combination ofthese histograms, which results in 4×4×5 = 80
length description. Only the intensity component is employed for edge detection. And the-norm distance is employed to measure thefeature distance between two images. SIFTflow descriptors characterize view-invariant andbrightness-independent image structures. Matching SIFT descriptors allows establishing
meaningful correspondences across image withsignificantly different image content. Furthermore, the pixel displacement (indicating by
the SIFT correspondence matching) should bespatial coherent, which means that close-by
pixels should have similar displacement. BDS

72

tries to capture that two signals (original andretargeted images) are considered to be “visually similar” is as many as possible patches of
one image (at multiple scales) are contained inanother image, and vice versa. EMD is basedon the minimal cost that must be paid to transform one distribution into the other. The signature or histogram, which represents a set offeature clusters, is viewed as the histogram
distribution. The point is the central value inbin of the histogram, and is to indicate the corresponding proportion. The definition of cluster is open. The color, position, and texture information can be employed to obtain the feature clusters. Only the size of the clusters in thefeature space needs to be limited. EMD is defined as the minimum work normalized by the
total flow to convert one histogram to another.
The performances evaluated on the database are shown in Table VIII. It can be observed that all of the existing metrics perform
poorly on the database. For the EMD, thecomposed histogram only represents the feature distribution of the image, which cannot
accurately depict the object shape and thecontent information of the image. Therefore,
the shape distortions and content information
loss, introduced during the retargeting process,are not effectively described. BDS tries tocapture how much information one image
conveys of the other image in a bidirectional
way. However, although it is claimed that the
spatial geometric relationship is considered bya multiple scale approach, the order-relationship can still not be preserved, such as the
local-order of each pixel or patch. Therefore,
the dissimilarity metric of BDS does not accurately depict the object shape distortion either. SIFT-flow employs the SIFT descriptor
to detect the correspondence between two images. It is claimed that the order-relationship
Table VIII Metric performance on image retargeting database
EH

EMD

BSE

SIFT-flow

LCC

0.342 20.276 0 0.289 6

0.314 1

SROCC

0.328 80.290 4 0.288 7

0.289 9RMSE

12.68612.97712.81712.922China Communications • May 2013

of the pixels or patches is captured. However,the content information loss during the retargeting process is not considered. EH employs
the edge histograms to describe the image, which
are organized in order for comparison. EH cansomehow represent the object shape information in the image. Same as the SIFT- flow, thecontent information loss is not accounted.
What have been described above are thereasons why the metrics available cannot perform effectively for retargeted images. It canbe observed that most of the quality metrics
try to extract several features from the originaland retargeted images, and compare the differences for quality assessment. However, thesefeatures cannot accurately capture the shape
distortions and content information loss, whichhave been introduced into the retargeted images. Therefore, in order to develop a more
effective quality metric for retargeted images,
the shape distortions and content information
loss should be first accurately captured and thenfused together, as a suggested direction of future research.

this paper was partially supported by the Research Grants Council of the Hong Kong SAR,
China (Project CUHK 415712); and the Ministry of Education Academic Research Fund
(AcRF) Tier 2 in Singapore under Grant No.
T208B1218.

References
[1]
[2]

[3]

[4]

[5]

IV. CONCLUSION
In this paper, an overview and comparison of
objective quality metrics for visual signals wasprovided, for both traditional and newly emerged visual signals, as well as giving comments
and discussion whenever appropriate. Specifically, we highlighted the new trends of traditional visual signal quality assessment; thoserelated to newly emerged visual signals havebeen discussed in a more detailed way, andsuch new forms of visual signals being discussed include scalable and mobile video, HDR
image, image segmentation, 3D images/video,
and retargeted images. We have also shared ourviews in some future development in the areas.

[6]

[7]

[8]

[9]

[10]

ACKNOWLEDGEMENT
The authors would like to thank the reviewers
for their detailed reviews and constructive
comments, which have helped to improve thequality of this paper. The work described in
China Communications • May 2013

[11]

ITU-T-P.800. Methods for Subjective Determination of Transmission Quality[S], 1996.
ITU-R BT.500-11. Methodology for the Subjective Assessment of the Quality of Television
Pictures[S], 2000.
LIN Weisi, KUO J C. Perceptual Visual Quality
Metrics: A Survey[J]. Journal of Visual Communication and Image Representation, 2011,22(4): 297-312.
LIU T J, LIN Weisi, KUO J C.-C. Recent Developments and Future Trends in Visual Quality
Assessment[C]// Proceedings of Asia-PacificSignal and Information Processing Association Annual Submit and Conference, 2011(APSIPA ASC 2011): October 18-21, 2011.
Xi’an, China, 2011.
ENGELKE U, ZEPERNICK H. Perceptual-Based
Quality Metrics for Image and Video Services: A
Survey[C]// Proceedings of the 3rd EuroNGI
Conference on Next Generation Internet Networks: May 21-23, 2007. Trondheim, Norway,
2007: 190-197.
WINKLER S, MOHANDAS P. The Evolution ofVideo Quality Measurement: from PSNR toHybrid Metrics[J]. IEEE Transactions on Broadcasting, 2008, 54(3): 660-668.
WINKLER S. Analysis of Public Image andVideo Databases for Quality Assessment[J].
IEEE Journal of Selected Topics in Signal Processing, 2012, 6(6): 616-625.
WANG Zhou, BOVIK A C, SHERKH H R, et al.
Image Quality Assessment: from Error Visibility to Structural Similarity[J]. IEEE Transactions
on Image Processing, 2004, 13(4): 600-612.
MA Lin, LI Songnan, NGAN K N. Visual Horizontal Effect for Image Quality Assessment[J].
IEEE Signal Processing Letters, 2010, 17(7): 627630.
LIU Anming, LIN Weisi, NARWARIA M. Image
Quality Assessment Based on Gradient Similarity[J]. IEEE Transactions on Image Processing, 2012, 21(4): 1500-1512.
LI Songnan, ZHANG Fan, MA Lin, et al. Image
Quality Assessment by Separately Evaluating
Detail Losses and Additive Impairments[J].
IEEE Transactions on Multimedia, 2011, 13(5):935-949.

73

[12]

[13]

[14]

[15]

[16]

[17]

[18]

[19]

[20]

[21]

[22]

[23]

[24]

WU Jinjian, LIN Weisi, SHI Guangming, et al. A
Perceptual Quality Metric with Internal Generative Mechanism[J]. IEEE Transactions on
Image Processing, 2013, 22(1): 43-54.
SHEIKH H R, BOVIK A C. Image Information
and Visual Quality[J]. IEEE Transactions on
Image Processing, 2006, 15(2): 430-444.
DALY S. The Visible Difference Predictor: An
Algorithm for the Assessment of Image Fidelity[M]. Cambridge, CA, USA: Digital Images
and Human Vision, MIT Press, 1993.
WINKLER S. Digital Video Quality: Vision
Models and Metrics, John Wiley and Sons[M].
Portland: Book News Inc., 2005.
ITU-T Recommendation J.247. Objective Perceptual Multimedia Video Quality Measurement in the Presence of a Full Reference[S],2008.
YANG X K, LING W S, LU Z K, et al. Just Noticeable Distortion Model and Its Applications
in Video Coding[J]. Signal Processing: Image
Communication, 2005, 20(7): 662-680.
MA Lin, NGAN K N, ZHANG Fan, et al. Adaptive Block-Size Transform Based Just-Noticeable Difference Model for Images/Videos[J].
Signal Processing: Image Communication, 2011,26(3): 162-174.
SESHADRINATHAN K, BOVIK A C. Motion
Tuned Spatio-Temporal Quality Assessment of
Natural Videos[J]. IEEE Transactions on Image
Processing, 2010, 19(2): 335-350.
ZHANG Fan, MA Lin, LI Songnan, et al. Practical Image Quality Metric Applied to Image
Coding[J]. IEEE Transactions on Multimedia,
2011, 13(4): 615-624.
NARWARIA M, LIN Weisi. SVD-Based Quality
Metric for Image and Video Using Machine
Learning[J]. IEEE Transactions on Systems, Man,
and Cybernetics, Part B: Cybernetics, 2012,42(2): 347-364.
LI Songnan, MA Lin, NGAN K N. Full-Reference Video Quality Assessment by Decoupling Detail Losses and Additive Impairments[J].
IEEE Transactions on Circuit System for Video
Technology, 2012, 22(7): 1100-1112.
LI Songnan, MA Lin, NGAN K N. Video Quality
Assessment by Decoupling Additive Impairments and Detail Losses[C]// Proceedings ofthe 3rd International Workshop on Quality ofMultimedia Experience (QoMEX 2011): September 7-9, 2011. Mechelen, Belgium, 2011:90-95.
LI Songnan, NGAN K N. Subtractive Impairment, Additive Impairment, and Image Visual
Quality[C]// Proceedings of 2010 IEEE International Symposium on Circuits and Systems

74

[25]

[26]

[27]

[28]

(ISCAS 2010): May 30-June 2, 2010. Paris,
France, 2010: 3373- 3376.
DAMERA-VENKATA N, KITE T D, GEISLER W S,
et al. Image Quality Assessment Based on a
Degradation Model[J]. IEEE Transactions on
Image Processing, 2000, 9(4): 636-650.
LARSON E C, CHANDLER D M. Most Apparent
Distortion: Full-Reference Image Quality Assessment and the Role of Strategy[J]. Electronic Imaging, 2010, 19(1): 1-21.
ROUSE D M, PEPION R, HEMAMI S S, et al.
Image Utility Assessment and a Relationshipwith Image Quality Assessment[J]. Human Vision Electronic Imaging, 2009, 7240:
724010-1-724010-14.
NARWARIA M, LIN Weisi. SVD-Based Quality
Metric for Image and Video Using Machine
Learning[J]. IEEE Transactions on Systems,

[29]

[30]

[31]

[32]

[33]

[34]

[35]

Man, and Cybernetics, Part B, 2012, 42(2):347-364.
NARWARIA M, LIN Weisi, CETIN A E. Scalable
Image Quality Assessment with 2D Mel-Cepstrum and Machine Learning Approach[J].
Pattern Recognition, 2011, 45(1): 299-313.
NARWARIA M, LIN Weisi. Objective Image
Quality Assessment Based on Support Vector
Regression[J]. IEEE Transactions on Neural
Networks, 2010, 21(3): 515-519.
LUO Huitao. A Training-Based No-Reference
Image Quality Assessment Algorithm[C]//
Proceedings of 2004 International Conference
on Image Proceeding (ICIP 2004): October
24-27, 2004. Singapore, 2004: 2973-2976.
SURESH S, BABU V, SUNDARARAJAN N. Image Quality Measurement Using Sparse Extreme Learning Machine Classifier[C]// Proceedings of the 9th International Conference
on Control, Automation, Robotics and Vision,
2006 (ICARCV 2006): December 5-8, 2006.
Singapore, 2006: 1-6.
LIU T J, LIN Weisi, KUO C.-C J. A Multi-Metric
Fusion Approach to Visual Quality Assessment[C]// Proceedings of 2011 the 3rd International Workshop on Quality of Multimedia
Experience (QoMEX): September 7-9, 2011.
Mechelen, Belgium, 2011: 72-77.
NARWARIA M, LIN Weisi. Video Quality Assessment Using Temporal Quality Variations
and Machine Learning[C]// Proceedings of2011 IEEE International Conference on Multimedia and Expo (ICME): July 11-15, 2011.
Barcelona, Spain, 2011: 1-6.
WOLF S, PINSON M. Low Bandwidth Reduced
Reference Video Quality Monitoring System[C]// Proceedings of the 1st International Workshop on Video Processing and Quality Metrics

China Communications • May 2013

for Consumer Electronics: January, 2005.
[36]

[37]

TAGLIASACCHI M, VALENZISE G, NACCARI M,

Reference Perceptual Quality Assessment of

ity Approximation for Videos Corrupted by

JPEG Compressed Images[C]// Proceedings ofChannel Errors[J]. Multimedia Tools Applica-

2002 International Conference on Image Pro-

tion, 2010, 48(3): 471-492.

cessing (ICIP 2002): September 22-25, 2002.

LE CALLET P, VIARD-GAUDIN C, BARBA D. A

Rochester, NY, USA, 2002: I-477-I-480.
[49]

tive Image Sharpness Metric Based on theTransactions on Neural Network, 2006, 17(5):Notion of Just Noticeable Blur (JNB)[J]. IEEE

1316-1327.

Transactions on Image Processing, 2009, 18(4):PINSON M.H, WOLF S. A New Standardized

717-728.
[50]

Reference Quality Assessment Using Nature

2004, 50(3): 312-322.

Scene Statistics: JPEG 2000[J]. IEEE Transa-

REDI J.A, GASTALDO P, HEYNDERICKX I, et al.

ctions on Image Processing, 2005, 14(11): 1918-

Reference Assessment of Perceived Image

1927.
[51]

Blind Image Quality Assessment without Hu-

Video Technology, 2010, 20(12): 1757-1769.

man Training Using Latent Quality Factors[J].

LI Qiang, WANG Zhou. Reduced-Reference Im-

IEEE Signal Processing Letters, 2012, 19(2):malization-Based

Image

Representation[J].

75-78.
[52]

ceptual DCT Statistics Based Blind Image

essing, 2009, 3(2): 201-211.

Quality Metric[J]. IEEE Signal Processing Let-

WANG Zhou, WU Guixing, SHERIKH H.R, et al.

ters, 2010, 17(6): 583-586.
[53]

Image Processing, 2006, 15(6): 1680-1689.

lity Assessment: from Natural Scene Statistics
to Perceptual Quality[J]. IEEE Transactions on

Using Reorganized DCT-Based Image Repre-

Image Processing, 2011, 20(12): 3350-3364.
[54]

Image and Video Quality Estimation: Applica-

2011, 13(4): 824-829.

tions and Human-Motivated Design[J]. Image

MA Lin, LI Songnan, NGAN K N. Reduced-

Communication, 2010, 25(7): 469-481.
[55]

for proposals on scalable video coding exten-

on Circuit System Video Technology, 2012,

sions of high efficiency video coding (HEVC)

MA Lin, LI Songnan, ZHANG Fan, et al. Re-[R]. 2012.
[56]

jective Evaluation of Scalable Video Coding

in Reorganized DCT Domain[J]. IEEE Transac-

for Content Distribution[C]// Proceedings oftions on Multimedia, 2011, 13(4): 824-829.

the International Conference on Multimedia

WANG Zhou, BOVIK A C. Reduced- and no-(MM’10): October 25-29, 2010. Firenze, France,tural Scene Statistics Model Approach[J]. IEEE

2010: 65-72.
[57]

ZHAI Guangtao, CAI Jianfei, LIN Weisi, et al.

Signal Processing Magazine, 2011, 28(6): 29-

Cross-Dimensional Perceptual Quality Asses-

40.

sment for Low Bit-Rate Videos[J]. IEEE Transac-

LIANG Luhong, WANG Shiqi, CHEN Jianhua, etal. No-Reference Perceptual Image Quality[47]

LEE J S, SIMONE F D, RAMZAN N, et al. Sub-

duced-Reference Image Quality Assessment

Reference Image Quality Assessment: the Na-[46]

Video Coding Experts Group (VCEG). Joint callpressed Video Sequences[J]. IEEE Transactions
22(10): 1441-1456.

[45]

HEMAMI S S, REIBMAN A R. No-Reference

sentation[J]. IEEE Transactions on Multimedia,

Reference Video Quality Assessment of Com-

[44]

MOORTHY A K, BOVIK A C. Blind Image Qua-

MA Lin, LI Songnan, ZHANG Fan, et al. Reduced- Reference Image Quality Assessment[43]

SAAD M A, BOVIK A C, CHARRIER C. A Per-IEEE Journal of Selected Topics in Signal Proc-

Quality-Aware Images[J]. IEEE Transactions on[42]

MITTAL A, MURALIDHAR G S, GHOSH J, et al.

Quality[J]. IEEE Transactions on Circuit System

age Quality Assessment Using Divisive Nor-[41]

SHEIKH H R, BOVIK A C, CORMARK L. No-

Quality[J]. IEEE Transactions on Broadcasting,Color Distribution Information for Reduced-[40]

FERZLI R, KARAM L J. A No-Reference Objec-

Objective Video Quality Assessment[J]. IEEE

Method for Objectively Measuring Video[39]

WANG Zhou, SHEIKH H R, BOVIK A C. No-

et al. A Reduced-Reference Structural Similar-

Convolutional Neural Network Approach for[38]822-833.
[48]

tions on Multimedia, 2008, 10(7): 1316-1324.
[58]

MOORTHY A K, CHOI L K, BOVIK A C, et al.

Metric Using Gradient Profiles for JPEG 2000

Video Quality Assessment on Mobile Devices:[J]. Signal Processing: Image Communication,

Subjective, Behavioral and Objective Stud-2010, 25(7): 502-516.

ies[J]. IEEE Journal of Selected Topics in Signal

BRANDAO T, QUELUZ M P. No-Reference Image Quality Assessment Based on DCT Domain Statistics[J]. Signal Processing, 2008, 88(4):China Communications • May 2013

Processing, 2012, 6(6): 652-671.
[59]

YEGANEH H, WANG Zhou. Objective Quality
Assessment of Tone Mapped Images[J]. IEEE75

USA: University of California, Berkeley, 2002.

Transactions on Image Processing, 2013, 22(2):657-667.
[60]

[71]

YEGANEH H, WANG Zhou. Objective Quality

tion of Clustering Methods[J]. Journal of theAssessment of Tone Mapping Algorithms[C]//American Statistical Association, 1971, 66(336):846-850.

Proceedings of the 17th IEEE International
Conference on Image Processing (ICIP 2010):

[61]

[72]

Comparing Two Hierarchical Clusterings[J].

2010: 2477-2480.

Journal of the American Statistical Association,
1983, 78(833): 553-569.

AYDIN T O, MANTIUK R, MYSZKOWSKI K, et al.
[73]

Assessment[J]. ACM Transactions on Graphics,
CADIK M, HAJDOK O, LEJSEK A, et al. Evalua-of Computer Vision, 2005, 1: 394-400.
[74]

Segmentation Quality by Adaptive Ground//www.cgg.cvut.cz/members/cadikm/tmo.

Truth

LIU Jianqing, YANG Y. Multiresolution Color

Computer Science, 2012, 7574: 287-300.
[75]

IEEE Transactions on Pattern Analysis andMachine Intelligence, 2004, 26(5): 530-549.
[76]

ter, 1998, 19(8): 741-747.

Complex Wavelet Structural Similarity: A New
Image Similarity Index[J]. IEEE Transactions on

Entropy-Based Objective Segmentation Eva-

Image Processing, 2009, 18(11): 2385-2401.
[77]

MOORTHY A K, SU Chechun, MITTAL A, et al.

SPIE - Electronic Imaging Storage and Re-

Subjective Evaluation of Stereoscopic Imagetrieval Methods and Applications for Multi-

Quality[J]. Signal Processing: Image Commu-

REN Xiaofeng, MALIK J. Learning a Classifica-

nication, 2012.
[78]

LEBRETON P, RAAKE A, BARKOWSKY M, et al.

tion Model for Segmentation[C]// Proceed-

Evaluating Depth Perception of 3D Stereo-

ings of the 9th IEEE International Conference

scopic Videos[J]. IEEE Journal of Selected Topics in Signal Processing, 2012, 6(6): 710-720.
[79]

BOSC E, PEPION R, LE CALLET P, et al. Towards

CHRISTENSEN H I, PHILLIPS P J. Empirical Eva-a New Quality Metric for 3-D Synthesized View

luation Methods in Computer Vision[M]. Sin-

Assessment[J]. IEEE Journal of Selected Topics

gapore: World Scientific Publishing Company,
2002.

in Signal Processing, 2011, 5(7): 1332-1343.
[80]

HEWAGE C T E R, MARTINI M G. Reduced-

MARTIN D, FOWLKES C, TAL D, et al. A Data-

Reference Quality Assessment for 3D Videobase of Human Segmented Natural Images

Compression and Transmission[J]. IEEE Trans-

and Its Application to Evaluating Segmenta-actions on Consumer Electronics, 2011, 57(3):tion Algorithms and Measuring Ecological
Statistics[C]// Proceedings of the 8th IEEE In-

1185-1193.
[81]

MOORTHY A K, BOVIK A C. A Two-Step Fra-

ternational Conference on Computer Vision,

mework for Constructing blind Image Quality

2001 (ICCV 2001): July 7-14, 2001. Vancouver,Indices[J]. IEEE Signal Processing Letters, 2011,

BC, Canada, 2001: 416-423.
FREXENET J, MUNOZ X, RABA D, et al. Yet

17(2): 587-599.
[82]

BENOIT A, LE CALLET P, CAMPISI P, et al.

Another Survey on Image Segmentation: Re-

Quality Assessment of Stereoscopic Images[J].

gion and Boundary Information Integration

EURASIP Journal on Image and Video Proc-[C]// Proceedings of the 7th European Conference on Computer Vision-Part III (ECCV’02):

76

SAMPAT M P, WANG Zhou, GUPTA S, et al.

ZHANG Hui, FRITTS J E, GOLDMAN S A. An

16, 2003. Nice, France, 2003: 10-17.

[70]

MARTIN D R, FOWLKES C C, MALIK J. Learning
Local Brightness, Color and Texture Cues[J].

on Computer Vision, 2003 (ICCV): October 13-

[69]

in

BORSOTTI M, CAMPADELLI P, SCHETTINI P C.

media, 2004, 5307: 38-49.

[68]

Notes

to Detect Natural Image Boundaries Using

luation Method for Image Segmentation[J].

[67]

Lectures1994, 16(7): 689-700.

mentation Results[J]. Pattern Recognition Let-[66]

Composition[J].

Pattern Analysis and Machine Intelligence,

Quantitative Evaluation of Color Image Seg-

[65]

PENG Bo, ZHANG Lei. Evaluation of Imagetion of Tone Mapping Operators[EB/OL]. http:Image Segmentation[J]. IEEE Transactions on[64]

UNNIKRISHNAN R, HEBERT M. Measures ofSimilarity[J]. IEEE Workshop on Applications

2008, 27(3): 69.

[63]

FOWLKES E B, MALLOWS C L. A Method for

September 26-29, 2010. Hong Kong, China,

Dynamic Range Independent Image Quality[62]

RAND W M. Objective Criteria for the Evalua-essing, 2009: 1-13.
[83]

HEWAGE C T E R, MARTINI M G. Reduced-

May 28-31, 2002. Copenhagen, Denmark, 2002:

Reference Quality Metric for 3D Depth Map

408-422.

Transmission[C]// Proceedings of 3DTV-Con-

MARTIN D R. An Empirical Approach to Grou-

ference: The True Vision- Capture, Transmis-

ping and Segmentation[D]. San Francisco,sion and Display of 3D Video (3DTV-CON),

China Communications • May 2013

2010: June 7-9, 2010. Tampere, Finland, 2010:
1-4.
[84]

Dense Correspondence across Different Scen-

Perceptual Quality Assessment for Stereos-

es[C]// Proceedings of the 10th European Con-

copic Images Based on 2D Image Quality

ference on Computer Vision: Part III (ECCV’08):

Metrics and Disparity Analysis[C]// Proceed-

October 12-18, 2008. Marseille, France, 2008:

ings of the 5th International Workshop on

28-42.
[95][87]

Summarizing Visual Data Using Bidirectional

USA, 2010: 281-316.

Similarity[C]// Proceedings of IEEE Conference

GORLEY P, HOLLIMAN N. Stereoscopic Imageon Computer Vision and Pattern Recognition

Quality Metrics and Compression[J]. Stereos-

2008 (CVPR 2008): June 23-28, 2008. Ancho-

SHEN Lili, YANG Jiachen, ZHANG Zhouyun.

rage, Alsaka, USA, 2008: 1-8.
[96]

al. PatchMatch: A Randomized Correspon-

Multiple Channel HVS Model[C]// Proceed-

dence Algorithm for Structural Image Edit-ings of the 2nd International Congress on

ing[C]// Proceedings of the International Con-

Image and Signal Processing, 2009 (CISP’09):

ference and Exhibition on Computer Graphics

October 17-19, 2009. Tianjin, China, 2009: 1-4.

and Interactive Techniques (SIGGRAPH): Aug-

YANG Jiachen, HOU Chunping, ZHOU Yuan, et

ust 3-7, 2009. New Orleans, Louisiana, USA,

Stereo Images[C]// Proceedings of 3DTV Con-

[91]

PELE O, WERMAN M. Fast and Robust EarthMover’s Distances[C]// Proceedings of 2009

sion and Display of 3D Video, 2009: May 4-6,

IEEE 12th International Conference on Com-2009. Potsdam, Germany, 2009: 1-4.

puter Vision: September 29-October 2, 2009.

ZHU Zhongjie, WANG Yuer. Perceptual Disto-

Kyoto, Japan, 2009: 460-467.
[98]

RUBNER Y, TOMASI C, GUIBAS L J. The Earthtion[J]. WSEAS Transactions on Signal Proces-Mover’s Distance as a Metric for Image Ret-

sing, 2009, 5(7): 241-250.

rieval[J]. International Journal of Computer

AKHTER R, SAZZAD Z M P, HORITA Y, et al. No

Vision, 2000, 40(2): 99-121.

Reference Stereoscopic Image Quality Asses-[90]

2009: 24.
[97]

ference: The True Vision - Capture, Transmis-

rtion Metric for Stereo Video Quality Evalua-[89]

BARNES C, SHECHTMAN E, FINKELSTEIN A, et

Stereo Picture Quality Estimation Based on aal. Objective Quality Assessment Method of[88]

SIMAKOV D, CASPI Y, SHECHTMAN E, et al.

2010): January 13-15, 2010. Scottsdale, Arizona,

copic Displays and Applications, 2008, 6803.
[86]

LIU Ce, YUEN J, TORRALBA A, et al. SIFT Flow:

YOU Junyong, XING Liyuan, PERKIS A, et al.

Video Processing and Quality Metrics (VPQM[85]

nology, 2001, 11(6): 703-715.
[94]

sment[J]. Stereoscopic Displays and Applica-

Biographies

tions, 2010, 7524: 752407-752407-12.

MA Lin, received his B.E. and M.E. degrees in com-

RUBINSTEIN M, GUTIERREZ D, SORKINE O, etputer science from Harbin Institute of Technology

al. A Comparative Study of Image Retarget-(HIT), China in 2006 and 2008, respectively. He ising[C]// Proceedings of the 3rd ACM SIGG-

currently pursuing his Ph.D. degree at the Depart-

RAPH Conference and Exhibition on Com-

ment of Electronic Engineering, Chinese University ofputer Graphics and Interactive Techniques inHong Kong (CUHK), Hong Kong, China. He was a

Asia: December 15-18, 2010. Hong Kong, China,

recipient of the Best Paper Award in Pacific-Rim

2010: 160.

Conference on Multimedia in 2008. He was awarded

MA Lin, LIN Weisi, DENG Chenwei, et al. Im-

Microsoft Research Asia Fellowship in 2011. His re-age Retargeting Quality Assessment: A Study

search focuses on image/video quality assessmentof Subjective Scores and Objective Metrics[J].

and compression.

IEEE Journal of Selected Topics in Signal Pro[92][93]

cessing, 2012, 6(6): 626-639.

DENG Chenwei, received his Ph.D. degree in signal

MA Lin, LIN Weisi, DENG Chenwei, et al. Study

and information processing from Beijing Institute ofof subjective and Objective Quality Assess-

Technology (BIT), Beijing, China in 2009. From May

ment of Retargeted Images[C]// Proceedings

2009 to August 2012, he was a Postdoctoral Researchof 2002 International Conference Symposium

Fellow at the School of Computer Engineering, Nan-on Circuits and Systems (ISCAS 2012): Mayyang Technological University (NTU), Singapore.

20-23, 2012. Seoul, Korea, 2012: 626-639.

Currently, he is an Associate Professor with the

MANJUNATH B S, OHM J, VASUDEVAN V V, et

School of Information and Electronics, Beijing Insti-

al. Color and Texture Descriptors[J]. IEEE Tran-tute of Technology, China. His research interests in-sactions on Circuits Systems for Video Tech-clude image/video coding, perceptual modeling, andChina Communications • May 201377

computer vision. *The corresponding author. Email:Lecturer in 2006-2007.

cwdeng@bit.edu.cn
LIN Weisi, received his B.S. and M.S. degrees from78

King Ngi Ngan, is a Chair Professor at the Depart-

Zhongshan University, China, and a Ph.D. degree inment of Electronic Engineering, Chinese University ofcomputer vision from King’s College, London Univer-Hong Kong (CUHK), Hong Kong, China. He holds

sity, UK. He is now an Associate Professor in thehonorary and visiting professorships of numerous

School of Computer Engineering, Nanyang Techno-

universities in China, Australia and South East Asia.

logical University (NTU), Singapore. Prior to this, he

His research interests are in video coding and visual

was the Laboratory Head and the Acting Department

signal processing, and he published extensively inManager in the Institute for Infocomm Research,these areas. Prof. Ngan is a Fellow of IEEE (USA),

Singapore. His areas of expertise include perceptual

Institute of Educational Technology (UK), and Institu-

modeling, video compression, multimedia quality ev-

tion of Engineers Australia, and an IEEE Distinguishedaluation and computer vision.

China Communications • May 2013