590

IEEE TRANSACTIONS ON MULTIMEDIA, VOL. 18, NO. 4, APRIL 2016

Free-Energy Principle Inspired Video Quality
Metric and Its Use in Video Coding
Long Xu, Member, IEEE, Weisi Lin, Fellow, IEEE, Lin Ma, Member, IEEE, Yongbing Zhang, Member, IEEE,
Yuming Fang, Member, IEEE, King Ngi Ngan, Fellow, IEEE, Songnan Li, Member, IEEE, and Yihua Yan

Abstract—In this paper, we extend the free-energy principle tovideo quality assessment (VQA) by incorporating with the recent
psychophysical study on human visual speed perception (HVSP).
A novel video quality metric, namely the free-energy principle
inspired video quality metric (FePVQ), is therefore developed andapplied to perceptual video coding optimization. The free-energy
principle suggests that the human visual system (HVS) can actively
predict “orderly” information and avoid “disorderly” information
for image perception. Basically, “orderly” is associated with theskeletons and edges of objects, and “disorderly” mostly concerns
textures in images. Based on this principle, an image is separated
into orderly and disorderly regions, and processed differently
in image quality assessment. For videos, visual attention, or
fixation, is associated with the objects with significant motion
according to HVSP, resulting in a motion strength factor in
the FePVQ so that the free-energy principle is extended into
spatio-temporal domain for VQA. In addition, we investigate
the application of the FePVQ in perceptual rate distortion
optimization (RDO). For this purpose, the FePVQ is realized withlow computational cost by using the relative total variation modeland the block-wise motion vectors of video coding to simulate thefree-energy principle and the HVSP, respectively. The experimental
results indicate that the proposed FePVQ is highly consistent
with the HVS perception. The linear correlation coefficient andSpearman’s rank-order correlation coefficient are up to 0.8324
and 0.8281 on the LIVE video database. Better perceptual quality
Manuscript received July 6, 2015; accepted January 25, 2016. Date of publication February 3, 2016; date of current version March 15, 2016. This work was
supported in part by the National Natural Science Foundation of China (NSFC)
under Grant 61232016, Grant U1405254, Grant 61572461, Grant 61202242,
Grant 61571212, and Grant 11433006, in part by CAS “100-Talents” (Dr. X.
Long), in part by the NSFC Guangdong Joint Fund under Grant U1201255 andGrant U1301257, and in part by the Research Grants Council of the Hong Kong
SAR under Project CUHK 415712. The associate editor coordinating the reviewof this manuscript and approving it for publication was Prof. Klara Nahrstedt.
(Corresponding author: Lin Ma.)
L. Xu is with the Key Laboratory of Solar Activity, National Astronomical
Observatories, Chinese Academy of Sciences, Beijing 100012, China, and alsowith the School of Computer and Software, Jiangsu Engineering Center of Network Monitoring, Nanjing University of Information Science and Technology,
Nanjing 210044, China (e-mail: lxu@nao.cas.cn).
W. Lin is with the Department of Computer Engineering, Nanyang Technological University, Singapore 639798 (e-mail: WSlin@ntu.edu.sg).
L. Ma is with Huawei Noah’s Ark Lab, Hong Kong 999077, China(e-mail: forest.linma@gmail.com).
Y. Zhang is with the Graduate School at Shenzhen, Tsinghua University,
Shenzhen 518005, China (e-mail: zhang.yongbing@sz.tsinghua.edu.cn).
Y. Fang is with the School of Information Technology, Jiangxi University of Finance and Economics, Nanchang 330032, China (e-mail:fa0001ng@e.ntu.edu.sg).
K. N. Ngan and S. Li are with the Department of Electronic Engineering,
Chinese University of Hong Kong, Hong Kong 999077, China (e-mail: knngan@ee.cuhk.edu.hk; snli@ee.cuhk.edu.hk).
Y. Yan is with the Key Laboratory of Solar Activity, National Astronomical
Observatories, Chinese Academy of Sciences, Beijing 100012, China (e-mail:yyh@nao.cas.cn).
Color versions of one or more of the figures in this paper are available online
at http://ieeexplore.ieee.org.
Digital Object Identifier 10.1109/TMM.2016.2525004

of encoded video sequences is achieved by FePVQ-motivated RDO
in video coding.
Index Terms—Free-energy principle, perceptual video coding
optimization, video coding, visual quality assessment (VQA).

I. INTRODUCTION
URING recent years, there has been an increasing interestin visual quality assessment of image/video. Since humans
are the ultimate receivers of visual signal being processed in mostsituations, the most accurate way of assessing image/video quality (VQ) is to ask humans for their opinions on the quality of theimage/video, which is known as subjective visual quality assessment. However, subjective visual quality assessment is very time
consuming, laborious and expensive. Moreover, it is infeasible tohave subjects’ intervention with in-loop and on-service processes.
Therefore, objective quality assessment targeting at automatically
predicting visual quality by computer model is in demand.
Objective quality assessment metrics can be classified into two
categories: signal fidelity measures and perceptual visual quality
metrics (PVQMs). The signal fidelity measures, like mean squareerror (MSE), signal-to-noise ratio (SNR) and peak SNR (PSNR),
generally are poor predictor of perceived visual quality. PVQMs,
which are designed to measure the perceived visual quality, havebeen intensively developed during the last decades. Some wellknown PVQMs include moving pictures quality metric [1], perceptual distortion metric [2], Sarnoff just noticeable difference (JND)[3] vision model, digital video quality [4], and scalable wavelet
based video distortion index [5]. In [6], You et al. proposed a visual perception model based on foveated vision for video quality
assessment (VQA). In [7], an advanced foveal imaging model was
proposed by incorporating visual attention into [6]. In [8], Lu et al.
proposed a set of heuristic fuzzy rules that use both relative andabsolute motion information to account for motion suppression andvisual attention. It was shown that these rules are effective in improving VQA performance of the standard MSE/PSNR measures
as well as the SSIM [9]–[11] approach. In [12], the contributionof spatial and temporal factors and their interaction were exploredby machine learning [13], [14] and a low-complexity VQA using
temporal quality variation is proposed.
With recent studies on brain theory and neuroscience, the freeenergy principle has been found to be existing in action, perception
and learning of human brain [15], [16]. It can be incorporated into
existing image quality assessment (IQA) metrics with the forms
of JND [17], importance weight for pooling local quality measurement, and so on. The free-energy principle claims that any
self-organizing system that is at equilibrium with its environment
must minimize its free energy. The essence of this principle is asort of mathematical formulation of how adaptive systems resist
a natural tendency to disorder. The free-energy principle indicatesthat the human visual system (HVS) can actively predict the sensory information and avoid the residual uncertainty/disorder forD

1520-9210 © 2016 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See http://www.ieee.org/publications standards/publications/rights/index.html for more information.

XU et al.: FEPVQ AND ITS USE IN VIDEO CODING

image perception and understanding [17]. In other words, the HVS
tries to extract as much information as possible to minimize theuncertainty of an input scene. However, it is impossible to fully
process all of the sensation information and thus it has to discardsome uncertainties in a visual scene. There have been a few efforts
to apply this principle to visual quality assessment [17], [18]. In[17], an IQA metric was proposed inspired by the Bayesian brain
theory [19] and the free-energy principle which indicates that thehuman brain works with an internal generative mechanism (IGM)
for visual information perception and understanding. Regarding
IGM, the HVS acts as an inference system that actively predicts
the visual sensation and aviods the residual uncertainty/disorder.
And, an auto-regression model was adopted to decompose a visual
scene into two portions: predicted sensory content and residual uncertain content. Then, they were processed differently by existing
IQA metrics. In [18], a new psychovisual image quality metric is
developed based on free-energy principle. However, there have notyet been considerations of free-energy principle in application of
VQA. Compared to image signal, video signal is depicted as asignal in a three-dimensional (3-D) field (x, y , z ). To investigate
the free-energy principle in video signal processing, we have toextend spatial orderly and disorderly terms into spatio-temporal
domain beyond [17]. Specifically, for inter-frame predicted videocoding, the motion estimation (ME) [20] and motion compensation (MC) revise the spatial signal extensively, where the highly
textured regions would be disorderly and noisy after ME/MC withthe repetitive pattern of textures removed by ME/MC process. As
a result, the free-energy principle needs to be further explored in
depth in video coding.
Our study targets at a VQA metric which is computationallyefficient, and can be easily integrated into video coding process tohave a close-loop solution of video coding optimization. Consequently, we adopt the pooling of blocked MSE, which is the only
quality measurement in video coding, in our proposed scheme. Our
study is inspired by the recent revealed free-energy principle [15],[16] and the psychophysical study by Stocker and Simoncelli on
human visual speed perception (HVSP) [21], [22]. In HVSP, the
HVS was first modeled as an efficient information extractor [23].
To achieve such efficiency, the HVS should pay more attention on
the areas/regions that contain more information in a visual scene.
For video signal, it is believed that object motion is associatedwith visual attention. Second, the visual perception was modeled
as an information communication process and the HVS was anerror-prone communication channel. The amount of informationthat can be perceived at the receiver end depends on the noise levelof the communication channel (the HVS), so the internal noise in
the HVS results in perception uncertainty. These two models suggest the spatial visual attention and the temporal visual attention,
respectively in video signal perception. In addition, they separate
the video signal into orderly and disorderly parts (sensory information and perception uncertainty) in spatio-temporal domain so thatthey could further finetune visual perception by applying different
policies to the two separated video signal parts.
In this paper, a spatio-temporal weight function is defined andincorporated as weighting factors for pooling block based MSE for
VQA. The block size is 4 × 4 which corresponds to the smallest processing unit of hybrid video coding (e.g., H.264/AVC and HEVC).
Thus, a novel VQA metric, namely free-energy principle inspired
video quality metric (FePVQ) is proposed for VQA. As a variant of MSE, FePVQ can be easily incorporated into hybrid videocoding to have a close-loop solution of video coding optimization.
To reach a low computational cost, the available information ofvideo coding is reused in FePVQ as much as possible. We reuse591

the motion vectors to approximate HVSP for motion perception.
In addition, we employ a computationally efficient model, namely
relative total variation (RTV) [24] to distinguish orderly and disorderly information instead of the one [17] which was applied toimage and was unallowable to video processing due to extremely
high computational cost. The contributions of this paper lie in thefollowing respects:
1) extending the free-energy principle accounting for IQA into
spatio-temporal video signal quality assessment;2) incorporating the recent psychophysical study on HVSP into
VQA for measuring motion speed perception;3) developing a low-complexity VQ metric for perceptual videocoding optimization; and4) simplifying the free-energy principle and HVSP models by
employing the RTV and reusing motion vectors of videocoding for saving computations.
The rest of this paper is organized as follows. Section II presents
the proposed FePVQ algorithm in detail. Section III investigates
the perceptual RDO of video coding based on FePVQ. Section IV
evaluates the performance of FePVQ for predicting perceptual VQ,
and the R-D efficiency of FePVQ-based video coding optimization
by extensive experimental results. Finally, a brief conclusion is
given in the last section.

II. FREE-ENERGY PRINCIPLE INSPIREDVIDEO QUALITY METRIC
According to free-energy principle [15], [16], orderly and disorderly regions are loosely associated with structural edges andtextures, respectively in visual scene. The structural edges could
provide more semantics information of a visual scene. For example, the structural edges of an image could clearly reveal the visual objects contained, such as peoples, animals, flowers, cars andbuildings. Thus, they are most important to the HVS and should
be preserved as much as possible in digital image processing. The
textures of a scene are usually the surface of objects, such as texture
patterns of wallpaper, grass, tree and road surface, which containdetails of objects, and could further augment the objects with more
appealing properties, such as fine texture, smooth grayscale transition and rich color, and make them vivid to human perception.
Thus, structure and texture jointly render the users impressive perspective of visual scenes. However, for bit rate constraint usages,
such as image/video coding, the quality of these two visual forms
cannot be guaranteed at the same time. Generally, they are contradictory due to the constant bit rate in the communication systems.
In these scenarios, the structure regions should be given the priority of bit quota compared with texture regions. Specifically, concerning free-energy principle in motion-compensated inter-frame
video coding (named hybrid video coding, such as H.264/AVC), thehighly textured regions may become disorderly with noise property
after inter-frame ME/MC, as the repetitive pattern of textures havebeen removed by ME process. As a result, only structural edges
after ME/MC still keep orderly and reveal clear profiles of visual objects. Thus, it is essential to process these two visual forms
differently.
In this paper, 3-D gradient of video sequences is investigated as a
3-D cube consisting of tiles which are individual frames. In the 3-D
cube, the moving objects would stretch out a plane along temporal
trajectory. As a result, the video clip would vary along a certain
spatio-temporal direction. Each pixel in the cube is associated witha 3-D gradient represented by a triple parameter (∂x, ∂y , ∂z ),where ∂x, ∂y and ∂z are the partial derivatives along x, y andz axes, respectively. ∂x and ∂y are computed from the spatial

592

IEEE TRANSACTIONS ON MULTIMEDIA, VOL. 18, NO. 4, APRIL 2016

Fig. 1. Temporal and spatial (horizontal and vertical) gradients of video sequences (left: temporal gradient for video sequence; center: spatial verticalgradient; right: spatial horizontal gradient).

displacement of one pixel along horizontal and vertical directions,
respectively. ∂z is computed from the temporal displacement of
two neighboring frames. Fig. 1 shows the examples of (∂x, ∂y, ∂z )on two video sequences.

A. Textural Strength in Spatio-Temporal Gradient Domain
Generally, the gradient is directly used to compute the mutual
correlation/similarity as in SSIM. In addition, SSIM is usually computed on pixel-basis, which requires more computations than blockwise metrics. For better integration with video coding, the proposed
FePVQ uses the gradients as weights for MSE [25], [26] to simulate
how the HVS responses to visual signals. Nowadays, PSNR/MSE
is still widely used, especially for video compression. Thus, thequality metric in the variant form of MSE can be easier to be integrated into current visual application systems. In this work, we
propose the texture strength at macroblock (MB) basis to depict the
local texture properties for quality assessment, which is defined as
T Sm =

 |∂x F |q + |∂y F |q + |∂z F |q(1)q ∈M B

where F is the input signal, ∂x, ∂y and ∂z represent the partial
derivatives, q denotes one pixel. Usually, the regions with detailed
visual content, such as fine texture, moving objects, have the large
gradient, so the texture strength is large. Besides spatial gradient,temporal gradient defined by the difference between two consecutive frames is also included in (1). From Fig. 1, we had the sameobservation that the texture regions and moving objects had large
gradient. Thus, these three gradients, ∂x, ∂y and ∂z are combined
to measure texture strength in (1).
Based on the knowledge that detailed texture regions can conceal
more distortions than smooth regions [25], [26], a VQ metric isdefined as the MSE weighted by texture strength [25] as
V QTmS =1
× M SEm ; m = 0, 1, 2, . . . , M − 1
T Sm

(2)

where T Sm is a weight for the mth block of a frame, V QTmS
represents visual quality of the mth block and M is the total numberof blocks in a frame.

B. Structure Strength by Considering Free-Energy Principle
In hybrid video coding, such as H.264/AVC, ME is widely used toeliminate temporal redundancy between successive frames for interframe coding. The signal after ME/MC is of much less energy thanthe original one. That’s why video can be compressed dramatically.
However, there are still large residual coefficients for structuralFig. 2. Extraction of structures from textures. (a)-(d) are original images,“house,” “linghthouse,” “sailing2” (LIVE image database), and “tractor” (extracted from tractor YUV sequence), and (e)-(h) are extracted structure from(a)-(d).

edges and fast moving objects, which consume a large part of the
bit budget. Considering this aspect of the free-energy principle inhybrid video coding, texture after ME/MC is largely disorderly. The
disorderly signal is similar in characteristic as noise, where there
is little correlation between pixels. The structural signal mostly
concerns the skeleton of objects in image. From (2), it can be
deduced that the larger the texture strength, the less the visualquality loss under the same MSE. Referring to (1), it can be observedthat both structure and texture regions are of large texture strength.
Since the HVS is highly adapted to extract structural information, it
is necessary to distinguish the structure information from the texture
information so that they can be processed differently in VQA andvideo coding. The algorithm of [17] is of significant computational
complexity although it can be used to separate structure and texture
ideally. In [24], two models, pixel-wise windowed total variation
and windowed inherent variation were defined as⎧⎪⎪⎪⎨ Dx (p) =



⎪
⎪
⎪
⎩ Lx (p) =q ∈R (p)gp,q (∂x F(3)
q ∈R (p)gp,q (∂x F

where ∂x represents the partial differential operator on the input
signal F , R(p) is a neighboring region around the pixel p, Dx (p)and Lx (p) indicate the pixel-wise windowed total variation andwindowed inherent variation are computed along x direction, andgp,q is a weighting function of Gaussian. Here, Dx (p) counts theabsolute spatial difference within the window R(p), while Lx (p)can measure the gradient consistency of R(p). Dy (p) and Ly (p)can be computed in the same way as (3), but along y direction.
Then, a RTV model was defined as L(p) divided by D(p) for
structure-texture decomposition. A window only including texture
generally has a smaller inherent variation L( p) than that including
both texture and structural edges, since the gradient directions of thestructural region are consistent while the gradient directions of the
texture region are random. Fig. 2 shows the extracted structure fromtexture by using the RTV, where the original images (see Fig. 2(a)–(d)) have plentiful textures, grass, water and uneven ground, andthe structural edges of these images are accurately extracted fromtexture background as shown in Fig. 2(e)–(h). In this study, theXU et al.: FEPVQ AND ITS USE IN VIDEO CODING

593

RTV is further extended to temporal domain to classify structure
and texture of video signals. Denote the three-dimension gradientsof video as (∂x, ∂y , ∂z ), the structure strength (SS) of each MB is
defined as
SSm =





(∂x F )q +q ∈M B(∂y F )q +q ∈M B(∂z F )q(4)

q ∈M B

where SSm represents SS of a block. Different from T Sm definedin (1), SSm depends on the factor whether the gradient directions(“+” or “−”) in a block are coincident or not. The blocks containing
only texture generally lead to a smaller SSm while those blocks
containing structural edges have a larger SSm .
Observing Fig. 1, which show temporal gradients for two video
sequences, we can notice the obvious skeleton of each video frame.
In addition, the skeleton is closely related to the boundaries ofsignificant objects, which indicates that the structural edges canbe separated from the background in an image by using temporal
gradient information. To further enhance the fidelity of SS, thetemporal gradient ∂z is included in (4) additionally. Furthermore,
the temporal gradient can represent temporal saliency very well.
Intuitively, (4) is a supplement to (1) since (1) cannot handle thedifference between texture and structure regions.
Regarding disorderly and orderly regions [17], they are basically corresponding to texture and structure, respectively in ME/MC
based hybrid video coding. In [17], the HVS responses to disorderly and orderly signals differently, which is integrated into a JND
model. It is believed that texture regions usually become disorderly after ME/MC in hybrid video coding, so they have a larger
JND threshold, and contribute less than structure ones to overall
perceptual visual quality. From (4), SSm of the texture regions isless than that of structure regions since the latter has inconsistentgradients.

C. Motion Strength Considering Human Visual Speed
Perception to the HVS
In video, a moving object should be associated with visual attention and can be used for predicting visual fixations [29]. This isbecause an object with significant motion with respect to the background would be a strong surprisal. However, as the backgroundmotion is too large, the HVS cannot identify the objects with sameaccuracy as in static background, which would result in humanperception uncertainty [21], [22]. For a natural video, a scene usually lasts for several seconds, where an almost same backgroundremains for all the frames within the scene. For simplicity, only
the relative motion represented by motion vectors is considered inthis work for the sake of computational efficiency. In video coding,motion vectors are available, so they are utilised to approximate theperceptual motion model without extra computations.
Assume motion vector (vr = (vx , vy )), which is given at N × N
block basis, e.g., a 4 × 4 block in H.264/AVC. When the multiplereferences are used, we first project vr onto its nearest forward
reference by vr = vr /d, where d represents distance from current
frame to its reference regarding to vr . Then, the motion strength isdefined as
M Sm = α ×log |vr (i, j)| + βi,j ∈M B|vr (i, j)| =vx (i, j)2 + vy (i, j)2d(i, j)2(5)

where M Sm is defined at MB basis, (vx , vy ) is the motion vectorof a 4 × 4 block and d(i, j) is the distance from current frame to its
reference; α and β are two constant parameters for fine-tuning the
profile of M Sm . The logarithm in (5) accounts for “Weber-Fechnerlaw” which states that the resolution of perception diminishes forthe stimuli of greater magnitude, specifically, the subjective sensation is proportional to the logarithm of the stimuli intensity.
Referring to [22] and [21], (5) can be explained as the selfinformation of motion vector (relative motion) whose prior distribution was assumed as a power-law function
p(v) = Cv −α(6)where α > 0 and C > 0 are two constant parameters; we use v toreplace |v | (motion vector length as defined in (5)) for simplicity.
The most common approach for evaluating empirical data against
a hypothesised power-law distribution is to observe that the powerlaw implies the linear form
− log p(v) = α log v + β(7)where β = − log C , “−” before log p(x) indicates self-informationof p(x). Although (7) is in a linear form with respect to α and
β , the linear regression cannot be relied on to estimate α and βsince it would result in significant systematic biases [31], [32].
Thus, the maximum likelihood estimation is employed to estimate
α and β from empirical data [31], [32]. By collecting a set of
motion vectors {vr (i)} from practical video coding, the maximum
likelihood estimation is performed as⎧⎪⎨α = 1+ n⎪⎩ni=1 log

vi

−1

vm in

(8)

α −1
β = − log C = − log ((α − 1)vm
in )where vi , i = 1, . . . , n is the observed value of v , vm in is the minimum value of v . It should be noticed that vm in corresponds not tothe smallest value of v measured but to the smallest for which thepower-law behaviour holds. We collect motion vectors by coding
the first 16 frames (about half a second for 30 f/s) of the standard
video sequences: “Foreman,” “Football,” “Mobile,” “Bus,” “Tennis” and “Flower” (CIF). The frames are coded in “P” frame byusing quarter pixel ME and only one forward reference. A pair of
α and β can be computed from (8) for each sequence, and then the
averages of them (α = 4.55, β = 4.20) are used to initialize α andβ in (5). We found that α and β are related to motion vector scaleand specific video coding system closely. That is why the values of
α and β are much different from those in [22]By combining (1), (4), (5) and local MSE, a new VQA metric,i.e., FePVQ is designed at MB basis as
V Qm =

a × SS b
M Smm× M SEm
c
T Sm

(9)

where M Sm , SSm and T Sm representing motion strength, SS andtexture strength, respectively, are weighting factors of local MSE;a, b and c are constants to finetune the profiles of M Sm , SSm and
T Sm , respectively. They are handpicked and set to 1.25, 1.25 and1.2 in our experiments. We found that this setting is competitive
among all the testings. However, more systematic way to choosethese parameters could recur to optimization methods. Equation (9)gives visual quality score of each MB, and the visual quality scoreof a frame is computed by summing up the scores of all the MBs.

594

IEEE TRANSACTIONS ON MULTIMEDIA, VOL. 18, NO. 4, APRIL 2016

III. PERCEPTUAL VIDEO CODING OPTIMIZATION
The proposed FePVQ is naturally associated with quantization
parameter (QP) since MSE in (9) can be theoretically formulated
by QP function [33]. Thus, it can be directly integrated into videocoding systems to obtain a closed-form analytic solution for perceptual video coding optimization. Since FePVQ is designed basedon blocks, it is suitable for optimizing any block based process invideo coding, such as block-based ME, intra prediction, inter prediction, and RDO. In video coding, multiple modes are provided toadapt to video content variation. Different modes represent different
block sizes, prediction types, transforms and partitions of an MB.
For MBs with simple textured video content, a larger size mode ispreferred. On the contrary, the MBs with fine textured video content
may benefit from the small partitions of an MB. The best partitionmode is decided by RDO, which seeks the best tradeoff between
distortion and bit cost as [34], [35]J = D+λ×R(10)

where D is measured in MSE for conventional RDO, R is thenumber of bits for coding an MB, and λ is known as the Lagrange multiplier which controls the tradeoff between D and R.
With the same distortion, the mode with the least bit cost would be
the best mode. Equivalently, the mode with the least MSE would
be the best mode under the same bit cost. For the perceptual optimization, a perceptual measurement of distortion instead of MSE
is desirable for RDO. In [36], the authors proposed to use SSIM
instead of MSE to make mode decision and RDO. However, theexisting RDO scheme [34] which was theoretically optimized for
MSE measurement may not be optimal to SSIM. The proposed
FePVQ is extended from MSE, and consequently it is completely
compatible with the existing RDO scheme. In our previous work[37], the optimal perceptual VQ control was investigated for bit rate
constrained video coding task. In this paper, we go deep into the
bit rate constrained optimization problem of video coding. Firstly,
perceptual RDO is studied by replacing D measured by MSE in(10) with FePVQ as
J = V Qm + λ × Rm=

M Sm × SSm
× M SEm + λ × Rm
T Sm

(11)

= pm × M SEm + λ × Rm

M Sm × SSm
Q2
× m = pm × Q2m
T Sm
12Referring to (11), pm changes the weight of MSE in RDO, which
would revise the final mode selection, and therefore obtain better perceptual quality for same bit cost. In addition, the QP alsochanges in a certain range so as to have more choices for perceptual optimization. Specifically, for pm larger than 1.1, the RDO is
performed asmin{J (s, T )}J (s, t) = V Qm (s, t) + λ(t) × Rm (s, t)s ∈ [Q − 2, Q − 1, Q];t ∈ skip/direct, 16 × 16, 16 × 8, . . . , 4 × 8, 4 × 4(14)

where t and s represents mode and QP, respectively. For pm lessthan 0.9, the RDO is performed asmin{J (s, T )}J (s, t) = V Qm (s, t) + λ(t) × Rm (s, t)s ∈ [Q + 2, Q + 1, Q];t ∈ skip/direct, 16 × 16, 16 × 8, . . . , 4 × 8, 4 × 4.

(15)

Such a process can achieve a better tradeoff between perceptual
visual quality and bit cost.
In FePVQ, significant structural edges would have larger SSm
and thus increase the weight of visual quality contribution in awhole image. This situation is consistent to the perception processof the HVS to an input image. This is why FePVQ is much better
than the previous scheme [37]. By integrating FePVQ into video
encoders, the structural edges of an image can be largely preserved
while the quality of disorderly texture regions is compromised tosave more bits for coding structural edges. Thus, better perceptual
visual quality of video encoding can be achieved under the sametarget bits quota.

IV. EXPERIMENTAL RESULTS AND DISCUSSIONS

which seeks the best tradeoff between perceptual visual quality
measured by FePVQ and bit rate cost. Secondly, according to masking property of the HVS, which describes why similar artifacts are
more visible in certain region of a frame while they are hardly noticeable in other regions, the QP is adapted accordingly to different
video contents to achieve consistent visual quality in a whole visual
scene.
With the assumptions of uniform quantization scheme and uniform DCT coefficients distribution, MSE is usually modeled byQ2 /12 [33]. Thus, the proposed FePVQ can be modeled theoretically by
V Qm =

median value of pm and clipped into [0.75, 1.25]. The distortions ofregions with simple texture background, strong structural edge andhigh motion strength are more sensitive to the HVS. For consistentvisual quality, after obtaining QP of each frame, denoted by Qf ,we could compute a new QP for each MB by referring to Qf√
Q m = Q f / pm .
(13)

(12)

which indicates that the VQs of MBs are different from each other
under the same quantization. The underlying reason is that different
kinds of input visual signals are perceived differently by the HVS.
So {pm } actually provides a visual perception map of a frame
consisting of various kinds of visual signals. In practice, pm inboth (11) and (12) acts as a scaling factor, which is normalized by a

In Part A below, we evaluate the performance of FePVQ to predict visual quality of images and videos. Then, it is integrated into
HM14.0 reference software of HEVC to optimize video encoding
from the perspective of perceptual visual quality. In Part B, we show
how FePVQ benefits video coding optimization. Part C discusses
the associated computational complexity analysis.

A. Performance of FePVQ Metric
The performance of a VQA metric can be evaluated by depicting the relationship of the predicted objective quality scores andthe provided subjective quality ratings [difference mean opinion
score (DMOSs)]. The DMOS is provided by subjective viewing
tests where the viewers are requested to give their opinions on
visual quality of distorted visual signals. The DMOSs and associated images/videos consist of subjective quality databases to be thebenchmarks for evaluating user-defined objective quality metrics.
The used databases in our experiments are introduced briefly in
Table I. We follow the procedure [38] recommended by video quality expert group to evaluate the performance of VQA approaches.

XU et al.: FEPVQ AND ITS USE IN VIDEO CODING

595

TABLE IIMAGE SUBJECTIVE QUALITY DATABASES
Database

Description

LIVE Image [39]There are 29 reference images, and totally 779 distorted images with five distortions types: JPEG, JPEG2000, white noise, Gaussian blur,and simulated Rayleigh fading channel.
The TID2008 contains 25 reference images and 1700 distorted images (25 reference images × 17 types of distortions × 4 levels of
distortions). Distortion types mostly concern all kinds of noises, blur and JPEG/JPEG2000 compression and transmission error andblock-wise distortions.
There are 30 reference images, and totally 866 distorted images generated from six different types of distortion: JPEG, JPEG2000,
Global contrast decrements, Gaussian blur, addictive Gaussian white noise, and addictive Gaussian pink noise at four to five different
levels of distortion.
Ten videos: 250 frames, 768×432, 25/50f/s, YUV 4:2:0 and progressive; 150 distorted videos, the distortion types include: transmission
over wireless networks, transmission over error-prone IP networks, MPEG-2 compression and H.264 compression; camera patterns: still,camera pan, circular camera motion and zoom in; ownership: Image & Video Engineering (LIVE) Laboratory, University of Texas atAustin.
10 high-resolution reference videos, 1920×1088, YUV 4:2:0 and progressive; 128 distorted ones generated using 4 types of distortions:
MPEG-2 compression, Dirac wavelet compression, H.264 compression and packet loss on the H.264 streaming through IP networks.
12 high-quality reference videos, 832×480, YUV 4:2:0 and progressive; 216 distorted videos with six different types of distortion, i.e.,
H.264 compression, HEVC compression, MJPEG compression, Wavelet-based compression, H.264 bitstream over wireless channel and
AWN. Each distortion type has three different distortion levels.

TID2008 [40]CSIQ [41]LIVE Video [45]

IVP Video [46]CSIQ Video [47]TABLE IIPERFORMANCE COMPARISON ON MULTIPLE DATABASES
Database
TID2008

CSIQ

LIVE Image

PLCC
SROCC
RMSE
PLCC
SROCC
RMSE
PLCC
SROCC
RMSE

PSNR

SSIM

MSSIM

VIF

ESSIM

FSIM

FePVQ (18)0.57060.58001.10200.80010.80570.15750.87440.881313.2300

0.79120.79650.37400.86130.87560.13340.82420.911215.4430

0.84250.85280.72990.89980.91380.1145
0.94300.94459.09560.80900.7496
0.78880.92770.9193
0.09800.95980.9631
7.67340.88080.88560.28900.87630.88790.09680.95470.96118.11800.84970.85740.32200.91200.92420.10770.96040.9652
7.59700.84930.86070.32300.81330.82670.10780.93180.93759.8990The monotonic logistic function with five parameters β1 , β2 , β3 ,
β4 , β5 is used to map visual quality index computed from VQA
metrics into final objective score Vj as
Vj = β1 × 0.5 −1
1 + eβ 2 ×(x j −β 3 ) )+ β4 × xj + β5 .

(16)
The parameters are determined by some parametric minimization
approaches, such as least squares. Three statistical measurements,
linear correlation coefficient (LCC), Spearman’s rank order correlation coefficient (SROCC) and RMSE are used to measure thedegree of correlation between DMOS and predicted values of VQA
metric. Given two sets A and B , LCC is defined as the correlation
coefficient of A and B as
LCC(A, B) = ni=1 (Ai − Ā)(Bi − B̄)
 n
n
2
2
i=1 (Ai − Ā)i=1 (Bi − B̄)(17)where Ā and B̄ represent the means of A and B , respectively.
Intrinsically, LCC measures the accuracy of a VQA metric to predict DMOS. SROCC evaluates the prediction monotonicity, and
RMSE measures the error during the fitting process. Larger LCC
and SROCC values indicate the better correlation between objective
and subjective scores, while smaller RMSE means smaller error of
VQA prediction, therefore a better performance.
1) Performance of FePVQ on Images: Although (9) is designed for VQA, it can be directly adapted as
V Qm =SSm
× M SEm
T Sm(18)

for IQA, where T Sm and SSm are associated with image insteadof video. We compare (18) with several state-of-the-art IQA metrics: PSNR, SSIM [9], MSSIM [10], VIF [41], FSIM [42] andESSIM [43] over LIVE image [39], TID2008 [40], and CSIQ [41]databases. The statistics of LCC and SROCC are tabulated in Table
II. It shows that (18) can improve LCC and SROCC up to 0.9318
and 0.9375, respectively on LIVE image database [40], which is significantly beyond PSNR/MSE whose LCC and SROCC are 0.8744
and 0.8813, respectively. The reason is the use of texture strengthand SS inspired by the free-energy principle of HVS.
2) Performance of FePVQ on Videos: To evaluate FePVQ on
videos, the LIVE video [45], Image Video Processing (IVP) video[46] and CSIQ video [44] databases are used. In this experiment, theproposed FePVQ in (9) for videos is compared with state-of-theart metrics including PSNR, SSIM [9], MSSIM [10], FSIM [42],ESSIM [43], MOVIE [45], VSNR [46], NTIA-VQM [48], VIF[41] and our previously proposed one [37]. The LCC and SROCC
statistics on LIVE video database are tabulated in Table III. As
PSNR, SSIM, MSSIM, VSNR and VIF only provide frame-level
quality scores, the final quality index of video is generated byaveraging frame-level quality scores. From Table III, it can beobserved that PSNR performs poorly since it is loosely related to
the HVS perception. The VSNR is also not satisfactory since itmeasures distortion as to the HVS perception in wavelet domain,
while MPEG-2 and H.264/AVC are associated with quantization
distortion in DCT domain. In addition, VSNR is an image quality
metric accounting for spatial distortion. For VQA, the temporal
information is very important and needs to be highlighted. This is596IEEE TRANSACTIONS ON MULTIMEDIA, VOL. 18, NO. 4, APRIL 2016

TABLE IIIPERFORMANCE COMPARISON OF FEPVQ AND
BENCHMARKS ON LIVE VIDEO DATABASE
Algorithm
PSNR
SSIM
MSSIM
VSNR
VIF
WSNR
ESSIM
FSIM
NTIAVQM
MOVIE
L. Xu
[37]
FePVQ
(9)

LCC
SROCC
LCC
SROCC
LCC
SROCC
LCC
SROCC
LCC
SROCC
LCC
SROCC
LCC
SROCC
LCC
SROCC
LCC
SROCC
LCC
SROCC
LCC
SROCC
LCC
SROCC

Wireless

IP

H.264

MPEG-2All Data

0.67680.67050.4732
0.53860.68410.72910.69500.69400.59290.53750.67970.64580.44200.4056
0.72890.74350.77770.76850.83600.81090.76170.75330.83330.80730.47790.4296
0.53710.47410.68380.64490.73640.69300.6364
0.55330.72180.69320.53460.51580.74160.70900.70870.67610.75660.71570.73570.72370.67870.74170.58830.47730.61090.65850.69240.73430.61870.64050.64880.63770.58240.57330.66750.6396
0.69640.69470.72950.73880.79050.76640.70850.66420.89830.87250.40880.39390.58150.56900.63150.68100.67820.58740.67270.6346
0.58530.61540.5671
0.5682
0.66410.69410.86310.8414
0.79690.77330.55500.56370.73480.75130.56900.55270.50250.5326
0.67620.73510.68840.67150.57600.55790.67760.62960.50170.49580.70130.73220.77410.75270.81010.78900.74100.7211
0.83260.8279also the reason why SSIM, MSSIM and VIF perform successfully
in image quality evaluation, but not so well for VQA.
FePVQ outperforms all the competitors, which means that it caneffectively predict the perceptual quality of the distorted videos.
Both motion strength and SS in (9) account for the success of
FePVQ. As compared to our previous work [37], FePVQ is improved due to incorporating the additional SS. From Table III,FePVQ outperforms MOVIE with LCC and SROCC being 0.8324
and 0.8281, respectively. Especially, the LCC and SROCC can be up
to 0.8979 and 0.8728 on H.264/AVC compression, which are muchbetter than all the competitors. Since the video coding, specifically
H.264/AVC, is highly concerned in this work, so the good performance of FePVQ for H.264/AVC compression is indeed desirable.
It should be pointed out that although FePVQ is more competitiveon H.264/AVC compression, it is a general approach for VQA. In
Fig. 3, the scatter-plots after mapping are compared against thedifferent VQA approaches over the LIVE VQ database. It can beobserved that for FePVQ, the sample points scatter more closelyaround the fitted line. It means that the values predicted by FePVQ
correlate better with the subjective ratings, specifically the DMOS
values, demonstrating a better performance than other metrics.
From Table III, the performance of FePVQ is the best on “H.264”
videos (high correlations), and performs less well (not as competitive with benchmarks on the other distortions [“IP,” “Wireless” and“MPEG-2”)]. From Table III, we can see that the performance ofthe proposed FePVQ is not as good as MOVIE on “IP,” “Wireless”
and “MPEG-2.” To explain this observation in depth, we checked
the videos encoded by “Wireless,” “IP,” “H.264” and “MPEG-2,”
and extracted some frames shown in Fig. 4. From Fig. 4, the ”Wireless” and “IP” have the unnatural blockiness, structural edges due
to packet loss and some error concealment techniques, such as
copy the neighor block. These two distortions are usually charactered by unnatural blockiness, structural edges. The free-energy
priciple is more likely applicable to natural texture, structures andobject edges, however it fails to those kinds of unnatural artifacts.

Thus, FePVQ on “Wireless” and “IP” performs not as good as it on“H.264” and “MPEG-2.” Comparing “MPEG-2” with “H.264,” we
can notice that the severe blockiness exists in “MPEG-2,” as shownin Fig. 4(c), while it is not obvious in “H.264” (blur artifact is more
obvious) as shown in Fig. 4(d). This phenomenon is caused by several reasons: 1) ME with variable block size and multiple references
result in better prediction, and thus smaller residual in “H.264,”
which finally reduces blockiness caused by block-based transform
followed by quantization; 2) the smaller size DCT transform in“H.264” (4 × 4 and 8 × 8), “MPEG-2” uses 8 × 8 DCT; 3) improved deblocking filter (loop-filtering) in “H.264.” From Fig. 4(c),blockiness containing a special textural and structural pattern which
however is more likely to be “artificial” but not “natural” that thefree-energy principle accounts for. Thus, FePVQ performs betteron “H.264” than “MPEG-2.”
IVP video database is established by the IVP Laboratory of theChinese University of Hong Kong. The distorted videos are generated by four types of distortions: MPEG-2 compression (MPEG-2),
Dirac wavelet compression (Dirac), H.264 compression (H.264)
and packet loss on the H.264 streaming through IP networks (IP).
We tested the VQ metrics on this database and reported LCC andSROCC statistics in Table IV. From Table IV, it is noticed that
PSNR performs better than SSIM, which is because IVP database
was optimized on PSNR [46]. FePVQ is the best among all the compared metrics on H.264 and MPEG-2 distortions since it is specifically optimized for block based video codings, such as H.264,
MPEG-2 and HEVC. MOVIE performs better than FePVQ on
Dirac distortion, which may be because MOVIE also employs a
sub-bands signal decomposition as the Dirac does. Across all distortion types, it can be observed FePVQ is the best among all thecompared methods.
The CSIQ video database developed in last year is the newest
VQ database. It includes the newest HEVC compression distortion. Besides HEVC, H.264, Motion JPEG compression (MJPEG),
Wavelet-based compression using SNOW codec [47] (SNOW),
H.264 bitstream over wireless channel (Wireless) and additive white
noise (AWN) distortions are included. We also tested FePVQ andthe benchmarks on this database and reported LCC and SROCC
statistics in Table V. From Table V, FePVQ is significantly better
than all the benchmarks on H.264 and HEVC distortions, which is
because FePVQ is specifically designed for video coding optimization. FePVQ is also the best on MJPEG distortion. For Wireless andSnow distortions, MOVIE performs better than FePVQ. MOVIE isdefined on the Gabor sub-bands of signals, which make it more efficient on Snow because Snow also concerns the wavelet sub-bands
decomposition. For AWN distortion, it can be observed that PSNR
and SSIM perform better than others. The statistics across all the
distortion types in Table V shows that FePVQ is the best among all
the compared metrics.

B. Perceptual Video Encoding Optimization
Replacing MSE measurement of conventional encoder by
FePVQ in RDO, perceptual optimization of video encoding can be achieved. We use HM14.0 reference software of
HEVC [49] under the default configuration (IntraPeriod=−1,
MaxCUWidth/Height=64, MaxPartitionDepth=4, QuadtreeTU
Log2MaxSize=5, QuadtreeTULog2MinSize =2, QuadtreeTU
MaxDepthInter/Intra=3, DecodingRefreshType=0, GOPSize=4, FastSearch=1, SearchRange=64, RDOQ=1, RDOQTS=1) toconduct the experiment. Six standard video sequences (see Table VI) of HEVC are tested. They are with plenty of scenes, includ-XU et al.: FEPVQ AND ITS USE IN VIDEO CODING

597

Fig. 3. Scatter plots of the DMOS values versus model predictions on the LIVE VQ database. Each sample point represents one test video. (The circle and“+” indicate the distorted video sequences under wireless network and IP network, respectively. The star indicates H.264/AVC encoded video sequence, while
the triangle indicates the MPEG-2 compressed one.) First row from left to right: (a) PSNR, (b) SSIM, (c) MSSIM, and (d) VSNR; second row from left to right:(e) VIF, (f) NTIA-VQM, (g) MOVIE, and (h) the proposed FePVQ (9).

Fig. 4. Different characteristics of each kind of distortions (all frames are from video sequence “Pedestrian Area”; (a) is the 85th frame from “Wireless” distortion,(b) is the 170th frame from “IP” distortion, and (c) and (d) are the 27th frame from “MPEG-2” and “H.264” distortions, respectively).

ing indoors, outdoors, natural scenery, surveillance and busy street.
Each video sequence has 300 frames, and resolution ranges widely.
The constant QP coding (QP = 24, 28, 32 and 36) is performed inthis experiment.
The R-D comparison is conducted between the original RDO[34] and the proposed one. The original RDO means the traditional
MSE based optimization, and the proposed one employs FePVQ
for RDO. Significant R-D improvement can be achieved by the

proposed RDO over the original one. On average, an improvement
about 0.25 dB VQ and 3.4% bit rate saving can be achieved (seeAppendix). In this case, the R-D improvement is measured by
FePVQ. To avoid the bias that the R-D improvement is just becauseof the different measurements instead of the RDOs, NTIA-VQM[48] is also used to measure R-D performance. The R-D statistics
are listed in Table VII, where the smaller the NTIA-VQM value,
the better the visual quality. From Table VII, the proposed RDO is

598

IEEE TRANSACTIONS ON MULTIMEDIA, VOL. 18, NO. 4, APRIL 2016

TABLE IVPERFORMANCE COMPARISON OF FEPVQ AND
BENCHMARKS ON IVP VIDEO DATABASE
Distortion
Dirac
H.264
MPEG-2
IP
All

LCC
SROCC
LCC
SROCC
LCC
SROCC
LCC
SROCC
LCC
SROCC

TABLE VIIPERFORMANCE COMPARISON BETWEEN THE PROPOSED PERCEPTUAL RDO
AND THE ORIGINAL RDO (“IPPP” ENCODING STRUCTURE, FOR A FAIR
COMPARISON, BOTH OF THE TWO RDOS USE NTIA-VQM FOR
EVALUATING VISUAL QUALITY)

PSNR

SSIM

NTIA-VQM

MOVIE

FePVQ

0.826
0.846
0.771
0.799
0.613
0.700
0.668
0.639
0.682
0.709

0.739
0.789
0.664
0.629
0.608
0.572
0.536
0.511
0.546
0.565

0.865
0.884
0.868
0.864
0.786
0.787
0.535
0.466
0.611
0.643

0.888
0.870
0.823
0.845
0.858
0.842
0.823
0.824
0.880
0.879

0.881
0.867
0.861
0.856
0.873
0.856
0.837
0.819
0.911
0.884

Original

Distortion
H.264
Wireless
MJPEG
Snow
AWN
HEVC
All

LCC
SROCC
LCC
SROCC
LCC
SROCC
LCC
SROCC
LCC
SROCC
LCC
SROCC
LCC
SROCC

PSNR

SSIM

NTIA-VQM

MOVIE

FePVQ

0.802
0.835
0.851
0.802
0.509
0.460
0.759
0.769
0.906
0.949
0.785
0.805
0.579
0.565

0.755
0.717
0.716
0.722
0.734
0.747
0.716
0.733
0.894
0.897
0.523
0.580
0.516
0.558

0.919
0.916
0.801
0.806
0.647
0.641
0.874
0.840
0.884
0.918
0.906
0.915
0.789
0.769

0.897
0.904
0.886
0.882
0.887
0.882
0.900
0.898
0.843
0.855
0.933
0.937
0.806
0.788

0.920
0.912
0.881
0.873
0.896
0.887
0.889
0.875
0.869
0.848
0.943
0.941
0.821
0.810

(kb/s)(dB)(kb/s)

(dB)

(%)

-VQM

447.22221.82
119.4367.79
2062.931148.40706.05452.67
70.60
43.06
28.18
20.06
6598.663312.211709.36894.9228653.02
15797.31
9399.33
5974.74
1838.74
936.03542.64339.090.11970.22020.35790.49550.09110.17810.28970.41780.10870.2056
0.31890.44970.12790.23030.35730.49360.07510.16780.28010.39640.15850.26580.39350.5287433.07214.36114.8964.57
1999.711112.30
684.27438.18
67.71
40.68
26.93
19.33
6399.74
3211.711657.72867.4427792.09
15322.92
9115.775795.30
1783.33
906.81525.34328.910.11610.22020.35270.4927
0.09170.17480.28930.42460.1066
0.19950.31380.44430.12320.22610.35140.48840.0716
0.16650.28090.39780.15010.25730.38490.5124

−3.16
−3.36
−3.80
−4.75
−3.06
−3.14
−3.08
−3.20
−4.09
−5.54
−4.46
−3.66
−3.01
−3.03
−3.02
−3.07
−3.00
−3.00
−3.02
−3.00
−3.01
−3.12
−3.19
−3.00
−3.41

−0.00360.0000−0.0052−0.00280.0007−0.0033−0.0004
0.0068−0.0021−0.0060−0.0051−0.0054−0.0047−0.0042−0.0060−0.0052−0.0035−0.00130.00080.0013−0.0084−0.0085−0.0086−0.0163−0.0038TABLE VI
SEQUENCE INFORMATION
Sequence

Resolution

Frame rate

Flowervase
FourPeople
Mobisode
ParkScene
PeopleOnStreat
Vidyo
Keiba

832 × 480
1920 × 1080
416 × 240
1920 × 1080
2560 × 1600
1280 × 720
832 × 480

30.00
60.00
30.00
24.00
30.00
60.00
30.00

superior to the original one with 3.41% bit rate saving and 0.0038
VQ improvement measured by NTIA-VQM.
We draw the visual perception map {pm } [referring to (12)] in
Fig. 5 for the test sequences, which shows a good agreement between high perception and significant structural edges, boundaries
of motion objects. For mapping {pm } onto original images, we
compute the visual perception map for each pixel by overlapping
blocks size of 8 × 8, so the same size of visual perception map as
original image is given in Fig. 5. It can be observed that:
1) the clear profiles of objects can be successively outlined by
the visual perception maps;
2) for static objects, the visual perception map does not give thehigh visual perception value as the same as that of moving
objects. For example, Fig. 5(b) shows a surveillance video

People-OnStreat

Vidyo

AverageNTIA-VQM

Δ NTIA

Flowervase

ParkScene

Bit rate

Δ Bit
rate

Bit rate

Mobisode

NTIA
-VQM

R-D gain

Sequence

FourPeople

TABLE VPERFORMANCE COMPARISON OF FEPVQ AND
BENCHMARKS ON CSIQ VIDEO DATABASE

Proposedin which a man steps out of elevator. The elevator is static
relative to surveillance camera, so it is with low visual perception; and3) for the natural scenes, the significantly structural edges can be
successfully distinguished from the high texture background,e.g., Fig. 5(c) and (h), the trunk of trees, human bodies, horses
are obvious against the background.
For evaluating the subjective visual quality improvement, we
conducted the subjective evaluation in IVP Lab of CUHK [46].
The evaluation was performed in a studio room with lighting condition conforming to the BT.500 standard [50]. The display monitor
is a 65” Panasonic plasma display (TH-65PF9WK). 15 subjects participated in the subjective test. All of them are non-experts. Theireyesight was either normal or had been corrected to be normal
with spectacles. Each observer assessed 6 source videos and 48
distorted videos. A single-stimulate method [51] with absolute category rating (ACR) scale was used where each video (including thereference) occurred once in a random order, and yet the two successive videos come from different source videos so as to remove
contextual and memory effects in quality evaluation. The ACR scale
employs a five-category discrete quality judgment, specifically 5excellent, 4-good, 3-fair, 2-poor, and 1-bad. At the beginning of thetest, three videos were arranged as the training videos to stabilize
the observers’ opinion. Subjective rating of the compressed video
was subtracted from that of the reference video. The difference
values were processed using the method described in the BT.500
standard [50] to derive the DMOS.

XU et al.: FEPVQ AND ITS USE IN VIDEO CODING

Fig. 5.

599

Visual perception maps [(a), (d) Mobisode; (b), (e) ParkScene; (c), (f) Keiba; all frames are the 11th frame coded by “P”].
TABLE VIIICOMPUTING TIME OF FEPVQ AND THE BENCHMARKS

MetricAverageprocessing
time (insecond)Fig. 6. Subjective quality comparison between the proposed perceptual RDO
and the original one.

Five uncompressed, high quality source videos of natural scenes
are used to create the distorted videos using four different QPs (QP= 24, 28, 32 and 36) for the standard RDO of HEVC and the proposed perceptual RDO, respectively. All videos contain 300 frames.
The subjective experiments are performed to prove the better subjective visual quality of the proposed RDO over the traditional one.
The subjective evaluation result is shown in Fig. 6. The horizontal
axis indicates the encoded video sequences in different bit-rates.
The red histograms are the DMOS values of video sequences generated by the proposed RDO by employing FePVQ, while the blue
ones are the DMOS values from the original RDO of HEVC. The
green error bar indicates the variance of the subjective rating scoresfor each encoded sequence. The smaller DMOS value indicatesthe better visual quality of the encoded video sequence. It can beobserved that the proposed RDO can outperform the other one on
most of the test sequences.

MSNTIAFePVQPSNR SSIM SSIM VSNR VQM ESSIM FSIM VIF MOVIE
(9)
4

24

60

26

57

64

84

636

6320

45

SSIM, MS-SSIM, VSNR, VIF, NTIA-VQM, ESSIM, FSIM,
MOVIE and FePVQ is given in Table VI. All codes are implemented using MATLAB (without specific optimization) except for
MOVIE which uses C++. In Table VIII, the time complexity ismeasured in second and the value represent the average time pereach video sequence.
From Table VIII, FePVQ [(9)] is superior to MOVIE,
VIF, MS-SSIM, ESSIM, FSIM and NTIA-VQM in terms of computational complexity. In addition, it would be better when applied
to video coding, since the MSEs of MBs have been computed in
RDO stage of video encoding. Furthermore, FePVQ builds a relation between perceptual visual quality and quantization of video
compression in order to guide the perceptual video encoding besides assessing visual quality. The complexity of VIF and MOVIE
majorly comes from their subbands decomposition. VIF uses overcomplete steerable pyramid decomposition. MOVIE uses 3-D Gabor filters to decompose the video locally into more than 100 spatiotemporal channels. The complexity makes their uses in video coding
impractical.

V. CONCLUSION
C. Computational Complexity Analysis
Regarding the computational complexity of FePVQ, for a
250-frame 768 × 432 sequence (LIVE video Database) on a
3G Hz quad-core CPU with 6G RAM, processing time of PSNR,

Although much research has been done in IQA and VQA, it
is important to derive a well-grounded perceptual measurement
suitable for video coding. In this paper, a novel VQA metric, FePVQ
is proposed for VQA. The free energy principle in neuroscience is600IEEE TRANSACTIONS ON MULTIMEDIA, VOL. 18, NO. 4, APRIL 2016

first formulated into IQA for better representing structural edges ofvideo frames. Second, the RTV accounting for separating orderly
and disorderly regions is employed to implement the free-energy
principle due to its low computational complexity. Thirdly, the
HVSP accounts for the visual speed perception of video signal.
Since FePVQ is closely related with MSE, it can be easily integrated
into a RDO process of hybrid video coding for the purpose ofperceptual video coding optimization. The remarkable merit ofthis work over the previous algorithms is to introduce free-energy
principle and HVSP into VQA, contributing a better representationof orderly and disorderly signals in visual scenes, and therefore abetter performance.

REFERENCES[1] C. van den Branden Lambrecht and O. Verscheure, “Perceptual quality
measure using a spatio-temporal model of the human visual system,” inProc. SPIE, vol. 2668, 1996, pp. 450–461.
[2] S. Winkler, “A perceptual distortion metric for digital color video,” inProc. SPIE, Jan. 1999, pp. 175–184.
[3] J. Lubin, “The use of psychophysical data and models in the analysis of display system performance,” in Digital Images Human Vision,
A. B. Watson, Ed. Cambridge, MA, USA: MIT Press, 1993, pp. 163–
178.
[4] A. Watson, J. Hu, and J. McGowan, “Digital video quality metric basedon human vision,” J. Electron. Imag., vol. 10, no. 1, pp. 20–29, Jan.
2001.
[5] M. Masry, S. S. Hemami, and Y. Sermadevi, “A scalable wavelet-based
video distortion metric and applications,” IEEE Trans. Circuits Syst. Video
Technol., vol. 16, no. 2, pp. 260–273, Feb. 2006.
[6] J. You, T. Ebrahimi, and A. Perkis, “Modeling motion visual perception
for video quality assessment,” in Proc. ACM Multimedia, 2011, pp. 1293–
1296.
[7] J. You, T. Ebrahimi, and A. Perkis, “Attention driven foveated video
quality assessment,” IEEE Trans. Image Process., vol. 23, no. 1, pp. 200–
213, Jan. 2014.
[8] Z. K. Lu, W. Lin, X. K. Yang, E. P. Ong, and S. S. Yao, “Modelingvisual attention’s modulatory afterreffects on visual sensitivity and quality evaluation,” IEEE Trans. Image Process., vol. 14, pp. 1928–1942,
Nov. 2005.
[9] Z. Wang, H. R. Sheikh, A. C. Bovik, and E. P. Simoncelli, “Image quality assessment: From error visibility to structural similarity,” IEEE Trans. Image Process., vol. 13, no. 4, pp. 600–612, Apr.
2004.
[10] Z. Wang, E. P. Simoncelli, and A. C. Bovik, “Multi-scale structural similarity for image quality assessment,” in Proc. 37th IEEE Asilomar Conf.
Signals, Syst. Comput., Nov. 2003, pp. 1398–1402.
[11] A. K. Moorthy and A. C. Bovik, “Efficient video quality assessment along
temporal trajectories,” IEEE Trans. Circuit Syst. Video Technol., vol. 20,
no. 11, pp. 1653–1753, Nov. 2010.
[12] M. Narwaria, W. S. Lin, and A. M. Liu, “Low-complexity video quality
assessment using temporal quality variations,” IEEE Trans. Multimedia,
vol. 14, no. 3, pp. 525–535, Jun. 2012.
[13] B. Gu, V. S. Sheng, K. Y. Tay, W. Romano, and S. Li, “Incremental
support vector learning for ordinal regression,” IEEE Trans. Neural Netw.
Learning Syst., vol. 26, no. 7, pp. 1403–1416, Jul. 2015.
[14] B. Gu, V. S. Sheng, Z. Wang, D. Ho, S. Osman, and S. Li, “Incremental
learning for γ-support vector regression,” Neural Netw., vol. 67, pp. 140–
150, 2015.
[15] K. Friston, J. Kilner, and L. Harrison, “A free energy principle for thebrain,” J. Physiol., vol. 100, nos. 1–3, pp. 70–87, 2006.
[16] K. Friston, “The free-energy principle: A unified brain theory?,” Nature
Rev. Neurosci., vol. 11, no. 2, pp. 127–138, 2010.
[17] J. Wu, G. Shi, W. Lin, A. Liu, and F. Qi, “Just noticeable difference estimation for images with free-energy principle,” IEEE Trans. Multimedia,
vol. 15, no. 7, pp. 1705–1710, Nov. 2013.
[18] G. T. Zhai, X. L. Wu, X. K. Yang, W. S. Lin, and W. J Zhang, “A
psychovisual quality metric in free-energy principle,” IEEE Trans. Image
Process., vol. 21, no. 1, pp. 41–52, Jan. 2012.
[19] D. C. Knill and R. Pouget, “The Bayesian brain: The role of uncertainty inneural coding and computation,” Trends Neurosci., vol. 27, pp. 712–719,
2004.

[20] Z. Q. Pan, Y. Zhang, and S. Kwong, “Efficient motion and disparity estimation optimization for low complexity multiview video coding,” IEEE
Trans. Broadcast., vol. 61, no. 2, pp. 166–176, Jun. 2015.
[21] A. A. Stocker and E. P. Simoncelli, “Noise characteristics and prior expectations in human visual speed perception,” Nature Neurosci., vol. 9,
pp. 578–585, 2006.
[22] Z. Wang and Q. Li, “Video quality assessment using a statistical modelof human visual speed perception,” J. Opt. Soc. Amer. A, Opt. Image Sci.
Vis., vol. 24, no. 12, pp. B61–B69, Dec. 2007.
[23] E. P. Simoncelli and B. Olshausen, “Natural image statistics and neuralrepresentation,” Ann. Rev. Neurosci., vol. 24, pp. 1193–1216, 2001.
[24] L. Xu, Q. Yan, Y. Xia, and J. Y. Jia, “Structure extraction from texture via
relative total variation,” ACM Trans. Graphics, vol. 32, no. 6, Nov. 2012,
Art. no. 139.
[25] A. Bhat, S. Kannangara, Y. F. Zhao, and I. Richardson, “A full reference
quality metric for compressed video based on mean squared error andvideo content,” IEEE Trans. Circuits Syst. Video Technol., vol. 22, no. 2,
pp. 165–173, Feb. 2012.
[26] Y.-F. Ou, Z. Ma, and Y. Wang, “Perceptual quality assessment of video
considering both frame rate and quantization artifacts,” IEEE Trans. Circuits Syst. Video Technol., vol. 20, no. 3, pp. 286–298, Mar. 2010.
[27] Z. Wang, L. Lu, and A. C. Bovik, “Foveation scalable video coding with
automatic fixation selection,” IEEE Trans. Image Process., vol. 12, no. 12,
pp. 243–254, Feb. 2003.
[28] M. E. J. Newman, “Power laws, Pareto distributions and Zipf’s law,”
Contemporary Phys., vol. 46, no. 5, pp. 323–352, Sep. 2005.
[29] A. Clauset, C. R. Shalizi, and M. E. J. Newman, “Power-law distributions
in empirical data,” SIAM Rev., vol. 51, no. 4, pp. 661–703, 2009.
[30] L. Xu, X. Y. Ji, W. Gao, and D. B. Zhao, “Laplacian distortion model(LDM) for rate control in video coding,” in Proc. Multimedia 8th Pacific
Rim Conf. Adv. Multimedia Inf. Process., Dec. 2007, pp. 638–646.
[31] G. J. Sullivan and T. Wiegand, “Rate-distortion optimization for video
compression,” IEEE Signal Process. Mag., vol. 15, no. 6, pp. 74–90,
Nov. 1998.
[32] L. Xu, D. Zhao, X. Ji, L. Deng, S. Kwong, and W. Gao, “Window levelrate control for smooth visual quality and smooth buffer occupancy,” IEEE
Trans. Image Process., vol. 20, no. 3, pp. 723–734, Mar. 2011.
[33] S. Q. Wang, A. Rehman, Z. Wang, S. W. Ma, and W. Gao, “SSIMmotivated rate-distortion optimization for video coding,” IEEE Trans.
Circuits Syst. Video Techn. vol. 22, no. 4, pp. 516–529, Apr. 2012.
[34] L. Xu, S. N. Li, K. N Ngan, and L. Ma, “Consistent visual quality control
in video coding,” IEEE Trans. Circuits Syst. Video Technol., vol. 23,
no. 6, pp. 975–989, Jun. 2013.
[35] The Video Quality Experts Group, “Final report from the video
quality experts group on the validation of objective quality metrics
for video quality assessment,” 2000. [Online]. Available: http://www.
its.bldrdoc.gov/vqeg/projects/frtv phaseI[36] H. R. Sheikh, Z. Wang, L. Cormack and A. C. Bovik, “LIVE image quality assessment database release 2,” 2005. [Online]. Available:
http://live.ece.utexas.edu/research/quality[37] N. Ponomarenko et al., “TID2008-A database for evaluation of fullreference visual quality assessment metrics,” Adv. Modern Radioelectron.,
vol. 10, pp. 30–45, 2009.
[38] E. C. Larson and D. M. Chandler, “Most apparent distortion: Full-reference
image quality assessment and the role of strategy,” J. Electron. Imag.,vol. 19, no. 1, Mar. 2010, Art. no. 011006.
[39] H. R. Sheikh and A. C. Bovik, “Image information and visual quality,”IEEE Trans. Image Process., vol. 15, no. 2, pp. 430–444, Feb. 2006.
[40] L. Zhang, L. Zhang, X. Mou, and D. Zhang, “FSIM: A feature similarity
index for image quality assessment,” IEEE Trans. Image Process., vol. 20,
no. 8, pp. 2378–2386, Aug. 2011.
[41] X. D. Zhang, X. C. Feng, W. W. Wang, and W. F. Xue, “Edge strength
similarity for image quality assessment,” IEEE Signal Process. Lett.
vol. 20, no. 4, pp. 319–322, Apr. 2013.
[42] K. Seshadrinathan, R. Soundararajan, A. C. Bovik and L. K. Cormack, “LIVE video quality database,” 2009. [Online]. Available:
http://live.ece.utexas.edu/research/quality/live_video.html[43] F. Zhang, S. Li, L. Ma, and K. N. Ngan, “IVP video quality
database,” 2011. [Online]. Available: http://ivp.ee.cuhk.edu.hk/research/
database/subjective/index.shtml.
[44] P. V. Vu and D. M. Chandler, “ViS3: An algorithm for video quality
assessment via analysis of spatial and spatiotemporal slices,” J. Electron.
Imag., vol. 23, no. 1, pp. 13–16, Feb. 2014.
[45] K. Seshadrinathan, R. Soundararajan, A. C. Bovik, and L. K. Cormack,“Study of subjective and objective quality assessment of video,” IEEE
Trans. Image Process., vol. 19, no. 6, pp. 1427–1441, Jun. 2010.

XU et al.: FEPVQ AND ITS USE IN VIDEO CODING[46] D. M. Chandler and S. S. Hemami, “VSNR: A wavelet-based visual signalto-noise ratio for natural images,” IEEE Trans. Image Process., vol. 16,
no. 9, pp. 2284–2298, Sep. 2007.
[47] F. Bellard et al., “FFMPEG tool,” Nov. 15, 2012. [Online]. Available:
http://www.ffmpeg.org[48] M. Pinson and S. Wolf, “A new standardized method for objectively
measuring video quality,” IEEE Trans. Broadcast., vol. 50, no. 3,
pp. 312–322, Sep. 2004.
[49] G. J. Sullivan, J.-R. Ohm, W.-J. Han, and T. Wiegand, “Overview of thehigh efficiency video coding (HEVC) standard,” IEEE Trans. Circuits
Syst. Video Technol., vol. 22, no. 12, pp. 1649–1668, Dec. 2012.
[50] “Methodology for the subjective assessment of the quality of television
pictures,” ITU-R Rec. BT.500-11, ITU, Geneva, Switzerland, 2002.
[51] “Subjective video quality assessment methods for multimedia applications,” ITU-T Rec. P.910, ITU, Geneva, Switzerland, 2008.

Long Xu (M’13) received the M.S. degree in appliedmathematics from Xidian University, Xi’an, China,
in 2002, and the Ph.D. degree from the Institute ofComputing Technology, Chinese Academy of Sciences, Beijing, China.
He was a Postdoc with the Department of Computer Science, City University of Hong Kong, Hong
Kong, China, and the Department of Electronic Engineering, Chinese University of Hong Kong, Hong
Kong, China, and from July 2009 to December 2012.
From January 2013 to March 2014, he was a Postdoc
with the School of Computer Engineering, Nanyang Technological University,
Singapore. He is currently with the Key Laboratory of Solar Activity, National
Astronomical Observatories, Chinese Academy of Sciences. His research interests include image/video processing, solar radio astronomy, wavelet, machinelearning, and computer vision.
Dr. Xu selected into the 100-Talents Plan, Chinese Academy of Sciences,
2014.

Weisi Lin (S’91–M’92–SM’00–F’16) received the
Ph.D. degree from King’s College, London University, London, U.K.
He was previously the Lab Head of Visual Processing, Institute for Infocomm Research, Singapore. He
is currently an Associate Professor with the Schoolof Computer Engineering, Institute for Infocomm Research. His research interests include image processing, perceptual signal modeling, video compression,and multimedia communication, in which he has authored or coauthored more than 130 journal papers
and 200 conference papers, filed seven patents, and authored two books.
Prof. Lin is a Fellow of the IET. He is an Associate Editor for the IEEE
TRANSACTION IMAGE PROCESSING, the IEEE SIGNAL PROCESSING LETTERS,
and the Journal of Visual Communication and Image Representation, and a past
Associate Editor for the IEEE TRANSACTION MULTIMEDIA. He has also beena Guest Editor for eight special issues in international journals. He has been a
Technical Program Chair for IEEE ICME 2013, PCM 2012, and QoMEX 2014.
He chaired the IEEE MMTC Special Interest Group on QoE (2012–2014). He
has been an invited/panelist/keynote/tutorial speaker in more than ten international conferences, as well as a Distinguished Lecturer of Asia-Pacific Signal
and Information Processing Association (2012–2013).

601

Lin Ma (M’13) received the B.E. and M.E. degrees
from the Harbin Institute of Technology, Harbin,
China, in 2006 and 2008, respectively, and the Ph.D.
degree from the Department of Electronic Engineering, Chinese University of Hong Kong, Hong Kong,
China, in 2013.
He is currently a Researcher with Huawei Noah’s
Ark Lab, Hong Kong, China. He was a Research Intern with Microsoft Research Asia, Beijing, China,
from October 2007 to March 2008. He was a Visiting
Student with the School of Computer Engineering,
Nanyang Technological University, Singapore, from July 2011 to September
2011. His research interests include the areas of deep learning and multimodal
learning, specifically for image and language, image/video processing, and quality assessment.
Yongbing Zhang (M’13) received the B.A. degree
in english and the M.S. and Ph.D. degrees in computer science from the Harbin Institute of Technology, Harbin, China, in 2004, 2006, and 2010,
respectively.
He is currently an Associate Professor with the
Graduate School at Shenzhen, Tsinghua University,
Shenzhen, China. His current research interests include video processing, image and video coding,
video streaming, and transmission.
Prof. Zhang was the recipient of the Best Student
Paper Award at IEEE International Conference on Visual Communication andImage Processing in 2015.
Yuming Fang (M’13) received the B.E. degree from
Sichuan University, Chengdu, China, the M.S. degree from Beijing University of Technology, Beijing,
China, and the Ph.D. degree in computer engineeringfrom Nanyang Technological University, Singapore.
He is currently an Associate Professor with the
School of Information Technology, Jiangxi University of Finance and Economics, Nanchang, China. He
was also a Visiting Postdoc Research Fellow with the
IRCCyN laboratory, PolyTech’ Nantes & University
Nantes, Nantes, France, the University of Waterloo,
Waterloo, ON, Canada, and Nanyang Technological University, Singapore. His
research interests include visual attention modeling, visual quality assessment,
image retargeting, computer vision, and 3-D image/video processing.
Prof. Fang was a Secretary of the Ninth Joint Conference on Harmonious
Human Machine Environment (HHME2013). He was also a Special Session
Organizer at VCIP 2013 and QoMEX 2014.
King Ngi Ngan (M’79–SM’91–F’00) received the
Ph.D. degree in electrical engineering from theLoughborough University, Loughborough, U.K.
He is currently a Chair Professor with the Department of Electronic Engineering, Chinese University of Hong Kong, Hong Kong, China. He waspreviously a Full Professor with the Nanyang Technological University, Singapore, and the Universityof Western Australia, Crawly, WA, Australia. Since
2012, he has been a Chair Professor at the University of Electronic Science and Technology, Chengdu,
China, under the National Thousand Talents Program. He holds honorary andvisiting professorships at numerous universities in China, Australia, and South
East Asia. He has published extensively, including three authored books, sevenedited volumes, and more than 370 refereed technical papers, and he has editednine special issues in journals. He holds 15 patents in the areas of image/videocoding and communications.
Prof. Ngan is a Fellow of the IET (U.K.), and IEAust (Australia), and an
IEEE Distinguished Lecturer for 2006–2007. He was an Associate Editor of theIEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, theJournal on Visual Communications and Image Representation, the EURASIP
Journal of Signal Processing: Image Communication, and the Journal of Applied Signal Processing. He chaired and co-chaired a number of prestigious
international conferences on image and video processing including the 2010
IEEE International Conference on Image Processing, and served on the advisory and technical committees of numerous professional organizations.

602

Songnan Li (S’10–M’12) received the B.S. and M.P.
degrees in computer science and technology fromthe Harbin Institute of Technology, Beijing, China,
in 2004 and 2006, respectively, and the Ph.D. degreefrom the Department of Electronic Engineering, Chinese University of Hong Kong (CUHK), Hong Kong,
China, in 2012.
He joined the CUHK as a Research Assistant in2007. From 2012 to 2014, he was a Postdoc with the
Department of Electronic Engineering, CUHK. He iscurrently a Research Assistant Professor in the samedepartment. His research interests include image and video processing, RGBD
computer vision, and visual quality assessment.

IEEE TRANSACTIONS ON MULTIMEDIA, VOL. 18, NO. 4, APRIL 2016

Yihua Yan received the B.E and M.E. degrees
from Northwestern Polytechnical University, Xi’an,
China, in 1982 and 1985, respectively, and the Ph.D.
degree from the Dalian University of Technology,
Dalian, China, in 1990.
He is currently a Professor and Chief Scientist of
Solar Radio Research and the Director of the Key
Laboratory of Solar Activity and the Solar Physics
Division, National Astronomical Observatories, Science Academy of Sciences, Beijing, China. He wasa Foreign Research Fellow. He is currently the President of the International Astronomical Union Division E: Sun & Heliosphere,
Science Academy of Sciences, from 2015 to 2018 with the National Astronomical Observatory of Japan, Tokyo, Japan, from 1995 to 1996, and an Alexandervon Humboldt Fellow with the Astronomical Institute, Wurzburg University,
Wurzburg, Germany, from 1996 to 1997. His research interests include solar
magnetic fields, solar radio astronomy, space solar physics, and radio astronomical methods.