IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 25, NO. 7, JULY 2016

2943

Low-Delay Rate Control for Consistent Quality
Using Distortion-Based Lagrange Multiplier
Miaohui Wang, Member, IEEE, King Ngi Ngan, Fellow, IEEE, and Hongliang Li, Senior Member, IEEE

Abstract— Video quality fluctuation plays a significant rolein human visual perception, and hence, many rate control
approaches have been widely developed to maintain consistentquality for video communication. This paper presents a novel
rate control framework based on the Lagrange multiplier inhigh-efficiency video coding. With the assumption of constantquality control, a new relationship between the distortion andthe Lagrange multiplier is established. Based on the proposed
distortion model and buffer status, we obtain a computationallyfeasible solution to the problem of minimizing the distortion
variation across video frames at the coding tree unit level.
Extensive simulation results show that our method outperforms
the rate control used in HEVC Test Model (HM) by providing
a more accurate rate regulation, lower video quality fluctuation,and stabler buffer fullness. The average peak signal-to-noise ratio(PSNR) and PSNR deviation improvements are about 0.37 dB
and 57.14% in the low-delay (P and B) video communication,
where the complexity overhead is ∼4.44%.
Index Terms— Rate control, distortion model, Lagrange
multiplier, HEVC.

I. I NTRODUCTION
ITH the rapid development of visual communicationin recent years, efficient video compression techniques
have been considered by the multimedia community. To meetthis demand, ITU-T/SG16/Q6 (VCEG) and ISO/IEC
JTC1/SC29/WG11 (MPEG) have established the Joint
Collaborative Team on Video Coding (JCTVC) to develop
the latest video coding standard – High Efficiency Video
Coding (HEVC) [1]. As HEVC involves many sophisticatedcoding features and techniques, it has significantly improved
the compression performance in comparison with the previous
video coding standards, such as MPEG-2, H.263, MPEG-4
and H.264/Advanced Video Coding (AVC) [2]. For example,
HEVC supports diverse block sizes, flexible quad-tree

W

Manuscript received December 28, 2014; revised May 13, 2015,
January 10, 2016, and February 22, 2016; accepted April 5, 2016. Dateof publication April 11, 2016; date of current version May 11, 2016. This
work was supported in part by the Research Grants Council, Hong Kong,
under Grant CUHK415712, in part by the National Natural Science
Foundation of China under Grant 61525102, and in part by the Programfor Science and Technology Innovative Research Team for Young Scholars
in Sichuan Province, China, under Grant 2014TD0006. The associate editor
coordinating the review of this manuscript and approving it for publication
was Prof. Ricardo L. De Queiroz.
M. Wang and K. N. Ngan are with the Department of Electronic
Engineering, The Chinese University of Hong Kong, Hong Kong (e-mail:wang.miaohui@gmail.com; knngan@ee.cuhk.edu.hk).
H. Li is with the School of Electronic Engineering, University ofElectronic Science and Technology of China, Chengdu 610051, China (e-mail:hlli@uestc.edu.cn).
Color versions of one or more of the figures in this paper are available
online at http://ieeexplore.ieee.org.
Digital Object Identifier 10.1109/TIP.2016.2552646

structure and efficient filters. However, it is worth notingthat the above video standards only specify the syntax of
decoding the bitstream. Meanwhile, the compressed video
qualities depend considerably on implementing the rate
control scheme on the encoder side. Thus, computational
rate control algorithms are widely investigated after theinternational video standards have been developed.
Normally, various video applications, like video broadcasting and video surveillance, are transmitted via a constant
bit rate (CBR) channel. To maintain a short-term constant
output bit rate in the CBR channel, traditional codecs
adopt a uniform bandwidth allocation scheme in a groupof pictures (GOP). However, the number of encoding bitschanges from frame to frame owing to the time-varying video
complexity. In this case, it is infeasible to adjust encoding
parameters to achieve the exact fixed bandwidth for each frame
in a GOP. Consequently, an associate encoder buffer is usuallyemployed to regulate the output bits before transmitting. If thechannel bandwidth is less than the output bit rate, the encoded
bits will accumulate in the encoder buffer. When the size ofthe accumulated bits is too large, the encoder needs to skip
some frames to alleviate the buffer delay and avoid the buffer
overflow. Conversely, if the channel bandwidth is larger thanthe output bit rate, this indicates that some channel is wasted,
and it may cause buffer underflow. Since the buffer overflow
and underflow result in undesirable effects on the video quality
fluctuation, it is essential to control the bit rate to maintaina consistent quality over the entire video sequence.
In real-time video communications, rate control becomes
more challenging as it needs to satisfy low-latency of transmitting video data. In such a case, the encoder buffer must
maintain a very small size, and therefore, the encoder requiresa more accurate bit allocation and coding settings to reduce
the fluctuations of buffer fullness as well as to avoid theundesirable buffer overflow and underflow. The conventional
rate control schemes usually involve two steps: (1) thetarget bit is allocated to each basic unit according to its
relative complexity and the buffer status, and (2) quantization
parameter (QP) is computed by rate-quantization (R-Q)
models, such as quadratic model [3] and ρ-domain model [4].
It should be pointed out that with traditional R-Q models, QP
can determine the bits for residue information (i.e., quantizedtransformed coefficient (QTC)) but not for non-residue
information (i.e., partition mode, motion and other header
information). Although the overhead bits can be predictedfrom the previous frames, the accuracy of prediction isstill not well addressed in both H.264/AVC and HEVC

1057-7149 © 2016 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

2944

owing to time-varying video complexity. On the other hand,similar to H.264/AVC, the reference software of HEVC
utilizes rate distortion optimization (RDO) to determine theencoding settings so as to obtain the best rate-distortion (R-D)
performance. To solve this RDO problem, the Lagrangian
method is used to achieve the optimal trade-off between
rate and distortion on the encoder side. Additionally, it canbe seen that the Lagrange multiplier affects the total bits,
including both the residue and non-residue bits. In the
HEVC reference software, simulation results indicate that theLagrange multiplier is more sensitive, simple and efficient incontrolling the output target bits in comparison with QP, andnot surprisingly, an interesting relationship between the target
bits and Lagrange multiplier λ (also called Rate-lambda orR-λ model) was developed for HEVC in JCTVC-K0103 [5].
In the R-λ model, λ is first determined by the target
bits. Then, QP is computed by a logarithmic function of λin the HEVC reference software. Although the R-λ method
has shown a significant improvement in comparison withconventional methods, there are two main difficulties.
• Inaccurate bit allocation: The number of bits for a coding
tree unit (CTU) relies heavily on the frame budget,the previous overhead bits and the weights of CTUs
in the current frame, which is explicitly determined
one-by-one in a raster-scanning order. In practice, theframe bandwidth can be easily consumed by the first
several CTUs due to inaccurate estimation of the model
parameters. Hence, the buffer state greatly affects thetarget bits for the subsequent CTUs in a frame, and theraster-scanning bit allocation scheme can cause inaccurate
bit rates and adverse effects on the overall quality control.
• Inaccurate λ estimation: As mentioned earlier, since
there is the bit allocation problem, λ adjustment isfrequently applied for CTUs to achieve the frame budget.
Specifically, an extremely large (or small) λ is usually
computed according to the target bits and buffer fullness.
To maintain λ in a reasonable range, many “bound” and“crop” operations are employed on the encoder side.
For example, the HEVC reference software allows the
maximum λ variation in terms of QP up to 10 between
two successive frames which results in a large quality
fluctuation. Thus, inaccurate λ greatly degrades the final
rate control result.
Although some improvements [6], [7] in the R-λ models
have been reported recently, the two above-mentioned
limitations are still not being addressed satisfactorily. This
paper proposes a new distortion based Lagrange multipliermethod to improve the HEVC rate control in low-delay
communication. The major innovations of this paper can besummarized in three aspects. Firstly, with the assumptionof consistent video quality coding, a new relationshipbetween the distortion and λ is established, which canbe used to control the video quality fluctuations via λ.
Secondly, a computationally feasible solution to the problemof minimizing distortion across the video frames at the CTUlevel is obtained based on the above distortion model, which
can avoid the raster-scanning bit allocation. Thirdly, according
to buffer fullness, the CTU level λ is adjusted such that theIEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 25, NO. 7, JULY 2016

target bits satisfy the overall bandwidth in low-delay videocommunications.
The rest of the paper is organized as follows. We introduce
the related work in Section II. The new distortion modeland the associated rate model are described in Section III.
The proposed rate control algorithm is then implemented in
Section IV. The simulation results are presented in Section V,
and Section VI contains the concluding remarks.
II. R ELATED W ORK IN HEVC
Recently, many rate control methods have been consideredin the HEVC video coding system. In this section, we givea brief review of the different rate models to facilitate thestudy of the rate control algorithms. In HEVC, the rate control
models can be roughly categorized into R-Q and R-λ models.
A. R-Q Models
Choi et al. proposed a pixel-wise unified R-Q
model (URQ) [8] that is a direct extension of the quadratic
R-Q rate control method in the early HEVC reference
software (HM6.0). The quadratic R-Q model is first developed
for hybrid video coding by Chiang and Zhang [3] which
has been widely studied in H.264/AVC [9]. Similar to thequadratic R-Q method in H.264/AVC, in URQ, the target bit
is allocated based on the mean absolute difference (MAD) based complexity, and the quantization step is computed on
the encoder side. A detailed description of URQ can be foundin their recent work [10]. Although URQ shows a better
performance to control the output bits than the anchor HM6.0,
two major limitations of the quadratic model are not avoided.
Firstly, the quadratic model only determines the bits for the
residue information but not for the non-residue information.
Secondly, the quadratic model contains an inter-dependency
relationship between the RDO process and the computationof quantization step, which results in a well-known “chicken
and egg” dilemma [11].
Additionally, a general R-Q model is studied in HEVC,
which is based on the number of QTCs. He and Mitra [4]proposed the ρ-domain rate model for simple block structure
based video coding, where ρ is the percentage of zeros among
QTCs. The ρ-domain model is based on the observationthat there is an approximated linear relationship between
bits and ρ in a single transform block size. Recently,
Wang et al. developed a quadratic ρ-domain based Rate-GOP
method [12] in HEVC. With Rate-GOP algorithm, QP can firstbe determined by the picture order count (POC), and then thebits of non-zero QTCs are simulated by a quadratic functionof quantization step. Although simulation results have shown
its superiority compared to traditional rate control methods,
further investigation is necessary to verify the ρ-domain modelin HEVC. Specifically, in [13] and [14], experiments show
the facts that signal characteristics with respect to the coding
unit depth levels are considerably different. In addition, thevariances of the transform coefficients of intra frames are
significantly different from that of inter frames. Thus, in the
HEVC encoder, the ρ-domain methods should be modeled
separately due to the depth of the coding units and the types of

WANG et al.: LOW-DELAY RATE CONTROL FOR CONSISTENT QUALITY USING DISTORTION-BASED LAGRANGE MULTIPLIER

encoded frames. Unfortunately, this is still not well addressedin the Rate-GOP method.

2945

that both R and D are differentiable everywhere, λ can be
expressed by
∂D
.
(3)
∂R
As shown in equation (2), R and D for a given encoding
block or frame depend on the scaling factor λ. Conventionally,
λ is obtained from the input QP [21]. The relationship between
R-Q and D-Q (i.e., distortion-quantization) models has beenstudied and used for rate control in H.263, H.264/AVC and
MPEG-4 codecs. However, it is interesting to note that QP only
determines the residue bits, but λ determines the overall ratecost, including both residue and non-residue bits. Furthermore,λ also affects the encoding mode selection, such as partitions
and motions. Thus, in this section, the essential goal is tomodel the effect of λ on distortion and rate. We concentrateon modeling distortion and rate for motion-compensatedprediction (MCP) frames. It should be noted that similar
models can be established for intra frames, but the related
models will not be derived here because the intra frames are
not frequently used in low-delay video communications.
λ=−B. R-λ Models
To avoid the aforementioned problems, the Lagrange
multiplier based rate control methods have been studiedin H.264/AVC and HEVC. With the RDO scheme on theencoder side, larger λ is related to larger distortion andfewer bit, and the converse holds for smaller λ. Furthermore,since QP adjustments will cause additional overhead bits,
λ is used to adjust the rate and distortion during theactual encoding stage. Taking into account this observation,
Jiang and Ling [15], [16] proposed to adjust λ adaptively
according to the rate cost. In [17], Wang and Yan establisheda new relationship between λ and MAD. Since the MAD has
been integrated into the quadratic R-Q model in H.264/AVC,
it indicates that the relationship between rate and λ has been
implicitly established.
Nowadays, Li et al. [5] (see a detailed description in [18])developed a new relationship between rate and λ. Compared to
the R-Q method, the R-λ model considers the overall bit rate,
including both the residue and non-residue bits. Additionally,as the R-λ method outperforms the URQ method in terms ofbit estimation accuracy and video quality control, it has been
recommended for HEVC by the JCTVC. However, it should
be pointed out that the R-λ model only considers the target
bit but ignores the characteristics of video data in framelevel rate control. Thus, the frame-content complexity has
been considered to improve the R-λ model. For example,
Wang and Karczewicz [6] proposed to use summation ofabsolute transformed differences (SATD) to measure thecomplexity of an intra frame. In [7], Wang and Ngan proposeda gradient based R-lambda (GRL) model for intra frame
rate control. Simulation results show that the SATD andgradient can be used to effectively measure the frame-content
complexity and enhance the performance of intra-frame rate
control.
III. M ODELING R ATE AND D ISTORTION
IN H IGH E FFICIENCY V IDEO C ODING
Rate-distortion optimization has been widely studied inblock-based video coding systems, such as in the reference
software of H.264/AVC and HEVC. The R-D relationshipindicates that the higher the rate R is, the lower is distortion D,
and vice versa. Thus, the fundamental problem in rate control
is to minimize the distortion subject to a given rate constraint
Rmax [19].
min D, s.t. R ≤ Rmax ,(1)

Usually, this constrained problem can be solved by the
Lagrangian optimization method [20] in hybrid video coding.
The Lagrangian cost function is
J = D + λ × (R − Rmax ) ,(2)

where λ is the Lagrange multiplier. Additionally, it is notedthat the R-D curve is convex in video coding, and if we assume

A. Distortion ModelingIn the past decades, many R-D models [22]–[24] have been
proposed for rate control. For example, in H.264/AVC and
MPEG-4, the R-D curves can be modeled by a logarithmic
expression. Specifically, this logarithmic model has been
proved assuming a high-rate environment (e.g., above
0.5 bits/pixel) [25]. However, in the HEVC reference software,
since the encoding efficiency is greatly improved, thetraditional R-D model is not appropriate for the simulation of
the encoder in the low-rate case. In the meantime, as shown inequation (3), the more accurate the R-D model is, the betterλ can be obtained. Therefore, in HEVC, a more interesting
hyperbolic R-D model is proposed in equation (4), where thismodel was introduced by Mallat in 1998 [26].
D (R) = K × R −C ,(4)where C and K are the model parameters related to thecharacteristic of the video source. Putting equation (4) into
equation (3), we know that λ can be re-written as∂D= C × K × R −C−1
∂R
1C+1
= C × K− C × D C = γ D τ ,

λ=−

(5)

where γ and τ are both coding constants. The distortion
measure D is the mean squared error (MSE) between
the original and reconstructed CTUs or frames. MSE isa mathematically tractable and fast-to-compute full-reference (FR) quality metric, which has been widely used inmodern block-based video compression.
To validate the relationship between D and λ, wehave compressed several video sequences in the low-delaycase using the HEVC reference software HM10.0 and thesimulation results are shown in Fig. 1. For several values of λ,we plot D(λ) as a function of λ and fit the data via the formulain (5), where the average values are used. This experiment

2946

IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 25, NO. 7, JULY 2016

Fig. 1. Relationship between D and λ. The color curves are the fitting resultsof λ = γ D τ . Each sample point represents one test video, and the data points
with the same color denote the encoding results associated to the same video
sequence.

confirms that in the HEVC video coding system, the distortion
curves can be approximated by a power formula.
From now on, we will focus on the consistent video
quality modeling in which the more interesting D(λ)
relationship between two consecutive frames is established.
The distribution of distortion between two consecutive frames
is crucial in maintaining consistent quality control in videocoding. For consistent quality videos, it is observed thatthe distortion of the current CTU is distributed similarly
to that of the co-located position in the previous frame.
Hence, we incorporate the history of coding information into
the distortion model to avoid the complexity of parameter
estimation in equation (5). In addition, to maintain a consistentprev
quality, we assume Dicurr ≈ Di . Putting equation (5)D currD

Fig. 2. Difference between the actual distortion and the estimated distortion
per CTU of the 11th frame for (a) BasketballDrill, (b) BQMall, (c) BQTerraceand (d) Cactus (QP=32).
TABLE IC OMPARISONS OF D IFFERENT D ISTORTION M ODELS

previ≈ λ iprev , where
into this assumption, we can get λcurrii
are the distortion and the Lagrange multiplier
Dicurr and λcurriprev
for the ith CTU in the current frame. Similarly, Di
and
prev
λi
are for the co-located CTU in the previous frame. With
this assumption, we
can construct a heuristic linear distortion
prev
Di
model Dicurr ≈ λ prev
× λcurr.
ii
In order to remove large fluctuation during encoding, both
prev
prev
and Dican be computed as the weighted averages
λi
N
prev
from the previous frames [14] such as λi= N1l k=l 1
 Nl
prev
prev
prev
ωk λk
and Di= N1l k=, where Nl is the
1 ωk Dk
total number of the most recent encoded frames, and ωk is theassociated weight. On the other hand, from the implementation
point of view, we need to reduce the computational complexity
and save the storage of the encoder. Thus, we make a trade-off
between model accuracy and implementation complexity, i.e.,
N

prev
prev
prev
= N1
λk , and Di
is the actual distortion of
λi

k= 1

the ith CTU in the previous frame. Consequently, a linear
distortion model is completed as,σ
prev
Dicurr =× Di× λcurr,(6)iN
1  prev
λkN
k=1

where σ is a scaling factor that reduces the speed of distortion
changes between the co-located CTUs.

Extensive experiments have been performed to verify
the proposed distortion model, and four typical results arepresented in Fig. 2 that shows the distortion estimation resultsfor each CTU in low-delay video application. The encoding
frame is the 11th frame. The distortion of each CTU in thecurrent frame is estimated by the co-located distortion in
the 10th encoded frame. The scaling factor σ is chosen to
reduce the estimation error. For example, σ = 1.30, 1.34,
1.26 and 1.33 are the optimal values (in MSE sense) for(a), (b), (c) and (d), respectively. However, in this paper,
we do not explore the use of scaling factor for a better
estimation. Instead, as we will see later in Section IV-A,
σ is not required in the computation of λ at the CTU level.
More detailed numerical experiments are tabulated in Table I.
Pearson’s correlation coefficient (PCC) and normalized root
mean square error (NRMSE) are computed to measure theestimation accuracy and estimation error, respectively. The
NRMSE is defined as in

2
N 
1
i=1 Dact,i − Dest,i×,
NRMSE =
(Dact,max − Dact,min )
Nct u
(7)

WANG et al.: LOW-DELAY RATE CONTROL FOR CONSISTENT QUALITY USING DISTORTION-BASED LAGRANGE MULTIPLIER

where Nct u is the number of encoded CTUs in a frame,
Dact and Dest are the actual distortion and estimated distortionof the ith CTU, Dact,max and Dact,min are the maximum
and minimum values of Dact , respectively. Table I shows
the comparison between the proposed method and a typicaldistortion model that uses the neighboring information
to estimate the current one. Typically, the neighboringcurr
is considered as the predictedreconstructed error Di−1
curr . The
distortion of the current Dicurr , i.e., Dicurr = Di−1
PCCs of the proposed linear distortion method (6) range
from 0.91 to 0.99. In addition, the average NRMSE of our
method is 0.0674, while the neighboring method is 0.2287.
It can be seen that our distortion model is able to achieve
higher estimation accuracy than the traditional neighboring
estimation method.
B. Rate Modeling
In HEVC, the rate of the ith CTU in a frame is a functionof λ as shown in equation (5). For convenience, equation (5)is re-written asβ

Ri = αi × λi i ,(8)where Ri is the target budget of a CTU, and αi and βi are thecorresponding model parameters.
Extensive simulation results in JCTVC-K0103 have shownthat R(λ) in equation (8) is sufficient to represent therelationship between R and λ on the HEVC reference
software. In HEVC, given a target budget Ri fora CTU, the associated λi is computed using equation (8).
Consequently, the quantization parameter used for quantizing
the transform coefficients is computed by a natural logarithmic
formulaQ Pi = a × ln(λi ) + b,

(9)where Q Pi is the quantization parameter for the ith CTU,
and both a and b are constants, which are empirically set
as 4.2 and 13.71 in the reference software of HEVC HM10.0,
respectively.

model in equation (6) and Ri with equation (8), respectively,
the constrained MINAVE problem can be formulated as:
λ∗1 , . . . , λ∗N =1  curr  curr 
Dk
λk
N
N

arg minλcurr,λcurr ,...,λcurr
N ,1 N 2
curr ≤R
R
λ
(
) max
k
k= 1
k

=

N


arg minλcurr,λcurr ,...,λcurr ,k=1
1 N 2 curr N≤Rmax
k=1 Rk λk

(

k= 1

σ
prev
× λcurr,
prev × Dk
k
λk

)

(10)
where λ∗i is the optimal value of λcurr.
iEquation (10) is the key formulation for consistent quality
control. In equation (10), the average distortion and the target
budget constraint are convex function and convex set of, respectively. According to the Lagrange
the variable λcurritheory [32], there is a unique solution λ∗ = [λ∗1 , λ∗2 , . . . , λ∗N ]that can be obtained from equation (10). One way tosolve it is to use the Karush-Tuhn-Tucker (KKT) condition.
Unfortunately, it is difficult to obtain a closed-form solutionof equation (10) from its Lagrangian cost function, since thecomplex rate model is involved. Another way is to search
the possible set of λ∗ that satisfies the KKT condition ofequation (10). One should find if there is a certain λ∗ that leads
to a solution satisfying the KKT condition. This exhaustive
search method is infeasible for practical low-delay video
applications. Clearly, a more efficient solution is needed.
Based on the observation that the budget is a power function
curr
to relax thewith respect to the variable curr  λi , we propose
N
N≤ Rmax to k= 1 Rk λcurr≤
constraint
k= 1 Rk λk
k

N
Rmax
with the inequality of arithmetic and geometricNmeans. Then, we rewrite the problem in equation (10) as inequation (11). In equation (11), we can find a closed-form
solution of λ∗ . However, it is noted that we should consider
the buffer state and frame budget, andhence curr we need toN
λ≤ Rmax in theadjust λ∗ to guarantee that k=Rk1
kpractical encoder HM10.0. The proposed method is formulated
as follows.
λ∗1 , . . . , λ∗N =IV. R ATE C ONTROL W ITH D ISTORTION -BASEDL AGRANGE M ULTIPLIER
A. Proposed Distortion-Based Lagrange Multiplier
Quality variation in a compressed video signal has an
essential impact on the human visual perception [27]–[30],so the goal of the consistent quality control is to minimize thedistortion variation across video frames within the constraintsof frame rate, bandwidth, and delay requirement. Based on
the aforementioned distortion and rate models, we investigate
the problem of optimal bit allocation from the viewpoint ofminimizing average (MINAVE) distortion [31] for the HEVC
encoder system. Specifically, we want to find an expression for
the Lagrange multipliers that minimizes the average distortion1 N
k= 1 Dk subject to the summation of the target budgetsNN
k= 1 Rk which is not larger than the frame target budget
Rmax . In addition, replacing Di with the proposed distortion

2947

argminλcurr,λcurr,...,λcurr
1
2
N ,
N
Rmax
k=1 ln(Rk )≤N ln
N
prev

×Dk

N


σ

prev
λ
k=1 k

× λcurr.
k(11)

The Lagrangian cost function of equation (11) can be
expressed in equation (12), which is an unconstrained problem.
In addition, if Ri is replaced with equation (8), then we have
argminL λcurr, λcurr, . . . , λcurr
1
2
N ,u
λcurr
> 0,i= 1,...,N,u≥ 0
i

=

N


σ
prev
× λcurr
prev × Dkkλcurr
λi > 0,i= 1,...,N,u≥ 0 k= 1 kargmin

+u

N

k= 1

ln (Rk ) − N ln

Rmax
N

2948

=

IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 25, NO. 7, JULY 2016

N


σ
prev
× λcurr
prev × Dkkλcurr >0,αi >0,βi <0, k=1 λkiarg min

TABLE IIS YMBOLS U SED IN THE D ISTORTION -BASEDL AGRANGE M ULTIPLIER A LGORITHM

i=1,...,N,u≥0

+u

N





ln (αk ) + βk ln λcurr− N ln
k

k=1

Rmax
N

,
(12)

where u is the Lagrange multiplier. In Lagrange’s theory, it
has been shown that if there is a u ∗ such that equation (12)achieves the minimum value at λ∗ = [λ∗1 , λ∗2 , . . . , λ∗N ], thenλ∗ is also an optimal solution to equation (11).
In addition, since equation (12) is minimizing a convex anddifferentiable function on a convex set, the KKT condition
guarantees that the KKT point λ∗ is an optimum solution.
Consequently, after some straightforward manipulations(see Appendix), we obtain the optimal λ∗ in equation (13).

λ∗i =

−βi
prev × e
Di


N

−
ln αk
N ln Rmax
N
k=1
N

βk
k=1

−βk
prev
Dk

βk

.

(13)

As mentioned before, the summation of the CTU bit budgets
R λ∗i in equation (11) should satisfy the frame budget
constraint. Thus, we can adjust λ∗i to a proper λ∗C T U,i such thatN∗
k= 1 RC T U,k λC T U,k ≤ Rmax . In general, adjacent frames
in a video sequence have very high correlations that guaranteethat we can predict the associated model parameters for thecurrent frame from the previous encoded one. Furthermore,equal bit allocation scheme is used for each frame in the
proposed method. To maintain a proper bit budget for thecurrent frame, we use a scaling factor χ of the previous frame
to update λ∗i in equation (14). The scaling factor χ is obtained
prev
prev
from the actual budget Ract and target budget Rt ar of theprevious frame.
prevχ = Rt ar R prev .
(14)act
To satisfy the frame budget, we adjust the CTU level target
bit budget as RC T U,i = χ Ri . As a result, when replacing
RC T U,i and Ri with equation (8), we can obtain the CTUlevel λ∗CU T ,i as1λ∗CU T ,i = χ βi λ∗i ,(15)where λ∗CU T ,i is used to encode the current CTU.
We conducted experiments to evaluate the proposed
method (15) in low-delay video communications. The standard
video sequences with different resolutions and frame rates are
encoded. Tables V and VI summarize the simulation results of
the actual bit rates and the target ones. Experimental results
show that the proposed method can achieve a higher average
bits estimation accuracy.
B. Proposed Rate Control Algorithm
The traditional three-level bit allocation scheme has beenvery successful in the H.264/AVC rate control. As a result,
JCTVC-K0103 and other rate control proposals in the

HEVC codec follow the same structure, such as GOP level,
frame level and CTU level. Inspired by these pioneering works,
we adopted the same approach of JCTVC-K0103 with thedetailed algorithm as described as Algorithm 1. To easily
understand the following steps, the reader is referred to the
JCTVC-K0103 implementation in the HM10.0 software. The
constant exponents (e.g., in Step 2.2 and 3.3) are empirically
set in our paper. Symbols used in the proposed method aretabulated in Table II.
V. S IMULATION R ESULTS
To evaluate the performance of the proposed consistentquality control approach, we compare it with the state-of-theart techniques on HM10.0 platform [33]. The experiments
are conducted on a dual-core (i3-2100@3.10G Hz) workstation with RAM 4GB that is also used to measure thecomputational complexity of our method. In the simulation,
the rate control based encoder parameters are set as
follows: RateControl (enabled), NumLCUInUnit (enabled),
LCULevelRateControl (enabled), RCLCUSeparateModel(enabled), InitialQP (disabled), KeepHierarchicalBit (disabled)
and RCForceIntraQP (disabled). All the other encoder settings
are set identically for all methods. We have performed thefollowing four methods.
• HM10.0: JCTVC-K0103 rate control algorithm has been
implemented and enabled for all tests in this section.
• Choi et al. [10]: Choi’s method [10] (i.e., JCTVC-H0213)
has been implemented and compared in HM10.0.
• Lee et al. [14]: a frame-level rate control method isimplemented and tested in HM10.0.
• Proposed
method: the proposed distortion-based
Lagrange multiplier method is implemented in HM10.0.
In the experiment, the first two frames (i.e., the first
intra and inter frame) are encoded by the HM10.0 scheme,
which are used to collect the corresponding model parameters.
The proposed method was evaluated with two bandwidths(i.e., low and high bit rate) under the low-delay configurations(i.e., P and B Main coding profile). In both coding
structures, all the representative standard sequences with

WANG et al.: LOW-DELAY RATE CONTROL FOR CONSISTENT QUALITY USING DISTORTION-BASED LAGRANGE MULTIPLIER

Algorithm 1 Proposed distortion-based rate control method

2949

In the results, the standard deviation of Peak Signal-to-Noise
Ratio (PSNR) and Structural SIMilarity (SSIM) [34] are themeasures for the smoothness of video quality. Besides theabove measures, the quality change between adjacent framesis also employed, i.e.,
1  L DY,k − DY,k−1 ,(16)Vavg =
k=2
L −1
where L is the length of the coded video frames, and DY,kcan be the luminance Y-PSNR value or Y-SSIM of thekth frame. Since the Y-PSNR is frequently used in comparison,
we do not explicitly distinguish PSNR and Y-PSNR in thispaper. The additional complexity is measured by(17)
Tavg = |Tmet hod − Tanchor | Tanchor × 100%,
where Tmet hod and Tanchor are the total computational
complexities of the candidate method and HM10.0,
respectively. The buffer size is set as
Bu f f er = Delay × Target ,unique characteristics in the format of 4:2:0 YUV wereused to simulate low-delay communications. The intra frame
period (IFP) is set about 0.5 fps.

(18)

where Delay is the delay time for the real-time video bitstream,
and Target is the channel bandwidth. Delay is set at about
0.3 seconds in the simulation.
Obviously, high quality and low buffer occupancy cannot
be achieved concurrently due to a contradiction between themin low-delay communications. In order to achieve low-latency,
the buffer size Bu f f er is set as small as possible. According toequation (18), the buffer occupancy is mainly determined by
the target bits, which can be adjusted by changing the target
bits as well as the video quality. Fig. 3 shows four typicalbuffer occupancy results under the low-delay P configuration,
which demonstrates the superiority of the proposed approachthat can achieve lower buffer occupancy and less buffer
fluctuation in comparison with HM10.0. Such an improvement
is beneficial to enhance both the video quality and the buffer
occupancy level. The buffer fullness curves in Fig. 3 show thatthe proposed method has no buffer overflow whereas HM10.0
can lead to the buffer overflow. Meanwhile, our method canprovide less buffer variation (e.g., no buffer underflow) incomparison with Choi et al. [10] and Lee et al. [14]. It isnoted that our work here is not to study the effect of buffer
overflow (or underflow) and frame-skip, but to examine therole of the low buffer occupancy in the low-delay transmission.
We believe that if the frame-skip is enabled, HM10.0 would
give a worse quality fluctuation due to the loss of high-quality
reference frames.
Besides controlling the buffer occupancy, high quality
is also desirable in low-latency communication. In Fig. 4,
we compare the overall rate-distortion performance of the
proposed algorithm with that of HM10.0, Choi et al. [10]and Lee et al. [14] under the low-delay P configuration.
Experiments show that R-lambda based methods (i.e., HM10.0
and our method) generally gives a better rate-distortion
performance compared to that of R-Q based methods (i.e.,
Choi et al. [10] and Lee et al. [14]). Furthermore, it canbe seen that our method outperforms the other methods
for all of bit rates. In Fig. 4 (b), our method can achievea significant video quality improvement. The reason is that the2950

IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 25, NO. 7, JULY 2016

Fig. 3.
Comparison of frame-by-frame buffer occupancy: (a) BlowingBubbles (0.8 Mbps), (b) BQSquare (0.8 Mbps), (c) BQMall (3.2 Mbps) and(d) PartyScene (3.2 Mbps).

BQSquare sequence contains zoom motion, and HM10.0 has
poor estimation of encoding parameters in the raster-scanning
bit allocation, λ adjustment, and quality control. On the other
hand, our method establishes a novel relationship between
distortion and λ. Based on the new distortion model, we
obtain a computationally feasible solution to the problemof obtaining optimal λ for consistent video quality that can
avoid the use of raster-scanning bit allocation. Consequently,
the proposed method can achieve a better compromise
between quality variation and buffer occupancy in comparison
with HM10.0. Other simulation results also verify similar
performance.
Fig. 5 shows the subjective comparison of video quality
for Baskballpass (i.e., from frame 106 to 115) under
the low-delay P configuration. The comparisons of thereconstructed frames in Fig. 5 validate the visual quality
performance of the proposed method. As shown in the
top two rows of Fig. 5, it can be seen that our method has
a better quality in the texture area, where the numbers on thescoreboard clearly demonstrate the superiority of our method.
In addition, distortion maps show a subjective comparison, andsmall distortion indicates higher video quality. We can observethat the fourth row of Fig. 5 is more homogeneous compared tothe third row, where the brightness is the error measure. Fig. 6
shows the case for BQSquare from frame 106 to 115. In the
results of BQSquare, the maximum SSIM variation is 0.0376
in HM10.0 while the maximum SSIM variation is 0.0230 in
our method. Simulations have presented similar performancefor all other sequences that we have tested.
Since the SSIM score is considered to be closer tohuman evaluation than the PSNR value, simulation resultsof subjective comparison in terms of SSIM are tabulated inTables III and IV. In the simulation, we first compute theframe-by-frame SSIM score for each video sequence, and thenthe standard deviation of SSIM and the average SSIM change
between adjacent frames are used as the measures of video
quality variation. In Tables III and IV, “Avg.”, “Std. dev.” and“Vavg ” represents the average SSIM, the standard deviationof SSIM and the average quality change between adjacentframes, respectively. It can be seen that compared to HM10.0
and Choi et al. [10], our method can achieve the highest
average SSIM scores and the lowest SSIM variations in boththe low-delay P and B Main coding structures. Meanwhile,
the average SSIM score of Choi et al. [10] is worse than that

WANG et al.: LOW-DELAY RATE CONTROL FOR CONSISTENT QUALITY USING DISTORTION-BASED LAGRANGE MULTIPLIER

Fig. 4.

2951

Rate-distortion curves: (a) BlowingBubbles, (b) BQSquare, (c) BQMall and (d) PartyScene.

Fig. 5. Subjective visual quality comparisons of BasketballPass at bit rate 0.4 Mbps: (a) ∼ (j) HM10.0, proposed method, distortion map of HM10.0, anddistortion map of the proposed method (from top to bottom).

of the HM10.0 algorithm. The detailed simulation results ofSSIM can be found in Table III and Table IV.
We conducted the subjective experiment to evaluate
visual quality as suggested in [30]. There are total
fifteen subjects who participated in the test. The mean
opinion score (MOS) takes the five-point scale rule
(i.e., 5-excellent, 4-good, 3-fair, 2-poor, and 1-bad). Five
representative videos are selected as the test source, including
the following content characteristics, such as fast and slow

motion, high and low resolution. The distorted videos are thelow bit-rate reconstructed results from Table V. The results areshown in Fig. 7, where the bigger MOS value means the better
visual quality of the decoded video stream. It can be observedthat the proposed method outperforms the other methods.
For easy comparison, the simulation results of PSNR
and the variations thereof are tabulated in Table V. In the
low-delay P coding structure, the average PSNR improvementof our method is about 0.35 dB and 0.59 dB in comparison

2952IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 25, NO. 7, JULY 2016

Fig. 6. Subjective visual quality comparisons of BQSquare at bit rate 0.4 Mbps: (a) ∼ (j) HM10.0, proposed method, distortion map of HM10.0, anddistortion map of the proposed method (from top to bottom).

TABLE IIIC OMPARISONS OF SSIM VALUES IN THE
L OW-D ELAY P C ODING S TRUCTURE

TABLE IVC OMPARISONS OF SSIM VALUES IN THE
L OW-D ELAY B C ODING S TRUCTURE

with HM10.0 and Choi et al. [10], respectively. The averagerate control errors of HM10.0, Choi et al. [10] and our
method are 1.54%, 1.40% and 0.08%, respectively. For thequality variation measurement in terms of standard deviation,
HM10.0, Choi et al. [10] and our method are 2.52, 2.21
and 1.31, respectively. In addition, for the quality variation
measurement in terms of Vavg , HM10.0, Choi et al. [10] andour method are 0.50, 1.03 and 0.21, respectively. As can beobserved from the simulation results, the proposed method
generally outperforms HM10.0 and Choi et al. [10] in termsof the average PSNR and PSNR variations in the low-delay
P coding structure.

Fig. 7. Subjective quality comparisons between the proposed method andthe traditional algorithms.

In order to obtain a better coding efficiency, HEVC
supports the low-delay B coding structure. The associatedsimulation results are shown in Table VI. In our method,model parameters updates in equation (13) only rely on thenearest previous frame. The average PSNR improvement ofour method is about 0.39 dB and 1.24 dB, compared toHM10.0 and Choi et al. [10], respectively. Meanwhile, theaverage standard deviations of PSNR are 2.57, 1.74 and 1.34
for HM10.0, Choi et al. [10] and our method, respectively,
while the quality variation Vavg s of PSNR are 0.48, 0.95
and 0.21, respectively. It can be seen that the proposed method
can provide a more consistent video quality compared toHM10.0 and Choi et al. [10] in terms of the PSNR variations
under the low-delay B coding structure.
To get a consistent quality control, we employ additional
operations to compute a distortion-based λ for each CTU. Tavg
is considered as a factor to measure its complexity. Compared
to HM10.0 in the low-delay P coding structure, the Tavg valueof our method is about 3.90%, while that of Choi et al. [10]is about 27.57% as shown in Table V. In the low-delay B
coding structure, the Tavg value of our method is about 4.98%,
while Choi et al. [10] is about 22.52%. It can be seen that thecomplexity of our method is lower than that of Choi et al. [10]in both the low-delay P and B coding structures. There
are two reasons. One is that the model parameters in theWANG et al.: LOW-DELAY RATE CONTROL FOR CONSISTENT QUALITY USING DISTORTION-BASED LAGRANGE MULTIPLIER

2953

TABLE VS IMULATION R ESULTS IN THE L OW-D ELAY P C ODING S TRUCTURE

TABLE VIS IMULATION R ESULTS IN THE L OW-D ELAY B C ODING S TRUCTURE

proposed algorithm can be easily computed. The other being
the computational complexity of the R-lambda based rate
control is marginal when compared to that of the whole
encoding system. Furthermore, it should be pointed out that thereal-time concept is from the algorithm design instead of thecomputer-based implementation here. Therefore, considering
the improvement in video quality, one can conclude that the
proposed algorithm outperforms HM10.0 in low-delay videocommunications.
VI. C ONCLUSION
This paper focuses on consistent quality control for
HEVC and introduces an efficient distortion-based Lagrange
multiplier approach in low-latency video communications.
Using the distortion of co-located CTU in the previous frame,a new relationship between distortion and λ is established andemployed to control the video quality fluctuations. Based on
the proposed distortion model, we obtain a computationallyfeasible λ in the minimization of the total distortion subject
to a given bit rate. When considered jointly with thebuffer state, the CTU level λ is further adjusted such thatthe target bits satisfy the overall bandwidth of low-delay
video communication. As demonstrated in the simulation
experiments, the proposed rate control method outperforms
state-of-the-art techniques in terms of bit rate regulation, video
quality fluctuation and encoder buffer fullness.
A PPENDIXThe Karush-Tuhn-Tucker (KKT) condition of equation (12) is (A.1), as shown at the top of the next page.
N

prev
σ
× λcurr
It is easy to check thatprev × Dk
k
λ
k= 1

k


N 



Rmax
ln (αk ) + βk ln λcurr
−
N
ln
and
k
N
k= 1

are convex

functions with respect to the variable λcurr. In this case, we
k

 ∗ ∗
∗
know that if the point λ1 , λ2 , . . . , λ N satisfies the above
KKT conditions, it is a globe minimum solution for the
optimization problem (12).
prev
In practice, the distortion and the previous λi (i =1, . . . , N) is a non-negative value, and hence we know thatprev
σ
= 0 for any i ∈ [ 1, 2, . . . , N ]. In this case,prev × Di
λ
i

2954

IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 25, NO. 7, JULY 2016

⎧
∂L
σ
βi
prev
⎪
⎪
= prev × Di
+ u curr = 0, i = 1, . . . , N.
(i )
⎪
curr
⎪
∂λi
λi
λi
⎪
⎪
⎪
⎨
N 



Rmax
= 0, u ≥ 0. (ii )uln (αk ) + βk ln λcurr− N ln
k
N
⎪
k=1
⎪
⎪
prev
prev
⎪
⎪
λcurr
> 0, λi> 0, Di> 0,(iii )⎪i⎪⎩σ > 0, αi > 0, βi < 0, i = 1, . . . , N.

we know u = 0 in the condition (i ). As a result, λcurr
cani
be expressed as
λcurr
=
i

u
−βi
(−βi ) u
× prev .
prev = σDi
prev
prev × Di
λ
λ

(A.2)

σ

i

i

Substituting equation (A.2) into the condition (ii ), we can getNln (αk ) + βk ln

u

σ
prev
λ
k

k=1

k
× D−β
prev

Rmax
.
N

= N lnk(A.3)
Equation (A.3) can be further rewrote as

N ln

u

σ
prev
λi

Rmax
N

−

N


ln αk

k=1
N


=e

−βk
prev
Dk

βk

βk

.

k=1

(A.4)

Using equations (A.2) and (A.4), λcurr
is formulated as
i
=
λcurr
i

u

σ
prev
λi

×

−βi
prev
Di
N ln

=

−βi
prev × e
Di



Rmax
N

−

N


ln αk

k=1
N


βk

k=1

−βk
prev
Dk

βk

.

(A.5)

ACKNOWLEDGMENT
The author would like to thank the helpful comments givenby the anonymous reviewers.
R EFERENCES[1] G. J. Sullivan, J.-R. Ohm, W.-J. Han, and T. Wiegand, “Overview
of the high efficiency video coding (HEVC) standard,” IEEE Trans.
Circuits Syst. Video Technol., vol. 22, no. 12, pp. 1649–1668,
Dec. 2012.
[2] T. Wiegand, G. J. Sullivan, G. Bjøntegaard, and A. Luthra, “Overview
of the H.264/AVC video coding standard,” IEEE Trans. Circuits Syst.
Video Technol., vol. 13, no. 7, pp. 560–576, Jul. 2003.
[3] T. Chiang and Y.-Q. Zhang, “A new rate control scheme using quadratic
rate distortion model,” IEEE Trans. Circuits Syst. Video Technol., vol. 7,
no. 1, pp. 246–250, Feb. 1997.
[4] Z. He and S. K. Mitra, “Optimum bit allocation and accurate rate control
for video coding via ρ-domain source modeling,” IEEE Trans. Circuits
Syst. Video Technol., vol. 12, no. 10, pp. 840–849, Oct. 2002.
[5] B. Li, H. Li, L. Li, and J. Zhang, “Rate control by R-lambda model for
HEVC,” document JCTVC-K0103, Shanghai, China, 2012.
[6] X. Wang and M. Karczewicz, “Intra frame rate control based on SATD,”
document JCTVC-M0257, Incheon, South Korea, 2013.
[7] M. Wang, K. N. Ngan, and H. Li, “An efficient frame-content based
intra frame rate control for high efficiency video coding,” IEEE Signal
Process. Lett., vol. 22, no. 7, pp. 896–900, Jul. 2015.

(A.1)[8] H. Choi, J. Nam, J. Yoo, D. Sim, and I. Bajić, “Rate control based on
unified RQ model for HEVC,” document JCTVC-H0213, San Jose, CA,
USA, 2012.
[9] Y. Liu, Z. G. Li, and Y. C. Soh, “A novel rate control scheme forlow delay video communication of H.264/AVC standard,” IEEE Trans.
Circuits Syst. Video Technol., vol. 17, no. 1, pp. 68–78, Jan. 2007.
[10] H. Choi, J. Yoo, J. Nam, D. Sim, and I. V. Bajić, “Pixel-wise unified
rate-quantization model for multi-level rate control,” IEEE J. Sel. Topics
Signal Process., vol. 7, no. 6, pp. 1112–1123, Dec. 2013.
[11] X. Jing, L.-P. Chau, and W.-C. Siu, “Frame complexity-based
rate-quantization model for H.264/AVC intraframe rate control,”IEEE Signal Process. Lett., vol. 15, pp. 373–376, 2008,
doi: 10.1109/LSP.2008.920010.
[12] S. Wang, S. Ma, S. Wang, D. Zhao, and W. Gao, “Rate-GOP based rate
control for high efficiency video coding,” IEEE J. Sel. Topics Signal
Process., vol. 7, no. 6, pp. 1101–1111, Dec. 2013.
[13] B. Lee and M. Kim, “Modeling rates and distortions based on a mixtureof Laplacian distributions for inter-predicted residues in quadtree codingof HEVC,” IEEE Signal Process. Lett., vol. 18, no. 10, pp. 571–574,
Oct. 2011.
[14] B. Lee, M. Kim, and T. Q. Nguyen, “A frame-level rate control scheme
based on texture and nontexture rate models for high efficiency videocoding,” IEEE Trans. Circuits Syst. Video Technol., vol. 24, no. 3,
pp. 465–479, Mar. 2014.
[15] M. Jiang and N. Ling, “Low-delay rate control for real-time H.264/AVC
video coding,” IEEE Trans. Multimedia, vol. 8, no. 3, pp. 467–477,
Jun. 2006.
[16] M. Jiang and N. Ling, “On Lagrange multiplier and quantizer adjustment
for H.264 frame-layer video rate control,” IEEE Trans. Circuits Syst.
Video Technol., vol. 16, no. 5, pp. 663–669, May 2006.
[17] M. Wang and B. Yan, “Lagrangian multiplier based joint three-layer
rate control for H.264/AVC,” IEEE Signal Process. Lett., vol. 16, no. 8,
pp. 679–682, Aug. 2009.
[18] B. Li, H. Li, L. Li, and J. Zhang, “λ domain rate control algorithm
for high efficiency video coding,” IEEE Trans. Image Process., vol. 23,
no. 9, pp. 3841–3854, Sep. 2014.
[19] Z. Chen and K. N. Ngan, “Recent advances in rate control for videocoding,” Signal Process., Image Commun., vol. 22, no. 1, pp. 19–38,
2007.
[20] A. Ortega and K. Ramchandran, “Rate-distortion methods for image
and video compression,” IEEE Signal Process. Mag., vol. 15, no. 6,
pp. 23–50, Nov. 1998.
[21] T. Wiegand, H. Schwarz, A. Joch, F. Kossentini, and G. J. Sullivan,“Rate-constrained coder control and comparison of video coding
standards,” IEEE Trans. Circuits Syst. Video Technol., vol. 13, no. 7,
pp. 688–703, Jul. 2003.
[22] L.-J. Lin and A. Ortega, “Bit-rate control using piecewise approximatedrate-distortion characteristics,” IEEE Trans. Circuits Syst. Video
Technol., vol. 8, no. 4, pp. 446–459, Aug. 1998.
[23] M. Wang and M. van der Schaar, “Operational rate-distortion modelingfor wavelet video coders,” IEEE Trans. Signal Process., vol. 54, no. 9,
pp. 3505–3517, Sep. 2006.
[24] Y.-K. Tu, J.-F. Yang, and M.-T. Sun, “Rate-distortion modeling for
efficient H.264/AVC encoding,” IEEE Trans. Circuits Syst. Video
Technol., vol. 17, no. 5, pp. 530–543, May 2007.
[25] G. J. Sullivan and T. Wiegand, “Rate-distortion optimization for video
compression,” IEEE Signal Process. Mag., vol. 15, no. 6, pp. 74–90,
Nov. 1998.
[26] S. Mallat and F. Falzon, “Analysis of low bit rate image transform
coding,” IEEE Trans. Signal Process., vol. 46, no. 4, pp. 1027–1042,
Apr. 1998.
[27] Z. Chen and K. N. Ngan, “Towards rate-distortion tradeoff in real-timecolor video coding,” IEEE Trans. Circuits Syst. Video Technol., vol. 17,
no. 2, pp. 158–167, Feb. 2007.

WANG et al.: LOW-DELAY RATE CONTROL FOR CONSISTENT QUALITY USING DISTORTION-BASED LAGRANGE MULTIPLIER[28] L. Xu, D. Zhao, X. Ji, L. Deng, S. Kwong, and W. Gao, “Window-levelrate control for smooth picture quality and smooth buffer occupancy,”
IEEE Trans. Image Process., vol. 20, no. 3, pp. 723–734, Mar. 2011.
[29] J. Hou, S. Wan, Z. Ma, and L.-P. Chau, “Consistent video quality control
in scalable video coding using dependent distortion quantization model,”IEEE Trans. Broadcast., vol. 59, no. 4, pp. 717–724, Dec. 2013.
[30] L. Xu, S. Li, K. N. Ngan, and L. Ma, “Consistent visual quality control
in video coding,” IEEE Trans. Circuits Syst. Video Technol., vol. 23,
no. 6, pp. 975–989, Jun. 2013.
[31] G. M. Schuster, G. Melnikov, and A. K. Katsaggelos, “A review ofthe minimum maximum criterion for optimal bit allocation among
dependent quantizers,” IEEE Trans. Multimedia, vol. 1, no. 1,
pp. 3–17, Mar. 1999.
[32] D. A. Pierre, Optimization Theory With Applications. New York, NY,
USA: Dover, 2012.
[33] (2014). HEVC Reference Software. [Online]. Available: http://hevc.
kw.bbc.co.uk/trac/browser/jctvc-hm/tags[34] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image
quality assessment: From error visibility to structural similarity,” IEEE
Trans. Image Process., vol. 13, no. 4, pp. 600–612, Apr. 2004.

Miaohui Wang (S’13–M’16) received the
Ph.D. degree from the Department of Electronic
Engineering, Chinese University of Hong Kong,
Hong Kong, in 2015. From 2014 to 2015, he wasa Visiting Scholar with the Innovation Laboratory,
InterDigital Inc., San Diego, CA, USA. He received
the Best Master Thesis Award in Shanghai (2011)
and Fudan University (2012), China.
He has authored or co-authored numerous
technical papers in international journals andconferences. His current research interests covera wide range of topics related with video compression and transmission,
computer vision and machine learning, including transform coding, rate
control, image restoration, and denoising, and deepneuron-network-based
applications. He is a member of the IEEE Circuits and Systems Society.

2955

King Ngi Ngan (M’79–SM’91–F’00) received
the Ph.D. degree in electrical engineering fromLoughborough University, Loughborough, U.K.
He was a Full Professor with Nanyang
Technological University, Singapore, and with the
University of Western Australia, Perth, Australia.
He is currently a Chair Professor with the
Department of Electronic Engineering, Chinese
University of Hong Kong, Hong Kong. He holds
honorary and visiting professorships with numerous
universities in China, Australia, and South East Asia.
He has published extensively, including three authored books, six edited
volumes, over 300 refereed technical papers, and has edited nine special
issues in journals. He holds ten patents in image or video coding andcommunications.
Dr. Ngan is a fellow of IET, U.K., and IEAust, Australia, and was anIEEE Distinguished Lecturer from 2006 to 2007. He has served as an
Associate Editor of the IEEE T RANSACTIONS ON C IRCUITS AND S YSTEMSFOR V IDEO T ECHNOLOGY , the Journal on Visual Communications andImage Representation, the EURASIP Journal of Signal Processing: Image
Communication, and the Journal of Applied Signal Processing. He has
chaired a number of prestigious international conferences on video signal
processing and communications, and has served on the advisory andtechnical committees of numerous professional organizations. He co-chaired
the IEEE International Conference on Image Processing, Hong Kong,in 2010.
Hongliang Li (SM’12) received the Ph.D. degree
in electronics and information engineering fromXi’an Jiaotong University, Xi’an, China, in 2005.
He was a Research Associate with the
Visual Signal Processing and Communication
Laboratory (VSPC), Chinese University ofHong Kong (CUHK), Hong Kong, from2005 to 2006. From 2006 to 2008, he wasa Post-Doctoral Fellow with VSPC, CUHK. He iscurrently a Professor with the School of Electronic
Engineering, University of Electronic Science andTechnology of China, Chengdu, China. He has authored or co-authorednumerous technical articles in international journals and conferences. He isa Co-Editor of the book entitled Video Segmentation and its Applications(Springer, 2011). His research interests include image segmentation,
object detection, image and video coding, visual attention, and multimedia
communication system.
Dr. Li is a member of the Editorial Board of the Journal on Visual
Communications and Image Representation, and the Area Editor of Signal
Processing: Image Communication. He served as a Technical Program
Co-Chair in ISPACS 2009, the General Co-Chair of the ISPACS 2010,
the Publicity Co-Chair of the IEEE VCIP 2013, the Local Chair of theIEEE ICME 2014, and a TPC Member in a number of international
conferences, such as ICME 2013, ICME 2012, ISCAS 2013, PCM 2007,
PCM 2009, and VCIP 2010. He serves as a Technical Program Co-Chair of
the IEEE VCIP 2016. He was selected as the New Century Excellent Talent
in University, Chinese Ministry of Education, China, in 2008.