US008559511B2(12) United States Patent(10) Patent No.:

Ngan et al.
(54)

(75)

US 8,559,511 B2(45) Date of Patent:

METHOD AND APPARATUS FOR VIDEO

6,611,608 B1

Oct. 15, 2013

8/2003 WI} 6t a1~

CODING BY ABT-BASED JUST NOTICEABLE

zoog/bgfégg? 221*

DIFFERENCE MODEL

2007/0110327 A1 *Inventors: King Ngi Ngan, Hong Kong (HK); Lin2007/0257988 A1
11/2007 Ong et al.
2010/0086063 A1 * 4/2010 Haskell et al. ......... .. 37 5/240.27

Ma’ Hong Kong (HK); wai'K‘len

351x221 ~~~~~~~~~~~~~~~~ ~ 375/ 24016
5/2007

Han ............................ .. 382/251

OTHER PUBLICATIONS

Chain, Hong Kong (HK); Yu Liu, Hong
Kong

Jia,Y.; Lin, W.; Kassim, A.A.; Estimating Just-Noticeable Distortion
forVideo; IEEE Tran. on Circuits and Systems forVideo Technology,(73) Assignee: Hong Kong Applied Science andTechnology Research InstituteCompany Limited’ Hong Kong Science.
.
.
park’shann’ New Terntones(*)

Notice:Subject to any disclaimer, the term of thispatent is extended or adjusted under 35

U.S.C. 154(b) by 728 days.
(21)

Appl. No.: 12/750,401

Justnoticeable Distortion Model for H.264/AVC; IEEE Int’l Sympo
Sium on Circuits and Systems 2009, pp‘ 609-612,

Primary Examiner * Mehrdad Dastouri
Assistant Examiner * Kristin Dobbs

Flled:

Mar‘ 30’ 2010(65)(74) Attorney, Agent, or Firm * Ella Cheong Hong Kong;

Prior Publication Data
Us 2011/0243228 A1(51) Int‘ Cl‘
H043 1/66
H04N 7/12(52)

Just-Noticeable Difference Pro?le for Images; Proc. 10th P.R. Conf.
on Multimedia. Advances in MIP, pp. 1208-1218, (2009).
Mak, Chun-Man; Ngan, King Ngi; Enhancing Compression Rate by* cited by examiner

_

(22)

Jl11~ 2006, V01~ 16, N0~ 7, PP 820-829
Ma, Lin;_ Ngan, King Ngi; Adaptive Block-Size Transform Based

H04N11/02
H04N 11/04
US‘ C1‘

Sam T' Ylp

Oct. 6, 2011(57)
ABSTRACTThe present invention relates to method and apparatus forvideo coding by ABT-based just noticeable difference (JND).
For building the just noticeable difference model, spatial(200601)(200601)

content information (SCI) is used to represent the spatial(200601)(200601)

appearance similarity betWeen one macroblock and its sub
blocks and the motion characteristic distance (MCD) is used
to represent the motion characteristics similarity betWeen one

USPC ~~~~~~~~~~~ ~~ 375/240-13; 375040-12; 37504002

macroblock and its sub-blocks. For intra frames, the balance(58)

Field of Classi?cation Search
None
See application ?le for Complete Search history-(56)

References Cited

strategy based on the obtained SCI of the macroblock is used
to generate the ABT-based JND model. For inter frames, thebalanced strategy based on the obtained SCI and MCD of themacroblock is used to generate the ABT-based JND model.
Using the ABT-based JND model, the residual coef?cients foreach block in a frame is ?ltered to obtain a reduced set of

US. PATENT DOCUMENTS
5,426,512 A
5,559,900 A
5,629,780 A

residual coef?cients for transmission Without degradation invisual quality.

6/1995 Watson
9/1996 Jayant et a1.
5/1997 Watson

17 Claims, 6 Drawing Sheets

Input video

lntra

Prediction

30

40

Generation of
ABT-based JND

US. PatentOct. 15, 2013

Sheet 1 of6

Input video 1

[J10

1,21

US 8,559,511 B2/;l/22

lnt'raPrediction

lnter I
Prediction

/\)30
/\)40
Adaptive Block-size 'Generation ofTransform

ABT-based JNDl
ABT-based JND (JFiltering

i

50

' '

Rate Distortion l»)

60

Optimization

1

Entropy Coding 1») 70

FIG. 1

US. Patent

0a. 15, 2013

Sheet 2 of6

Block Types

1

[310
Inter

lntra or Inter?

V

US 8,559,511 B2[J21

PsychpphysicalExperiment

Psychophysical
16x16

Experiment

/

11

16x16

8x8
8x8

f}

33

‘'

PsychophysicalPsychophysicalExperiment

Experlment
v

V

4x4

4x4

FIG. 2

US. PatentOct. 15, 2013

US 8,559,511 B2

Sheet 3 0f 6

J00

ABT
301
I

4x4
Residual

302

[310

V

303

Residualv '/~/
16x16
Residualv/311

Selected JND

g 4x4 JND’JNDExtension

8x8 JND,‘V I331
4x4 fil’gered

62

Residual

16x16 JND’"232 :
8x8 filt'ered
Residualp933

16x16 filiered
Residual
340

FIG. 3

US. Patent

0a. 15, 2013

Sheet 4 of6

US 8,559,511 B2

11

I410

FIG. 4

U S. Patent

0a. 15, 2013

Sheet 5 of6

US 8,559,511 B2

520

522

FIG. 5

US. PatentOct. 15, 2013

Sheet 6 of6

US 8,559,511 B2

ABT-b/ased JND
Input
video

data
O—>

_

—>

model

601

generation unit

1

ABT

ABT

/~/

_'

Pred|ct|on mode
determination

—>

604

unit —> based JND

unit

'

60?,“

filter unitl

J05

Rate distortion
optimization unit

i

N606

Entropy I

coding unit

i

Coded
data

FIG. 6

US 8,559,511 B2
1

2

METHOD AND APPARATUS FOR VIDEO
CODING BY ABT-BASED JUST NOTICEABLE
DIFFERENCE MODEL

threshold, the corresponding IND threshold Will be sub
tracted from the non-Zero residual coef?cient.
In terms of the nature of the inventive IND models, unlike

the existing IND models Which are generated in image
TECHNICAL FIELD

domain, or are based on ?xed block-siZe DCT for IND models Which are generated in frequency domain, the present

The present invention relates generally to digital video
processing. In particular, the present invention relates to digi

invention employs ABT-based IND models Which are basedon different block siZes. Additionally, the ABT-based IND

tal video coding including encoding and decoding.

models take properties of hoW humans perceive things visu

BACKGROUND

ally into consideration over different block-siZe based transforms. The ABT-based IND model is determined based on

Due to the advance in imaging devices, more and more

spatial content information (SCI) and the motion character
istics distance (MCD).

videos (sequences of images, also knoWn as, motion pictures)are being generated anytime and anyWhere. Moreover, in the

In terms of the applications of the inventive IND models,
unlike the existing IND models Which are used for image/pursuit of higher resolution, the video siZe keeps increasing.

video Watermarking and quality assessment, the present
These all involve a large amount of data to be handled. This
invention uses IND models to enhance the video coding e?iposes a huge challenge in processes such as displaying, transciency and reduce the bit-rates.
mitting and storing the videos. Therefore, there is a need forFurthermore, the present invention performs video coding
video coding techniques so that video can be compressed 20
using the ABT-based IND models Without tuning the quantiWithout any loss to its quality.
Regarding the quality, less distortion is desired and theZation matrix according to a pre-de?ned image quality metric.

distortion is traditionally measured by quantitative metrics

One aspect of the present invention is to extend the ?xed
siZe IND model to various block siZes. The IND model forsuch as mean square error (MSE) and peak signal-to-noise

ratio (PSNR). HoWever, these quantitative metrics may not 25 various block siZes is used to capture the properties of hoW
re?ect hoW a human observer perceives an image and hoW
humans perceive things visually for different siZe transforms.
severe the distortion is in the eyes of the human observer.
Therefore, there is a need to further increase the compression

spatial content information (SCI) Which depicts the spatial

ratio Without introducing additional visual distortion by tak

appearance similarity betWeen one macroblock and its sub

Another aspect of the present invention is to determine

ing the properties of hoW humans perceive things visually into 30 blocks.
consideration. In other Words, a subjective test of hoW

Another aspect of the present invention is to determine

humans perceive an image is important. Therefore, there is amotion characteristic distance (MCD) Which depicts theneed to maintain a user’ s visual perception While performing

motion characteristics similarity betWeen one macroblock

video processing more e?iciently.
and its sub-blocks.
Furthermore, pursuant to the development of video stan 35
Another aspect of the present invention is to make use ofdards, an increasing number of features are incorporated in
the balanced strategy betWeen different block-siZe based
current and future video standards. One of these features is totransforms for image or intra frames based on the SCI of the
alloW various block siZes in video coding processes, forexample, adaptive block-siZe transform (ABT). This helps toimprove image quality by considering the image contents. 40
Therefore, there is a need to enable ABT in video coding, forexample, in compliance With video standards such as H.264/
AVC.
SUMMARY OF THE INVENTION

The present invention generates the ABT-based Just
Noticeable Difference (IND) model based on the spatial con
tent information and motion characteristics distance, Which45

macroblock.
Another aspect of the present invention is to make use of
the balanced strategy betWeen different block-siZe basedtransforms for inter frames based on the SCI and MCD of themacroblock.

Another aspect of the present invention is to perform
residual ?ltering for different siZe transforms to preserve thesame visual quality With bit-rate reduction.
Other aspects of the present invention are also disclosed as

illustrated by the folloWing embodiments.

BRIEF DESCRIPTION OF THE DRAWINGS
can be applied to video coding schemes by residual ?ltering 50
methods. The IND model is also knoWn as the IND pro?le.
Since the IND model can be represented in the form of a
These and other objects, aspects and embodiments of thisclaimed invention Will be described hereinafter in morematrix, the IND model is also referred as a IND map.
details With reference to the folloWing draWings, in Which:
One of the advantages of the present invention is that asmuch as 50% bit rate can be saved by using ABT-based IND 55
FIG. 1 shoWs a ?owchart of an exemplary embodiment ofmodel for video coding Without degradation in the subjective

video coding provided by the present invention.

visual quality. The present invention makes use of the prop

FIG. 2 shoWs a ?owchart of an exemplary embodiment ofgeneration of an ABT-based IND model.
FIG. 3 shoWs a ?oWchart of an exemplary embodiment ofer‘ties of hoW humans perceive things visually to reduce the bit

rate Without introducing noticeable visual distortions. A sys
tem that models the process of humans perceiving things 60 ABT-based IND ?ltering.
FIG. 4 shoWs an exemplary allocation of test DCT sub
visually is knoWn as a visual perception model (VPM). One
bands in a 16x16 block and a 4x4 block respectively in one
example of a visual perception model is the Just Noticeable
embodiment.
Difference (IND) model. The ABT-based IND model is built
FIG. 5 depicts motion vectors of macroblocks and theand gives the IND threshold for every block in a frame so thatthe non-Zero residual coef?cients are ?ltered out if their val 65 corresponding subblocks to illustrate the concept of MCD
ues are less than the IND thresholds. If the value of a non-Zeroconsistency for the temporal balanced strategy for interframes.
residual coef?cient is larger than the corresponding IND

US 8,559,511 B2
3

4

FIG. 6 depicts a block diagram for an exemplary embodi
ment of a video coding apparatus according to one embodi

LetY(u,v) be the (u,v)th transformed residual coef?cient of
the residual X,. If the Y(u,v) has a magnitude smaller thanJX(u,v), the transformed residual coef?cient Y(u,v) can be

ment.

discarded. If the Y(u,v) has a magnitude larger than JX(u,v),
the transformed residual coef?cient Y(u,v) is subtracted by
JX(u,v). The processing of the transformed residual coef?

DETAILED DESCRIPTION OF THE INVENTION
FIG. 1 shows a ?owchart of an exemplary embodiment ofcient Y(u,v) is also known as a ?ltering step 150 (ABT-based
JND ?ltering). In one embodiment, the ABT-based JND ?lvideo coding provided by the present invention. In one
embodiment, video coding is performed on an input video
110 and the input video 110 is encoded. To encode the input

tering step 150 is implemented before quantiZation and the?ltering computation is de?ned in the equation (4) below:

video 110, the JND model for each block of various blocksiZes in each frame in the input video 110 is computed.
Examples of possible block siZes are, but not limited to, 4x4,
8x8 and 16x16.
For each macroblock, a prediction is computed based oneither intra prediction 121 or inter prediction 122. For
example, inter prediction 122 is used when the frame is a P

0otherwise

where Y]. is the JND-?ltered transformed block. The sign

frame or a B frame. Intra prediction 121 is based on different
block siZes such as 4x4, 8x8, 16x16 and is also known asfunction ensures Y(u,v) to be negative or positive after ?lter
different block-size intra prediction. Inter prediction 122 is 20 ing. Since the residual coef?cients are ?ltered, the recon
structed block X]-' is computed by the equation (5) below:
based on different block siZes such as 4x4, 8x8, 16x16 and isalso known as different block-size inter prediction.
A difference block is generated from the difference
where Y?q is the quantiZed Yj.
between the prediction and the block. The difference block is
The distortion d between the reconstructed block X1.‘ andalso known as residual. The residual is partitioned into dif 25
the block X is de?ned in the equation (6) below:
ferent con?gurations of non-overlapping blocks for encoding
depending on the block siZe of the residual. The following
d:llX,-’—Xll1(6)
information are also computed:
where the distortion d is equal to the sum of absolute(1) Block types, for example, inter, or intra;(2) Block categoriZations, for example, plane, edge, or 30 difference (SAD) between the reconstructed and the original
block. Other difference metrics such as, but not limited to,
texture; and(3) Motion vectors for blocks of different siZes.
sum of squared difference (SSD) or sum of absolute trans
The above information is employed to compute the Just
formed difference (SATD) can also be used for rate distortion
Noticeable Difference (IND) model 140. According to one of
optimization (RDO) 160.
Let E be the DCT transformed difference between thethe embodiments, it is assumed that the JND model 140 is 35
original block and reconstructed block and E is computed ininitially developed for an 8><8 discrete cosine transformthe equation (7) below:(DCT) block based on Jia’s model as described in Jia, Y.; Lin,W.; Kassim, A. A.; Estimating Jusl-Noliceable Dislorlionfor
Wdeo; IEEE Transactions on Circuits and Systems for Video

Technology, Vol. 16, No. 7, pp. 820-829, July 2006, and the 40
disclosure of which is incorporated herein by reference in its

E is ?ltered by JX as in the above equation (4) and the?ltering is computed by the equation (8) below,

entirety. The JND model 140 for 8x8 DCT block can beextended to other block siZes as described below.

Let X be the block to be encoded, XP be the prediction of X.
The residual X,, as de?ned in the equation (1) below, is 45

transformed, quantized, and encoded.

if IEW, WI > 1x04, v),
Ej-(u, v) =0otherwise

XfX-XP(1)
For the term “?ltering”, the following operations is per
The residual X, are transformed by adaptive block-sizetransform (ABT) 130, also known as, different block-size 50 formed: If the E(u,v) has a magnitude smaller than JX(u,v), theDCT transformed difference E(u,v) can be discarded. If thetransform, for example, by different block-size discreteE(u,v) has a magnitude larger than JX(u,v), the DCT transcosine transform (DCT) to compute the transformed residual
formed difference E(u,v) is subtracted by JX(u,v). Then theY. The transformed residual, Y, is de?ned in the equation (2)distortion is computed from the inverse transform of E], as
below:
55 shown in the equation (9) below, and used in the rate distor
Y:T(Xr)(2)
tion optimiZation 160:
where T represents the ABT on residual X,. The quantiZedtransformed residual,Yq, is de?ned in the equation (3) below:
If rate distortion optimiZation 160 is enabled, the predic(3)60

where Q represents the quantiZation operation.

tion mode chosen for the block is the one that minimiZes the

RD (rate distortion) cost. For example, the RD cost is de?ned
as the equation (10) below:

The ABT-based JND model 140 is computed for the block
X The ABT-based JND model 140, denoted as JX(u,v), is amatrix or a map containing JND thresholds over which theobserver starts to notice any distortion for each residual coef?cient. The computation of the IND thresholds will be further
discussed later.

65

where 7» is a Lagrangian multiplier and L is the actual bit

length of encoding the block with that prediction mode. For
example, if the RD cost for inter prediction is lower than that

US 8,559,511 B2
5

6

for intra prediction, inter prediction Will be chosen as theprediction mode for that block.

IND model. The resulting ABT-based IND model is based on

The ABT-based IND ?ltering step 150 can be vieWed as an

adaptive quantization step in Which the quantiZer of eachblock is determined from its spatial and temporal character
istics. Because the ABT-based IND ?ltered block Will

undergo normal encoding procedure, the bitstream is totallycompliant to the H.264 standard and can be decoded correctly

by any decoder.
The selected transform mode together With the ?lteredresidual coef?cients is coded by the entropy coding 170. After
different block-size transforms and the ABT-based IND ?ltering are performed, the total distortion of the macroblockand its required bits are computed. In accordance With thetotal distortion of the macroblock and its required bits, thebest transform mode (best transform siZe) is determinedaccording to the rate distortion optimiZation. The ABT-based
IND model 140 is related With the transform mode, no otherthe block siZe of 16x16 according to the psychophysicalexperiment 221 based on the block siZe of 16x16. If thespatial content information of the 16><16 block is not consis
tent, then the 16><16 block is divided into 8x8 blocks. Subse

quently, for each 8x8 block, if the spatial content informationof the 8x8 block is consistent according to the spatial bal
anced strategy 222, then the block siZe of the DCT is selected
to be 8x8. The 8x8 DCT is utiliZed to generate the ABT-based
IND model. The resulting ABT-based IND model is based on
the block siZe of 8x8. If the spatial content information of the
8x8 block is not consistent, then the block siZe of the DCT isselected to be 4><4, the 4x4 DCT is utiliZed to generate the

ABT-based IND model. The resulting ABT-based IND modelis based on the block siZe of 4x4 according to the psycho
physical experiment 223 based on the block siZe of 4x4.
For an inter block, the spatial and temporal balanced strat
egies 230 are used to determine if the spatial content infor
bits need to be encoded.
FIG. 2 shoWs a ?owchart of an exemplary embodiment of 20 mation (SCI) of the 16><16 block is consistent and if the
motion characteristic distance (MCD) of the 16><16 block isgeneration of the ABT-based IND model. In order to generate
the ABT-based IND model, We need to determine the block
type 210. If it is an inter block, a balanced strategy in terms ofconsistent. If both the SCI and the MCD are consistentaccording to the spatial and temporal balanced strategies 230,

both spatial and temporal characteristics is applicable. If it isthen the block siZe of the DCT is selected to be 16><16. The
an intra block, only spatial balanced strategy is applicable. In 25 16><16 DCT is utiliZed to generate the ABT-based IND model.
one embodiment, the 8x8 DCT based IND model is used andis extended to different block-size DCT based IND models
such as 16><16 DCT based IND model and 4x4 DCT based

The resulting ABT-based IND model is based on the block

IND model. The details concerning adaptive block-size transis not consistent, then the 16><16 block is divided into 8><8

siZe of 16x16 according to the psychophysical experiment231 based on the block siZe of 16x16. Ifthe SCI or the MCD

form (ABT) based just-noticeable difference (IND) model 30 blocks. Subsequently, for each 8><8 block, the spatial andfor images are described in Ma, Lin; Ngan, King N.; Adaptive
temporal balanced strategies 232 are used to determine if theBlock-Size Transform Based Just-Noticeable Dlference Pro
SCI of the 8x8 block is consistent and the MCD of the 8x8?le for Images; Proceedings of the 10th Paci?c Rim Confer
block is consistent. If both the SCI and the MCD are consis
ence on Multimedia: Advances in Multimedia Information
tent according to the spatial and temporal balanced strategies
Processing, Session 13: Advanced Image Processing Tech 35 232, then the block siZe ofthe DCT is selected to be 8x8. The
niques; Page 1208-1218; 2009, the disclosure of Which is
8><8 DCT is utiliZed to generate the ABT-based IND model. If
incorporated herein by reference to its entirety.
the SCI or the MCD is not consistent, then the block siZe of
An automatic IND model can be determined in the image
the DCT is selected to be 4><4. The 4><4 DCT is utiliZed todomain, or transform domain, such as DCT and discrete
generate the ABT-based IND model. The resulting ABT
Wavelet transform (DWT), or even the combination of these 40 based IND model is based on the block siZe of 4x4 accordingtWo schemes. To accurately describe hoW humans perceiveto the psychophysical experiment 233 based on the block siZethings visually, a IND model generated in the transformof 4x4.
domain, Which is also knoWn as the subband-based IND, may
Spatial Balanced Strategy

incorporate all the major affecting factors. The major affect

Iust Noticeable Difference (IND) is the smallest detectableing factors include, but are not limited to, spatial contrast 45 difference betWeen a starting and secondary level of a parsensitivity function (CSF), luminance adaptation effect andticular sensory stimulus in psychophysics. IND is also
contrast masking.
regarded as a difference limen or a differential threshold. In
ABT alloWs different transform siZes. Different transformgeneral, the IND model can be expressed in form of thesiZes are adopted according to the general strategy that larger

folloWing equation (11):transforms are used to provide better energy compaction and 50

better preservation of details, While smaller transforms could
prevent more ringing artifacts during compression. The

(11)
Where
T
JNDS
is
the
spatial
IND
and
T
JNDT
is
the
temporalpresent invention alloWs the ABT-based IND model to make
IND modulation factor. In the inter frames, the intra predic
use of the different performance of hoW humans perceive
things visually if different transform siZes are used. During 55 tion is also needed for processing the blocks Which are notsuitable for motion estimation. Because these blocks depend
the extension, a psychophysical experiment is carried out tomoreon their spatial neighboring blocks than the temporalparameteriZe the spatial CSF to take account of the properties
frames. Consequently, these blocks are regarded as containof hoW humans perceive things visually. The content of aning spatial information only and are modeled by the spatial
image is also adaptively adjusted to decide Which transform
siZe is employed to yield the resulting ABT-based IND 60 IND model Which folloWs the spatial balanced strategy.

The spatial IND model in the DCT domain is determinedmodel.
by the folloWing equation (12):For an intra block, the spatial balanced strategy 220 is used
to determine if the spatial content information (SCI) is con
sistent in the block. If the spatial content information of a
16x16 block is consistent according to the spatial balanced 65
Where (m,n) denotes the position of DCT block in an
strategy 220, then the block siZe of the DCT is selected to be
image, (i,j) indicates the DCT subband (i,j), Than-c is the basic
16><16. The 16><16 DCT is utiliZed to generate the ABT-based
visibility threshold generated from spatial contrast sensitivity

US 8,559,511 B2
8

7Contrast Masking

function (CSF); 011M," is the luminance adaptation; 01cm, is thecontrast masking; and T(m,n,i,j) is the spatial JND threshold

The basic visibility threshold, T basic, is de?ned as theContrast masking, 01cm, is used to refer to the reduction in
the visibility of one visual component in the presence ofanother one. For example, noise Will be less visible in a
textured region but Will be more visible in a smooth region or

inverse of the contrast sensitivity of spatial frequency Whichis the sensitivity of human visual perception over spatial

categorization and is given by the folloWing equation (20):as obtained.

Basic Visibility Thresholdan edge region. Therefore, 01cm varies according to block

frequency and is modeled by the following equation (13):(20)

05cm :

Where (1) is the speci?ed spatial frequency. As JND thresh
old is the inverse of the sensitivity modeled by the equation(13), the basic JND threshold is modeled by the equation (14)‘l‘,for (12 +112) 5 16 in Plane and Edge Block

W .n[4 {l (beloW:'“n“m’

C011,”. 1. 1')

Team-4mm.mmmmmm

1036]]’

othersl4

Tbasic(ma n, i, j) =

(

)

1,
20

Where \y =forPlane and Edge Block

2.25, for (12 +112) 5 16 in Texture Block

1.25, for (12 +112) > 16 in Texture Block

Where s:0.25 denotes the summation effect factor, and 4),,4)]. are the DCT normalization factors as de?ned in the equation (15) beloW:

As one embodiment, the block categorization is the one
25

implemented in the image domain. Firstly, Canny edge detec
tor is utilized to mark the edge pixels in the image domain.
Secondly, based on the number of edge pixels in the block, theblock can be categorized into three types, namely PLANE,
EDGE, and TEXTURE, respectively. For example, for a30

N stands for the dimension of the DCT block. And 4),].

16x16 macroblock, block categorization is de?ned according
to:denotes the directional angle of the corresponding DCT coef?cient as shoWn in the equation (16) beloW:

PLANE(16) 35

Z < 16(21)

Edge

Where (1),,’- is the corresponding spatial frequency of DCT
subband (i,j) and is de?ned in the equation (17) beloW:

BlockfCategorization:

EDGE

1652 s52
Edge

TEXTURE

(17)

Edge

40

Where 6,C and By are horizontal and vertical visual angles

respectively and are given by the folloWing equation (18):Where 2 edge is the number of edge pixels in a given mac
roblock.
In one embodiment, the temporal JND model is determined

Where Rd indicates the ratio of vieWing distance to image

by the folloWing equation (20):height, and Ph is the image height (in pixel). The parameters(a, b, c) are de?ned and computed in the folloWing paragraphs.

1,

Luminance Adaption
Luminance adaption, 0111"", is used to represent hoW easily

TmDT =

50

1.07f1,

a difference can be noticed against different luminance. For

f, < Scpd &f, <10 Hz(22)

107M610), f, < Scpd &f, z 10 HZ,

f, > Scpds

example, there is a higher visibility threshold in either dark or
bright regions to denote the fact that it is more dif?cult to
Where ft:fsxvx+fsyvy; fsx, fsy, are horizontal spatial fre
notice any difference, While there is a loWer visibility thresh
old in medium bright regions to denote the fact that it is easier 55 quency and vertical spatial frequency respectively; and vx, vy
is the velocity of object motion on the retina plane along theto notice any difference. The luminance adaption is based onhorizontal direction and along the vertical direction, respec
the average intensity lave of the DCT block and is given by thetively.

folloWing equation (19):60

alum =(6O - avg/150 +1[m s 601

60 < [m < 170(1a,, -170)/425 + 1[m z 170

After the extension of the 8x8 to 16x16 as Well as to 4><4,(19)

three JND pro?les based on different block-size transforms
Will have been obtained. The smaller block-size DCT Would
result in better energy compaction and detailed information
65

Where lave is the average intensity value of the Whole DCT
block.

Balanced Strategy for JND Pro?le BetWeen Different
Block-Size Transforms

preservation, While the larger block- size DCT could represent
the content and characteristics of local image blocks more

e?iciently.

US 8,559,511 B2
10Spatial Balanced Strategies for Intra Frames
(23)

2
MCD =

TABLE 1I‘:0(Balanced strategy among 16x16, 8x8 and 4x4 JND pro?les)where MVLZ- is the motion vector of sub-block, while
MVMJ- is the motion vector the macroblock.
If MCD is smaller than a threshold, and the spatial content
information is consistent, a larger DCT based JND will beutiliZed. Otherwise, smaller DCT based JND will be

16x16

macroblock type8x8 sub-block typeJND Pro?lePLANE

All 8x8 sub-blocks are
PLANE

16x16 macroblock PLANE

Otherwise

Table 2

employed.

EDGE

All 8x8 sub-blocks are

16x16 macroblock EDGE

FIG. 5 depicts motion vectors of macroblocks and thecorresponding sub-blocks to illustrate the concept of MCDTEXTUREEDGE
Otherwise
All 8x8 sub-blocks are
TEXTURE
Otherwise

Table 2consistency for the temporal balanced strategy for inter

16x16 macroblock TEXTURE

15

Table 2frames. FIG. 5 shows a current frame 520 and its reference
frame 510. The macroblock 521 in the current frame 520 has

a motion vector pointing to the corresponding macroblock
511 in the reference frame 510. The macroblock 522 in thecurrent frame 520 has a motion vector pointing to the corre

For intra frames, the image is ?rstly divided into 16x16 20 sponding macroblock 512 in the reference frame 510. It ismacroblocks. The block categorization is employed as theshown that the motion vectors of the sub-blocks of macrob
basis for balancing different JND pro?les. As shown in table
lock 521 are more consistent with the motion vector of the
1, the ?rst column represents the 16x16 macroblock type,
macroblock 521 when compared with those motion vectors ofwhile the second column represents the 8x8 sub-block cat
egoriZation. And the third column is the decision for gener

25

the sub-blocks of macroblock 522. It is likely that the MCD
for macroblock 522 is larger than the MCD for macroblockating the resulting JND pro?le. For example, if the 16x16

521 and even over the threshold.

macroblock is marked as PLANE and all the 8x8 sub-blocks
in the macroblock are also marked as PLANE, the 16x16

Psychophysical Experiment

FIG. 4 shows an exemplary allocation of test DCT subbands in a 16x16 block and a 4x4 block respectively in one
DCT-based JND model is adopted for its advantage in energy
compaction and preserving detail information. Otherwise, the 30 embodiment. In one embodiment of a 16x16 block 410, test
DCT subbands are represented by shaded cells, such as cell
JND pro?le determination process will continue as in the411, and are allocated as depicted in the ?gure. In one
following Table 2:embodiment of a 4x4 block 420, test DCT subbands are
represented by shaded cells such as cell 421 and are allocated
TABLE 235 as depicted in the ?gure. As shown in FIG. 4, the test DCT(Balanced strategy among 16x16 8x8 and 4x4 JND pro?les) — continued

subbands are allocated such that at least one test DCT subband is located on each row and on each column, the test DCT

8x8 sub-blocktype

4x4 sub-block typesubbands cover low frequencies, middle frequencies and highJND Pro?lefrequencies, and at least one DCT subband is located on each
PLANE

All 4x4 sub-blocks are PLANE

8x8 sub-block PLANE

Otherwise

4x4 sub-block

EDGE

All 4x4 sub-blocks are EDGE

8x8 sub-block EDGE

Otherwise

4x4 sub-blocksubband, a plurality of amplitudes of noise for each subband

TEXTUREAll 4x4 sub-blocks are
TEXTURE

8x8 sub-block
TEXTUREare chosen based on the pre-designed measurements. Viewers

Otherwise

4x4 sub-block40

diagonal. Noise is added to certain spatial frequency sub
bands of 16x16 and 4x4 DCT individually. For each test DCT

45

For example, if the 8x8 sub-block is marked as PLANE andgive their opinions on whether the noise can be perceived. Ifyes, the noise amplitude will be reduced. In contrast, the noise
amplitude will be increased if the noise cannot be perceived.
After the experiment, the just noticeable difference thresholds for the selected DCT subbands can be obtained.
The JND model has directionality, which presents as most

all the 4x4 sub-blocks in the 8x8 sub-block are also marked as

PLANE, 8x8 DCT-based JND model will be adopted for each 50
sensitive to the horizontal and vertical frequency components
8x8 sub-block in the 16x16 macroblock according to theand least sensitive to the diagonal components. This percep
different information and characteristics of the 8x8 sub
tual performance is also named as the “oblique effect.”
block, for example, whether the block type is PLANE orDuring parameteriZation, at least one DCT subband is
EDGE or TEXTURE. Otherwise, the 4x4 DCT-based JND

model will be employed for each 4><4 sub-block, by consid

located on each row and on each column, at least one DCT
55

ering different content information and characteristics of thesubband is located on each diagonal, and the DCT subbands
are preferred to cover low frequencies, middle frequencies,

4x4 sub-blocks, for example, whether the block type isand high frequencies.

PLANE or EDGE or TEXTURE.

Least mean squared error is employed to ?t the obtained

JND threshold to the equation (24) below:

Temporal Balanced Strategy for Inter Frames
60

Temporal properties between a block and its sub-blocks areused for the balanced strategy for inter frames. In one embodi
ment, the motion vector is utiliZed to represent the motion(a. b. c) = argminz [Tn-j - That-Am. n. 1'. 012(24>

characteristic of the object. Motion Characteristic Distance(MCD) represents the motion characteristic similarity 65
between one macroblock and its sub-blocks. The MCD isdetermined by the following equation (23):where T61" is the IND threshold obtained from the psycho

physical experiment.

US 8,559,511 B2
1112FIG. 3 shows a ?owchart of an exemplary embodiment ofputes the ABT-based IND model for the inter frame based on

theABT-based IND ?ltering. In one embodiment, 4><4 residu
als 301, 8x8 residuals 302, and 16x16 residuals 303 are
obtained from the ABT 300 according to different transform
siZes of DCT.

spatial content information and temporal characteristics
according to the spatial and temporal balanced strategy as
described above.
The ABT-based IND ?lter unit 604 ?lters one or more

residual coe?icients by discarding one or more residual coef?cients Which are beloW the IND thresholds of the ABT
based IND model so that a signal With ?ltered residual coef

Through the generation of IND model 310, the selected
IND model 311 is generated, Which may be 4><4 IND model,
8x8 IND model, or 16x16 IND model. Furthermore, there are?cients is output. Based on the ?ltered residual coef?cients,
the rate distortion optimiZation unit 605 determines the ratetWo types of IND models for each transform siZe, namely
spatial IND model for intra macroblock and temporal IND
model for inter macroblock.
In order to make the IND model compatible With the ABTtransforrn, the selected IND model is extended to different
block-size based IND model by IND extension 312. In IND
extension 312, the information such as the block type, block

distortion cost to select the best prediction mode by loWering
the rate distortion cost if rate distortion optimiZation unit isenabled.

The entropy coding unit 606 then computes the coded data
after coding the ?ltered residual coef?cients.
Embodiments of the present invention may be imple
categoriZation, and motion vectors for different blocks aremented in the form of softWare, hardWare, application logic or
utiliZed to generate different block-size based IND models
a combination of softWare, hardWare and application logic.
for each block. Therefore, 4><4 IND model 321, 8x8 IND
The softWare, application logic and/or hardWare may residemodel 322, and 16x16 IND model 323 are obtained for ?l20 on integrated circuit chips, modules or memories. If desired,tering the transformed residual coe?icients.

part of the softWare, hardWare and/or application logic may
reside on integrated circuit chips, part of the softWare, hard
if Eryp s JNDW,

Ware and/or application logic may reside on modules, and(25)part of the softWare, hardWare and/or application logic may
reside on memories. In one exemplary embodiment, theapplication logic, softWare or an instruction set is maintainedon any one of various conventional non-transitory computer
Equation (25) represents the ?ltering process. “typ”
denotes the block siZe for the DCT transform and the IND
readable media.
Processes and logic ?oWs Which are described in this speci
model. Ely}, is the transformed residual coe?icients according
to different block siZes, INDWP is the generated IND, Which 30 ?cation can be performed by one or more programmable25

may be 4><4 IND model 321, 8x8 IND model 322, or 16x16

processors executing one or more computer programs to per

IND model 323. El”, is the ?ltered coef?cients.
After ?ltering, 4><4 ?ltered residual coe?icients 331, 8x8form functions by operating on input data and generating
output. Processes and logic ?oWs can also be performed by?ltered residual coef?cients 332, or 16x16 ?ltered residual
special purpose logic circuitry, e.g., an FPGA (?eld program
coef?cients 333 are obtained. The residual coef?cients 35 mable gate array) or an ASIC (application-speci?c integrated

together With the prediction mode information Will be pro
cessed by the rate distortion optimiZation (RDO) 340 forchoosing the best prediction mode. Therefore, When rate dis
tortion is enabled, the prediction mode chosen for the block is
the one that minimiZes the rate distortion cost. In one embodi
ment, the rate distortion cost is de?ned as R:d+}\,L

circuit).
Apparatus or devices Which are described in this speci?
cation can be implemented by a programmable processor, acomputer, a system on a chip, or combinations of them, by
40

operating on input date and generating output. Apparatus or
devices can include special purpose logic circuitry, e.g., an
FPGA (?eld programmable gate array) or an ASIC (applica
tion-speci?c integrated circuit).Apparatus or devices can also

Where 7» is a Lagrangian multiplier and L is the actual bit

length of encoding the block With that prediction mode.
FIG. 6 depicts a block diagram for an exemplary embodi

include, in addition to hardWare, code that creates an execu

ment of a video coding apparatus provided by the present 45 tion environment for computer program, e. g., code that con
invention. Various components as described beloW can be
implemented as one or more processors. The prediction mode

stitutes processor ?rmWare, a protocol stack, a database man

agement system, an operating system, a cross-platform

determination unit 601 operates on input video data and deter

runtime environment, e.g., a virtual machine, or a combina

mines Whether a frame in the video signal is an inter frame ortion of one or more of them.

an intra frame according to the prediction mode of the frame. 50
Processors suitable for the execution of a computer pro
The ABT unit 603 performs adaptive block transform on thegram include, for example, both general and special purpose
input video data to generate DCT transformed signal of vari
microprocessors, and any one or more processors of any kindous block siZes. On the other hand, the ABT-based INDof digital computer. Generally, a processor Will receive
model generation unit 602 generates an ABT-based IND
instructions and data from a read-only memory or a random
model based on the folloWing information as described 55 access memory or both. The elements of a computer generally

above:

include a processor for performing or executing instructions,(1) Block types, for example, inter, or intra;(2) Block categoriZations, for example, plane, edge, or
texture; and(3) Motion vectors for blocks of different siZes.
The ABT-based IND model contains one or more IND

thresholds as determined by a psychophysical experimentbased on the visual perception model. The ABT-based IND
model generation unit 602 computes the ABT-based IND

and one or more memory devices for storing instructions anddata.
Computer-readable medium as described in this speci?ca60tion may be any media or means that can contain, store,

communicate, propagate or transport the instructions for use
by or in connection With an instruction execution system,
apparatus, or device, such as a computer. A computer-readable medium may comprise a computer-readable storage

model for the intra frame based on spatial content information 65 medium that may be any media or means that can contain or

according to the spatial balanced strategy as described above.
The ABT-based IND model generation unit 602 also com

store the instructions for use by or in connection With an
instruction execution system, apparatus, or device, such as a

US 8,559,511 B2
13

14

computer. Computer-readable media may include all forms of
nonvolatile memory, media and memory devices, including
by Way of example semiconductor memory devices, e.g.,sense. Rather, it is possible to make variations and modi?cations Without departing from the scope of the present invention as de?ned in the appended claims.

EPROM, EEPROM, and ?ash memory devices; magnetic
disks, e. g., internal hard disks or removable disks; magneto

What is claimed is:

optical disks; and CD-ROM and DVD-ROM disks.

1. A video coding apparatus, comprising:A computer program (also knoWn as, e.g., a program, soft
Ware, softWare application, script, or code) can be Written ina unit for providing a video signal;a prediction mode determination unit that determines

any programming language, including compiled or interpreted languages, declarative or procedural languages, and it

Whether a frame in the video signal is an inter frame or an

intra frame according to the prediction mode of thecan be deployed in any form, including as a stand-alone
program or as a module, component, subroutine, object, orother unit suitable for use in a computing environment. A

frame;an adaptive block-siZe transform (ABT)-based just noticeable difference (JND) model generation unit that makes

computer program can be deployed to be executed on one
computer or on multiple computers that are located at one

use of one or more performance properties of hoW

single site or distributed across multiple sites and intercon
nected by a communication netWork.
Embodiments and/or features as described in this speci?
cation can be implemented in a computing system thathumans perceive things visually for different block-siZe

includes a back-end component, e.g., as a data server, or that 20

tains one or more JND thresholds as determined by aincludes a middleWare component, e.g., an application server,or that includes a front-end component, e.g., a client com

psychophysical experiment based on the visual percep
tion model; andputer having a graphical user interface or a Web broWser

an ABT-based JND ?lter that ?lters one or more residual

through Which a user can interact With one embodiment as

described in this speci?cation, or any combination of one or 25

coef?cients by discarding one or more residual coef?
cients Which are beloW the IND thresholds of the ABT

more such back-end, middleWare, or front-end components.
The components of the system can be interconnected by any

based JND model and subtracting the JND thresholds

form or medium of digital data communication, e. g., a com

than the JND thresholds of the ABT-based JND model sobased transforms and computes block types, block cat
egoriZations and motion vectors for block of different
siZes to generate an ABT-based JND model Which confrom one or more residual coef?cients Which are larger

munication netWork. Examples of communication netWorks
include a local area netWork (“LAN”) and a Wide area netthat a signal With ?ltered residual coef?cients is output;
30

Work (“WAN”), e.g., the lntemet.
The computing system can include clients and servers. A

Wherein the one or more JND thresholds are thresholds

over Which an observer of the output signal With ?lteredresidual coe?icients starts to notice any distortion; andclient and server are generally remote from each other andtypically interact through a communication netWork. The
relationship of client and server arises by virtue of computer 35
programs running on the respective computers and having aclient-server relationship to each other.

Wherein the one or more JND thresholds are modeled by

The Whole speci?cation contains many speci?c implemen
tation details. These speci?c implementation details are notmeant to be construed as limitations on the scope of the 40 Where:

invention or of What may be claimed, but rather as descrip

Than-Am, n, i, j) is a basic JND threshold;(m, n) denotes the position of DCT block in an image;tions of features speci?c to particular embodiments of theinvention.

s denotes the summation effect factor;

Certain features that are described in the context of sepa
rate embodiments can also be combined and implemented as 45
a single embodiment. Conversely, various features that aredescribed in the context of a single embodiment can also be4), and 4)]. are discrete cosine transform (DCT) normaliZa
tion factors;4),]. denotes directional angle of corresponding DCT coef

implemented in multiple embodiments separately or in any
suitable subcombinations. Moreover, although features may

001-’,- is corresponding spatial frequency of DCT subband (i,be described as acting in certain combinations and even ini?cient;J ,50y denotes the oblique effect factor; andtially claimed as such, one or more features from a combina
tion as described or a claimed combination can in certaina, b and c are selected so as to achieve a least mean squared

cases be excluded from the combination, and the claimed

physical experiment With the values obtained by com

error in ?tting JND thresholds obtained from a psycho

combination may be directed to a subcombination or varia

2. puting
The video
Than-GU11,
codingn: isapparatus as claimed in claim 1,
tion of a subcombination. Although various aspects of the 55
Wherein:invention are set out in the independent claims, other aspectsof the invention comprise other combinations of features
the ABT-based JND model generation unit computes the
ABT-based JND model for the block of intra type based
from the embodiments and/ or from the dependent claims With
on spatial content information according to a spatial
the features of the independent claims, and not solely the
combinations explicitly set out in the claims.
balanced strategy.
Certain functions Which are described in this speci?cation
3. The video coding apparatus as claimed in claim 2,
Wherein:may be performed in a different order and/or concurrently
the ABT-based JND model of a larger block siZe is selectedWith each other. Furthermore, if desired, one or more of theif the spatial content information is consistent betWeen aabove-described functions may be optional or may be com
bined.
block and the sub-block thereof.
The above descriptions provide exemplary embodiments
4. The video coding apparatus as claimed in claim 2,
Wherein:of the present invention, but shouldnot be vieWed in a limiting

US 8,559,511 B2
15

16

the spatial content information is determined by block
categorization based on the amount of edge pixels in

Wherein the one or more JND thresholds are modeled byeach block of the frame.
5. The video coding apparatus as claimed in claim 1,
Wherein:the ABT-based JND model generation unit computes the
ABT-based JND model for the block of inter type basedon spatial content information and temporal character
istic according to a spatial and temporal balanced strat

Where:

Than-Am, n, i, j) is a basic JND threshold;(m, n) denotes the position of DCT block in an image;s denotes the summation effect factor;
egy.
4), and 4)]. are discrete cosine transform (DCT) normaliza
6. The video coding apparatus as claimed in claim 5,
tion factors;
Wherein:4),]. denotes directional angle of corresponding DCT coef
the ABT-based JND model of a larger block size is selected?cient;if the spatial content information is consistent and the001-’,- is corresponding spatial frequency of DCT subband (i,temporal characteristic is consistent.
J ,7. The video coding apparatus as claimed in claim 5y denotes the oblique effect factor; andWherein:a, b and c are selected so as to achieve a least mean squared
error in ?tting JND thresholds obtained from a psycho
the spatial content information is determined by blockphysical experiment With the values obtained by com
categorization based on the amount of edge pixels in 20
each block of the frame.
11.puting
The method
Tbasicunsforn: coding
isa video signal as claimed in
8. The video coding apparatus as claimed in claim 5
claim 10, further comprising:
Wherein:computing the ABT-based JND model for the block of intra
the temporal characteristic is determined by a motion char 25
type based on spatial content information according to a

spatial balanced strategy.

acteristic distance betWeen a block and the subblocks
thereof.

12. The method for coding a video signal as claimed inclaim 11, Wherein:

9. A method for coding a video signal, comprising:processing the video signal using the video coding appa
ratus of claim 1.

30

10. A method for coding a video signal, comprising:providing a video signal to be coded;
determining Whether a frame in the video signal is an interframe or an intra frame according to the prediction mode 35of the frame;
generating an adaptive block-size transform (ABT)-based
just noticeable difference (JND) model by making use of
one or more performance properties of hoW humans

perceive things visually for different block-size based 40
transforms and computes block types, block categoriza
tions and motion vectors for block of different sizes togenerate the ABT-based JND model Which contains one
or more JND thresholds as determined by a psycho

physical experiment based on the visual perception 45
model; and?ltering one or more residual coef?cients by discarding
one or more residual coef?cients Which are beloW thethe ABT-based JND model of a larger block size is selected
if the spatial content information is consistent betWeen a
block and the sub-block thereof.
13. The method for coding a video signal as claimed inclaim 11, Wherein:
the spatial content information is determined by block
categorization based on the amount of edge pixels ineach block of the frame.
14. The method for coding a video signal as claimed inclaim 10, further comprising:computing the ABT-based JND model for the block of intertype based on a spatial content information and a tem

poral characteristic.
15. The method for coding a video signal as claimed inclaim 14, Wherein:
the ABT-based JND model of a larger block size is selected
if the spatial content information is consistent and thetemporal characteristic is coherent according to a spatial

and temporal balanced strategy.
16. The method for coding a video signal as claimed inclaim 14, Wherein:
the spatial content information is determined by block
categorization based on the amount of edge pixels in

ABT-based JND thresholds of the ABT-based JND
model and subtracting the JND thresholds from one or 50
each block of the frame.
more residual coef?cients Which are larger than the IND
17. The method for coding a video signal as claimed inthresholds of the ABT-based JND model so that a signal
claim 14, Wherein:
the temporal characteristic is determined by a motion charWith ?ltered residual coef?cients is output;
acteristic distance betWeen a block and the subblocks
Wherein the one or more JN D thresholds are thresholds
thereof.
over Which an observer of the output signal With ?ltered 55
residual coef?cients starts to notice any distortion; and