2008/12/17

RST invariant digital
image watermarking &digital watermarking
based audiovisual quality
evaluation

Outline•••Digital watermarking
RST invariant image watermarking
Audiovisual quality evaluation based on watermarking

Jiying Zhao, Ph.D.
School of Information Technology & Engineering

University of Tsukuba -- 12/16/2008

Introduction todigital watermarking
•
•
•
•
•
•

What is digital watermarking
Host signals
Applications of watermarking
How it works
Requirements for watermarking
Some examples

Digital watermarking•

Digital Watermarking technology allows users to embed somedata into digital contents such as still image, movie and audio
data.

•

When data is embedded, it is not written at header part but
embedded directly into digital media itself by changing media
contents data.

University of Tsukuba -- 12/16/2008

University of Tsukuba -- 12/16/2008

Host signals for
digital watermarking

Applications ofdigital watermarking

•
•
•
•
•

•

Image
Video
Audio
Software
…

•

•
•
•

University of Tsukuba -- 12/16/2008

Rights management– Copyrights protection– Content distribution, tracking and monitoring
Authentication– Assure contents integrity– Prevent unauthorized alternation of contents– Detect alternation location in the contents
Contents management– Indexing & retrieval
Access/copy control– Prevent unauthorized copy, playback of multimedia contents
Quality evaluation

University of Tsukuba -- 12/16/2008

1

2008/12/17

How it works

Technical requirements
for watermarking(IBM TRL)

Invisibility

Robustness

University of Tsukuba -- 12/16/2008

Capacity

University of Tsukuba -- 12/16/2008

Examples of watermarking

Broadcast monitoring(example: IBM TRL)

Original

Watermarked

University of Tsukuba -- 12/16/2008

University of Tsukuba -- 12/16/2008

Owner identification

Content authentication

•

•

•
•

An invisible watermark is very resistant to destruction under any
image manipulation.
Verifies ownership of media suspected of misappropriation.
Digital detection of the watermark would indicate the source of
the image.

University of Tsukuba -- 12/16/2008

•
•
•
•

A fragile watermark will be destroyed when the image is manipulated digitally in
any way.
It proves authenticity of media.
If the watermark is still intact, then the image has not been “doctored” .
If the watermark has been destroyed, then the image has been tampered with.
Semi-fragile watermark, tell-tale watermark

University of Tsukuba -- 12/16/2008

2

2008/12/17

RST invariant
image watermarking [1][3][6]••Fourier transformf ( x, y)    F (u, v)

Rotation, Scaling, and Translation (RST) are the mostchallenging attacks to image watermarking.
We have proposed several novel RST invariant image
watermarking algorithms.

• “An image rectification scheme and its
applications in RST invariant digital image
watermarking”

University of Tsukuba -- 12/16/2008

Fourier transform – shift invariance2
f ( x  a, y  b)    F (u, v) exp(  j(au  bv))
N

University of Tsukuba -- 12/16/2008

Fourier transform -- rotationf r ( x, y)  f (( x cos a  y sin a), ( x sin a  y cos a))

Fr (u, v)  F ((u cos a  v sin a), (u sin a  v cos a))

University of Tsukuba -- 12/16/2008

Fourier transform -- scaling1u v
f (ax, by)   F( , )
ab a b

University of Tsukuba -- 12/16/2008

Log-polar mapping•

Using log-polar coordinates:u  e  cos v  e  sin •

The magnitude of the Fourier spectrum:

I ' (  , )  •

University of Tsukuba -- 12/16/2008

2

I (   ln  ,   )Advantage:– Image scaling results in a translational shift of lnσ along the log-radius ρaxis.
– Image rotation results in a cyclical shift of α along the angle θ axis.
– Image translation has no effects in LPM domain.

University of Tsukuba -- 12/16/2008

3

2008/12/17

Log-polar mapping (example)

Matching template•

We cut a small block as a matching template from the LPM
domain or the spatial domain of the original image.

Original
image

DFT

Cut a
template

LPM

optional
DFT: discrete Fourier transform
LPM: log-polar mapping

Cut atemplate



y
x

•


Cartesian domain

The key technique is to match the template in the log-polar domain with thewatermarked image having undergone RST attacks.

LPM domain

University of Tsukuba -- 12/16/2008

University of Tsukuba -- 12/16/2008

The matching results of the five filters

Five traditional filters•

Classical matched filter

G( ,  )  AG ( ,  )e••(without rotation or scaling)
Binary phase-only filter

GBPOF ( ,  )  e j BPOF (  , )Amplitude-only filter

where

GA ( ,  )  AG ( ,  )

 00 ,
 BPOF ( ,  )   0
180 ,Gr  0Gr  0(a) Classical matched filter(b) Amplitude-only filter(c) Inv erse filter

Inverse filter
GI ( ,  ) • j G (  , )• j G (  , )Gr stands for the real part of
the Fourier transform G (  ,  )e
AG ( ,  )Phase-only filterG ( ,  )  e j G (  , )(d) Phase-only filter

University of Tsukuba -- 12/16/2008

The matching results of the five filters(with rotation and scaling)

Phase-only filtering method•••(g) Classical matched filter(e) Binary phase-only filter

University of Tsukuba -- 12/16/2008(h) Amplitude-only filter(i) Inverse filter

Phase information is more important than the amplitude
information in preserving the visual intelligibility.
Correlation detection is only optimal in the case that the signal
can be modeled as additive white Gaussian noise.
Using only the phase information of the matching template
and the LPM spectrum of the watermarked image undergone
RST transformations.

r  IFFT [ F ( ,  )  G ( ,  )]
*

where

F ( ,  )  e
G ( ,  )  e(j) Phase-only filter

University of Tsukuba -- 12/16/2008

 j F (  , ) j G (  , )(k) Binary phase-only filter

University of Tsukuba -- 12/16/2008

42008/12/17

Rectification(rotation and scaling parameters)

The matching results of our method••

Compute the cross-correlation between the template and thewatermarked image in the LPM domain.
Suppose the coordinates of the peak is ( 1 ,1 ) . We know theoriginal position of the template is ( 0 ,0 )The translations in the LPM domain are:•    1  0
    1  0
The rotation and scaling parameters in the spatial domain are:•(a) Without rotation or scaling
360 0    ' N'ln( rmax )  M' ' e(b) With rotation and scaling

University of Tsukuba -- 12/16/2008

University of Tsukuba -- 12/16/2008

Scaling without rotation

1(a) Rotated by 45 degrees counterclockwise without scaling(c) Scaled by 0.72(e) Rotated by 45 degrees counterclockwise after being scaled by
0.7062

1 Application 1: Spatial domain
2 Application 2: Fourier domain
3 Application 3: LPM domain(b) Correlation with (a). The peak is(d) Correlation with (c). The peak is
at (185,371).
at (250,401).
University of Tsukuba -- 12/16/2008(f) Correlation with (e). The peak is
at (186,401).

3

University of Tsukuba -- 12/16/2008* The original template position is (250,370)

Rotation with cropping

1

3

Scaling and rotation

2

University of Tsukuba -- 12/16/2008

1

3

2

University of Tsukuba -- 12/16/2008

5

2008/12/17

Image quality evaluation based onwatermarking [7]JPEG compression

1

2

3

University of Tsukuba -- 12/16/2008

University of Tsukuba -- 12/16/2008

Limitation of existing quality metrics

Aim of developing watermarking based
quality metric••

A common limitation of the widely used quality metrics, such as
PSNR, wPSNR and Watson model: need access to the originalwork– All of them are based on point-to-point difference calculation
between the original work and degraded work in spatial
domain or in frequency domain– This definitely results in big inconveniences at the receiverside in a communication system when the original work isreal-time signals such as TV signals

To find a quality estimation method which can accurately
estimate image/video qualities without the access to the original
image/video– Digital watermarking based metric can be a good candidate
since it can evaluate the image/video quality by simply
evaluating the degradation of the extracted watermark– Watermark is much smaller than the original work, which iseasier to transmit and makes it possible for real-time quality
evaluation

University of Tsukuba -- 12/16/2008

University of Tsukuba -- 12/16/2008

Main idea of developing watermarkingbased quality metric

Main idea of developing watermarking based
quality metric••

The proposed scheme is based on DWT•The different watermark embedding vulnerability for different coverwork

Use the degradation of watermark to evaluate the quality
degradation of the cover work
– The watermark is invisibly embedded throughout the coverwork– The watermark is inseparable from the cover work, whichmeans the watermark will undergo the same transformations
and distortions as the cover work
– The watermark can be embedded with suitable vulnerability
so that the degradation of the watermark can reflect thequality degradation of the cover work

University of Tsukuba -- 12/16/2008– Watermark embedding bits
• Middle frequency• High/low frequency– Watermark embedding strength•

Need to:– Find a balance between robustness and fidelity– Improve accuracy of the results

University of Tsukuba -- 12/16/2008

6

2008/12/17

Automatic control system designed for thewatermarking scheme

The proposed watermarking-based image
quality evaluation scheme
Original image
Feedforward control

Empirical
information

Feedforward
control

3-level DWTInitial quantization
parameters

Emprical watermark
portionController
Feedback control

Reference
input+

Controller

Implementer

Measured
outputControlled process

Quantization parameter
adjuster

Adjusted quantization
parameters

Watermark embedder

3-level IDWT

Increment/Decrement

Watermarked image

The quantization parameter(s)needed to be adjusted

distortion-

Degraded watermarked image

The most contributing
block(s)Disturbances
Original
watermark3-level DWTClassic quality metric

Watermark extractor

Calculated quality

Gradient calculator

SensorN++

TDR

Acceptable ?

Estimation error

Ideal curve

Y

Estimated quality

MappingFinished

University of Tsukuba -- 12/16/2008

University of Tsukuba -- 12/16/2008

Quantization method employed by thewatermark embedder

Feedback control

The watermark extraction and quality
measurement
Degraded watermarked
image

Original position before
embedding

Quantization
coefficients

0

-5


1

Q ( e)  
0



1

-4

0

-3

1

-2

0

-1

1

0

0

1

Final position
after embedding

1

2

3

0

1

A 4

B 5

3-level DWT

Original watermark

TDR

Ideal curve

Mapping

TDR (True Detection Rate) for the watermark degradation

The watermarking based quality
measurement -- ideal curve & mapping
18

16

16

14

14

12

12

10

10

JND

18

8

8

6

6

4

4

2

TDR 

Number of successfully detected watermark bits
Number of watermark bits used for embedding

0
0.2

2

0.3

0.4

0.5

0.6
TDR

0.7

0.8

0.9

1

0
0.2

0.3

0.4

0.5

0.6
TDR

0.7

0.8

0.9

1

Linear interpolation:
T(j)

TC

T(j+1)Q(j)QE

Q(j+1)

QE  Q ( j ) University of Tsukuba -- 12/16/2008

Qualityevaluation

University of Tsukuba -- 12/16/2008

JND

The image/video quality will be estimated by evaluating the degradation of thewatermark

Watermarkextraction

Estimated quality

The watermarking based quality
measurement•

Watermark extractor
DWT coefficien t  even quantizati on parameter 
DWT coefficien t  odd
 quantizati on parameter 

University of Tsukuba -- 12/16/2008•

Quantization parameterTC  T ( j ) Q ( j )  Q ( j  1)
| T ( j )  T ( j  1) |Mean Absolute Error (MAE) Q (i)  Q (i)
N

MAE 

i 1

C

E

N

University of Tsukuba -- 12/16/2008

7

2008/12/17

The principle behind the adjustment ofwatermark vulnerability

JND

Ideal curve

Ideal
curve

JND

Evaluation --- Original watermark

UpperboundsTestedresultP

Adjusted
resultLowerboundsThe 32 by 32 original watermark
Tested curve

0

B

A

TDR

0

CD E

TDR

The estimation errors on the sample distortions were reduced at the same time

University of Tsukuba -- 12/16/2008

University of Tsukuba -- 12/16/2008

Ideal curve used for estimating PSNR

Experiment result 1 – estimating PSNR
60

60

55

55

50
Calculated PSNR (dB)Calculated PSNR (dB)

50
45
40
35

45
40
35
30

30

25

25
20
20

25

30

35
40
45
Estimated PSNR (dB)

50

55

60

20
20

25

MAE = 2.4733 dB

University of Tsukuba -- 12/16/2008

35
40
45
Estimated PSNR (dB)

50

55

60

MAE = 0.7877 dB

University of Tsukuba -- 12/16/2008

Experiment result 2 – estimating
wPSNR
75

75

70

70

65

65

Calculated wPSNR (dB)Calculated wPSNR (dB)

Ideal curve for estimating wPSNR

60
55
50
45
40

30
30

60
55
50
45
40

35

35

35

40

45
50
55
60
Estimiated wPSNR (dB)

65

70

75

MAE = 2.2982 dB

University of Tsukuba -- 12/16/2008

30

30
30

35

40

45
50
55
60
Estimiated wPSNR (dB)

65

70

75

MAE = 0.7967 dB

University of Tsukuba -- 12/16/2008

8

2008/12/17

Experiment result 3 – estimating
Watson JND

Ideal curve for estimating Watson JND
20

20

18

18

16

16

14

14

16

Calculated JND

14

JND

12
10

Calcualted JND

18

12
10
8

12
10
8

6

6

4

4

8
6

0

4
2
0
0.2

2

2

0

0

2

4

6

8
10
12
Estimated JND

14

16

18

20

0

2

MAE = 1.5608 JND
0.3

0.4

0.5

0.6
TDR

0.7

0.8

0.9

4

6

8
10
12
Estimated JND

14

16

18

20

MAE = 0.5440 JND

1For JND  8 , MAE = 0.4057 JND
University of Tsukuba -- 12/16/2008

University of Tsukuba -- 12/16/2008

Speech quality evaluation based onwatermarking [8]Watermarking scheme

University of Tsukuba -- 12/16/2008

University of Tsukuba -- 12/16/2008

Watermark bit embedding

Discrete wavelet transform

CoefficientLevelsSignal

Low Pass

Filters

SpeechHigh Pass

DWT

Blocking

Level 1

Block 1

Level 2Block 2

Coefficient
Selection……

Select first 50 largest
coefficients to embedwatermark……
Level mcA1

Block ncD1

Quantization -- Embed one bit (0 or 1) on each coefficient
Low Pass

Filters

cA2

University of Tsukuba -- 12/16/2008

High Pass

cD2

University of Tsukuba -- 12/16/2008

92008/12/17

Experimental results

Experimental results

4.5

4.5
4.5

3
2.5
2
1.5
1
1

1.5

2

2.5

3

3.5

4

4.5

4
3.5
3
2.5
2
1.5
1
1

1.5

DWMOS
(a) MP3 compression (100) samples

2

2.5

3

3.5

4

4.5

4
3.5
3
2.5
2
1.5
1
1

DWMOS
(b) Gaussian noise

ITU-T (P.862) PESQMOS

3.5

ITU-T (P.862) PESQMOSITU -T (P.862) PESQMOSITU-T (P.862) PESQMOS

4.5
4

1.5

2

2.5

3

3.5

4

4.5

2
1.5
1
0.5
0
-0.5
-0.5

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

DWMOS

(100) samples(c) Low-pass filtering (80) samples

Research in process
The watermarking-based image quality algorithm is applying to
the MPEG-2 / H.264 video codec.
The research on quality estimation against noise pollution,
packet loss, and other signal processing are in process.

•The experiment to simulate subjective visual quality is underconsideration.

•

NSERC Strategic Project: Quality evaluation and enhancementof audiovisual signals based on digital watermarking

University of Tsukuba -- 12/16/2008

Selected Journal Papers(RST invariant image watermarking)

1.
2.
3.
4.
5.

6.

University of Tsukuba -- 12/16/2008

D.Zheng, S.Wang, and J.Zhao, An RST invariant image watermarking algorithm withmathematical modeling and analysis, to appear in IEEE Transactions on Image Processing.
D.Zheng, Y.Liu, J.Zhao and Abdulmotaleb El Saddik, A Survey of RST Invariant Image
Watermarking Algorithms, ACM Computing Surveys. Vol. 39, No. 2, Article 5, pp. 1-91, June
2007.
Y.Liu, D.Zheng, and J.Zhao, An image rectification scheme and its applications in RST
invariant digital image watermarking, Kluwer Journal: Multimedia Tools and Applications,Vol. 34, No. 1, pp. 57-84, July 2007.
D.Zheng, Y.Liu, and J.Zhao, RST invariant digital image watermarking based on a newphase-only filtering method, Elsevier Journal: Signal Processing, Vol.85, No.12, pp.23542370, December 2005.
Y.Liu, D.Zheng, and J.Zhao, A Rectification Scheme for RST Invariant Image
Watermarking, IEICE Transactions on Fundamentals of Electronics, Communications andComputer Sciences, Special Section on Cryptography and Information Security, Vol. E88-A,
No.1, pp. 314-318, January 2005, LETTER.
D.Zheng, J.Zhao, and A.El Saddik, RST Invariant Digital Image Watermarking Based on
Log-Polar Mapping and Phase Correlation, IEEE Transactions on Circuits and Systems forVideo Technology, Special Issue on Authentication, Copyright Protection and Information
Hiding, Vol. 13, Issue 8, pp. 753-765, August 2003.

University of Tsukuba -- 12/16/2008

Selected Journal Papers

Acknowledgement

7.

•(quality evaluation based on digital watermarking)Sha Wang, Dong Zheng, Jiying Zhao, Wa James Tam, and Filippo Speranza,
An image quality evaluation method based on digital watermarking, IEEE
Transactions on Circuits and Systems for Video Technology, Vol.17, No.1,
pp.98-105, January 2007.
8. Libin Cai, Ronghui Tu, Jiying Zhao, and Yongyi Mao, Speech quality
evaluation: a new application of digital watermarking, IEEE Transactions on
Instrumentation and Measurement, Vol. 56, No. 1, pp. 45-55, February 2007.
9. Sha Wang, Dong Zheng, Jiying Zhao, Wa James Tam, and Filippo Speranza,
An Accurate Method for Image Quality Evaluation Using Digital Watermarking,
IEICE Electronics Express (ELEX), Vol.2, No.20, pp.523-529, October 2005.
10. Libin Cai and Jiying Zhao, Evaluation of speech quality using digitalwatermarking, IEICE Electronics Express (ELEX), Vol.1, No.13, pp. 380-385,
October 2004.

(d) Packet loss (100) samples

DWMOS --- predication of MOS

University of Tsukuba -- 12/16/2008

•

3
2.5

DWMOS

DWMOS --- predication of MOS

•

4
3.5

Graduate students– Dong Zheng (RST invariant image watermarking, image quality
evaluation)•– Yan Liu (RST invariant image watermarking)– Ronghui Tu (audio watermarking, speech quality evaluation)– Libin Cai (speech quality evaluation)– Sha Wang (image/video quality evaluation)
Colleagues– Dr. Wa James Tam (Communications Research Centre Canada(CRC))– Dr. Filippo Speranza (CRC)

University of Tsukuba -- 12/16/2008

University of Tsukuba -- 12/16/2008

10

2008/12/17

Acknowledgement
井上 智雄 博士
筑波大学井上研究室
Research Center for Knowledge Communities
知的コミュニティ基盤研究センター

University of Tsukuba -- 12/16/2008

University of Tsukuba -- 12/16/2008

11