VIDEO FINGERPRINTING BASED ON MOMENT INVARIANTS CAPTURING
APPEARANCE AND MOTION
Regunathan Radhakrishnan and Claus Bauer
Dolby Laboratories Inc, 100 Potrero Ave San Francisco CA 94103
ABSTRACTIn this paper, we propose two video ﬁngerprinting methods
that are robust to both geometric and non-geometric modiﬁcations on content. Both of the proposed methods are basedon computation of moment invariants as features from concentric circular regions. The two methods differ in the way
they capture appearance and motion information from video.
In one method, we capture motion information by computing a difference image between the current video frame and atemporal average video frame computed from a past windowof video frames. This method captures appearance by computing moment invariants from concentric circular regions ofa video frame. In the second method, we capture appearance and motion by projecting features onto two sets of basis
functions and explicitly capture how the moment invariants
change over the regions and over time. We present experimental results on both of these video ﬁngerprinting comparing their performance in terms of robustness against attacks
and sensitivity to content.
Index Terms— Moment Invariants, Video ﬁngerprinting,
SVD

and motion information. Moment invariants are global measures of the image surface that are robust against translation,
rotation and scale change attacks and were originally proposed by Hu in 1962 for recognition of characters [5].The
proposed methods differ from [1],[2],[3] in the fact that they
do not perform any speciﬁc transform on input video. Unlike
the local features based method proposed in [4], the proposed
methods derive only one ﬁngerprint codeword per video
frame. We present experimental results on a 150hr databaseand compare their performance in terms of robustness andsensitivity.
2. PROPOSED METHODS BASED ON MOMENTINVARIANTS
In this section we describe the proposed two video ﬁngerprinting methods that capture appearance and motion in different ways. Both of the methods are based on moment invariants as features extracted from concentric circular regions.
We will ﬁrst describe the parts of the processing blocks thatare common to both of these methods. Figure 1 illustrates theprocessing blocks in the proposed methods.

1. INTRODUCTION
Video ﬁngerprinting methods that are robust against both nongeometric and geometric attacks has been an active area of
research in the recent years. Past work in this area can bebroadly classiﬁed into two approaches. The ﬁrst class of approaches transform the input video data into a transform domain (e.g Radon transform, Fourier Mellin Transform) thatis invariant under geometric operations before extracting robust features from this domain[1],[2],[3]. The second classof approaches represent the video frame as a collection of local features such as SIFT [4]. The robustness of this classof methods is due to the redundancy introduced by the multiple ﬁngerprint codewords derived per video frame from corresponding local features. For instance, cropping attack may
get rid of some of the local features but there is sufﬁcient information in the remaining local features for successful identiﬁcation.
In this paper, we propose two video ﬁngerprinting methods that are based on moment invariants. The two methods
differ from one another in the way they capture appearance

978-1-4244-4291-1/09/$25.00 ©2009 IEEE

1532

Fig. 1. Proposed Video Fingerprinting Methods.
Step 1: The input video is ﬁrst temporally downsampled
to a reference frame rate. This makes comparison of signatures easier when the original video and the processed video
do not have the same frame rate. This means that we extract
signature at certain time interval (Tint ) only. Let Fj represent

ICME 2009

the closest video frame for time step ’j’. The signature fortime step ’j’ is then extracted based on features from Fj .
Step 2: The frame Fj is downsampled to reference spatial resolution say, (120*160). This helps in dealing with spatial resolution changes. The registration between the originalvideo and spatially scaled video is not disturbed as long as
the aspect ratio is not changed. Then, we perform letterbox
detection and removal from the input frame Fj .
Step 3: A sub-image is cropped out from the downsampled Fj image. This region is selected so as to allow for text
overlay in a portion of the original picture and to allow forplacement of logo or graphics around the corners. The ﬁngerprint of the processed video will not be affected as long as theselected region does not contain any new graphics content.
Let us represent this image as Fjc .
Step 4: A low-pass ﬁltering operation is performed on
Fjc to improve the robustness of extracted features. A simple
low-pass ﬁlter using the average of 3*3 neighborhood of thecurrent pixel could be used. Let us represent the low passﬁltered image by FjL .
These are the preprocessing steps common to both of the
proposed methods. In the following subsection, we describe
the remaining steps (5a-8a) in the ﬁrst proposed method “MIdiff”. This method captures captures appearance by computing moment invariants as features from concentric circular regions. It captures motion by computing a difference image
between the current frame and a temporal average frame computed from a past window of frames.
2.1. Method 1:MI diff
Step 5a:The low-pass ﬁltered image FjL captures the appearance of the current frame. In order to capture information
about motion of objects in the video, we create a difference
image Dj . The difference image is derived according to:
Dj = Aj (k, l) − FjL (k, l);k = 1, 2...H; l = 1, 2...W ;Here
W is the width and H is the height of FjL . In our implementation, W = 160 and H = 120. And Aj is the temporal
average image obtained by averaging pixels from a windowof past ’T’ decoded frames. Aj is computed according tojAj (k, l) = T1 i=j−T FiL (k, l). The motivation for computing Dj as a difference between the current frame (FjL )and a temporally averaged image (Aj ) is the following. If
the difference is computed from just one previous decodedframe then the values of Aj are susceptible to change underframe rate conversion attacks. The temporal averaging operation prevents dependence on just one frame. At the end of thisstep, we have two matrices from which features are to be extracted: one capturing the appearance of the current decodedframe (FjL ) and another capturing the motion aspects of thecurrent frame (Dj ).
Step 6a: In this step, we create regions in the two matrices(FjL and Dj ) before extracting features from each of the regions. Figure 2(a) shows an example of a case where we haveﬁve concentric regions R1 , R2 ...R5 with radii r1 , r2 , ...r5 re-1533

spectively. We pick the inner region R1 and select the otherregions such that area increments are equal to the area of region R1 . One advantage of these concentric circular regions
is that the content of the image within each region stays thesame irrespective of the amount of rotation.
Step 7a: In this step, we extract ’M’ features from each of
the ’N’ regions demarcated in the previous step. The featurematrix derived from the regions of FjL is denoted as FjM . FjL
is of dimension H ×W and FjM is of dimension N ×M where
H and W represent the height and width of the downsampled
video frame and N and M represent the number of regions
and the number of features extracted from each region. Similarly, the ’M’ features are extracted from ’N’ regions of thematrix Dj to create DjM . The robustness of the extracted ﬁngerprints depend on the robustness of the M features extracted
from each of the regions. We compute a set of seven moment
invariants as semi-global features from these regions that arerobust against translation, rotation and scale change attacks.
Please see [5] for details on moment invariants.
Step 8a: In this step, we have two input matrices FjM
and DjM each representing how the extracted features fromeach region change as one proceeds from the inner region R1
to the outer region RN . The signature generation procedure
is identical for both matrices (FjM and DjM ). Therefore, we
explain this bit extraction procedure for one of them, say, FjM .
In order to generate K1 bits from FjM , we ﬁrst create K1
vectors (P1 , P2 ...PK1 ) that have the same dimension as FjM .
The matrix FjM is projected onto this set of K1 vectors asshown in the equation below:
Hk =

M
N 


Q(i, j) ∗ Pk (i, j)

i=1 j=1

Here, the matrix Q is either FjM or DjM and M is thenumber of features per region and N is the number of regions.
The signature bits are then derived by thresholding the K1
projections [6].
Similarly, we create K2 hash bits from DjM bits. Then,ﬁnally our ﬁngerprint is of length (K1 + K2 ) bits and is theconcatenation of the two sets of hash bits from FjM and DjM .
The pseudo-random matrices for generating K1 hash bits andthe pseudo-random matrices for generating K2 hash bits aredifferent. In our implementation, K1 and K2 were set to be18.
2.2. Method 2:MI-row-col-basis
In the previous subsection, we outlined the processing steps
that capture appearance and motion for “MI diff” method.
Here we describe the remaining processing steps (5b-7b)that capture appearance and motion for “MI-row-col-basis”
method.
Step 5b:In order to capture appearance explicitly from FjL ,we ﬁrst compute M moment invariants from N concentric
circular regions to obtain FjM . FjM is a N × M matrix with

(a)

(b)

Fig. 2. a: Concentric regions from a frame b: SVD basis
patterns for capturing appearance
N rows representing the N regions and M columns representing the M moment invariants. In our implementation N = 5
and M = 7. In a second step, we transform the columnsof this matrix using a set of Basis vectors (B1 , B2 , ...BN ) to
explicitly capture how the moment invariants change as we
proceed from the inner region (R1 ) to the outer region (RN ).
The Basis vectors (B1 , B2 , ...BN ) are obtained using SVD(Singular Value Decomposition) of a training dataset ofﬂine.
Notice from Figure 2(b), that the basis vectors capture different patterns. For example B1 captures a monotonicallyincreasing pattern of the feature. Therefore, if a feature in
FjM matrix has this monotonically increasing pattern then its
projection onto this basis B1 would be the highest and allother projections would be small. Thus by transforming thecolumns of the matrices FjM using such a basis, we capture
the patterns of feature variation from region R1 through RN
directly. Let us represent the transformed Matrix by FjT .
Step 6b:In order to capture motion information explicitly,
we ﬁrst vectorize FjT and ﬁll a buffer Mj containing vectorized FjT from a window of ’G’ past video frames. Mj is
G × 35 where the 35(5 × 7) elements are from vectorized
FjT and there are G of them from corresponding G frames. In
a second step, we transform the rows of this matrix using a
set of Basis vectors (V1 , V2 , ...VG ) to explicitly capture howthese 35 features vary over frames 1 through G. Similar to thevectors (B1 , B2 , ...BN ), the Basis vectors (V1 , V2 , ...VG ) are
obtained using SVD of a training dataset ofﬂine.Let us represent the transformed matrix by MjT .
Step 7b:Finally, we create hash bits from MjT by projecting it onto (K = 36) pseudo-random matrices (P1 , P2 , ...PK )and thresholding the projections. This step is similar to thelast step in “MI-diff” method[6].
3. EXPERIMENTAL RESULTS
In this section, we present experimental results comparing theperformance of the proposed two approaches (“MI diff” and“MI-row-col-basis”) for capturing appearance and motion using moment invariants as features.

1534

We created a 150hr database of ﬁngerprints from a datasetof reference videos using both methods. We created modiﬁed versions of some of the reference clips to be used as
query videos. The modiﬁcations included non-geometric attacks such as compression, spatial scaling, frame-rate conversion and also the geometric attacks such as rotation, aspect ratio conversion and cropping. The number of hours ofquery videos was about 55hrs and was used to illustrate therobustness of the proposed methods against both geometricand non-geometric attacks. We also set aside about 22hrs ofcontent that was not part of the reference database to illustrate
the sensitivity of the proposed methods to content.
First, we present the results on the sensitivity property ofthe two methods. From the 55hrs of modiﬁed query video
ﬁngerprints, we performed close to 300,000 queries for every 8s of video. For every query ﬁngerprint, we record thepercentage of bit errors (BER) between the query ﬁngerprint
and matching ﬁngerprint in the reference database. Then,
we compute the probability distribution of the BER for thisdataset. Let us denote this distribution as “IN DB pdf”. Now,
we perform close to 100,000 queries from the 22 hrs of content that is not part of the reference database of content andagain record the BER between the query ﬁngerprint and closest matching ﬁngerprint in the reference database for this experiment as well. Then, we compute the probability distribution of the BER for this dataset. Let us denote this distribution
as “NOT IN DB pdf”.
MI diff BER distribution

MI−row−col−basis BER distributions

0.012

0.014
IN DB pdf
NOT IN DB pdf

IN DB pdf
NOT IN DB pdf
0.012

0.01

0.01
0.008
0.008
0.006
0.006
0.004
0.004
0.002

0

0.002

0

0.05

0.1

0.15

(a)

0.2

0.25

0.3

0.35

0

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

(b)

Fig. 3. Comparison of BER distributions for MI diff (a) andMI-row-col-basis(b).x-axis:BER y-axis:density
Figure 3 compares the “IN DB pdf” and “NOT IN DB
pdf” for both “MI diff” and “MI-row-col-basis”. Notice thatin both cases the two pdfs have little overlap. The BER of
modiﬁed versions are smaller than the BER of content thatis not part of the reference database. This implies that byselecting a BER threshold one can compute the probability ofmiss (PM ) and the probability of false alarm (PF ) for thesemethods. In case of “MI-row-col-basis”,for a BER thresholdof 0.2, PM = 1.98% and PF = 0.06%. In case of “MI-diff”,
for a BER threshold of 0.15,PM = 1.49% and PF = 2.47%
Based on these measures, we can see that “MI-row-colbasis” outperforms “MI diff” in terms of sensitivity. This may
be attributed to the fact that “MI-row-col-basis” uses 3s worth

of video to capture motion information whereas “MI diff”
uses only 1s worth of video for the same. Both approaches
perform in a similar fashion in terms of robustness as theirprobability of miss is comparable. “MI-row-col-basis” requires a total of 7.5s of query video for each query to performthe matching (54 codewords of ﬁngerprint at 12fps and 3s ofvideo for motion information). On the other-hand, “MI diff”
requires only 5.5s of query video to perform the matching (54codewords of ﬁngerprint at 12fps and 1s of video for motion
information).
Second, we present the results on the robustness property of the two methods. From the 55hrs of modiﬁed queryvideo ﬁngerprints, we performed close to 300,000 queries forevery 8s of video. For every query ﬁngerprint, we record
the percentage of bit errors (BER) between the query ﬁngerprint and matching ﬁngerprint in the reference database. Table 1 presents the comparison of the performance of two approaches in terms of average BER for non-geometric attacks(compression, spatial scaling, frame rate conversion), rotation
attacks (2,3,10 and 45 degrees of rotation) and aspect ratio
conversion attacks (4:3 to 16:9 aspect ratio change).
Table 1. Average BER for Approach 1 and Approach 2
in case of non-geometric attacks, rotation and aspect ratio conversion; A:Non-Geometric Attacks; B:Aspect ratio
C:Rotation
A

B

C

MI-row-col-basis

0.07280.18260.1323MI diff

0.06030.11500.0832

Note that in terms of robustness against Non-geometricattacks as well as rotation and aspect ratio conversion, “MIrow-col-basis” and “MI diff” perform in a similar fashion.
Although, “MI diff” has lower average BER in all cases, theBER for “MI-row-col-basis” in all these attacks is smaller
than the chosen threshold of 0.2. Next, we present the performance of both the approaches for another geometric attack(cropping) in Table 2. Here, we record the average BER for
increasing amount of cropping from the edges of a picture.
In this Table, the cropping amount is expressed in terms ofπ(r22 −r12 ). Here r2 is the radius that covers thepercentage as πr22whole picture and the region between r2 and r1 is the region
that is cropped out. As one increases the cropping percentage
from 0% to 14.44% and all the way up to 55.55%, the average
BER for both approaches does not degrade gracefully. This is
as expected because both approaches attempt to capture how
moment invariants change from the inner circular region tothe outer circular region. For the cropped video, the boundaries of these circular regions are at different locations thanthey were for the original reference video clip.
Unlike these two methods, if the ﬁngerprint were derived
from a list local features identiﬁed from a decoded video
frame as in [4], then there would be many redundant entries

1535

in the database for a particular video frame. This would
translate into more robustness against cropping as some of
the local features are likely to survive the cropping operation.
Table 2. Degradation of average BER for cropping attacks(C: cropping ratio; M1: MI-row-col-basis; M2: MI diff)C00.14440.30550.43750.5555M1

0.07990.21990.28050.29050.2736M2

0.04630.12380.18070.22010.2487

4. CONCLUSION
We proposed two video ﬁngerprinting methods that are basedon moment invariants computed from concentric circular regions. The methods differ in the way they captured motion
and appearance information.In one method, we capture motion information by computing a difference image between
the current video frame and a temporal average video frame
computed from a past window of video frames. This method
captures appearance by computing moment invariants fromconcentric circular regions of a video frame. In the second
method, we capture appearance and motion by projecting
features onto two sets of basis functions and explicitly capture how the moment invariants change over the regions andover time. Based on our experimental results, we conclude
that both the approaches are robust against geometric andnon-geometric modiﬁcations on content. Their performanceagainst cropping attack is not as robust as the performance ofa local features based method.
5. REFERENCES[1] J.S.Seo, J.Haitsma, T.Kalker and C.D.Yoo, “A robust
image ﬁngerprinting system using the radon transform,”
Signal Processing:Image Communication, vol. 19, pp. pp
325–339, 2004.
[2] A.Swaminathan, Y.Mao, and M.Wu, “Robust and securehashing for images,” IEEE Trans. on Info. Forensics andSecurity, vol. 1, pp. 215–230, 2006.
[3] R.Radhakrishnan and C.Bauer,“Robust video ﬁngerprinting based on subspace embedding,” Proc. of
ICASSP, 2008.
[4] G.Willems, T.Tuytelaars, L.V.Gool, “Spatio-temporalfeatures for robust content-based video copy detection,”
Proc. of ACM MM, 2008.
[5] Hu,M.K, “Visual pattern recognition by moment invariants,” IRE Trans. Info. Theory, vol. IT-8, pp. 179–187,
1962.
[6] J.Fridrich and M.Goljan, “Robust hash functions for digital watermarking,” ITCC, 2000.