AUDIO SIGNATURE EXTRACTION BASED ON PROJECTIONS OF SPECTROGRAMSRegunathan Radhakrishnan, Claus Bauer, Corey Cheng and Kent Terry
Dolby Laboratories Inc
100 Potrero Ave, San Francisco, CA
Email: {regu.r,cb,cnc,kbt}@dolby.com
ABSTRACT
Content-based signatures are designed to be a robust bitstream representation of the content so as to enable content
identi¿cation even though the original content may go throughvarious signal processing operations. In this paper, we propose a novel content-based audio signature extraction methodthat captures temporal evolution of the audio spectrum. The
proposed method, ¿rst, divides the input audio into overlapping chunks and computes a spectrogram for each chunk.
Then, it projects each of the spectrograms onto random basis vectors to create a signature that is a low-dimensional bitstream representation of the corresponding spectrogram. Our
experimental results show the robustness and sensitivity of the
proposed content-based audio signature extraction method forvarious signal processing operations on audio content.
1. INTRODUCTION
The Internet increasingly serves as a platform to publish andsell multimedia content. The piracy of multimedia content in
the Internet creates a need for technologies that can identify
copyrighted content. Content based signature extraction has
been identi¿ed as a candidate technology. As pirates mightpublish multimedia content that has been processed in order to render a non-manual identi¿cation of pirated content
more dif¿cult, the extracted signature should largely remain
the same as the content goes through various signal processing operations. Also, the signature should be compact for theef¿ciency and scalability of the identi¿cation algorithm thatuses the signature.
Past work on content-based audio signature extraction generally consists of the following two steps: the ¿rst step extracts features that represent the underlying content and thesecond step generates a compact bitstream representation of
the features. In [1], Burges et al propose a dimensionality reduction method called Distortion Discriminant analysis (DDA)
to obtain a low-dimensional representation of the log spectrum of the audio signal. In [2], Chun-Shien Lu proposesa audio characterization method based on one dimensional
wavelet transform coef¿cients. In [3], Batlle et al propose
a signature for song identi¿cation from noisy broadcast audio

1-4244-1017-7/07/$25.00 ©2007 IEEE2110

based on MFCC feature extraction followed by Viterbi decoding from a sequence of pre-trained HMMs. In [4], Haitsma et
al propose a robust audio signature based on the fact that thesign of energy differences (simultaneously along the time andfrequency axes) is a property that is very robust to many kindsof processing.
Unlike prior approaches, our proposed method treats thewhole spectrogram as a pattern and extracts signature bits tocapture the temporal evolution of the audio spectrum. It thus
avoids the additional step of extracting features from a spectrogram or spectrum and also can record more information
about the spectrogram. It attempts to represent the underlying
content as a sequence of spectrograms. The proposed method¿rst divides the input audio into overlapping chunks and computes a spectrogram for each chunk. Then, it projects each of
the spectrograms onto random basis vectors to create a signature that is a low-dimensional bitstream representation of thecorresponding spectrogram. This method of projection onto
random vectors was originally proposed for still image hashing in [5]. The concatenation of the sequence of extracted bitsfrom the spectrogram of each chunk serves as the identi¿er forunderlying content.
We present the proposed audio signature extraction method
in section 2 and provide experimental results in section 3.
2. AUDIO SIGNATURE EXTRACTION
The audio signature extraction consists of the feature extraction and the calculation of the signature bits.
2.1. Audio Feature Extraction
The audio feature extraction attempts to represent the underlying content as a sequence of spectrograms. Note that thespectrogram itself captures the temporal evolution of the spectrum of the signal. Towards this end, we divide the input audio into overlapping chunks and create a spectrogram fromeach of the chunks. Then, we create a coarse spectrogramby averaging along both time and frequency. This operation
provides robustness against small changes in the spectrogram
along time and frequency. Note that the coarse spectrogram
created could choose to emphasize certain parts of the spectrum more than others.

ICME 2007

Fig. 1. Audio Feature Extraction
The generation of the coarse spectrogram itself is illustrated in detail in Figure 1. The input audio is ¿rst divided
into chunks of duration Wfk with an overlap of Wr between
adjacent chunks. For each chunk of audio data ([fk ) we compute a spectrogram with certain time resolution and frequencyresolution. Then, we tile the computed spectrogram [fk withtime-frequency blocks. Finally, we sum up the magnitude of
the spectrum within each of the time-frequency blocks to obtain a coarse representation of the spectrogram. Let us represent the spectrogram by S. We obtain a coarse representation (T) of S by averaging the magnitude of frequency coef¿cients in time-frequency blocks of size Zi ×Zw . Here, Zi is
the size of block along frequency and Zw is the size of blockalong time. Let F be the number of blocks along frequency
axis and T be the number of blocks along time axis and hence
T is of size (F*T). T is computed as given below:T(n> o) =1Zi  ZwnZiXoZwXto change slowly with small changes in features. This would
enable us to allow for certain signal processing operations on
the content which do not change the content but only slightly
disturb the features. We use one such robust hash function
proposed in [5] for generating the hash bits from the featurematrix, Q. We generate K random vectors each with the same
dimensions as the matrix, Q (P × Q ). The matrix entries are
uniformly distributed random variables in [0> 1]. The state of
the random number generator is set based on a key. Let us denote these random vectors by S1 > S2 > ===SN each of dimension(P × Q ). We compute the mean of matrix Sl and subtract itfrom each matrix element in Sl (i goes from 1 to K). Then, thematrix Q is projected onto these K random vectors as shown
below:
Kn =P X
QX
l=1 m=1

T(l> m)  Sn (l> m)(1)

Here Kn is the projection of the matrix Q onto the random vector Sn . Using the median of these projections ( Kn ,n = 1> 2> ===N) as a threshold, we generate K hash bits for thematrix Q. We generate a hash bit ’1’ for nwk hash bit if theprojection Kn is greater than the threshold. Otherwise, we
generate a hash bit of ’0’.
3. EXPERIMENTAL RESULTS
3.1. Test Content

V(l> m)l=(n1)Zi m=(o1)Zwn = 1> 2===I ; o = 1> 2===W
Here, i and j represent the indices of frequency and timein the spectrogram and k and l represent the indices of thetime-frequency blocks in which the averaging operation isperformed.
Finally, we create a low-dimensional representation of acoarse spectrogram (T) of the input audio frame by projecting
the spectrogram onto random vectors, which can be thoughtof as basis vectors. The projection onto random vectors isexplained in section 2.2.
2.2. Robust Hash
This block takes as input the matrix Q, and creates the signature by generating K hash bits. We use a robust hash function
for this purpose as small perturbations in the audio featurescaused by signal processing operations such as compression,¿ltering, etc. would not change the hash bits drastically. TheRobust hash function also serves to reduce the bit-rate of thesignature stream. Let us represent the dimensions of the matrix Q by (P ×Q ). By using a robust hash, instead of sending(P × Q ) values we only send few bits. A robust hash function is unlike a regular cryptographic hash function. A cryptographic hash function changes its output for every single bitchange in the input. However, we would like our hash output

The test content used for the performance assessment of theproposed audio signature extraction consisted of 36 one-minuteaudio clips of a variety of content types in various formats.
The audio content included stereo at a sampling rate of 48khz.
Each of the test clips were processed in a variety of ways
to simulate typical signal processing operations. We considered the following processing operations: Downmixing,
Gain modi¿cation by +/-15db, Equalization, Dynamic Range
Compression, Dolby Digital Encoding/Decoding (at 192 and128kbps for stereo), MPEG1 layer II encoding and decoding
at 128 and 256kbps for stereo. Overall, we created 645 test
cases for audio.
3.2. Parameter Settings
The signature extraction parameters for audio were set as described below. The time-frequency resolution for the spectrogram was set to be 10× 20 (W = 10> I = 20). The number ofrandom vectors for the hash was chosen to be 18. The chunksize (Wfk ) was chosen to be 90ms and the step size (Wr ) was
set to be 512 samples at 48khz sampling rate. The comparisonof two audio signatures was based on the hamming distance.
These parameters were chosen empirically based on performance on a wide variety of content.
3.3. Robustness of proposed audio signature
In this section, we present experimental results to show therobustness of the proposed audio signature to various signal

2111

DD128

DRC1

0.8
0.7

limit

Gain+15db

1

1

0.9

0.9

0.9

0.8

0.8

0.8

0.7

0.7

0.6

0.6

0.5

0.5

0.4

0.4

0.3

0.3

0.2

0.2

0.1

0.1

0.7

0.6
0.5

0.6
0.5

0.4

0.4
0.3

0.3

0.2
0.1
0−5

05

1015

20

0−5

05

1015

20

0−5

0.2
0.1

05

1015

20

0
−5

0

5

1015

20

Fig. 2. Left:Robustness against compression Right: Robustness against DRC

Fig. 3. Left:Robustness against Limiting Right: Robustness
against Volume Change

processing operations on the original content. Towards that
end, we ¿rst extract the proposed audio signature from original audio content and then compare that against the signature
extracted from processed (attacked) audio content using hamming distance measure. Finally, we look at the histogram ofhamming distance for each of the following signal processing
operations. A robust signature would have a distribution thatis heavily skewed near hamming distance value of zero.

3.3.3. Equalization

3.3.1. Audio Compression
We performed the following ¿ve compression attacks on theoriginal content: Dolby Digital Encoding and Decoding cycle
at 192kbps (DD192) and at 128kbps (DD128) and MPEG1Layer II Encoding and Decoding cycle at 256kbps (MP1LII256)
and at 128kbps (MP1LII128) and Dolby E encoding and decoding (DE20). Figure 2 shows the distribution of hamming
distances for DD128 compression attack. From the ¿gure, it
can be observed that 1 or 2 bit Àips occur for almost 20% of
the time with DD128 attack. Also, from Table 1, one can observe that MP1LII128 compression attack causes most number of bits to be Àipped in the signature and DD192 causes theleast number of bits to be Àipped in the signature. One canalso conclude that as one allocates more bits for compression,
fewer signature bits change. Another interesting observationis that MPEG1 LayerII compression at 256kbps (MP1LII256)
causes more bitÀips in the signature than Dolby Digital Encoding at 192kbps.
3.3.2. Dynamic Range Compression
Dynamic range compression attacks were evaluated by applying multi-band compression methods to the original content(DRC1 and DRC2). These represented moderate to aggressive compression of dynamic range. Figure 2 shows the distribution of hamming distances for one of the dynamic range
compression attacks (DRC1). The distribution has a weight ofclose to 1 for a hamming distance value of zero. Also, fromtable 1, observe that both DRC1 and DRC2 cause only 0.04%
and 0.05% of the bits to be Àipped.

2112

An equalization attack was tested by applying an equalization
method (consisting of +/-6dB equalization processing acrossvarious frequency bands) to the source content. Figure 4 shows
the distribution of hamming distances for the equalization attack. From the ¿gure, one can observe that close 40% of thetime there can 1 or 2 bit Àips in the extracted signature if
equalization is performed. Also, from table 1, observe thatequalization causes 3.87% of the bits to be Àipped which isone of the highest among the test cases. Since the spectrum
is shaped during equalization, it causes some of the values in
the coarse spectrogram to change. This consequently causes
more percentage of the bits to be Àipped.
3.3.4. Gain Change
A volume change attack was evaluated by applying a +/-15dB
gain to the original content (Gain+ and Gain-). Figure 3 shows
the distribution of hamming distances for volume change attacks. Also, from table 1, observe that positive gain change by
15db causes 1.7% of the bits to be Àipped whereas negative
gain change causes only 0.35% of the bits to be Àipped. Weobserved that a volume change of +15db results in clipping of
the sample values and which in turn can affect the spectrum
and result in more bit changes in the signature than in the caseof volume change by -15db.
3.3.5. Limit
We increased the gain on original audio by 6db ¿rst and thenpassed it through a limiter and ¿nally reduce the gain by 6dbagain. Figure 3 shows the distribution of hamming distances
for this processing (limit). As is shown in table 1, this processing causes only 0.04% of the bits to be Àipped.
3.3.6. Addition of Noise
A noise attack was evaluated by adding a relatively high levelof noise ( -25dB) to the source content. Figure 4 shows thedistribution of hamming distances for addition of noise attack. Also, from table 1, observe that addition of noise causes
4.74% of the bits to be Àipped which is the highest among allattacks. The reason for this is the following. Recall that theEqualization

Noise

0.7

0.7

0.6

0.6

0.5

0.5

0.4

0.4

0.3

0.3

0.2

0.2

0.1

0.1

0.9
pdf of hamming distancefor source file through various
processing

0.8
0.7
0.6
0.5
0.4

pdf of hamming distance for
twodifferent files

0.3
0−5

05

10

15

20

0−5

05

1015

20

0.2
0.1

Fig. 4. Left:Robustness against equalization Right: Robustness against noiseAttack
DD128
DD192
DE20DRC1DRC2Equalization
GainGain+Limit
MP1LII128MP1LII256
NoiseB
53730
24692
23036
1070
1316
90594
8792
37716
1069
79314
45945
116134C
135950135950135924135990135990129756135990122694135990135999135999135978

0−50

5

10

1520

Fig. 5. Comparison of hamming distance histograms

D
0.02190.01000.00940.00040.00050.03870.00350.01700.00040.03240.01870.0474the hamming distance value 8, and has very little overlap withthe other. This means that the proposed signature is sensitive(unique) to the underlying content it represents and canserve as a robust audio content identi¿er.
4. CONCLUSION

Table 1. Robustness of Proposed Audio Signature for various Signal Processing Operations; B: Q xp=ElwHuuruv C:E, Vlj = 18
Q xp=I udphv D:EHU = (VljF)
spectrogram was computed from 100ms chunks of audio. If
the audio source contains lots of silence periods like speechpauses, the spectrograms in these regions change considerably when noise is added. This results in bit errors in thoseparts of the signal. Note, from Figure 4, that the hamming
distance between the original and attacked signature can beas high as 10 for this reason. We are basically comparing the¿ngerprints of the noise signal to silent segments in the original which results in such a high value of hamming distance.
For the aforementioned attacks, we have seen the robustness of the proposed signature for various signal processing
operations in terms of percentage number of bit Àips for eachkind of processing (see Table 1). Now, we would like to see
how sensitive (unique) the proposed signature is to the underlying content it represents. Figure 5 shows the histogram ofhamming distances between signatures of an audio (say clipA) and its various processed versions (clip A compressed anddecompressed, clip A through an equalizer, clip A throughdynamic range compression etc). Note that this histogramis skewed close to 0 showing the robustness to various processing. The ¿gure also shows the histogram of hamming
distances between signatures of two different audio ¿les (sayclip A and another clip B). This histogram is centered around2113

We proposed a novel audio signature extraction method that isrobust to various signal processing operations. The proposed
method divides the input audio into overlapping chunks andcreates a spectrogram from each chunk to capture the temporal evolution of the spectrum. Then, the extracted spectrograms are projected onto random basis vectors to generatesignature bits. Thus, the underlying audio content is represented compactly as a sequence of spectrograms. Our experimental results show that common signal processing operations including compression, equalization etc cause only lessthan 5% of the signature bits to Àip. We also show the sensitivity of the signature by comparing the signatures of twodifferent ¿les. In our future work, we will study the tradeoff between signature bit rate and robustness. We would also
study the effect of time scale modi¿cation attacks on the signatures.
5. REFERENCES[1] Burges, C.J.C. Platt, J.C. and Jana, S, “Distortion discriminant analysis for audio ¿ngerprinting,” IEEE Transactions on Speech and Audio Processing, May 2003.
[2] Chun-Shien Lu, “Audio ¿ngerprinting based on analyzing time-frequency localization of signals,” Proc. ofMMSP, May 2002.
[3] E.Batlle, J.Masip and E. Guaus, “Automatic song identi¿cation in noisy broadcast audio,” Proc. of SIP, Aug
2002.
[4] J.Haitsma and T.Kalker, “A highly robust audio ¿ngerprinting system,” Proc. of ISMIR, 2002.
[5] J.Fridrich and M.Goljan, “Robust hash functions for digital watermarking,” Proc. of ITCC, May 2000.