Presentation of Information Synchronized with the Audio Signal
Reproduced by Loudspeakers Using an AM-based Watermark
Akira Nishimura
Department of Media and Cultural Studies, Faculty of Informatics,
Tokyo University of Information Sciences
1200–1, Yatoh-cho, Wakaba-ku, Chiba-city, Chiba, JAPANakira@rsch.tuis.ac.jp

Abstract
Reproducing stego audio signal via a loudspeaker anddetecting embedded data from a recorded sound from a
microphone are challenging with respect to the application of data hiding. A watermarking technique using
subband amplitude modulation was applied to a system that displays text information synchronously withthe watermarked audio signal transmitted in the air.
The robustness of the system was evaluated by a computer simulation in terms of the correct rate of datatransmission under reverberant and noisy conditions.
The results showed that the performance of detection
and the temporal precision of synchronization were sufficiently high. Objective measurement of the watermarked audio quality using the PEAQ method revealed
that the mean objective difference grade obtained from100 watermarked music samples exhibited an intermediate value between the mean ODGs of 96-kbps and
128-kbps MP3 encoded music samples.

payload than copyright management systems.
The author has developed a new audio watermarking technique based on subband amplitude modulation.
Evaluation of this watermarking system was conducted
for data hiding in music [4] and in speech signals [6].
The system is robust against perceptual audio codings,
additive Gaussian noise, spectral modifications, reflections, and reverberations. Deterioration of the soundquality resulting from embedding is relatively small because modulation masking and modulation detection
interference occur in the human auditory system.
In the present paper, a technique for displaying
embedded data synchronously with the watermarked
sounds reproduced by loudspeakers is presented. The
technique can be used to display Karaoke lyrics or captions of foreign films synchronously with the music orthe sounds in the film. The robustness of the watermark with respect to background noise and reflections
caused by air transmission is examined. The objective
audio quality for the watermarked audio signals is also
evaluated.

1. Introduction
2
Applications of audio watermarking are not restricted to the area of copyright management. Anotherapplication of audio watermarking is the inclusion ofaugmentation data[2]. For example, if an announcement or music broadcasted from loudspeakers in a public area could contain the text form of the announcement or advertisement messages embedded in it, theembedded data could be extracted in real-time fromthe transmitted sound detected by a microphone. A
receiver for the watermarked audio signal could be a
personal digital assistant or a cellular phone.
Such an application requires the watermarking system to be robust against reverberations, reflections,and background noise. It also requires a greater data

2.1

Audio watermarking based on amplitude modulation
Embedding process

At the beginning of the embedding process, a hostsignal H(t), which is the length of one data frame period, is split into 2n subband signals hm (t) using anequal-bandwidth filter bank:H(t) =2nX

hm (t).

(1)m=1Sinusoidal amplitude modulations (SAMs) at a relatively low modulation frequency (f -Hz) are applied tothe adjacent subband signals h2m (t) and h2m+1 (t) in
the opposite phase. An embedding key produced bya known pseudorandom number generator arbitrarily
classifies the n subband pairs into k subband groups.
It also defines the random initial phase angles r(m) of
the SAMs for each subband pair. The output of an amplitude modulated subband pair xim (t), which belongs
to the i-th subband group, is given by
xim (t) = h2m (t) 1+A(m) sin(2πf t+r(m)+p(i) +
h2m+1 (t) 1−A(m) sin 2πf t+r(m)+p(i) , (2)where A(m) is the depth of the SAM of the m-th
subband pair. Embedded information is encoded byphase shift keying (PSK), defined as the difference between the phase angles of the SAM of the first subband
group and that of the i-th subband group, p(1) and p(i)(i = 2, ..., k). Quarterly PSK encodes 2-bit information(Di = 0, 1, 2, 3) into every π/2 phase angle of p(i).
(0i = 1;
p(i) =πDi2

i = 2, ..., k.

(3)

As a result, 2(k − 1) bits of information are embedded per data frame period. Multiplex watermarking
can be applied using different modulation frequencies
simultaneously. Finally, a watermarked signal X(t) is
obtained by summing all of the amplitude-modulated
signals xm (t).
X(t) =nX

xm (t).

(4)

m=1

Synchronization of the data frames is achieved byinverting the relative phase of the SAMs between successive frames for the first subband group.

2.2

Extraction process

which belongs to the i-th subband group produces
an accumulated AM waveform Gi (τ ), which emphasizes the AM waveform. Consequently, the modulation depth A(m) in the embedding process can be kept
small. Initial phase differences between the first andthe i-th subband group, that is, the embedded information, are obtained by comparing the phase angles of
the FFT spectra calculated from G1 (τ ) and Gi (τ ).
The intensity of a watermark, which is defined as thedepth of the SAM A(m), was determined relative to theextracted modulation power of the subband signal of
the host signal. This means that the A(m) value thatgenerates equal power of the effective value of Gm (τ )for the non-watermarked signal is set to 0 dB [5].

2.3

Detecting the starting point of a data
frame

Before decoding the PSK data, the starting point of
the embedded data frame must be detected. A rectangular temporal window of the data frame length T is
iteratively applied to the modulation waveform G1 (τ )extracted from the first subband group. The starting point of the windowing is denoted by u in Eq. 6.
Then, F (u) is derived by subtracting the synchronized
addition of the modulation waveforms in the odd-orderwindows from the synchronized addition of the modulation waveforms in the even-order windows.
Ru = {G1 (u), G1 (u + 1), ..., G1 (u + T − 1)}.
X
X
Ru+(2v+1)T .
Ru+2vT −F (u) =

(6)
(7)

v=0

v=0

The Fourier amplitude of F (u), which corresponds tothe modulation frequency f , denoted by AMPf F (u) ,exhibits a maximum when the position of the window
overlaps completely the position of the frame. Consequently, the starting point of the data frame y is givenby
y = argmax AMPf F (u) .
(8)uThe amplitude envelopes of the subbands
Em (τ )(m = 1, 2, ..., 2n) of the watermarked frame
signal can be derived from the amplitude spectrumof the half-overlapped running FFT, where τ is theperiod of time defined as half the FFT length. The
embedded modulation waveform Gm (τ ) is extracted
by calculating the logarithmic ratio of the amplitudeenvelopes extracted from adjacent subband signals.

The above-described method is applicable when the
duration of the watermarked signal is at least 2T .
The detection performance is improved by using alonger watermarked signal, especially when a severe
disturbance is applied to the watermarked signal. The
present implementation utilizes a watermarked signal
length of 6T to calculate F (u).

(5)

Displaying
informationsynchronously with watermarked sounds

After compensation of the initial phase differencer(m) for each Gm (τ ), synchronized addition of Gm (τ ),Table 1 shows an example of the definition of therecord of embedding data. The record consists of six

Gm (τ ) = log

E2m (τ ).
E2m+1 (τ )3

Table 1. Data definition of the embedding
record.
Auxiliary Relative Starting Relative
EndingIndex fordata
starting
time inendingtime ininformation
frame
the frameframe
the frame to display
2 bits6 bits4 bits6 bits4 bits7 bits4Computer simulation of noisy and reverberant conditions

A computer simulation was performed in order toconfirm that the data embedded in 100 pieces of popular music (RWC-MDB-P2001 [1]) could be successfully extracted under noisy and reverberant conditions.
An AM- based watermark of the embedding condition
shown in Table 2 is embedded in the initial 60-second
left-channel signal of each piece of music.
The impulse response of the reverberation room having a reverberation time of 1.3 seconds (the ‘ir130.dat’
file in the RWCP Sound Scene Database in a real acoustic environment) was convolved with the watermarked
music signals. One of the five types of backgroundnoise was added to the reverberant watermarked music.
These background noises consisted of noises recorded
in an airport lounge, a crowded intersection, an underground corridor, a station platform, and a lowpass
noise that had a cutoff of 500 Hz and a slope of –9
dB/oct. The same noises were used in the previous research [6]. The overall signal-to-noise-ratio (SNR) was
15 dB. The role of the impulse response is to supply

Table 2. Watermarking conditions.
Parameters
Valuesembedding bit rate
43 bpssampl. freq. [Hz]44,100embedding region
below 11,025 Hz
subband pairs (n)64subband groups (k)17frame period
3smod. freq. [Hz]1.67, 2, 2.33, 3
watermarking intensity+12 dBthe band-limited characteristics of a loudspeaker and a
microphone, not only to supply reflections and reverberations.
The watermarking performance was evaluated interms of the number of data frames that achieved a
bit error rate lower than the error correction limit of
21 bits. The accuracy of synchronization was evaluated in terms of the deviation of the detected boundary from the correct boundary of the data frame. Since
these performance indices depend on the number offrames from the beginning of the watermarked music,
the results are shown as a function of the frame number. Figure 1 shows the ratio of the correctly recovered
data frames. Figure 2 shows the mean deviation fromthe correct boundary location. Each datum is collectedfrom 500 simulated conditions (100 pieces of music versus five background noises).
The ratio of correctly detected data frames is insufficient in the initial part of the music. This is causedby the relatively large deviations of the detected frameboundaries from the correct timing and the small power
level observed in the introduction of the music. However, after the fifth frame, the synchronization and detection performances are sufficient for the host signalof various types of popular music.

Correct detection rate

data fields of 29 bits in total. BCH coding is used to
encode 29 bits of the record into 127 bits with the error
correction limit of 21 bits. Bit sequences in the data
fields are interpreted as unsigned integer values. The
length of each data field depends on the requirementsof the application. The second field defines the position of the target frame to begin display relative to theembedded frame. The exact timing to begin display
during the target frame is defined as the ratio from the
beginning of the frame. The end timing of display isdefined in the same manner as the start timing. The
index data field points to the entity of the information
for display, such as a song lyric or a movie caption thatis already stored and indexed in the machine that runs
the detection program.
Synchronization of displaying embedded data isbased on real-time detection of the embedding frame
segment from the watermarked signal. After frame
detection, all decoded record data are stored in thebuffer memory for the time being. Each time the incoming watermarked signal is transported from the AD
converter, the timing data of the buffered records arescanned to display the indexed information.

1
0.95
0.9
0.85
0.8
0.75
0.7
5

10

15

20

Frame number [T] (T = 3 s)

Figure 1. Correct rate of detected frames.
Each datum was collected from 500 simulated conditions.

Deviation from thecorrect frame boundary [s]

6

0.2
0.1
0
-0.1
-0.2
-0.3
-0.4
5

10

15

20

Frame number [T] (T = 3 s)

Figure 2. Deviation from the correct frameboundary. Error bars show ± 1 standard errors.

5

Objective audio quality measurement
for watermarked music

Objective Difference Grade

A method for the objective measurement ofperceived audio quality (ITU-R Recommendation
BS.1387), referred to hereinafter as PEAQ, uses a number of psycho-acoustical measures that are combined
to provide a measure of the quality difference between
two instances of a signal (a reference and a test signal).
The implementation of the basic version of PEAQ by
Kabel [3] was used to assess the objective quality degradation of the watermarked music used in the previous
section. The PEAQ measurement outputs the Objective Difference Grade (ODG), which corresponds to the
Subjective Difference Grade obtained from the assessment procedure of the subjective quality degradation
defined in ITU-R BS.1116-1.
Figure 3 shows the averaged result of PEAQ measurement for watermarked music, and 96-kbps and 128kbps MP3 encoded music. The mean ODG obtained
from the watermarked music is an intermediate value
between the 96-kbps and 128-kbps MP3 encoded music. In other words, the degradation in quality of themusic impaired by watermarking is slightly annoying
but still tolerable.
Very annoying: -4
Annoying: -3
Slightly
annoying: -2
Perceptible,
but not annoying: -1
Imperceptible: 0
Watermarked
43 bpsMP3
96 kbps
Degradation

MP3
128 kbps

Figure 3. Mean ODGs for 100 pieces of musicfrom RWC-MDB-P2001. Error bars show ± 1
standard errors.

Discussion

Practical implementation of the system requires embedding the same record iteratively. The limitation ofthis scheme is that recorded data must be embedded
before being displayed. The optimal number of iterations for each embedding record can be determined
automatically depending on the host signal.
In an application of displaying captions of a film, it
is difficult to handle relatively long periods of silencein the soundtrack. A solution is to use a long synchronization frame divided into several data frames. The
embedding parameters of the system are flexible, allowing the desired embedding bit rate to be to realized
if the length of the frame is altered.

7

Summary

A watermarking technique using subband AM wasapplied to the synchronous display of text information
with the audio signal. The robustness of the system
was evaluated by a computer simulation in terms ofthe correct data transmission rate under reverberant
and noisy conditions. The results showed that the performance of detection and temporal precision are sufficient. Objective measurement of the watermarked audio quality was conducted using the PEAQ method.
The mean ODG obtained from the 100 watermarked
music was an intermediate value between the meanODGs of 96-kbps and 128-kbps MP3 encoded music.

References[1] M. Goto, H. Hashiguchi, T. Nishimura, and R. Oka.
Rwc music database: Popular, classical, and jazz music
databases. In Proceedings of the 3rd International Conference on Music Information Retrieval (ISMIR 2002),
pages 287—288, 2002.
[2] D. Gruhl, A. Lu, and W. Bender. Echo hiding. In
Proceedings of the First International Workshop on Information Hiding LNCS 1174, pages 295—315, 1996.
[3] P. Kabal. An examination and interpretation of ITU-R
BS.1387: Perceptual evaluation of audio quality. TSP
Lab Technical Report, Dept. Electrical & Computer Engineering, 2002.
[4] A. Nishimura. Audio watermarking based on sinusoidal
amplitude modulation. In Proceedings of ICASSP 2006,
IV, pages 797—800. IEEE, 2006.
[5] A. Nishimura. Audio watermarking based on subband amplitude modulation. In Proceedings of the 2006
Symposium on Cryptography and Information Security,
number 3F4-2. IEICE, 2006.
[6] A. Nishimura. Data hiding for speech sounds using subband amplitude modulation robust against reverberations and background noise. In Proceedings of IIH-MSP
2006, pages 7–10. IEEE, 2006.