Data hiding in speech sounds using subband amplitude modulation
robust against reverberations and background noise
Akira Nishimura
Department of Media and Cultural Studies, Faculty of Informatics,
Tokyo University of Information Sciences
1200–1, Yatoh-cho, Wakaba-ku, Chiba-city, Chiba, JAPANakira@rsch.tuis.ac.jp
Abstract
Data hiding in audio signals can be used for transmitting auxiliary information related to the content
of the audio signal. Such an application requires a
greater data payload than music copyright management systems. Also, it is diﬃcult to extract the embedded data from sounds played through a loudspeakerand detected by a microphone because of additive background noise, reﬂections and reverberations and the
band-limited characteristics of loudspeakers and microphones. In this study, a watermarking technique using
subband amplitude modulation was evaluated by computer simulation in terms of robustness against background noises and reverberations. The eﬀects of amplitude modulation on the articulation scores of 125
vowel-consonant-vowel (VCV) syllables were also investigated. The results showed that reverberant speech
signals with various background noises having a SNRof 10 dB can transmit more than 90% of embedded data
at 48 bps, with only a small deterioration in the syllable
identiﬁcation scores.

1. Introduction
Audio watermarking techniques are usually associated with content protection and digital rights management of music. In such applications, the embedded
data is generally the copy control code and the content identiﬁcation code. Therefore, the robustness ofwatermarks to perceptual codecs, additive noise andlowpass ﬁltering is critical when watermarked music is
distributed by computer networks or broadcasting.
Another application of audio watermarks is the inclusion of augmentation data[1]. For example, if anannouncement broadcast from loudspeakers in a publicarea could have a text form of the announcement embedded in it, the embedded data could be extracted inreal-time from the speech signal that is detected by the
microphone. Such an application would be invaluablefor hearing-impaired people. A receiver for the watermarked audio signal could be a portable device such asa personal digital assistant or a cellular phone. Suchan application requires the watermarking system tobe robust against reverberations, reﬂections and background noise. It also requires a greater data payload
than copyright management systems, while speech has
a narrower frequency band compared with that for music, which will result in a low embedding data rate.
Moreover, a certain degree of quality degradation is
acceptable in the case when the cover data is speech,
while clarity is essential for watermarked speech.
However, very few attempts have been made to
produce watermarking systems for speech or to verify the robustness of watermarks against reverberations. Generally speaking, most audio watermarking
techniques that adopt relatively short embedding timeframes, say below 1 s, cannot be used in reverberant
spaces, since the delayed signal frame masks and interferes with the following frame. Such watermarking
techniques include spread-spectrum, echo-hiding andphase-manipulations technologies.
The author has developed a new audio watermarking technique based on subband amplitude modulation.
This system is robust against perceptual audio codings,
additive Gaussian noise and spectral modiﬁcations [2].
It is also robust against reﬂections and reverberations,since it applies relatively slow amplitude modulation ina long embedding frame of several seconds.
Evaluation of this watermarking system was initiallyconducted for data hiding in music[2]. However, thewatermarking system can also be used for data hidingin speech signals.

In this paper, the eﬀectiveness of watermarking
based on amplitude modulation in applying speech signals is examined in terms of the embedding data rate,
the detection rate and the correct identiﬁcation scoreof listeners.

Synchronization of the data frames is achieved byinverting the relative phase of the SAMs between successive frames for the ﬁrst subband group.

2

The amplitude envelopes of the subbands
Em (τ )(m = 1, 2, ..., 2n) of the watermarked frame
signal can be derived from the amplitude spectrumof the half-overlapped running FFT, where τ is theperiod of time deﬁned as half the FFT length. The
embedded modulation waveform Gm (τ ) is extracted
by calculating the logarithmic ratio of the amplitudeenvelopes extracted from adjacent subband signals.

2.1

Audio watermarking based on amplitude modulation
Embedding process

At the beginning of the embedding process, a hostsignal H(t), which is the length of one data frame period, is split into 2n subband signals hm (t) using anequal-bandwidth ﬁlterbank:H(t) =2nhm (t).

Sinusoidal amplitude modulations (SAMs) at a relatively low modulation frequency (f -Hz) are applied
to the adjacent subband signals h2m (t) and h2m+1 (t)in opposite phase. An embedding key produced bya known pseudorandom number generator arbitrarily
classiﬁes the n subband pairs into k subband groups.
It also deﬁnes the random initial phase angles r(m) of
the SAMs for each subband pair. The output of an amplitude modulated subband pair xim (t), which belongs
to the i-th subband group, is given by
xim (t) = h2m (t) 1+A(m) sin(2πf t+r(m)+p(i) +
h2m+1 (t) 1−A(m) sin 2πf t+r(m)+p(i) , (2)where A(m) is the depth of the SAM of the m-th subband pair. Embedded information is encoded by phaseshift keying, deﬁned as the diﬀerences between thephase angles of the SAM of the ﬁrst subband groupand that of the i-th subband group, p(1) and p(i)(i = 2, ..., k). 4-phase shift keying encodes 2-bit information (Di = 0, 1, 2, 3) to every π/2 phase angle ofp(i).
0
i = 1;
p(i) =πDii = 2, ..., k.
(3)2
As a result, 2(k − 1) bits of information are embedded per data frame period. Multiplex watermarking
can be applied using diﬀerent modulation frequencies
simultaneously. Finally, a watermarked signal X(t) is
obtained by summing up all the amplitude-modulated
signals xm (t).
n
m=1

xm (t).

(4)

Extraction process

Gm (τ ) = log

(1)

m=1

X(t) =

2.2

E2m (τ ).
E2m+1 (τ )(5)After compensation of the initial phase diﬀerencer(m) for each Gm (τ ), synchronized addition of Gm (τ )which belongs to the i-th subband group produces
an accumulated AM waveform Gi (τ ) which emphasizes the AM waveform. Consequently, the modulation depth A(m) in the embedding process can be kept
small. Initial phase diﬀerences between the ﬁrst andthe i-th subband group, that is the embedded information, are obtained by comparing the phase angles of
the FFT spectra calculated from G1 (τ ) and Gi (τ ).
The intensity of a watermark is deﬁned as the depthof the SAM A(m). In a previous study, A(m) was determined relative to the weighted modulation power [2]or the extracted modulation power of the subband signal of the host signal [3]. In the present study, thevalues of A(m) for all m are equal and they are simply
determined as parameter values, because the calculation cost should be minimized for real-time embedding.

2.3

Finding a starting point of the data
frame

Before decoding the phase shift keying data, thestarting point of the embedded data frame must be
detected. A rectangular temporal window of the data
frame length T is iteratively applied to the modulation waveform G1 (τ ) extracted from the ﬁrst subband
group. The starting point of the windowing is denotedby u in Eq. 6. Then, F (u) is derived by subtracting thesynchronized addition of the modulation waveforms in
the odd-order windows from the synchronized additionof the modulation waveforms in the even-order windows.
Ru = {G1 (u), G1 (u + 1), ..., G1 (u + T − 1)}.

(6)



Ru+2vT −v=0

Ru+(2v+1)T .

(7)

v=0

The Fourier amplitude of F (u) which correspondsto
the modulation frequency f , denoted by AMPf F (u) ,exhibits a maximum when the position of the window
overlaps completely with the position of the frame.
Consequently, the starting point of the data frame yis given by(8)y = argmax AMPf F (u) .

(SNR) was 10 dB and 20 dB. The watermarking performance was evaluated in terms of the correct bit rateof the extracted data when compared with that of theembedded data. No error correction was performed on
the embedded data.

-20

Relative power [dB]

F (u) =u3

Computer simulation of noisy and reverberant conditions

-30
-40
-50
-60

A computer simulation was performed to conﬁrm
that the embedded data in speech signals can be successfully extracted in noisy and reverberant conditions.
The watermarking conditions used in this simulation
are shown in Table 1.
Table 1. Watermarking conditions.
Parameters
Valuesembedding bit rate
64 bps
48 bpssampl. freq. [Hz]22050←
embedding region below 6034 Hz←
subband pairs (n)140←subband groups (k)25←frame period
3s
4s
mod. freq. [Hz] 1, 1.67, 2.33, 3 1, 1.5, 2, 2.5
Speech segments having a duration of 36 seconds
and spoken by 12 female and 10 male speakers wereselected from the Continuous Speech Database for Research (Vol. 1) published by the Acoustical Society ofJapan. These speech segments were used as the coveraudio data. Their sampling frequencies were converted
from 16 kHz to 22.05 kHz.
In the simulation, a room impulse response recorded
in a variable reverberation room having a reverberation
time of 1.3 seconds (the ‘ir130.dat’ ﬁle in the RWCP
Sound Scene Database in a real acoustic environment)
was convolved with the watermarked speech signals.
One of the ﬁve types of background noise was added
to the reverberant watermarked speech. These background noises consisted of noise recorded in an airport
lounge, a crowded intersection, an underground corridor, a station platform and a lowpass noise which had a
cutoﬀ of 500 Hz and a slope of –9 dB/oct. (this is representative of a typical average spectrum of backgroundnoise). The averaged spectra of the background noises
are shown in Fig. 1. The overall signal-to-noise-ratio-70

Airport Lounge
Intersection
Underground Corridor
Platform of Station
Lowpass 500 Hz, -9 dB/oct.
100

1000

10000

Frequency [Hz]Figure 1. Averaged power spectra of the additive background noises.
The impulse response was recorded in a variable
reverberation room, which is a diﬀerent environment
from those of the background noises. However, the roleof the impulse response is to supply the band-limited
characteristics of a loudspeaker and a microphone, notonly to supply reﬂections and reverberations.
Figure 2 shows the medians of the bit detection
rate for 110 conditions (22 speakers by ﬁve backgroundnoises) as a function of the watermark intensity A(m).
The error bars denote the 10th to 90th percentile ofthe detection rate obtained in the 110 conditions. The
results of the detection rates demonstrate that if data
is embedded at 48 bps and an AM depth of more than0.5, more than 88% data bits survive for more than
90% conditions of the speaker and background noise
combinations at a SNR of 10 dB. They also reveal thatembedding at 64 bps is eﬀective for high SNR conditions.

4

Identification test for watermarked
VCV syllables

Identiﬁcation tests for watermarked VCV syllables
were conducted to investigate the eﬀect of watermark
on articulation scores.
125 watermarked VCV syllables, consisting of ﬁve
ﬁrst vowels (a, i, u, e, o), 25 Japanese consonants andthe second vowel ‘a’, were presented via a headphone(Sennheiser HDA-200) diotically in a soundproof room.
The conditions of watermarking were the same as those51

Detection rate

0.95
0.9
0.85
0.8
64 bps, SNR = 10 dB
64 bps, SNR = 20 dB
48 bps, SNR = 10 dB
48 bps, SNR = 20 dB

0.75
0.7

0.3

0.4

0.5

0.6

0.7

AM(m): AM depth

Figure 2. Medians of bit detection rate. Error
bars denote 10th to 90th percentile of the detection rate obtained in 110 conditions.

Mean correct identification rate

given in Table.1 except that the embedding frequency
region was under 8 kHz. The subjects were asked totype what they heard using the keyboard of a computer. The watermark intensity (A(m) in Eq.2) was
set to 0.4, 0.6 and 0 (no watermarking). The VCV syllables were presented in random order with or withouta low-pass noise which had a cutoﬀ frequency of 500
Hz and a slope of –9 dB/oct. at a SNR 10 dB. Overall SPL of the VCV syllables was 72 dB. Five subjects
with normal hearing participated in the same experiment which was conducted twice on separate days.
The mean articulation scores of the 125 VCV syllables are shown in Fig. 3. The error bars denote theminimum and maximum scores of the subjects. Since
speech intelligibility is generally higher than the articulation scores of syllables, no serious deterioration in theintelligibility of the watermarked speech is expected.

no additional noise
SNR 10 dB

1Discussion

The advantage of the identiﬁcation test of the VCVsyllables is availability of confusion matrix analysis.
Analysis of confusion matrix shows that the misidentiﬁcation rates of the syllables which were misidentiﬁed when there was no watermark but backgroundnoise was present increase when the intensity of watermarking increases and background noise is added. The
ﬁnding of occasional misidentiﬁcation of the ﬁrst andsecond vowels in all conditions and the dependence of
misidentiﬁcation of the consonants on the ﬁrst vowel,
indicates that amplitude modulation impairs temporalchanges which result from coarticulation between theﬁrst vowels and the consonants. Detailed analysis of
the confusion matrix will enable improvements on theembedding algorithm.
In the case of an audio watermark that carries text
data of the speech, the payloads of embedded data, including a practical error correction scheme, should bein the range of 200 to 300 bps in order to achieve areal data rate of approximately 100 bps, which represents the average text data rate transmitted by speech.
Since the present data rate is four or ﬁve times smaller
than the above requirement, eﬀective data compression for embedding and improved watermark detection
is required.

6

Summary

A watermarking technique using subband amplitude
modulation was evaluated by a computer simulationin terms of robustness against background noises, reﬂections and reverberations. The eﬀects of amplitude
modulations on the articulation scores of 125 VCV syllables were also investigated. The results showed thatreverberant speech signals with various backgroundnoises having a SNR of 10 dB can transmit more than
90% of embedded data at 48 bps, with only a small
deterioration in syllable identiﬁcation scores.

0.95

References

0.9
0.85
0.8

0.6

0.4
A(m) : AM depth

0

Figure 3. Mean articulation scores of 125 VCVsyllables. Error bars denote minimum andmaximum among the five subjects.

[1] D. Gruhl, A. Lu, and W. Bender. Echo hiding 1996. In
Information Hiding, pages 295—315, 1996.
[2] A. Nishimura. Audio watermarking based on sinusoidal
amplitude modulation. In Proceedings of ICASSP 2006,
IV, pages 797—800. IEEE, 2006.
[3] A. Nishimura. Audio watermarking based on subband amplitude modulation. In Proceedings of the 2006
Symposium on Cryptography and Information Security,
number 3F4-2. IEICE, 2006.