INTERSPEECH 2006 - ICSLP

Speaker Identification under Noisy Environments
by Using Harmonic Structure Extraction and Reliable Frame Weighting
Hiromasa Fujihara† , Tetsuro Kitahara† , Masataka Goto‡ ,Kazunori Komatani† , Tetsuya Ogata† , and Hiroshi G. Okuno††

Dept. of Intelligence Science and Technology
Graduate School of Informatics, Kyoto University
Sakyo-ku, Kyoto 606-8501, Japan{fujihara,kitahara,komatani,ogata,okuno}@kuis.kyoto-u.ac.jp

Abstract‡

National Institute of Advanced Industrial
Science and Technology (AIST)
Tsukuba, Ibaraki 305-8568, Japanm.goto@aist.go.jp

environmental noises by extracting the harmonic structure of aspeech signal from given audio signals and then resynthesizing
the speech signal using a sinusoidal model. This method is robust to unknown nonstationary noises because it does not assume
the noise characteristics. On the other hand, the reliable frame
weighting method assigns a higher reliability to frames that areless influenced by environmental noises, and uses this reliability
as a weight for each frame when identifying the speaker. Thismethod is robust to sudden noises because the contribution fromhighly distorted frames during speaker identification was reduced.
The rest of this paper is organized as follows. The next section
describes our method for speaker identification, and Section 3 describes results of our experiments. Finally, Section 4 draws conclusions and points out future directions.

We present methods for automatic speaker identification in noisy
environments. To improve noise robustness of speaker identification, we developed two methods, the harmonic structure extraction
method and the reliable frame weighting method. The harmonic
structure extraction method enables the speaker of input speech
signals to be identified after environmental noise has been reduced.
This method first extracts harmonic components of the speechfrom the sound mixtures and then resynthesizes a clean speech signal by using a sinusoidal model driven by harmonic components.
The reliable frame weighting method then determines how eachframe of the resynthesized speech is reliable (i.e. little influenced
by environmental noises) by using two Gaussian mixture models
for the speech and noise. The speaker can be robustly identified
by attaching importance to reliable frames. Experimental resultswith thirty speakers showed that our method was able to reducethe influences of environmental noise and achieved an error rate of
10.7%, while the error rate for a conventional method was 18.9%.
Index Terms: speaker identification, noise robustness, voice extraction, voice reliability, Gaussian mixture model.

2. Speaker Identification Robust To
Environmental Noise
Figure 1 shows an overview of our method that identifies a speakerfrom an input speech signal with background noises. The following describes each step of the method with a focus on the harmonic structure extraction method and the reliable frame weighting method, which are important for the robustness to noises. Although the main idea of these two robust methods is based on ourwork on automatic singer identification from polyphonic musicalaudio signals[4], this is the first paper that validates the effectiveness of the idea in identifying a speaker name in speech signals.

1. Introduction
Automatic speaker identification is increasing in importance since
it can be used in many applications, such as speaker indexing
and voiceprint verification. Speaker identification is also usefulfor human-robot interaction. For example, speaker identification
will enable a robot to communicate with any speakers by identifying the voice of a speaker or by detecting speaker’s change.
In a real environment, however, traditional speaker identification
techniques for clean speech voices cannot be used because speech
voices are distorted by environmental noises.
To improve the noise robustness of speaker identification, twopopular methods have been studied. A spectral subtraction (SS)method [1] reduces a background noise by subtracting the power
spectrum of the background noise from an observed power spectrum. A hidden Markov model (HMM) composition method [2, 3]
creates noise-added speech HMMs by combining clean speech
HMMs with noise HMMs. The noise-added speech HMMs canthen be used to identify the speaker. These two methods, however,are not robust to sudden and nonstationary noises because they assume that the power spectrum or HMMs of the background noise
are known in advance.
To solve this problem, we propose the harmonic structure extraction method and the reliable frame weighting method. The
harmonic structure extraction method can reduce the influence of

2.1. Harmonic Structure Extraction
To reduce the environmental noise in a given audio signal, we use
a speech resynthesis technique based on harmonic structure. This
technique consists of the following three steps:
1. Estimating the fundamental frequency (F0) of the speechusing Goto’s PreFEst method [5].
2. Extracting the harmonic structure corresponding to the estimated F0 in each frame.
3. Resynthesizing the audio signal (waveform) corresponding
to the speech using a sinusoidal synthesis.
We thus obtain a waveform that corresponds to dominant speech.
2.1.1. Fundamental Frequency (F0) Estimation
We use Goto’s PreFEst [5] for estimating the F0 of the speech. The
PreFEst method estimates the most predominant F0 in frequencyrange-limited sound mixtures. Since the speech tends to have

1459

September 17-21, Pittsburgh, Pennsylvania

INTERSPEECH 2006 - ICSLP(allowable) F0 range, and w(t) (F ) is the weight of a tone modelthat satisfies
Z

Input Audio Signal

Fli

Extracting Harmonic Structure

Resynthesis

SpeechGMM

Reliability CM

Noise
GMM
Speaker name determination

F

Since the PreFEst was originally designed to estimate themelody and base lines in musical audio signal, it assumes that an
observed spectrum is a mixture of harmonic sounds. Although
PreFEst might not perform at its optimum when applied to speech
signals in a noisy environment, the estimation result is still useful
if used with the reliable frame weighting method for robustness
against F0 estimation errors, which is described later.
2.1.2. Extracting Harmonic Structure
Based on the estimated F0, we extract the power and phase of theF0 component and harmonic components. For each component,
we allow an |r| cent error and extract the peak in the allowed area.
The power, Al , phase, θl , and frequency, Fl of the l th overtone(l = 1, . . . , 20) can be represented as=argmax |S(F )|==(lF · (1 − 2 1200 ) ≤ F ≤ lF · (1 + 2 1200 )), (6)|S(Fl )|,
(7)arg S(Fl ),
(8)

F

r

Output (Speaker’s name)AlθlFigure 1: Method overview

rwhere S(F ) denotes the spectrum and F denotes the F0 estimatedby the PreFEst. We set r to 20 in our experiments.
Figure 2 shows an example of the F0 estimation and harmonic
structure extraction. Figure 2 (a) shows an original spectrum andits envelope, and Figure 2 (b) shows an extracted spectrum andits envelope. These figures show that the spectral envelope of theextracted spectrum (b) correctly represents the formants of speech,
when compared with that of the original spectrum (a).

the most predominant harmonic structure in middle- and highfrequency regions, we can estimate the F0 of the speech by applying the PreFEst.
A summary of the PreFEst is described below. Hereafter, x
is the log-scale frequency denoted in units of cents (a musicalinterval measurement) and (t) means time. Given a power spec(t)trum, Ψp (x), we first apply the band-pass filter (BPF) that isdesigned so that it covers most of the dominant harmonics in typical speech voice. The filtered frequency components can be rep-

2.1.3. Resynthesis

 (t)resented as BP F (x)Ψ p (x), where BP F (x) is the BPF’s frequency response to the speech. To enable statistical methods to be
used, we represent each of the bandpass-filtered frequency components as a probability density function (PDF), called an observed(t)
PDF, pΨ (x):We resynthesize the audio signals of the speech from the extracted
harmonic structure by using a sinusoidal model[6]. Resynthesizedaudio signals are expressed ass(t) =LX

Al cos(ωl t + θl ),

(9)

l=1

(1)

where Al , θl , and Fl represent the power, phase, and frequency ofthe l th overtone and t is time.

Then, we consider each observed PDF to have been generated froma weighted-mixture model of the tone models for all possible F0s,
which is represented as follows:
Z Fhp(x|θ(t) ) =w(t) (F )p(x|F )dF(2)

2.2. Feature Extraction
We use mel-cepstral coefficients of the LPC spectrum, called linear prediction derived mel-cepstral coefficients (LPMCCs), because we have found that, in the context of singer identification,
the LPMCCs can better represent vocal characteristics better thanstandard mel-frequency cepstral coefficients (MFCCs) [4].
The LPC spectrum is a spectral envelope calculated by linear
prediction (LP) analysis. The LP analysis [7][8] is a method thatFlθ(t) = {w(t) (F )|Fl ≤ F ≤ Fh},(4)

The tone model represents a typical harmonic structure and indicates where the harmonics of the F0 tend to occur. Then, we estimate w(t) (F ) using the EM algorithm and regard it as the F0’s
PDF. Finally, we obtain the most dominant F0 F (t) by the following equation:
F (t) = argmax w(t) (F )
(5)

Fl

(t)
BP F (x)Ψp (x)(t)
pΨ (x) = R ∞.
(t)BP F (x)Ψp (x)dx−∞w(t) (F )dF = 1.

Fhi

Reliable Frame Weighting

for each frame

Harmonic Structure Extraction

F0 Estimation(3)

where p(x|F ) is the PDF of the tone model for each F0, and Fhand Fl are defined as the lower and upper limits of the possible

1460

linear power

INTERSPEECH 2006 - ICSLPTable 1: Dataset
ASJ-JNAS[9]ATR phonetically-balanced sentences
30 speakers (male: 15, female: 15)
Training: 10 utterances
Evaluation: 40 utterancesTraining: Lobby of the great hall
Evaluation: Party venue

Speech data
0

2000

4000
Frequency [Hz]

6000

8000

Noise datalinear power(a) An original spectrum and its envelope.

0

2000

4000
Frequency [Hz]6000

Table 2: Speech analysis
Sampling
16 kHz, 16 bit
Window function Hamming
Frame length
160 msFrame period
10 ms
Feature vector
12th order LPMCC
based on 20th order LPC analysis

8000(b) An extracted spectrum and its envelope.
Figure 2: Example of F0 estimation and harmonic structure extraction. The envelope of each spectrum is calculated using the linear
prediction (LP) analysis.
estimates the transfer function of a vocal tract, assuming that the
input audio signal only contains a human voice. In the LP model,
when a descrete signal, s(n) is provided, we predict the signal as alinear combination of its previous samples. The predicted sW (n)value is given bysW (n) =p
X

αi sW (n − i) + g(n),2.4. Speaker Name Determination
The name of the speaker is determined based on the 64-mixture
GMMs. We train the GMMs λ1 , · · · , λI in advance for each registered speaker by using the EM algorithm. Let X = {xt |t =1, . . . , T } be a time series of feature vectors of input signals andp(x|λi ) be the likelihood of the GMM for the speaker i. Finally,
the name of the speaker is eventually determined using the following equation:

(10)

i=1

where p represents the order of a predictor, a set of αi are definedas the linear prediction coefficients (LPCs), and g(n) represents
an error in the model. The LPCs are estimated by minimizing themean squared prediction error of g(n). We used 20th-order LPC
in this paper.
In order to calculate the LPMCC, we apply the mel-cepstral
analysis to the LPC spectrum. In addition to the role of orthogonalization, the LPMCCs are superior to the LPC in terms of suitability to the human auditory system, which is a benefit of themel-frequency scale. In this paper, we derive the LPMCC by computing the MFCC from the LPC spectrum for the sake of simplicity
of implementation. We set the order of the LPMCC to 15 in thispaper.

i3.1. Conditions
To confirm the effectiveness of our methods, we conducted experiments on text-independent speaker identification.
Figure 1 shows the data used in this experiment. We used
ASJ-JNAS ATR phonetically-balanced sentences[9] for training
and evaluation datasets. The duration of each utterance was approximately 4 seconds. All noise-added data, x(t), were createdby mixing clean speech data, xS (t), with noise data, xN (t), at 0dB SNR according to the following equations:

We introduce two different Gaussian mixture models (GMMs), a
speech GMM λV and a noise GMM λN . The speech GMM λV
is trained using feature vectors extracted from speech signals, andthe noise GMM λN is trained using feature vectors extracted fromnoise signals. We train each GMM by using the EM algorithm (weuse 64-mixture GMMs in our experiments.). Given a feature vector, x, the likelihood that feature vector x is like speech or a noise
can be expressed as p(x|λV ) and p(x|λN ). If the feature vector,
x, is less influenced by environmental noise (i.e., more reliable),p(x|λV ) will be higher and p(x|λN ) will be lower. We therefore
define reliability of the feature vector, x, as follows:p(x|λV )w(t) (F ),p(x|λV ) + p(x|λN )(12)

3. Experiments

2.3. Reliable Frame Weighting

CM(x) =1 X
CM(xt ) log p(xt |λi )T t=1
Ts = argmax

x(t)

=

k

=

SPOW

=

1 X
(xS (t))2 ,TS t=1

(15)

NPOW

=

TN
1 X
(xN (t))2 ,TN t=1(16)

xS (t) + kxN (t),r
SPOW − SNR
10 10 ,
NPOW

(13)
(14)

TN

where TN and TS are the number of samples in the speech andnoise sample sequences, respectively. The noise data used in the
training were different from those used in the evaluation.
As the training data for the reliable frame weighting, we used
274 utterances from 274 speakers taken from the ASJ-JNAS ATR(11)

where w(t) (F ) is the weight (relative amplitude) of the tone modelat the F0 F estimated by the PreFEst in this frame.

1461

INTERSPEECH 2006 - ICSLP

4. Conclusion

Table 3: Experimental results under five different conditions. The“extract.” and “weight.” denote the harmonic structure extraction method and the reliable frame weighting method, respectively.
The “Clean” means that the F0 was estimated from clean speech
signals to show the upper limit of the performance. “” and “×”mean that the harmonic structure extraction method or the reliable
frame weighting method were performed and not performed, respectively.
Conditions
(i)
(ii)
(iii)
(iv)
(v)
Noise
with noise (SNR 0 dB)
w/o noise
F0 estimation
PreFEstClean
extract.
×
×weight.
××××
Error rate (%)
10.7 26.5 18.9
5.1
1.5

We described the harmonic structure extraction method and the reliable frame weighting method for automatic speaker identificationin noisy environments. In our experiments with thirty speakers, wefound that our method achieved a 43% decrease in error rate andconfirmed the robustness and effectiveness of our methods.
In the future, we plan to extend our method to be able to deal
with various severer SNR conditions. Furthermore, we plan toimplement this method to robots to achieve robust human-robot
speech interaction.

5. Acknowledgements
This research was partially supported by the Ministry of Education, Culture, Sports, Science and Technology (MEXT), Grantin-Aid for Scientific Research (A), No.15200015, and Informatics
Research Center for Development of Knowledge Society Infrastructure (COE program of MEXT, Japan). We thank Kazuyoshi
Yoshii (Kyoto University) and Takuya Yoshioka (NTT) for theirvaluable discussions.

phonetically-balanced sentences. Those 274 speakers were notincluded in the 30 speakers used in the evaluation. We trained
speech/noise GMMs using those data after the harmonic structure
extracting method had been performed. We did not perform the reliable frame weighting method on the training data of each speaker
and used the whole duration of each utterance, while the harmonic
structure extraction method was performed.
We conducted experiments under the following five conditions:(i) with extraction using F0s estimated by PreFEst and weighting(proposed),(ii) with extraction using F0s estimated by PreFEst and without
weighting,(iii) without both extraction and weighting (baseline),(iv) with extraction using correct F0s estimated from clean speech
signal and without weighting (for reference),(v) using clean speech signal for both training and evaluation
without both extraction and evaluation (for reference).
Conditions (iv) and (v) were experiments for reference since theyused information that could only be obtained from (normally unknown) clean speech signals.

6. References[1] Boll, S. F., “Suppression of Acoustic Noise in Speech using Spectral Subtraction,” IEEE Transaction on Acoustic,
Speech and Signal Processing, Vol. ASSP-27, 113–120,
1979.
[2] Rose, R. C., Hofstetter, E. M., and D. A. Reynolds, “Integrated Models of Signal and Background with Application to Speaker Identification in Noise,” IEEE Transaction on
Speech and Audio Processing, Vol. 2, no. 2, 245–257, 1994.
[3] Gales, M. J. F., and Young, S. J., “Robust Speech Recognition using Parallel Model Combination,” IEEE Transaction
on Speech and Audio Processing, Vol. 4, 352–359, 1996.
[4] Fujihara, H., Kitahara, T., Goto, M., Komatani, K., Ogata, T.,
and Okuno, H. G., “Singer Identification Based on Accompaniment Sound Reduction and Reliable Frame Selection,”
In Proceedings of the 6th International Conference on Music
Information Retrieval (ISMIR2005), 329–336, 2005.

3.2. Result and Discussion[5] Goto, M., “A real-time music-scene-description system:
predominant-F0 estimation for detecting melody and basslines in real-world audio signals,” Speech Communication,
Vol. 4, no. 4, 311–329, 2004.

Table 3 lists experimental results. The comparison of noised-added
speech signals (condition v) with clean speech signals (conditioniii) shows that the error rate increased significantly from 1.5% to18.9%. This increase indicates a bad influence of environmental
noises. When our two methods were used together (condition i),
the error rate decreased from 18.9% to 10.7%, which is approximately a 43 % reduction in error rate. These results confirmed theimproved robustness of our methods to environmental noise.
When we only preformed the harmonic structure extraction
method with the F0s estimated from the clean speech (conditioniv), the error rate was very low at 5.1%. On the other hand, when
we only performed the harmonic structure extraction method with
the F0s estimated by PreFEst (condition ii), the error rate was
26.5%, which was even worse than the baseline. This implies thatthe reliable frame weighting method has high potential, while thisalso implies that the F0 estimation error has affects the identification error. If we performed the reliable frame weighting method in
addition to the harmonic structure extraction method (condition i),
the error rate decreased from 26.5% to 10.7%. These results showthat the reliable frame weighting method could reduce the negative
influence of F0 estimation errors.

[6] Moorer, J. A., “Signal processing aspects of computer music:
A survey,” In Proceedings of the IEEE, Vol. 65, no. 8, 1108–
1137, 1977.
[7] Atal, B. S., “Effectiveness of linear prediction characteristicsof the speech wave for automatic speaker identification andverification,” the Journal of the Acoustical Society of America, Vol. 55, no. 6, 1304–1312, 1974.
[8] Shikano, K., “Evaluation of LPC spectral matching measuresfor phonetic unit recognition,” Technical Report CMU-CS96-108, CMU, Computer Science Department, 1986.
[9] Ito, K., Yamahoto, M., Takeda, K., Takezawa, T., Matsuoka, T., Tokayashi, T., Shikano, K., and Itahashi, S.,“JNAS: Japanese speech corpus for large vocabulary continuous speech recognition research,” Journal of Acoustic Society Japan (E), Vol.20, No.3, 199–206, 1999.

1462