IEEE TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, VOL. 18, NO. 6, AUGUST 20101507

Speech Spectrum Modeling for Joint Estimation ofSpectral Envelope and Fundamental Frequency
Hirokazu Kameoka, Member, IEEE, Nobutaka Ono, Member, IEEE, and Shigeki Sagayama, Member, IEEE

Abstract—Although considerable effort has been devoted to both
fundamental frequency ( 0 ) and spectral envelope estimation in
the field of speech processing, the problem of determining 0 andspectral envelopes has largely been tackled independently. If 0
were known in advance, then the spectral envelope could be estimated very reliably. On the other hand, if the spectral envelope
were known in advance, then we could obtain a reliable 0 estimate. 0 and the spectral envelope, each of which is a prerequisiteof the other, should thus be estimated jointly rather than independently in succession. On this basis, we develop a parametric speech
spectrum model that allows us to estimate the 0 and spectral envelope simultaneously. We confirmed experimentally the significant
advantage of this joint estimation approach for both 0 estimationand spectral envelope estimation.
Index Terms—Expectation–maximization (EM) algorithm,
estimation, spectral envelope estimation, speech analysis.

0

I. INTRODUCTIONPECTRAL envelope estimation and fundamental freestimation play very important roles in aquencywide range of speech processing activities including speechcompression, speech recognition, and synthesis. Conventionally, considerable effort has been dedicated to tackling these
problems independently. By contrast, this paper describes theand spectral envelope simultaimportance of estimating theneously. For this purpose, we develop a new speech analyzer
based on a parametric speech source-filter model that includes
and spectral envelope parameters.
both
In the filter-based speech synthesis framework, we sometimes
assume a speech production model based on a linear systemwith a single pulse train as the input, which represents a vocal
cord excitation. In this model, since the input power spectrum
is assumed to have harmonic components with equal powers, asmooth function passing through the prominent spectral peaks
corresponds approximately to the power spectrum of the vocal

S

Manuscript received January 17, 2008; revised September 16, 2008. First
published November 17, 2009; current version published July 14, 2010. The
associate editor coordinating the review of this manuscript and approving it for
publication was Prof. Gaël Richard.
H. Kameoka is with the Media Information Laboratory, NTT Communication
Science Laboratories, Kanagawa 243-0198, Japan (e-mail: kameoka@eye.brl.
ntt.co.jp).
N. Ono, and S. Sagayama are with Graduate School of Information Scienceand Technology, The University of Tokyo, Tokyo, 113-8656, Japan (e-mail:onono@hil.t.u-tokyo.ac.jp; sagayama@hil.t.u-tokyo.ac.jp).
Color versions of one or more of the figures in this paper are available online
at http://ieeexplore.ieee.org.
Digital Object Identifier 10.1109/TASL.2009.2036287

tract impulse response. In this paper, when we refer to “spectralenvelope,” we assume that it follows this definition.
Many techniques can be used to estimate spectral envelopes.
For example, linear predictive coding (LPC) can be understoodas a spectral envelope extractor in the sense that it tries to fit anall-pole spectrum to the observed power spectrum using a spectral distortion measure associated with a statistical distribution
assumption related to excitation source signals [1]–[5]. For instance, when an excitation source signal is assumed to follow a
Gaussian distribution, the corresponding distortion measure iscalled the Itakura–Saito distance, which imposes larger penalties on negative deviations than on positive deviations of the
all-pole spectrum from the observed spectrum. If we can ensurethat an excitation source signal follows an assumed distribution,
LPC can be a good estimator of a spectral envelope. However,when changes, in general the statistics of the excitation signal
will change correspondingly. For this reason, we must underwith a different assumption regarding thetake LPC for each
is unknown we cannot decidestatistical distribution, but ifwhat distribution should be assumed in the first place.
The cepstrum method [6] can also be used to estimate thespectral envelope by low-pass filtering a log-amplitude spectrum interpreted as a signal. But this estimator is also sensitive
to : the envelope estimate tends to descend into the space between the harmonics.
The discrete all-pole modeling method [7] aims to mitigate
dependency problem in LPC, and is designed to fit anthisall-pole spectrum to a discrete set of harmonic components exestimation) prior to the analysis. A generalized
tracted (viaversion of the discrete all-pole approach to the estimation ofmoving average and autoregressive moving average models is
proposed in [8]. Similarly, the discrete cepstrum method [9]is an extension of the cepstrum method, and directly estimatesthe cepstral coefficients where only the extracted harmonics are
considered to be the observed data. The regularized discrete
cepstrum method [10], [11] applies a regularization techniqueto the discrete cepstrum approach to impose smoothness conditions on the spectral envelope estimates. STRAIGHT [12],another state-of-the-art technique, begins by estimating . It
then determines an appropriate length for the analysis window
estimate, and applies discrete Fourier analaccording to theysis. These methods have been shown to provide high-precision
estimaspectral envelope estimation results with a reliabletion preprocessor.
estimation
On the other hand, although a huge number ofalgorithms have been developed (see review articles by Hess[13], [14]), their reliability is still limited. One critical issue indetermination is how to avoid subharmonic or pitch halving

1558-7916/$26.00 © 2010 IEEE
Authorized licensed use limited to: NTT ATSUGI. Downloaded on August 17,2010 at 19:43:48 UTC from IEEE Xplore. Restrictions apply.

1508

IEEE TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, VOL. 18, NO. 6, AUGUST 2010

errors. In a mathematical sense, the period of the signal,
which is the inverse of , is defined as the minimum value. However, this definition applies
such thatstrictly only to a perfectly periodic signal and speech departs
from perfect periodicity. Therefore, we would like to find the. Difficulties inminimum value such thatclude finding a way to measure the degree of periodicity of signals. , which is likely to be the smallest member of an infinite
set of time shifts that leave the signal “almost” invariant, cannot
be determined easily because if is the true period, we obtainfor all. We note here that
corresponds to the period of the subharmonic. One effective way
to avoid subharmonic errors is to introduce a smoothness measure/constraint for spectral envelopes (that is, limit the variation in partial amplitude across the frequency axis) [15]–[20].
To clarify this point, let us consider a situation where we are
estimate obtained with an
extractor. If thisgiven anestimate corresponds to(i.e. double the true period), for example, it amounts to assuming that the speech spectrum has a
harmonic structure with zero power for all the odd-order harmonics. We may conjecture that the spectral envelope of such aspectrum would be nonsmooth. The smoothness measure therefore signals the irregularity of the speech spectrum when thevalue is half the true .
determined
From the above discussions we can draw the following
determination is the more
conclusion: the more reliable theaccurate the spectral envelope estimation becomes, and, on theother hand, the more reliable the spectral envelope estimation
determination becomes. Givenis the more reliable theestimation and spectral
this chicken-and-egg relationship,
envelope estimation should preferably be performed together
rather than independently in succession. This is the standpoint
we adopt in this paper for formulating a combined model of thespectral envelope and spectral fine structure based on .

the Fourier transform of its analytic signal representation isagain a pulse sequence given by

II. FORMULATION OF PROPOSED METHODThis approximation is justified under the sparseness assumption whereby the power spectrum of the sum of multiple signal
components is approximately equal to the sum of the power
spectra generated independently from the components. The accuracy of this approximation increases as the interferences between the harmonics decrease, where the cross termis such thatis sufficiently smaller. Assuming
to be a Gaussian window,
thanis also a Gaussian distribution function with a frequency spread of such that

A. Speech Spectrum Modeling
A short-time segment of a speech signal
can be modeled
as an output of the linear system of the vocal tract impulse rewith the source excitationsuch thatsponse
(1)

(4)
where
is theparameter,
the Dirac delta function, and runs over the integers. Multiplying
by the vocal
and then taking the convolutract frequency responseof the window function with the frequency responsetion yields the complex spectrum of the short-time segment ofvoiced speech(5)
We will use the approximation of its power spectrum as a modelof the speech spectrum ( Fig. 1)(6)

is a window function. In the Fourier
where is time anddomain, the above equation is written as
(2)

(7)

,
,
, and
are thewhere is the frequency,
,
,
, and
, respectively.
Fourier transforms ofLetting the excitation source signal
be a pulse sequencewith a pitch period such that

We see from (6) that the power of each harmonic component
is determined according to the spectral envelope function. As we want
to be a smooth and non-negative
function of , we introduce the following Gaussian mixture
function (see Fig. 2):

(3)

(8)

Authorized licensed use limited to: NTT ATSUGI. Downloaded on August 17,2010 at 19:43:48 UTC from IEEE Xplore. Restrictions apply.

KAMEOKA et al.: SPEECH SPECTRUM MODELING FOR JOINT ESTIMATION OF SPECTRAL ENVELOPE AND FUNDAMENTAL FREQUENCY

1509

Y!

Fig. 3. Compound model of spectral envelope and fine structure j ( )j .
Fig. 1. Linear system approximation model in the power spectrum domain.

spectrum should be flat. Now, if becomes sufficiently large
for the tails of adjacent Gaussians to cover each other, the harmonic structure disappears and the model given by (10) appears
fairly similar to a white spectrum. However, as the approximation given in (6) becomes less accurate in this case, a more
careful modeling of unvoiced speech should be investigated in
the future. Another reason for treating as a free parameter isthat it has been effective in avoiding the local minimum problem
during the optimization process, which will be described later.
Now, we would like to find the optimal estimates ofsuch that the present model best
fits the observed spectrum.

H!

Fig. 2. Spectral envelope model: j ( )j .

with

B. Parameter Optimization(9)
The underlying model for the vocal tract filter with respect tothe above spectral envelope model is shown in Appendix I. The
scale parameter determines the gain of the spectrum model.
From (6) –(8), the speech spectrum can be written concisely as(10)where is assumed to be bounded for convenience. We noticefrom (10) that the spectral model we present here is a compoundmodel of two Gaussian mixture models, which represent the
spectral envelope and the spectral fine structure, respectively(see Fig. 3). Note that the present model is assumed to contain the null frequency component. Although the null frequency
component is usually not considered a partial, this contributes
comduring the model fitting process to preventing theponent from being pulled into the low-frequency region when alow-frequency noise component is present.
Until now, we have assumed voiced speech with a harmonic
structure, but by treating in (10), which has thus far been constant, as a free parameter, the model can also approximate an
unvoiced speech spectrum fairly well. In linear system-based
speech synthesis, a white noise signal is often used as the excitation input to synthesize unvoiced speech so the input power

Based on the fact that the present power domain model iscomposed of multiple Gaussian distribution functions with the
mixture weights constrained by each other, the current model fitting problem can be viewed as a natural1 extension of Gaussian
mixture model estimation using the expectation–maximization(EM) algorithm [21]. This is one advantage we may derive fromdealing with the power-domain observation.
, namely the
Suppose that we are given an observation
Fourier transform of a short-term segment of a speech signal.
The current problem is to minimize the distortion measure beand. We define
tween nonnegative functions
the distortion measure as Csiszár’s -divergence [22](11)
which is often used for non-negative matrix factorization(NMF) [23]. It simply reduces to the KL divergence when
. Since the model
contains the parameters that characterize the spectral envelope
structure and the spectral fine structure, this optimization leads
to a joint estimation of the spectral envelope and . Note that
this measure is not derived from the stochastic modeling ofa time domain signal but defined for the sake of convenience
when deriving an efficient optimization process.
is represented in the form
Now, recall that

(12)
1The maximum-likelihood estimation of GMM parameters basically amounts
to the problem of fitting a Gaussian mixture density function to a data histogramwith the KL divergence as the fitting criterion.

Authorized licensed use limited to: NTT ATSUGI. Downloaded on August 17,2010 at 19:43:48 UTC from IEEE Xplore. Restrictions apply.

1510

IEEE TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, VOL. 18, NO. 6, AUGUST 2010

Based on this fact, we will show that the model parameters canbe efficiently estimated with an EM-like iterative algorithm.
such thatFirst, for any weight functions

cannot be minimized with respect to
We notice thata closed-form expression because of the third term

in

(18)
(13)
More specifically, this part reads

and
(14)
we obtain the following inequation:(19)
from which we find thatis nonlinear in ,and. However, as this term is the sum of the heights of the samwith an interval , it is convenient topled points of. Approximating
approximate it with the integral
the Gaussian integral with the Riemann sums with subintervalsof an equal length of :
(15)
(20)

by invoking Jensen’s inequality based on the concavity of thelogarithm function such that
leads us to

(21)
where

By using
to denote the upper bound of
the right-hand side of the inequation (15), equality
holds if and only if(16)since the left-hand side of (20) is 1. Substituting (21) into (19),
it follows that

, i.e.,

(22)

(17)

We can thus conjecture that the third term of the divergence) depends very weakly on ,
, and
. The
(and
update equations for the parameters other than and can thus
be obtained approximately by maximizing the term

Equation (17) is obtained by setting the variation of the funcwith respect toat zero. If we are able
tionalto decrease
with respect to , thencan be decreased iteratively in the following way.
is given by (17) with an arbitrary , theWhenbecomes equal to. Then, theobjective function
parameter that decreases
while keeping
fixed will necessarily decrease, since inequation (15)guarantees that the original objective function is always even
. Therefore, by repeatingsmaller than the decreased
the update ofby (17) and the update ofthat de, the objective function, which is bounded fromcreases
below by zero, decreases monotonically and converges toa stationary point. This approach is often referred to as the
majorization method [24] and has been adopted for various
optimization problems including NMF [23].

(23)Now, the parameter update equations for ,,, ,,and are obtained as follows (see the Appendix for their derivations):

..
.

..
.

..

.

..
.

Authorized licensed use limited to: NTT ATSUGI. Downloaded on August 17,2010 at 19:43:48 UTC from IEEE Xplore. Restrictions apply.

(24)

KAMEOKA et al.: SPEECH SPECTRUM MODELING FOR JOINT ESTIMATION OF SPECTRAL ENVELOPE AND FUNDAMENTAL FREQUENCY

1511

TABLE I
ACCURACIES OF F ESTIMATION

Fig. 4. Observed power spectra of voiced (top) and unvoiced (bottom) speech
and the corresponding spectral envelope estimates.

(25)

(26)

(27)

(28)

and the superscript refers to the iterwhereation cycle. Some examples of the estimated envelope
withcan be seen in Fig. 4.
III. EXPERIMENTAL EVALUATIONS
A. Single Voice

Estimation

extractor, we tested our
To confirm its performance as an
method on ten Japanese speech data of male (“ myi”) and female (“fym”) speakers from the ATR speech database [25] andextractor “YIN” [26] for comparison.
chose the well-known
All the speech data were around 5 s long, monaural, and sampled

at 16 kHz. All the power spectra were computed using a Hanning window that was 32 ms long with a 10-ms overlap. The
Gaussians, andspectral model was made using
Gaussians. The
the envelope model was made using. Threenumber of free parameters was therefore
independent sets of iterations were run for each analysis frame
and the starting conditions were different in each case. The initial values of were set at 47, 94, and 141 Hz, respectively,
and of these conditions, the converged parameter set that gave
the minimum value was considered to be the global optimum.
As the present optimization algorithm usually converged after
ten iterations, each search was run for ten iterations. The initialvalues were determined uniformly, and andwere initialized at 31 and 313 Hz, respectively.
was initially set at
8000/m Hz.
estimation task, we defined two error criteria,
For annamely deviations over 5% and 20%, from the hand-labeledreference as fine and gross errors, respectively. The former
criterion shows how precisely the proposed analyzer is able toand the latter reveals the robustness with respect toestimate
pitch doubling/halving pitch errors. The areas where reference
were given by zero were not considered in the accuracy
computation. As a second evaluation, we took the average ofandfor the entire
the cosine measures between
analysis interval to confirm the appropriateness of the choicesof distortion measure for minimization and of the model for
expressing speech power spectra. These results can be seen in
Table I. The numbers in the brackets in Table I are the results
obtained with YIN. The source code was kindly provided to
us by its authors. The results confirm that our method is as
accurate as YIN when it comes to roughly estimating , andsignificantly outperforms YIN as regards precise estimation.
Thus, our method would be especially useful when a very
estimate is required, which is exactly the case with
preciseestimates.
spectral envelope estimation algorithms that use
We should note however that the parameters used for YIN may
not do it full justice. The results seem to be rather good fora frame-by-frame algorithm, which encourages us to embed
this envelope structured model into the parametric spectrogram model proposed in [27], [28] to exploit the temporalconnectivity of speech attributes and thus realize a further
improvement.
(the number of Gaussian
Although we have chosen andcomponents for the spectral envelope model and the spectral

Authorized licensed use limited to: NTT ATSUGI. Downloaded on August 17,2010 at 19:43:48 UTC from IEEE Xplore. Restrictions apply.

1512

IEEE TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, VOL. 18, NO. 6, AUGUST 2010

fine structure model) experimentally, we infer that the perforextractor would depend strongly on the model
mance as anorder. Determining the model order remains an open problem
that must be investigated.
B. Synthesis and Analysis
Here we evaluate the accuracy of spectral envelope estimation. To accomplish this, we must use speech signals as the experimental data whose true spectral envelope is known in advance. For this purpose, we created several synthetic speech signals, which were made using three types of linear filter, namely
an all-zero filter, an all-pole filter and a pole-zero filter, and the
input excitation. The input excitation we used here is a linear
modulates linearly fromchirped single pulse signal, whose
100 to 400 Hz within 2 seconds. We chose filters with the following characteristics:

All zero 1

All zero 2

All pole
Fig. 5. Frequency responses of the synthesis filters. (a) All-zero type (1). (b)All-zero type (2). (c) All-pole type. (d) Pole-zero type.

Pole zeroThe frequency responses of the respective filters can be seen in
Fig. 5. As a measure for assessing the accuracy of the spectralenvelope estimation we chose the “Spectral Distortion (SD),”defined by(29)

where refers to the frequency-bin index,
is the true(reference) spectral envelope, andis the spectral envelope estimate.
The experimental results are shown in Fig. 6. Fig. 6(a)–(d)are the results when the tests were undertaken using the data
created respectively by all-zero (1), all-zero (2), all-pole, andpole-zero. Each graph shows the transitions of SD values withinof the input excitation modtwo seconds during which theulates from 100 to 400 Hz. These graphs show that as theof the input increases, conventional methods such as 40-order
LPC and LPC cepstrum tend to obtain poorer results. This isperhaps because the envelope estimates descend into the spacevalues. The accuracy of the enbetween the partials for highvelope estimates obtained with the proposed method does notvalues. This is obviously
seem to deteriorate even with highbecause the proposed method attempts to estimate the spectral

Authorized licensed use limited to: NTT ATSUGI. Downloaded on August 17,2010 at 19:43:48 UTC from IEEE Xplore. Restrictions apply.

KAMEOKA et al.: SPEECH SPECTRUM MODELING FOR JOINT ESTIMATION OF SPECTRAL ENVELOPE AND FUNDAMENTAL FREQUENCY

1513

TABLE II
PREFERENCE SCORE (%) OF THE SYNTHESIZED SPEECH GENERATEDBY THE PROCEDURE DESCRIBED IN SECTION III-C

LPC envelope is too smooth to realize a good fit with the true envelope, and the cepstrum method always seems to obtain poorer
results.
C. Analysis and Synthesis

Fig. 6. Comparison of the accuracies of spectral envelope estimation with theproposed method and the conventional methods. Each graph shows the transitions of SD values during two seconds. (a) All-zero type (1). (b) All-zero type(2). (c) All-pole type. (d) Pole-zero type.

fine structure at the same time. On the other hand, the 14-order

As the advantage of the filter-type speech synthesis framework is its flexibility as regards modifying speech, here we evaluate the basic performance of the present method as a speech
modifier.
We performed a psychological experiment to evaluate the intelligibility of pitch-modified synthesized speech generated byusing the parameters obtained with the present and LPC analyzers. The parameters extracted using the present analyzer
were transformed into a synthesized speech signal using thetime-domain representation of the present model described inAppendix I. As we see from (30), a vocal tract impulse response
can be constructed employing the estimates of , , and, and using randomly chosen. Then, we can simply generate a synthesized signal by convolving a pulse train (with an. As theinterval of a certain period) with the constructed
test set, we used speech data consisting of five vowels (/a/, /i/,/u/, /e/, /o/) and 40 randomly chosen words uttered by a female
speaker that were excerpted from the same database. Analyses
were performed using a Hanning window that was 32 ms longwith a 10-ms overlap. The dimension of the parameters for the
proposed model was set at 45. In this experiment, the LPC order
was set at 45 so that the number of degrees of freedom would
be the same in both models, thus allowing for a fairer comparwere extracted by using theison. For the LPC analysis, thesupplementary
extraction tool included in the Snack Sound
Toolkit.2 Each synthesized speech used for the evaluation was
excited with an estimated vocal tract characteristic by a pulse
sequence at intervals with a different pitch period from the original one. The pitch periods were modified to 80% and 120% of
the pitch periods obtained from the original speech. We let ten
listeners choose the speech that they thought was the more intelligible and obtained a preference score for the results using
the proposed analyzer. The preference score, shown in Table II,shows that the intelligibility of the synthesized speech generated
by the present analyzer is higher than that generated by the LPC
analyzer.
2[Online]. Available: http://www.speech.kth.se/snack/Authorized licensed use limited to: NTT ATSUGI. Downloaded on August 17,2010 at 19:43:48 UTC from IEEE Xplore. Restrictions apply.

1514

IEEE TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, VOL. 18, NO. 6, AUGUST 2010

It should be noted that as LPC models of order 12 or 14 areoften considered to be appropriate for the synthesis of speechvowels, different settings for the LPC model may have yielded
better performance.
IV. CONCLUSION
determination and spectral
In this paper, we formulated
envelope estimation as a joint optimization problem with respect to a composite function model of the spectral envelope
and the spectral fine structure. The experiments confirmed theextractor, spectral enveeffectiveness of our method as anlope extractor, and speech modifier.
The extension of the present model to concurrent utterancesof multiple speakers is straightforward. We can construct amixed speech spectrum model by mixing the speech spectrum
models introduced in this paper each of which has its own degree of freedom. The derivation of the optimization algorithm
is exactly the same as the derivation described in this paper.
In future, we will examine its application to monaural speech
separation.

=0

Fig. 7. Equations (30) and (31) with 
for (a) M(a) Single Gaussian case. (b) Gaussian mixture case.

= 1 and (b) M = 2.

APPENDIX IDERIVATION OF SPECTRAL ENVELOPE MODEL(32)

Here we show the underlying model for the vocal tract
filter that leads to the spectral envelope model introduced in
Section II-A.
Because human vocal tracts are at most about 20 cm long, it
would be fairly natural to assume that vocal tract impulse re. Based on this assumption,sponses are dominant around
we consider modeling a vocal tract impulse response by superimposing several Gabor functions, all of which are centered at. Using an analytic signal representation, we define theimpulse response
as
(30)
where
,
,
, and
represent the carrier frequency,
amplitude modulation parameter, scale, and starting phase forthe carrier sinusoid of the th Gabor function, respectively. Itshould be noted that in a strict sense this model is not valid as
regards allowing the filter to be acausal. Now, the Fourier trans(see Fig. 7) is written as
form of(31)
The power spectrum is then given as

Assuming thatis an independent random variable uniformly, the second term can be
distributed on the interval
canceled out by taking the expectation ofwith respectsuch thattoand use (32) toWe shall assume that
express spectral envelopes. By imposing a scale constraint(33)
we finally arrive at (8).
We notice from an inspection of (30) and (32) that constraining each Gabor function in (30) so that they are localized
densely around, which can be accomplished simply byat a large value, amounts to imposing a smoothness
setting
constraint on the spectral envelope, as each Gaussian function.
in (32) spreads in proportion to
APPENDIX IIDERIVATION OF UPDATE EQUATIONS
Update equations for guaranteeing approximately the noncan be derived in various ways, only two of
increase ofwhich are described in this section owing to space limitations.
One way involves adopting a pure coordinate descent apwith respect
proach. That is, we seek to find the minimum of
to each parameter while keeping the other parameters fixed at, , and , the update
the newest update values. As for ,equations can be achieved approximately by maximizing (23).
Taking the partial derivative of (23) with respect to each of theparameters and setting at zero, we obtain the following:

where

Authorized licensed use limited to: NTT ATSUGI. Downloaded on August 17,2010 at 19:43:48 UTC from IEEE Xplore. Restrictions apply.

(34)

KAMEOKA et al.: SPEECH SPECTRUM MODELING FOR JOINT ESTIMATION OF SPECTRAL ENVELOPE AND FUNDAMENTAL FREQUENCY

As for and ,can be directly minimized with (26) and(28).
The second way is to adopt an approximation technique. Thisapproach also adopts the coordinate descent approach but differs
in that we further approximate (23) with(35)
which can be minimized jointly with respect toandthrough a linear simultaneous equation. This approximation isbased on the assumption that
. In this way, we
obtain the update equations given in Section II-B.

ACKNOWLEDGMENT
The authors would like to thank Dr. J. Le Roux for fruitful
discussions and the reviewers for their constructive suggestions
that have helped improve the manuscript.

REFERENCES[1] F. Itakura and S. Saito, “Analysis synthesis telephony based upon
the maximum likelihood method,” in Proc. 6th Int. Congr. Acoust.
(ICA’68), 1968, C-5-5, C17-20.
[2] B. S. Atal, “Effectiveness of linear prediction characteristics of the
speech wave for automatic speaker identification and verification,” J.
Acoust. Soc. Amer., vol. 55, no. 6, pp. 1304–1312, 1974.
[3] M. N. Murthi and B. D. Rao, “All-pole modeling of speech based on
the minimum variance distortionless response spectrum,” IEEE Trans.
Speech Audio Process., vol. 8, no. 3, pp. 221–239, May 2000.
[4] D. Giacobello et al., “Sparse linear predictors for speech processing,”
in Proc. Interspeech, 2008.
[5] E. Denoël and J.-P. Solvay, “Linear prediction of speech with a least
absolute error criterion,” IEEE Trans. Acoust., Speech, Signal Process.,
vol. ASSP-33, no. 6, pp. 1397–1403, Dec. 1985.
[6] A. V. Oppenheim and R. W. Schafer, “Homomorphic analysis of
speech,” IEEE Trans. Audio Electroacoust., vol. AU-16, no. 2, pp.
221–226, Jun. 1968.
[7] A. El-Jaroudi and J. Makhoul, “Discrete all-pole modeling,” IEEE
Trans. Signal Process., vol. 39, no. 2, pp. 411–423, Feb. 1991.
[8] R. Badeau and B. David, “Weighted maximum likelihood autoregressive and moving average spectrum modeling,” in Proc. Int. Conf.
Acoust., Speech, Signal Process. (ICASSP’08), 2008, pp. 3761–3764.
[9] T. Galas and X. Rodet, “An improved cepstral method for deconvolution of source-filter systems with discrete spectra: Application
to musical sound signals,” in Proc. 1991 Int. Comput. Music Conf.
(ICMC’90), 1990, pp. 82–84.
[10] O. Cappé, J. Laroche, and E. Moulines, “Regularized estimation of cepstrum envelope from discrete frequency points,” in Proc. IEEE Workshop Appls. Signal Process. Audio Acoust. (WASPAA’95), 1995.
[11] M. Campedel-Oudot, O. Cappé, and E. Moulines, “Estimation of thespectral envelope of voiced sounds using a penalized likelihood approach,” IEEE Trans. Speech Audio Process., vol. 9, no. 5, pp. 469–481,
Jul. 2001.
[12] H. Kawahara, “Speech respresentation and transformation using adaptive interpolation of weighted spectrum: Vocoder revisted,” in Proc. Int.
Conf. Acoust., Speech, Signal Process. (ICASSP’97), 1997, vol. 2, pp.
1303–1306.
[13] W. J. Hess, Pitch Determination of Speech Signals. Berlin, Germany:
Springer-Verlag, 1983.
[14] W. J. Hess, “Pitch and voicing determination,” in Advances in Speech
Signal Processing, S. Furui and M. M. Sohndi, Eds. New York:
Marcel Dekker, 1992, pp. 3–48.

1515[15] A. P. Klapuri, “Multiple fundamental frequency estimation based on
harmonicity and spectral smoothness,” IEEE Trans. Speech Audio
Process., vol. 11, no. 6, pp. 804–816, Nov. 2003.
[16] C. Yeh, A. Roebel, and X. Rodet, “Multiple fundamental frequency
estimation of polyphonic music signals,” in Proc. 2005 IEEE Int.
Conf. Acoust., Speech, Signal Process. (ICASSP’05), 2005, vol. 3, pp.
225–228.
[17] T. Virtanen and A. Klapuri, “Separation of harmonic sounds using
linear models for the overtone series,” in Proc. 2002 IEEE Int. Conf.
Acoust., Speech, Signal Process. (ICASSP’02), 2002, vol. 2, pp.
1757–1760.
[18] F. Bach and M. Jordan, “Discriminative training of hidden markov
models for multiple pitch tracking,” in Proc. 2005 IEEE Int. Conf.
Acoust., Speech, Signal Process. (ICASSP’05), 2005, vol. 5, pp.
489–492.
[19] G. Cauwenberghs, “Monaural separation of independent acousticalcomponents,” in Proc. 1999 IEEE Symp. Circuit Syst. (ISCAS’99),
1999, pp. 62–65.
[20] R. J. Leistikow, H. D. Thornburg, J. O. Smith, III, and J. Berger,“Bayesian identification of closely-spaced chords from single-frame
STFT peaks,” in Proc. 17th Int. Conf. Digital Audio Effects (DAFx04),
2004, pp. 5–8.
[21] A. P. Dempster, N. M. Laird, and D. B. Rubin, “Maximum likelihoodfrom incomplete data via the EM algorithm,” J. R. Statist. Soc., ser. B,
vol. 39, no. 1, pp. 1–38, 1977.
[22] I. Csiszár, “ I -divergence geometry of probability distributions
and minimization problems,” Ann. of Probability, vol. 3, no. 1, pp.
146–158, 1975.
[23] D. D. Lee and H. S. Seung, “Algorithms for non-negative matrix factorization,” in Proc. 2000 Adv. Neural Inf. Process. Syst. (NIPS’00),
2000, pp. 556–562.
[24] K. Lange, D. Hunter, and I. Yang, “Optimization transfer using surrogate objective functions,” J. Comp., Graph. Statist., vol. 9, no. 1, pp.
1–20, 2000.
[25] H. Kuwabara, K. Takeda, Y. Sagisaka, S. Katagiri, S. Morikawa, andT. Watanabe, “Construction of a large-scale Japanese speech databaseand its management system,” in Proc. 1989 IEEE Int.. Conf. Acoust.,
Speech, Signal Process. (ICASSP’89), 1989, pp. 560–563.
[26] A. de Cheveigné and H. Kawahara, “YIN, a fundamental frequency
estimator for speech and music,” J. Acoust. Soc. Amer., vol. 111, no. 4,
pp. 1917–1930, 2002.
[27] H. Kameoka, T. Nishimoto, and S. Sagayama, “A multipitch analyzer
based on harmonic temporal structured clustering,” IEEE Trans. Audio,
Speech, Lang. Process., vol. 15, no. 3, pp. 982–994, Mar. 2007.
[28] J. Le Roux, H. Kameoka, N. Ono, A. de Cheveigné, and S. Sagayama,“Single and multiple pitch contour estimation through parametric
spectrogram modeling of speech in noisy environments,” IEEE Trans.
Audio, Speech, Lang. Process., vol. 15, no. 4, pp. 1135–1145, May
2007.

Hirokazu Kameoka (M’07) received B.E., M.E.,
and Ph.D. degrees all from the University of Tokyo,
Tokyo, Japan, in 2002, 2004, and 2007, respectively.
He is currently a Research Scientist at NTT
Communication Science Laboratories, Kanagawa,Japan. His research interests include computational
auditory scene analysis, acoustic signal processing,speech analysis, and music applications.
Dr. Kameoka is a member of the Institute ofElectronics, Information and Communication Engineers (IEICE), the Information Processing Societyof Japan (IPSJ), and the Acoustical Society of Japan (ASJ). He was awarded
the Yamashita Memorial Research Award by IPSJ, the Best Student Paper
Award Finalist at 2005 IEEE International Conference on Acoustics, Speech,
and Signal Processing (ICASSP2005), the 20th Telecom System Technology
Student Award by the Telecommunications Advancement Foundation (TAF)in 2005, the Itakura Prize Innovative Young Researcher Award by ASJ, 2007
Dean’s Award for Outstanding Student in the Graduate School of Information
Science and Technology by the University of Tokyo, 1st IEEE Signal Processing Society Japan Chapter Student Paper Award in 2007, the Awaya Prize
Young Researcher Award by ASJ in 2008, and the IEEE Signal Processing
Society 2008 SPS Young Author Best Paper Award in 2009.

Authorized licensed use limited to: NTT ATSUGI. Downloaded on August 17,2010 at 19:43:48 UTC from IEEE Xplore. Restrictions apply.

1516

IEEE TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, VOL. 18, NO. 6, AUGUST 2010

Nobutaka Ono (M’02) received the B.E., M.S., andPh.D. degrees in mathematical engineering and information physics from the University of Tokyo, Tokyo,
Japan, in 1996, 1998, and 2001, respectively.
He has been with the Graduate School of Information Science and Technology, University ofTokyo, as a Research Associate since 2001, and asa Lecturer since 2005. His research interests include
acoustic signal processing, speech processing, music
processing, sensing and measurement, and auditorymodeling.
Dr. Ono was the Secretary of the Technical Committee of Psychological andPhysiological Acoustics in Japan from 2006 to 2009. He is a member of the Institute of Electronics, Information and Communications Engineers (IEICE), theInstitute of Electrical Engineers of Japan (IEEJ), the Acoustical Society of Japan(ASJ), the Society of Instrument and Control Engineers (SICE), and the Information Processing Society of Japan (IPSJ). He received the Sato Prize Paper
Award from ASJ in 2000, the Igarashi Award at the Sensor Symposium on Sensors, Micromachines, and Applied Systems from IEEJ in 2004, the Awaya Prize
Young Researcher Award from ASJ in 2007, and the Best Paper Award at theInternational Symposium on Industrial Electronics (ISIE) in 2008.

Shigeki Sagayama (M’82) received the B.E., M.E.,
and Ph.D. degrees from the University of Tokyo,
Tokyo, Japan, in 1972, 1974, and 1998, respectively,
all in mathematical engineering and information
physics.
He joined Nippon Telegraph and Telephone Public
Corporation (currently, NTT) in 1974 and started his
career in speech analysis, synthesis, and recognition
at NTT Labs in Musashino, Japan. From 1990, he
was Head of the Speech Processing Department,
ATR Interpreting Telephony Laboratories, Kyoto,
Japan where he was in charge of an automatic speech translation project.
In 1993, he was responsible for speech recognition, synthesis, and dialog
systems at NTT Human Interface Laboratories, Yokosuka, Japan. In 1998,
he became a Professor of the Graduate School of Information Science, Japan
Advanced Institute of Science and Technology (JAIST), Ishikawa. In 2000,
he was appointed Professor at the Graduate School of Information Scienceand Technology (formerly, Graduate School of Engineering), the Universityof Tokyo. His major research interests include the processing and recognitionof speech, music, acoustic signals, handwriting, and images. He was theleader of anthropomorphic spoken dialog agent project (Galatea Project) from2000–2003.
Prof. Sagayama received the National Invention Award from the Institute ofInvention of Japan in 1991, the Chief Official’s Award for Research Achievement from the Science and Technology Agency of Japan in 1996, and otheracademic awards including Paper Awards from the Institute of Electronics, Information and Communications Engineers, Japan (IEICEJ) in 1996 and fromthe Information Processing Society of Japan (IPSJ) in 1995. He is a member ofthe Acoustical Society of Japan, IEICEJ, and IPSJ.

Authorized licensed use limited to: NTT ATSUGI. Downloaded on August 17,2010 at 19:43:48 UTC from IEEE Xplore. Restrictions apply.