2007 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics

October 21-24, 2007, New Paltz, NY

SINGLE CHANNEL SPEECH AND BACKGROUND SEGREGATION THROUGH
HARMONIC-TEMPORAL CLUSTERING
Jonathan Le Roux1,2 , Hirokazu Kameoka1 † , Nobutaka Ono1 , Alain de Cheveigné2 , Shigeki Sagayama1
1

Graduate School of Information Science and Technology, The University of Tokyo, Tokyo, Japan
2
CNRS, Université Paris 5, and Ecole Normale Supérieure, Paris, France{leroux,kameoka,onono,sagayama}@hil.t.u-tokyo.ac.jp, Alain.de.Cheveigne@ens.fr

ABSTRACTThe design of effective algorithms for single-channel analysis ofcomplex and varied acoustical scenes is a very important andchallenging problem. We present here the application of the recently introduced Harmonic-Temporal Clustering (HTC) framework to single channel speech enhancement, background retrieval
and speaker separation. HTC processing relies on a precise parametric description of the voiced parts of speech derived from thepower spectrum. We explain the positioning of the algorithm inside the Computational Acoustic Scene Analysis (CASA) area,
describe the theoretical background of the method, show throughpreliminary experiments its basic feasibility, and discuss potential
improvements.
1. INTRODUCTION
The design of an effective method for the analysis of complex andvaried acoustical scenes is a very important and challenging problem. Many applications, such as automatic speech recognition(ASR) or speaker identiﬁcation, would for example beneﬁt fromthe ability of such a system to reduce acoustic interferences which
often occur simultaneously with speech in real environments. Being able to locate and extract a portion of an acoustical scene, oron the contrary to cancel it, would also lead to very appealing applications such as instrument separation inside a multipitch track,background music recovery or voice activity detection (VAD).
Although there exist general methods for signal separation orenhancement in multisensor frameworks, based for example on independent component analysis or spatial ﬁltering, single-channel
solutions are necessary for many applications, such as in telecommunication, analysis of monaural CD recordings, automatic news
search or background music determination in television programs,
for example. Implementing the single-channel separation problem in computers has proven to be extremely challenging. However, human listeners are able to concentrate on listening to a speciﬁc target sound without difﬁculty even in the situation where
many speakers are talking at the same time, and they are are able
to do so even in monaural situations. This fact has persuaded
many scientists that the human auditory system has a signiﬁcantability to actively recognize the external environment, in otherwords to perform an auditory scene analysis (ASA) and has been
attracting interest since Bregman’s book was published [1]. In[1], Bregman has shown through experiments the psychologicalevidences that the auditory system segregates the acoustic signal† Now with NTT Communication Science Laboratories, NTT Corporation, Atsugi, Japan

978-1-4244-1619-6/07/$25.00 ©2007 IEEE

279

into spectrogram-like pieces, called auditory elements, which aregrouped into auditory streams according to several grouping cues.
Recent efforts are being directed towards the reproduction of thisability of the auditory system in computers, in a framework called“Computational Auditory Scene Analysis (CASA)” [2, 3, 4, 5].
The main focus of today’s CASA research is to develop a source
separation method based upon the grouping cues suggested by
Bregman. More speciﬁcally, the main purpose is to extract usefulfeatures (for example, the fundamental frequency F0 ) or to restore
the target signal of interest by performing the segregation process
and grouping process through a computational algorithm.
Many methods utilizing the grouping cues have been proposed(see [5] for a list of references), in most of which the grouping
process is usually implemented in two steps. Instantaneous features are ﬁrst extracted at each discrete time point, which corresponds to the grouping process in the frequency direction, anda post-processing is then performed on these features to reduce
errors and/or obtain continuous tracks, through hidden Markov
model (HMM), multiple agents, or some dynamical system suchas Kalman ﬁltering, corresponding to the grouping process in thetime direction. Considering that from an engineering point of view,
it should be more efﬁcient to perform the analysis in both time andfrequency directions simultaneously, we formulated a uniﬁed estimation framework “Harmonic-Temporal Clustering” (HTC) for
the two dimensional structure of time-frequency power spectra [5],in contrast to the conventional strategy.
In our previous works [5, 6], we presented an F0 estimation
algorithm based on HTC for both music and speech. We present
here the next step in the application of HTC, single-channel speech
signal processing for speech enhancement, background retrieval,and speaker separation. We use the estimated HTC models to build
masking functions which we apply to the power spectra to extractspeech, retrieve the background and separate two speakers.
2. INTRODUCTION OF HTC
We brieﬂy introduce here the HTC framework. More details on thetheory, the implementation and the performance of F0 estimation
through HTC can be found in [5, 6, 7].
2.1. General HTC method
Consider the wavelet power spectrum W (x, t), where x is logfrequency and t is time, of a signal recorded from an acousticalscene. The problem is to approximate it as well as possible as
the sum of K parametric source models qk (x, t; Θ), where Θ isthe set of model parameters, modeling the power spectrum of K
“objects” each with its own F0 contour μk (t).

As described in [5], each source model is expressed as a Gaussian Mixture Model with constraints on the characteristics of thekernel distributions: supposing there is harmonicity with N partials modeled in the frequency direction, and that the power envelope is described using Y kernel functions in the time direction,
we can rewrite each source model as
qk (x, t; Θ) =

−1
N Y
X
X

Skny (x, t; Θ),(1)with kernel densities Skny (x, t; Θ) which are assumed to have thefollowing shape:
wk vkn uknye
2πσk φkTime

Log-frequency

n=1 y=0

Skny (x, t; Θ) October 21-24, 2007, New Paltz, NYPower density

2007 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics(x−μk (t)−log n)2(t−τk −yφk )2
−
−
2σ 2
2φ2
k
k

,

(2)
where the parameters wk , vkn and ukny are normalized to unity.
A graphical representation of a HTC source model qk (x, t; Θ) canbe seen in Fig. 1.
2.2. Speech modeling
In order to model the spectrum of a speech utterance, we makeseveral assumptions. First, we suppose that the F0 contour issmooth and deﬁned on the whole interval: we will not makevoiced/unvoiced decisions, and F0 values are assumed continuous. Previously, HTC was applied to musical signals with piecewise constant F0 contours. For speech, it is necessary to model
continuously varying contours. For that, we chose to use cubic
spline functions as a general class of smooth functions. To model
the variation over time of the shape of the spectral envelope, thespeech segment is modeled as a succession in time of slightly overlapping source models sharing a common F0 contour. In the single
speaker case, all the source models inside the HTC model share thesame F0 contour: μk (t) = μ(t), ∀k, while for multiple speakers,
according to the number of F0 contours that we want to estimate,
we group source models into subsets sharing a common F0 contour. As the structure is assumed harmonic, the model takes advantage of the voiced parts of the speech utterance.
It is shown in [5] and [6] that prior distributions can be introduced on the parameters, and that the HTC model can be efﬁciently
optimized using an EM-like algorithm to minimize the “distance”
between the parametric HTC model and the observed spectrogrammeasured by the I-divergence between them.
2.3. Noise ModelingWe introduce a noise model to cope with the broadband background noise which can be a disturbance in the process of clustering the harmonic portions of speech. The idea to design thismodel was that detecting the harmonic parts of the spectrogram ina noisy background corresponds to searching for thin and harmonically distributed “islands” which emerge from a “sea” of noise.
We thus chose to model the noise using a mixture of Gaussian
distributions with large ﬁxed variance and with centers ﬁxed on agrid, the only parameters being the mixing weights and the ratio
of noise power inside the whole spectrogram. The noise model parameters are optimized simultaneously with the other parametersin the same EM-like framework [6].
3. SPECTROGRAM MODIFICATION
Speech enhancement, background retrieval and speaker separation
can be performed very simply through HTC, based on the classic

978-1-4244-1619-6/07/$25.00 ©2007 IEEE

280

Figure 1: Proﬁle of a HTC source model qk (x, t; Θ)idea of masking function, often used in CASA oriented methods.
3.1. Ratio of HTC models
When the noise to be cancelled is broadband, an estimation of
the spectrogram where the interference has been cancelled can beobtained from the original spectrogram by looking, at each point(x, t), at the proportion of the “clean” part inside the whole parametric model. In the same way, in the multiple speakers case, whenseveral speech models are used simultaneously, the ratio of one
speech model inside the whole at each time-frequency bin can beused as a mask. This corresponds to the E-step in the EM algorithm. In the course of the optimization of the model, it is actually on this “cleaned” part of the spectrogram that the F0 contour
estimation is performed, during the M-step, enabling the F0 estimation to perform well even in very noisy environments or within
multiple speakers [6].
3.2. Direct use of HTC models
However, when the noise to be cancelled is not a broadband noise,
we can expect that the noise model will not be able to cope totallywith the background noise. It might thus as well happen that the
total power of the noise model is very low in absence of broadband
noise, and using the ratio of speech model in the whole would thenmake less sense, as this ratio would almost always be close to 1,
thus leading to a very ineffective mask. Introducing noise models to cope with more types of background noise could be a way
to deal with that problem, but it is limited by several problems:
background noises could be of inﬁnitely many kinds, and the multiplication of models would lead to a larger computation cost, andmight also conﬂict with the estimation of the speech model. It
might thus be simpler and more effective to look directly at theestimated speech model itself to build a masking function. The
speech model has been designed to encompass the acoustic characteristics of speech, and has by construction a harmonic structure.
To use the speech model as a mask, we use a “ﬁltered” versionof the speech model which broadens its peaks:
Q̃(x, t) =1,

1 + ( Q(x,t))p(3)

where Q is the speech model, normalized such that its maximum
is 1,  a small constant, typically between 10−3 and 10−1 , and p a
constant which tunes the broadening of the peaks.
The interference can also be retrieved using a masking function obtained through HTC. We simply consider 1 − Q̃, where Q̃
is deﬁned as in (3), as a mask function to apply to the noisy spectrogram to retrieve the interference part from the mixture.
In all cases, the modiﬁed power spectrogram is coupled withthe phase of the noisy spectrogram to obtain an estimation of thedenoised complex spectrogram. An inverse transform is then used
to synthesize the denoised signal back.

2007 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics

MixtureHu-Wang
p = 1,  = 0.005
p = 1,  = 0.1
p = 2,  = 0.1

n0
-3.27
16.34
-3.98
-6.04
-6.51

Table 1: SNR results (dB) for the enhanced speech.
n1
n2
n3
n4
n5
n6
-4.08 10.20 4.39
4.05
-5.83
1.90
7.83 16.71 8.32 10.88 14.41 16.89-0.54 15.11 6.63
4.90
-5.76
1.91
5.61 11.39 7.79
6.69
-5.31
4.20
7.61
9.56
7.24
6.53
-5.57
4.72

3.3. STFT and Wavelet Spectrograms
The HTC models are optimized to ﬁt the wavelet power spectrumof an utterance. The basic idea to synthesize an enhanced or denoised speech, is, as described above, to modify the wavelet power
spectrum and use an inverse wavelet transform. So far, a Gabortransform has been used for HTC analysis in [5] and [6], and a ﬁrst
option is to perform the analysis-synthesis with this transform. As
argued in [7], the HTC framework is naturally designed to ﬁt a
power spectrogram obtained with a constant-Q ﬁlterbank based on
an analyzing wavelet whose Fourier transform is of the form
8
`
´2 !
>
<exp − log ω(ω > 0)Ψ(ω) = Ψ∗ (ω) =
. (4)
4σ 2
>
:
0
(ω  0)One can thus also try to perform the analysis-synthesis with thisanalysing wavelet.
But as the models obtained on the wavelet spectrogram haveparametric expressions, it is also possible to generate masking
functions in the linear-frequency domain, and simply use STFT
and inverse STFT, for example by overlap-add method, to generate a modiﬁed speech or background. Although this process is
expected to lead lower-quality results than wavelet-based synthesis due to misﬁtting between the analysis and synthesis methods,
it is much faster, and still gives interesting preliminary results on
the basic performance of the method.
3.4. Relation to adaptive comb ﬁltering methods
Comb ﬁltering ﬁrst requires a robust F0 estimation, and according
to [4], suffers from the fact that it retains too much interference
as it passes through all frequency components close to the multiples of target F0 . The HTC frameworks, on the opposite, features
an embedded estimation of the F0 , and estimates separately thepowers and shapes of the harmonics. HTC also estimates the parameters simultaneously in the time and frequency directions, thus
integrating continuity in the model and making it more robust, on
the contrary to comb ﬁltering methods. In the multiple speakercase especially, if the speech of two speakers are harmonically related, a comb ﬁlter method will inevitably fail, while HTC can still
perform well, as shown in Section 4.3.

October 21-24, 2007, New Paltz, NY

n7
6.57
11.97
7.01
7.66
7.06

n8
10.53
14.44
10.65
9.86
8.68

n9
0.75
5.27
0.69
-0.47
-0.98

masking, even if it introduces artifacts or slightly modiﬁes speech,
which is not advantageous in an SNR way.
4.1. Speech enhancement
We used a corpus of 100 mixtures of voiced speech and interference [2], commonly used in CASA research. There are 10 interferences: n0, 1-kHz pure tone; n1, white noise; n2, noise bursts;
n3, “cocktail party” noise; n4, rock music; n5, siren; n6, trill telephone; n7, female speech; n8, male speech; and n9, female speech.
The results are shown in Table 1, for different mask settings on various types of interferences. Each value in the table represents theaverage SNR for one interference mixed with 10 target utterances.
Mixture designates the Signal-to-Interference Ratio in the original mixture, and Hu-Wang stands for the state-of-the-art algorithm
presented in [4]. The HTC enhanced speech is generated using
Eq. (3) with p and  as indicated in the ﬁrst column. 40 iterationsof HTC were performed, and the parameters were as in [6], witha speech model and a noise model, apart from the number of harmonics considered, which was set to 40. We note that according
to the type of interference, different settings lead to better results.
This is due to the fact that these settings change the sharpness ofthe peaks of the mask, introducing a trade-off between Signal-toInterference Ratio (SIR) and Signal-to-Artifact Ratio (SAR). According to the acoustical properties of the interference to reduce, itis thus possible to use different values for the parameters.
For n1, n2, n3 and n4, the results are promising, with results
close to the Hu-Wang algorithm for the ﬁrst three. It is so far
less effective on the other interferences. For n0 and n5, which areinterferences with a strong localized signal overlapping with harmonics of the speech utterance, our method failed as the speechmodel mistook the interference for a harmonic and rose the powerof the corresponding Gaussian functions. This should be dealt within the future, for example by using a stronger constraint on the
power of the harmonics and on the shape of the power envelope in
the time direction. The analysis of SIR and SAR results tends toshow that our algorithm reduces the interference very effectively,
but creates artifacts. The question whether these artifacts are perceptually signiﬁcant or not should be further investigated, as is the
improvement of the quality of the synthesized speech, especiallyusing inverse wavelet transforms. We shall note in particular that,
contrary to the algorithm by Hu and Wang, our method does notseem to generate musical noise.

4. EXPERIMENTS
We performed three types of experiments to conﬁrm the basic effectiveness of our method for speech enhancement, background
retrieval and speaker separation, using STFT spectrograms. We
used the SNR as a quantitative measure of the performance of ouralgorithm, and tested different settings for the mask functions. We
shall stress the fact that the SNR, although it is an easy-to-computeobjective value, may not give a full idea of the performance ofa CASA system, and may especially differ signiﬁcantly from aperceptive evaluation. The human ear tends to prefer a stronger

978-1-4244-1619-6/07/$25.00 ©2007 IEEE

281

4.2. Speech cancellation for background enhancement
The HTC framework can be used not only for speech enhancement, but also for speech cancelling and background retrieval, as
explained in section 3.2. There are very interesting potential applications to this task, such as the retrieval of speech in the background, or background music retrieval. This last issue is of particular importance: in an acoustical scene where someone is speaking with music playing in the background, being able to “clean”
the background music from the speech would ease the automatic

2007 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics

Table 2: SNR results (dB) for the retrieved background.
n3
n4
n7
n8
n9
Mixture-4.39 -4.05 -6.57 -10.53 -0.75
p = 3, 3 = 5.10−4
0.90 -0.01 -2.99-4.85 -6.44
p = 3, 3 = 2.10−3
0.39
0.28 -2.62-5.12 -5.36

Frequency (Hz)

2000

500
0
0.88

1.32

1.76

Figure 2: Clean spectrogram of speaker v0Frequency (Hz)

1500
1000
500
0
0

0.44

0.88

1.32

1.76

Time (s)Figure 3: Clean spectrogram of speaker n9
2000
1500
1000
500
0
0

0.44

0.88

1.32

1.76

Time (s)Figure 4: Spectrogram of the mixture v0n9
2000
1500
1000
500
0
0

0.44

0.88

1.32

1.76

Time (s)Figure 5: Estimated spectrogram of speaker v0
2000
1500
1000
500
0
0

5. CONCLUSIONS

0.44

2000

Frequency (Hz)By using two speech models, we showed in [6] that the F0 contours of concurrent speech by two speakers with close average
power can be effectively estimated through HTC. As explainedin Section 3.1, the speech of each speaker can be reconstructedusing the proportion of its model in the whole. We performed aseparation experiment on two mixtures. On the ﬁrst one, v0n9
from Cooke’s database, we obtained an improvement from an initial SNR of 0dB to 6dB for both speakers. This is a very difﬁculttask as the harmonics of the speakers almost constantly overlap.
The clean spectrograms of the utterances v0 (male speaker) and n9(female speaker) can be seen in Fig. 2 and Fig. 3 respectively, andtheir mixture in Fig. 4. The corresponding spectrograms extractedusing HTC from the mixture v0n9 can be seen in Fig. 5 and Fig. 6
respectively. An exponent 0.3 was used as non-linear scaling forthe ﬁgures. The second mixture constitutes of a male Japanese
speaker uttering “aoi” and a female Japanese speaker uttering “oiwo ou”, the female speaker being 5dB stronger. We obtained an
improvement from the initial −5dB to +3dB for the male speaker,
and from the initial +5dB to 9.3dB for the female speaker.

1000

Time (s)Frequency (Hz)4.3. Speaker Separation

1500

0

Frequency (Hz)recognition of copyrighted material inside television programs forexample. Another interesting application is the automatic cancellation of the vocal part inside a music piece to produce karaokeaccompaniments at lower cost and from the real song, thus with abetter quality than the usual MIDI accompaniments.
The results for the interferences which retrieval is of potential
interest for applications are presented in Table 2. Mixture designates the Interference-to-Signal Ratio in the original mixture. To
our knowledge, no results by previous methods are available on
this task. Background is retrieved through Eq. (3) with p and 
as indicated. A substantial increase in SNR could be obtained forn3, n4, n7 and n8. For an interference constituting of other speechwith close average power such as n9, we will see in the next sectionthat using two speech models leads to better results.

0.44

0.88

1.32

1.76

Time (s)We presented several applications of the HTC framework for single channel speech signal processing problems and showed its
basic effectiveness for tasks as various as speech enhancement,
speech cancellation for background retrieval, with potential applications in background music retrieval, and speaker separation.
Future works include the improvement of the quality of the output sound by using wavelet transforms instead of STFT ones, athorough study of the performance of HTC compared to previous
works, both on voiced speech and on data containing unvoiced
speech, the use of auditory-based distortion measures and an evaluation through subjective listening tests.
6. REFERENCES[1] A. S. Bregman, Auditory Scene Analysis. MIT Press, Cambridge, 1990.
[2] M. P. Cooke, “Modeling Auditory Processing and Organisation,” Ph.D. dissertation, University of Shefﬁeld, 1993.

978-1-4244-1619-6/07/$25.00 ©2007 IEEE

October 21-24, 2007, New Paltz, NY

282

Figure 6: Estimated spectrogram of speaker n9[3] D. Ellis, “Prediction-driven Computational Auditory Scene
Analysis,” Ph.D. dissertation, MIT, 1996.
[4] G. Hu and D. Wang, “Monaural speech segregation. based on
pitch tracking and amplitude modulation,” IEEE. Trans. Neural Networks, vol. 15, pp. 1135–1150, 2004.
[5] H. Kameoka, T. Nishimoto, and S. Sagayama, “A multipitch
analyzer based on harmonic temporal structured clustering,”
in IEEE Trans. on Audio, Speech and Language Processing,vol. 15, no. 3, 2007, pp. 982–994.
[6] J. Le Roux, H. Kameoka, N. Ono, A. de Cheveigné, and
S. Sagayama, “Single and multiple F0 contour estimation
through parametric spectrogram modeling of speech in noisy
environments,” in IEEE Trans. on Audio, Speech and Language Processing, vol. 15, no. 4, 2007, pp. 1135–1145.
[7] H. Kameoka, “Statistical Approach to Multipitch Analysis,”
Ph.D. dissertation, The University of Tokyo, 2007.