A SPARSE COMPONENT MODEL OF SOURCE SIGNALS AND ITS APPLICATION TO
BLIND SOURCE SEPARATION
Yu Kitano, Hirokazu Kameoka† , Yosuke Izumi, Nobutaka Ono, Shigeki Sagayama
Graduate School of Information Science and Technology, The University of Tokyo, Japan{kitano,izumi,onono,sagayama}@hil.t.u-tokyo.ac.jp†NTT Communication Science Laboratories, NTT Corporation,
kameoka@eye.brl.ntt.co.jp
ABSTRACTIn this paper, we propose a new method of blind source separation(BSS) for music signals. Our method has the following characteristics: 1) the method is a combination of the sparseness-based modelof source signals and the factorized basis model in nonnegative matrix factorization (NMF), 2) it is assumed that only one basis which
structure source signals is active at each time-frequency bin of theobserved signals, in order to degrade the degree of freedom, 3) parameter estimation algorithm is based on the EM algorithm regarding the index of the only one active basis as the hidden variable. We
develop the formulation at a different point from NMF and show
source separation performance in some simulation experiments.
Index Terms— Blind source separation, nonnegative matrix
factorization, EM algorithm, sparseness of source signals, timefrequency masking
1. INTRODUCTION
Blind source separation (BSS) for music signals has been investigated as the basic technique of automatic music transcription, music equalizer and music information retrieval (MIR) [1]. In suchcase that music performance is recorded using few microphones orremixed to cut a CD, mixing process of sources is unknown. Thenthe aim of BSS is to separate the mixed music signals into eachsource.
There are roughly two approaches for BSS. One approach isto use the spatial information of sources. In overdetermined case,where there are more sensors than sources, we can separate observed signals into each source signal using independent component
analysis (ICA). However music signals are often stereo recordings
and there are more sources than sensors in real environments, therefore we must take into account the source separation in underdetermined case. There are many methods of BSS in underdeterminedcase based on the sparseness of sources, which have been discussed[2, 3, 9]. The other approach is to use the spectral information ofsources. Nonnegative matrix factorization (NMF) is one of the methods to use it, and has been investigated as a separation method ofmonaural audio signal [6]. In the framework of NMF, it is assumed
that acoustic signals in amplitude (or power) domain are composedof just so many components (music signals often satisfy this assumption.), and target signal can be factorized into the spectral patterns
which structure the signal and their activations at nonnegative domain. Various extensions of NMF have been researched [5, 7, 8].
In this paper, we propose a new method of BSS for music signal. We have a new assumption that only one basis which structure

978-1-4244-4296-6/10/$25.00 ©2010 IEEE4122

frequency [Hz]

8000

6000

4000

2000

00

1

2

3

4

5times [s]Fig. 1. Spectrogram of mixed signal which includes three sources(red, green, blue).
sources is active at each time-frequency bin, and develop a formulation in combination of the sparseness-based model of source signals
and the factorized basis model in NMF to improve separation performance by using the spatial and spectral information of sources.
2. EXISTING APPROACH OF SPARSENESS-BASED BSS
The most typical method of sparseness-based BSS is time-frequency
masking. Each source signal has the sparse energy in time-frequency
domain as Fig. 1 shows, and it is often assumed only one of them is
active on each time-frequency component. Now we consider observed signal Yω,t (ω = 1, . . . , Ω is a frequency bin index, and(l)t = 1, . . . , T is a time frame index) as mixed L source signals Sω,t(l = 1, . . . , L is a source index) in time-frequency domain by the(l)Short-Time Fourier Transform (STFT). The mask Φω,t to extract thesource signal l can be written as(1 (iω,t = l)(l)Φω,t =,(1)0 (iω,t = l)
where iω,t is the index of the active source in (ω, t), and estimatedsource signal l can be written as
(l)

(l)

Ŝω,t = Φω,t Yω,t .

(2)

The parameter iω,t can be determined by clustering the power ratios
and time delays between left and right channel of observed signals
at each time-frequency bin [2], or by fitting some distribution using
the EM algorithm in the feature domain [3].
There is a little different approach based on the EM algorithm(L)(R)[9]. Let Yω,t = (Yω,t , Yω,t )T be observed signals recorded byICASSP 2010

two microphones in time-frequency domain. Yω,t can be written as„«1(i)Yω,t =(3)Sω,tω,t + Nω,t ,jδω
e iω,twhere δl is the time delay of the source signal l between two mi(L)(R)
crophones and Nω,t = (Nω,t , Nω,t )T is the observation error. If
Nω,t follows some distribution, each parameter can be determinedby applying the EM algorithm regarding the active source index iω,tas the hidden variable. This algorithm gives the probability thateach source is active in each time-frequency bin, and the probability
presents the partition function and works as a soft mask for separation.
3. PROPOSED METHOD
3.1. Observation based on a sparse component model(L)is the index of the active basis in (ω, t), and lω,tis thewhere kω,t∈
index of the source signal which is composed of the basis, kω,tKlω,t . And for simplicity we abbreviate the frequency and time in, lω,t. In this paper we call this model a sparse compodex of kω,tnent model, and this framework is a different approach from NMF.
Thanks to the assumption, the degree of freedom of observation
model can be degraded, and the parameters Hω,k , Ut,k can be determined uniquely.

3.2. Applying the EM algorithm to parameter estimation
We can decide these parameters in the framework of Maximum
A Posteriori (MAP) estimation. For simplicity, it is defined asH ≡ (Hω,k )Ω×T , U ≡ (Ut,k )T ×K , ﬃ ≡ (φω,t,k )Ω×T ×K ,Ψ ≡ {H, U, ﬃ}, ‹ ≡ {δ1 , · · · δL }. By Bayes’ theorem, the posterior distribution of observed signals Yω,t can be written asp(Ψ, ‹, σ 2 |Y) ∝ p(Y|Ψ, ‹, σ 2 )p(Ψ, ‹, σ 2 ),(R)Let Yω,t = (Yω,t , Yω,t )T be observed signals recorded by twomicrophones in time-frequency domain. Approximately Yω,t canbe written as«
L „X
1(l)
(4)
Sω,t + Nω,t ,Yω,t =jδl ωe
l=12

where p(Y|Ψ, ‹, σ ) and p(Ψ, ‹, σ ) are the likelihood and priordistribution of the parameters Ψ. For convenience, we assume thatthe prior distribution of H, U, ﬃ, σ 2 and ‹ are independent, thatp(H), p(ﬃ), p(σ 2 ) and p(‹) are uniform, and that U follows a generalized Gaussian distribution given by(l)where Sω,t is the source signal l, δl is the time delay of the source(L)(R)
signal l between two microphones, and Nω,t = (Nω,t , Nω,t )T is
the observation error which includes reverberation and backgroundnoise. And Nω,t is assumed to follow that`´(5)Nω,t ∼ NC 0, σ 2 I ,where NC (—, Σ) is the complex Gaussian distribution such asx ∼ NC (—, Σ) =1 −(x−—)H Σ−1 (x−—).
eπ|Σ|k∈Klp(U) =k∈Kl(k)where cω,t is a component which structures acoustic signals, Hω,kis a static magnitude spectral pattern called a basis, Ut,k is a timevarying activation, and φω,t,k is a phase spectrum. Kl is the class(l)of bases which structure Sω,t and k (k = 1, . . . , K) is the index ofbases. And we assume
XHω,k = 1(8)ωin order to avoid an indeterminacy in the scaling. Some of the authors proposed this factorized model and its parameter estimation
algorithm in the framework called complex NMF [7].
(k)Here, it is assumed that the component cω,t composed of eachbasis has sparse energy in time-frequency domain and that only one
basis is active in (ω, t) of observed signals. Then Yω,t can be written as„«1Yω,t =(9)Hω,k Ut,k ejφω,t,k + Nω,t ,jδl ωe4123

Yt,kp|Ut,k |1
e− bp ,
−12Γ(1 + p )b(11)where p (0 < p ≤ 2) and b (> 0) are the parameters that decide
the shape of the distribution. The distribution promotes sparsity of
U so that source signals can be composed of as few bases as possible. Then our goal is to obtain the parameter θ ≡ {Ψ, ‹, σ 2 } thatmaximizes
Ξ(θ) ≡ log p(Y|Ψ, ‹, σ 2 ) + log p(U).

(6)

Now we introduce factorized basis model of NMF to this mixture
model in order to utilize the spectral information of source signals.
Acoustic signals are assumed to be composed of just so many components which have their own spectral patterns in amplitude domain,(l)and source signal Sω,t can be factorized asX (k)X(l)
Sω,t =cω,t =Hω,k Ut,k ejφω,t,k ,(7)

(10)

2(12)

It is very difficult to maximize Ξ(θ) directly because the logarithmic posteriori distribution has hidden variable k in eq. (9), however the parameter θ can be determined applying the EM algorithm
as same way as Izumi et al [9]. For parameter estimation including missing data, the EM algorithm introduces an auxiliary function
called the Q function defined using a tentative parameter. On theassumption that only one basis is dominant at each time-frequency
bin, the likelihood p(Yω,t |θ) can be written asp(Yω,t |θ) =|θ)p(Yω,t , kω,tp(Yω,t |Ψk , δl , σ 2 )=,p(kω,t |Yω,t , θ)p(kω,t|Yω,t , θ)(13)

represents the index of the dominant basis in (ω, t) andwhere kω,tsatisfies kω,t∈ Klω,t . p(Yω,t |Ψk , δl , σ 2 ) represents the likelω,tlihood of the observation Yω,t when the basis k is in the direction
δl (k ∈ Kl ), and can be written aslog p(Yω,t |Ψk , δl , σ 2 ) = − log(πσ 4 )−˛˛21 ˛(l)jφω,t,k ˛−aHUe˛Y
˛ ,ω,tω,kt,kω
σ2(14)(l)where aω = (1, exp(jδl ω))T . From eqs. (11) - (14), the Q function can be written as
Q(θ, θ (n) ) =X
k,ω,t(n)mk,ω,t log p(Yω,t |Ψk , δl , σ 2 ) −p
X Ut,kt,kbp,(15)

where(n)mk,ω,t(n)=p(k|Ψ(n) , δ (n) , σ 2=(n)(n)(n)
p(Yω,t |Ψk , δl , σ 2 ).
P(n)(n)2 (n) )k p(Yω,t |Ψk , δl , σ, Yω,t )

(16)
(17)

The parameter θ is estimated by sequential iteration of two steps:(n)• E-step: calculate Q(θ, θ (n) ) (update mω,t,k by eq.(17))• M-step: update θ(n) by θ(n+1) = argmax Q(θ, θ (n) ).
θAnd we are led to get the update rules of θ by differentiating the Q
function with respect to each parameter. Update rules of the param(n)eters except mω,t,k are summarized as follows:
˛P(n+1)Hω,k(n+1)
Ut,k˛(n)(n) ˛ H(l) (n) ˛˛t mk,ω,t Ut,k ˛Yω,t (aω )=2
P=2PP(n)(n) 2t mk,ω,t (Ut,k )˛,(18)

And our framework can estimate not only source signals but also(k)each component. In the same way as eq. (26), a component cω,t ofsource l can be written as, (19)

mk,ω,t (Hω,t Ut,k )2(aω )H Yω,t(k)·.
ĉω,t = P22
k mk,ω,t (Hω,t Ut,k )˛(n)(n) ˛ H(l) (n) ˛˛ω mk,ω,t Hω,k ˛Yω,t (aω )(n)(n) 2(n) p−22
ω mk,ω,t (Hω,k ) + λpσ (Ut,k )=“”HH(l) (n)arg Yω,t(aω),(20)δl=argmax Q(θ, θ (n) ),(21)` 2 ´(n+1)σ=˛˛2
X m(n)
k,ω,t ˛ (n) ˛˛‰ω,t,k ˛ ,2ΩT

(22)

(n+1)
φω,t,kδlk,ω,t(n)(l)(n)(n) jφ(n)where λ = b−p , ‰ω,t,k = Yω,t − (aω )(n) Hω,k Ut,k e ω,t,k . The
update of δl is done by calculating the Q function for all the discreteδl and selecting the maximum because update rule cannot be solved
analytically. In addition, after every update of H, it is standardized
as satisfies eq. (8). And from these update rule, the non-negativity of
H and U is preserved if we start with nonnegative initial conditions
for them.
3.3. Mask design based on Wiener filter for separation
There are several methods to extract source signals from observed
signals using the estimated parameter θ. Here we design a mask
based on Wiener filter for separation in order to utilize the information of power spectra that source signals have. Now the expectationof power spectrum of source signal l can be calculated asih(l)E |Sω,t |2=

X(k)p(k)|cω,t |2 +X

X

p(k ) · 0(23)k ∈K/ lk∈Kl=

mk,ω,t (Hω,k Ut,k )2 ,(24)k∈Kl(l)so source signal Sω,t can be written as(l)Ŝω,t==Fig. 2. alignment of sources and sensors.

ih(l)
(l)
E |Sω,t |2(aω )H Yω,t,(25)P h (l) 2 i ·2
l E |Sω,t |P2(l)(aω )H Yω,tk∈Kl mk,ω,t (Hω,t Ut,k )P·. (26)
22k mk,ω,t (Hω,t Ut,k )

4124

(l)

(27)

4. EXPERIMENT
We evaluated the separation performance of proposed method inreverberant environments through simulations. There were threesources and two microphones in a simulated room illustrated Fig. 2.
The observed signals were calculated by mirror method. We synthesized the source signals of RWC database using MIDI (MusicalInstrument Digital Interface). The instruments of sources are Tenor
Sax, Bass, and Piano. STFT was calculated using Hanning windowthat was 64ms long with a 32ms overlap at a sampling rate of 16kHz.
The algorithm was run for 40 iterations. The parameters were set up
at p = 2, λ = 0.01, K = 30. How to give the initial values of Hand U is as following:
1. Initialize H and U by applying NMF to the observed signals,
2. Reconstruct components at left and right channels of eachfactorized basis using the phase spectrum of the observed signals,
3. Classify K components into source signals by comparing signals of left and right channel using cross-correlation.
As a comparison, we used the DUET by Yilmaz et al. [2] andthe soft masking method using the EM algorithm by Izumi et al.
[9]. We used the improvement of SNR and SIR as measures ofsource separation performance. In this paper,
SIR and SNR wereP
calculated by SIRi = | < si , ŝi > |/ j=i | < sj , ŝi > | andSNRi = |si |/|ŝi − si |, where si represents a vector of the true
ith source signal in the time domain, ŝi is the estimated one andsi = si · < si , ŝi > /|ŝi |2 . Table 1 ,2 and Fig 3 show results. Whilein case that TR = 66.7 ms Izumi’s method and proposed method
show better performance for source separation, in case of long reverberation time, proposed method has best performance among thesemethods.
(l)Fig. 4 shows the spectrogram of source signal Sω,t , estimated(l)(k)
source Ŝω,t and a component of the source cω,t . We can see a spec(k)tral pattern on the spectrogram of cω,t . Then it is indicated that ourmethod can not only separate the observed signals into sources but
also factorize the bases which structure the observed signals.



Avg.
10.3
13.1
8.6
4.3
9.1
11.0

(TGSWGPE[=M*\?

Table 1. Source separation results (SNR[dB])
Condition
Method
s1
s2
s3
DUET
15.0 5.9 10.0
TR = 66.7[ms]Izumi
13.3 9.9 16.1
proposed 11.7 4.7
9.4
DUET
12.7 -3.1
3.4
TR = 273[ms]
Izumi
8.1
5.9 13.3
proposed 10.1 8.5 14.6







Avg.
23.7
27.1
23.0
16.8
24.2
26.8

(TGSWGPE[=M*\?

Table 2. Source separation results (SIR[dB])
Condition
Method
s1s2
s3
DUET
31.4 19.4 20.4
TR = 66.7[ms]Izumi31.3 19.8 30.2
proposed 24.4 25.1 19.6
DUET
32.0 12.3
6.1
TR = 273[ms]Izumi
32.9 11.1 28.8
proposed 28.5 18.8 33.2


6KOG=U?





6KOG=U?






(TGSWGPE[=M*\?

504KORTQXGOGPV=F$?

 &7'6+\WOKRTQRQUGF





6KOG=U?

Fig. 4. Spectrogram of source signal (top) and estimated signal (center) and an estimated component of the source (bottom).

TGXGTDGTCVKQPVKOG=OU?

Fig. 3. Average SNR improvement under various conditions.

5. CONCLUSION
In this paper, we proposed a new method of source separation formusic signals based on the assumption that only one basis which
structures source signals is active at each time-frequency bin, anddeveloped a parameter estimation method based on the EM algorithm in the same way as [9]. The hidden variable corresponds tothe active basis index in our formulation. Our approach is to use thespatial and spectral information, and we confirmed that our method
has better performance to separate the observations in long reverberant environments through a simulation experiment, and that bases
which structure sources can be learned in our framework. We plan
to introduce a noise model appropriate for reverberation and extend
the bases to 2-dimensional components with temporal structure like[5].
6. REFERENCES[1] E. Vincent, H. Sawada, P. Bofill, S. Makino and J. P. Rosca,“First Stereo Audio Source Separation Evaluation Campaign:
Data, Algorithms and Results,” Proc. ICA’07, pp. 552–559,
2007.
[2] O. Yilmaz and S. Rickard, “Blind Separation of Speech Mix-4125

tures via Time-Frequency Masking,” IEEE Transaction on Signal Processing, Vol. 52, No. 7, pp. 1830–1847, 2004.
[3] M. Mandel, D. Ellis and T. Jebara, “ An EM algorithm for localizing multiple sound sources in reverberant environments,”
Proc, NIPS’06, 2006.
[4] D. D. Lee and H. S. Seung, “Algorithms for nonnegative matrix
factorization,” Proc. NIPS’00, pp. 556–562, 2000.
[5] P. Smaragdis, “Non-negative Matrix Factor Deconvolution; Extraction of Multiple Sound Sources from Monophonic Inputs,”
Proc. ICA’04, pp. 494–499, 2004.
[6] T. Virtanen, “Monaural Sound Source Separation by NonNegative Matrix Factorization with Temporal Continuity andSparseness Criteria,” IEEE Transactions on Audio, Speech andLanguage Processing, Vol. 15, No. 3, pp. 1066–1074, 2006.
[7] H. Kameoka, N. Ono, K. Kunio and S. Sagayama, “Complex
NMF: A New Sparse Representation for Acoustic Signals,”
Proc. ICASSP’09, pp. 3437–3440, 2009.
[8] A. Ozerov and C. Févotte, “ Multichannel Nonnegative Matrix Factorization in Convolutive Mixtures. With Application toBlind Audio Source Separation,” Proc. ICASSP’09, pp. 3137–3140, 2009.
[9] Y. Izumi, N. Ono and S. Sagayama, “Sparseness-Based 2ch BSS
using the EM Algorithm in Reverberant Environment,” Proc.
WASPAA’07, pp. 147–150, 2007.