REAL-TIME BLIND EXTRACTION OF DOMINANT TARGET SOURCES
FROM MANY BACKGROUND INTERFERENCE SOURCES
Hiroshi Sawada

Ryo Mukai

Shoko Araki

Shoji Makino

NTT Communication Science Laboratories, NTT Corporation
2-4 Hikaridai, Seika-cho, Soraku-gun, Kyoto 619-0237, Japan{sawada,ryo,shoko,maki}@cslab.kecl.ntt.co.jp
Note that xj (t) = N
k=1 xjk (t). The task should be performed only with the M observed signals x 1 , . . . , xM .
The number of target sources Q and the total number ofsources N are unknown. The number Q is assumed to be
no more than the number of sensors M , and N may belarger than M .

ABSTRACTThis paper presents a method for enhancing target sources of interest and suppressing background interference sources. The target sources are assumed to be close to sensors, to have dominant
powers at the sensors, and to have non-Gaussianity. The method
is based on a two-stage process where independent component
analysis (ICA) is first employed in each frequency bin and thentime-frequency masking is used to improve the performance further. We propose a new sophisticated method for deciding thenumber of target sources and then selecting their frequency components. Experimental results for cocktail party situations arepresented to show the effectiveness of the method. We also describe a real-time implementation of the method.

The first issue is how to extract target sources s 1 , . . . , sQ
blindly. Even if the total number of sources N could belarger than M , independent component analysis (ICA) [1,2] with an N = M assumption produces M componentsthat maximize an ICA criterion. We assume that the target
sources are non-Gaussian, close to sensors, and dominant
in the mixtures. Therefore, we expect that some of the
M components correspond to target sources s 1 , . . . , sQ
whose ICA criteria are high.

1. INTRODUCTION

We employ ICA in the frequency domain [4–7]. This isbecause it is efficient [7] and can also be used with timefrequency masking, which is discussed below. An additional operation that should be performed is the selectionof each target component in every frequency bin. This is
known as the permutation problem of frequency-domain
BSS [8]. We solve this problem by clustering the basis
vectors (6) produced by ICA after some normalization.
Moreover, the number Q of dominant target sources canbe decided by examining the variances of the clusters.

The technique for estimating individual source components from their mixtures at sensors is known as blindsource separation (BSS) [1, 2]. With audio applications
such as speech enhancement, the sources do not necessarily have equal significance. We often want to extract
only a few sources that are close to sensors, have dominant powers, and/or have interesting features.
Based on this idea, we have proposed a method for blindly
extracting one dominant target source and suppressing theinterference sources [3]. This paper generalizes the method[3] so that the number of target sources can be more thanone. Let us formulate the task. Suppose that a few close
target sources s1 , . . . , sQ and other background sources
sQ+1 , . . . , sN are convolutively mixed and observed at M
≥ 2 sensors
xj (t) =

N 

k=1

hjk (l) sk (t − l), j = 1, . . . , M,

The next issue is that some interference still remains in
the extracted frequency components when N > M . Timefrequency masking is known to be efficient for sources
with sparseness in the time-frequency domain, such as
speech. The performance depends on how well we canspecify the time-frequency slots where the target sourceis active. Although many methods [9–12] for specifying
masks have been proposed, they are basically applied totwo sensors. We propose a new criterion for specifying
masks that can be applied to any number (≥ 2) of sensors.

(1)lwhere hjk (l) represents the impulse response from source
k to sensor j. The goal is to have output signals y i (t), i =1, . . . , Q that are the components of s i measured at a selected sensor J:xJi (t) =hJi (l) si (t − l), i = 1, . . . , Q. (2)This paper also presents a real-time implementation of the
proposed method. More accurately, it is a blockwise batch
implementation [13], where ICA is applied to a block ofa few seconds but the system input-output delay is kept atless than a second.

l

73

STFT

ISTFT

Basis vector
clustering

ICA

Time-frequency
masking

Figure 1: Flow of proposed method
components is equal to M :2. FLOW OF PROPOSED METHOD

y(f, τ ) = W(f ) x(f, τ ),Figure 1 shows the flow of the proposed method. This section explains each part except “Basis vector clustering”,
which is explained in more detail in Sec. 3.

where W is an M × M separation matrix and y(f, τ ) =[y1 (f, τ ), . . . , yM (f, τ )]T is the vector of independent components. Then, we calculate the inverse of W to obtainbasis vectors

2.1. Frequency domain operations[a1 , · · · , aM ] = W−1 , ai = [a1i , . . . , aMi ]T .

First, time-domain observed signals x j (t) sampled at frequency f s are converted into frequency-domain time-series
signals xj (f, τ ) with an L-point STFT:

xj (τ + r) win(r) e−j2πf r ,By multiplying both sides of (5) by W , the observation
vector x(f, τ ) is represented by a linear combination ofbasis vectors a1 , . . . , aM :(3)x(f, τ ) =r=−L/2Mai (f )yi (f, τ ).

(7)

i=1

where f ∈ {0, L1 fs , . . . , L−1
L fs } is a frequency, win(r)is a window that tapers smoothly to zero at each end, andτ is a new index representing time.
The following operations are performed in the frequency
domain. There are two advantages to this. First, the convolutive mixtures (1) can be approximated as instantaneous mixtures at each frequency:
xj (f, τ ) ≈

(6)

−1

L/2−1

xj (f, τ ) =

(5)

N


hjk (f )sk (f, τ ),Some of these independent components correspond to thecomponents of dominant sources. However, the correspondence is not clear at this stage because of the permutation ambiguity of ICA. Thus, basis vector clustering(Sec. 3) is performed to solve this permutation ambiguity.
2.3. Time-frequency masking
In a general case where the total number of sources N islarger than the number of sensors M , independent components y1 (f, τ ), . . . , yQ (f, τ ) produced by ICA have someresiduals caused by the limitation of spatial filtering [3].
Time-frequency masking is used to reduce such residuals,
and is performed by

(4)

k=1

where hjk (f ) is the frequency response from source k tosensor j, and sk (f, τ ) is a frequency-domain time-series
signal of sk (t) obtained by the same operation as (3). The
second advantage is that the sparseness of a source signal
becomes prominent in the time-frequency domain if thesource is colored and non-stationary such as speech.
After several operations, which are explained in the following subsections and in Sec. 3, time-domain output signals yi (t) are obtained by an inverse STFT (ISTFT) :1ỹi (f, τ ) ej 2πf ryi (τ + r) =L·win(r)L−11
f ∈{0, L fs , ...,Lỹi (f, τ ) = Mi (f, τ ) yi (f, τ ), i = 1, . . . , Q(8)where 0 ≤ Mi (f, τ ) ≤ 1 is a mask specified for eachtime-frequency slot (f, τ ). The masks should have large
values for time-frequency slots where the target sourceis active. We specify masks based on the angle θ i (f, τ )between ai (f ) and x(f, τ ) calculated in the space transformed by a whitening matrix V(f ) = R −1/2 , R =x(τ ) x(τ )H τ . The angle is calculated by

fs }for i = 1, . . . , Q.

θi (f, τ ) = arccos
2.2. Independent component analysis (ICA)|bHi (f ) z(f, τ )|,||bi (f )|| · ||z(f, τ )||(9)where z(f, τ ) = V(f )x(f, τ ) is a whitened sample andbi (f ) = V(f )ai (f ) is the basis vector in the whitened
space. The angle would be close to 0 degrees if the target source is active in the time-frequency slot. Then, we

To extract the components of dominant sources, we apply ICA to observation vectors x(f, τ ) = [x 1 (f, τ ), . . . ,xM (f, τ )]T by assuming that the number of independent74

calculate a mask by using a logistic function
1,(10)
Mi (θi (f, τ )) =g(θ1 + e i −θT )where θT and g are parameters specifying the transition
point and its steepness, respectively. Typical values are(θT , g) = (0.333π, 20). As θT becomes smaller, the residual power that appears in ỹ i decreases but the musicalnoise in yi increases.

Framewise
Filtering &Masking

STFT

ISTFT

Blockwise

Whitening

Basis vector
clustering

ICA

3. BASIS VECTOR CLUSTERING
Figure 2: System structure for a real-time implementation

Basis vector clustering has two purposes: to solve the permutation ambiguity in (7) and to select the target sources.
The previous method [3] selects the cluster with the minimum variance as the only target source. This paper generalizes this idea so that the system decides the number Q
of dominant target sources, and then selects those target
sources.
We normalize all basis vectors a i (f ), i = 1, . . . , M , forall frequency bins f = 0, L1 fs , . . . , L−1
L fs such that they
form clusters, each of which corresponds to an individual
source. This normalization is performed by selecting areference sensor J and calculating
arg[aji (f )/aJi (f )](11)āji (f ) ← |aji (f )| exp j
4f c−1 dmax

for the other sources [3].
To identify target source clusters, we sort C 1 , . . . , CM sothat their variances are arranged in ascending order:
J1
JQ
JM≤ ··· ≤≤ th var ≤ · · · ≤
,
|C1 |
|CQ |
|CM |

where th var is a predefined threshold for specifying theset of target sources. Then, to align the permutation ambiguities, we renumber the indexes of the basis vectors by
ak (f ) ← aΠf (k) (f ),Πf = argminΠQ||āΠ(k) (f ) − ck ||2 .

(17)

k=1

(12)

We also renumber independent components y 1 (f, τ ), . . . ,yM (f, τ ) accordingly.

to āi (f ) = [ā1i (f ), . . . , āMi (f )]T .
After normalizing all the basis vectors, we employ a clustering algorithm to find clusters C 1 , . . . , CM formed bynormalized vectors āi (f ). The centroid c k of a cluster Ck
is calculated byā/|Ck |, ck ← ck /||ck ||,
(13)
ck ←

4. REAL-TIME IMPLEMENTATIONThis section describes a real-time implementation of themethod. Our current implementation is mainly with Matlab and partially with C, and performed on an ordinary
desktop or notebook computer that has a single CPU. Thus,
we employ a blockwise batch algorithm [13]. Figure 2
shows the system structure. The upper path, which consists of STFT, Filtering & Masking and ISTFT, is operated framewise, where the input-output delay is kept as
small as a few STFT frames. The lower path, which includes Whitening, ICA, and Basis vector clustering, isoperated blockwise to ensure learning convergence andalso to interleave operations for many frequency bins. A
typical configuration is as follows: STFT frame size =128 ms, frame shift = 32 ms, learning block size = 98 frames(3.232 s), block interval = 49 frames (1.568 s). This configuration means that Whitening and ICA for each frequency bin and basis vector clustering are performed every 1.568 s.

ā∈Ckwhere |Ck | is the number of vectors in C k . The clustering
criterion is to minimize the total sum J of the squared
distances between cluster members and their centroid
M


J =
Jk , Jk =||ā − ck ||2 .
(14)
k=1

(16)

where Πf : {1, . . . , Q} → {1, . . . , M } is a one-to-one
mapping decided for each frequency f bywhere c is the propagation velocity and d max is the maximum distance between the sensor J and a sensor ∀ j ∈{1, . . . , M }. We then apply unit-norm normalizationāi (f ) ← āi (f ) / ||āi (f )||

(15)

ā∈Ck

This minimization can be performed efficiently with the
k-means clustering algorithm [14].
Once we have found M clusters C 1 , . . . , CM , we need
to identify the clusters that correspond to dominant target sources s1 , . . . , sQ . We decide that a cluster Ck with asmall variance Jk /|Ck | belongs to the set of target sources.
The rationale behind this criterion is that the direct-path(nearfield) mixing model is more valid for s 1 , . . . , sQ than75

445 cm

Target position

Target source
candidates

90cm

Table 1: Average SIR improvement (dB)

25cm

b120

InputSIRi
Only ICAICA and T-F masking

80cm
270cm

355 cm

70cm

a120

80cm

a120 b120 c170
−3.9
12.5
15.1

−3.6 −5.9
13.6 14.5
16.5 17.6

c170 (170 cm height)
2.5cm

Mic. 3Mic. 1

4cm

2.5cm
2.3cm

Mic. 4

7. REFERENCES
Interference
speech sources

3.5cm Mic. 2• Microphones arranged
3-dimensionally
150cm[1] S. Haykin, Ed., Unsupervised Adaptive Filtering (Volume
I: Blind Source Separation), John Wiley & Sons, 2000.
[2] A. Hyvärinen, J. Karhunen, and E. Oja, Independent Component Analysis, John Wiley & Sons, 2001.
[3] H. Sawada, S. Araki, R. Mukai, and S. Makino, “Blindextraction of a dominant source signal from mixtures ofmany sources,” in Proc. ICASSP 2005, Mar. 2005, vol. III,
pp. 61–64.
[4] P. Smaragdis, “Blind separation of convolved mixtures in
the frequency domain,” Neurocomputing, vol. 22, pp. 21–
34, 1998.
[5] L. Parra and C. Spence, “Convolutive blind separation of
non-stationary sources,” IEEE Trans. Speech Audio Processing, vol. 8, no. 3, pp. 320–327, May 2000.
[6] L. Schobben and W. Sommen, “A frequency domain blind
signal separation method based on decorrelation,” IEEE
Trans. Signal Processing, vol. 50, no. 8, pp. 1855–1865,
Aug. 2002.
[7] H. Sawada, R. Mukai, S. Araki, and S. Makino,“Frequency-domain blind source separation,” in Speech
Enhancement, J. Benesty, S. Makino, and J. Chen, Eds.,
pp. 299–327. Springer, Mar. 2005.
[8] H. Sawada, R. Mukai, S. Araki, and S. Makino, “A robust
and precise method for solving the permutation problem offrequency-domain blind source separation,” IEEE Trans.
Speech Audio Processing, vol. 12, no. 5, pp. 530–538, Sept.
2004.
[9] Ö. Yilmaz and S. Rickard, “Blind separation of speech
mixtures via time-frequency masking,” IEEE Trans. Signal
Processing, vol. 52, no. 7, pp. 1830–1847, July 2004.
[10] S. Araki, S. Makino, A. Blin, R. Mukai, and H. Sawada,“Underdetermined blind separation for speech in real environments with sparseness and ICA,” in Proc. ICASSP
2004, May 2004, vol. III, pp. 881–884.
[11] N. Roman and D. Wang, “Binaural sound segregation for
multisource reverberant environments,” in Proc. ICASSP
2004, May 2004, vol. II, pp. 373–376.
[12] D. Kolossa and R. Orglmeister, “Nonlinear postprocessing
for blind speech separation,” in Proc. ICA 2004 (LNCS
3195), Sept. 2004, pp. 832–839.
[13] R. Mukai, H. Sawada, S. Araki, and S. Makino, “Blindsource separation for moving speech signals using blockwise ICA and residual crosstalk subtraction,” IEICE Trans.
Fundamentals, vol. E87-A, no. 8, pp. 1941–1948, Aug.
2004.
[14] R. O. Duda, P. E. Hart, and D. G. Stork, Pattern Classification, Wiley Interscience, 2nd edition, 2000.

80cm
80cm

25cm
25cm

• Reverberation time: RT60 = 130 ms• Room height: 250 cm• Loudspeakers and microphones at height of 120cm except c170

Figure 3: Experimental conditions

5. EXPERIMENT
To evaluate the separation performance for multi-target
cases, we performed experiments in a batch mode. We
measured impulse responses h jk (l) under the conditions
shown in Fig. 3. The speaker positions simulated a cocktail party situation. Mixtures at the microphones weremade by convolving the impulse responses and 6-second
English and Japanese speech sampled at 8 kHz. We usedfour 3-dimensionally arranged microphones (M = 4). The
system knew only the maximum distance (4 cm) between
the reference microphone (Mic. 1) and the others. Experiments were conducted with 10 combinations of 9 speech(3 targets + 6 background interference sources). We usedth var = 0.015 to specify the set of target sources by (15).
Table 1 shows the average signal-to-interference ratio (SIR)
improvements obtained solely with ICA, and with a combination of ICA and T-F masking. Even in such hard situations, the system succeeded in enhancing and separating
the target sources.

6. CONCLUSION
This paper described some extensions of our previously
reported method [3] that blindly extracts a close target
source from many background interference sources. The
first extension makes it possible for the system to handle
plural target sources by examining the cluster variances ofnormalized basis vectors. The second extension involves
the realization of a real-time implementation.

76