Blind separation and localization of speeches in a
meeting situation
Hiroshi Sawada

Shoko Araki

Ryo Mukai

Shoji Makino

NTT Communication Science Laboratories, NTT Corporation
2-4 Hikaridai, Seika-cho, Soraku-gun, Kyoto 619-0237, Japan
Email: {sawada,shoko,ryo,maki}@cslab.kecl.ntt.co.jp

I. I NTRODUCTION
The theory and algorithms of blind source separation (BSS)and independent component analysis (ICA) became popular in
the signal processing community, as many textbooks [1]–[3]have been published. One of the well recognized applicationsof ICA/BSS is the separation of speeches mixed in a realroom reverberant environment (i.e. solving a cocktail party
problem). The difficulty of this problem lies in the fact thatthe mixing system is not simply instantaneous but convolutive.
Thus, additional effort has been devoted to the separation of
convolutive speech mixtures by many researchers in recent
years, as reported in many papers [4]–[12]. Some of these papers describe effective methods that can attain successful separation, namely an improvement in the signal-to-interference
ratio (SIR) of 10∼15 dB or more, for real recorded speech
mixtures. However, the situations are commonly well set up
so that these conditions are satisfied:
1) The mixing system is roughly time-invariant.
2) All speakers are active most of the time.
3) The number of speakers is known, and less than or equal
to the number of microphones.
This paper reports a trial where BSS techniques are applied
to a meeting situation. The goal is to obtain the enhanced
spoken utterances of each speaker and to identify the location of each speaker from the signals observed at multiple
microphones. It should be noted that the above conditions are
not necessarily met in a meeting situation. First, it is hard toassume time-invariance of the mixing system for hundreds or
thousands of seconds of observed signals. Our approach here isto apply block processing to signals of several seconds where
time-invariance can be assumed. Second, speaker activity is
non-stationary in a meeting, and the second condition in theabove list is basically not met. If a person speaks continuously
in one block and then falls silent throughout the next block,the separation filters of the BSS system change drastically.
If such situations happen frequently, a speaker in the specific
BSS output may change across blocks. We call this problem a1­4244­0785­0/06/$20.00

Separated signals

Mixtures

Abstract— The technique of blind source separation (BSS) has
been well studied. In this paper, we apply the BSS technique,particularly based on independent component analysis (ICA), toa meeting situation. The goal is to enhance the spoken utterances
and to estimate the location of each speaker by means of multiple
microphones. The technique may help us to take the minutes ofa meeting.

Block-wiseBSS(e.g. 10 s)

Permute outputs(if necessary)

Mixingsystemestimation

Direction-ofarrival (DOA)
estimation
Arraygeometry

Fig. 1.

Global flow of the method

block-wise permutation problem, and propose a solution basedon speaker localization in this paper.
As related work, there are many meeting room projects (e.g.
[13]–[15]). The goal of such projects is to develop techniques
for the transcription, summarization and understanding of ameeting. Of the many research topics in this field, speakerdiarization [16], [17], which identifies when each participant
speaks, is one that is strongly related to the goal of this paper.
However, they generally assumed that at most one speakeris active at a time, and BSS techniques have not yet been
substantially considered in these projects.
This paper is organized as follows. Section II presents an
overview of the proposed method, which basically consists ofblock-wise BSS and speaker localization. An implementationof the block-wise BSS is presented in Sec. III. Since weemploy a small size microphone array in this trial, where
the microphone spacing is around 4 cm, we focus only on
the direction of each speaker in the processing of speakerlocalization. Therefore, a method for estimating the directionof-arrival (DOA) of each speaker is presented in Sec. IV. Also,
the clustering of such estimated DOAs for aligning block-wisepermutations is explained in the same section. Experimental
results are shown in Sec. V, and Sec. VI concludes this paper.
II. G LOBAL FLOW
Figure 1 shows the global flow of the proposed method. We
apply block-wise BSS to several seconds (e.g. 10 seconds)of observed mixtures, to obtain separated signals for each

1407

STFT

ISTFT

ICA

Scaling

Permutation

Fig. 2.

Structure of frequency-domain BSS

block. As discussed in the introduction, there is a possibility
that block-wise permutation occurs. The problem is solved
by estimating the DOA for each speaker in each block. To
accomplish this, we first estimate the mixing system from the
BSS results. And then the DOA of each speech is estimated
together with the array geometry. Finally, we permute theoutputs of block-wise BSS if necessary so that the same
speaker appears at the same BSS output.
III. B LOCK - WISE BSS
We can apply any method/approach [4]–[12] to the blockwise BSS in Fig. 1. We here adopt a frequency-domain
approach similar to [6]–[10] for convolutive BSS. This section
explains the procedure in detail.
Figure 2 shows the system structure. First, sensor observations x1 (t), . . . , xM (t) in the time domain sampled at
frequency f s are converted into frequency-domain time-series
signals x1 (f, t), . . . , xM (f, t) by a short-time Fourier transform (STFT) with frame size L:

which minimizes E{||x − Ay||2 }. The notation · H represents
the conjugate transpose of ·.
The next step is to solve the internal permutation problem, which is caused by the fact that ICA (2) is employedindependently in each frequency bin and ICA solutions have
permutation ambiguities. Many methods have been proposed[6]–[10] for the internal permutation problem. However, we
employ a recently developed approach in the experiments,
which is explained in detail in a paper under review [18].
Whatever method we use, we calculate a permutation Π f :{1, . . . , N } → {1, . . . , N } for each frequency bin f , and thenthe separated signals and the basis vectors are permuted byyi (f, t) ← yΠf (i) (f, t), ai (f ) ← aΠf (i) (f ),∀i, f, t.

(6)

so that the separated components y i originating from the same
source have the same index at all frequency bins.
Next, scaling ambiguities of the ICA solutions are alignedby adjusting y i (f, t) to the observation x J (f, t) of a selectedreference sensor J ∈ {1, . . . , M }:yi (f, t) ← aJk (f )yi (f, t),∀i, f, t.

(7)

We see in (4) that aJi (f )yi (f, t) is a part of xJ (f, t).
Finally, time-domain output signals y i (t) are calculated byapplying an inverse STFT (ISTFT) to the separated signals
yi (f, t).

L/2−1

xj (f, t) ←

xj (t + q) win(q) e−ı2πf q ,(1)

IV. DOA ESTIMATION AND CLUSTERING

q=−L/2for all discrete frequencies ∀ f ∈ F = {0, L1 fs , . . . , L−1
L fs },
and for time t which is now down-sampled with a distanceequal to the frame shift. We typically use a window win(q)that tapers smoothly to zero at each end, such as a Hanningwindow win(q) = 12 (1 + cos 2πq
L ).
Next, ICA is employed in each frequency bin:y(f, t) = W(f ) x(f, t),

∀

f ∈F,

(2)

T

where x = [x1 , . . . , xM ] is an observation vector, y =[y1 , . . . , yN ]T is a separated signal vector, and W is an N ×M
separation matrix. We can apply any instantaneous ICA [1]–[3] for the calculation of W. Then, we calculate a matrix A
whose columns are basis vectors a i ,A = [a1 , · · · , aN ], ai = [a1i , . . . , aMi ]T ,(3)in order to represent the vector x by a linear combination of
the basis vectors:x(f, t) = A(f ) y(f, t) =Nai (f )yi (f, t),∀f ∈ F.

(4)

i=1

If W has the inverse, the matrix is given simply by A = W −1 .
Otherwise it is calculated as a least-mean-square estimator
A = E{xyH }(E{yyH })−1 ,(5)This section presents a method for estimating the DOAs
of sources from the result of block-wise BSS. The flow is
depicted in the lower half of Fig. 1.
A. Mixing system estimation
First, the mixing system is estimated from the BSS results.
If we take the frequency-domain approach, the frequency
responses of the mixing system have already been estimatedas A(f ) by (5). If we take another approach (e.g. time-domain
BSS) for the block-wise BSS and thus obtain just the set oftime-domain mixtures x 1 (t), . . . , xM (t) and separated signals
y1 (t), . . . , yN (t), the mixing system could be estimated as amatrix of filters A(l) by performing a least-mean-square estimation in the time domain. However, this is computationallydemanding if the filters are long. An efficient alternative way
is to convert the time-domain signals into frequency-domain
time-series signals by STFT (1), and apply the same equation(5) to the frequency-domain vectors x(f, t) and y(f, t) toestimate A(f ) for each frequency f .
As shown in (3), the columns of matrix A are called basis
vectors. The i-th basis vector a i = [a1i , . . . , aMi ]T is an
estimation of the way in which the source i was observedat all the sensors. This means that we can estimate the source
directions by analyzing the basis vectors.

1408

source3
4

sensors
Fig. 5.
Fig. 3.

DOA estimation with multiple sensor pairs

Time-difference-of-arrival (TDOA) estimation

sourcesensors
Fig. 4.

DOA estimation with two sensors

B. TDOA estimation
Before estimating the DOA of a source,
we estimate thetime-difference-of-arrival (TDOA) τ ijj of source i between
sensors j and j  (Fig. 3). We employ the well-known GCCPHAT (Generalized Cross Correlation PHAse Transform)
function [19], [20] for TDOA estimations. In the original
formulation, frequency-domain sensor observations x j (f, t)
and xj  (f, t) are used in the GCC function. In our scenario,
however, sensor observations may contain multiple sourcecomponents, which disturb the estimation. On the other hand,the basis vectors ai (f ), f ∈ F, of source i represent information specific only to source i. Thus, we use basis vector
elements aji (f ) and aj  i (f ) corresponding to sensors j andj  instead. The formula

12

τijj = argmaxτ

 aji (f )a∗j  i (f )fe|aji (f )a∗j  i (f )|ı2πf τ

,(8)

where ·∗ denotes the complex conjugate of ·, estimates the
TDOA of source i between sensors j and j  .
C. DOA estimation with array geometryThen, by incorporating the array geometry information, the
DOA of source i can be estimated. Let us define a DOA vector⎡⎤cos θi cos φiqi = ⎣ sin θi cos φi ⎦sin φithat has unit norm ||q i || = 1. The azimuth and elevationof the corresponding source i are represented by θ i and φi ,respectively.
If we start by considering a simple two-sensor case, suchas that shown in Fig. 4, the DOA vector of source i should
satisfy
pTjj qi = τijj · v(9)where pjj is a 3-dimensional vector representing the locationof sensor j relative to that of sensor j  , τijj is the TDOA
estimated by (8), and v represents the signal velocity. The
possible solutions for the DOA vector q i in (9) represent acone, not a direction [10].
If we have more sensor pairs that provide additional equations, the direction can be specified as the intersection of
the cones given by the multiple equations. Figure 5 showsa situation where we have four sensors and take three sensor
pairs into consideration. In such a case, we have the following
simultaneous linear equations for the DOA vector q i :⎡ T ⎤⎡ 12 ⎤p12
τi⎣ pT23 ⎦ qi = ⎣ τi23 ⎦ v .
pT34
τi34
In most cases, such simultaneous linear equations do not havean exact solution for q i because it is hard to obtain precise
array geometry information, and hard to estimate TDOAs
correctly in a real reverberant situation. Hence we practicallycompromise with an approximated solution. An efficient way[10] is to multiply the Moore-Penrose pseudo-inverse of thearray geometry matrix
⎡ T ⎤
p12
D = ⎣ pT23 ⎦pT34
to obtain

⎡

⎤
τi12
qi = D+ ⎣ τi23 ⎦ v ,τi34

and then normalize it to unit-norm
qi
.
qi =
||qi ||
D. DOA clustering and block-wise permutation alignment
So far, we have calculated DOA vectors q i for each blockand for each output of the block-wise BSS. Figure 6 shows
an example of calculated DOAs. These data were obtained forthe situation shown in Fig. 9, where we have four speakers
and four microphones. The DOA estimations are simply represented by the azimuth values θ i in degrees, and the elevations
φi are all zero, since all the microphones were placed ina 2-dimensional space in this case. There are four different
plot symbols, each of which corresponds to each of the BSS
outputs. As seen in Fig. 6, the BSS outputs of the speakers
interchange at some block transition points. This shows anexample of the block-wise permutation problem.

1409

90

90
DOA (degree)

180

DOA (degree)

180

0

−90

−180
0

−90

50

100
150
Time (second)

−180
0

200

Unsorted DOA estimations from the results of block-wise BSS.

Fig. 8.

50

100
150
Time (second)

200

Sorted DOA estimations from the results of block-wise BSS.
Aisle

Fig. 6.

0Partition 305 cm

Source 2

Screen

cm

Desk

90

90

cm

Wall

110 cmClustering DOA estimations into four classes

11

0

cm

In order to align these block-wise permutations, we apply aclustering algorithm to the DOA vectors to identify the DOA
cluster for each speaker. There are many clustering algorithms
that can be used for this purpose. In this paper we employ asimple k-means algorithm [21], assuming that the number of
speakers is known a priori. Figure 7 shows the clustering result
applied to the DOA estimations shown in Fig. 6. In this figure,
DOA vectors are plotted on a unit circle since the microphonearray was 2-dimensional.
Based on the clustering result, we can permute the blockwise BSS outputs so that the same speaker appears at the same
BSS output. Figure 8 shows the sorted DOA estimations. We
observe that the misalignments found in Fig. 6 are reduced.
Sometimes, someone is silent for the whole time in a block,and consequently multiple DOA estimations belong to one
cluster in the block. In such a case, we preserve only the
DOA estimation whose corresponding separated signal is theloudest, and reassign the other DOA estimations to otherclusters.
V. E XPERIMENTS
Figure 9 shows the meeting situation to which we applied
the BSS system. We have four speakers and four microphones
together with some noise sources such as desktop computers(PC) and a projector. Each of the four human speakers was
sitting on a chair. The four microphones were arranged in
the horizontal 2-dimensional plane about 4 cm apart. The
reverberation time of the room was around RT 60 = 350 ms.

Aisle

Source 4

PC1

Source 3

White Board

PC2

270 cmFig. 7.

TowelsWallm

Projector

100c

140 cmSource 1

Partition

Fig. 9.

Meeting situation

We recorded mixed sounds consisting of human speech andPC/projector noise as the input signals for the BSS system. We also measured the impulse response h ji (l) from asource i to a microphone j for a quantitative evaluation of
the separation performance in terms of signal-to-interference
ratio (SIR) improvement. The improvement was calculated by
OutputSIRi − InputSIRi for each output i. These two types of
SIRs are defined by 
hJi (l)si (t − l)|2
t|
l


(dB),
InputSIRi = 10 log10
2
t|
k=i
l hJk (l)sk (t − l)|2t |yii (t)|OutputSIRi = 10 log10  2t|k=i yik (t)|(dB),

where si is the i-th source, J ∈ {1, . . . , M } is the index of a
selected reference sensor, and y ik (t) is theNcomponent of s kthat appears at output y i (t), i.e. yi (t) = k=1 yik (t).
Examples of recorded mixtures and separated sounds areshown in Figs. 10 and 11, respectively. The utterances of the
speakers were well separated, and also the PC/projector noise
was reduced, as a result of BSS. In an informal listening
test, however, we observed that some leaks could still beheard. Estimated DOAs for recorded mixtures have already1410

VI. C ONCLUSION
This paper discussed the application of BSS techniques toa meeting situation. The proposed approach consists of blockwise BSS and DOA clustering. The separation performanceis good in terms of SIR improvement (Table I) and withthe waveforms (Fig. 11). However, we can still hear anotherspeaker’s utterances and their reverberations in the separated
signals. Post-processing is a way to reduce such interference.
Other future work will include an online implementation andtracking, a combination with a speech recognition system, andthe application to underdetermined cases where the speakers
outnumber the microphones.
R EFERENCES
Fig. 10. Four channels of observed mixed signals at the four microphones
in the meeting situation. 30 second excerpts.

Fig. 11. Separated signals generated from the mixed signals shown in Fig. 10
with the BSS system.

been shown in Fig. 8, whose result was obtained through theintermediate results shown in Figs. 6 and 7.
An evaluation in terms of the SIR was conducted byusing the measured impulse responses and speech signals
drawn from a speech database. Table I shows the average
SIR improvements for 8 different speech combinations. We
performed tests using three different lengths (30, 60, and 90
seconds) of total data, while the block length for BSS was
10 seconds in all cases. To observe the effect of block-wisepermutation, we compare the results that we obtained whenblock-wise permutations were aligned and untouched. As thetotal data length increases, the SIR improvement decreases in
the untouched case. This is why the chance for block-wisepermutation increases as the total data length increases.
TABLE ISIR IMPROVEMENTS . AVERAGE OF 8 SPEECH COMBINATIONS .
Block-wise permutation
Aligned
Untouched

30 seconds
14.99 dB
9.87 dB

60 seconds
15.54 dB
7.98 dB

90 seconds
14.29 dB
7.62 dB[1] T.-W. Lee, Independent Component Analysis - Theory and Applications.
Kluwer Academic Publishers, 1998.
[2] A. Hyvärinen, J. Karhunen, and E. Oja, Independent Component Analysis. John Wiley & Sons, 2001.
[3] A. Cichocki and S. Amari, Adaptive Blind Signal and Image Processing.
John Wiley & Sons, 2002.
[4] S. C. Douglas, H. Sawada, and S. Makino, “A spatio-temporal FastICAalgorithm for separating convolutive mixtures,” in Proc. ICASSP 2005,
vol. V, Mar. 2005, pp. 165–168.
[5] W. Kellermann, H. Buchner, and R. Aichner, “Separating convolutive
mixtures with TRINICON,” in Proc. ICASSP 2006, vol. V, May 2006,
pp. 961–964.
[6] P. Smaragdis, “Blind separation of convolved mixtures in the frequency
domain,” Neurocomputing, vol. 22, pp. 21–34, 1998.
[7] L. Parra and C. Spence, “Convolutive blind separation of non-stationary
sources,” IEEE Trans. Speech Audio Processing, vol. 8, no. 3, pp. 320–
327, May 2000.
[8] N. Murata, S. Ikeda, and A. Ziehe, “An approach to blind source separation based on temporal structure of speech signals,” Neurocomputing,vol. 41, no. 1-4, pp. 1–24, Oct. 2001.
[9] H. Sawada, R. Mukai, S. Araki, and S. Makino, “A robust and precise
method for solving the permutation problem of frequency-domain blindsource separation,” IEEE Trans. Speech Audio Processing, vol. 12, no. 5,
pp. 530–538, Sept. 2004.
[10] R. Mukai, H. Sawada, S. Araki, and S. Makino, “Frequency-domain
blind source separation of many speech signals using near-field andfar-field models,” EURASIP Journal on Applied Signal Processing, vol.
2006, pp. Article ID 83 683, 13 pages, 2006.
[11] A. Hiroe, “Solution of permutation problem in frequency domain ICA
using multivariate probability density functions,” in Proc. ICA 2006
(LNCS 3889). Springer, Mar. 2006, pp. 601–608.
[12] T. Kim, H. T. Attias, S.-Y. Lee, and T.-W. Lee, “Blind source separation
exploiting higher-order frequency dependencies,” IEEE Trans. Audio,
Speech and Language Processing, (accepted for future publication).
[13] [Online]. Available: http://nist.gov/speech/tests/rt/rt2002/[14] [Online]. Available: http://www.icsi.berkeley.edu/Speech/mr/[15] [Online]. Available: http://chil.server.de/servlet/is/101/[16] D. Ellis and J. Liu, “Speaker turn segmentation based on betweenchannel differences,” in Proc. ICASSP 2004 NIST Meeting Recognition
Workshop, 2004, pp. 112–117.
[17] D. A. Reynolds and P. Torres-Carrasquillo, “Approaches and applicationsof audio diarization,” in Proc. ICASSP 2005, vol. V, Mar. 2005, pp. 953–
956.
[18] H. Sawada, S. Araki, and S. Makino, “Measuring dependence of binwise separated signals for permutation alignment in frequency-domain
BSS,” in Proc. 2007 IEEE International Symposium on Circuits andSystems (ISCAS 2007), 2007, (submitted).
[19] C. H. Knapp and G. C. Carter, “The generalized correlation method
for estimation of time delay,” IEEE Trans. Acoustic, Speech and Signal
Processing, vol. 24, no. 4, pp. 320–327, Aug. 1976.
[20] M. Omologo and P. Svaizer, “Use of the crosspower-spectrum phase inacoustic event location,” IEEE Trans. Speech Audio Processing, vol. 5,
no. 3, pp. 288–292, May 1997.
[21] R. O. Duda, P. E. Hart, and D. G. Stork, Pattern Classification, 2nd ed.
Wiley Interscience, 2000.

1411