BLIND SPEECH SEPARATION IN A MEETING SITUATIONWITH MAXIMUM SNR BEAMFORMERS
Shoko Araki†‡

Hiroshi Sawada†

Shoji Makino†‡† NTT Communication Science Laboratories, NTT Corporation
2-4 Hikaridai, Seika-cho, Soraku-gun, Kyoto 619-0237, Japan‡ Graduate School of Information Science and Technology, Hokkaido University
Kita 14, Nishi 9, Kita-ku, Sapporo-shi, Hokkaido 060-0814, Japan
Email: shoko@cslab.kecl.ntt.co.jp
ABSTRACT
We propose a speech separation method for a meeting situation,
where each speaker sometimes speaks and the number of speakers
changes every moment. Many source separation methods have already been proposed, however, they consider a case where all the
speakers keep speaking: this is not always true in a real meeting.
In such cases, in addition to separation, speech detection and theclassiﬁcation of the detected speech according to speaker become
important issues. For that purpose, we propose a method that employs a maximum signal-to-noise (MaxSNR) beamformer combinedwith a voice activity detector and online clustering. We also discuss
the scaling ambiguity problem as regards the MaxSNR beamformer,
and provide their solutions. We report some encouraging results fora real meeting in a room with a reverberation time of about 350 ms.
Index Terms— Speech separation, maximum SNR beamformer,
scaling ambiguity, voice activity detector, online clustering.
1. INTRODUCTION
This paper considers speech separation in a meeting situation, where
sources are sometimes active but silent for most of the observation period. Recently, many blind source separation methods havebeen proposed (e.g.,[1–3]). However, most of them assume that all
sources keep speaking during the observation. However, this is notalways the case in a real situation such as a conversation or a meeting. In such a scenario, the meeting recognition has recently beenstudied and it has been pointed out that the voice activity detection(VAD) for each speaker is one of the important topics (e.g., [4]). Beyond the VAD, this paper tries to recover each speech stream with alinear ﬁltering: beamforming.
Let us formulate the task. Suppose that N ≥ 2 speech sources
s1 , . . . , sN are convolutively mixed and observed at M sensors,
PP
xj (t) = N
l hjk (l) sk (t − l) + nj (t), j = 1, . . . , M, (1)k=1where hjk (l) represents the impulse response from source k to sensor j, and nj (t) is the observed stationary background noise at sensor j. The speech sk (t) are intermittent signals. Our goal is to obtain
estimates yk of each source signal sk separately from the sensor observations x without information about the number of sources N , thespeech sources sk or the mixing process hjk .
Because we have N ≥ 2 speeches, the other speakers act as thenon-stationary noises for a target signal. Therefore, it is difﬁcult toapply a single channel noise reduction approach (e.g.,[5]). Such an
approach cannot enhance each source separately, either. When N isgiven, we can separate signals with, for instance, independent component analysis (e.g.,[2]) or a binary mask approach with k-means

1­4244­0728­1/07/$20.00 ©2007 IEEE

clustering (e.g.,[6]). If hjk or the steering vector of the target signal
is available, widely used beamformers such as the minimum variance beamformer (MVBF) [7] can enhance the target signal.
However, in this paper, we consider a scenario where none ofthese previous knowledge is available. In such a case, speech detection and the classiﬁcation as well as separation become important issues. To realize such a system, we propose a signal extraction
method with a maximum signal-to-noise (MaxSNR) beamformer [7,8] combined with a VAD and online clustering. The MaxSNR beamformer maximizes the ratio between the signal power and interferenceand-noise power, which are estimated in “target active” and “target
silent (interferences-and-noise)” periods, respectively. These periods are estimated by the VAD and online clustering. With the online clustering, we can classify speech signals even if the number ofspeeches changes every moment.
The MaxSNR beamformer is attractive because it does not need
information on the target location, such as the steering vector or hjk ,required in widely used beamformers e.g., the MVBF [7]. However,the MaxSNR beamformer has the scaling ambiguity problem: since
the MaxSNR beamformer does not have any constraint for its gain
to the target direction, the beamformer gains at different frequencies
may differ. This characteristic is problematic when we apply the
MaxSNR beamformer to wide-band signals such as speech mixtures[8]. The authors of [8] proposed methods for solving the scaling
problem, however, their proposals still need (rough) target location
information. It weakens the merit of the MaxSNR beamformer. We
propose a method that uses a simple linear estimation for the scaling
ambiguity problem.
We report some encouraging results of experiments conductedduring a meeting in a room with reverberation time of about 350 ms.
2. PROPOSED METHODThis paper employs a time-frequency domain approach. With a T point short-time Fourier transform (STFT), (1) is converted into:P
xj (f, τ ) = N(2)
k=1 hjk (f )sk (f, τ ) + nj (f, τ ),or in vector notation,
x(f, τ ) =PN

k=1 hk (f )sk (f, τ ) + n(f, τ )(3)where hjk (f ) is the frequency response from source k to sensor j,sk (f, τ ) and nj (f, τ ) are the STFTs of a source sk and noise nj
respectively. f ∈ {0, T1 fs , . . . , T T−1 fs } is a frequency (fs is the
sampling frequency) and τ (= 1, · · · , K) is a time-frame index. The
vectors are x = [x1 , . . . , xM ]T , hk = [h1k , . . . , hM k ]T and n =[n1 , . . . , nM ]T .

I ­ 41ICASSP 2007

That is, we calculate a(f ), which minimizes the following cost function:(8)
G(a(f )) = E{||x(f, τ ) − a(f )yk (f, τ )||2 }.
This is a linear least-mean-squares estimation problem [10]. Therefore, an optimal a(f ) can be obtained by setting the differentiation∂G(a(f ))at zero:∂a(f )

Max. SNR Beamformer
VAD
Clustering

Featureextraction

Fig. 1. Flow for proposed method.

a(f ) =

Figure 1 shows the system ﬂow of our proposed method. As the
body of our system, we employ a maximum signal-to-noise (MaxSNR)
beamformer [7]. It maximizes the ratio between the output powers
for the target-active and the target-silent periods. When such a beamformer wk (f ) is obtained for source k, the k-th output signal can beobtained by(4)yk (f, τ ) = wkH (f )x(f, τ ) .
Let P = {1, . . . , K} be the whole period of K observations
x(f, 1), . . . , x(f, K) at each frequency, PTk ⊂ P be the targetactive period when the target source sk is active, and PIk ⊂ P be
the target-silent period when the target sk is NOT active but interferences and noise may active. In this paper we assume PTk ∪ PIk = P.
In this section, ﬁrst we describe the MaxSNR beamformer, andthen how to estimate PTk and PIk for the MaxSNR beamformer.
2.1. Maximum SNR beamformer
The design criterion for the beamformer wk (f ) is to maximize theratio λ(f ) of the output power between the target-only period PTk
and the interference-and-noise-only period PIk :λ(f ) =

E{|yk (f, τ )|2 }P k
T

E{|yk (f, τ )|2 }P kI=

wkH (f )RkT (f )wk (f )wkH (f )RkI (f )wk (f )(5)where RkT (f ) and RkI (f ) are the correlation matrices of observations
1 X
x(f, τ )xH (f, τ ),RkT = E{x(f, τ )xH (f, τ )}P k =T|PTk |
k
τ ∈PT

RkI

1 X
= E{x(f, τ )x (f, τ )}P k =x(f, τ )xH (f, τ )I|PIk |kHτ ∈PI

where |P| denotes the number of elements of the set |P|.
By differentiating λ(f ) with wk (f ) and setting it at 0, we have
RkT (f )wk (f ) = λ(f )RkI (f )wk (f ).

(6)

Obtaining the maximum λ(f ) corresponds to calculating the largest
eigenvalue of the generalized eigenvalue problem (6), and the corresponding eigenvector e(f ) gives the solution for the MaxSNR beamformer(7)
wk (f ) = e(f ).
(6) is simpliﬁed to an eigenvalue problem by multiplying both sides
by [RkI (f )]−1 .
2.2. Scaling determination
The MaxSNR beamformer does not have any constraint for its gain,
and so the beamformer gain provided by (7) has a scaling ambiguity. This characteristic should be compensated for if the MaxSNR
beamformer is applied to wide-band signals such as speech.
Inspired by the deﬂation based blind source separation algorithm[9], we propose to compensate wk (f ) so that the output yk (f, τ )becomes as close as observations:x(f, τ ) ≈ a(f )yk (f, τ ) = a(f )wkH (f )x(f, τ ).

Rx (f )wk (f )E{yk∗ (f, τ )x(f, τ )}= H,
E{|yk (f, τ )|2 }wk (f )Rx (f )wk (f )(9)where Rx (f ) = E{x(f, τ )xH (f, τ )} is the observation correlation
matrix. The scale compensated beamformer is given by a selecting
J-th component aJ ,wk (f ) ← aJ wk (f ).

(10)

2.3. Target and noise period estimation
For the MaxSNR beamformer, the PTk and PIk estimations are very
important. These estimations correspond to the target and interference classiﬁcation. However, the well-known k-means clustering is
difﬁcult to use in a meeting situation where the number of speakers
may change. Moreover, the noise-only periods when no speaker isactive degrade the speaker classiﬁcation performance.
In order to handle such issues, this paper proposes a method that
employs a voice activity detector (VAD) and online clustering of thetime-difference of arrival (TDOA) information. First, time frames τ ,where no speaker is active (and noise may exist), are removed with aVAD. Then the remaining time frames τ , where speakers are active,
are classiﬁed into the target period and the interference period with
online clustering.
Step1: Voice activity detection (VAD)
First, we detect the noise-only period PN ⊂ PIk by using the
VAD proposed by Sohn et al [11]. The noise-only period PN andthe speech-exist period PS are determined by the ML based decisionrule:if
Λ(τ ) > η then τ ∈ PS
(11)
else
τ ∈ PNwhere(T −1)fs /TX{γ(f, τ ) − log γ(f, τ ) − 1},
Λ(τ ) =
f =0

γ(f, τ ) = ||x(f, τ )||2 /σ(f, τ ) is a posteriori SNR, σ(f, τ ) is an
estimated noise variance, and η is a threshold [11].
Step2: Feature extraction
As the speech-exist period PS = P − PN includes the target
source and other interferences, it should be classiﬁed into target period PTk and interference period. These periods are determined insteps 2 and 3.
In order to classify the target (k-th signal) and interferences, we
utilize the time differences of arrival (TDOA) at sensors j and j  .

TDOA qjj
 (τ ) can be estimated by using the generalized cross correlation method with the phase transform (GCC-PHAT) [12]X xj (f, τ )x∗j  (f, τ ) j2πf q (τ ) = argmaxq 
e
.
(12)
qjj
|xj (f, τ )x∗j  (f, τ )|f

We can use a TDOA (column) vector q (τ ), which consists of theqjj (τ ) of all the sensor pairs for the clustering. Instead, we utilized
the direction of arrival (DOA) vector q(τ ) [13]. When the source azimuth is θ(τ ) and the elevation is φ(τ ), the DOA vector can be written as q(τ ) = [cos θ(τ ) cos φ(τ ), sin θ(τ ) cos φ(τ ), sin φ(τ )]T .
The DOA vector q(τ ) is calculated by the TDOA information q (τ )I ­ 42

AislePartition

Source 3 (60o)

Microphones

Screen140 cm

90
cm

4 cm
Desk

Projector

90

cm

11
0

The signal-to-interference plus noise-ratio (SINR) was calculated byP|yii (t)|2
SINRi = 10 log10 P P t
[dB],
(16)
2
t|
k=i yik (t) + yn (t)|PC1

Source 1 (- 40o)White Board

PC2

270 cmSource 2 (- 120o)cm

3.2. Evaluation measures

Wall

Wall

110 cmAisle

from the projector, the personal computers PC1 and PC2 (see Fig. 2).
We also tried a recorded meeting in the room shown in Fig. 2. During the recording, source 1 was standing, and sources 2 and 3 weresitting. As it was a real meeting, the sources moved. The frame size
T for STFT was 2048 (128 ms), and the frame shift was 256 (16 ms).

305 cmwhere y (t) is the sk component that appears at outputˆ yi (t): yi (t) =˜PN ikH
k=1 yik (t), and yn (t) the noise output: yn = IFFT wi (f )n(f, t) .
The sound quality was evaluated by the signal to distortion ratio(SDR):
P2
t |xJi (t)|
[dB],
(17)
SDRi = 10 log10 P
2
t |xJi (t) − βyii (t − Δ)|P
where xJk (t) = l hJk (l) sk (t − l), β and Δ are parameters used
to compensate for the amplitude and phase difference between xJi
and yii . In the simulation, we tried six speaker combinations andaveraged all the results.

Partition

Fig. 2. Room setup. The reverberation time was around 350 ms..
and the given sensor coordinate information D [13]:q(τ ) = cD− q (τ ).

(13)−

where c is the propagation velocity of the signals and denotes theMoore-Penrose pseudo-inverse.
Step3: Clustering
To divide PS into the target period PTk and interference period,
we then perform clustering for the extracted features q(τ ) of all
time-frame τ ∈ PS . As we do not know the number of sources N
to be separated, we employ the online clustering algorithm (leaderfollower clustering) [14].
Let us select the clusters which have enough elements, and nameone of them Ck . Now we set the target period PTk asτ ∈ PTkifq(τ ) ∈ Ck .

3.3. Results(14)

Finally the interference-and-noise-only period PIk is determined as
PIk = P − PTk .

(15)

Method summary
1. Estimate noise-only period PN and speech-exist period PS
with VAD.
2. Calculate the feature q(τ ) for each frame τ .
3. For each k (k = 1, · · · , N ),1) Determine the target period PTk from PS with (14).
2) Decide the interference-and-noise-only period PIk with (15).
3) Calculate a MaxSNR beamformer wk (7) with PIk and PTk .
4) Compensate the beamformer gain with (10).
5) Calculate the output with (4).
There may be many options with respect to the decision for PTk
and PIk . For example, in this paper, we deﬁned PTk and PIk for every
time frame τ . We can also determine these periods for every timefrequency (f, τ ) by using the VAD (11) and DOA vector (13) at eachtime-frequency point. Such an option may work well when speech
signals often overlap.
3. EXPERIMENTS
3.1. Setup
Experiments were performed in the room shown in Fig. 2 whose reverberation time was around 350 ms. DOAs θ for sources 1, 2 and 3
were −40◦ , −120◦ and 60◦ , respectively. Simulated meeting observations were made by following (1) with the impulse responses
hjk (l) and the noise nj (t) measured in the room, and English speechsources sk (t) sampled at 16 kHz. The active time for each sourcein the simulation is shown in Fig. 3. We utilized the recorded noise

In the simulation, there were overlaps between speakers as shown in
Fig. 3, and Fig. 4. Figure 5 illustrates the estimated DOA θ(τ ) for
the speech-exist period PS (black dots) and noise-only period PN(gray dots). The estimation succeeded for most of the periods, however, the VAD sometimes failed the noise-only period PN estimation. Moreover, since we employed GCC, we can estimate only one
DOA at each frame τ even when speech signals overlapped. However, we can separate each signal successfully as shown in Fig. 5 andTable 1. We can say that, for designing the MaxSNR beamformer,
the target period PTk and the interference-and-noise-only period PIk
estimation does not have to be so accurate.
Figure 7 shows the effect of the scaling method (10). (a) is thesignal spectrum for a source s1 of one second and (b) is that for aseparated signal y1 without scaling. Without scaling, the spectrum(b) is severely distorted. This is why the SDR was poor without
scaling (see SDR’ in Table 1). With the scaling method, we canrestore the spectrum to that of speech (Fig. 7(c)).
We also tried to separate the signals for a recorded meeting. The
top of Fig. 9 shows one of the observations x1 . There was heavy
noise and speeches sometimes overlapped. Figure 8 illustrates theestimated DOA θ(τ ) for the speech-exist period PS (black dots) andnoise-only period PN (gray dots). Because this is a real recording,the source 3 (θ ≈ −40◦ ) was moving. As we cannot calculate SINR
and SDR for a real recording, we checked the output waveforms (inFig. 9). We can see that each speech was extracted successfully.
4. CONCLUSION
We proposed a method for extracting speech from a meeting recording with the MaxSNR beamformer. We also proposed a method forthe target active/silence period estimation with a VAD and TDOA
clustering, and the scaling compensation method for the MaxSNR
beamformer. We conﬁrmed that our proposed method works well
for a meeting situation. In this paper, the method was applied to
the whole 30-second long observations, that is, it worked in a batchmode. However, the results with our method, which includes the online clustering, encourage us to implement this approach in real-time
conﬁguration.

I ­ 43

0
í100
0

5

10

15
20
time (sec.)

25

30

10

15

20

25

30

DOA [deg.]

3000

4000

10
1000

2000

3000

4000

(c)

5

10

15

20

25

30

5

10

5

10

15

20

15
20
time (sec.)

25

25

30

10

15
20
time (sec.)

25

20

25

30

5

10

15

20

25

30

5

10

15
20
time (sec.)

25

30

Fig. 6. Separated signals in the simulated meeting.
Table 1. Simulation results. SDR’: without scaling solver (10).
SINR[dB]
9.1

SDR[dB]
12.9

í100

2
X1 0
í2
0
2
y 0
1
í2
0
2
y2 0
í2
0
2
y3 0
í2
0

1

15

SDR’[dB]
3.8

5. REFERENCES[1] S. Haykin, Ed., Unsupervised Adaptive Filtering (Volume I: Blind
Source Separation), John Wiley & Sons, 2000.
[2] H. Sawada, R. Mukai, S. Araki, and S. Makino, “Frequencydomain blind source separation,” in Speech Enhancement, J. Benesty,
S. Makino, and J. Chen, Eds., pp. 299–327. Springer, Mar. 2005.
[3] O. Yilmaz and S. Rickard, “Blind separation of speech mixtures via
time-frequency masking,” IEEE Trans. Signal Processing, vol. 52, no.
7, pp. 1830–1847, July 2004.
[4] D. Ellis and J. Liu, “Speaker turn segmentation based on betweenchannel differences,” in Proc. NIST Meeting Recognition Workshop,
2004, pp. 112–117.

4000

5

10

15
20
time (sec.)

25

30

Fig. 8. Estimated DOA for speech-exist period (black) and noise-only
period (gray) for the recorded meeting.

Fig. 5. Estimated DOA for speech-exist period (black) and noise-only
period (gray) for the simulated meeting.

10

3000

0

0

30

5

2000
Frequency [Hz]

100

30

í100
5

1000

Fig. 7. Signal spectra for (a) source s1 , (b) separated signal y1 without scaling, (c) separated signal y1 with scaling.
DOA [deg.]

5

0

2
0
í2
0
2
y2 0
í2
0
2
y 0
3
í2
0

2000

(b)

0
0

100

y

1000

10

Fig. 4. Original signals and a mixture x1 in the simulated meeting.

0

0
0
20

0
0
20

Fig. 3. True speech active periods for the simulated meeting.
2
S1 0
í2
0
2
S2 0
í2
0
2
0
S3
í2
0
2
X1 0
í2
0

(a)

10

Spectrum

DOA [deg.]

20

100

5

10

15

20

25

30

5

10

15

20

25

30

5

10

15

20

25

30

5

10

15
20
time (sec.)

25

30

Fig. 9. A recorded and the separated signals for the recorded meeting.
[5] Y. Ephraim and D. Malah, “Spech enhancement using a minimum
mean-square error short-time spectral amplitude estimator,” IEEE
Trans. Acoust. Speech, Signal Processing, vol. 32, pp. 1109–1121,
1984.
[6] S. Araki, H. Sawada, R. Mukai, and S. Makino, “A novel blindsource separation method with observation vector clustering,” inProc. IWAENC 2005, Sept. 2005, pp. 117–120.
[7] H. L. Van Trees, Ed., Optimum Array Processing, John Wiley & Sons,
2002.
[8] E. Warsitz and R. Haeb-Unbach, “Controlling speech distortion in
adaptive frequency-domain principal eigenvector beamforming,” inProc. IWAENC 2006, 2006.
[9] A. Cichocki, R. Thawonmas, and S. Amari, “Sequential blind signal
extraction in order speciﬁed by stochastic properties,” Electronics Letters, vol. 33, no. 1, pp. 64–65, 1997.
[10] T. Kailath, A. H. Sayed, and B. Hassibi, Linear Estimation, Prentice
Hall, 2000.
[11] J. Sohn, N. S. Kim, and W. Sung, “A statistical model-based voiceactivity detection,” IEEE Signal Processing Letters, vol. 6, no. 1, pp.
1–3, 1999.
[12] C. H. Knapp and G. C. Carter, “The generalized correlation method
for estimation of time delay,” IEEE Trans. Acoust. Speech and Signal
Processing, vol. 24, no. 4, pp. 320–327, 1976.
[13] S. Araki, H. Sawada, R. Mukai, and S. Makino, “DOA estimation for
multiple sparse sources with normalized observation vector clustering,”
in Proc. ICASSP2006, May 2006, vol. 5, pp. 33–36.
[14] R. O. Duda, P. E. Hart, and D. G. Stork, Pattern Classiﬁcation, Wiley
Interscience, 2nd edition, 2000.

I ­ 44