1IWAENC 2006 – PARIS – SEPTEMBER 12-14, 2006

BLIND SPEECH SEPARATION BY COMBINING
BEAMFORMERS AND A TIME FREQUENCY BINARY MASK
1,2Jan Cermak, 1Shoko Araki, 1Hiroshi Sawada and 1Shoji Makino
2cermak4@kn.vutbr.cz
NTT Communication Science Laboratories, NTT Corporation
2-4 Hikaridai, Seika-cho, Soraku-gun, Kyoto 619-0237, Japan
2
Brno University of Technology, Faculty of Electrical Engineering and Communication, Department of Telecommunication, Purkynova 118, 612 00 Brno, Czech Republic
1ABSTRACTThis paper describes a new method for blind speech separation(BSS) of convolutive mixtures. Our approach is based on awidely used speech enhancement method called beamforming.
We utilize this technique for BSS by combining a beamformer
and a time-frequency binary mask (TFBM) in one system. We
propose two different approaches using the same basis butwith a different setup. The first approach is designed for(over-)determined configurations, i.e. the number of sensors isequal to or greater than the number of sources. The second
approach is designed for underdetermined configurations, i.e.
the sources outnumber the sensors. Experimental results showthat the proposed approach provides better results than thesole use of a conventional TFBM or a conventional beamformer.

1. INTRODUCTION
BSS aims to separate speech signals from their mixtures
without any a priori information about the source position, room acoustics, mixing processes etc. Speech separation methods can be distinguished into two groups
according to applicability for only an (over-)determinedcase, or for both (over-)determined and underdeterminedcases.
A well known representative of (over-)determined methods is independent component analysis ICA [1,2]. ICA isa statistical BSS method relying only on statistical independence of the source signals. Another possible (over)determined approach is a multiple-beamformer. Beamformer [3] performs spatial filtering by forming a directivity pattern of an array with M sensors in order to emphasize a target signal sk(t) arriving from a given direction and to suppress signals arriving from other directions (jammers). In order to separate N sources, a set ofN different beamformers must be employed. However,this method may also be used for an underdeterminedcase but the jammer suppression is not efficient because

only M-1 minimums (null patterns) can be designed in a
directivity pattern.
The critical problem of set of multiple-beamformers orof beamformer itself is that the speech separation is notblind, because beamformer needs a priori information
about the target signal sk(t), e.g. a mixing vector or atleast its approximation (steering vector).
Time-frequency binary mask (TFBM) [4,5] is a BSS
approach that can be applied even to an underdeterminedcase because the TFBM method relies on time-frequency
sparseness. However it has a musical noise problem due
to zero padding in the time-frequency domain.
In this paper, we present a BSS method that overcame
the above mentioned limitations. Our method removes
and significantly reduces musical noise for (over)determined and underdetermined cases, respectively.
This is achieved by combining multiple-beamformers
and TFBM. We use TFBM for mixing vector estimationin order to make multiple-beamformers blind. Furthermore, TFBM is exploited for reducing the number ofjammers so that only M-1 jammers are included in thesignal mixtures at the beamformer input even in the underdetermined case. Beamformer then separates the target signal, thus the musical noise is reduced.
2. MIXING MODEL AND CONVENTIONALAPPROACHES
2.1. Mixing Model
We consider M sensors observing N sources as convolutive mixtures
Nx j (t ) = ∑∑ h jk (l ) sk (t − l ), j = 1,", M,(1)

k =1 lwhere xj(t) is the signal observed by the j-th microphone,t is the discrete time index, sk(t) is the k-th source signaland hjk(t) represents the impulse response from the k-th2IWAENC 2006 – PARIS – SEPTEMBER 12-14, 2006

source to the j-th sensor. The mixing model (1) in thetime-frequency domain becomesNx ( f , τ ) ≈ ∑ h k ( f ) sk ( f , τ )

(2)

k =1

where x(f,τ)=[ x1(f,τ),…, xM(f,τ)]T is an observation vector and hk(f)=[ h1k(f),…, hMk(f)]T is the vector of the impulse responses (mixing vector). If the source signals
are sparse, which holds for speech signals [5], thesources rarely overlap in the time-frequency domain and(2) can be approximated asx( f ,τ ) ≈ h k ( f ) sk ( f ,τ ),(3)where sk(f,τ) is the dominant source at the timefrequency point (f,τ).
2.2. Conventional beamformer
A beamformer is a set of filters wjk(f) that perform spatial filtering. The enhanced output signal in the timefrequency domain is obtained asyk ( f ,τ ) = w k ( f )T x( f ,τ ),(4)where wk(f)=[w1k(f),…,wMk(f)]T. We describe the minimum variance (Frost) beamformer [3]. The filters wjk(f)are designed by using

mixtures xj(t) are transformed into the time-frequency
domain by the short-time Fourier transform (STFT).
Observation vector x(f,τ) is then normalized so that it
forms clusters corresponding to individual sources sk(t).
Normalization is undertaken in block NORMAL by
selecting reference sensor J and counting⎡ arg ( x j ( f ,τ ) / xJ ( f ,τ ) ) ⎤x j ( f ,τ ) = x j ( f ,τ ) exp ⎢ − j⎥,(7)
2πfc −1 d max⎣⎢⎦⎥x( f ,τ )x ( f ,τ ) = ,(8)x( f ,τ )where dmax is the maximal distance between the reference sensor J and sensor j ∈ {1,..., M} andx( f ,τ ) = [ x1 ( f ,τ ),..., xM ( f ,τ )]T .
⎡ x1 (t ) ⎤⎢⎥⎢ # ⎥⎣⎢ xM (t ) ⎦⎥Conventional TFBM

STFTy ( f ,τ )x( f ,τ )2.3. Conventional TFBM
A conventional TFBM is shown in Fig. 1. The principleof the TFBM is introduced in [4, 5] and therefore wedescribe the TFBM briefly here. First, the time domainy (t )NORMALx ( f ,τ )CLUSTERINGckM k ( f ,τ )CkMASKBACKNORMAL*⎛⎞
R k ( f ) −1 a k ( f )wk ( f ) = ⎜(5)
⎟ ,H−1⎝ ak ( f ) R k ( f ) ak ( f ) ⎠where H is the complex conjugate transpose, * is theconjugation, Rk(f)=E[x(f,τ)x(f,τ)H] is the correlation
matrix of observation vector x(f,τ), and E[] is the mean
operator. ak(f) is a steering vector representing an approximation of hk(f) for an anechoic environmenta k ( f ) = ⎡⎣ e − j2πf τ1k ,..., e − j2πf τ Mk ⎤⎦ ,(6)where τjk is the time delay between the arrival of sk(t) atsensor j and at reference sensor J. τjk is dependent on thetarget location, the sensor array geometry and thepropagation velocity c. However, it is difficult to determine a precise steering vector due to the misalignmentof the sensor array or reverberation in a real situation.
Instead of ak(f), we can employ hk(f), which contains thecorrect array geometry and reverberation information.
Moreover, Rk(f) can be substituted by the correlation
matrix of the observation vector in the jammer onlyperiod R k ( f ) . These changes both result in improved
performance. However, the hk(f) measurement is notrealistic in practice and furthermore the estimation of
the jammer only period constitutes a difficult problem
especially when the jammers are non-stationary signals.

ISTFThˆ k ( f )aˆ k ( f )Figure 1. Time-frequency binary mask (TFBM).
Normalized vector x ( f ,τ ) = [ x1 ( f ,τ ),..., xM ( f ,τ )]T isthen clustered in order to find N clusters C1,…,CN corresponding to individual source signals. Note that thenumber of sources N must be known beforehand. Clustering is achieved by minimizing the objective function
Nℑ = ∑∑ x ( f ,τ )∈C d k ( f ,τ ),kk =1(9)where dk(f,τ) is the distance from the normalized vectorx ( f ,τ ) to the cluster centers ck=[ c1k,…, cMk]Td k ( f ,τ ) = 1 − real( x ( f ,τ )T c k * ).
(10)
ℑ is minimized by the following updates⎧⎫
Ck = ⎨ x ( f ,τ ) k = argmin (di (f ,τ )) ⎬ ,(11)i⎩⎭ck = E [ x( f ,τ ) ]x∈C , c k = ck / ck ,(12)kwhere E[]x∈Ck is the mean operator for the members of acluster Ck. The vector of separated target signalsy ( f ,τ ) = [ y1 ( f ,τ ),..., y N ( f ,τ )]T is obtained byy k ( f ,τ ) = M k ( f ,τ ) xP ( f ,τ ),(13)where P ∈ {1,..., M} is an arbitrary sensor index andMk(f,τ) is a time-frequency binary mask extracting thetime-frequency points of cluster Ck.
x ( f ,τ ) ∈ C k⎧1M k ( f ,τ ) = ⎨.
(14)otherwise⎩0
Finally, the vector of the separated target signals

3IWAENC 2006 – PARIS – SEPTEMBER 12-14, 2006y ( f ,τ ) is transformed back into the time domain by
inverse STFT (ISTFT).

3. PROPOSED APPROACH FOR(OVER-) DETERMINED CASE
Figure 2 shows the flow of our method. The time domain mixtures xj(t) are first transformed into the timefrequency domain by STFT. Observation vector x(f,τ) isthe beamformer and TFBM input. The function of theTFBM, the jammer selection (JS) block, and the correlation (CORR) block is to estimate the jammer correlation
ˆ ( f ) and hˆ ( f ) or aˆ ( f ) , where ^ stands formatrix Rkkk
estimated value. Finally, the vector of the separated
target signals y(f,τ)=[ y1(f,τ),…, yN(f,τ)]T is transformed
back into the time domain by ISTFT.
⎡ x1 (t ) ⎤⎢⎥⎢ # ⎥⎢⎣ xM (t ) ⎥⎦x( f ,τ )STFTy ( f ,τ )BEAMFORMERhˆ k ( f )TFBMyˆ ( f ,τ )ISTFTy (t )

ˆ (f)RkCORRnˆ k ( f ,τ )JSFigure 2. Proposed method for (over-)determined case.
ˆ ( f ) estimation:
Jammer correlation matrix Rk
In contrast to a conventional TFBM (13), we computethe separated signal matrix yˆ ( f ,τ ) = [yˆ 1 ( f ,τ ),...,yˆ N ( f ,τ )]whereyˆ k ( f ,τ ) = M k ( f ,τ )x( f ,τ )T =T[ yˆ1k ( f ,τ ),..., yˆ Mk ( f ,τ )] . The separated signal matrixyˆ ( f ,τ ) is needed in order to satisfy the requirement of
M channel signals for beamformer design. The jammersignals nˆ k ( f ,τ ) = [nˆ1k ( f ,τ ),..., nˆMk ( f ,τ )]T are estimatedin the JS block asnˆ jk ( f ,τ ) =N

∑ yˆ ( f ,τ ).

b=1,b ≠ kjb(15)ˆ ( f ) = E(nˆ ( f ,τ )nˆ ( f ,τ ) H ) is counted in theFinally, Rkkk
CORR block.

Mixing vector hˆ k ( f ) estimation:
In our approach, we extend the TFBM to estimate themixing vector hˆ k ( f ) or steering vector aˆ k ( f ) . This ispossible because the cluster centroids ck, obtained
through the normalization and clustering process, represent an estimation of mixing vector hk(f) [6]. This canbe derived from (3), (7), (8) and (12) as
ck = E[ x ( f ,τ )]x∈Ck = E[h k ( f )] f ,(16)
where h k ( f ) is a normalized mixing vector. As regards(16), the mixing vector can be obtained through backnormalization by using (8) and (7)ckexp ⎡⎣ − j2πfd max c −1 arg( ck cJ ) ⎤⎦ .
hˆ k ( f ) =(17)M
Steering vector aˆ k ( f ) (6) can also be derived by substituting the time delay τˆ k = d max c −1 arg( ck cJ ), obtainedfrom (17), where τˆ k = [τˆ1k ,...,τˆMk ] .

4. PROPOSED APPROACH FOR
UNDERDETERMINED CASE
The method proposed in the previous section may also
be used for an underdetermined case but it results in low
signal-to-interference ratio (SIR), see (19), as describedin section 1. In order to achieve a high SIR, our underdetermined approach implements a jammer reduction,
which reduces the number of jammers N in the signal
mixtures at the beamformer input, so that N=M-1 . The
flow of our proposed method is shown in Fig. 3.
Jammer reduction:In Fig. 3, the JS block counts the reduced jammer signal
mixture vector nˆ ′k ( f ,τ ) = [nˆ1′k ( f ,τ ),..., nˆM′ k ( f ,τ )] andthe estimated (pre-separated) target signal yˆ k ( f ,τ ) thatshould be enhanced by the beamformer. The reducedjammer signal mixture nˆ ′jk ( f ,τ ) is a summation ofN=M-1 jammer signals estimated by the TFBM. The
jammer reduction is achieved bynˆ ′jk ( f ,τ ) =N∑

b=1,b ≠ k , b∉z jyˆ jb ( f ,τ ),(18)where z j ∈ {1...N}N − M is a set of N-M jammer indexes.
Jammer indexes zj represent jammer signals that should
not be included in nˆ ′jk ( f ,τ ) . Jammer indexes are determined by the selection criterion. Different selection criteria can be used. In our approach we select the jammer
signals with the fewest cluster members in order tominimize musical noise.
The beamformer input xˆ k ( f ,τ ) = nˆ ′k ( f ,τ ) + yˆ k ( f ,τ ) is
the summation of the reduced jammer mixture nˆ ′k ( f ,τ )and the estimated target signal yˆ k ( f ,τ ) .
Beamformer design is then based on the reduced jamˆ ′ ( f ) = E[nˆ ′ ( f ,τ )nˆ ′ ( f ,τ ) H ]mer correlation matrix Rkkk
and on the estimated mixing vector hˆ k ( f ) or estimatedsteering vector aˆ k ( f ) .
5. EXPERIMENTS
We performed experiments for both determined (M=3,N=3) and underdetermined (M=3, N=4) cases in a roomwith a reverberation time of 120 ms, see Fig. 4. The
source signals were 5-second English and Japanesespeeches. The STFT frame size was L=512, frame shift
L/4 and the sampling frequency fs=8 kHz. Separation
performance was evaluated by the SIR and signal-todistortion ratio (SDR)

4IWAENC 2006 – PARIS – SEPTEMBER 12-14, 2006nˆ ′k ( f ,τ )⎡ x1 (t ) ⎤⎢⎥⎢ # ⎥⎢⎣ xM (t ) ⎥⎦STFTx( f ,τ )yˆ ( f ,τ )TFBM

JSCORRyˆ k ( f ,τ )+xˆ k ( f ,τ )ˆ′ (f )
Rk

BEAMFORMERy ( f ,τ )ISTFTy (t )hˆ k ( f )Figure 3. Proposed method for underdetermined case.
SIR k = 10 log10

SDR k = 10 log10

E ⎡⎣ yk (t ) 2 ⎤⎦⎡ N
⎤E ⎢ ∑ yi (t ) 2 ⎥⎣ i =1,i ≠ k⎦2
E ⎡⎣ xPk (t ) ⎤⎦

[dB],

(19)

[dB],
(20)
2E ⎡( xPk (t ) − αyk (t − D) ) ⎤⎣⎦where yi(t) are the jammer components that appear in theoutput target signal, yk(t) is the output signal without any
contribution from the jammers and xPk (t ) =∑ l hPk (l )sk (t − l ) . Coefficients α and D compensate theamplitude and delay, respectively.
880 cmMicrophone (120cm height, omni-directional)

30°

Mic. 380 cm

d=4 cm200 cm

S350 cm 225°

S2

Mic. 1d0°
S1300°

282cm

6. CONCLUSION

ddS4135°

Table 1. Experimental results for 3 microphone setting.
DesignAverage SIR [dB] Average SDR [dB]DUDDUDMethodCB
5.1
2.3
7.3
7.2
TFBM
10.9
9.6
10.6
9.0
Proposal
14.6
9.3
13.6
10.8

Microphone setup
Mic. 2375 cm••

Loudspeaker (120cm height)
Reverberation time: 120 ms
Room height: 240cm

the CB but the zero padding in the time-frequency domain means that we hear large musical noise, especiallyin the UD case. On the other hand, the proposed methods achieve higher separation performance with muchless and without musical noise for UD and D cases, respectively. Furthermore, the limitations of the conventional approaches, described in section 1, are overcome.

Figure 4. Room setup.
Table 1 shows the separation results for the determinedcase (D), when sources S1, S2 and S3 were used, and forthe underdetermined case (UD). We used different BSS
approaches in order to compare them. A conventional
beamformer (CB) was designed as described in section
2.2, namely using steering vector ak(f) with given target
locations and observation correlation matrix Rk(f). Notethat separation is not performed blindly. The conventional TFBM setup corresponds to the approach outlined
section 2.3. Finally, we used our proposed method,ˆ ( f ) and hˆ ( f ) , R
ˆ ′ ( f ) in the Dwhich exploits hˆ ( f ) , Rkkand UD cases, respectively.
The CB setup did not achieve good results because theˆ ( f ) was not usedjammer only correlation matrix R
kand furthermore the steering vector ak(f) could not reflect the room reverberation or sensor misalignment.
The TFBM achieved higher SIR and SDR values thanWe introduced a new BSS approach for both (over)determined and underdetermined cases by assuming
source sparseness. Our method combines beamformers
and TFBM and provides with better separation performance than conventional techniques. Because thebeamformer is the core of the separation, the musicalnoise is significantly reduced and removed in underdetermined and (over-)determined cases, respectively. Furthermore, the computation time is not significantly increased revealing the potential for real time application.
7. REFERENCES
[1]
[2]

[3]
[4]
[5]
[6]

A. Hyavarinen, J Karhunen and E. Oja, Independent
Component Analysis, John Wiley & Sons, 2001.
H. Sawada, R. Mukai, S. Araki and S. Makino, “Frequency-domain blind source separation,” in Speech
Enhancement, J Benesty, S. Makino and J. Chen, Eds.
Springer, Mar. 2005.
D. Johnson, D. Dudgeon, Array Signal Processing,
Prentice Hall, 1993.
S. Araki, H. Sawada, R. Mukai, S. Makino, “A novel
blind source separation method with observation vector
clustering,” in Proc. IWAENC2005, Sept. 2005.
O. Yilmaz, S. Rickard, “Blind separation of speech
mixtures via time-frequency masking,” IEEE Trans. onSP, vol. 52, no. 7, pp. 1830-1847, 2004.
S. Araki, H. Sawada, R. Mukai, S. Makino, “DOA
estimation for multiple sparse sources with normalized
observation vector clustering,” in Proc. ICASSP2006,
May 2006.