BLIND SOURCE SEPARATION BASED ON A BEAMFORMER ARRAY AND TIMEFREQUENCY BINARY MASKING
1,2

Jan Cermak, 1Shoko Araki, 1Hiroshi Sawada and 1Shoji Makino

1

NTT Communication Science Laboratories, NTT Corporation
2-4 Hikaridai, Seika-cho, Soraku-gun, Kyoto 619-0237, Japan
2
Brno University of Technology, Faculty of Electrical Engineering and Communication,
Department of Telecommunication, Purkynova 118, 612 00 Brno, Czech Republic
ABSTRACTThis paper deals with a new technique for blind source
separation (BSS) from convolutive mixtures. We present a
three-stage separation system employing time-frequency
binary masking, beamforming and a non-linear postprocessing technique. The experiments show that thissystem outperforms conventional time-frequency binary
masking (TFBM) in both (over-)determined andunderdetermined cases. Moreover it removes the musicalnoise and reduces interference in time-frequency slots
extracted by TFBM.
Index Terms— Array signal processing, speechprocessing, speech enhancement

and the final separation was accomplished by multiple
beamformers. Another approach to reducing musical noise
is to employ a soft mask instead of a binary mask [4].
In this paper we present a technique that overcomes thelimitations of TFBM described above. Our method removes
the musical noise and suppresses the interference in all
time-frequency slots. A block diagram of the proposed
three-stage system is shown in Fig. 1. A pre-separation
process using TFBM is employed to design the
beamformers blindly as described in section 3. The core of
the separation technique is the beamformer array (BA)
described in section 4. In section 5 we present a method forsignal enhancement, indicated as ENH in Fig. 1, that further
improves the separation performance.
2. MIXING MODEL

1. INTRODUCTION
Blind source separation has been intensively studied inrecent years due to its many potential applications [1]. We
address BSS as a technique used for separating signals fromtheir mixtures when the mixing process is unknown. The
separation techniques can basically be divided into twogroups according to the number of sources N and sensors M,
namely, (over-)determined cases when N d M andunderdetermined cases when N ! M .
Time-frequency binary masking [2] is a versatile
approach capable of dealing with both (over-)determined
and underdetermined cases that relies on signal sparsenessand the assumption of independent sources. TFBM is basedon estimating N time-frequency binary masks from thesignal mixtures observed at M sensors. The target signals
are then easily separated by multiplying the time-frequency
binary masks with one of the signal mixtures. However
TFBM leads to short tones randomly distributed in theseparated target signal over the time and frequency andknown as musical noise. Moreover interference remains in
the extracted time-frequency slots.
We have proposed a system combining TFBM andadaptive beamformers [3] to ease the above-mentioned
limitation. TFBM was exploited as a pre-separation process1­4244­0728­1/07/$20.00 ©2007 IEEE

The convolutive mixing model is used to describe signal
observations in a real room environment, i.e. anenvironment with reverberations. We consider this modelwith M sensors observing N sourcesN

x j (t )¦¦ h (l )s (t  l ),
jk

k 1

k

j 1,! , M,(1)lwhere xj(t) is the signal observed by the j-th sensor, t is thediscrete time index, sk(t) is the k-th source signal and hjk(t) is
the impulse response from the k-th source to the j-th sensor.
The mixing model (1) in the time-frequency domain
becomesNx( f ,W ) | ¦ h k ( f ) sk ( f ,W )

(2)

k 1

where x( f ,W ) [ x1 ( f ,W ),! , xM ( f ,W )]T is an observation
vector and h k ( f ) [ h1k ( f ),! , hMk ( f )]T is a mixing vector.
If the source signals are sparse, which holds for speech
signals [5, 2], the sources rarely overlap in the timefrequency domain and (2) can be approximated asx( f ,W ) | h k ( f ) sk ( f ,W ),(3)where sk(f,W) is the dominant source at the time-frequency
point (f,W).

I ­ 145ICASSP 2007hˆ k ( f )ª x1 (t ) º
«
»
« # »
¬« xM (t ) ¼»㪪㪫㪝㪫x( f ,W )㪫㪝㪙㪤yˆ k ( f ,W )㪙㪘

M k ( f ,W )y ( f ,W )㪜㪥㪟y ( f ,W )㪠㪪㪫㪝㪫

ª y1 (t ) º
«
»
« # »
«¬ yN (t ) »¼

Fig. 1. Block diagram of the proposed system. STFT = short time Fourier transform, ISTFT = inverse STFT.
3. BEAMFORMER DESIGN WITHOUT A PRIORIINFORMATION
The main part of our separation system is the BA, which
consists of minimum variance (Frost) beamformers. Thissection describes how to design the Frost beamformers
blindly. The spatial filter for enhancing source sk(t) is
designed by using [6],§·R 1 ( f )a k ( f )wk ( f ) ¨ H
(4)
¸,
1
© a k ( f )R ( f )a k ( f ) ¹

where R ( f ) E[x( f ,W )x H ( f ,W )] is the correlation matrixof the observation vector, E[] is the expectation operator, H
is the complex conjugate transpose and a k ( f )[e  j2ʌf W1k ,..., e  j2ʌf W Mk ] is the given steering vector. Wjk is thetime delay between the arrival of sk(t) at sensor j and atreference sensor J. The critical problem with the Frost
beamformer (4) is that the design is not performed blindly.
To design the beamformer blindly and to further
improve its performance, we have proposed using TFBM toestimate the mixing vector hˆ k ( f ) and the correlation matrix
ˆ (f)of the observation vector in the jammer only period R
k[3]. Jammer only period is a time period when the target
signal sk (t ) is not active. Note that R ( f ) and a k ( f ) can beˆ ( f ) and hˆ ( f ) in (4), respectively. The
substituted by Rkkremainder of this section reviews the way to estimate
ˆ ( f ) and hˆ ( f ) .
RkkFirst, we estimate time-frequency binary mask
M k ( f ,W ) so that the pre-separated signalyˆ k ( f ,W ) [ yˆ1k ( f ,W ),..., yˆ Mk ( f ,W )]T M k ( f ,W )x( f ,W ) (5)becomesanestimationofthesource,i.e.
yˆ k ( f ,W ) | h k ( f ,W ) sk ( f ,W ) . M k ( f ,W ) extracts the timefrequency slots of cluster Ck whose members are estimatedto belong to the source signal sk ( f ,W )x( f , W )  C k­1
M k ( f ,W ) ®
.
(6)otherwise¯0
The cluster Ck can be estimated by using, for example,
DUET [2], clustering the normalized observation vector [7]or the direction-of-arrival estimations [8]. Note that unlike
conventional TFBM [2, 7] where the binary mask is applied
to one observation only, the source separation (5) isaccomplished over the whole observation vector x( f ,W ) inorder to use the output vector yˆ k ( f ,W ) for designing the
Frost beamformers.

ˆ ( f ) and hˆ ( f ) for theThen we estimate Rkkbeamformer (4). Let yˆ k ( f ,W ) be the target component, andyˆ b ( f ,W ), b (1,! , N, b z k ) be interference for the source k.
ˆ ( f ) is estimated by
The correlation matrix Rkˆ ( f ) E[nˆ ( f ,W )nˆ H ( f ,W )]
R
(7)
k
k
k
where nˆ k ¦ bz yˆ b ( f ,W ), and zk is a set of jammerkindexes whose size is less than or equal to M-1 [3]. For
example if N=M=3 and k=1 then zk {2, 3} .
To estimate the mixing vector hˆ k ( f ) , we want tominimize the criterion
(h k ( f )) E[x( f ,W )  hˆ k ( f ) yˆ Jk ( f ,W )] .
(8)
Setting the derivative w(h k ( f )) / wh k ( f ) to zero leads tothe solution
E[x( f ,W ) yˆ J*k ( f ,W )]hˆ k ( f ),(9)E[| yˆ Jk ( f ,W ) |2 ]where * denotes conjugation.

4. BEAMFORMER ARRAYIn [3] and Section 3, we utilized just one beamformer
w k ( f ) to obtain the k-th output yk ( f ,W ) , whichcorresponds to an estimation of the target sk ( f ,W ) . In thispaper, we propose BA, see Fig. 2, which utilizes D
beamformers (k-th column in BA in Fig. 2) to estimate thek-th target signal yk ( f ,W ) .
Let us assume that our goal is to extract just one signal
from the signal mixture. In this case BA consists only of
one column (D beamformers). The main idea behind BA is
to compose D different mixtures from the pre-separated
signals provided by TFBM, which are then filtered by D
beamformers. All these beamformers are designed toenhance the target signal. Each input mixture includes the
pre-separated target signal and different pre-separated
jammers. The important point is that all the jammers must
be used at least once. As a result of beamforming we obtain
the enhanced target signal D times. Finally, we add together
all the outputs of the beamformers. Note that we used all
pre-separated signals (all time-frequency slots) andtherefore the BA output contains no musical noise.
Now, we describe the BA mathematically. The BA has
N columns and D rows. D is a binomial coefficientestimated by§ N-1·(N  1)!
D= ¨
.
(10)
¸
Z
©
¹ Z!(N-1-Z)!

I ­ 146

Each cell of the BA is a beamformer given asˆ 1 ( f )hˆ ( f )§·Rkw kd ( f ) ¨ H kd
¸¸ ,
(11)

1
¨ hˆ ( f )Rˆˆ
kd ( f )h k ( f ) ¹© k
where d is the row index and k is the column index, and theindex of the target signal i.e. each column is responsible forseparating one target signal.
ˆ (f)
ˆ (f)
hˆ 1 ( f ) Rhˆ N ( f ) R11
N1
y11 ( f ,W ) x N1 ( f ,W )y N1 ( f ,W )x 11 ( f ,W )
㫎㪈㪈
㫎㪥㪈

㪅㪅㪅

㪅㪅㪅

ˆ ( f ) 㪅㪅㪅hˆ 1 ( f ) R
1Dx 1D ( f ,W )
㪂
㫎㪈㪛

ˆ (f)
hˆ N ( f ) R
NDx ND ( f ,W )㪂㫎㪥㪛

­ D M k ( f ,W ) 1.
¯ E M k ( f ,W ) 0㪪㪚㪘㪣㪠㪥㪞

y1 ( f ,W )y N ( f ,W )Fig. 2. Beamformer Array.
The input of a single beamformer isx kd ( f ,W ) yˆ k ( f ,W )  nˆ kd ( f ,W ),(12)where nˆ kd ( f ,W ) is a vector of a mixture ofZ  ^1,! , M  1` jammer signals. nˆ kd ( f ,W ) is counted bynˆ kd ( f ,W ) ¦ yˆ b ( f ,W ),(13)bz kdwhere zkd is the d-th Z-subset [9] of T k . The set T k{q 1! N, q z k} contains indexes of all the jammer signalsˆ ( f ) E[nˆ ( f ,W )nˆ H ( f ,W )] isfor the k-th target signal. R
kd
kd
kd
estimated similarly to (7) from a mixture of jammers.
Let us take an example where N=4, M=3, Z=1 and k=1.
The size of the BA is N=4 and D=3. The set of all jammer
indexes is T1 {2, 3, 4} and the 1-subsets are z11=2, z12=3,
z13=4. The input signals of the beamformers arex 11 ( f ,W ) yˆ 1 ( f ,W )  yˆ 2 ( f ,W ), x 12 ( f ,W ) yˆ 1 ( f ,W )  yˆ 3 ( f ,W )and x 13 ( f ,W ) yˆ 1 ( f ,W )  yˆ 4 ( f ,W ) .
In
onecolumn,theoutputsignalsy kd ( f ,W ) w kd H ( f )x kd ( f ,W ) of each beamformer are added
together and then scaled with the number of rows D using
block SCALING in Fig. 2Dy k ( f ,W )

kd

d 1

,

In the third section, ENH, the output vector y ( f ,W ) of BA, isfurther enhanced. We propose a one-channel enhancement
nonlinear technique based on knowledge of the timefrequency binary mask M k ( f ,W ) . The enhancement
improves the interference suppression in the time-frequency
slots of the target signal y k ( f ,W ) where M k ( f ,W ) 0 . Thiscan be easily achieved byyk ( f ,W ) [ k ( f ,W ) y k ( f ,W ),(15)
where the enhancement factor [ k ( f ,W ) is

[ k ( f ,W ) ®

㪪㪚㪘㪣㪠㪥㪞¦ y ( f ,W )5. ENHANCEMENT(14)

D
where y ( f ,W ) [ y1 ( f ,W ),! , y N ( f ,W )] is the output vectorof the BA. Single beamformer outputs can be added due tothe constraint of the Frost beamformer w kd H ( f )h k ( f ) 1,
d . The summation (14) is important for two reasons.
Firstly, the target signal is enhanced since, unlike thejammers, it is in phase. Secondly, the musical noise isremoved because all N pre-separated signals yˆ k ( f ,W ) are
included in the input signals x k1 ( f ,W ),! , x kD ( f ,W ) andtherefore y k ( f ,W ) includes no zero padding.

(16)

The coefficients D and E can assume three different
combinations
1) D ! 1, E 1 ,2) D 1, E  1 ,3) D ! 1, E  1 .
There are more possible ways to determine thesecoefficients. Note that this stage may cause musical noisee.g. by setting E 0 .

6. EXPERIMENTS
We performed experiments for a determined case (DC)(M=3, N=3) and an underdetermined case (UDC) (M=3,
N=4) in a room with a reverberation time of 120 ms, (seeFig. 3). The source signals were 5-second English andJapanese utterances. The STFT frame size was L=512, theframe shift was L/4, and the sampling frequency fs was 8kHz. The separation performance was evaluated in terms of
the signal-to-interference ratio (SIR) and signal-to-distortion
ratio (SDR)
E ª¬ ykk (t ) 2 º¼
SIR k 10 log10
[dB],
(17)
ª N
º
E « ¦ ykb (t ) 2 »¬ b 1,b z k¼
2ªº
E ¬ xPk (t ) ¼
SDR k 10 log10
[dB],
(18)
2
E ª xPk (t )  J ykk (t  ' ) º¬¼
where ykb (t) are the jammer components that appear in thek-th output target signal, ykk (t) is the output signal without
any contribution from the jammers and xkP (t )¦ l hPk (l )sk (t  l ) . P is an arbitrarily selected sensor.
Coefficients J and ' compensate the amplitude and delay,
respectively.
The average results for the DC, when sources S1, S2 andS3 were used, and for the UDC are summarized in the Table
1. We used TFBM as a performance reference for BA
evaluation. The size of the BA was N=3, D=3 in DC and
N=4, D=3 in the UDC. We use the notation BAZ-E in Table
1, where Z  ^1, 2` is the size of Z-subsets and ȕ is a constantI ­ 147

enhancement value (16) for all time-frequency slots. Į was
set at one i.e. we used the second possible enhancement
combination introduced in the previous section.
880 cmMicrophone (120cm height, omni-directional)••

Loudspeaker (120cm height)
Reverberation time: 120 ms
Room height: 240cm

30㫦

Microphone setup
Mic. 2dMic. 1dd0㫦Mic. 3S1

d=4 cm375 cm

S4

S3
135㫦
S2
282cm

50 cm 225㫦
80 cm

7. CONCLUSION

200 cm300㫦

Fig. 3. Room setup.
We can see from Table 1 that the BA without
enhancement (BA1-1 and BA2-1) outperforms conventional
TFBM [7] in both the DC and UDC. BA1-1 achieves ahigher SIR than BA2-1 because there are fewer jammersin x kd ( f ,W ) but on the other hand BA1-1 causes more
distortion. The musical noise is removed in both setups.
Applying the third stage ENH of our system, we cansee that the SIR increases further with decreasing EBut wecan also observe the decreasing SDR and furthermore themusical noise becomes audible. By setting E the BA
output signals are binary masked, which is similar to the
output with conventional TFBM, i.e. we introduce a musicalnoise into the BA output signals. However, unlike TFBM,
our system reduces the interference even in time-frequency
slots, where M k ( f ,W ) 1 . This setup results in a lower
SDR but a much higher SIR than with conventional TFBM.
It is obvious from Table 1 that BA2-0 outperforms BA1-0,
which means that BA2-0 has less interference in
M k ( f ,W ) 1 time-frequency slots.

Table 1. Separation results.
DCUDCSIR [dB] 㩷 SDR [dB] 㩷 SIR [dB]  SDR [dB] 
TFBM㩷
BA1-1㩷
BA1-0.5㩷
BA1-0㩷
BA2-1㩷
BA2-0.5㩷
BA2-0㩷

10.93 㩷16.67 㩷18.39 㩷19.43 㩷14.61 㩷18.80 㩷23.77 㩷10.57 㩷10.93 㩷9.93 㩷
8.89 㩷
13.62 㩷12.04 㩷9.38 㩷

9.56 㩷
13.81 㩷14.93 㩷15.57 㩷11.00 㩷13.93 㩷16.44 㩷audible sooner in the BA1-ȕ setup. We can say that the BA
with Z=1 better suppresses the jammers in M k ( f ,W ) 0
than the BA with Z=M-1. In contrast, the BA with Z=M-1
suppresses the jammers better in M k ( f ,W ) 1 .
These results were anticipated and we can draw thegeneral conclusion that a BA designed with Z=1 gives
satisfactory results even as a two-stage system without
enhancement. The BA designed with Z=M  1 has moreinterference in the time-frequency slots where M k ( f ,W ) 0
hence the third stage highly improves the performance.

8.97 㩷
9.48 㩷
8.60 㩷
7.74 㩷
11.44 㩷9.75 㩷
7.89 㩷

We introduced a new three-stage BSS approach thatprovides high separation performance. We have shown thata BA removes the musical noise caused by conventional
TFBM. Moreover the interference in the extracted timefrequency slots of the target signal is reduced. The third
stage of our system allows us to control the level of musicalnoise and interference in the output signal. This is aversatile approach to signal enhancement that is capable offurther improvement.

8. REFERENCES[1] S. Haykin, Ed., Unsupervised Adaptive Filtering (Volume I:
Blind Source Separation), John Wiley & Sons, New York, 2000.
[2] O. Yilmaz, S. Rickard, “Blind separation of speech mixtures
via time-frequency masking,” IEEE Trans. on SP, vol. 52, no. 7,
pp. 1830-1847, 2004.
[3] J. Cermak, S. Araki, H. Sawada and S. Makino, “Blind speech
separation by combining beamformers and time frequency binarymask,” in Proc. IWAENC2006, Sept. 2006.
[4] S. Araki, H. Sawada, R Mukai and S. Makino, “Blind sparsesource separation with spatially smoothed time-frequency
masking,” in Proc. IWAENC2006, Sept. 2006.
[5] P. Bofill and M. Zibulevsky, “Blind separation of more sources
than mixtures using sparsity of their short-time Fourier transform,”in Proc. ICA2000, pp. 87-92, June 2000.
[6] D. Johnson, D. Dudgeon, Array Signal Processing, Prentice
Hall, Upper Saddle River, 1993.
[7] S. Araki, H. Sawada, R. Mukai, S. Makino, “A novel blindsource separation method with observation vector clustering,” inProc. IWAENC2005, pp. 117-120, Sept. 2005.
[8] Z. Yermeche, N. Grbic, I. Claesson, “Speech enhancement ofmultiple moving sources based on subband clustering time-delay
estimation,“ in Proc. IWAENC2006, Sept. 2006.

We can also see from Table 1 that the BA1-ȕ setup ismuch less influenced by ȕ changes than the BA2-ȕ setup.
Furthermore by decreasing ȕ , the musical noise becomes[9] Mathworld – k-subset, Online: http://mathworld.wolfram.com/k-Subset.html, April 21, 2004.

I ­ 148