UNDERDETERMINED BLIND SPEECH SEPARATIONWITH DIRECTIVITY PATTERN BASED CONTINUOUS MASK AND ICA
Shoko Araki

Shoji Makino

Hiroshi Sawada

Ryo Mukai

NTT Communication Science Laboratories, NTT Corporation
2-4 Hikaridai, Seika-cho, Soraku-gun, Kyoto 619-0237, Japan
Email: shoko@cslab.kecl.ntt.co.jp

Blind source separation (BSS) is an approach that estimatesoriginal source signals si (n) only from observations x j (n) without
source or mixing process information.
In this paper, we consider the BSS of speech signals observedin a real environment, i.e., the BSS of convolutive mixtures ofspeech. Recently, many methods have been proposed to solve the
BSS problem of convolutive mixtures (e.g., [1]). However, mostof these methods consider the determined or overdetermined case.
In contrast, we focus on the underdetermined BSS problem where
source signals outnumber the sensors, especially the case of threesources and two sensors.
There are two approaches with which to realize underdetermined BSS. Both approaches rely on the sparseness of source signals. One involves the clustering of time-frequency points with binary masks [2], and the other is based on ML estimation, where thesources are estimated after mixing matrix estimation [3–5]. Since
separation in a real environment has already been tried with the former method, we decided to watch a binary mask approach [2]. If
the signals are sufficiently sparse, that is, most of the samples of asignal are almost zero, we can assume that the sources rarely overlap. Rickard and Yilmaz [2] employ this assumption and extract
each signal using a time-frequency binary mask (BM). However,the use of binary masks means that there is too much discontinuous zero-padding to the extracted signals, and they contain loudmusical noise.
To overcome this, we have proposed combining binary masks
and ICA (BMICA) to solve the underdetermined BSS problem [6].
In the paper, we estimate the time points when only one source isactive using the sparseness assumption. Then, we remove this single source from the observations with a binary mask and apply ICA
to the remaining mixtures to obtain separated signals. The single
source removal with this method causes less discontinuous zeropadding than with the BM method, because the BMICA extracts
more time-frequency points than the BM method. Therefore, we

sN

x1
hji

xj
xM

Mixingprocess

Separated
signalsy1..... .....

1. INTRODUCTION

si

Observed
signals..... .....

We propose a method for separating speech signals when sources
outnumber the sensors. In this paper we mainly concentrate on
the case of three sources and two sensors. Some existing methods
employ binary masks to extract the signals, and therefore, the extracted signals contain loud musical noise. To overcome this problem, we propose the utilization of a directivity pattern based continuous mask, which removes a single source from the observations,and independent component analysis (ICA) to separate the remaining mixtures. Experimental results show that our proposed method
can separate signals with little distortion even in a real reverberant
environment of TR =130 ms.

..... .....

Sourcesignalss1
s2

ABSTRACTy2
ykyN

Separatingprocess

Figure 1: Block diagram of underdetermined BSS. N > M.
have been able to use the BMICA to obtain separated signals withlittle distortion. However, as the BMICA still employs a binary
mask for one source removal, the zero-padding to the separated
signals still remains. Moreover, heuristic parameters are used when
designing the binary mask shapes not only for the BMICA but also
for the BM method.
In this paper, we propose a new method for one source removal that employs a directivity pattern based continuous mask
(DCmask). First, using the source sparseness, we estimate the direction of arrival (DOA) of each source. We then remove the single
source from the observations with a DCmask, which has a small
gain for the DOA of one source and a large gain for other directions. We next apply ICA to the remaining mixtures. Since thisDCmask is not binary, discontinuous zero-padding to the extracted
signals does not occur. Moreover, we do not need any parameters
for designing the DCmask.
Experimental results show that our method can separate signals with little distortion even in a real reverberant environment of
TR =130 ms.
2. PROBLEM DESCRIPTIONIn real environments, N signals observed by M sensors are modL
eled as convolutive mixtures x j (n) = ∑N
i=1 ∑l=1 h ji (l) si (n − l +1) ( j = 1, · · · , M), where si is the signal from a source i, x j is thesignal observed by a sensor j, and h ji is the L-taps impulse response
from a source i to a sensor j (see Fig. 1). Our objective is to obtainseparated signals yk (n) (k = 1, · · · , N) using only the information
provided by observations x j (n). Here, we consider the underdetermined case N > M. In this paper N = 3 and M = 2. Moreover,the sources are speech signals, i.e., the sources are assumed to be
mutually independent and sufficiently sparse in the time-frequency
domain.
This paper employs a time-frequency domain approach because speech signals are more sparse in the time-frequency domain than in the time-domain [5] and convolutive mixture prob-

1991

10
10
99

(a) TR = 0ms

S1@1969Hzoccurency

88
77

θ1

Separation of S2 and S3

66
55
44
33

S3

22
11
000
0

20

20

40

40

~
θ1

60

60

80

80

100

100

~
θ2

120

120

[1st stage] X1 [2nd stage]

X2

^ 23

160

^ 23

X =M23X

X2

23

23^ 23

Y =W X

Y2
Y3

Separation of S1 and S2^ 12[1st stage] X1 [2nd stage]

θ~ DOA (deg.)
θ3
140

^ 23

X1

S2

180

140 160 180^ 12

X =M12X

Figure 2: Example histogram. A male-male-female combination
with STFT frame size T = 512. TR = 0 ms.
lems can be converted into instantaneous mixture problems at eachfrequency. In the time-frequency domain, mixtures are modeled asX(ω , m) = H(ω )S(ω , m), where H(ω ) is a 2×3 mixing matrix
whose j-i component is a transfer function from a source i to a sensor j, S(ω , m) = [S1 (ω , m), S2 (ω , m), S3 (ω , m)]T and X(ω , m) =[X1 (ω , m), X2 (ω , m)]T denote short-time Fourier transformed
source and observed signals, respectively. ω is the frequency andm is the time-dependence of the short-time Fourier transformation(STFT). Moreover, we write short-time Fourier transformed separated signals as Y(ω , m) = [Y1 (ω , m),Y2 (ω , m),Y3 (ω , m)]T .
3. CONVENTIONAL METHODS
Several methods have been proposed (e.g., [2–7]) for solving theunderdetermined BSS problem, and they all utilize source sparseness. If most of the samples of a signal are almost zero, we saythat this signal is sparse. For a detailed experimental analysis ofsparseness, see [7].
3.1 Conventional method 1: with only binary masks (BM)Some methods use the sparseness assumption and extract eachsignal using time-frequency binary masks (e.g.,[2]). When signals
are sufficiently sparse, it can be assumed that sources do not overlap very often. This condition is closely discussed in [2]. Using
this non-overlap assumption, they extract each source by selecting the time points at which there is only one signal. One way of
estimating such time points is to use the level difference and thephase difference between the observations. In this paper, we utilize
omni-directional microphones, therefore we use the phase differX (ω ,m)ence ϕ (ω , m) = 6 X1 (ω ,m) between the observations.
2

Using ϕ (ω , m), we estimate the direction of arrival (DOA) forϕ (ω ,m)ceach time point m by calculating θ (ω , m) = cos−1 ω d , where cis the speed of sound and d is the microphone spacing, and draw ahistogram of the DOA θ (ω , m) (see Fig. 2). Each peak corresponds
to each source in the histogram for each frequency. Let these peaks
be θ˜1 , θ˜2 and θ˜3 where θ˜1 ≤ θ˜2 ≤ θ˜3 (Fig. 2), and the signal fromθ˜r be S˜r (r = 1, 2, 3).
We can extract each signal with a binary mask1 θ˜r − ∆ ≤ θ (ω , m) ≤ θ˜r + ∆r
MBM (ω , m) =(1)0 otherwiser (ω , m)X (ω , m) where j=1 or 2.
by calculating Yr (ω , m) = MBM
jHere, ∆ is an extraction range parameter.
Although we can extract each signal using this binary mask(1), such extracted signals are discontinuously zero-padded by thebinary masks, and therefore, we hear musical noise in the outputs.
Moreover, the performance depends on the heuristic parameter, ∆.

^ 12

X2

12

12^ 12

Y =W X

Y1
Y2

Figure 3: System setup
3.2 Conventional method 2: with binary masks and ICA(BMICA)To overcome the musical noise problem, we have proposed using
both a binary mask and ICA (BMICA) [6]. The BMICA has twostages (Fig. 3).
In the first stage, unlike the conventional method 1 where single
source is extracted, we remove one source from the mixtures usinga binary mask
1 θmin ≤ θ (ω , m) ≤ θmaxpq(ω , m) =(2)
MBMICA0 otherwise
by calculatingpq
X̂ pq (ω , m) = MBMICA(ω , m)X(ω , m).
(3)
In (2), θmin and θmax are extraction range parameters that cover
the direction range of two sources, and in (3), X̂ pq (ω , m) =[X̂1pq (ω , m), X̂2pq (ω , m)] are expected to be mixtures of S˜p and S˜q .
Then, as we can expect the remaining mixtures X̂ pq to consist of
only two signals, we apply a standard ICA to these remaining mixtures in the second stage.
In the 1st stage, we expect that the zero-padding of the extracted signals to cause less trouble because we extract more timefrequency points than conventional method 1. Therefore we obtained a result with less musical noise [6].
However, as BMICA still employed a binary mask for one
source removal, the zero-padding of the separated signals still remained. Moreover, we have to find a reasonable θmin and θmax in(2). This is not an easy problem and we relied on manual setting.

4. PROPOSED METHOD[1st stage] One source removal with new DC mask:
In this paper, we propose a new method (Fig. 4) which employs
a directivity pattern based continuous mask (DCmask) in the 1ststage. The DCmask has a small gain for the DOA of one sourceand preserves signals from other directions. As shown in Fig. 2
and Sec. 3, DOA is a powerful clue with which to estimate theactive source in each frame and to extract or remove one signal.
Therefore the DCmask is a natural alternative. Because this newmask is not binary, it completely avoids any zero-padding to theextracted signals.
One way to obtain such a mask is to utilize the gain of thedirectivity pattern of a null beamformer (NBF) which makes a null
toward the estimated direction θ̃r of one source.
First we estimate the source directions with a histogram suchas that in Fig. 2. Then, we form a matrix
exp ( jωτ11 ) exp ( jωτ12 ),(4)
HNBF (ω ) =exp ( jωτ21 ) exp ( jωτ22 )

1992

θ1

[1st stage]
DCmask
X1

S2
X2
S3

pq
DC

M

pq

MDC

[2nd stage]
ICA

1

^ pq

0.8

X1

^ pq

X2

@3125Hz

0.9

Yp

0.7

pq

W

|F3|

0.6

Yq

gain

S1

(p=r, q=r)

DOAest.
with Fig.2 ~

|F1|

0.5

0.4

0.3

θ

0.2

0.1

Figure 4: Block diagram of proposed method. Only one path isdrawn for visibility.

0

0

0

θ

S3

S2

90

4 cm
o

3.55 m135o

5.1 Experimental conditions
We conducted anechoic tests and reverberant tests. For the anechoic tests (TR = 0 ms), we simulated the recording in an anechoic room using the mixing matrix H ji (ω ) = exp ( jωτ ji ), where

180

180

DOA (deg.)~
θ32.25 m45

1.75 mS1

Omni-directional microphones(height : 1.39 m)
Room height : 2.50 m

Figure 6: Room for reverberant tests. TR = 130 ms.

(7)

5. EXPERIMENTS

~
θ2

160

160

o

(6)

[2nd stage] Separation of remaining sources by ICA:
Because the remaining signals X̂ pq are expected to be mixturesof two signals, we can use 2×2 ICA to separate X̂ pq . The separation process can be formulated as
Y pq (ω , m) = W pq (ω )X̂ pq (ω , m),(8)pq
where X̂ is the masked observed signal obtained by (7),Y pq (ω , m) = [Yp (ω , m),Yq (ω , m)]T is the separated output signal,and W pq (ω ) represents a (2×2) separation matrix. Wpq (ω ) is
determined so that Yp (ω , m) and Yq (ω , m) become mutually independent by ICA.
Note that we need two paths to obtain three separated signals(see Fig. 3) because our system has only two outputs.
Although we consider the case of two sensors (M = 2) and threesources (N = 3) here, we can expand our DCmask method to theunderdetermined case of N ≤ (the number of nulls formed by M
sensors) + (the number of outputs of a standard ICA) = (M − 1) +
M.

140

140

1.1 m

(5)

It should be noted that the DCmask is applied to both channels(Fig. 4), because ICA in the 2nd stage needs two inputs that mainpq (ω , m)tain the mixing matrix information. Here, our new mask MDC
changes only the gains of inputs X1 and X2 and preserves theirphases. Therefore, we can preserve to some extent the spatial information on the sources, and a standard ICA can work in the 2ndstage.

120

1.80 mThis is our new mask, the DCmask. Figure 5 shows an example ofthe shape of a DCmask.
Finally, the one source removal is achieved bypq(ω , m)X(ω , m).
X̂ pq (ω , m) = MDC

100

100 120

Loudspeakers(height : 1.35 m)

Here, we use the gain of the directivity pattern of NBF as our mask,(p 6= r, q 6= r).

80

80

4.45 mk=1

pq
MDC(ω , m) = |Fr (ω , θ (ω , m))|60

60

Figure 5: Example mask pattern.

c is the speed of sound. Where, one of the θ̃i values should be theestimated DOA θ˜r of signal to be removed, and the other should
be a different direction from which the signal’s gain and phase areconstrained at 1.
The directivity pattern of the NBF, W(ω ) = H−1
NBF (ω ), is obtained by
Fr (ω , θ ) = ∑ Wrk (ω ) exp ( jω dk cos θ /c).

40

40

~
θ1

d
where τ ji = cj cos θ̃i , d j is the position of the j-th microphone and

M

20

20

d

τ ji = cj cos θi , d j is the position of the j-th microphone, and θi isthe direction of the i-th source. The source directions were 45◦ , 90◦
and 135◦ .
For the reverberant tests, we used speech data convolved with
impulse responses recorded in a real room (Fig. 6) whose reverberation time was TR = 130 ms.
As the original speech, we used three Japanese sentences spoken by three male and three female speakers. We investigated threecombinations of speakers: male-male-female (m-m-f), male-malemale (m-m-m), and female-female-female (f-f-f).
The STFT frame size T was 512 and the frame shift was 256
at a sampling rate of 8 kHz. We used θmin = θ˜2 − ∆ and θmax =23180◦ for MBMICA(BMICA 23), and θmin = 0◦ and θmax = θ˜2 + ∆12
for MBMICA (BMICA 12). The ∆ value in the conventional methods’
binary masks was 15◦ in DOA.
The adaptation rule of ICA
 that we used in our experiments
was Wi+1 (ω ) = Wi (ω ) + η diag hΦ(Y)Y H i − hΦ(Y)Y H i
6· Wi (ω ), where Φ(y) = φ (|y|) · e j· (y) , φ (x) = tanh(gx) andg = 100. To solve the permutation problem of frequency domain
ICA, we employed the DOA and correlation approach [8], and tosolve the scaling problem of frequency domain ICA, we used theminimum distortion principle [9].

5.2 Performance measures
We used the signal to interference ratio (SIR) as a measure of
separation performance, and the signal to distortion ratio (SDR) asa measure of sound quality:

1993

SIRi = 10 log∑n y2isi (n)∑n (∑i6= j yis j (n))2(9)

Table 1: Percentage of each source power. TR =0ms.
combi.s
m-m-f
m-m-m
f-f-f

12 =|F |DCmask MDC
3
S1 :75, S2 :23, S3 :2
S1 :71, S2 :24, S3 :5
S1 :76, S2 :22, S3 :2

Table 3: Percentage of each source power. TR =130ms.

23 =|F |DCmask MDC
1
S1 :2, S2 :22, S3 :76
S1 :4, S2 :24, S3 :72
S1 :2, S2 :23, S3 :75

combi.s
m-m-f
m-m-m
f-f-f

Table 2: Results of TR =0 ms simulations. ‘Conv. 1’: with conventional method 1, ‘BMICA 12’ and ‘BMICA 23’: with conventional method 2, ‘DCICA 12’ and ‘DCICA 23’: with proposed
method.
SIR1 SIR2 SIR3 SDR1 SDR2 SDR3
Conv. 1
18.0
8.9
18.4
7.9
11.5
8.3
BMICA 12 12.6
5.9
18.1
15.2
BMICA 23
6.1
13.0
13.6
17.4
DCICA 12 16.1
4.8
15.2
12.5
DCICA 23
4.6
16.2
12.8
15.4

∑n x2ksi (n)∑n (xksi (n) − α yisi(n − D))2

6. CONCLUSION

(10)

,

where yi is the estimation of si , and yis is the output of the wholejseparating system at yi when only s j is active, and xks = hki ∗ si (∗i
is a convolution operator). α and D are parameters to compensate
for the amplitude and phase difference between xks and yis .
i23 =|F |DCmask MDC
1
S1 :8, S2 :38, S3 :54
S1 :7, S2 :29, S3 :64
S1 :4, S2 :41, S3 :55

Table 4: Results of reverberant tests. TR =130 ms.
SIR1 SIR2 SIR3 SDR1 SDR2 SDR3
Conv. 1
12.3
6.3
11.0
5.0
13.9
5.8
BMICA 12
9.8
5.5
7.8
15.9
BMICA 23
5.5
9.2
14.5
9.3
DCICA 12 11.4
3.2
6.7
6.3
DCICA 23
4.0
11.7
12.3
8.4
[dB]

[dB]
SDRi = 10 log

12 =|F |DCmask MDC
3
S1 :69, S2 :23, S3 :8
S1 :65, S2 :23, S3 :12
S1 :66, S2 :29, S3 :5i

5.3 Experimental results
5.3.1 Applicability of ICA at the 2nd stage
Before trying to separate signals with our method, we investigated the applicability of ICA at the 2nd stage, because if the contribution of the third signal is large in X̂ pq , we cannot use a standard ICA algorithm at the 2nd stage. Table 1 shows the percentage
12 and M 23 . Two signals areof each source power extracted by MDC
DCdominant and the contributions of the third signal are small. Therefore, we can say that we can use ICA in the 2nd stage.
5.3.2 Separation results
Table 2 shows the experimental results we obtained for TR = 0
ms. The SIR and SDR values were averaged over three speakercombinations. The first row shows the results obtained by conventional method 1; the rows labeled ‘BMICA’ show the results
obtained by conventional method 2, and the rows labeled ‘DCICA’show the results obtained with our proposed method.
With conventional method 1, the SDR values were unsatisfactory, and a large musical noise was heard. In contrast, with our
proposed method (DCICA), we were able to obtain high SDR values without any serious deterioration in the separation performance
SIR. Compared with conventional method 2 (BMICA), although
SDR values were a bit degraded, we hear no musical noise with
DCICA. Some sound samples can be found at our web site [10].
Tables 3 and 4 show the results of reverberant tests for
TR = 130 ms. In the reverberant case, due to the decline ofsparseness, the performance with all methods was worse than when
TR = 0 ms. However, we were able to obtain higher SDR values
with DCICA than with conventional method 1 even in a reverberant environment without musical noise.
It should be noted that it remains difficult to separate signals at
the center position with any method.

We proposed utilizing a directivity pattern based continuous
mask and ICA for BSS when speech signals outnumber the sensors. Our method avoids discontinuous zero-padding, and therefore, can separate the signals with no musical noise. Moreover, our
new mask does not involve any heuristic parameters.
REFERENCES[1] S. Haykin, Unsupervised adaptive filtering, John Wiley &Sons, 2000.
[2] S. Rickard and O. Yilmaz, “On the W-Disjoint orthogonalityof speech,” Proc. ICASSP2002, vol.1, pp. 529-532, 2002.
[3] F. J. Theis, C. G. Puntonet, E. W. Lang, “A histogram-based
overcomplete ICA algorithm,” ICA2003, pp. 1071-1076, 2003.
[4] L. Vielva, D. Erdogmus, C. Pantaleon, I. Santamaria, J. Pereda
and J. C. Principe, “Underdetermined blind source separation
in a time-varying environment,” Proc. ICASSP2002, pp. 30493052, 2002.
[5] P. Bofill and M. Zibulevsky, “Blind separation of more sources
than mixtures using sparsity of their short-time Fourier transform,” Proc. ICA2000, pp. 87-92, 2000.
[6] S. Araki, S. Makino, A. Blin, R. Mukai and H. Sawada, “Blindseparation of more speech than sensors with less distortion by
combining sparseness and ICA,” Proc. IWAENC2003, pp. 271274, 2003.
[7] A. Blin, S. Araki and S. Makino, “Blind source separation
when speech signals outnumber sensors using a sparsenessmixing matrix combination,” Proc. IWAENC2003, pp. 211214, 2003.
[8] H. Sawada, R. Mukai, S. Araki and S. Makino, “A robust
and precise method for solving the permutation problem offrequency-domain blind source separation,” Proc. ICA2003,
pp. 505-510, 2003.
[9] K. Matsuoka and S. Nakashima, “A robust algorithm for blindseparation of convolutive mixture of sources,” Proc. ICA2003,
pp. 927-932, 2003.
[10] http://www.kecl.ntt.co.jp/icl/signal/araki/nbficademo.html

1994