Underdetermined Blind Separation of Convolutive Mixtures of Speech
by Combining Time-frequency Masks and ICA
Shoko Araki †Shoji Makino †Audrey Blin †‡Ryo Mukai †Hiroshi Sawada ††NTT Communication Science Laboratories, NTT Corporation
2-4 Hikaridai, Seika-cho, Soraku-gun, Kyoto 619-0237, Japan
Email: shoko@cslab.kecl.ntt.co.jp‡
Université du Québec, INRS-EMT
800 de la Gauchetière Ouest, Suite 6900, Montréal, Québec, H5A 1K6, Canada

Blind source separation (BSS) is an approach that estimatesoriginal source signals si (n) only from observations xj (n) without source or mixing process information.
In this paper, we consider the BSS of speech signals observedin a real environment, i.e., the BSS of convolutive mixtures ofspeech. Recently, many methods have been proposed to solve the
BSS problem of convolutive mixtures (e.g., [1]). However, mostof these methods consider the determined or overdetermined case,i.e., the number of sensors is equal to or greater than the numberof signals. In contrast, we focus on the underdetermined BSS
problem where source signals outnumber sensors.
It is our understanding that there are two approaches to realize
the underdetermined BSS. Both approaches rely on the sparsenessof source signals. One is the clustering of time-frequency points
with binary masks [2], and the other is based on ML estimation,
where the sources are estimated after the mixing matrix estimation [3–5]. Because separation in a real environment has been
tried with the former method, we have decided to watch a binarymasks approach [2]. If the signals are sufficiently sparse, thatis, most of the samples of a signal are almost zero, we can assume that the sources rarely overlap. [2] uses this assumption andextracts each signal using a time-frequency binary mask. However, due to these binary masks, their method results in too muchdiscontinuous zero-padding of the extracted signals, and so theextracted signals are severely distorted.
To overcome this problem, we propose utilizing both a binary

Mo4.D.1s2
sisN

Separated
signalsy1

x1
hji

Mixingprocessxj..... .....

1. Introduction

Observed
signals..... .....

We propose a method for separating speech signals when sources
outnumber sensors. Several methods have already been proposed
for solving the underdetermined blind separation problem for convolutive mixtures, and they utilize the sparseness of speech signals. Some methods employ binary masks to extract the signals,
and therefore, their extracted signals contain loud musical noise.
To overcome this problem, we propose utilizing both a binary
mask and independent component analysis (ICA). First, using
sparseness, we estimate the time points when only one source isactive. Then, we remove this single source from the observations
and apply ICA to the remaining mixtures. Experimental results
show that our proposed method can separate signals with little
distortion even in reverberant conditions of TR =130 and 200 ms.

Sourcesignalss1..... .....

Abstract

xM

y2
yk
yN

Separatingprocess

Figure 1: Block diagram of underdetermined BSS. N > M .
mask and independent component analysis (ICA). First, using abinary mask, we estimate the time points when only one sourceis active. Then, we remove this single source from the observations and apply ICA to the remaining mixtures in order to separate the signals. This single source removal does not cause severe
zero-padding of the separated signals, therefore we can improvetheir sound quality. Experimental results show that our method
can separate signals with little distortion even in real reverberant
environments of TR =130 and 200 ms.

2. Problem description
In real environments, N signals observed
M sensors are
Pby
N PP
h (k)
modeled as convolutive mixtures xj (n) =k=1 jii=1
si (n − k + 1) (j = 1, · · · , M ), where si is the signal from asource i, xj is the signal observed by a sensor j, and hji is the P taps impulse response from a source i to a sensor j (see Fig. 1).
Here, we consider the underdetermined case N > M . In thispaper N = 3 and M = 2. Moreover, in this paper, the sources
are speech signals, i.e., the sources are assumed to be mutuallyindependent and sufficiently sparse in the time-frequency domain.
This paper employs a time-frequency domain approach because speech signals are more sparse in the time-frequency domain than in the time-domain [5] and convolutive mixture problems can be converted into instantaneous mixture problems ineach frequency. In the time-frequency domain, mixtures are modeled as (ω, m) = (ω) (ω, m), where (ω) is a 2×3 mixing matrix whose j-i component is a transfer function from asource i to a sensor j, (ω, m) = [S1 (ω, m), S2 (ω, m), S3 (ω, m)]T ,(ω, m) = [X1 (ω, m), X2 (ω, m)]T and (ω, m) = [Y1 (ω, m),

I - 321











10
10

@1969Hz

99

88

88

77

77

66
55
44

(b) TR = 200ms@1969Hz

55
44

3

3

22

22

S3

3

11

11

20

20

40

40

60

60

80

80

~
θ1

100

100

120

120

140

160

θ 000
180

140 160 180~ DOA (deg.)
θ3

~
θ2

θ1

0

20

20

40

40

60

60

80

80

~
θ1

100

100

~
θ2

120

120

140

160

~ DOA (deg.)θ3

3. Conventional methods:
with binary masks onlyStandard ICA cannot be applied to underdetermined cases because it assumes that a mixing matrix is invertible. Several methods have been proposed (e.g., [2–5]) for solving an underdetermined BSS problem, and they utilized source sparseness.
If most of the samples of a signal are almost zero, we say that
this signal is sparse. When signals are sufficiently sparse, we canassume that the sources overlap at rare intervals. For a detailed
analysis of sparseness, see [6].
Some conventional methods use the sparseness assumption
and extract each signal using time-frequency binary masks. Because we can assume that sources do not overlap very often, we
can extract each source by selecting the time points at which there
is only one signal. One way of estimating such time points is to
use the level difference of the observations and the phase difference between the observations. In this paper, we utilize omnidirectional microphones, therefore we use the phase difference
1 (ω,m)ϕ(ω, m) = 6 X
between the observations.
X2 (ω,m)Using ϕ(ω, m), we estimate the direction of arrival (DOA)
for each time point m by calculating θ(ω, m) = cos−1 ϕ(ω,m)c,
ωdwhere c is the speed of sound and d is the microphone spacing,and make a histogram of the DOA θ(ω, m). Each peak corresponds to each source in the histogram for each frequency. Let
these peaks be θ˜1 , θ˜2 and θ˜3 where θ˜1 ≤ θ˜2 ≤ θ˜3 (Fig. 2), andthe signal from θ˜ξ be S˜ξ (ξ = 1, 2, 3).
We can extract each signal with a binary mask
1
0

X2

^ 23

θ˜ξ − ∆ ≤ θ(ω, m) ≤ θ˜ξ + ∆otherwise(1)

by calculating Yξ (ω, m) = Mξ (ω, m)Xj (ω, m) where j=1 or 2.
Here, ∆ is an extraction range parameter: if ∆ is small the separation performance is good but the distortion is large, in contrast, if∆ is large the musical noise problem is reduced but the separation
performance deteriorates.
Although we can extract each signal using this binary mask(1), such extracted signals are discontinuously zero-padded by the

X =M23
X
ICA

^ 23

X2

23

23^ 23

Y =W X

Y2
Y3

^ 12

^ 12

X =M12
X
ICA

^ 12

X2

12

12^ 12

Y =W X

Y1
Y2

Figure 3: System setup
binary masks, and therefore, we hear musical noise in the extracted output.

4. Proposed Method:
Combination of binary mask and ICA[1st stage] X1 [2nd stage][1st stage] X1 [2nd stage]180

140 160 180

Y2 (ω, m),Y3 (ω, m)]T show Fourier transformed source, observed
and separated signals, respectively. ω is the frequency and m is
the frame index.
Our objective is to estimate separated signals (ω, m) using
only the information provided by observations (ω, m).

^ 23

X1

Separation of S1 and S2

θ

Figure 2: Example histogram. (a) TR = 0 ms and (b) TR = 200 ms.
A male-male-female combination with DFTsize T = 512.

Mξ (ω, m) =Separation of S2 and S3

S2

66

3

000
0

S1

10
10

(a) TR = 0ms

occurency

occurency

99

To overcome this musical noise problem, we propose using
both a binary mask and ICA. Our method has two stages (Fig. 3).
In the first stage, unlike the conventional approach, we remove
one source from mixtures using the signals’ sparseness. By thisremoval, it is expected that their zero-padding of the extracted
signals to be less trouble because we extract more time-frequency
points than the conventional approach. Moreover, because we can
expect the remaining mixtures to consist of only two signals, we
can apply a standard ICA to these remaining mixtures in the second stage. Because these separated signals are not highly zeropadded, we can expect less musical noise.
[1st stage] One source removal:
Instead of extracting each source as in conventional approaches,
we remove only one source from the mixtures with a binary mask
1 θmin ≤ θ(ω, m) ≤ θmax
pq
(2)
MICA (ω, m) =0 otherwise
by calculatingpq
ˆ pq (ω, m) = MICA
(ω, m) (ω, m).
(3)
In (2), θmin and θmax are extraction range parameters, and in
pq(3), ˆ (ω, m) = [X̂1pq (ω, m), X̂2pq (ω, m)] are expected to be
mixtures of S˜p and S˜q . For instance, if S˜1 can be removed from23
the observations with a mask MICA
we can use ICA to separate S˜2and S˜3 in the next stage. In this case θmin and θmax in (2) canbe θ˜1 < θth1 = θmin < θ˜2 , θmax = 180◦ (see Fig. 4), where θ˜1and θ˜2 are estimated in the same way as in the previous section.
Similarly, when S˜3 is to be removed from the observations with a12mask MICA, θmin = 0◦ , θ˜2 < θth2 = θmax < θ˜3 .
Because our system has only two outputs, both removals shouldbe performed to obtain three separated signals (see Fig. 3).
[2nd stage] Separation of remaining sources by ICA:
pq
Because the remaining signals ˆ are expected to be mixpq
tures of two signals, we can use 2×2 ICA to separate ˆ . The
separation process can be formulated aspqpqpq(ω, m) =(ω) ˆ (ω, m),(4)
pq
where ˆ is the masked observed signal obtained by (3),pq
(ω, m) = [Yp (ω, m), Yq (ω, m)]T is the separated output
pq
signal, and pq (ω) represents a (2×2) separation matrix.
(ω)
is determined so that Yp (ω, m) and Yq (ω, m) become mutuallyindependent.

I - 322(a) TR=0 ms
12
MICA (area 12)
M1

area 23

S1: 50area 12

θ

~
θ

1.1 m

180

3

7
1
8
5
2
2
32
91
91
97
60
4
S1: 3 S2: 47 S3: 50

S2: 48 S3: 2

135
2.25 marea 23
area 12

87
4
9

66
86
24
5
10
9
removedS1: 36Omni-directional microphones(height : 1.39 m)
Room height : 2.50 m

Figure 5: Room for reverberant tests.

removedx2ksi (n).
(xksi (n) − αyisi (n − D))2n

SDRi = 10 log P[%]

Figure 4: Source power in each area (%). A male-male-female
combination.

5. Experiments
5.1. Experimental conditions
In our experiments, we utilized the set-up shown in Fig. 5. For
tests of TR = 0 ms, we simulated a recording using the mixingdmatrix Hji (ω) = exp (jωτji ), where τji = cj sin θi , dj is the
position of the j-th microphone, and θi is the direction of the i-thsource.
For the reverberant tests, we used speech data convolved with
impulse responses recorded in a real room whose reverberation
time was TR = 130 and 200 ms.
As the original speech, we used three Japanese sentences spoken by three male and three female speakers. We investigatedthree combinations of speakers: male-male-female, male-malemale, and female-female-female.
The DFT frame size T was 512 and the frame shift was 256
at a sampling rate of 8 kHz. The ∆ value in the conventional
method’s binary masks (1) was 15◦ in DOA. We used θth1 =2312θ˜2 − ∆ for MICA
(area 23), and θth2 = θ˜2 + ∆ for MICA
(area12), where ∆ was also 15◦ . With this ∆ value the conventional
method and our proposed method provided compatible signal tonoise ratio (SIR).
The adaptation rule of ICA utilized in our experiments wasHi − hΦ( ) H i
i+1 (ω) =i (ω) + η diag hΦ( )






6

j· ( ), φ(x) = tanh(gx)·i (ω), where Φ( ) = φ(| |) · e
and g = 100 [7]. To solve the permutation problem of frequency
domain ICA, we employed the DOA and correlation approach [8],and for solving the scaling problem of frequency domain ICA, weused the minimum distortion principle [9].

n

(

i6=j

yisj (n))2

(5)

n

(6)

5.3. Experimental results
5.3.1. Sparseness assessments
The histograms we used to design the binary masks are shownin Fig. 2. Because of the sparseness property of speech signals,
the signals are well localized and the histogram has sharp peaks
for TR = 0 ms [Fig. 2 (a)]. However, in a reverberant case thepeaks are not so sharp [Fig. 2 (b)]. This shows that signals overlap
each other in a reverberant case and it becomes more difficult to
use the sparseness assumption than when TR = 0 ms.
We can also see this overlap in Fig. 4, which shows the power
content by percentage of each signal when ∆ = 15◦ . When
TR = 0 ms [Fig. 4 (a)], S1 , S2 and S3 are dominant in the areas of M1 , M2 and M3 , respectively. When TR = 200 ms [Fig. 4(b)], however, the other signals’ overlaps increase in each area. It
is difficult for the sparseness assumption to held in a reverberantcase.
The degree of sparseness affects the performance of the 2ndstage of our method. Figure 4 also shows the percentage of each
1223
. The contributions ofand MICA
source power in the areas of MICA
the third signal are small and two signals are dominant even whenTR =200 ms. Therefore, we can say that we can use ICA in the
2nd stage of our method.
5.3.2. Effect of one source removal
power eliminated by the zero-padding
PTable 1 shows
Pthe signal
x1si (n)2 −ŝi (n)2n P
ncaused by binary masks, where sˆi (n) =2n

We used the signal to interference ratio (SIR) as a measure of
separation performance, and the signal to distortion ratio (SDR)
as a measure of sound quality:P

The permutation is solved before calculating SIR and SDR, i.e.,
yi is the estimation of si , and yisj is the output of the wholeseparating system at yi when only sj is active, and xksj is theobservation obtained by microphone k when only sj is active. α
and D are parameters to compensate for the amplitude and phasedifference between xksi and yisi .

5.2. Performance measures

P 2
yis (n)SIRi = 10 log P Pn i

S1

[%]

5
3
13
13
6
11
43
80
89
86
44
7
S1: 10 S2: 58 S3: 32

45

1.75 mremovedS2: 56 S3: 8

S2

90

4 cm

(b) TR=200 ms
S1
S2
S3

S3

1.80 m3.55 m

58
99
37
1
5
0
removed

91
7
2

θth2

Loudspeakers(height : 1.35 m)

o

S1
S2
S3

1

M3

o

0

M2
∆ ∆
θth1 ~
θ2o~θ

4.45 m23

MICA (area 23)

x1si (n)IDFT[M (ω, m)X1si (ω, m)]. In the sparseness only case [i.e.,M (ω, m) = Mi (ω, m)], a large part of the signal power was
eliminated by the binary mask. By contrast, with our proposedpq
method, signal power eliminated by MICA
(ω, m) was inferior. This
result convinces us that the adverse effect of zero-padding was
mitigated by using our method.

I - 323

Table 1: Power lost by binary masks (in %). (a) TR =0 ms, (b)TR =130 ms, (c) TR =200 ms. A male-male-female combination,
12
23
MICA
MICA
mask
M1 M2 M3(a) output Y1
Y2
Y3 Y1 Y2 Y2 Y3[%]
15 8.7 15 1.4 3.9 4.9 0.9
12
23
MICA
MICA
mask
M1 M2 M3(b) output Y1
Y2
Y3 Y1 Y2 Y2 Y3[%]
39 6.0 28 1.6 2.5 3.6 4.4
12
23
MICA
MICA
mask
M1 M2 M3(c) output Y1
Y2
Y3 Y1 Y2 Y2 Y3[%]
48 9.2 37 4.1 4.5 4.7 8.8
5.3.3. Separation results
Table 2 shows the experimental results we obtained for TR = 0
ms. The first row shows the results obtained solely using binarymasks, and the second and third rows show the results obtainedwith our proposed method. With only a binary mask, the SDR
values were unsatisfactory, and a large musical noise was heard.
In contrast, with our proposed method, we were able to obtainhigh SDR values without destroying the separation performance
SIR and the musical noise was reduced.
Moreover, Tables 3 and 4 show the results of reverberant tests
when TR = 130 and TR = 200 ms, respectively. In reverberantcases, due to the decline of sparseness, the performance with bothmethods was worse than when TR = 0 ms. However, we wereable to obtain higher SDR values with our proposed method thanwith the conventional method even in reverberant environments.
Some sound samples can be found at our web site [10].

[8] H. Sawada, R. Mukai, S. Araki and S. Makino, “A robust
and precise method for solving the permutation problem offrequency-domain blind source separation,” 4th International
Symposium on ICA and BSS (ICA2003), pp. 505-510, 2003.
[9] K. Matsuoka and S. Nakashima, “A robust algorithm for blindseparation of convolutive mixture of sources,” 4th International Symposium on ICA and BSS (ICA2003), pp. 927-932,
2003.
[10] http://www.kecl.ntt.co.jp/icl/signal/araki/underdeter
minedBSSdemo.html
Table 2: Results of TR =0 ms simulations. ‘Conv.’: with conventional method, ‘area 12’ and ‘area 23’: with our method. ∆=15◦ .
male-male-female[dB]
SIR1 SIR2 SIR3 SDR1 SDR2 SDR3
Conv.
18.4 11.6 17.3
9.0
11.5
9.4
area 12 15.0
8.4
18.7
14.4
area 23
7.9
14.6
13.9
18.5
male-male-male
SIR1 SIR2 SIR3 SDR1 SDR2 SDR3
Conv.
14.4
5.5
17.4
5.5
9.2
6.0
area 12
9.2
2.5
16.8
14.7
area 23
3.7
11.3
10.5
12.4
female-female-female
SIR1 SIR2 SIR3 SDR1 SDR2 SDR3
Conv.
21.3
9.7
20.5
9.3
13.9
9.6
area 12 13.6
6.8
18.9
16.6
area 23
6.8
13.2
16.3
21.3

6. Conclusion
We proposed utilizing both a binary mask and ICA for BSS
when speech signals outnumber sensors. Our method avoids excessive zero-padding, and therefore, can separate the signals withlittle distortion in reverberant environments of TR =130 and 200 ms.

7. References[1] A. Hyvärinen, J. Karhunen and E. Oja, Independent Component Analysis, John Wiley & Sons, 2001.
[2] S. Rickard and O. Yilmaz, “On the W-Disjoint orthogonalityof speech,” Proc. ICASSP2002, vol.1, pp. 529-532, 2002.
[3] F. J. Theis, C. G. Puntonet, E. W. Lang, “A histogram-based
overcomplete ICA algorithm,” 4th International Symposium
on ICA and BSS (ICA2003), pp. 1071-1076, 2003.
[4] L. Vielva, D. Erdogmus, C. Pantaleon, I. Santamaria,
J. Pereda and J. C. Principe, “Underdetermined blindsource separation in a time-varying environment,”
Proc. ICASSP2002, pp. 3049-3052, 2002.
[5] P. Bofill and M. Zibulevsky, “Blind separation of more
sources than mixtures using sparsity of their short-time
Fourier transform,” 2nd International Workshop on ICA and
BSS (ICA2000), pp. 87-92, 2000.
[6] A. Blin, S. Araki and S. Makino, “Blind source separation
when speech signals outnumber sensors using a sparsenessmixing matrix combination,” Proc. IWAENC2003, pp. 211214, 2003.
[7] H. Sawada, R. Mukai, S. Araki and S. Makino, “Polar
coordinate based nonlinear function for frequency domain
blind source separation,” Proc. ICASSP2002, pp. 1001-1004,
2002.

I - 324

Table 3: Results of reverberant tests. TR =130 ms.
male-male-female[dB]
SIR1 SIR2 SIR3 SDR1 SDR2 SDR3
Conv.
10.4
6.8
10.7
4.8
13.3
6.2
area 12
9.5
6.1
8.8
15.9
area 23
6.4
8.9
14.4
9.5
male-male-male
SIR1 SIR2 SIR3 SDR1 SDR2 SDR3
Conv.
10.6
2.9
8.6
3.7
12.5
4.4
area 12
7.7
2.5
5.9
14.3
area 23
2.2
8.9
12.7
7.9
female-female-female
SIR1 SIR2 SIR3 SDR1 SDR2 SDR3
Conv.
16.0
9.2
13.7
6.4
16.0
6.9
area 12 12.2
7.9
8.6
17.6
area 23
7.9
9.7
16.5
10.6
Table 4: Results of reverberant tests. TR =200 ms.
male-male-female[dB]
SIR1 SIR2 SIR3 SDR1 SDR2 SDR3
Conv.
8.3
5.9
8.6
3.4
11.3
4.9
area 12
8.1
5.4
6.0
12.6
area 23
5.5
7.5
13.6
6.5
male-male-male
SIR1 SIR2 SIR3 SDR1 SDR2 SDR3
Conv.
7.7
1.9
5.4
1.1
10.6
4.0
area 12
6.6
2.3
4.2
11.6
area 23
1.7
2.4
10.5
5.4
female-female-female
SIR1 SIR2 SIR3 SDR1 SDR2 SDR3
Conv.
10.2
6.0
8.2
3.3
13.5
4.8
area 12
9.6
5.5
5.8
14.0
area 23
5.6
9.0
13.6
7.1