➠➡
UNDERDETERMINED BLIND SEPARATION FOR SPEECH IN REAL ENVIRONMENTS
WITH SPARSENESS AND ICA
Shoko Araki †Shoji Makino †Audrey Blin †‡Ryo Mukai †Hiroshi Sawada ††NTT Communication Science Laboratories, NTT Corporation
2-4 Hikaridai, Seika-cho, Soraku-gun, Kyoto 619-0237, Japan
Email: shoko@cslab.kecl.ntt.co.jp‡
Université du Québec, INRS-EMT
800 de la Gauchetière Ouest, Suite 6900, Montréal, Québec, H5A 1K6, CanadasisN

Separated
signals
y1

x1
hji

xj

..... .....

0-7803-8484-9/04/$20.00 ©2004 IEEE

s2

Observed
signals..... .....

1. INTRODUCTION
Blind source separation (BSS) is an approach that estimatesoriginal source signals si (n) only from observations xj (n) without source or mixing process information.
In this paper, we consider the BSS of speech signals observedin a real environment, i.e., the BSS of convolutive mixtures ofspeech. Recently, many methods have been proposed to solve
the BSS problem of audio signals in real environments (e.g., [1]).
However, most of these methods consider the determined or overdetermined case, i.e., the number of sensors is equal to or greater than
the number of signals. In contrast, this paper focuses on the underdetermined BSS problem where source signals outnumber sensors.
It is our understanding that there are two approaches with which
to realize underdetermined BSS. Both approaches rely on the sparseness of source signals. One involves the clustering of time-frequency
points with binary masks [2], and the other is based on ML estimation, where the sources are estimated after mixing matrix estimation [3–5]. Since separation in a real environment has already been
tried with the former method, we decided to employ a binary mask
approach [2]. If the signals are sufficiently sparse, that is, most of
the samples of a signal are almost zero, we can assume that thesources rarely overlap. [2] uses this assumption and extracts eachsignal using a time-frequency binary mask. However, due to thesebinary masks, their method result in too much discontinuous zeropadding of the extracted signals, and so the extracted signals areseverely distorted.
To overcome this problem, we propose utilizing both a sparseness approach and independent component analysis (ICA). First,
using sparseness, we estimate the time points when only one source

Sourcesignalss1..... .....

ABSTRACTIn this paper, we propose a method for separating speech signals
when there are more signals than sensors. Several methods havealready been proposed for solving the underdetermined problem,and some of these utilize the sparseness of speech signals. Thesemethods employ binary masks to extract the signals, and therefore,
their extracted signals contain loud musical noise. To overcome
this problem, we propose combining a sparseness approach and independent component analysis (ICA). First, using sparseness, weestimate the time points when only one source is active. Then, we
remove this single source from the observations and apply ICA to
the remaining mixtures. Experimental results show that our proposed sparseness and ICA (SPICA) method can separate signals
with little distortion even in reverberant conditions of TR =130 and
200 ms.

xM

Mixingprocessy2
ykyN

SeparatingprocessFig. 1. Block diagram of underdetermined BSS. N > M .
is active. Then, we remove this single source from the observations and apply ICA to the remaining mixtures in order to separate the signals. This single source removal does not cause severe
zero-padding of the separated signals, therefore we can improvetheir sound quality. Experimental results show that our sparsenessand ICA (SPICA) method can separate signals with little distortioneven in real reverberant environments.
2. PROBLEM DESCRIPTIONIn real environments, N signals observed by M sensors are
N L
h (l)
modeled as convolutive mixtures xj (n) =i=1
l=1 jisi (n−l+1) (j = 1, · · · , M ), where si is the signal from a sourcei, xj is the signal observed by a sensor j, and hji is the L-taps impulse response from a source i to a sensor j (see Fig. 1). Our objective is to obtain separated signals yk (n) (k = 1, · · · , N ) using
only the information provided by observations xj (n). Here, we
consider the underdetermined case N > M . In this paper N = 3
and M = 2. Moreover, the sources are speech signals, i.e., thesources are assumed to be mutually independent and sufficiently
sparse in the time-frequency domain [2–6].
This paper employs a time-frequency domain approach because speech signals are more sparse in the time-frequency domain than in the time-domain [5, 6] and we can convert convolutive mixture problems into instantaneous mixture problems in eachfrequency. In the time-frequency domain, mixtures are modeled as(ω, m) = (ω) (ω, m), where (ω) is a 2×3 mixing matrix whose j-i component is a transfer function from a source i to asensor j, and (ω, m) = [S1 (ω, m), S2 (ω, m), S3 (ω, m)]T and(ω, m) = [X1 (ω, m), X2 (ω, m)]T denote short-time Fouriertransformed source and observed signals, respectively. ω is the frequency and m is the frame index. Moreover, we write short-time
Fourier transformed separated signals as (ω, m) = [Y1 (ω, m),
Y2 (ω, m), Y3 (ω, m)]T .

III - 881











ICASSP 2004

➡

➡
10
10

(a) TR = 0ms

1010@1969Hz

99

88

88

77

77

occurency

occurency

99

66
55
44

22

11

11

40

40

~
θ1

60

60

80

80

100

100

~
θ2

120

120

140

160

180

θ 000

140 160 180~ DOA (deg.)
θ3

Separation of S2 and S30[1st stage] X1 [2nd stage]

X2

^ 23

S2

S3

20

40

40

60

60

~
θ1

80

80

~
θ2

100

100

120

120

140

160

^ 12

^ 12

X
X =M12
ICA

Fig. 2. Example histogram. (a) TR = 0 ms and (b) TR = 200 ms.
A male-male-female combination with DFTsize T = 512.

23^ 23

Y3

X2

12

12^ 12Y1

Y =W X

Y2

Fig. 3. System setup
source from mixtures using the signals’ sparseness. By this removal, it is expected that their zero-padding of the extracted signals to be less trouble because we extract more time-frequency
points than the conventional approach. Moreover, because we can
expect the remaining mixtures to consist of only two signals, we
can apply ICA (e.g., [8]) to these remaining mixtures in the second
stage. Because these separated signals are not highly zero-padded,
we can expect less musical noise.
[1st stage] One source removal:
Instead of extracting each source as in conventional approaches,
we remove only one source from the mixtures with a binary maskpq
(ω, m) =MICA

θmin ≤ θ(ω, m) ≤ θmaxotherwise

1
0

(2)

by calculatingpq
ˆ pq (ω, m) = MICA
(ω, m)(ω, m).
(3)
In (2), θmin and θmax are extraction range parameters, and in (3),ˆ pq (ω, m) = [X̂ pq (ω, m), X̂ pq (ω, m)] are expected to be mix1
2
tures of S˜p and S˜q . For instance, if S˜1 can be removed from the23
observations with a mask MICA
we can use ICA to separate S˜2 andS˜3 in the next stage. In this case θmin and θmax in (2) can beθ˜1 < θth1 = θmin < θ˜2 , θmax = 180◦ (see Fig. 4), where θ˜1and θ˜2 are estimated in the same way as in the previous section.
Similarly, when S˜3 is to be removed from the observations with a12, θmin = 0◦ , θ˜2 < θth2 = θmax < θ˜3 .
mask MICA
Because our system has only two outputs, both removals shouldbe performed to obtain three separated signals (see Fig. 3).
[2nd stage] Separation of remaining sources by ICA:
pq
Because the remaining signals ˆ are expected to be mixpq
tures of two signals, we can use 2×2 ICA to separate ˆ . The
separation process can be formulated aspqpqpq(ω, m) =(ω) ˆ (ω, m),(4)
pq
where ˆ is the masked observed signal obtained by (3),pq
(ω, m) = [Yp (ω, m), Yq (ω, m)]T is the separated output sigpqpq(ω) represents a (2×2) separation matrix.
(ω)
nal, andis determined so that Yp (ω, m) and Yq (ω, m) become mutuallyindependent.
i (ω) +
 In this paper, the adaptive rule is i+1 (ω) =η diag Φ( ) H  − Φ( ) H i (ω), where Φ( ) =








j· ( )





φ(| |) · e, φ(x) = tanh(gx) and g = 100 [9]. To solve
the permutation problem of frequency domain ICA, we employed
the DOA and correlation approach [10], and for solving the scaling
problem of frequency domain ICA, we used the minimum distortion principle [11].


To overcome this musical noise problem, we propose using
both sparseness and ICA. Our method has two stages (Fig. 3). In
the first stage, unlike the conventional approach, we remove one

23

Y =W X^ 12~ DOA (deg.)θ3

4. PROPOSED METHOD: COMBINATION OFSPARSENESS AND ICA (SPICA)

X2[1st stage] X1 [2nd stage]180

140 160 180

3. CONVENTIONAL METHODS:SPARSENESS APPROACH
Standard ICA cannot be applied to underdetermined cases because it assumes that a mixing matrix is invertible. Several methods have been proposed (e.g., [2–5]) for solving an underdetermined BSS problem, and they utilized source sparseness.
If most of the samples of a signal are almost zero, we say that
this signal is sparse. When signals are sufficiently sparse, we canassume that the sources overlap at rare intervals. For a detailed
analysis of sparseness, see [7].
Some conventional methods use the sparseness assumption
and extract each signal using time-frequency binary masks. Because we can assume that sources do not overlap very often, we
can extract each source by selecting the time points at which there
is only one signal. One way of estimating such time points is to
use the level difference of the observations and the phase difference between the observations. In this paper, we utilize omnidirectional microphones, therefore we use the phase difference
1 (ω,m)between the observations X1 (ω, m) andϕ(ω, m) =  X
X2 (ω,m)
X2 (ω, m).
Using ϕ(ω, m), we can estimate the direction of arrival (DOA),
for each time point m by calculating θ(ω, m) = cos−1 ϕ(ω,m)c
ωdwhere c is the speed of sound and d is the microphone spacing.
When we plot this DOA θ(ω, m), we can see three peaks in thehistogram for each frequency. Let these peaks be θ˜1 , θ˜2 and θ˜3where θ˜1 ≤ θ˜2 ≤ θ˜3 (Fig. 2), and the signal from θ˜ξ be S˜ξ(ξ = 1, 2, 3).
We can extract each signal with a binary mask
1 θ˜ξ − ∆ ≤ θ(ω, m) ≤ θ˜ξ + ∆(1)Mξ (ω, m) =0 otherwise
by calculating Yξ (ω, m) = Mξ (ω, m)Xj (ω, m) where j=1 or 2.
Here, ∆ is an extraction range parameter: if ∆ is small the separation performance is good but the distortion is large, in contrast, if∆ is large the musical noise problem is reduced but the separation
performance deteriorates.
Although we can extract each signal using this binary mask(1), such extracted signals are discontinuously zero-padded by thebinary masks. Therefore, we hear considerable musical noise in
the extracted output.

^ 23

X =M23X
ICA

Y2

Separation of S1 and S2

θ
20

^ 23

X1

44
33

20

θ1

55

22

20

S1

@1969Hz

66

33

000
0

(b) TR = 200ms

III - 882

➡

➡

58
99
37
1
5
0
removed91
7
2

area 23

S1: 50area 12

θth2

2

~
θ

3

Loudspeakers(height : 1.35 m)

θ

1.1 m

7
1
8
5
2
2
32
91
91
97
60
4
S1: 3 S2: 47 S3: 50

S2: 48 S3: 2

removed135
2.25 marea 23
area 12

87
4
9

66
86
24
5
10
9
removedS1: 36

removedRoom height : 2.50 m

Fig. 5. Room for reverberant tests.
[%]

Fig. 4. Source power in each area (%). A male-male-female combination.
5. EXPERIMENTS
5.1. Experimental conditions
In our experiments, we utilized the set-up shown in Fig. 5. For
tests of TR = 0 ms, we simulated a recording using the mixingdmatrix Hji (ω) = exp (jωτji ), where τji = cj sin θi , dj is the
position of the j-th microphone, and θi is the direction of the i-thsource.
For the reverberant tests, we used speech data convolved with
impulse responses recorded in a real room whose reverberation
time was TR = 130 and 200 ms.
As the original speech, we used three sentences spoken by
three male and three female speakers. We investigated three combinations of speakers: male-male-female, male-male-male, andfemale-female-female.
The DFT frame size T was 512 and the frame shift was 256 ata sampling rate of 8 kHz. We constructed a histogram of θ(ω, m)for each frequency (see Fig. 2), and smoothed it slightly to findθ˜1 , θ˜2 and θ˜3 . The ∆ value in the conventional method’s binary
23(area
masks (1) was 15◦ in DOA. We used θth1 = θ˜2 −∆ for MICA
12˜23), and θth2 = θ2 + ∆ for MICA (area 12), where ∆ was also
15◦ . With this ∆ value the conventional method and our proposed
method provided compatible signal to noise ratio (SIR).
5.2. Performance measures
We used the signal to interference ratio (SIR) as a measure of
separation performance, and the signal to distortion ratio (SDR) asa measure of sound quality: 2
yis (n)SIRi = 10 log  n i
n

(



i=j

yisj (n))2x2ksi (n).
(xksi (n) − αyisi (n − D))2n

SDRi = 10 log nS1

Omni-directional microphones(height : 1.39 m)

5
3
13
13
6
11
43
80
89
86
44
7
S1: 10 S2: 58 S3: 32

S2: 56 S3: 8

45

1.75 m

[%]

S2

90

4 cm

(b) TR=200 ms
S1
S2
S3

S3

1.80 m180
3.55 m

S1
S2
S3

1

M3

o

0

M2
∆ ∆
θth1 ~
θo~θ

4.45 m23

MICA (area 23)o(a) TR=0 ms
12
MICA (area 12)
M1

(5)
(6)

The permutation is solved before calculating SIR and SDR, i.e.,
yi is the estimation of si , and yisj is the output of the whole separating system at yi when only sj is active and sk (k = j) is silent,and xksj is the observation obtained by microphone k when only
sj is active. α and D are parameters to compensate for the amplitude and phase difference between xksi and yisi . To evaluate

the conventional method (sparseness only method), we calculated
SIR and SDR using the measurements from both microphones, andadopted the better value.
5.3. Experimental results
5.3.1. Sparseness assessments
The histograms we used to design the binary masks are shownin Fig. 2. Because of the sparseness property of speech signals,
the signals are well localized and the histogram has sharp peaks
for TR = 0 ms [Fig. 2 (a)]. However, in a reverberant case thepeaks are not so sharp [Fig. 2 (b)]. This shows that signals overlap
each other in a reverberant case and it becomes more difficult to
use the sparseness assumption than when TR = 0 ms.
We can also see this overlap in Fig. 4, which shows the power
content by percentage of each signal when ∆ = 15◦ . When
TR = 0 ms [Fig. 4 (a)], S1 , S2 and S3 are dominant in the areas of M1 , M2 and M3 , respectively. When TR = 200 ms [Fig. 4(b)], however, the other signals’ overlaps increase in each area. It
is difficult for the sparseness assumption to held in a reverberantcase.
The degree of sparseness affects the performance of the 2ndstage of our method. Figure 4 also shows the percentage of each
12
23
and MICA. The contributions ofsource power in the areas of MICA
the third signal are small and two signals are dominant even whenTR =200 ms. Therefore, we can say that we can use ICA in the 2ndstage of our method.
5.3.2. Effect of one source removal
Table 1 shows the signal power eliminated by the zero-paddingnx1si (n)2 −



n

n

x1si (n)2ŝi (n)2

caused by binary masks, where sˆi (n) =IDFT[M (ω, m)X1si (ω, m)]. In the sparseness only case [i.e.,M (ω, m) = Mi (ω, m)], a large part of the signal power was eliminated by the binary mask. By contrast, with our proposed method,pq(ω, m) was inferior. This result
signal power eliminated by MICA
convinces us that the adverse effect of zero-padding was mitigatedby using our method.
5.3.3. Separation results
Table 2 shows the experimental results we obtained for TR = 0 ms.
The first row shows the results obtained solely using the sparseness, and the second and third rows show the results obtained with
our SPICA method. With sparseness only, the SDR values were
unsatisfactory, and a large musical noise was heard. In contrast,with our proposed method, SPICA, we were able to obtain highIII - 883

➡

➠
Table 1. Power lost by binary masks (in %). (a) TR =0 ms, (b)TR =130 ms, (c) TR =200 ms. A male-male-female combination,
12
23
MICA
MICA
mask
M1 M2 M3(a) output Y1
Y2
Y3 Y1 Y2 Y2 Y3[%]
15 8.7 15 1.4 3.9 4.9 0.9
12
23
MICA
MICA
mask
M1 M2 M3(b) output Y1
Y2
Y3 Y1 Y2 Y2 Y3[%]
39 6.0 28 1.6 2.5 3.6 4.4
12
23
MICA
MICA
mask
M1 M2 M3(c) output Y1
Y2
Y3 Y1 Y2 Y2 Y3[%]
48 9.2 37 4.1 4.5 4.7 8.8
SDR values without any serious deterioration in the separation performance SIR and the musical noise was reduced.
Moreover, Tables 3 and 4 show the results of reverberant tests
when TR = 130 and TR = 200 ms, respectively. In reverberantcases, due to the decline of sparseness, the performance with bothmethods was worse than when TR = 0 ms. However, we wereable to obtain higher SDR values with SPICA than with the conventional method even in reverberant environments. Some soundsamples can be found at our web site [12].
It should be noted that it is not easy to separate signals at thecenter position by either method.
6. CONCLUSION
We proposed combining a sparseness approach and ICA (SPICA)
for BSS when speech signals outnumber sensors. Our method
avoids excessive zero-padding, and therefore, can separate the signals with little distortion in reverberant environments of TR =130
and 200 ms.

[9] H. Sawada, R. Mukai, S. Araki and S. Makino, “Polar coordinate based nonlinear function for frequency domain blindsource separation,” Proc. ICASSP2002, pp. 1001-1004, 2002.
[10] H. Sawada, R. Mukai, S. Araki and S. Makino, “A robust
and precise method for solving the permutation problem offrequency-domain blind source separation,” Proc. ICA2003
pp. 505-510, 2003.
[11] K. Matsuoka and S. Nakashima, “A robust algorithm
for blind separation of convolutive mixture of sources,”
Proc. ICA2003, pp. 927-932, 2003.
[12] http://www.kecl.ntt.co.jp/icl/signal/araki/underdeter
minedBSSdemo.html
Table 2. Results of TR =0 ms simulations. ‘Conv.’: with conventional method, ‘area 12’ and ‘area 23’: with SPICA. ∆=15◦ .
male-male-female[dB]
SIR1 SIR2 SIR3 SDR1 SDR2 SDR3
Conv.
18.4 11.6 17.3
9.0
11.5
9.4
area 12 15.0
8.4
18.7
14.4
area 23
7.9
14.6
13.9
18.5
male-male-male
SIR1 SIR2 SIR3 SDR1 SDR2 SDR3
Conv.
14.4
5.5
17.4
5.5
9.2
6.0
area 12
9.2
2.5
16.8
14.7
area 23
3.7
11.3
10.5
12.4
female-female-female
SIR1 SIR2 SIR3 SDR1 SDR2 SDR3
Conv.
21.3
9.7
20.5
9.3
13.9
9.6
area 12 13.6
6.8
18.9
16.6
area 23
6.8
13.2
16.3
21.3

7. REFERENCES[1] A. Hyvärinen, J. Karhunen and E. Oja, Independent Component Analysis, John Wiley & Sons, 2001.
[2] S. Rickard and O. Yilmaz, “On the W-Disjoint orthogonalityof speech,” Proc. ICASSP2002, vol.1, pp. 529-532, 2002.
[3] F. J. Theis, C. G. Puntonet, E. W. Lang, “A histogrambased overcomplete ICA algorithm,” ICA2003, pp. 10711076, 2003.
[4] L. Vielva, D. Erdogmus, C. Pantaleon, I. Santamaria, J. Pereda
and J. C. Principe, “Underdetermined blind source separation
in a time-varying environment,” Proc. ICASSP2002, pp. 30493052, 2002.
[5] P. Bofill and M. Zibulevsky, “Blind separation of more sources
than mixtures using sparsity of their short-time Fourier transform,” Proc. ICA2000, pp. 87-92, 2000.
[6] M. Aoki, M. Okamoto, S. Aoki, H. Matsui, T. Sakurai and
Y. Kaneda, “Sound source segregation based on estimating incident angle of each frequency component of input signals acquired by multiple microphones,” Acoust. Sci. & Tech, vol. 22,
no. 2, pp. 149-157, 2001.
[7] A. Blin, S. Araki and S. Makino, “Blind source separation
when speech signals outnumber sensors using a sparsenessmixing matrix combination,” Proc. IWAENC2003, pp. 211214, 2003.
[8] S. Araki, R. Mukai, S. Makino, T. Nishikawa and
H. Saruwatari, “The fundamental limitation of frequency domain blind source separation for convolutive mixtures ofspeech,” IEEE Trans. on SAP, vol. 11, no. 2, pp. 109-116,
2003.

III - 884

Table 3. Results of reverberant tests. TR =130 ms.
male-male-female[dB]
SIR1 SIR2 SIR3 SDR1 SDR2 SDR3
Conv.
10.4
6.8
10.7
4.8
13.3
6.2
area 12
9.5
6.1
8.8
15.9
area 23
6.4
8.9
14.4
9.5
male-male-male
SIR1 SIR2 SIR3 SDR1 SDR2 SDR3
Conv.
10.6
2.9
8.6
3.7
12.5
4.4
area 12
7.7
2.5
5.9
14.3
area 23
2.2
8.9
12.7
7.9
female-female-female
SIR1 SIR2 SIR3 SDR1 SDR2 SDR3
Conv.
16.0
9.2
13.7
6.4
16.0
6.9
area 12 12.2
7.9
8.6
17.6
area 23
7.9
9.7
16.5
10.6
Table 4. Results of reverberant tests. TR =200 ms.
male-male-female[dB]
SIR1 SIR2 SIR3 SDR1 SDR2 SDR3
Conv.
8.3
5.9
8.6
3.4
11.3
4.9
area 12
8.1
5.4
6.0
12.6
area 23
5.5
7.5
13.6
6.5
male-male-male
SIR1 SIR2 SIR3 SDR1 SDR2 SDR3
Conv.
7.7
1.9
5.4
1.1
10.6
4.0
area 12
6.6
2.3
4.2
11.6
area 23
1.7
2.4
10.5
5.4
female-female-female
SIR1 SIR2 SIR3 SDR1 SDR2 SDR3
Conv.
10.2
6.0
8.2
3.3
13.5
4.8
area 12
9.6
5.5
5.8
14.0
area 23
5.6
9.0
13.6
7.1