International Workshop on Acoustic Echo and Noise Control (IWAENC2003), Sept. 2003, Kyoto, Japan

Blind Separation of More Speech than Sensors with Less Distortion
by Combining Sparseness and ICA
Shoko Araki

Shoji Makino

Audrey Blin

Ryo Mukai

Hiroshi Sawada

NTT Communication Science Laboratories, NTT Corporation
2-4 Hikaridai, Seika-cho, Soraku-gun, Kyoto 619-0237, Japan
Email: shoko@cslab.kecl.ntt.co.jp
Abstract

271

x1
hji

Mixingprocess

xj

Separated
signalsy1..... .....

Blind source separation (BSS) is an approach that estimates original source signals s i (n) using only informationon the mixed signals x j (n) observed in each input channel.
This technique can be used for noise robust speech recognition and high-quality hearing aid systems. It may also
become a clue to auditory scene analysis.
In this paper, we consider the BSS of speech signals observed in a real environment, i.e., the BSS of convolutive
mixtures of speech. We focus particularly on the underdetermined BSS problem, that is, the case of the number ofsource signals outnumbers the number of sensors.
Several methods have already been proposed for solving the underdetermined problem and some of these utilize
the sparseness of speech signals [1, 2, 3]. If the signals aresparse enough, that is, most of the samples of a signal arealmost zero, we can assume that the sources rarely overlap. Sparseness approaches use this assumption and extract
each signal using time-frequency binary masks. However,
due to these binary masks, their methods result in too muchzero-padding to the extracted signals, and so the extracted
speeches are severely distorted and sound unnatural.
To overcome this problem, i.e., to reduce the distortionof the extracted signals, we propose combining a sparse-sN

Observed
signals..... .....

1. Introductionsi..... .....

We propose a method for separating speech signals withlittle distortion when the signals outnumber the sensors.
Several methods have already been proposed for solving
the underdetermined problem, and some of these utilize
the sparseness of speech signals. These methods employ
binary masks that extract a signal at time points where thenumber of active sources is estimated to be only one. However, these methods result in an unexpected excess of zeropadding and so the extracted speeches are severely distorted and have loud musical noise. In this paper, we propose combining a sparseness approach and independent component analysis (ICA). First, using sparseness, we estimate
the time points when only one source is active. Then, we
remove this single source from the observations and apply
ICA to the remaining mixtures. Experimental results showthat our proposed sparseness and ICA (SPICA) method canseparate signals with little distortion even in a reverberant
condition.

Sourcesignals
s1
s2

xM

y2
yk
yN

Separatingprocess

Figure 1: Block diagram of underdetermined BSS. N > M .
ness approach and independent component analysis (ICA).
First, using sparseness, we estimate the time points when
only one source is active. Then, instead of extracting only
one signal from the observations, we remove this single
source from the observations and apply ICA to the remaining mixtures in order to separate the signals. This removal
does not cause severe zero-padding to the separated signals,
therefore we can improve the sound quality of the separated
signals. Experimental results show that our sparseness andICA (SPICA) method can separate signals with little distortion even in an echoic environment.

2. Problem description
In real environments, N signals observed by M sensors
PN PP
are modeled as convolutive mixtures x j (n) = i=1 k=1
hji (k)si (n−k +1) (j = 1, · · · , M ), where s i is the signal
from a source i, x j is the signal observed by a sensor j, andhji is the P -taps impulse response from a source i to a sensor j (see Fig. 1). Here, we consider the underdeterminedcase N > M . In this paper N = 3 and M = 2. Sources
are assumed to be mutually independent and sparse.
This paper employs a time-frequency domain approach
because speech signals are more sparse in the time-frequency
domain [3, 4] and we can convert convolutive mixture problems into instantaneous mixture problems in each frequency.
In the time-frequency domain, mixtures are modeled asX(ω, m) = H(ω)S(ω, m), where H(ω) is a 2×3 mixing matrix whose j-i component is a transfer function froma source i to a sensor j and (ω, m) = [S 1 (ω, m), S2 (ω, m),
S3 (ω, m)]T , (ω, m) = [X1 (ω, m), X2 (ω, m)]T and (ω, m) =[Y1 (ω, m), Y2 (ω, m), Y3 (ω, m)]T show a Fourier transformed




12

40

@1797Hz(a)@1797Hz

(b)
35

10

30

8

frequency

frequency

source, observed and separated signals, respectively. ω isthe frequency and m is the frame index.
Our objective is to estimate separated signals Y (ω, m)using only the information provided by observations X(ω, m).
In this paper, the sources are speech signals, i.e., the sources
are sufficiently sparse in the time-frequency domain [1][4].

6

4

25

20

15

10

2
5

0

3. Conventional methods with sparseness

20

20

40

40

60

60

80

80

100

120

140

160

180

θ

0

100 120 140 160 180

200

0

0

20

40

60

0

20

40

60

DOA
smoothing

The standard ICA cannot be applied in underdeterminedcases because it assumes that a mixing matrix is invertible.
Several separation methods employing source sparsenesshave been proposed for use when there are more sources
than sensors [1, 2, 3].
If most of the samples of a signal are almost zero, we
say that this signal is sparse. When signals are sparse enough,we can assume that the sources overlap at rare intervals. We
can assume the sparseness of the speech signals especiallyin the time-frequency domain. For a detailed analysis ofsparseness, see [5].
Sparseness approaches use this assumption and extract
each signal using time-frequency binary masks. Because
we can assume that sources do not overlap very often, we
can extract each source by selecting the time points at which
there is only one signal. One way of estimating such time
points is to use the level difference of the observations andthe phase difference between the observations. In this paper, we utilize omni-directional microphones, and we use
1 (ω,m)the phase difference ϕ(ω, m) = 6 X
X2 (ω,m) between the observations X1 (ω, m) and X2 (ω, m).
Using ϕ(ω, m), we can estimate the direction of arrival(DOA) for each time point m by calculating θ(ω, m) =, where c is the speed of sound and d is thecos−1 ϕ(ω,m)c
ωd
microphone spacing. When we plot this DOA θ(ω, m), we
can see three peaks in the histogram for each frequency.
Let these peaks be θ˜1 , θ˜2 and θ˜3 where θ˜1 ≤ θ˜2 ≤ θ˜3 , andthe signal from θ˜ξ be S˜ξ (ξ = 1, 2, 3) (Fig. 2).
We can extract each signal with a binary maskMξ (ω, m) =10θ˜ξ − ∆ ≤ θ(ω, m) ≤ θ˜ξ + ∆otherwise(1)

by calculating Y ξ (ω, m) = Mξ (ω, m)Xj (ω, m) where j=1
or 2. Here, ∆ is an extraction range parameter: if ∆ is small
the separation performance is good but the distortion (musical noise) becomes large, on the other hand, if ∆ is large
the musical noise problem is overcome but the separation
performance deteriorates.
Although we can extract each signal using this binary
mask (1), such extracted signals are discontinuously zeropadded by the binary mask. Therefore, we can hear considerable musical noise in the extracted output.

272

~
θ1

80

100

80

100

120

140

160

120 140 160 180

~ DOA
θ2

~
θ3

Figure 2: Example of (a) histogram and (b) smoothed histogram.
Anechoic, female-male-male combination. DFTsize T =512.

4. Proposed Method: Combination ofsparseness and ICA (SPICA)
To overcome this musical noise problem, we propose using both sparseness and ICA. Our method has two stages(see Fig. 3). In the first stage, contrary to the conventional approach, we remove one source from mixtures using the signals’ sparseness. By this removal, it is expectedthat their zero-padding to be unimportant because we extract more time-frequency points than the conventional approach. Moreover we can expect the remaining mixtures
to consist of only two signals. Therefore, in the second
stage, we can apply ICA (e.g., [6]) to these remaining mixtures. Because these separated signals are not highly zeropadded, we can expect less musical noise.
[1st stage] One source removal:
Here, we utilize omni-directional microphones, therefore, we use the phase difference between the observations
to set θ˜1 , θ˜2 and θ˜3 shown in the previous section.
Instead of extracting each source as in conventional approaches, we remove only one source from the mixtureswith a binary maskM pq (ω, m) =
ICA

1
0

θmin ≤ θ(ω, m) ≤ θmaxotherwise(2)

by calculatingˆ (ω, m) = M pq (ω, m)
ICA(ω, m).

180

θ

0

(3)

In (2), θmin and θmax are extraction range parameters, andin (3), X̂(ω, m) is expected to be mixtures of S˜p and S˜q .
For instance, as in case 1, if S˜1 can be removed from the23
observations with a mask M ICA
we can use ICA to sep˜˜arate S2 and S3 in the next stage (see Fig. 3). In thiscase θmin and θmax in (2) can be θ˜1 < θth1 = θmin <θ˜2 , θmax = 180◦ (see Fig. 4). Similarly in case 2, when S˜3
12is to be removed from the observations with a mask M ICA,◦ ˜˜θmin = 0 , θ2 < θth2 = θmax < θ3 .
Because our system has only two outputs, both removals
should be performed to obtain three separated signals (seeFig. 3).
[2nd stage] Separation of remaining sources by ICA:
Because the remaining signals X̂ are expected to be
mixtures of two signals, we can use 2×2 ICA to separate

4.45 mθ1

Case1: Separation of S2 and S3[1st stage] X1 [2nd stage]

X2

^

S2

S3

Loudspeakers(height : 1.35 m)

^

X1

^

X=M23
X
ICA

Y2

Y=WX

1.1 m

Y3

Case2: Separation of S1 and S2^[1st stage] X1 [2nd stage]
^

^

X=M12
X
ICA

Y=WX

male
2.25 m4 cm

Y11.75 m

^

X2

male

1.80 m^X2

3.55 mS1

Y2

femaleOmni-directional microphones(height : 1.39 m)

Figure 3: System setup

Room height : 2.50 m

Figure 5: Room for echoic tests.

23

12
ICA

M

(case 2)
M1
~
θ

0
S1
S2
S3

1

90
8
2

M2
∆ ∆
θth1 ~
θ2

98
48
2
49
0
3
remove

case1
case2

MICA (case 1)

S1: 50

M3
θth2

~
θ

3

θ

(Fig. 5) and added them to obtain the mixtures.
We constructed a histogram of θ(ω, m) for each frequency, and smoothed it slightly (see Fig. 2). Then we divided the histogram into three parts, determined the peakfor each part, and set these peaks as θ˜1 , θ˜2 and θ˜3 . The
DFT frame size T was 512 and we used a frame shift of
256.

180

4
6
0
5
94
30
1
3
3
64
99
92
S1: 3 S2: 47 S3: 50

S2: 48 S3: 2

remove[%]Figure 4: Source power in each area (%).

5.2. Performance measures

X̂. The separation process can be formulated as Y (ω, m) =W (ω)X̂(ω, m), where X̂ is the masked observed signal
obtained by (3), Y (ω, m) = [Y ξ1 (ω, m), Yξ2 (ω, m)]T (ξ1 , ξ2
=1,2,3) is the separated output signal, and W (ω) represents a (2×2) unmixing matrix. W (ω) is determined sothat Yξ1 (ω, m) and Yξ2 (ω, m) become mutually independent. This calculation is carried out independently at eachfrequency.
i+1 (ω) =i (ω) +
 In this paper, the adaptive rule isη diag hΦ( ) H i − hΦ( ) H ii (ω), where Φ(y) =6φ(|y|) · ej· (y ) , φ(x) = tanh(gx) and g = 100 [7]. For
solving the permutation problem of frequency domain ICA,
we utilized the direction of arrival approach [8], and forsolving scaling problem of frequency domain ICA, we used
the minimum distortion principle [9].








5. Experiments
5.1. Experimental conditions
For the anechoic tests, we simulated the recording in an
anechoic room using the mixing matrix H ji (ω) = exp (jωτji ),dwhere τji = cj sin θi , dj is the position of the j-th microphone, θi is the direction of the i-th source, and c is the
speed of sound. We simulated a pair of omni-directional
microphones with an inter-element spacing of 4 cm. The
sampling rate was 8 kHz. The speech signals arrived fromthree directions, 50 ◦ (female), 100 ◦(male) and 135 ◦ (male).
For the reverberant tests, we recorded each speech signal in a real room whose reverberation time was T R =130 ms

273

To evaluate the effectiveness of our approach, we used
the signal to interference ratio (SIR) as a measure of separation performance, and the signal to distortion ratio (SDR)
as a measure of sound quality:
P 2
yis (n)(4)
SIRi = 10 log P nP i 2
i6=j

SDRi = 10 log P

n

P

n

yisj (n)x2 (n)
n ksi(xksi (n) − αyisi (n − D))2(5)

where the permutation is solved before calculating SIR and
SDR, i.e., yi is the estimation of si , and yisj is the outputof the whole separating system at y i when only s j is active
and sk (k 6= j) does not exist, and x ksj is the observation
obtained by microphone k when only s j exists. α and D
are parameters to compensate for the amplitude and phasedifference between x ksi and yisi . To evaluate the conventional method (sparseness only method), we calculated SIR
and SDR using the measurements from both microphones,
and adopted the better value.
5.3. Experimental results
Figure 4 shows the power content by percentage of eachsignal when ∆ = 10 ◦ . From the upper three rows of Fig. 4,
we can see that the observations in θ˜2 − ∆ ≤ θ(ω, m) ≤180◦ mainly contain the signals S˜2 and S˜3 , on the other
hand, the observations in 0 ◦ ≤ θ(ω, m) ≤ θ˜2 + ∆ mainlycontain the signals S˜1 and S˜2 . From this, we set θth1 =θ˜2 − ∆ for S˜1 removal (case 1), and θ th2 = θ˜2 + ∆ for S˜3removal (case 2). In Fig. 4, the percentages of each powerTable 1: Power lost by binary masks (in %)mask
output
[%]

M1
Y1
17

M2
Y2
14

M3
Y3
23

12
MICA
Y1 Y2
2.5 5.7

Table 2: Results of anechoic simulations. Signals are at50◦ , 100◦ , 135◦ . Sparse: with only sparseness, Case 1: withSPICA (S˜1 was removed for ICA), Case 2: with SPICA (S˜3 wasremoved for ICA).

23
MICA
Y2 Y3
8.1 0.7

female-male-maleextracted by the binary mask (2) with these thresholds arealso shown. Because the proportion of the third signal isvery small, we can say that we can use an ICA at the 2ndstage of our method.
TableP
1 shows thePsignal2power eliminated by the zerosi (n)2 −ŝi (n)n P
npadding
caused by binary masks, wheres (n)2nisˆi (n) = IDFT[Mi (ω, m)Si (ω, m)]. If we use these mask
thresholds for separation, i.e., in the sparseness only case,around 20% of signal power was diminished by the binary
mask. By contrast, with our method, the signal power eliminated by M pq (ω, m) was inferior. This result convinces
ICA
us that the adverse effect of zero-padding was mitigated byusing our method, that is, one source removal.
Table 2 shows the experimental results of anechoic simulations. The first row shows the results obtained solely
using the sparseness, the second and third rows show theresults obtained with our SPICA method. With sparseness
only, the SIR values were high but the SDR values were unsatisfactory. A large musical noise was heard in this case.
In contrast, with SPICA, we were able to obtain high SDRvalues without any serious deterioration in the separation
performance SIR. Now the musical noise decreased.
Moreover, we conducted reverberant tests (T R =130 ms)
with settings of 50 ◦ (female), 100 ◦ (male), 135 ◦ (male),and 45◦ (female), 90 ◦ (male), 135 ◦ (male). The results areshown in Table 3. Even in a reverberant environment, we
obtained reasonable results with our proposed SPICA.
It should be noted that it is hard to separate the signal
at the center position by both methods.
Some sound samples can be found at our web site [10].

6. Conclusion
We proposed a method for separating more speech signals than sensors by combining a sparseness approach andICA (SPICA). Our method avoids over-zero-padding, andtherefore, can separate the signals with less distortion in areverberation of 130 ms.

7. References[1] S. Rickard and O. Yilmaz, “On the W-Disjoint orthogonalityof speech,” Proc. ICASSP2002, vol.1, pp. 529-532, 2002.
[2] L. Vielva, D. Erdogmus, C. Pantaleon, I. Santamaria,
J. Pereda and J. C. Principe, “Underdetermined blindsource separation in a time-varying environment,” Proc.
ICASSP2002, vol. 3, pp. 3049-3052, 2002.
[3] P. Bofill and M. Zibulevsky, “Blind separation of more
sources than mixtures using sparsity of their short-time
Fourier transform,” Proc. ICA2000, pp. 87-92, 2000.
[4] M. Aoki, M. Okamoto, S. Aoki, H. Matsui, T, Sakurai and
Y. Kaneda, “Sound source segregation based on estimating

274

SparseCase 1Case 2

SIR1
17.6
17.5

SIR2
11.6
8.8
8.8

SparseCase 1Case 2

SIR1
13.1
12.6

SIR2
7.7
4.5
4.0

SparseCase 1Case 2

SIR1
23.6
16.5

SIR3
17.3
13.6
-

SDR1
7.3
20.8

[dB]
SDR2
9.3
12.5
11.8

SDR3
8.5
16.2
-

SDR2
8.1
9.6
13.5

SDR3
4.6
10.0
-

SDR2
11.9
15.3
14.2

SDR3
8.5
15.6
-

male-male-male
SIR3
15.6
10.4
-

SDR1
4.3
17.7

female-female-female
SIR2
11.3
8.0
8.0

SIR3
18.2
13.0
-

SDR1
8.5
21.5

Table 3: Results of reverberant tests (TR =130ms) for a femalemale-male combination. Sparse: with only sparseness, Case 1:
with SPICA (S˜1 was removed for ICA), Case 2: with SPICA (S˜3
was removed for ICA).
50◦ , 100◦ , 135◦
SparseCase 1Case 2

SIR1
9.9
9.0

SIR2
4.7
4.1
3.3

SIR3
8.3
9.6
-

SDR1
4.0
9.4

[dB]
SDR2
8.3
9.1
10.3

SDR3
4.8
6.9
-

SDR2
9.7
10.9
11.5

SDR3
3.5
8.4
-

45◦ , 90◦ , 135◦
SparseCase 1Case 2

SIR1
12.1
8.4

SIR2
5.8
5.3
4.2

SIR3
13.8
12.1
-

SDR1
4.3
7.6

incident angle of each frequency component of input signals
acquired by multiple microphones,” Acoust. Sci. & Tech, vol.
22, no. 2, pp. 149-157, 2001.
[5] A. Blin, S. Araki and S. Makino, “Blind source separation
when speech signals outnumber sensors using a sparsenessmixing matrix combination,” Proc. IWAENC2003, 2003.
[6] S. Araki, R. Mukai, S. Makino, T. Nishikawa and H.
Saruwatari, “The fundamental limitation of frequency domain blind source separation for convolutive mixtures ofspeech,”IEEE Trans. on SAP vol. 11, no. 2, pp. 109-116,
2003.
[7] H. Sawada, R. Mukai, S. Araki and S. Makino, “Polar coordinate based nonlinear function for frequency domain blindsource separation,” Proc. ICASSP2002, pp. 1001-1004, May
2002.
[8] H. Sawada, R. Mukai, S. Araki, S. Makino, “A Robustand Precise Method for Solving the Permutation Problem of Frequency-Domain Blind Source Separation,” Proc.
ICA2003 pp. 505-510, Apr. 2003.
[9] K. Matsuoka and S. Nakashima, “A robust algorithm forblind separation of convolutive mixture of sources,” Proc.
ICA2003, pp. 927-932, Apr. 2003.

[10] http://www.kecl.ntt.co.jp/icl/signal/araki/underdeter
minedBSSdemo.html