International Workshop on Acoustic Echo and Noise Control (IWAENC2003), Sept. 2003, Kyoto, Japan

Blind Source Separation when Speech Signals Outnumber Sensors
using a Sparseness - Mixing Matrix Estimation (SMME)
Audrey Blin
Shoko Araki
Shoji MakinoNTT Communication Science Laboratories, NTT Corporation
2-4 Hikaridai, Seika-cho, Soraku-gun, Kyoto 619-0237, JAPANe-mail: blin@cslab.kecl.ntt.co.jp
Abstract
This paper focuses on underdetermined blind source
separation (BSS) of three speech signals mixed in a realenvironment from measurements provided by two sensors.
Underdetermined BSS is a problem that has not yet been
intensely studied and so far no satisfying solution has beenobtained. The major issue encountered in previous work
relates to the occurrence of distortion, which affects a
separated signal with loud musical noise. To overcome thisproblem, we propose combining sparseness with the use ofan estimated mixing matrix. First, we use a geometricalapproach to perform a preliminary separation and to detect
when only one source is active. This information is then used
to estimate the mixing matrix, which allows us to improve
our separation. Experimental results show that thisSparseness - Mixing Matrix Estimation (SMME) provides
separated signals of better quality (less distortion, lessmusical noise) than those extracted without using theestimated mixing matrix.

1. Introduction
BSS refers to the problem of recovering signals fromseveral observed linear mixtures. The adjective “blind”
stresses the fact that the source signals are not observed andthat no information is available about how the sources are
mixed.
In a determined problem a common way of solving the
BSS issue was to use Independent Component Analysis(ICA) [1], or to estimate and then invert the mixing matrix
modeling the system. But, here, where sources outnumber
sensors, the mixing matrix is no longer square and theprevious solutions are not feasible. In fact, more informationis needed to solve an underdetermined problem.
Up to now, solving the BSS problem in anunderdetermined case has mainly consisted in assuming thatthe speech signals were sufficiently sparse [2-6]. However,
due to unexpected discontinuous zero-padding, suchseparated signals have considerable distortion, and therefore
a loud musical noise is heard.
In [5], an estimated mixing matrix was used for solving thedetermined BSS problem. Our suggestion for eliminating thedistortion matter is to combine sparseness with a mixing
matrix estimation. Indeed we can obtain more information
about the signals to be separated and hence reduce the zeropadding effect, from which the musical noise originates.
Whereas Vielva et al., Rickard and Yilmaz worked on anunderdetermined instantaneous case employing sparseness[2,3,4], and Deville on a determined instantaneous caseutilizing a mixing matrix estimation [5], here, we are dealing
with underdetermined BSS in a convolutive case.
In this paper, essential investigations with regard tosparseness are described in Sec. 3. Section 4 is a step-by-steppresentation of the proposed method, whose results areprovided in Sec. 5. The paper concludes in Sec. 6.

2. Problem statements and notations
In this paper, we consider speech mixtures observed in areal room. In this case, as speeches are mixed with theirreverberation, the observed vectors xj (j=1..M) can bemodeled as convolutive mixtures of the source signals si(i=1..N) as follows:Nx j (t ) = ∑ h ji * si (t )

(1)

i =1

where hji is the impulse response from a source i to a sensor j.
In this paper, we deal with a case where N=3 sources and
M=2 sensors. Moreover, we assume that the source signals
are mutually independent and sparse: namely signals havelarge values at rare sampling points. We are using the Short
Time Fourier Transform (STFT) to convert our problem intoa linear instantaneous mixtures’ problem as well as toimprove the sparseness of the speech signals [4]. In the timefrequency
domain,oursystembecomes:X ( f , m) = H ( f ) S ( f , m) where f is the frequency, m theframe index, H(f) the 2 × 3 mixing matrix whose i-jcomponent is a transfer function from a source i to a sensor j,
X(f,m)=[X1(f,m), X2(f,m)]T and S(f,m)=[S1(f,m), S2(f,m),S3(f,m)]T, namely the Fourier transformed observed signals
and source signals, respectively.
Our aim is to estimate three speech signals frommeasurements provided by two sensors.

3. Sparseness Inquiries
The first definition of sparseness is that the more zero
samples contained in a source, the more sparse it is, whichmeans that the sources overlap at infrequent intervals.
Figure 1 is a histogram showing the number of sources thatare simultaneously active. It can be seen that the time points
where no sources are active are very numerous whereas thetime points where three sources are active are very
infrequent. We can infer from these observations that thesignals are sparse and that the three signals rarely overlap.

211

80

1

2

Proportion

1

150

Desired source power remaining (rj %)30lost energy => zero-padding

75
70
65
60
55
50
45

original data

40

128ms echoic data

35

200ms echoic data

recoverable energy

30
0

100

5

10

15
20
Threshold x (dB)25

30

Fig. 2: Approximate WDO against the threshold x for an fftsizeof 512 and a male-male-female combination.
50

3.2. The sparsest representation with STFT

4Frequency (kHz)Fig. 1: Histogram of the number of active sources: 0, 1, 2 or 3 fora male-male-female combination recorded with a reverberationof 200 ms and for an fftsize of 512.

3.1. Measure of overlapping
In this section, we investigated the sparseness moreclosely checked the degree of signal overlap by utilizing acriterion called Approximate W-Disjoint Orthogonality(WDO) defined by Rickard and Yilmaz [6].

To make the separation easier, we have to represent ourdata in the domain where their degree of sparseness is thebest. By plotting the Approximate WDO against the fftsize in
Fig.3, we can estimate the appropriate fftsize for the easiest
separation. Hence, from now, we will use an fftsize of 512
for a 8 kHz sampling rate in as much as this fftsize allows thebest recovery in terms of energy and therefore separated
signals are less distorted.
70
Desired source power remaining (rj %)0

We use a mask:

1 20 log(| S j ( f , m ) | / | Y j ( f , m ) |) > x (2)φ ( j , x ) ( f , m) = otherwise
0
where Yj(f,m) is the STFT of y j (t) =N∑ s (t) i.e. y (t) is

i =1,i ≠ j

ij

the summation of the sources interfering with source j. The
Approximate WDO is defined as:r j ( x) = 100 || φ ( j , x ) ( f , m) S j ( f , m) || / || S j ( f , m) || (3)22This measures the percentage rj of source j energy for timefrequency points where it dominates the other signals by rj %at x dB. From this criterion it emerges that, if we can predict
the time-frequency points at which a source dominates theothers by rj % at x dB, we should be able to recover rj % of
the energy of the original sources. If rj is sufficiently large,
we can separate signals with small distortion and vice-versa.
For example in Fig. 2, if we want a signal-to-interference
ratio of 20 dB, only around 50 % of the original power isrecoverable, which means that almost half the points are
zero-padded by a mask and such distortion cannot be avoided.
Moreover Fig. 2 shows that reverberant data have a lower
Approximate WDO than unreverberant data. Henceseparating reverberant data becomes more difficult.

65
60
55
50
45
40
35
30
25
20
32

64

128

256

fftsize

512

1024

2048

Fig.3: Approximate WDO of anechoic measured data against thefftsize for a threshold x of 10 dB and a sampling rate of 8 kHz.

4. Proposed method: Sparseness Mixing Matrix Estimation (SMME)
In previously reported methods [2-4], one of the major
drawbacks was the occurrence of distortion, i.e., musicalnoise. To overcome this issue, we propose a three-stepmethod. First, using the sparseness of speech signals, we
adopt a geometrical approach extracting the time points mwhen only one source is active [1rst step], then we estimate
the mixing matrix [2nd step] and finally we reconstruct thesignals when two sources are active [3rd step].
•[1rst step] Geometrical approachThis first step consists of detecting the frame indices mwhen only one of the three sources is active for eachfrequency bin f.
Scatter-plots of the measurements, as shown in Fig. 4,
comprise three main lines (if the sources are sparse enough).

212

According to Vielva et al. [4], these lines symbolize thedirections defined by the column vectors of the mixing
matrix. In other words, they can be seen as a representationof each source existing alone. In between two given
directions, we find the time-frequency points modeling oursystem when two sources (those linked to the above
directions) are active simultaneously.
By setting narrow areas each containing only one line,
such as areas 1, 2 and 3 in Fig. 4, we are able to determine
when only one source is active. At the same time we can also
reconstruct the signals for these time-frequency points. Thisis the method exploited in previous works [2, 3, 4]. However,as expected by using such a rough approach, the quality ofthe separated signals is not satisfactory. Since the rate ofrecoverable energy is too low (as shown in Fig. 6), we cannot
avoid an important zero-padding, which makes the signals
insufficiently continuous. As a result, considerable distortion
i.e., loud musical noise can be heard.
To overcome this lack of quality, we attempt to complete
our separation using a totally different approach, relying on
the knowledge of the mixing matrix.

area 2|X2(f,m)|S1

S3[3rd step] Reconstruction of time-frequency points
when two sources are active
At this stage, it should be noted that knowing the mixing
matrix does not enable us to separate the signals when threesources are active. This is because the mixing matrix is not
square and does not have any inverse. Deville has only
applied his method to a squared mixing matrix.
Nevertheless, it is still possible to rebuild the time-frequency
points when two sources are active, providing that for eachfrequency bin, we know the frame indices for which this caseoccurs. Once more this information is provided by thegeometrical approach employed in the first step. But thistime, instead of setting the limits very close to the observed
directions, we are considering much wider areas so as toenclose the points located between two given directions.
Indeed let us suppose that, for an estimated (f, m) detectedduring the first step, S1(f, m) is null (area 23 in Fig. 4), in thisarea, our system becomes:1
 X 1 ( f , m)  H ( f )22= X ( f , m )
 2
  H 12 ( f )1 H 23 ( f )  ⋅  H 12 ( f ) S 2 ( f , m) (6)H 13 ( f )   H 13 ( f ) S 3 ( f , m)

Now the mixing matrix is square and can thus be
inverted, leading to H12(f)S2(f,m) and H13(f)S3(f,m):−1
 H 12 ( f ) S 2 ( f , m )  H 1( f ) H 1( f )   X 1 ( f , m ) (7) ⋅23 H ( f ) S ( f , m ) =  22
3
 13
  H 12 ( f ) H 13 ( f )   X 2 ( f , m )Moreover, in this area, if the signals H12(f)S2(f,m) andH13(f)S3(f,m) are not too greatly zero-padded, we expect thatthe distortion of the estimated H12(f)S2(f,m) and H13(f)S3(f,m)
will not be that large. We proceed in the same way whenS3(f,m) is null.
It should be noted that, in Fig. 1, we have alreadyconfirmed that we do not often have three sources active
simultaneously.

area 1S2•

area 3area 23|X1(f,m)|Fig. 4: Scatter-plots of the mixtures at a frequency of 312 Hz formale-male-female combination, a reverberation of 130 ms and an
fftsize of 512.

5. Experiments
5.1. Experimental conditions•[2nd step] Estimation of mixing matrix
Deville recovers the mixing matrix by estimating a certain
cross-correlation parameter ratio over time-frequency zones
where only one source exists [5]. This ratio was then proved
to be equal to H2i/H1i (i=1, 2, 3).
In constrast to Deville, here we are working with aunderdetermined convolutive case, however his approachgave us the idea to model our system in the time-frequency
domain by: X 1 ( f , m)   H 1( f ) X ( f , m) =  21
 2
  H11 ( f )1
H 22 ( f )H12 ( f )1   H11 ( f ) S1 ( f , m) 
H 23 ( f )  ⋅  H ( f ) S ( f , m)
2
 12
H13 ( f )   H ( f ) S ( f , m) 
3
 13


1

(f,m) = H 1i (f)Si (f,m)2(f,m) = H 2i (f)Si (f,m)

S2 (male)(4)
S1 (female)

Therefore, using time-frequency points estimated in thefirst step when only Si (i=1, 2, 3) is active, we have:

S3 (male)Fig. 5: Experimental conditions(5)

whose ratio X2(f,m)/X1(f,m) provides one of the componentsof the mixing matrix H2i(f)/H1i(f).

The recordings were done in a room with little
reverberance (TR=130 ms) using a two-element array of
directional microphones 4 cm apart. The speech signals,
sampled at 8 kHz, came from three directions: 120° (male),213

90° (male) and 50° (female) and the distance between thesources and the sensors was L = 55 cm.

5.2. Mask justification
Figure 6 justifies our decision to use wide masks. Indeed if
we use narrow masks (e.g., area 3 in Fig. 4) as in the previous
method, the recoverable power is only around 45 % with athreshold of 10 dB whereas if we utilize wider masks (e.g.,area 23 in Fig. 4), we can recover over 60 % of this power.
Desired source power remaining (rj %)80

Table 1 shows the results we obtained from ourmeasurements. By “sparseness” we imply that we are
evaluating the performance of our speech signals when weare applying the narrow masks. “invH12” means that we areapplying our mixing matrix to area 12 comprising speech
signals 1 and 2. Likewise “invH23” means that we areapplying our mixing matrix to area 23 comprising speech
signals 2 and 3. Actually, we are comparing the conventional
method with our SMME method.
Table 1: SIR and SDR calculated in dB for different approaches,
fftsize=512, TR=130 ms

wide masksnarrow masks

70

SIR1

SIR2

SIR3

SDR1

SDR2

SDR3

sparseness

15.3

9.9

10.6

8.4

10.3

3.4

50

invH12

11.6

3.1

8.7

12.2

40

invH23no masks
60

30

20
0

5

10

15
Threshold (dB)

20

25

30

Fig. 6: Approximate WDO against the threshold, fftsize=512,
TR=130 ms.

To evaluate the separation performance of our method, we
have chosen to calculate the Signal-to-Interference Ratio(SIR) as a measure of separation performance and the Signalto-Distortion Ratio (SDR) as a measure of sound quality:∑ y (t)SIR = 10log∑ ∑ y (t)

7.6

12.5

7.2

As we can see, the use of our SMME method allows us
to obtain less distorted signals.
We performed informal listening tests and it isimportant to note that, for the third signal, although the two
SDR results are very close, much less musical noise is heard
when separation is undertaken using SMME than when only
sparseness is used.

Consequently the technique using wide areas makes it
possible to reduce the distortion of the separated signals,
which was our aim.

5.3. SIR and SDR calculations

3.3

6. Conclusion
We proposed a separation method for use when there are
more speech signals than sensors by combining a sparsenessapproach and an estimation of the mixing matrix. The firstexperimental results are very encouraging in terms of quality
and suggest that the SMME is an approach that deserves
serious investigation.

2

t

i

2

i≠ j

SDR i = 10log

(8)

isi

t

∑x
t

2
ksi

(t)

∑(x (t) − .\ (t − ϕ ))ksi

7. References

is j

2

isi

t

where the permutation is solved before calculating SIR and
SDR, i.e. yi(t) is the estimation of si(t), and yiSj is the outputof the whole separating system at yi when only sj is active andsk (k ≠ j) does not exist, and xkSj is the observation obtainedby microphone k when only sj exists. α is a constant thatcompensates for the amplitude difference and ϕ is an angle
that fits the phase difference between input xkSj and output
yiSj. To evaluate the performance of the previous method(sparseness only method), we calculated SIR and SDR using
both microphones’ measurements, and adopted the better
values.

5.4. Results[1]

A. Hyvarinen, J. Karhunen and E. Oja, “Independent
Component Analysis,” John Wiley & Sons, 2001.

[2]

P. Bofill and M. Zibulevsky, “Blind separation of more
sources than mixtures using sparsity of their short-time Fouriertransform,” Proc. ICA2000, pp. 87-92, 2000.

[3]

M. Zibulesky, B.A. Pearlmutter, P. Bofill and P. Kisilev,“Blind source separation by sparse decomposition in a signal
dictionary,” TR No. CS99-1, University of New Mexico,
Albuquerque, July 1999.

[4]

L. Vielva, D. Erdogmus, C. Pantaleon, I. Santanmaria, J.
Pereda and J. C. Principe, “Underdetermined blind source
separation in a time-varying environment,” Proc. ICASSP2002,
vol. 3, pp. 3049-3052,2002.

[5]

Y. Deville, “Temporal and time frequency correlation-based
blind source separation methods,” Proc. ICA2003, pp. 10591064, April 2003.

[6]

S. Rickard and O. Yilmaz, “On the approximate w-disjoint
orthogonality of speech,” Proc. ICASSP2002, vol.1, pp. 529532, 2002.

(9)

214