1IWAENC 2006 – PARIS – SEPTEMBER 12-14, 2006

BLIND SPARSE SOURCE SEPARATIONWITH SPATIALLY SMOOTHED TIME-FREQUENCY MASKING
1,2

Shoko Araki, 1 Hiroshi Sawada, 1 Ryo Mukai, and 1,2 Shoji Makino

shoko@cslab.kecl.ntt.co.jp
NTT Communication Science Laboratories, NTT Corporation
2-4 Hikaridai, Seika-cho, Soraku-gun, Kyoto 619-0237, Japan
2
Graduate School of Information Science and Technology, Hokkaido University
Kita 14, Nishi 9, Kita-ku, Sapporo-shi, Hokkaido 060-0814, Japan
1

ABSTRACT

2. PROBLEM DESCRIPTIONIn this paper, we handle the blind sparse source separation problem with time-frequency masks. Some existing methods employ
time-frequency binary masks to extract the signals, and therefore, the extracted signals tend to contain loud musical noise.
It causes from the winner-take-all property of the binary mask.
To overcome the problem, this paper proposes some non-binary
time-frequency masks that allow each time-frequency component to belong to several output signals. Experimental results
show that our proposed method can separate signals with little
distortion.

Suppose that sources s1 , . . . , sN are convolutively mixed andobserved at M sensors

1. INTRODUCTION
A time-frequency mask approach to the blind sparse source separation (BSS) is recently widely studied (e.g., [1, 2]). The timefrequency mask approach is attractive because it can handle anunderdetermined problem where the sources outnumber the sensors. With regards to these problems, we also have already proposed a time-frequency mask approach that is based on the observation vector clustering [3].
The time-frequency mask methods rely on the assumption ofsource sparseness. If signals are sufficiently sparse, that is, mostof the samples of each signal are almost zero, we can assume
that the sources rarely overlap. The time-frequency mask approaches employ this assumption and they usually extract eachsignal by applying a time-frequency binary mask to the observed
mixture. By using the time-frequency binary mask we can extract each signal from a mixture. However, the use of binarymasks leads to too much discontinuous zero-padding to the extracted signals, and therefore, they tend to contain loud musicalnoise, which is undesirable for audio applications. The zeropadding is caused by the binary mask’s winner-take-all property:
each time-frequency point of the observed mixture is allowed tobelong to only one extracted signal.
In order to reduce the musical noise problem, in this paper,
we propose some non-binary time-frequency masks that allow
each time-frequency component to belong to several output signals. Our non-binary masks are designed by using the cluster information of the observation vector clustering [3], therefore, themask has the spatially-smoothness. We show that our proposed
non-binary masks can reduce musical noise with no degradationof separation performance.

PP
xj (t) = N(1)
k=1
l hjk (l) sk (t − l), j = 1, . . . , M,
where hjk (l) represents the impulse response from source k tosensor j. In this paper, we look especially at a situation where
the number of sources N can exceed the number of sensors M(N > M ). We assume that N and M are known, and that thesensor alignment does not cause the spatial aliasing problem.
Our goal is to obtain separated signals yk (t) that are estimationsof sk calculated solely from M observations.
This paper employs a time-frequency domain approach. Using a short-time Fourier transform (STFT), the convolutive mixtures (1) can be converted to instantaneous mixtures at each frequency f :P(2)
xj (f, τ ) ≈ N
k=1 hjk (f )sk (f, τ ),or in vector notation,
x(f, τ ) ≈PN

k=1 hk (f )sk (f, τ ),(3)

where hjk (f ) is the frequency response from source k to sensor
j, sk (f, τ ) is the STFT of a source signal sk , and τ is a timeindex. We call x = [x1 , . . . , xM ]T an observation vector andhk = [h1k , . . . , hM k ]T is a vector of the frequency responses
from source sk to all sensors.
We assume the sparseness of sources in the time-frequency
domain. This assumption has been widely employed for solving
the underdetermined problem (e.g. [1, 2, 3]). When the signals
are sufficiently sparse, we can assume that the sources rarely
overlap at each time-frequency point, and (3) can be approximated asx(f, τ )≈hk (f )sk (f, τ ), k ∈ {1, · · · , N },(4)

where sk (f, τ ) is a dominant source at the time-frequency point(f, τ ). For instance this is true for speech signals in the timefrequency domain [1].
3. CONVENTIONAL BINARY MASK APPROACH
3.1. Method
Here we employ the method proposed in [3]. First, we normalize all observation vectors x(f, τ ) for all time-frequency points

2

IWAENC 2006 – PARIS – SEPTEMBER 12-14, 2006

x1(t)

...
xM(t)

]

x(f,τ)

yk(f,τ)Normalization

Mk(f,τ)STFT]

x(f,τ)

ISTFT

qP
M
2
where Ak =j=1 λjk . We can see that the normalized observation vector x̄(f, τ ) depends only on the source geometry
λjk and τjk of the source sk , which is dominant at the timefrequency point (f, τ ). Therefore, the normalized observation
vectors can be clustered based on the source geometry. In otherwords, the observation vectors x̄(f, τ ) in a cluster Ck are assumed to be the signal components coming from a physicallyspatially localized region.

yk(t)

Mask
designClustering
Ck

Figure 1: Flow of the time-frequency mask approach.
4. NON-BINARY MASK PROPOSALS(f, τ ). The normalization includes phase-normalization with respect to a sensor J and the frequency-normalization,»–arg[xj (f, τ )/xJ (f, τ )]x̄j (f, τ ) ← |xj (f, τ )| exp (5)4f c−1 dmax
where c is the propagation velocity and dmax is the maximum
distance between sensor J and a sensor ∀ j ∈ {1, . . . , M } [3].
Then, we apply unit-norm normalizationx̄(f, τ ) ← x̄(f, τ ) / ||x̄(f, τ )||(6)T

to x̄(f, τ ) = [x̄1 (f, τ ), . . . , x̄M (f, τ )] .
Then we find clusters C1 , . . . , CN formed by all normalized
vectors x̄(f, τ ). The clustering criterion is to minimize the totalsum J of the squared distances d2k between cluster members andtheir centroid:
J

=

M
X

X

d2k (f, τ )(7)=||x̄(f, τ ) − ck ||.

(8)

The clustering is realized by the following iterative updates:
Ck
ck

=
←

{x̄(f, τ ) | k= argmini d2i (f, τ )}E[x̄(f, τ )]x̄∈Ck , ck ← ck /||ck ||,4.1. MASK1: Sigmoid based mask
The first mask has the shape of the sigmoid function [5]. The
mask is defined as
Mk (f, τ )k=1 x̄(f,τ )∈Ck

dk (f, τ )The binary mask (11) can extract each signal from a mixture,
however, it causes much discontinuity in the extracted signals,
and therefore, the musical noise occurs. It arises as a result of
the binary mask’s winner-take-all property, as mentioned in Section 1. In order to relax the property, in this section, we propose
some non-binary time-frequency masks that allow each timefrequency component to belong to several output signals.
Here, we specify the mask based on the distances dk (f, τ )(8) between cluster members and their centroid, which was utilized in the clustering procedure (9). From the fact mentioned
in Sec. 3.2, our proposing non-binary masks become a spatiallysmooth mask.

(9)
(10)

=
=

Mk (dk (f, τ ))1
1 + exp [g(dk (f, τ ) − thk )](14)

where thk and g are parameters deciding the shape of the sigmoid function. The smaller thk , the more interference power issuppressed, but the more musical noise sound in the extracted
signals. An example sigmoid mask is shown in Fig. 2.

where E[·]x̄∈Ck is a mean operator for the members of a cluster
Ck . This minimization can be performed efficiently with the k4.2. MASK2: Baysian theorem based maskmeans clustering algorithm [4] with a given source number N .
Because each resulting cluster corresponds to an individual
As the second mask, we newly propose to use the Bayse’s thesource, finally, we have separated signals yk (f, τ ) = Mk (f, τ )xj (f, τ )orem by assuming each cluster member x̄(f, τ ) ∈ Ck has awherecomplex
Gaussian distribution,
1 x̄(f, τ ) ∈ Ckhi
(11)
Mk (f, τ ) =1
0 otherwisep(x̄|Ck ) =exp −(x̄ − ck )H Σ−1k (x̄ − ck )N(2π) |Σk |(15)then we have time-domain outputs yk (t) by using an inverse
where Σk is a covariance matrix of x̄(f, τ ) ∈ Ck . Note that
STFT (ISTFT).
the term (x̄ − ck )H Σ−1k (x̄ − ck ) in (15) is the square distance2dk normalized by Σk . With the Bayse’s theorem, the posterior
3.2. Rationale
probability of x̄(f, τ ) being x̄(f, τ ) ∈ Ck is given as
Let assume that an impulse response hjk (f ) is modeled asp (x̄(f, τ )|Ck ) P (Ck )P (Ck |x̄(f, τ )) =(16)
hjk (f ) ≈ λjk exp [−2πf τjk ].
(12)p(x̄(f, τ ))where λjk ≥ 0 and τjk are the attenuation and the time delay
where P (Ck ) is known priori (here we assume P (Ck ) = 1/N,∀ k),from source k to sensor j. If we assume an anechoic situationPand p(x̄(f, τ )) = N
k=1 p(x̄(f, τ )|Ck )P (Ck ).
and no-spatial aliasing, the parameters λjk and τjk are deterHere we use this posterior as our mask,
mined solely by the geometric condition of the sources and sensors. Considering (4) and (12), the normalized vector x̄(f, τ )Mk (f, τ ) = P (Ck |x̄(f, τ ))(17)
can be written as
PN»–
The mask has a property that k=1 Mk (f, τ ) = 1, because ofπ(τjk − τJk )λjkx̄j (f, τ ) ≈exp −
,(13)
the normalization with the denominator of (16). An example
Ak
2c−1 dmax

3

IWAENC 2006 – PARIS – SEPTEMBER 12-14, 2006

If we know the sensor configuration, we can employ the directivity pattern of a fixed null-beamformer (NBF) that makes nulls
towards the interference directions [6] as our non-binary mask.
This mask realizes a spatially smooth mask, straightforwardly.
STEP1: Direction estimation
First, we have to estimate the direction of arrivals (DOAs) qk of
all N sources, where qk is a 3-dimensional vector of a unit-norm
representing the direction of the source sk . DOAs are estimatedby the method described in [7]q̂k = −

2dmax +
P arg [ck ]π(18)where P = [p1 −pJ , · · · , pM −pJ ]T and pj is a 3-dimensional
vector representing the location of sensor j.
STEP2: Mask design by the directivity pattern of NBF
Then, we make an NBF that makes nulls towards N − 1 interference directions. Remember that we have only M < N sensors
in an underdetermined case. In order to form N − 1 nulls, we assume V > (N − 1) + 1 (virtual) sensors [6] of arbitral positionsof V virtual sensors p0j (j = 1, · · · , V ).
Then NBF W (f ) is calculated by W (f ) = H −1
NBF (f ), where
H NBFˆ (f ) is a (V × V ) matrix whose
ji-th element HNBFji (f ) =˜
exp −2πf c−1 (p0j − p0J )T q̂i , q̂1 = q̂k , and k is the index of
the source to be extracted. The directivity pattern of the NBF isobtained as a function of DOA q
Fk (f, q) =

V
X

c1

c2 c3

c4
d1

Figure 2: Sigmoid mask (MASK1). The horizontal axis is the
distance d1 from the centroid c1 .
1Mask gain for each time−frequency point

4.3. MASK3: Directivity pattern based mask

Mask gain

mask is shown in Fig. 3. It looks scattered. This is because evenif dk (f, τ ) = dk (f 0 , τ 0 ), it does not mean x̄(f, τ ) = x̄(f 0 , τ 0 )and therefore p(x̄(f, τ )|Ck ) 6= p(x̄(f 0 , τ 0 )|Ck ).

0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
0

1

2

C1

3

4

C2

C3

Normlized distance

5

6

7

C4

Figure 3: Baysian mask (MASK2).
The horizontal
axis is the normalized distance from the centroid c1 :q(x̄ − c1 )H Σ−1
1 (x̄ − c1 ), see section 4.2.

W1j (f ) exp [−2πf c−1 (p0j − p0J )T q]. (19)j=1

An example gain pattern of (19) is given by Fig. 4 (a).
By employing this gain pattern, our proposed mask becomesMk (f, τ ) = Fk (f, q(f, τ ))(20)

where q(f, τ ) is the estimated DOA of a observation x̄(f, τ ) at
each time-frequency [7].
We can modify the mask so that it has a small constant gain
except for a main beam [6]. Figure 4 (b) shows an example.
5. EXPERIMENTS

conducted a small subjective test with four listeners, since SDR
cannot evaluate musical noise [8]. It was a paired comparison
test, where we paired up a conventional binary mask output witha proposed smooth mask output, and each listener judged which
has less musical noise for many pairs.
5.2. MASKs’ parameters
For MASK1, g was 7.8, that realized as good SIR as with aconventional binary mask, and thk was the standard deviationof dk (f, τ ) ∈ Ck . For MASK2, P (Ck ) = 1/N,∀ k, that is we
assume the same priori for all k. For MASK3, the number of thevirtual sensors V = 4.

5.1. Experimental conditions
We evaluated the performance with 5-second English speech sources
and measured impulse responses in a room (Fig. 5). The observations were simulated by (1). The sampling rate was 8 kHz, the
STFT frame size L = 512 and the frame shift was L/2, L/4
and L/8. Here, we only tested the 3-microphones and 4-sourcescase.
The separation performance was evaluated in terms of thesignal-to-interference ratio (SIR) improvement [3]. Moreover,
we also evaluated the separated sound quality with the signal todistortion ratio (SDR) [3]. We investigated four speaker combinations and averaged the results for all outputs. Furthermore, we

5.3. Results
Results are shown in Table 1, where “BM” denotes the conventional binary mask (11), and MASK3’ means the modified
MASK3 with the same way as Fig. 4(b). By using non-binarymasks, we improved the SDR without degrading the SIR. Table
1 also have the result of the listening test with paired comparison(PC) for the frame shift=L/4. The PC values show the percentage of MASK’s superiority over the BM in musical noise
PC = 100 ×# of pairs where MASK is better than BM# of all pairs in the test

4

IWAENC 2006 – PARIS – SEPTEMBER 12-14, 2006

1.5

Table 1: Experimental results. SIR and SDR are in dB. “Shift”
stands for the frame shift. ”PC” is the result of the listening testwhen Shift=L/4.

Mask gain

(a)
1

0.5

0
0

50

100

150
200
Azimuth θ [deg.]

250

300

350

300

350

1.5

Mask gain

(b)
1

0.5

0
0

50

100

150
200
Azimuth θ [deg.]

250

Figure 4: (a) Example directivity based mask (MASK3) and (b)its modification (MASK3’) for extracting s1 in a setting shownin Fig. 5. The horizontal axis is the azimuth θ whose definitionis shown in Fig. 5.

880 cmMicrophone (120cm height, omni-directional)
Loudspeaker (120cm height)[Mic. setup]
4cm

S3375 cm

S2
282cm

200cm

SIRi
SDRi
MASK1 SIRi
SDRi
MASK2 SIRi
SDRi
MASK3 SIRi
SDRi
MASK3’ SIRi
SDRi

ShiftL/2 ShiftL/4 ShiftL/8 PC(%)
13.5
14.5
14.9
8.1
8.9
9.1
13.4
14.6
14.7
90.6
8.4
8.6
8.7
14.2
15.1
15.3
62.5
8.5
9.2
9.3
12.1
13.0
13.2
87.5
8.5
9.3
9.4
13.1
13.9
14.2
81.3
8.9
9.6
9.8

flected in SIR. This was because the interferences’ residual also stayed without zero-padding.
5. Our non-binary masks worked effectively when the frame
shift was a half or quarter of a frame size (Shift = L/2
or L/4). For the fine frame shift (Shift=L/8), the musical noise of BM was not so loud, and therefore, our nonbinary masks had small impact.

4cm
4cm

S4

BM

[DOA definition]

S1• Reverberation time: 120 ms• Room height: 240cm

Figure 5: Experimental setup. θk (k = 1, · · · , 4) are the azimuth
values of signals. The elevation φk = 0, ∀k.

where the numerator means # of pairs where a MASK{1,2,3,3’}
output has less musical noise than a conventional binary mask
output. The listening test results show that our non-binary masks
decrease the musical noise significantly.
The observed phenomena were as follows.
1. MASK1 achieved good performance in spite of its simple realization (14). It has only one parameter g and its
setting is not difficult.
2. The ability to reduce musical noise of MASK2 (baysianmask) was smaller than these of other masks. This is because that the MASK2 is a non-binary mask but it does
not have spatial continuity shown in Fig. 3.
3. SIR of MASK3 was not good because MASK3 has large
gains between null directions (see Fig. 4(a)). On the other
hand, MASK3’ (Fig. 4(b)) realized good performance in
SIR with less musical noise.
4. With our non-binary masks the interferences in outputs
were now understandable, although the effect is not re-

6. CONCLUSION
We proposed spatially smooth non-binary masks which allow
each time-frequency component to belong to several output signals. Although they do not assure the perfect time-continuity ofthe output signals, the proposed masks effectively reduced themusical noise without decreasing the separation performance.
7. REFERENCES[1] Ö. Yılmaz and S. Rickard, “Blind separation of speech mixtures via time-frequency masking,” IEEE Trans. Signal Processing, vol. 52, no. 7, pp. 1830–1847, July 2004.
[2] N. Roman, D. Wang, and G. J. Brown, “Speech segregation
based on sound localization,” Journal of Acoustical Societyof America, vol. 114, no. 4, pp. 2236–2252, 2003.
[3] S. Araki, H. Sawada, R. Mukai, and S. Makino, “Anovel blind source separation method with observation vector clustering,” in Proc. IWAENC 2005, Sept. 2005.
[4] R. O. Duda, P. E. Hart, and D. G. Stork, Pattern Classification, Wiley Interscience, 2nd edition, 2000.
[5] H. Sawada, S. Araki, R. Mukai, and S. Makino, “Blind extraction of a dominant source signal from mixtures of manysources,” in Proc. ICASSP2005, Mar. 2005, vol. III, pp. 61–
64.
[6] S. Araki, S. Makino, H. Sawada, and R. Mukai, “Source
extraction from speech mixtures with null-directivity patternbased mask,” in Proc. HSCMA2005, Mar 2005, pp. d1–d2.
[7] S. Araki, H. Sawada, R. Mukai, and S. Makino, “DOA estimation for multiple sparse sources with normalized observation vector clustering,” in Proc. ICASSP2006, May 2006,
vol. 5, pp. 33–36.
[8] N. Virag, “Single channel speech enhancement based on
masking properties of the human auditory system,” IEEE
Trans. on SAP, vol. 7, no. 2, pp. 126–137, 1999.