Musical noise reduction in time-frequency-binary-masking-based
blind source separation systems
2, 3, aJ. Čermák, 1S. Araki, 1H. Sawada and 1S. Makino

1

NTT Communication Science Laboratories, NTT Corporation, Kyoto, Japan
Institute of Radio Engineering and Electronics, Academy of Sciences, Prague, Czech
Republic3
Department of Telecommunication, Brno University of Technology, Brno, Czech Republic
E-mail: cermak4@kn.vutbr.cz
2

Abstract: Blind source separation (BSS) problem consists of estimating N
sources from M mixtures without using source and mixing information. In thispaper we focus on improving BSS method called time-frequency binary masking(TFBM). TFBM is a versatile approach due to its ability to separate signals inboth (over-)determined case ( N ≤ M ) and underdetermined case ( N > M ). The
cost we pay for the versatility is the musical noise which is introduced to theseparated signals by the binary masking. We introduce a method, which reducesmusical noise and improves separation performance in both (over-)determined
and underdetermined cases.

1. Introduction
Blind source separation methods can be employed in many applications e.g. teleconferences,pre-processing for speech recognition. It has been discussed by many authors [1, 2] lately but
most of the existing approaches are applicable just for a (over-)determined case (DC) [2].
Approaches for underdetermined case (UDC) suffer from musical noise [3] or can not be usedin unknown environment. Our aim is to develop an approach that is blind, does not causemusical noise and can be used even for UDC; see table 1 for comparison between different
source separation methods.

ICA
TFBM
AFB
ProposalDC
UDC
Musical noise
☺
☺
☺
☺
☺
☺
☺
☺
☺
Table 1: Comparison of source separation methods.

Blind
☺
☺
☺

Independent component analysis (ICA) [2, 4] is a statistical BSS method relying only on
statistical independence of the source signals. ICA is a well known representative of DC
which does not cause musical noise.
Time-frequency binary mask [3, 5] is a BSS approach that can be applied even to an UDC.
However it causes a musical noise problem due to zero padding in the time-frequency (TF)
domain.
Adaptive beamformer (ABF) [6] is a system performing spatial filtering by forming a
directivity pattern of an array with M sensors in order to emphasize a target signal sk(t)arriving from a given direction and to suppress signals arriving from other directions(jammers). In order to separate N sources, a set of N different ABFs (multiple-beamformer)
must be employed. The critical problem of ABF is that the speech separation is not blind,aThis work was done during the internship at NTT Communication Science Laboratories.

because ABF needs a priori information about the target signal sk (t ) , e.g. a mixing vector orat least its approximation (steering vector). Multiple-beamformer may also be used for an
UDC but the jammer suppression is not efficient because only M-1 minimums (null patterns)
can be designed in a directivity pattern.
In this paper we combine TFBM and multiple-beamformer into one system to reduce themusical noise caused by conventional TFBM while keeping its versatility. We employ the
TFBM as a pre-separation part and the core of the separation is done by the multiplebeamformer.
We will focus on general description of the proposed system. The detailed description of
single parts of our system [7] is beyond the scope of this article. The organization of thispaper is following. First we give a brief overview about the mixing system in section 2. In
section 3, the proposed system will be introduced and finally the experimental results will beshown in section 4.

2. Mixing process
We consider a mixing process that is taking place in unknown real environment i.e. we haveto deal with convolutive mixturesNx j (t ) = ∑∑ h jk (l ) sk (t − l ),k =1

j = 1,… , M,(1)lwhere x j (t ) is the signal observed by the j-th sensor, t is the discrete time index, sk (t ) is thek-th source signal and h jk (t ) is the impulse response from the k-th source to the j-th sensor.
By applying short time Fourier transform (STFT) the convolutive mixtures (1) are reduced to
instantaneous mixtures at each frequencyNx( f ,τ ) ≈ ∑ h k ( f ) sk ( f ,τ ),

(2)

k =1
T

x( f ,τ ) = [ x1 ( f ,τ ),… , xM ( f ,τ )]isanobservation
vectorandwhereT
h k ( f ) = [h1k ( f ),… , hMk ( f )] is an mixing vector.
In BSS problem the sparseness of the source signals is often considered. Sparse signals are
signals which have only few samples different from zero. Speech signal in time-frequency(TF) domain can be considered as a sparse signal [3] because the energy in voiced segments
is accumulated around the multiples of fundamental frequency resulting in low energy in allother TF slots. Further we consider that the sources sk ( f ,τ ) do not overlapN∏ s ( f ,τ ) = 0, ∀f ,τ .
k =1

k

(3)

The assumption (3) is not satisfied for simultaneous speech signals but it can still be used as
an approximation [3] for independent utterances because the speech signals are sparse in TF
domain and each speaker has different speech characteristics. Using weakened version of (3)as an approximation the mixing model in TF domain (2) becomesx( f ,τ ) ≈ h k ( f ) sk ( f ,τ ),(4)where sk ( f ,τ ) is the dominant source at the TF slot.

3. Proposed approach
We propose BSS method based on multiple-beamformer and TFBM. Block diagram of our
proposed system, shown in Figure 1, extracts one of N signals from M mixtures. TFBM is
exploited as a pre-separation process to design ABFs and to compose ABFs inputs. Each ABF
extracts then one target signal sk (t ) .

⎡ x1 (t ) ⎤x( f ,τ )
⎢
⎥
STFT
⎢
⎥
⎢⎣ xM (t ) ⎥⎦ABFxˆ k ( f ,τ )hˆ k ( f )+yk ( f ,τ )ISTFT yk (t )

ˆ (f)
R
k

yˆ k ( f ,τ )TFBM

JammerCorrelationnˆ k ( f ,τ ) reduction nˆ ′k ( f ,τ )Figure 1: Block diagram of proposed BSS system; ISTFT = inverse STFT.
3.1. Adaptive beamformer
ABF is a set of filters w jk ( f ) that perform spatial filtering. The enhanced output signal in thetime-frequency domain is obtained asyk ( f ,τ ) = w k ( f )xˆ k ( f ,τ ),(5)Twhere w k ( f ) = [ w1k ( f ),… , wMk ( f )] and xˆ k ( f ,τ ) = [ xˆ1 ( f ,τ ),… , xˆM ( f ,τ )] is the input vectorof ABF as described in section 3.3. Now let us consider that the mixing vector h k ( f ) and theobservation vector in the jammer only period (the time period when the source signal sk (t ) is
not active) is known. The filters w jk ( f ) can then be designed by using⎛⎞
R k ( f ) −1 h k ( f )wk ( f ) = ⎜
(6)
⎟,
H
−1
⎝ hk ( f ) R k ( f ) hk ( f ) ⎠where H is the complex conjugate transpose, R k ( f ) = E[n k (f ,τ )n k (f ,τ ) H ] is the correlation
matrix of the observation vector in the jammer only period n k (f ,τ ) and E[] is the mean
operator.
But mixing vector h k ( f ) and correlation matrix of the observation vector in the jammer only
period R k ( f ) are usually not known because measurement of h k ( f ) is not realistic inpractice and the estimation of the jammer only period constitutes a difficult problem
especially when the jammers are non-stationary signals. Therefore by conventional design of
ABF [6] an approximation of mixing vector for an anechoic environment (steering vector)
and correlation matrix of observation vector R ( f ) = E[x(f ,τ )x(f ,τ ) H ] is used instead ofh k ( f ) and R k ( f ) , respectively. This simplification is a standard design procedure although
it lowers the separation performance and makes the blind design impossible.
ˆ ( f ) in
Though all the difficulties we use (6) for designing ABF by estimating hˆ k ( f ) and Rkour proposed system; ^ stands for estimated value.
3.2. Multi-channel TFBM
Conventional TFBM
The principle of conventional TFBM is introduced in [3, 5] in detail. The main idea of TFBM
is based on the equation (4) saying that one source is dominant in each TF slot. If we are able
to estimate the dominant TF slots for each target signal sk ( f ,τ ) we can get a good estimationof the target signals yˆ k (f ,τ ) by simply extracting the dominant TF slots from an arbitrary
selected observation xP ( f ,τ ) , P ∈ {1,..., M} .
TFBM estimates the dominant TF slots by normalization of the observation vector x(f ,τ ) andclustering [5, 7]. Each cluster represents TF slots of an independent source. The estimatedclusters are then used for building a TF binary mask M k ( f ,τ ) . The extraction is then done bymultiplying the TF binary mask M k ( f ,τ ) with observation xP ( f ,τ ) . M k ( f ,τ ) = 1 when asource k is estimated as dominant source in ( f ,τ ) , otherwise M k ( f ,τ ) = 0 . Due to the TF
binary masking the TFBM outputs yˆ k (f ,τ ) are distorted by musical noise.

Multi-channel TFBM
In our system, we use the TFBM as a pre-separation system and the final separation is doneby multiple-beamformer. In order to design the multiple-beamformer the conventional TFBM
must be extended:
1) Estimation of observation in the jammer only period nˆ1 (f ,τ ) . This can be done bycombining the pre-separated signals yˆ k (f ,τ ) in TF domain. For example if N=3, M=3, P=1
and k=1 then the estimated target signal is yˆ1 (f ,τ ) and the estimated observation in jammer
only period is nˆ1 (f ,τ )=yˆ 2 (f ,τ )+yˆ3 (f ,τ ) . These signals are extracted from the mixture x1 (f ,τ ) .
2) Multi-channel extension. The extension can be achieved by multiplying the TF binary
mask M k ( f ,τ ) with whole observation vector x( f ,τ ) . This results in vector of pre-separated
target signal yˆ k (f ,τ )=[ yˆ k1 (f ,τ ),… ,yˆ kM (f ,τ )]T and observation vector in jammer only
period nˆ k (f ,τ )=[nˆk1 (f ,τ ),… ,nˆkM (f ,τ )]T .
3) Estimation of mixing vector hˆ k ( f ) . We estimate hˆ k ( f ) by back-normalization (inverse
process to normalization of observation vector x(f ,τ ) ) as described in [7].

a)

b)

c)
Figure 2: Spatial characteristic of ABF (vertical solid line is the direction of arrival (DOA) of
jammer signals, dash line is the DOA of the target signal, dot-dashed line is the DOA of the
excluded jammer) designed by our proposed system in a) DC, N=3, M=3,
R k ( f ) = E[nˆ k (f ,τ )nˆ k (f ,τ ) H ] ; b) UDC, N=4, M=3, R k ( f ) = E[nˆ k (f ,τ )nˆ k (f ,τ ) H ] - without
jammer reduction; c) UDC, N=4, M=3, R k ( f ) = E[nˆ ′k (f ,τ )nˆ ′k (f ,τ ) H ] - with jammer reduction.
3.3. Jammer reduction
Proposed approach shown in Fig. 1 can be used for both DC and UDC. The output of the
TFBM and the Jammer reduction block form the input signal of ABF

xˆ k (f ,τ )=nˆ ′k (f ,τ )+yˆ k (f ,τ ).
(7)
In DC the block Jammer reduction does not affect the signal. Thus we can writenˆ k (f ,τ )=nˆ ′k (f ,τ ) and xˆ k (f ,τ ) =x(f ,τ ) . Now let us have a look at spatial characteristic of the
ABF for DC in Fig. 2a. It can be seen that the jammers are suppressed well although thedesign is made blindly. However if we use the same approach in the UDC the jammers can
not be suppressed efficiently as shown in Fig. 2b.
In order to fit the minima of spatial characteristics in the direction of arrival of the jammer
signals we reduce the number of jammers in block Jammer reduction to M-1 jammers in UDC.
For example N=4, M=3 and k=1 then nˆ 1 (f ,τ )=yˆ 2 (f ,τ )+yˆ 3 (f ,τ )+yˆ 4 (f ,τ ) andnˆ 1′ (f ,τ )=yˆ 3 (f ,τ )+yˆ 4 (f ,τ ) . yˆ 2 (f ,τ ) was excluded from the vector nˆ 1′ (f ,τ ) . The excluded
signals are determined by the selection criterion. Different selection criteria can be used. In
our approach we exclude the jammer signals with the fewest dominant TF slots in order tominimize musical noise. See Fig. 2c for spatial characteristic of ABF when jammer reduction
is applied.

4. Experiments
We performed experiments for a DC (M=3, N=3) and an UDC (M=3, N=4) in a room with areverberation time of 120 ms, see Fig. 3 for room setup. The source signals were 5-second
English and Japanese utterances. The STFT frame size was L=512, the frame shift was L/4,
and the sampling frequency fs was 8 kHz. The separation performance was evaluated in termsof the signal-to-interference ratio (SIR) and signal-to-distortion ratio (SDR)
E ⎡⎣ ykk (t ) 2 ⎤⎦SIR k = 10 log10
[dB],
(8)
⎡ N
2⎤
E ⎢ ∑ ykb (t ) ⎥⎣ b =1,b ≠ k⎦
2E ⎡⎣ xPk (t ) ⎤⎦[dB],
SDR k = 10 log10(9)2E ⎡( xPk (t ) − γ ykk (t − Δ) ) ⎤⎣⎦where ykb (t ) are the jammer components that appear in the k-th output target signal, ykk (t ) is
the output signal without any contribution from the jammers and xkP (t ) = ∑ l hPk (l ) sk (t − l ) . P
is an arbitrarily selected sensor. Coefficients γ and Δ compensate the amplitude and delay,
respectively.
880 cmMicrophone (120cm height, omni-directional)

Microphone setup
Mic. 2ddMic. 3

S1
S4

300°

S3
135°
S2

50 cm 225°
80 cm

282cm

Figure 3: Room setup.

d=4 cm375 cm0°30°Mic. 1d

200 cm••Loudspeaker (120cm height)
Reverberation time: 120 ms
Room height: 240cm

Table 2 shows the separation results for the DC, when sources S1, S2 and S3 were used, andfor the UDC. We used different BSS approaches in order to compare them. An conventional
design ABF (CDABF), namely using steering vector with given target locations andobservation correlation matrix R ( f ) . Note that separation is not performed blindly. The
conventional TFBM setup corresponds to the approach outlined in the beginning of section
ˆ ( f ) for ABF design.
3.2. Finally, we used our proposed method, which exploits hˆ ( f ) , Rk
The CBABF setup did not achieve good results because the jammer only correlation matrix
ˆ ( f ) was not used and furthermore the steering vector could not reflect the roomRkreverberation or sensor misalignment. The TFBM achieved higher SIR and SDR values thanthe CBABF but the zero padding by the TF binary mask means that we hear large musicalnoise, especially in UDC. On the other hand, the proposed methods achieve higher separation
performance with much less and without musical noise for UDC and DC, respectively.
DC
UDC
Design
MethodSIR [dB]
SDR [dB]
SIR [dB]
SDR [dB]
CBABF
5.1
7.3
2.3
7.2
TFBM
10.9
10.6
9.6
9.0
Proposal
14.6
13.6
9.3
10.8
Table 2: Results for 3 mic. setting; input SIR=-3.1 dB in DC and input SIR=-4.8 dB in UDC.

5. Conclusion
In this paper, we have described a BSS approach applicable for both (over-)determined andunderdetermined cases by assuming source sparseness. In (over-)determined case BSS iscompletely done by the multiple-beamformer, which does not cause musical noise. In
underdetermined case the TFBM affect the multiple-beamformer input by reducing thenumber of jammers. Jammer reduction causes musical noise but in our approach just N-M
jammers are reduced (zero padded) instead of N-1 jammers as in conventional TFBM
approach and therefore the musical noise is significantly reduced. Furthermore our method
provides better separation performance than conventional techniques.

Acknowledgement: The paper has been supported by the National Research Project“Information Society” 1ET301710509 and by the Ministry of Education, Youth and Sports,
project number F2101/2006/61.

References
[1]
[2]

[3]
[4]
[5]
[6]
[7]

S. Haykin, Ed., Unsupervised Adaptive Filtering (Volume I: Blind Source Separation), John
Wiley & Sons, New York, 2000.
S. Makino, H. Sawada, R. Mukai and S Araki, “Blind source separation of convolutive
mixtures of speech in frequency domain,“ IEICE Trans. Fundamentals, vol.E88-A, no.7,
pp.1640-1655, July 2005.
O. Yilmaz, S. Rickard, “Blind separation of speech mixtures via time-frequency masking,”
IEEE Trans. on SP, vol. 52, no. 7, pp. 1830-1847, 2004.
A. Hyavarinen, J Karhunen and E. Oja, Independent Component Analysis, John Wiley &Sons, 2001.
S. Araki, H. Sawada, R. Mukai, S. Makino, “A novel blind source separation method withobservation vector clustering,” in Proc. IWAENC2005, Sept. 2005.
D. Johnson, D. Dudgeon, Array Signal Processing, Prentice Hall, 1993.
J. Cermak, S. Araki, H. Sawada and S. Makino, “Blind speech separation by combining
beamformers and time frequency binary mask,” in Proc. IWAENC2006, Sept. 2006.