➠➡

ROBUST REAL-TIME BLIND SOURCE SEPARATIONFOR MOVING SPEAKERS IN A ROOM
Ryo Mukai
Hiroshi Sawada
Shoko Araki
Shoji Makino
NTT Communication Science Laboratories, NTT Corporation
2–4 Hikaridai, Seika-cho, Soraku-gun, Kyoto 619–0237, Japanryo,sawada,shoko,maki@cslab.kecl.ntt.co.jp

This paper describes a robust real-time blind source separation (BSS) method for moving speech signals in a room.
Our method employs frequency domain independent component analysis (ICA) using a blockwise batch algorithm
in the first stage, and the separated signals are refinedby postprocessing using crosstalk component estimationand non-stationary spectral subtraction in the second stage.
The blockwise batch algorithm achieves better performance
than an online algorithm when sources are fixed, and the
postprocessing compensates for performance degradation
caused by source movement. Experimental results using
speech signals recorded in a real room show that the proposed method realizes robust real-time separation for moving sources. Our method is implemented on a standard PC
and works in realtime.

system to be completed. These techniques are useful for
realizing low-delay real-time BSS.
The blockwise batch algorithm achieves better separation performance than an online algorithm for fixed source
signals, but the performance declines for moving sources.
As we pointed out in [7], the solution of ICA works likean adaptive beamformer, which forms a spatial null towards
a jammer signal. This characteristic means that BSS using
ICA is fragile as regards a moving jammer signal but robustwith respect to a moving target signal. Utilizing this nature,
we can estimate residual crosstalk components even whena jammer signal moves. To compensate for the degradation
when a jammer signal moves, we employ postprocessing inthe second stage.
Experimental results using speech signals recorded ina room show the effectiveness of the method in realizing
robust real-time separation.

1. INTRODUCTION

2. ICA BASED BSS OF CONVOLUTIVE MIXTURES

Blind source separation (BSS) is a technique for estimating original source signals using only observed mixtures.
The BSS of audio signals has a wide range of applications including noise robust speech recognition, hands-freetelecommunication systems and high-quality hearing aids.
In most realistic applications, the source signal location may
change, and the mixing system is time-varying. Although alarge number of studies have been undertaken on BSS basedon ICA [1, 2], only few studies have been made on BSS for
moving source signals [3, 4, 5, 6]. Indeed an online algorithm can track a time-varying system, however, in general,its performance is worse than a batch algorithm when thesystem becomes stationary. Although we are dealing withmoving sources, we do not want to degrade the performancefor fixed sources.
In this paper, we propose a robust real-time BSS methodthat employs frequency domain ICA using a blockwise
batch algorithm in the first stage, and the postprocessing ofcrosstalk component estimation and non-stationary spectral
subtraction in the second stage. When we adopt a blockwise frequency domain ICA, we need to solve a permutation problem for every block, and this is a time consuming
process especially when the block length is short. We use analgorithm based on analytical calculation of null directions
to solve the permutation problem quickly. Another problem
inherent to batch algorithms is an input-output delay. To
reduce the delay, we use a technique for computing output
signal without waiting for the calculation of the separating

In this section, we briefly review the BSS algorithm thatuses frequency domain ICA and formulate a blockwise
batch algorithm including an online algorithm as a specialcase. We also describe a fast algorithm for solving permutation problems, which is necessary for real-time processing.

ABSTRACT

0-7803-7663-3/03/$17.00 ©2003 IEEE

2.1. Frequency domain ICA
When the source signals are        ), the signals
observed by microphone  are       , and theseparated signals are       ), the BSS model
can be described by the following equations:       (1)  

       

(2)where  is the impulse response from source  to microphone  ,   is the coefficient when we assume that a separating system is used as an FIR filter, and  denotes the
convolution operator.
A convolutive mixture in the time domain corresponds
to instantaneous mixtures in the frequency domain. Therefore, we can apply an ordinary ICA algorithm in the frequency domain to solve a BSS problem in a reverberant environment. Using a short-time discrete Fourier transform(STDFT) for (1), the model is approximated as:          
(3)
where,
is the angular frequency, and represents theframe index. The separating process can be formulated inV - 469ICASSP 2003

➡

➡

Observed
signals

Separated
signals

Ë½

+

 Ï
½

+

Ë¾

+

¾

+

Sourcesignals

Mixing system

½
¾

Separated signals
obtained by ICA

½ 
¾    ).

each frequency bin as:           

(4)is the source

where              
signal in frequency bin ,               denotes the observed signals,              is the estimated source signal,and    represents the separating matrix.    is determined so that      and     become mutually independent.
To calculate the separating matrix  , we use an optimization algorithm based on the minimization of the mutual information of  . The optimal  is obtained by thefollowing iterative equation using the natural gradient approach:               (5)where  is an index for the iteration,  is an identity matrix,
 is a step size parameter,  denotes the averaging operator, and  is a nonlinear function. Because the signals
have complex values in the frequency domain, we use a polar coordinate based nonlinear function, which is effective
for fast convergence especially when the number of input
data samples is small [8]:   

SpectralRefinedsubtractor output

 

´µ
½ 

½´µ  

½´µ 

Separating system

Fig. 1. Model of BSS system ( 





´µ
½ 



     

Fig. 2. Postprocessing for removing crosstalk component from  .
2.3. Scaling and permutation
Once we have completed the ICA for all frequencies, weneed to solve the permutation and scaling problems. Since
we are handling signals with complex values, the scaling
factors are also complex values. Thus the scaling can bedivided into phase scaling and amplitude scaling.
We use a directivity pattern based method to solve thepermutation and phase scaling problems. When we consider
a separating system as a microphone array, we can write directivity patterns for every frequency bin. The permutation
problem is solved so that the null directions are aligned. We
can estimate the directions of the source signals from thealigned directivity patterns, and the phase scaling problem
is solved so that the phase response of the estimated source
direction becomes zero.
In the following sections, we consider a two-input, twooutput convolutive BSS problem, i.e.,  == 2 (Fig. 1).
When  and the distance between the microphones
is sufficiently small to avoid spatial aliasing, the null directions    can be calculated analytically as:(6)where  is a gain parameter that controls the nonlinearity.
2.2. Blockwise batch algorithm
In order to track the time-varying mixing system, we update the separating matrix for each time block           , where  is the block size, and 
represents the block index  .
Koutras et al. have proposed a similar method in thetime domain [4]. When  equals the STDFT frame length,
this procedure can be considered an online algorithm in thefrequency domain.
We use the separating matrix of the previous block asthe initial iteration value for a new block, i.e.,          , where  is the number of iterations for (5).
We use a set of two null beamformers as the initial matrix     for the first block.
The batch algorithm has an inherent delay, because thecalculation of  needs to wait for the arrival of a data
block. Moreover, the calculation itself also takes time.
However, when the calculation is completed within  and
we use for separation of the signals in  , we can
avoid the delay for waiting and calculation. This technique
can reduce the input-output delay and is suitable for lowdelay real-time applications.

 ½    ¾           




 

  



(7)where      is an -th row vector of   , 
is the distance between microphones and  is the velocity
of sound [9]. This method does not require the directivity
pattern to be scanned, thus we can solve the permutation
problem quickly.
The amplitude scaling problem is solved by using
a slightly modified version of the method described in[10]. We calculate the inverse of the separating matrices    , and decide the scaling factors so that the normsof each column of     become uniform.
3. POSTPROCESSING FOR REFININGSEPARATED SIGNALS
In this section, we briefly summarize the procedure for estimating and subtracting the residual crosstalk component.
The algorithm is described in detail in [11]. Figure 2 showsa block diagram of the algorithm.
We consider that  is a target signal and  is a jammer
signal. The separated signal   consists of a straight component  derived from   and a crosstalk component  derived from  . If  is removed from   , the separation
performance improves.
We introduce FIR filters              in each frequency bin, where  is the lengthV - 470

➡

➡

4.45 mMicrophones(height: 1.39 m) -70-4030

3.55 m4cm

Table 1. Experimental conditions

Loudspeakers(height: 1.35 m)
TargetA signal

B

CommonSampling rate = 8 kHz
Window = hanning
Reverberation time  =130 ms
Frame length  = 1024 point (128 ms)
Frame shift = 256 point (32ms)
 = 100.0
Number of iterations  = 100
Block size  = 1 s
Frame length  = 1024 point (128 ms)
Frame shift = 64 point (8 ms)

ICA part

1.15 m70

2.25 mC
Jammersignal1.75 m D
Room height: 2.50 m

Postprocessing
partFig. 3. Layout of room used in experiments.

of the filter. We assume that the power of      canbe approximated as the output of the filter whose input is   . This is formulated as follows:           (8)The filters are updated by the following selectively normalized LMS algorithm.
     
(9)

17
16




Æ       
) (if(otherwise)










 



avoid numerical instability when is very small.
We estimate the power of the residual crosstalk component using (8) and (9), and finally, we obtain an estimationof the straight component as  by the following spectral
subtraction procedure:    (10)         
 (if       )(otherwise)








 







SIR (dB)

13

Blockwise batch

11

Online0.5

12

Block size Ì (s)Fig. 4. Average and standard deviation of SIR for fixedsources                    is an input vector and                is an estimation error. Here, is a step size parameter and Æ is a positive constant to



14

12



where



15

10
0.128 0.2   Æ  













4. EXPERIMENTS
4.1. Experimental conditions
To examine the effectiveness of the proposed method, we
carried out experiments using speech signals recorded ina room. The reverberation time of the room was 130 ms.
We used two omni-directional microphones with an interelement spacing of 4 cm. The layout of the room is shownin Fig 3. The target source signal was first located at A, andthen moved to B at a speed of 30 deg/s. The jammer signal
was located at C and moved to D at a speed of 40 deg/s.
The step size parameter  in (5) affects the separation
performance of BSS when the block size changes. We chose to optimize the performance for each block size. Other
conditions are summarized in Table 1.

We assumed the straight component  as a signal, andthe difference between the output signal and the straightcomponent as interference. We defined the output signal-

   

to-interference ratio (SIR  ) in the time domain as follows:       (dB)
Similarly, the input SIR (SIR ) is defined as,       SIR
       (dB)
SIR

 



   

 









 









 

(11)

(12)

We use SIR  SIR  SIR as a performance measure.
This measurement is consistent with the performance evaluation of BSS in which the crosstalk component is assumed
as interference. We measured SIRs with 30 combinations ofsource signals using three male and three female speakers,
and averaged them.
4.2. Performance for fixed sources
Although we are dealing with moving sources, we do notwant the performance for fixed sources to deteriorate. First,
we measured the BSS performance using ICA without postprocessing. Figure 4 shows the average and standard deviation of SIR for fixed sources (the target is at A and the
jammer at C in Fig. 3). This indicates that the blockwise
batch algorithm outperforms the online algorithm (in which
 is tuned to optimize the performance), when we use theupdate equation (5). In addition, the deviation of the batchalgorithm is smaller than that of the online algorithm. This
is why we adopt the blockwise batch algorithm in the firststage. We used  = 1.0 sec. in the following experiments.
4.3. Moving target and moving jammer
Before considering the result obtained with the postprocessing method, we investigate the BSS performance for moving sources using the blockwise batch algorithm. Figure 5
shows the SIR for a moving target (solid line) and for a moving jammer (dotted line). We can see that the SIR is not

V - 471

➡

➠

20

Blockwise batchwith postprocessing(proposed method)

20

Moving target

SIR (dB)

15

SIR (dB)

15

10

10

Moving jammer

5

5

Move
0

0

2

4

6

8

10

12

14

16

18

0

20

Time (s)Gain (dB)

0
-5
-10
-15
-20
-25
-30

-40

-20

0

Direction

20

(deg)

4000
3500
3000
2500
2000
1500
1000
40

60

500
80

4

6

8

10

12

14

16

18

20

0

)

Hz

y(

nc

e
qu

Fig. 7. Effect of postprocessing. Jammer signal moved fromC to D at 10 sec.
nique is useful for solving convolutive BSS problems in
realtime. Postprocessing using crosstalk component estimation and non-stationary spectral subtraction improves theseparation performance and reduces the performance deterioration when a jammer signal moves. Experimental results
using speech signals recorded in a room showed the effectiveness of the proposed method.

5

-60

2

Blockwise batch
without postprocessing

Time (s)Fig. 5. SIR of blockwise batch algorithm without postprocessing. Target and jammer signals moved at 10 sec.

-80

0

Move

Fre

Fig. 6. Directivity pattern of separating system obtained byfrequency domain ICA
degraded even when the target moves. By contrast, jammer
movement causes a decline in the SIR.
This can be explained by the directivity pattern of theseparating system obtained by ICA. The solution of frequency domain BSS works in the same way as an adaptive
beamformer, which forms a spatial null towards a jammer
signal (Fig. 6). Because of this characteristic, BSS using
ICA is robust as regards a moving target signal but fragile
with respect to a moving jammer signal.
4.4. Effect of postprocessing
The most important factor when estimating the crosstalkcomponent   using (8) and (9) is  , and  is estimatedrobustly even when  moves, because  is a target signal for  . Therefore, postprocessing works robustly evenwhen the jammer signal  moves.
Figure 7 shows the SIR of blockwise batch algorithmwith postprocessing when the jammer signal moves (solidline). We can see that the SIR is improved by the postprocessing, and the drop of the SIR when the jammer moves isreduced. This result shows that our postprocessing method
can compensate the fragility of the blockwise batch algorithm when a jammer signal moves. Although crosstalkcomponents still remaining in the postprocessed output signal sometimes make a musical noise, the power is muchsmaller than ordinary spectral subtraction.
5. CONCLUSION
We proposed a robust real-time BSS method for moving
source signals. The combination of the blockwise batch
and the postprocessing realizes a robust low-delay real-time
BSS. We can solve a permutation problem quickly by using analytical calculation of null directions, and this tech-

ACKNOWLEDGEMENT
We thank Dr. Shigeru Katagiri for his continuous encouragement.
6. REFERENCES[1] A. J. Bell and T. J. Sejnowski,“An informationmaximization approach to blind separation and blind deconvolution,” Neural Computation, vol. 7, no. 6, pp. 1129–
1159, 1995.
[2] A. Hyvärinen, J. Karhunen, and E. Oja, Independent Component Analysis, John Wiley & Sons, 2001.
[3] J. Anemüller and T. Gramss, “On-line blind separation ofmoving sound sources,” in Proc. of Intl. Conf. on Independent Component Analysis and Blind Source Separation(ICA’99), 1999, pp. 331–334.
[4] A. Koutras, E. Dermatas, and G. Kokkinakis, “Blind speech
separation of moving speakers in real reverberant environment,” in Proc. of ICASSP’00, 2000, pp. 1133–1136.
[5] I. Kopriva, Z. Devcic, and H. Szu, “An adaptive short-timefrequency domain algorithm for blind separation of nonstationary convolved mixtures,” in Proc. of IJCNN’01, 2001,
pp. 424–429.
[6] K. E. Hild II, D. Erdogmus, and J. C. Principe, “Blind source
separation of time-varying, instantaneous mixtures using an
on-line algorithm,” in Proc. of ICASSP’02, 2002, pp. 993–
996.
[7] S. Araki, S. Makino, R. Mukai, and H. Saruwatari, “Equivalence between frequency domain blind source separationand frequency domain adaptive null beamformers,” in Proc.
of Eurospeech’01, 2001, pp. 2595–2598.
[8] H. Sawada, R. Mukai, S. Araki, and S. Makino, “Polar coordinate based nonlinear function for frequency-domain blindsource separation,” in Proc. of ICASSP’02, 2002, pp. 1001–
1004.
[9] H. Sawada, R. Mukai, S. Araki, and S. Makino, “A robust
approach to the permutation problem of frequency-domain
blind source separation,” in Proc. of ICASSP’03, 2003, submitted.
[10] F. Asano and S. Ikeda, “Evaluation and real-time implementation of blind source separation system using time-delayed
decorrelation,” in Proc. of Intl. Workshop on Independent
Component Analysis and Blind Signal Separation (ICA’00),
2000, pp. 411–415.
[11] R. Mukai, S. Araki, H. Sawada, and S. Makino, “Removalof residual crosstalk components in blind source separation
using LMS filters,” in Proc. of NNSP’02, 2002, pp. 435–444.

V - 472