Eurospeech 2001 - Scandinavia

Real-time Sound Source Localization and Separation System and Its
Application to Automatic Speech Recognition
Futoshi Asano, Masataka Goto, Katunobu Itou and Hideki AsohAIST (former ETL), Japan

asano@etl.go.jp

Abstract

A real-time sound localization/separation system fornear- eld sound sources was constructed and evaluated ina real oce environment. As for the sound localization,
the experimental results showed that the direction of thetwo sources was estimated with high accuracy while therange of the sources was estimated with moderate accuracy. As for the sound separation, a recognition rate of
70% for an on-line recognizer on a network and of 90%
for an o -line recognizer were achieved, respectively.
1. Introduction

When using an automatic speech recognizer (ASR) in areal environment, its performance su ers deterioration
due to environmental noise or signals from other soundsources. The authors previously proposed a method ofsound source localization/separation using a microphonearray [1]. In this method, assuming a dialog between ahuman and a mobile robot under conditions of sound interference, the sound localization/separation in the neareld range (the robot, the speaker and the interference
are within a range of 1-2 m) was focused on. In the neareld, the wavefront becomes spherical and the range ofthe sources as well as the direction must be taken into
account for accurate source localization/separation. As
for the sound localization, by extending a high-resolution
spatial spectrum estimator, MUSIC [2], to that for a 2dimensional space, the range and the direction of sources
can be estimated. Based on the localization results, aseparation lter is designed using the minimum-variancebeamformer (MVBF) [3].
In this paper, a real-time system which realizes thismethod was constructed using DSPs and was evaluatedin a real oce environment.
2. Method

2.1. Model of Signal
The Fourier transform of the microphone input, Xm (k; t),is modeled asx(k; t) = [X1 (k; t); 1 1 1 ; X (k; t)] = A s(k; t) + n(k; t):(1)
The vector s(k; t) consists of the source spectrum S (k; t)as s(k; t) = [S1 (k; t); 1 1 1 ; S (k; t)] . The symbol k and tdenote the indices for the frequency and the time-frame,respectively. The vector n(k; t) denotes the backgroundnoise. The matrix A is the transfer function matrix, itsMNkTTkn(m; n) element being the transfer function of the direct

path from the nth source to the mth microphone. Thenth column vector of Ak is termed the location vector ofthe nth source.
2.2. Sound Localization
The spatial spectrum is obtained as an average of theMUSIC spatial spectrum asP (r; ) =1KXkkH=kLP (r; ; k):(2)The symbols kL and kH are indices for the lower andthe upper boundaries of the frequency range, and K =kH 0 kL + 1. The symbols r and  denote the range andthe direction, respectively. The MUSIC spatial spectrum
is given by1
P (r; ; k ) = H(3)
ja~k (r; )Enkj2 :wherea~ (r; ) = kaa ((r;r; ))k(4)R = E 3 E01;

(5)

kkk

is the normalized location vector for the scanning point(r; ) [1]. The matrix Enk consists of the eigenvectors
obtained by the eigenvalue decomposition of Rk as
kkkk

where Rk is the spatial correlation matrix de ned as

R = E[x(k; t)x (k; t)]:(6)
The eigenvectors are split as E = [E jE ] where E andE denote the sets of eigenvectors corresponding to theHkskknknksk

N dominant eigenvalues and the rest of the eigenvalues,respectively. The location of sources can be known fromthe peak of this spatial spectrum.

2.3. Sound Separation
By using MVBF, the spectrum of the nth sound sourceis recovered by the following ltering:S^n (k; t) = wH (k )x(k; t)(7)where01w = a^ R R0a^1a^kHn;kkn;kn;k:(8)The vector a^ n;k is the location vector for the nth source
estimated by the sound localization. Two modi ed versions of MVBF were employed in this paper.

Eurospeech 2001 - Scandinavia

MUSIC

Table 1: Di erences of MV1 and MV2.
MV2

absence of target
must be detectedhigh tracking capability
performance depends
on localization accuracy

Correlation Kk
Advantage high noise reduction
Disadvantage

FFTMV1QkA/DA/D,D/A

Table 2: Results of the benchmark test. The processing
time for 1 s data is shown.
Time [s]
0.86
0.34
0.16
0.52

Processor
DSP-B
HostHostDSP-AMV

1 =K01 a^a^ K01a^n;kkHn;kk;

MV

2 =Q01 a^a^ Q01a^k:Memory
B

Figure 2: Architecture of the DSP system.

(9)n;kn;kkHn;kMemoryADSP-B

PCI Bus(8) becomes the maximum likelihood (ML) estimator [3].
The matrix Kk denotes the spatial correlation that isestimated in the period where the target signal Sn (k; t)
is absent. The ML estimator wM V 1 yields a high noise
reduction performance with a relatively small amount ofdata compared with (8) and is termed \MV1" hereafter.
On the other hand, the authors previously proposed
the modi ed MVBF [1], in which the spatial correlation
Rk is replaced by the following synthesized spatial correlation:
Qk = A^ k A^ Hk + I:
(10)
The second modi ed MVBF using this Qk is termed\MV2", and is given byw

SharedMemory

Host PC

When the spatial correlation matrix Kk is used instead of Rk in (8) asw

FilterFigure 1: Block diagram of the proposed system.

DSP-A

FFT & Correlation
MUSICMVBF
Filtering

MVBFCOR(11)n;kAs described above, a salient feature of MV2 is theuse of Qk that is synthesized using the estimate of the^ k . The estimate of the localization can belocalization, A
obtained with less input observation (0.2-0.5s) [1] compared to that for Kk (more than 1 s). Thus, the advantage of this method is its faster capability of tracking
environmental changes compared with MV1. This is adesired feature for applications such as the mobile robot.
The di erences of MV1 and MV2 are summarized in Table 1.
The second term in (10) corresponds to the correlation for the background noise. The parameter functions
as weight in the weighted least-square problem. Whenthe real background noise is high, the reduction performance for the background noise can be raised by employing a larger .

3. System

3.1. Hardware
A block diagram of the system is depicted in Fig. 1. First,
the microphone-array input is Fourier-transformed andthe spatial correlation is calculated by the FFT and COR
modules. Then the spatial correlation Rk is sent to the^ ) is estimated.
MUSIC module and the source location (AIn the MVBF module, the beamformer coecients arecalculated. For MV1, the correlation in the absence ofthe target, Kk , is also estimated in the COR module andis sent to the MVBF module. The length of data used
for estimating Rk and that used for Kk is 1.0 s and 0.5s, respectively, in this paper. The estimated coecientsare transformed to the time domain by Fourier transformand the input signal is ltered in Filter with these timedomain coecients.
The system consists of dual DSPs (TI C6701) and thehost PC (Pentium III 600MHz). The architecture of thesystem is depicted in Fig. 2. The assignment of the computational resources and the results of the benchmark
tests are shown in Table 2.
The microphone array was circular in shape with adiameter of 0.5 m and M = 8 and was mounted on amobile robot XR-4000 as depicted in Fig. 4(b). The frequency range of averaging in (2) was [500, 3000] Hz.
3.2. Location Vector Database
Prior to the operation of the system, the location vectors for the possible source locations were measured as a
database for the source localization and separation. The
data were taken in the range of 60-160 cm as indicatedin Table 3. The total number of the data entries is 492
points. The database is used for ak (r; ) in (4) for the^ k in (10) for thesound localization, and a^ n;k in (9) and A
sound separation. The measured data and the detailsof the measurement are available from the RWCP soundscene database web-site [4].

Eurospeech 2001 - Scandinavia

3.0m

Table 3: Location vector database.
60-90 cm range
100-160 cm range

Direction
every 10 
every 5 

S1Rangeevery 10 cm
every 20 cm

3.3m
5.5m

S2

Table 4: Source Location
RealEstimatedSource 1
Angle Range
1080
1090

Source 2Angle Range
70120701401.4m(a) Con guration of sound sources

3.3. Online ASR System
The above sound source localization/separation system isconnected through a network to the ASR server currently
being developed by the authors as depicted Fig. 3. The
ASR system, denoted as RVCP-niNja hereafter, consistsof a discrete-HMM ASR engine (niNja) [5] and a network
interface with the network protocol specially designed forexchanging speech information through the network (Remote Voice Control Protocol, RVCP) [6]. This ASR system is useful for applications such as the mobile robot employed in this paper which have limited local resources.
By using RVCP, the ASR system itself can be distributed
over several computational resources on a network, a useful feature for the cases with heavy ASR load such as
multiple mobile robots.

(b) Scene of experiment
Figure 4: Experimental setup.

4. Evaluation

4.1. Condition4.2. Results

The evaluation experiment was conducted in an ordinaryoce. The con guration of the source and the microphone array is shown in Fig. 4. The location of the soundsources (loudspeakers) is shown in Table 4. The soundsources, S1 and S2, emit speech and music respectively.
The sound levels of S1, S2 and the background noise at
the position of microphone #1 were 68, 67, 48 [dBA], respectively. The background noise mainly consists of thenoise of the PC fan.

Figure 5 shows the obtained spatial spectrum. From thepeak position of this spectrum, the location of the sources
was estimated. The estimated location is shown in Table 4. It can be seen that the estimation of the directionis accurate while the range was estimated as being a littlelarger. This is considered to be mainly an e ect of roomre ections.
Figure 6 shows the recognition rate for 492 Japanesewords. For the sake of comparison, an o -line continuousHMM speech recognizer (HTK [7] with IPA phonetic
model [8]) was employed as well as RVCP-niNja. When
MV1 was employed, a high recognition rate was achieved.
On the other hand, when employing MV2, the recognition rate was reduced by around 20%. As described inTable 1, this is due to the accuracy of the sound localization. As indicated in Table 3, the location vectors were
measured every 5 in the range of 1-1.6 m. This means
that, even when the localization is the most accurate, theestimated location and the real location could di er by
6 2.5 .
Figure 7 shows the gain of MV2 in the vicinity of theinterference direction. From this gure, it can be seen
that, when the real and the estimated location di er by6 2.5 , the noise reduction performance is reduced toaround 10 dB at the higher frequencies. When higher
performance is required, a database with higher resolu-speech (RVCP)

LANWireless LAN

ASR results
ASR ServerNomad
XR-4000

Figure 3: Online ASR system.

Eurospeech 2001 - Scandinavia

100HTKniNja
Recognition Rate [%]8060

40200Single mic. MV1Method−10−1−20
5−20−15−101000

5−10−2−10−−1520−10−20−10−15−10−10−20−15−10−20−15−10−20

Figure 5: Spatial spectrum obtained by 2D-MUSIC.

−10−20−15Angle [deg]0−15−101000−1−20−15−20 −15−10200Angle [deg]−15−20−5300

0−200−20−15160
1401201008060Range [cm]−1−10 −15−20−10−20Power [dB]−1−150−20−10−15−20−155−5−150−1−20−
1010Peak1, Angle:10 Range:90

0 Peak2, Angle:70 Range:1405−12000

3000
4000
Frequency [Hz]

5000

6000

Figure 7: Gain of MV2 in the vicinity (610 ) of theinterference direction.
Automation of this detection is required.
When employing MV2 for the sound separation, theperformance was reduced by around 20%. This is a tradeo between high noise reduction performance and hightracking capability. MV2 requires only Rk . In the o line experiment, it was shown that the beamformer coefcients were able to be updated every 0.2 s [1]. However,
the evaluation of the tracking capability such as that formoving sources has not yet been conducted and remains
for future study.

MV2

Figure 6: ASR rate.
tion (e.g., every 2.5 ) should be employed.
5. Discussion and Conclusion

In this paper, a real-time system of sound source localization/separation for sources in the near- eld range wasdeveloped and evaluated. The system consists of dual
DSPs and a PC.
Regarding the sound localization, the directions of
the two sound sources were accurately estimated. As forthe range estimation, the resolution was lower comparedwith that of the direction estimation, due to the variation of phase di erence between the microphones being
smaller. The accuracy of the range estimation is evenlower than those of the o -line experiments in [1]. Thisis considered to be due mainly to the higher backgroundnoise and the early re ections as compared with the environment used in [1].
Regarding the sound separation, a high recognition
rate was achieved when MV1 was employed. Future work
on MV1 includes the detection of the presence/absenceof the target signal. MV1 requires both Rk (spatial correlation for the target being present) for determining thebeam (focus) location and Kk (spatial correlation for thetarget being absent) for determining the null location.
In the present study, the presence/absence of the target
signal is indicated to the system by a human operator.

6. References[1] Futoshi Asano, Hideki Asoh, and Toshihiro Matsui,\Sound source localization and separation in neareld," IEICE Trans. Fundamentals, vol. E83-A, no.
11, pp. 2286{2294, November 2000.
[2] R. O. Schmidt, \Multiple emitter location and signal parameter estimation," IEEE Trans. Antennas
Propag, vol. AP-34, no. 3, pp. 276{280, March 1986.
[3] Don H. Johnson and Dan E. Dudgeon, Array signal
processing, Prentice Hall, Englewood Cli s NJ, 1993.
[4] http://tosa.mri.co.jp/sounddb/.
[5] K. Itou, S. Hayamizu, K. Tanaka, and H. Tanaka,\System design, data collection and evaluation of aspeech dialog system," IEICE Trans. INF & SYST.,
vol. E76-D, no. 1, pp. 121{127, Jan. 1993.
[6] Masataka Goto, Katunobu Itou, Tomoyosi Akiba,and Satoru Hayamizu, \Speech completion: New
speech interface with on-demand completion assistance," in Proc. of HCI International 2001, 2001.
[7] http://htk.eng.cam.ac.uk/.
[8] T. Kawahara, T. Kobayashi, K Takeda, N. Minematsu, K. Itou, M. Yamamoto, A. Yamada, T. Utsuro, and K. Shikano, \Japanese dictation toolkit:
Plug-and-play framework for speech recognition
r&d," in Proc. of ICASSP'99, March 1999, pp. I{393.