SPEAKER INDEXING AND SPEECH ENHANCEMENT
IN REAL MEETINGS / CONVERSATIONS
Shoko Araki, Masakiyo Fujimoto, Kentaro Ishizuka, Hiroshi Sawada, Shoji Makino
NTT Communication Science Laboratories, NTT Corporation
2-4 Hikaridai, Seika-cho, Soraku-gun, Kyoto 619-0237, Japan
Speaker indexing

ABSTRACTVAD

This paper presents a speaker indexing method that uses a small
number of microphones to estimate who spoke when. Our proposed
speaker indexing is realized by using a noise robust voice activity
detector (VAD), a GCC-PHAT based direction of arrival (DOA) estimator, and a DOA classifier. Using the estimated speaker indexing information, we can also enhance the utterances of each speakerwith a maximum signal-to-noise-ratio (MaxSNR) beamformer. This
paper applies our system to real recorded meetings / conversations
recorded in a room with a reverberation time of 350 ms, and evaluates the performance by a standard measure: the diarization error rate (DER). Even for the real conversations, which have many
speaker turn-takings and overlaps, the speaker error time was verysmall with our proposed system. We are planning to demonstrate areal-time speaker indexing system at ICASSP2008.

MaxSNR Beamformer
Speech enhancement

Fig. 1. Flow for proposed method.
employed a maximum signal-to-noise-ratio (MaxSNR) beamformer,
which requires “target speaker speaking period” and “target silent(interferences-and-noise) period” information. That is, our enhancement method includes a speaker indexing system.
In this paper, first, we detail our speaker indexing method. Our
speaker indexing system consists of a noise robust voice activity detector (VAD), a GCC-PHAT based direction of arrival (DOA) estimator, and a DOA classifier. That is, our system estimates the
speaker indices by relying on the speaker seat locations. Our system is characterized by a VAD that is robust with respect to all typesof noise (i.e., stationary, non-stationary, and burst noise). In addition, we explain a speech enhancement method with the MaxSNR
beamformer. Although the authors of [2] employ a beamformer as afrontend for the speaker indexing, this paper proposes to utilize thespeaker indexing result for designing the beamformer coefficients.
Then, we report the speaker indexing and enhancement performance of our proposed system. We also evaluate the noise robust
VAD. We utilized real recordings of meetings / conversations in aroom whose reverberation time was 350 ms. We obtained encouraging results even for the recorded conversations, which usually havemore speaker turn-takings and overlaps than meetings.

Index Terms— Speaker indexing, diarization, voice activity detector, maximum SNR beamformer
1. INTRODUCTION
Meeting recognition has been studied [1–5] and it has been pointed
out that speaker indexing (sometimes called “diarization”), i.e., estimating who spoke when, is an important topic. The speaker indexing
information should be useful for such applications as speech recognition during minute taking and speech enhancement.
Let us formulate the task. Suppose that N ≥ 2 speech sources
s1 , . . . , sN are convolutively mixed and observed at M microphones,PP
xj (t) = N
l hjk (l) sk (t − l) + nj (t), j = 1, . . . , M, (1)k=1where hjk (l) represents the impulse response from source k to microphone j, and nj (t) is the observed stationary background noiseat microphone j. The speech sk (t) are intermittent signals. In thispaper we assume that the speakers do not change their seats during
one meeting / conversation. Our goals are (i) to give speaker indicesto each time point t, and (ii) to enhance each speaker utterance, without knowing the number of speakers N , the speech sources sk or the
mixing process hjk .
Recently, some papers (e.g., [1] and related papers) employ several microphones and utilize the time-difference of arrival information between microphones to improve speaker indexing. However,most previous work uses an undesigned microphone array. However, in the meeting, we can employ a small and precisely arranged
microphone array. By using such an array, we can utilize the speaker
position information more accurately, and easily relate the estimatedspeaker position to the meeting’s seating arrangement. Such an arrayis also portable, which is important for a minuting system.
As regards the meeting recognition task, we have proposed aspeech enhancement method for a meeting situation [6]. The paper

1-4244-1484-9/08/$25.00 ©2008 IEEE

Clustering

Featureextraction

2. PROPOSED METHOD
Figure 1 shows the system flow of our proposed method. This section explains each step in Fig. 1 closely.
Our system works in the time-frequency domain. That is, itutilizes the time-frequency representation xj (f, τ ) of the observations xj (t) (1), which is obtained by a short-time Fourier transform(STFT). Here f is a frequency and τ is a time-frame index.
2.1. Speaker indexing
2.1.1. Voice activity detector (VAD)
First, we detect automatically the periods of target human speeches
from a continuously observed signal by using VAD.
A block diagram of the VAD employed in this paper is shownin Fig. 2. In the figure, the VAD is constructed by using two stream

93

ICASSP 2008

Speechsignalfor τ = 1 : end { /* for all frame τ , do online clustering */if exist(centroid) & τ ∈ PS {if min ||ck − q(τ )|| < th1 { /* close enough to existing centroid */then k ← argmink ||ck − q(τ )|| /* find nearest centroid ck */Ck ← Ck ∪ q(τ ) /* add cluster member to cluster Ck */ck ← ck + μq(τ ) /* update centroid ck */}}if mod(τ /F 1)==0 { /* check new centroid(s) for every F 1 frames */if (the number of τ ∈ PS in bini ) > th2 in the last F 2 frames& min ||ck − bi || > th3 { /* if recent features concentrate in a DOA& it is far enough to existing centroids */then create new centroid cK+1 = bi /* K: # of existing clusters */}}}Observed
signalAmbient
noise
PARADE
PARAD

SpeechperiodSKF

Stream combination

VAD

Fig. 2. Block diagram of VAD. PARADE: a Periodic to Aperiodic
component RAtio-based DEtection, SKF: a switching Kalman filter.

speech / non-speech discriminators, i.e., periodic to aperiodic component ratio-based detection (PARADE) [7] and a switching Kalmanfilter (SKF)-based approach [8]. Each stream outputs the likelihood of speech / non-speech discrimination frame by frame, thus
the speech period PS is decided by using the adaptively weighted
sum of each likelihood [9].
PARADE is robust for burst noise and the SKF is robust for stationary and non-stationary noises. Therefore, by integrating them,
we can obtain a VAD that is robust for all types of noises, i.e., stationary noise, non-stationary noise, and burst noise.
Here, the SKF constructs a clean speech / silence state transition model (GMM: Gaussian mixture model) in advance, and sequentially updates the noise model without a priori knowledge of
ambient noise by using a Kalman filter when a signal is observed.
After the noise model updating, a noise adapted model, i.e., a modelwith a speech (clean speech + noise) state and a non-speech (silence+ noise) state is composed by using probability density functions(PDFs) of the clean speech state or the silence state and updated
noise model. With the method, the Kalman filter for noise updating
is formulated by using PDF parameters of the clean speech state or
the silence state. Consequently, two types of estimation (updating)results are given for the noise model by the selection of the clean
speech state or the silence state. This means that the state-space representation of the Kalman filter depends on the state selection, thus
the method has the characteristic of a switching Kalman filter. By
using the adapted model, we can construct the VAD that is robust toa variety of speech and time varying noise.
In this paper, the VAD results are given by binary labeling, i.e.,
the non-speech and the speech frames are labeled as 0 and 1, respectively. As we employ an array of multiple microphones, first weapply the VAD to each channel independently. Then, each outputtedbinary label is unified with a frame by frame logical sum operation.
The speech period PS is determined by PS = {τ |frames labeled as 1}.

2.1.2. Speaker indexing
Then, we partition the speech period PS into each speaker period
Pk (k = 1, · · · , N ). The estimated speaker periods Pk gives us the
speaker indexing result.

Fig. 3. Pseudo code of online clustering. bini is the pre-definedfeature range, bi is a representing vector for bini . In our implementation, bini = [10◦ (i − 1), 10◦ i) is defined in azimuth θ (see section
2.1.2), and bi is the center value of bini . The thresholds in our paper
are th1 = 15◦ and th3 = 30◦ in azimuth θ, F 1 = 20, F 2 = 500,
and th2 = 16 frames.
tion method with the phase transform (GCC-PHAT) [10]X xj (f, τ )x∗j  (f, τ ) j2πf q (τ ) = argmaxq qjj.
e|xj (f, τ )x∗j  (f, τ )|(2)f

We can use a TDOA (column) vector q (τ ), which consists of theqjj (τ ) of all the microphone pairs, however, in this paper we use
the DOA vector q(τ ) as a feature for simplicity of implementation.
The DOA vector q(τ ) is calculated by the TDOA informationq (τ ) and the given microphone coordinate information D [11]:q(τ ) = cD− q (τ ).

(3)−where c is the propagation velocity of the signals and denotes theMoore-Penrose pseudo-inverse. When the source azimuth is θ(τ )and the elevation is φ(τ ), the DOA vector can be written as q(τ ) =[cos θ(τ ) cos φ(τ ), sin θ(τ ) cos φ(τ ), sin φ(τ )]T .
Because we employed the GCC-PHAT, the feature q(τ ) is estimated frame-wise (not time-frequency slot-wise).
Clustering: To divide PS into individual speaker periods Pk , wethen perform clustering for the extracted features q(τ ) of all timeframes τ ∈ PS . In order to apply our method even when the numberof speakers N is unknown, we employ an online clustering algorithm (leader-follower clustering) [12]. The pseudo code is provided
in Fig. 3, where a new centroid is generated when a new speakerappears in a recording.
Each speaker period Pk is determined byτ ∈ Pkifq(τ ) ∈ Ck ,(4)

where Ck is the k-th cluster. This Pk is the speaker indexing result for the real-time display (see Fig. 5). For the speaker indexing
evaluation in Section 3, we removed the short fragments and shortpauses by smoothing Pk , and provide temporal information about
the speech-onset and speech-offset for each speaker.
2.2. Speech enhancement
For speech enhancement, this paper employs a MaxSNR beamformer
as in our previous paper [6]. The design criterion for the beamformer
wk (f ) is to maximize the ratio λ(f ) of the output power between
the target-speaker period Pk and the interference-and-noise-only period P̄k = P − Pk . That is, the MaxSNR beamformer is one of theapplications of the speaker indexing. The beamformer coefficients

Feature extraction: In this paper, we classify the direction of arrival (DOA) q(τ ) in the speech period PS . To estimate the DOA,
first we estimate the time differences of arrival (TDOA) qjj
 (τ ) for
all microphone pairs j and j by using the generalized cross correla-

94

Table 1. Conversation recordings. Each recording duration was five minutes.
Evaluation data ID#Speaker Overlap [%] #Turn-taking #Utterance
Noise sources
presentation rehearsal 1
4
1.4
40
119
PC1,2, projector, laughing voicepresentation rehearsal 2 (Q&A)
3
6.0
75
145
of other speakers
conversation
3
34.8
243
278
PC2
discussion
3
10.8
126
172
PC2, paper noisecrossword puzzle 1
4
18.6
149
185
PC2
crossword puzzle 2
4
13.0
183
218
PC2

Table 2. Experimental results of VAD [%]Aisle

Partition

Sohn
Proposed
FAR FRR Ave. DER FAR FRR Ave. DER
presentation rehearsal 1 41.8 14.3 28.1 20.0 12.8 24.3 18.6 22.2
presentation rehearsal 2 24.3 28.5 26.4 34.2 22.5 19.7 21.1 22.0
conversation
33.2 44.1 38.7 56.6 47.9 21.7 34.8 32.5
discussion
56.4 17.0 36.7 45.7 14.8 22.4 18.6 23.0
crossword puzzle 1
12.6 37.1 24.9 45.3 16.4 13.7 15.1 19.4
crossword puzzle 2
32.0 22.8 27.4 37.3 26.8 13.6 20.2 17.8

305 cm

Evaluation data ID

PC2

Microphones

140 cm4 cm

PC1
Projector

Wall

110 䌣䌭

Desk
790 cmWallstatistical model-based VAD technique.
The feature parameters for the PARADE-based VAD and SKFbased VAD in the proposed method were the 1st order periodic toaperiodic component ratio and the 24th order log-Mel spectra, respectively. These parameters were extracted by using a Hamming
window with a 64 ms frame length and a 32 ms frame shift length.
We trained the silence and clean speech GMMs for PARADE-based
VAD and SKF-based by using phonetically balanced Japanese sentences. The training data consisted of 5,050 utterances spoken by
101 speakers. Each GMM had 32 Gaussian distributions.
The evaluation criteria are the false acceptance rate (FAR) andthe false rejection rate (FRR):

Door

Fig. 4. Room setup. Small ellipses illustrate example speaker places.
The reverberation time was around 350 ms.
wk (f ) is obtained by the eigenvector e1 (f ), which corresponds tothe largest eigenvalue of the generalized eigenvalue problem [6]where RkT (f )P

RkT (f )wk (f ) = λ(f )RkI (f )wk (f ),(5)P= |P1k | τ ∈Pk x(f, τ )xH (f, τ ), RkI (f ) = |P1¯ |kHTτ ∈P¯k x(f, τ )x (f, τ ), x(f, τ ) = [x1 (f, τ ), . . . , xM (f, τ )] , and|P| denotes the number of elements of |P|.
The enhanced speech for the k-th speaker is obtained byyk (f, τ ) = wkH (f )x(f, τ ) .

(6)

3. SYSTEM EVALUATIONS
3.1. Setup
Experiments were performed in the room shown in Fig. 4 whose
reverberation time was around 350 ms. We recorded some conversations by three or four speakers in the room. The duration of each unitof recorded data was five minutes. The distance between the microphone array and the speakers was around 1 m. Personal computers(PC1, 2) and a projector in Fig. 4 could be the noise sources.
Because our recordings were conversations, they contain more
speaker turn-takings and speaker overlaps than usual meeting recordings. Table 1 summarizes the conversation situations. It can beseen that our data contains many speaker turn-takings and overlaps,
which make speaker indexing difficult and require speech enhancement. Reference VAD and speaker indexing labels were generated
by employing a hand-labeled transcription, which includes temporal
information about speech-onsets and speech-offsets.
The sampling rate was 16 kHz, the frame size for STFT was
64 ms, and the frame shift was 32 ms.

FAR = NF A /Nns × 100 [%], FRR = NF R /Ns × 100 [%] ,where Nns , Ns , NF A , and NF R are the total number of non-speechframes, the total number of speech frames, the number of non-speechframes detected as speech frames, and the number of speech frames
detected as non-speech frames, respectively. We also evaluated thediarization error rate (DER) for VAD [3] (in [3], it is defined as DER
for speech activity detection (SAD))
Wrongly estimated speech period length
DER =× 100[%].
Entire speech period length
Table 2 shows the VAD results. As seen in the table, the proposed method significantly improves the average FAR and FRR
rates and the DER compared with Sohn’s method. With the proposed
method, the factors that contributed to the improvement were the updating of Kalman filter-based noise model and the adaptive stream
combination. In particular, the expansion of the applicable noise environment based on the multi stream combination is the most crucial
factor for VAD in real environments.
3.3. Speaker indexing results
We evaluated the diarization error rate (DER),
Wrongly estimated speaker time length
DER =× 100[%],
Entire speaker time length
which is established by NIST [3]. The diarization error includes themissed speaker time (MST), the false alarm speaker time (FAT), andthe speaker error time (SET). We evaluated the DER after smoothing
the speaker indexing result Pk as mentioned in Section 2.1.2. If
the estimated number of speakers outnumbers the true number of
speakers, such ghost speaker periods were regarded as the SET.

3.2. VAD results
In the evaluation, we compare the VAD performance of the proposed
method with that of Sohn’s method [13], which is a widely used95

With Sohn’s VAD
DER MST FAT SET
presentation rehearsal 1 27.2 21.6 4.0 1.7
presentation rehearsal 2 35.0 27.6 4.5 2.9
conversation
61.3 30.6 13.7 17.1
discussion
45.0 24.5 17.7 2.8
crossword puzzle 1
45.0 30.7 7.7 6.6
crossword puzzle 2
47.6 34.3 9.2 4.1
Evaluation data ID

With proposed VAD
DER MST FAT SET
23.9 20.7 3.1 0.1
31.2 23.1 5.6 2.5
38.7 19.0 14.5 5.3
34.8 25.9 6.9 2.0
36.9 18.1 13.6 5.2
32.7 21.3 6.8 4.6

(A)

(C)

(B)
Seat ID

Table 3. Experimental results of speaker indexing [%]Time [Sec.]

Fig. 5. Result image. (A) recording at microphone 1, (B) speakerindexing result, (C) speaker positions with respect to the microphonearray (the center of the circle indicates the array position).

Table 3 summarizes the results. Please note that our recordings
contained many speaker turn-takings and overlaps. With our noise
robust VAD, we obtained better performance in the DER than with
Sohn’s VAD. In our implementation, we had small the SET rate.
This means that our online clustering stage worked successfully. The
DER results were affected by the high MST rate. The high MST rate
was because we employed frame-wise DOA as the feature, which
disregards the speaker overlap in one frame. The MST may be improved by using time-frequency-wise DOA information (e.g.,[11]).

Table 4. Speaker enhancement results in SINR and SDR [dB]. The
averaged input SINR was −3.1 [dB].
With Sohn’s VAD
SINR
SDR
11.8
16.6

With proposed VAD
SINR
SDR
11.9
16.2

3.4. Speech enhancement results
We also evaluated the speech enhancement performance provided
by the MaxSNR beamformer, where the target-speaker period Pk
and the interference-and-noise-only period P¯k were estimated by
the speaker indexing. Here, Pk was not smoothed. For a quantitative evaluation, we utilized simulated conversation data that wasmade by following eq. (1) with impulse responses hjk (l) and PC
and projector noise nj (t) obtained in the room (Fig. 4), and English speech sources sk (t) sampled at 16 kHz. Each data segment
lasted 90 s and the conversation involved three speakers. We tried six
speaker combinations. The performance measures were the signalto-interference plus noise ratio (SINR) and the signal to distortion
ratio (SDR) [6].
Table 4 shows the speech enhancement results. We obtained
high performance, although we did not smooth the speaker indexing
result Pk . Moreover, the performance does not depend on the VAD
methods. This suggests that the MaxSNR beamformer does not necessarily require the low MST and FAT rates, and that it just needs theaccurate speaker frame estimation (i.e., small SET rates).
We also confirmed informally that we can obtain good enhancement performance with our implemented system for the real recorded
conversation. Even when we have more speakers than microphones,
we can still expect that there are fewer speakers in a short data block(e.g, 5 seconds) in a real conversation. Therefore, it was still possible
to employ the MaxSNR beamformer.

consists of noise robust VAD, a GCC-PHAT based DOA estimatorand a DOA classifier, works well even for conversation recordings.
We also reported that speech enhancement with the MaxSNR beamformer can be realized by using roughly estimated speaker indexing
results. Our future work will include improving the speaker indexing
performance by employing time-frequency-wise DOA information.
5. REFERENCES[1] J. M. Pardo, X. Anguera, and C. Wooters, “Speaker diarization for
multi-microphone meetings using only between-channel differences,”in Proc. of MLMI’06 (LNCS 4299). Sept. 2006, pp. 257–264, Springer.
[2] X. Anguera, C. Wooters, and J. Hernando, “Acoustic beamforming
for speaker diarization of meetings,” IEEE Trans. Audio, Speech andLanguage Processing, vol. 15, pp. 2011–2022, Sept. 2007.
[3] http://www.nist.gov/speech/test beds/mr proj/[4] D. Ellis and J. Liu, “Speaker turn segmentation based on betweenchannel differences,” in Proc. of NIST Meeting Recognition Workshop,
2004, pp. 112–117.
[5] C. Busso, P. Panayiotis, G. Georgiou, and S. Narayanan, “Real-timemonitoring of participants’ interaction in a meeting using audio-visualsensors,” in Proc. of ICASSP’07, Apr. 2007, vol. II, pp. 685–688.
[6] S. Araki, H. Sawada, and S. Makino, “Blind speech separation ina meeting situation with maximum SNR beamformers,” in Proc. of
ICASSP’07, Apr. 2007, vol. I, pp. 41–45.
[7] K. Ishizuka, T. Nakatani, M. Fujimoto, and N. Miyazaki, “Noise robust
front-end processing with voice activity detection based on periodic toaperiodic component ratio,” in Proc. of Interspeech ’07, 2007, pp. 230–
233.
[8] M. Fujimoto and K. Ishizuka, “Noise robust voice activity detection
based on switching Kalman filter,” in Proc. of Interspeech ’07, Aug.
2007, pp. 2933–2936.
[9] M. Fujimoto, K. Ishizuka, and T. Nakatani, “A voice activity detection based on adaptive integration of multiple speech feature and signal
decision scheme,” in Proc. of ICASSP ’08, Mar. 2008, (in submitting).
[10] C. H. Knapp and G. C. Carter, “The generalized correlation method
for estimation of time delay,” IEEE Trans. Acoust. Speech and Signal
Processing, vol. 24, no. 4, pp. 320–327, 1976.
[11] S. Araki, H. Sawada, R. Mukai, and S. Makino, “DOA estimation for
multiple sparse sources with normalized observation vector clustering,”
in Proc. of ICASSP’06, May 2006, vol. 5, pp. 33–36.
[12] R. O. Duda, P. E. Hart, and D. G. Stork, Pattern Classification, Wiley
Interscience, 2nd edition, 2000.
[13] J. Sohn, N. S. Kim, and W. Sung, “A statistical model-based voiceactivity detection,” IEEE Signal Processing Letters, vol. 6, no. 1, pp.
1–3, 1999.

3.5. System
In our implementation, the VAD component was written in C andthe speaker indexing parts and display drawing (see Fig. 5) were realized by MATLAB6. 5. In the display (Fig. 5), microphone-1 observation (Fig. 5(A)) and the speaker indexing results (Fig. 5(B), (C))
are drawn. Our speaker indexing system can work in real-time on a
personal computer (AMD Athlon64, 2.4GHz). The real-time factorof the system, which is defined as (the total processing time)/(thetotal recording signal duration), was around 0.6.
4. CONCLUSION
We reported an evaluation of our proposed speaker indexing / enhancement system. Our proposed speaker indexing system, which96