A DOA BASED SPEAKER DIARIZATION SYSTEM FOR REAL MEETINGS
Shoko Araki, Masakiyo Fujimoto, Kentaro Ishizuka, Hiroshi Sawada, and Shoji Makino
NTT Communication Science Laboratories, NTT Corporation
2-4 Hikaridai, Seika-cho, Soraku-gun, Kyoto 619-0237, Japan
Email: shoko@cslab.kecl.ntt.co.jp

ABSTRACTThis paper presents a speaker diarization system that estimates who
spoke when in a meeting. Our proposed system is realized by usinga noise robust voice activity detector (VAD), a direction of arrival(DOA) estimator, and a DOA classifier. Our previous system utilized the generalized cross correlation method with the phase transform (GCC-PHAT) approach for the DOA estimation. Because the
GCC-PHAT can estimate just one DOA per frame, it was difficult
to handle speaker overlaps. This paper tries to deal with this issue
by employing a DOA at each time-frequency slot (TFDOA), and reports how it improves diarization performance for real meetings /conversations recorded in a room with a reverberation time of 350
ms.

GCC
-PHAT

[Previous method]
Clustering

VAD
Centroids

DOA
calculation

Thresholding[Proposed method I]

Index Terms— diarization, voice activity detector, direction ofarrivalFig. 1. Block diagram of previous method (framed by dashed line)
and proposed method (Method I).

1. INTRODUCTION

portable: it can run in real-time and it uses a small number of microphones. However, the system has a problem: it estimates just one
DOA per frame, because the previous approach employed the GCCPHAT. This means that only one speaker is detected even if multiple
speakers spoke in a frame. This causes a lot of missed speaker time(false rejection in VAD terminology), and degrades the performance.
Some other approaches by the ICSI project [1] and the CHIL project[2] also employ the GCC-PHAT technique, and therefore they also
experience the same problem as our approach.
In this paper, we try to improve the performance of our diarization system by using the DOA at each time-frequency slot (TFDOA).
The authors of [7] have successfully employed the TFDOA in aspeech enhancement scenario. In addition to the TFDOA, we try
to utilize the amplitude information of observations at each timefrequency slot, and a probabilistic representation of the VAD results.
The experimental results obtained for real recordings of meetings/ conversations show that such refinements improve the diarization
performance.

Meeting recognition has been studied [1, 2, 3, 4, 5] and it has been
pointed out that speaker diarization, i.e., estimating who spoke when,is an important topic. Speaker diarization information should be useful for such applications as speech recognition during minute taking
and speech enhancement.
Let us formulate the task. Suppose that N ≥ 2 speech sources
s1 , . . . , sN are convolutively mixed and observed at M microphones,PP
xj (t) = N
k=1
l hjk (l) sk (t − l) + nj (t), j = 1, . . . , M, (1)where hjk (l) represents the impulse response from source k to microphone j, and nj (t) is the observed stationary background noiseat microphone j. Speech sk (t) consists of intermittent signals. In
this paper we assume that the speakers do not change their seats during one meeting / conversation. Our goal is to estimate “who spokewhen” at each time point t, without knowing the number of speakers
N , the speech sources sk or the mixing process hjk . We work in thetime-frequency domain. That is, we utilize the time-frequency representation xj (f, τ ) of the observations xj (t) (1), which we can obtainby a short-time Fourier transform (STFT). Here f is a frequency andτ is a time-frame index.
Recently, we proposed a diarization system based on a noise robust voice activity detector (VAD), a generalized cross correlation
method with a phase transform (GCC-PHAT) based direction of arrival (DOA) estimator, and a DOA classifier [6]. That is, our diarization system relies on the speaker seat locations. The system worked
very well for real meetings / conversations, even when the system
did not know the number of speakers. The system is simple and978-1-4244-2338-5/08/$25.00 ©2008 IEEE29

2. PREVIOUS METHODThis section describes our previous approach [6], and points out its
problems.
2.1. Method
Figure 1 shows the system flow of our previous method [6]. With
the method, first the speech periods PS = {τ |bv (τ ) = 1} are estimated from a continuously observed signal by using a VAD, thenthe speech periods PS are determined for each speaker period Pk

HSCMA 2008

whereObserved
signalSpeechsignalφk (θ)=

1
0

if |θ − θk | ≤ thotherwise,(6)

θk is the centroid of the k-th cluster, and th is a threshold.

Ambient
noise
PARADE
PARAD

SpeechperiodStream combination

2.2. Problems with previous approachSKF

2.2.1. Problem 1: Frame-wise DOA

VAD

Because the previous approach employed the GCC-PHAT, it estimates one DOA per frame. The system cannot estimate multiple
DOAs even if there are some speakers in a frame. This causes a lotof missed speaker time (false rejection in VAD terminology), anddegrades the performance.

Fig. 2. Block diagram of VAD. PARADE: a Periodic to Aperiodic
component RAtio-based DEtection, SKF: a switching Kalman filter.
(k = 1, · · · , N ) by classifying the direction of arrival (DOA) information. That is, our previous method for estimating “who spokewhen” is based on the speaker positions.
As the VAD, we utilized the MUlti Stream Combination of Likelihood Evolution of VAD (MUSCLE-VAD) [8], which integrates
multiple speech features and a signal decision scheme. A block diagram of the VAD is shown in Fig. 2. The VAD is constructed byusing two stream speech / non-speech discriminators, i.e., periodic
to aperiodic component ratio-based detection (PARADE) [9] and aswitching Kalman filter (SKF)-based approach [10]. Each stream
outputs the likelihood of speech / non-speech discrimination frameby frame, thus the speech period PS is decided by using the adaptively weighted sum of each likelihood [8]. PARADE is robust for
burst noise and the SKF is robust for stationary and non-stationary
noises. Therefore, by integrating them, we can obtain a VAD thatis robust for all types of noises, i.e., stationary noise, non-stationary
noise, and burst noise.
The VAD results bv (τ ) are given by binary labeling, i.e., thespeech and non-speech frames are labeled 1 and 0, respectively. As
we employ multiple microphones, first we apply the VAD to eachchannel independently. Then, each outputted binary label is unified
with a frame by frame logical sum operation.
The DOA estimation was performed using the GCC-PHAT [11]X xj (f, τ )x∗j 0 (f, τ ) 2πf q000 (τ ) = argmaxq 0
e
,
(2)
qjj
|xj (f, τ )x∗j 0 (f, τ )|2.2.2. Problem 2: Directional noise
Thanks to the VAD, noise periods (non-speech periods) are estimated successfully in most of the time frames. However, directional
noise is sometimes detected as a speaker if the directional noise is detected as speech in terms of VAD, or if it becomes dominant in termsof the GCC-PHAT calculation when the other speaker is speaking
with small power, e.g. speech onset/offset, and unvoiced speech part.
Such false detections increase the false alarm speaker time (FAT, see
Section 4.2) and degrade overall performance.
2.2.3. Problem 3: Binary VAD
The VAD performance changes when we change the threshold forthe speech / non-speech discrimination. And, what is worse, overall
diarization performance depends on the VAD performance. If thefalse acceptance rate (FAR) of the VAD is high, there is little missed
speaker time (MST, see Section 4.2) but increased FAT. On the other
hand, when the false rejection rate (FRR) of the VAD is high, there
is little FAT but MST increases. Therefore, it is difficult to set apreferable VAD threshold for good diarization performance.
3. PROPOSED METHODSThis section proposes three refined methods to handle the problems
discussed in the previous section.

f0
where qjj
0 (τ ) is the time differences of arrival (TDOA) in between
a microphone pair j − j 0 . The DOA vector q(τ ) is calculated by the0
TDOA information q0 (τ ), which consists of the qjj
0 (τ ) of all the
microphone pairs, and the given microphone coordinate information
D [12]:q(τ ) = cD+ q0 (τ )(3)where c is the propagation velocity of the signals and + denotes theMoore-Penrose pseudo-inverse. When the source azimuth is θ(τ )and the elevation is φ(τ ), the DOA vector can be written asq(τ ) = [cos θ(τ ) cos φ(τ ), sin θ(τ ) cos φ(τ ), sin φ(τ )]T .
Here, we employ only the azimuth θ(τ ) for simplicity.
The individual speaker periods Pk are determined by clustering
the estimated DOA θ(τ ) at all speech frames τ ∈ PS :
τ ∈ Pkifq(τ ) ∈ Ck ,(4)

where Ck is the k-th cluster. This Pk is the speaker diarization result.
Equation (4) can be rewritten asτ ∈ Pkifbv (τ )φk (θ(τ )) = 1

(5)

30

3.1. Method I: Employ TFDOA
To solve problem 1, we employ the TFDOA. That is, instead of (2),the TDOA is estimated at each time-frequency slot by
1
0
0 (f, τ ) =arg [xj (f, τ )x∗j 0 (f, τ )]
(7)
qjj
2πf
and the TFDOA q(f, τ ) is estimated in the same way as (3). Here,we employ only the azimuth θ(f, τ ) for simplicity.
If τ is a speech frame τ ∈ PS and a sufficient number of frequency bins f have a DOA adjacent to the k-th centroid θk in frame
τ , then speaker k is regarded as speaking in the frame τ . That is, theindividual speaker periods Pk are determined byτ ∈ Pkifbv (τ )λ1k (τ ) > th1,

(8)

X

(9)

where th1 is a threshold,λ1k (τ ) =φk (θ(f, τ ))f

is a DOA voting, and φk (θ) has the same definition as (6). Figure 1
also shows a block diagram of this proposed method I.

AislePartition

305 cm

4. SYSTEM EVALUATION

PC
Wall

4.1. Setup

140 cmMicrophones

Wall

Desk

4 cm
3

790 cmMicrophone spacing
and direction definitions
6

9
12

Door

Fig. 3. Room setup. Small ellipses illustrate example speaker locations. The reverberation time was around 350 ms.
Table 1. Conversation recordings. Each recording lasted five minutes.
Evaluation data ID #Speaker Overlap [%] #Turn-taking #Utterance
crossword puzzle 1
4
18.6
149
185
crossword puzzle 2
4
13.0
183
218
discussion
3
10.8
126
172
conversation
3
34.8
243
278

Experiments were performed in the room shown in Fig. 3 whose
reverberation time was around 350 ms. We recorded some conversations between three or four speakers in the room. The duration ofeach unit of recorded data was five minutes. The distance between
the microphone array and the speakers was around 1 m. The personal computer (PC) in Fig. 3 could be the noise source. Figure 3
also illustrates our direction definition.
Because our recordings were conversations, they contain more
speaker turn-takings and speaker overlaps than usual meeting recordings. Table 1 summarizes the conversation situations. It can be seenthat our data contains many speaker turn-takings and overlaps, whichmake speaker diarization difficult. Reference diarization labels weregenerated by employing a hand-labeled transcription, which includes
temporal information about the speech onsets and speech offsets ofeach speaker.
The sampling rate was 16 kHz for VAD and 8 kHz for DOA
estimation. The frame size for STFT was 64 ms, and the frame shift
was 32 ms. The thresholds for the proposed methods were th1 = 40,
th2 = 0.2, and th3 = 0.4. They were decided in our preliminary
experiments.
4.2. Evaluation measure

3.2. Method II: Employ amplitude weight
To suppress the noise influence, here we assume that the signal tonoise ratio is sufficiently high, i.e., the amplitude of the speech spectrum is greater than that of the noise spectrum. By exploiting thisassumption, method II employs the amplitude weight for the DOA
voting:X |x1 (f, τ )|P
λ2k (τ ) =φk (θ(f, τ ))(10)
f |x1 (f, τ )|fwhere φk (θ) has the same definition as (6). From the amplitude
weight, it can be expected that the speech time-frequency slot garners a large number of votes and noise time-frequency slot receives
few votes.
The individual speaker periods Pk are determined in the same
way as in Method I:τ ∈ Pkifbv (τ )λ2k (τ ) > th2(11)where th2 is a threshold.
3.3. Method III: Employ probabilistic VAD
To handle the third problem, here we utilize the probability pv (τ ) of
speech activity at each frame τ . Such a probability can be estimatedwith a VAD. Here, we also employ the MUSCLE-VAD introducedin Section 2.1, and utilize the forward probability (see (11) in [8]).
As we employ an array of multiple microphones, first we calculate
the probability for each channel independently and then take the average.
The individual speaker periods Pk are determined byτ ∈ Pkif

pv (τ )pk (τ ) > th3(12)where th3 is a threshold and pk (τ ) is the probability of speaker kspeaking. Here as the probability pk (τ ), we employ the normalized
DOA voting result
λ2k (τ )pk (τ ) ≡ λ3k (τ ) = P.
(13)k λ2k (τ )31

We evaluated performance with the diarization error rate (DER),
Wrongly estimated speaker time length
DER =× 100[%],
Entire speaker time length
which was established by NIST [3]. The diarization error includes
the missed speaker time (MST), the false alarm speaker time (FAT),
and the speaker error time (SET) [3]. If the estimated number of
speakers exceeds the true number of speakers, such ghost speaker
periods were regarded as the SET. We also evaluated the DER aftersmoothing (hangover) the speaker diarization result Pk , where shortfragments and short pauses were removed.
4.3. Results and discussion
Table 2 summarizes the results. With the previous method, whichcannot estimate multiple DOAs in a frame, we had a large MST rate.
When using Method I, where multiple DOAs can be estimated, theMST rate decreases. Figure 4 shows an example illustration. With
the previous method (Fig. 4 (b)) we can estimate only one DOA foreach frame. On the other hand, several DOAs can be estimated in aframe with Method I (Fig. 4 (c)). However, the FAT rate becomes alittle worse with Method I. One reason for this is that the directional
noise (PC) can be falsely detected easily by Method I. Figure 5 (c)shows an example of such false detection of the directional noise(from direction 8).
With Method II, where we utilize the amplitude weight, we can
improve the FAT rate. Figure 5 shows this result. With the previous
method (Fig. 5 (b)) and the proposed Method I (Fig. 5 (c)), somedirectional noise from direction 8 was detected for 6 to 8 seconds.
By contrast, with Method II (Fig. 5 (d)), the directional noise is notdetected thanks to the amplitude weight.
The performance further improved when we used Method III,
which employs the probabilistic VAD result. When we test “crossword puzzle 2” data with Method II (Table 2 (b)), the FAR of the
VAD was 26.8% and the FRR of the VAD was 13.6%. If we use
the VAD of FAR=45.7% and FRR=2.8%, MST, FAT, SET and DER

Table 2. Diarization results [%] for each evaluation data. (Values inbracket indicate the DER after hangover.)(a) Crossword puzzle 1
MethodMST FAT SET DER
Previous
35.3 8.1 9.7 53.2 (41.1)
Method I
31.4 10.5 6.4 48.3 (37.0)
Method II 31.2 8.2 3.9 43.2 (28.5)
Method III 25.1 6.5 3.6 35.2 (21.9)

(b)
(c)

(b) Crossword puzzle 2
MethodMST FAT SET DER
Previous
28.4 12.7 8.9 50.1 (40.0)
Method I
24.2 17.8 5.2 47.2 (36.1)
Method II 23.9 15.1 3.1 42.2 (29.2)
Method III 19.6 13.4 2.6 35.5 (25.0)(c) Discussion
MethodMST FAT SETPrevious
43.5 5.1 2.1
Method I
38.5 9.0 1.5
Method II 37.8 7.8 1.5
Method III 38.5 5.5 1.5

DER50.6 (35.9)
49.0 (31.9)
47.0 (32.6)
45.5 (29.9)(d) Conversation
MethodMST FAT SETPrevious
38.2 7.5 8.0
Method I
30.0 13.1 7.6
Method II 30.4 12.3 6.2
Method III 33.2 9.5 6.1

DER
53.7 (40.7)
50.7 (37.7)
48.9 (32.3)
48.8 (34.3)(a)Fig. 4. Diarization result examples. (a) ground truth label, (b) withprevious method and (c) with Method I. Gray dots indicate noise
frames.

(a)

(b)

(c)

were 5.6, 30.7, 5.3, and 41.6, respectively. That is, the diarization
performance depends on the VAD performance as pointed out in
Section 2.2.3. With Method III, without tuning the VAD, we canobtain good diarization performance.

(d)5. CONCLUSION
We proposed methods for improving the performance of our meeting diarization system. Instead of the GCC-PHAT approach, we utilized the DOA at each time-frequency slot for the diarization. This
refinement reduces the missed speaker time, and improves the performance. In addition, we also showed that the use of the observation amplitude at each time-frequency slot effectively disregards
directional noise. We also showed that the use of a probabilistic representation of the VAD results produces high levels of performance.
6. ACKNOWLEDGMENT
The authors thank Mr. Taku Hasegawa of Toyohashi University ofTechnology for his help in conducting the experiments.
7. REFERENCES[1] X. Anguera, C. Wooters, and J. Hernando, “Acoustic beamforming
for speaker diarization of meetings,” IEEE Trans. Audio, Speech andLanguage Processing, vol. 15, pp. 2011–2022, Sept. 2007.
[2] D. Macho, J. Padrell, A. Abad, C. Nadeu, J. Hernando, J. McDonough,M. Wölfel, U. Klee, M. Omologo, A. Brutti, P. Svaizer, G. Potamianos, and S. Chu, “Automatic speech activity detection, source localization, and speech recognition on the CHIL seminar corpus,” in Proc.
of ICME’05, July 2005, pp. 876–879.
[3] http://www.nist.gov/speech/test beds/mr proj/[4] D. Ellis and J. Liu, “Speaker turn segmentation based on betweenchannel differences,” in Proc. of NIST Meeting Recognition Workshop,
2004, pp. 112–117.

32

Fig. 5. Diarization result examples. (a) ground truth label, (b) withprevious method, (c) with Method I and (d) with Method II. Gray
dots show noise frames.
[5] C. Busso, P. Panayiotis, G. Georgiou, and S. Narayanan, “Real-timemonitoring of participants’ interaction in a meeting using audio-visualsensors,” in Proc. of ICASSP’07, Apr. 2007, vol. II, pp. 685–688.
[6] S. Araki, M. Fujimoto, K. Ishizuka, H. Sawada, and S. Makino,“Speaker indexing and speech enhancement in real meetings / conversations,,” in Proc. of ICASSP’08, Mar. 2008, (to appear).
[7] M. L. Seltzer, I. Tashev, and A. Acero, “Microphone array post-filter
using incremental Bayes learning to track the spatial distributions ofspeech and noise,” in Proc. of ICASSP’07, Apr. 2007, vol. I, pp. 29–32.
[8] M. Fujimoto, K. Ishizuka, and T. Nakatani, “A voice activity detection based on adaptive integration of multiple speech feature and signal
decision scheme,” in Proc. of ICASSP ’08, Mar. 2008, (to appear).
[9] K. Ishizuka, T. Nakatani, M. Fujimoto, and N. Miyazaki, “Noise robust
front-end processing with voice activity detection based on periodic toaperiodic component ratio,” in Proc. of Interspeech ’07, 2007, pp. 230–
233.
[10] M. Fujimoto and K. Ishizuka, “Noise robust voice activity detection
based on switching Kalman filter,” in Proc. of Interspeech ’07, Aug.
2007, pp. 2933–2936.
[11] C. H. Knapp and G. C. Carter, “The generalized correlation method
for estimation of time delay,” IEEE Trans. Acoust. Speech and Signal
Processing, vol. 24, no. 4, pp. 320–327, 1976.
[12] S. Araki, H. Sawada, R. Mukai, and S. Makino, “DOA estimation for
multiple sparse sources with normalized observation vector clustering,”
in Proc. of ICASSP’06, May 2006, vol. 5, pp. 33–36.