2009 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics

October 18-21, 2009, New Paltz, NY

A PROBABILISTIC SPEAKER CLUSTERING FOR DOA-BASED DIARIZATION
Katsuhiko Ishiguro, Takeshi Yamada, Shoko Araki and Tomohiro Nakatani
NTT Communication Science Laboratories
Kyoto, 619-0237, Japanishiguro@cslab.kecl.ntt.co.jp
2. SYSTEM OVERVIEW AND RELATED WORKS

ABSTRACT
We present a probabilistic speaker clustering and diarization
model. Speaker diarization determines “who spoke when” fromthe recorded conversation of unknown number of people. We formulate this problem as the clustering of sequential auditory features generated by an unknown number of latent mixture components (speakers). We employ a probabilistic model which automatically estimates the number of speakers and time-varying speakerproportions. Experiments with synthetic and real sound recordings
confirm that the proposed model can successfully infer the number
and features of speakers and obtained better speaker diarization results than conventional models.

C
A

VAD
DOA

??Parameters??Number of clusters

PREPROCESSING
CLUSTERING

Index Terms— Probabilistic clustering, speaker diarization,
direction of arrival, variational BayesA
B

Speaker samples

RECORDED SPEECH

B

speechAB
C

CLASSIFICATIONDIARIZATONRESULTSC

1. INTRODUCTION
Figure 1: Diarization system overview
Speaker diarization, i.e. estimating “who spoke when”, is one of
key technologies for meeting recognition from audio recordings(e.g. [1]). Speaker diarization is useful for, among other functions,the auto-annotation of minutes, speech signal enhancements andhuman-computer interfaces.
A speaker diarization system first estimates the number of
speakers and their classification characteristics. After that, the system classifies the recorded signals into speaker-wise speech signal
fragments based on the estimated speaker information.
The main topic of the speaker diarization is speaker clustering: clustering of the auditory features into an unknown number ofclusters (speakers). We also note that on-line (incremental) clustering is necessary for real-system application [2]. We measure
the time diﬀerence in sound signal arrival to the microphone array.
Given this TDOA (time diﬀerence of arrival) feature and the microphone array geometry, we compute DOA (direction of arrival)features to estimate speaker location. TDOA and DOA features
have been proven to be useful in speaker clustering and diarization tasks [2, 3]. DOA-based diarization requires that the speakers
do not move during the conversation, but it is robust against voiceoverlapping and this is highly desirable for speaker diarization inmeeting situations.
In this paper, we introduce a new probabilistic model calleddLDA for speaker clustering and diarization. Our proposed model
estimates the number of speakers attending the conversation byclustering DOA features. In addition, the model formulates timevarying speaker proportions as a simple Markov model, depending on the previous time frame. Experiments on synthetic andreal-world recording data show that the proposed model is more
eﬀective than conventional models.

978-1-4244-3679-8/09/$25.00 ©2009 IEEE

Fig. 1 overviews a typical DOA-based speaker diarization system. A sound recording of the conversations by an unknown number of speakers is given to the system as a sequence of time frames.
We then extract features for speaker clustering from the sequenceof frames.
In our model, we combine two feature extractors (preprocessors). The first one, called VAD (voice activity detector) [4], computes the probability of the frame containing (any) voices. If thisprobability is low, the frame is assumed to be a noise frame, i.e.
an environmental noise, and is excluded from further processes.
The second and main extractor is DOA (direction of arrival). We
utilize the DOA extractor proposed in [5]. This feature extractor estimates i) the direction (angle) of the sound source from a
microphone array and ii) the power of the sound heard from that
direction. We expect that many high-power vocal signals will beemitted from the locations of the speakers.
Given the sequence of DOA features (power-orientation features from frames), the clustering processor infers the number andthe locations of the speakers. In the last step, the classification processor determines the utterance status of each speaker at each time
based on the clustering results.
As discussed earlier, the clustering is the most diﬃcult andchallenging part. We briefly review the clustering techniques usedin the previous models and clarify their problems.
In [2], the authors proposed a real-time and on-line diarization
system based on DOA features. They employed a simple on-line
clustering technique called leader-follower clustering [6]. This algorithm is simple and fast to compute, but has an apparent draw-

2412009 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics

back that the clustering result strongly depends on a time independent threshold that is diﬃcult to optimize a priori.
In [3] Bayesian Information Criteria (BIC) is used to find the
optimal number of clusters. The algorithm finds the optimal number by comparing the BIC scores before and after merging any
two clusters. It requires the computation of BIC improvements forevery possible merge, thus the computational load increases exponentially.
One of the most popular clustering models is Gaussian mixture model (GMM). However a simple GMM, where each Gaussian with parameter θ corresponds to a speaker, cannot deal withdynamics of time series.∑It assumes and estimates a time-invariantKmixture model G0 (θ) = k=1
βk δθk (θ) where βk denotes the mixing
ratio, θk a parameter of the cluster k, and δθk a delta function peakedat θk , respectively. However, the actual distribution at each timeframe is time varying because of “turn-takings” i.e. the number of
speakers who produce speech at time t is not constant. Suppose wehave a conversation of THREE speakers, and at time t only ONE
speaker speaks. The distribution at t is apparently diﬀerent fromthe time-invariant distribution which generates THREE speakers’
observations.

October 18-21, 2009, New Paltz, NY

described as follows:
Ht =

K
∑

πtk δθk(1)

k=1

wt,1 | a0 , b0 ∼ Beta (a0 , b0 ) ,Gt = (1 − wt ) Gt−1 + wt Ht =w1 = 1K∑

βtk δθk .

(2)
(3)

k=1

Gt is a mixture which represents who actually produces speeches
at the time frame t. As in (3), Gt is constructed as a linear interpolation of the previous distribution Gt−1 and a newly introduced
distribution (innovation measure) Ht , which is responsible for thechange in the sample distribution between t−1 and t. It is worth
noting that wt , an interpolation factor, is a probabilistic variablesampled from Beta distribution (2) and that by regulating wt thedLDA model can handle an irregular distribution changes, ranging
from small to significant changes.
It is beneficial to consider two extreme cases as follows. If we
set wt = 0 for all t, then all Gt is equivalent to G0 and the dLDA
model is reduced to the simple GMM [7]. On the other hand, setting wt = 1 for all t makes all the time frames independent fromeach other. In this case, we recover the latent Dirichlet allocation(LDA) model [8], which is popular in machine learning community. The dLDA is a natural extension of GMM and LDA.

3. PROPOSED MODEL3.1. Preprocessing

3.3. Dynamic LDA model: the generative model

We assume meeting situations where the speakers remained seatedaround a table, i.e. do not move during the meeting. We put anarray of three microphones on the table and represent the locationsof the speakers by angles from a reference( line. DOA features) are
D = 360 dimensional power vector ft = ft,−179 , . . . , ft,180 . Eachft,d denotes the estimated signal power from the direction (angle)
d[deg] at a time t.
To meet the model description, we convert this DOA feature
ft to a set of discrete samples xt = {xti ∈ R1 }. From each d ∈{−179, . . . , 180}, multiple samples xti = d are reproduced. The
number of samples ntd is proportional to ft,d . Going through thisconverting step ∑from d = −179 to d = 180, we have xt = {xti }containing nt = d ntd samples. The distribution of xti reflects thepower-orientation distribution at time t. Because human voices
have large power (large ft,d ), a sample concentration indicates alocation of a speaker. Therefore we can estimate the locations the
speakers by clustering samples.
Speaker clustering is understood as the on-line clustering ofsample set {x1 , . . . , xt }. We expect xti are partitioned into clusters.
Each cluster corresponds to a speaker, and has latent parameter θ
representing the location of the speaker. Samples xti are generated
from observation function F(θ). We would like to infer the numberof speakers, and their locations, and partition the sound fragments.
This is equivalent to finding the optimal number of clusters, theirparameters θ, and the assignment of xti to the clusters.

The generative model of dLDA is shown below.
θk | H ∼ H

k = 1, . . . , K
(α α
α0 )
0
0
πt | α0 ∼ Dirichlet, ,··· ,K K
Kwt,1 | a0 , b0 ∼ Beta (a0 , b0 ) , w1 = 1t∏(1 − wm )vtl = wl

(4)
(5)
(6)
(7)

m=l+1

dti | vt ∼ Multinomial (vt )( )zti | dti , πt ∼ Multinomial πdti
( )xti | zti , θk ∼ F θzti

(8)
(9)
(10)

In (4) we sample K parameters for clusters. As discussed, we
assume each cluster corresponds to a speaker, and its parameter
represents the speaker location. The dLDA puts a Dirichlet prioron the mixing ratio of K. If we set K large enough, we will have anappropriate number of “eﬀective” clusters that have a large mixing
ratio, and the mixing ratios of the other clusters will become “negligibly” small. i.e. the number of “eﬀective” clusters (speakers)
and their mixing ratios are automatically estimated.
Ht is generated from Dirichlet as in (5), and Gt is represented
as the weighted sum of Ht . From (3) it is easy to see that

 t
t
t ∏

∑
∑




(1 − wm )
vtl Hl .
(11)
wl Hl ,
Gt =






3.2. Dynamic LDA model: idea

l=1

In this paper, we adopt a new probabilistic model called dynamic
Latent Dirichlet Allocation (dLDA). The dLDA model not onlyinfers the number of clusters (speakers), but also flexibly deals
with the time evolution of the parameter (sample) distributions.
The unique point of dLDA is that it has a Markov property between
the distributions of two consecutive time frames. This property is

m=l+1

l=1

Rather than considering the temporal mixing ratio βtk directly, it iseasier to work on vt for inference. Next, we sample the interpolation factor wt (6) and compute vtl (l = 1, . . . , t) as in (7).
As in (11), Gt is a mixture of Hl with the mixing ratio vt . We
pick an innovation measure index dti = l, meaning that innovation242

2009 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics

measure Hl generates xti (8). Using πdti , we pick a cluster index
zti = k meaning the cluster with parameter θk is responsible forgenerating xti . In other words, zti is an index of the speaker that
produces the sound heard from the direction of xti at time t (9).
This model can represent voice overlapping situations naturally
because each zti may take a diﬀerent value independently.
Finally, xti is sampled from the observation distribution (function) F with picked parameter θk . We assume F is a one dimensional Gaussian (14). To maintain conjugacy, we use a NormalGamma distribution for the prior of parameter (15).
Please note that dLDA can be extended to dynamic Hierarchical Dirichlet Process (dHDP) [9] as the limit of K → ∞, where thefinite Dirichlet prior is replaced by an infinite stick-breaking prior.
However, it is diﬃcult to derive VB inference algorithm for dHDP.
[10] proposes a VB inference solution for finite approximated version of dHDP, but their model is in fact dLDA.
Another closely related model is recently proposed by Fox andothers [11]. The model formulates the infinite number of speakers
and time-dependent turn-takings in a form of extended HMM. The
main diﬀerence from our model is that their model does not allow
overlaps of speakers.

October 18-21, 2009, New Paltz, NY

Table 1: Specification of the real recording data sets
Data
CP1
CP2
DC
CN

# Speaker
4
4
3
3# Turn taking
149
183
126
243

LDA clusters# Utterance
185
218
172
278

dLDA clusters
12

10

probabilistic density

probabilistic density

12Overlap [%]
18.6
13.0
10.8
34.8

8
6
4
2
0
-0.5 -0.4 -0.3 -0.2 -0.1

0

0.1 0.2 0.3 0.4 0.5

10
8
6
4
2
0
-0.5 -0.4 -0.3 -0.2 -0.1

normalized direction0

0.1 0.2 0.3 0.4 0.5

normalized direction

Figure 2: Clustering results on the CP1 real record data set (fourspeakers). Vertical axis denotes the probability density, and thehorizontal axis denotes the normalized angle (location). Left: clustering results by LDA. Right: clustering results by dLDA.

3.4. On-line VB inference[sec] long recording, and was sampled at 32 [msec] intervals. We
merged 10 consecutive frames into a single frame, yielding four
sequences of T = 938.

There are two major approaches to solve the probabilistic model.
One is Gibbs sampling that is accurate but slow in terms of convergence, and the other is variational Bayes (VB) that is fast but
may be trapped in a locally optimal solution. We prefer the VB
inference for on-line speaker diarization system.
In this paper, we develop an on-line and incremental inference
by VB for dLDA. VB computes the variational (approximated)posterior q (·) of the hidden variables and parameters ( (16-20) in
the appendix). VB iteratively computes i) the posterior parametersof hidden variables (rtik and stil ) and ii) suﬃcient statistics.
We describe how to estimate the mixing ratios and the numberof speakers. First, we compute the expected number of samples
assigned to the cluster (= a speaker) k at time t as
kzt,k k =nt∑rtik .

4.2. Experiment on speaker clustering
In this experiment, we tested the performance of dLDA with regard
to the speaker clustering tasks, and compared it against the GMM
and the LDA models. The GMM and LDA are simulated by setting
all wt as wt ≈ 0 or wt ≈ 1, respectively. After the on-line clusteringof the record data up to the last time frame T , we examined theresultant number of clusters and their parameters.
We present the part of clustering results in Fig. 2 and Fig. 3. As
you can see, dLDA clustering obtained better results than GMM
and LDA models. We assume that this diﬀerence comes from theability to model the intermittent changes of speaker (cluster) distributions. The dLDA model could not achieve perfect clusteringwith the DC and CN datasets: the model produced extra clusters corresponding to noise inputs. However those “noise” clusters had smaller mixing ratios than the those of “speaker” clusters.
Therefore we can still improve the clustering of dLDA by carefully
studying the threshold (β̂k > K1 ).

(12)

i=1

Second, using kzt,k k we determine the posterior estimates of temporal mixing ratios β̂tk and global mixing ratio β̂k as follows:∑
kzt,k kkzt,k k, β̂k = ∑K t ∑.
(13)β̂tk = ∑Kt kzt,k k
k=1 kzt,k k
k=1
We count the number of “eﬀective clusters” (speakers) as thosethat have mixing ratio β̂k larger than chance level K1 . In manycases, the mixing ratio of other minor clusters are negligibly small.

4.3. Experiment on speaker diarization precision
Next, we evaluated the classification performance to verify the entire diarization system (Fig. 1). We employ the DER (diarization
error ratio) measure [12] for the evaluations. DER is a percentageof the error time length in the total sound recording length.
We test a simple classification rule based on the posterior ofsample assignments zti to estimate each speaker’s diarization activity (speaks or not).

4. EXPERIMENTS
4.1. Data Sets
The simulated data has 422 frames with 64 [msec] intervals, excluding no-speech periods based on VAD. We merged five consecutive frames into a larger frame in order to stabilize the distributions of xti within a time frame. Hereafter index t refers to thetime index after the merge, and T = 82. This data simulated theconversation by three people with speaker overlaps.
We used four data sets of the real recordings gathered in [2].
Their specifications are shown in Table 1. Each data set is a 300

k speaks at t
k does not speak at t

if kzt,k k > τ1otherwise&β̂tk > τ2

τ1 , τ2 are predefined thresholds. This simple classification rule isbased on the one used in [2], and eﬀectively suppress the “noise”
clusters estimated in CN and DC datasets (Fig. 4).

2432009 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics

GMM clusters

12

dLDA clusters

VB posteriors are defined as follows:109probabilistic density

10

probabilistic density

October 18-21, 2009, New Paltz, NY

8
6
4
2

q∗ (θk ) = NormalGamma (µ1 , β1 , c1 , d1 ) ,865q∗ (πt ) = Dirichlet(

4
3
2

0

0.1 0.2

0.3

0-0.5 -0.4 -0.3 -0.2 -0.1

0.4 0.50

0.1 0.2 0.3 0.4 0.5

normalized direction

normalized direction

T nm
α0 ∑ ∑
rmi1 smit , . . . ,+K m=t i=1

T nm
T nm
α0 ∑ ∑
α0 ∑ ∑
rmik smit , . . . ,rmiK smit , ),++K m=t i=1
K m=t i=1


nt
nt ∑
t−1
∑
∑


q∗ (wt ) = Beta a0 +stit , b0 +stim  ,1
0-0.5 -0.4 -0.3 -0.2 -0.1

(16)

7

Figure 3: Clustering results on the CN real record data set (three
speakers). Vertical axis denotes the probability density, and thehorizontal axis denotes the normalized angle (location). Left: clustering results by GMM. Right: clustering results by dLDA.

i=1

(17)

(18)

i=1 m=1q∗ (dti ) = Multinomial (sti1 , . . . , stil , . . . , stit ) ,q∗ (zti ) = Multinomial (rti1 , . . . , rtik , . . . , rtiK ) .

(19)
(20)

Parameters of the hidden variables’ posteriors (s, r) are computed
via EM algorithms. Required suﬃient statistics are:Table 2: DER measures (%) achieved in diarization experiments
Method[2]dLDA(proposed)

Simulate
7.1
6.7

CP1
21.9
21.6

CP2
25.0
22.3

DC
29.9
27.0

µ1 =

CN
34.3
30.9

c1 = c0 +Nk =Table 2 presents the computed DER measures; the DER values reported in [2] are shown for comparison. It is clear that the
proposed model basically outperforms the previous method. We
note that DERs of the GMM and the LDA models are much worse
than those of dLDA (results not shown).

nt
T ∑
∑
t=1 i=1

rtik ,

β0 µ0 + Nk x̃k,β0 + N k

β 1 = β0 + N k ,S̃
β0 Nk ( x̃k − µ0 )2+,2 β0 + N k
2
nt
T nt
T ∑
∑
1 ∑∑
rtik (xti − x̃k )2 .
x̃k =rtik xti , S̃ k =Nk t=1 i=1
t=1 i=1
Nk,2

d1 = d0 +B. REFERENCES[1] S. Tranter and D. Reynolds, “An overview of automatic speaker diarization systems,” IEEE Trans. ASLP, vol. 14, no. 5, pp. 1557–1565,
2006.

5. CONCLUSION[2] S. Araki, M. Fujimoto, K. Ishizuka, H. Sawada, and S. Makino, “A
DOA based speaker diarization system for real meetings,” in Proc.
HSCMA, 2008, pp. 29–32.

In this paper, we proposed a new probabilistic model for speakerdiarization tasks. We employ dynamic LDA (dLDA) model for
speaker clustering. The dLDA model automatically infer thenumber of clusters and data partitioning, and is able to handle
time varying cluster distributions. We developed an on-line inference algorithm, and experimentally confirmed the improvements
in DER performance yielded by the dLDA model.

[3] J. M. Pardo, X. Anguera, and C. Wooters, “Speaker diarization
for multi-microphone meetings using only between-channel diﬀerences,” in Proc. MLMI, 2006, pp. 705–711.
[4] M. Fujimoto, K. Ishizuka, and T. Nakatani, “A voice activity detection based on adaptive integration of multiple speech feature and signal decision scheme,” in Proc. IEEE ICASSP, 2008, pp. 4441–4444.

A. MODELS AND VB POSTERIORS[5] S. Araki, H. Sawada, R. Mukai, and S. Makino, “DOA estimation for
multiple sparse sources with normalized observation vector clustering,” in Proc. IEEE ICASSP, vol. 5, 2006, pp. 33–36.

The observation function F and the parameter prior H are:)(xi ∼ N ·; m, σ2 ,()
p m, σ−2 = NormalGamma (µ0 , β0 , c0 , d0 ) .

(14)[6] R. O. Duda, P. E. Hart, and D. G. Stork, Pattern Classification. John
Wiley & Sons, 2001.

(15)[7] S. Araki, T. Nakatani, H. Sawada, and S. Makino, “Blind sparsesource separation for unknown number of sources using Gaussian
mixture model fitting with Dirichlet prior,” in Proc. ICASSP, 2009,
pp. 33–36.
dLDA diarization

Ground truth[8] D. M. Blei, A. Y. Ng, and M. I. Jordan, “Latent Dirichlet allocation,”
J. Machine Learning Research, vol. 3, pp. 993–1022, 2003.

3

speaker index

speaker index

5

2

1

[9] L. Ren, D. B. Dunson, and L. Carin, “The dynamic hierarchical
Dirichlet process,” in Proc. ICML, 2008, pp. 824–831.

43[10] I. Pruteanu-Malinici, L. Ren, J. Paisley, E. Wang, and L. Carin, “Dynamic hierarchical Dirichlet process for modeling topics in timestamped documents,” IEEE Trans. PAMI, vol. submitted, 2008.

2
1

0

50

100

150

200

time[sec]

250

300

0

50

100

150

200

250

300

[11] E. Fox, E. Sudderth, M. Jordan, and A. Willsky, “An HDP-HMM forsystems with state persistence,” in Proc. ICML, 2008.

time[sec][12] NIST, “Spring 2007 (rt-07) rich transcription meeting recognition
evaluation plan,” 2007. [Online]. Available: http://www.nist.gov/
speech/tests/rt/2007/index.html

Figure 4: Diarization results on the DC real record data set (bestviewed in color). The horizontal axis denotes the actual time. Left:ground truth. Right: diarization results after dLDA clustering.

244