IEEE TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, VOL. 20, NO. 2, FEBRUARY 2012447

Probabilistic Speaker Diarization With Bag-of-Words
Representations of Speaker Angle Information
Katsuhiko Ishiguro, Member, IEEE, Takeshi Yamada, Member, IEEE, Shoko Araki, Member, IEEE,
Tomohiro Nakatani, Senior Member, IEEE, and Hiroshi Sawada, Senior Member, IEEE

Abstract—Speaker diarization determines “who spoke when”
from the recorded conversations of an unknown number of people.
In general, we have no a priori information about the number,the locations, or even the characteristics of the speakers. Additionally, speakers’ speech utterances vary dynamically becauseof turn-taking during the conversations. These conditions makethe speaker-clustering task extremely difficult. The problem becomes even harder if online (incremental) processing is required.
In this paper, we formulate the speaker-clustering problem as
the clustering of the sequential audio features generated by an
unknown number of latent mixture components (speakers). We
employ a probabilistic model that assumes time-sensitive speakermixtures at every time frame, which, surprisingly, suits the diarization scenario. We combine the time-varying probabilistic
model with direction of arrival (DOA) information calculated froma microphone array in a bag-of-words (BoW)-style feature representation. The proposed system effectively estimates the number
and locations of the speakers in an online manner based on thestandard Bayes inference scheme. Experiments confirm that the
proposed model can successfully infer the number and featuresof speakers and yield better or comparable speaker diarization
results compared with conventional methods in several datasets.
Index Terms—Bag-of-words (BOW), clustering, direction of arrival (DOA), latent Dirichlet allocation (LDA), speaker diarization,
microphone arrays, variational Bayes inference.

I. INTRODUCTION
HE automatic indexing of multi-party conversations suchas group meetings has been studied intensively because it
can allow their rapid retrieval from archives, automatic minutes
taking, and automatic speech enhancements in conversations.
Many multi-party conversations have been recorded in relation
to, for example, Augmented Multi-party Interaction with Distant Access (AMIDA) [1], Computers in the Human Interaction
Loop (CHIL) [2], and the NIST Rich Transcription Meeting

T

Manuscript received June 30, 2010; revised January 05, 2011; accepted April
21, 2011. Date of current version January 13, 2012. An earlier version of thiswork was presented at the 2009 Workshop on Applications of Signal Processing
to Audio and Acoustics (WASPAA 2009) as a workshop paper. The associate
editor coordinating the review of this manuscript and approving it for publication was Dr. Gerald Friedland.
K. Ishiguro, S. Araki, T. Nakatani, and H. Sawada are with the NTT Communication Science Laboratories, NTT Corporation, Kyoto 619-0237, Japan(e-mail: ishiguro.katsuhiko@lab.ntt.co.jp; araki.shoko@lab.ntt.co.jp; nakatani.
tomohiro@lab.ntt.co.jp; sawada.hiroshi@lab.ntt.co.jp).
T. Yamada is with NTT Science and Core Technology Laboratory Group,
NTT Corporation, Kanagawa 243-0198, Japan (e-mail: yamada.tak@lab.ntt.co.
jp).
Color versions of one or more of the figures in this paper are available online
at http://ieeexplore.ieee.org.
Digital Object Identifier 10.1109/TASL.2011.2151858

Recognition Project [3]. Also, a number of methodologies forthis problem have been proposed [4]–[6]. One essential function for indexing such recorded data is speaker diarization, i.e.,estimating “who spoke when” from audio recordings [7]–[9].
A speaker diarization system estimates the speaker segment
boundaries by classifying speech features, such as acoustic
and/or location features.
A key issue of speaker diarization is speaker clustering:clustering the features into an unknown number of clusters(speakers). As features, acoustic and location information are
widely employed. The former is the mel-frequency cepstralcoefficient (MFCC) features, which were used in [10], forexample. MFCC features are so strong that we are able todiscern the speakers very precisely if we can capture the soleclean utterances of each speaker in a stable manner. Friedland
and Vinyals [11] proposed a very fast diarization system. In
this paper, MFCC features were effectively combined witha Gaussian mixture model (GMM) model. However, MFCC
features tend to be degraded by environmental noise and theoverlap in different speakers’ utterances. This is inconvenient
for the speaker diarization task, where we expect the system torecognize complex dialogs with many overlaps in various environments. The latter, those adopted by this paper, are related to
the location information of the speakers, including time difference of arrival (TDOA) or direction of arrival (DOA), estimatedvia microphone arrays. DOA-based diarization, which requiresthat the speakers do not move during the conversation, is robust
against voice overlap, and this is highly desirable for speakerdiarization in meeting situations. Some papers [8], [12] successfully combined MFCC and TDOA features for diarization.
However, the use of solely TDOA or DOA features also has
been proven to be useful in speaker clustering and diarization
tasks [13], [14]. In this paper, we determine the relative timedifference of sound signal arrival between microphone pairs ofa microphone array following [13].
In most cases, we have no a priori information about the
number, the locations and the characteristics of the speakers.
We also note that online (incremental) clustering is necessary
for some applications. For example, in the teleconference
system we need instant speaker diarization for better speechenhancements of the speaker. Instant speaker diarization is
also required for human–computer interface situations suchas robots serving many persons. In both cases, participants inthe conversation may vary every moment, and the overlappingof utterances makes the problem even more difficult. The
diarization system needs to solve these problems on-site; thus,
online clustering is an important issue. It is evident that we

1558-7916/$31.00 © 2012 IEEE

448

IEEE TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, VOL. 20, NO. 2, FEBRUARY 2012cannot employ simple mixture models such as the GMM,
whose mixing ratio is fixed for all time frames, because the“mixture” of speakers’ speech fragments is time sensitive due
to turn-taking during the conversations.
In this paper, we adopted the new probabilistic model called
dynamic Latent Dirichlet Allocation (dLDA), which was proposed in [15]. Employing dLDA enables us to incorporate thedynamic properties of the conversation, especially turn-taking,into the model to represent time-frame-sensitive audio feature
distribution. The model formulates time-varying speaker mixtures as a simple Markov model that depends on the previous
time frame. Thanks to this time-varying nature of the dLDA
model, the proposed model can effectively estimate the numberof speakers participating in the conversation by clustering location information. We also incorporate a bag-of-words (BoW)style feature representation for audio signal processing, which
is attracting a lot of attention in other research fields such as natural language processing and computer vision. To utilize a BoW
representation in DOA-based speaker diarization, we propose
replacing the standard Dirichlet-Multinomial model [15], [16]with a Gaussian mixture model as in our previous research [17].
We validate the performance of the dLDA model with quantitative and qualitative experiments, compared with conventional
speaker diarization techniques.
In Section II, we give an overview of the proposed system
and related works in the literature. In Section III, we use theprobabilistic representation to describe the speaker-clustering
problem and compare several mixture models. In Section IV,
we introduce the dLDA model for speaker clustering. Section V
describes the Bayesian inference algorithm. Section VI presents
the experiments and the results, which confirm the superiorityof the proposed model. Section VII concludes this paper.
II. SYSTEM OVERVIEW AND RELATED WORKS
Fig. 1 gives an overview of a typical DOA-based speaker diarization system. A sound recording of a conversation held by
an unknown number of speakers is given to the system as a sequence of time frames. We then extract features for speaker clustering from the sequence of frames.
Our model is a combination of two feature extractors (preprocessors). The first, called voice activity detector (VAD) [18],computes the probability of a frame containing any voice. If thisprobability is low, the frame is assumed to be a noise frame,i.e., an environmental noise, and is excluded from further processing. The second, and main extractor, is DOA. We utilize the
DOA extractor proposed in [19]. This feature extractor estimates
1) the direction (angle) of the sound source from a microphonearray and 2) the power of the sound heard from that direction.
We expect that many high-power vocal signals will be emitted
from the locations of the speakers (see Appendix for details).
Given the sequence of DOA features (power-orientation features from frames), the clustering processor infers the number
and locations of the speakers. In the last step, the classification
processor determines the utterance status of each speaker at eachtime based on the clustering results.
As discussed earlier, clustering is the most challenging part.
We briefly review the clustering techniques used in previous

Fig. 1. Diarization system overview.

models and clarify their problems. In [13], the authors proposed a real-time (online) diarization system based on DOA
features. They employed a simple online clustering techniquecalled leader-follower clustering [20]. This algorithm is simple
and fast, but has an apparent drawback in that the clusteringresult strongly depends on a time-independent threshold thatis difficult to optimize a priori. In [14], Bayesian information
criteria (BIC) is used to find the optimal number of clusters.
The algorithm finds the optimal number by comparing the BIC
scores before and after merging any two clusters. It requires thecomputation of BIC improvements for every possible merge,
so the computation load increases exponentially.
In this paper, we seek yet another interpretation of the
speaker clustering problem, which naturally captures thenature of meeting conversations, based on speaker location
information.

III. NAIVE MODELING BY MIXTURE MODELSIn this paper, we study three mixture-based models: GMM,
LDA, and dLDA. First, we examine GMM and LDA, which
are less complicated than the dLDA model. We introduce theprobabilistic representations of these two, and study the weak
points of these models. In the next section, we introduce dLDA
to overcome the problems.

ISHIGURO et al.: PROBABILISTIC SPEAKER DIARIZATION WITH BoW REPRESENTATIONS

449

A. Bag-of-Words Representations
Throughout this paper, we adopt the BoW feature representation. BoW approaches first appeared in natural language processing and information retrieval research. BoW is a simple
representation of documents, each of which consists of many
words, that can handle high (thousands) cardinality orders. We
consider a dataset that is a collection of documents. BoW representation describes a document by a histogram of words: aggregate of the counts of word appearance regardless of the order ofwords in the sentences. Identifying the word histogram (or a distribution), we can easily analyze the contents of a document, andthis is useful for document handling such as search and abstraction. Though this representation is very simple, BoW is known
as a powerful technique for document analysis. These days,
other research fields such as computer vision (e.g., [21]–[25]),and very recently audio and acoustics processing [26], employ
BoW combined with the topic model, which we will discusslater in this section.
In this paper, we work on the BoW representation for speakerclustering and diarization based on speaker location information. A conversation recording is a collection of time frame-wiseobservations. Every time frame contains a set of information
about multiple speakers’ locations. We interpret this data structure as a variant of BoW representation. A series of consecutive
time frames (possibly one time frame) is considered as a document. Every document consists of a set of location information; therefore, speaker location information serves as a wordin documents. We analyze the location word histogram and try
to reveal “who spoke” in the given document. Repeating thisanalysis over all documents, we can identify “who spoke when”
from a conversation recording.
BoW is, in essence, discrete feature representation: counts ofword tokens. Therefore, we first quantize the original features
into discrete tokens, which serve as “words.” If we quantize
a visual feature such as SIFT [27], the quantized features arecalled “visual words.” Similarly, we consider “angle words” inthis paper.
The audio feature used in this paper is DOA. We assume
meeting situations in which the speakers remain seated arounda table, i.e., do not move during the meeting. We put an arrayof three microphones on the table and represent the locations of
the speakers by angles from a reference direction. DOA features
dimensional power vectors:are(1)
denotes the estimated signal power from direction
Eachat time . For details, please see the Appendix.
(angle)In accordance with the BoW representation, we convert DOA
into a set of discrete words
asfeature, multiple samples
in Fig. 2. From eachare reproduced. The division by
is for, is propornormalization purposes. The number of words,
tional to. Going through this conversion fromto, we have(2)

Fig. 2. Bag-of-Angle words representation of observations. We quantize theobserved DOA powers into discrete “word” samples x .

(3)
Now we have angle “words.” Next, we define a document.
In this paper, we merge some consecutive time frames as one
document (frame document). Each document has its own word
histogram, which summarizes all angle word occurrences in thetime frames within the document. The merging process serves
to smooth document-word distributions (the histogram) among
documents and to aid computational efficiency.
Finally, each frame document is represented by “Bag-ofAngle words.” Hereafter, we slightly abuse notations. An index
refers to the time index of the frame documents. We have“angle words” for the frame document . Each word (a sample)
denotes the angle of the speech fragment. The distribution
reflects the power-oriented distribution(or, histogram) ofat time . Because human voices have large power (large),a sample concentration indicates the location of a speaker.
Therefore, we can estimate the locations of the speakers byclustering these samples, namely angle words.
B. Gaussian Mixture Model
Standard mixture models are among the most popular statistical models for clustering, especially the GMM. Araki et al.
[28] studied the use of GMM for a batch (offline) speaker clustering scenario, which is not our goal. We explain how GMM
can be used in a fully probabilistic manner for online speakerclustering for BoW representation.
In this paper, speaker clustering is understood as the online.
clustering of observed sample set (BoW)
into clusters. Each cluster corresponds
We hope to partition
to a speaker, and latent parameter represents the location of
are generated from observation functhe speaker. Samples
tion. We would like to infer the number of speakers, andtheir locations. This is equivalent to finding the optimal number
to eachof clusters and their parameters , and assigning
cluster.

450IEEE TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, VOL. 20, NO. 2, FEBRUARY 2012

Fig. 3. Graphic models of the three mixture models. Circle nodes are random variables and square nodes are the constants. Shaded nodes denote the observed
values, and rounded rectangular “plates” are the repetitions. Arcs indicate the probabilistic dependencies between variables. (A) GMM, (B) LDA, and (C) dLDA.

It may be that this process can be tackled by an ordinary
GMM. Usually, BoW indicates that the observation distribution is a multinomial distribution [16]. This is because words(natural language words) are discrete symbols. However, eachis a continuous real value. Therefore, in thisangle-word,paper we adopt the Gaussian prior as the distribution ofwhich makes the model a GMM with BoW (angle words). Eachcluster corresponds to a Gaussian parameterized by . We asmixtures of Gaussians, and formulate the distribution
sumeof
as follows:(4)is the parameter of the th cluster.
is
whereis the variance of the th Gaussian.
is thethe mean, and.
mixing ratio of
For simplicity and clarity of later discussions, we describe
GMM in another but equivalent way. We introduce a hidden as.
is a mixture assignment variable; it indicates the cluster
signment variable for observed wordthat generated. We assume
is a -dimensional binary
vector, whose elements equal zero except for the element of 1.
, meaning the th element
Without confusion, we write; the others are zero.
of the vector
is a random variable that distributes proportional to the.
mixing ratio vector
Also assuming the prior distribution on , we have the following fully probabilistic GMM model for speaker clustering:
Dirichlet(5)

Multinomial

(6)
(7)

is a -dimensional hyperparameter for the Dirichlet distribution. The graphical model of the above GMM is shown in
Fig. 3(a). In the figures, circle nodes represent the random variables, and square nodes indicate constants. The shaded nodes

Fig. 4. Result of speaker cluster estimation by GMM model (ground truth: four
speakers). Each cluster (Gaussian) is assumed to be a speaker. The horizontal
axis is the angle value (normalized from 0.5 to 0.5).

0

are the observed data (angle words), and rounded rectangular“plates” denote repetition with respect to the indices below.
This simple GMM set up is easy to understand, and seems
well suited to the speaker clustering problem: running this, which indicate themodel includes inferring assignments
mixtures. Given theseclustering of the observations into
clustering assignments, we can easily infer the parameter of theth speaker cluster (mixture) parameter and its mixing ratio.
Now we present one part of the results of the experiments forbetter understanding. We build the above model, and estimate
all unknown variables ( , , and ) by Bayesian inference(discussed in the “Inference” section). Inference is performed in
an online manner: we iterate the inference process every time thebag-of-angle words of , a new frame document, is received.
Fig. 4 shows the final results of the GMM estimation for fourspeaker conversation data. As you can see, GMM yields many
more clusters than there are speakers (four); it fails to estimate
the number of speakers (clusters).

ISHIGURO et al.: PROBABILISTIC SPEAKER DIARIZATION WITH BoW REPRESENTATIONS

Fig. 5. Illustration of GMM model for speaker clustering in diarization tasks.
is assumed to represent time-varying observation
Time-invariant mixture, which reflects turn-taking.

x

GThis result is disappointing but understandable because a
simple GMM cannot deal with the dynamics of conversations.
Choosing GMM for speaker clustering is equivalent to assuming a time-invariant mixture model in the parameter space:
(8)
where
is a delta function that peaks at . Since each parameter corresponds to a speaker, GMM assumes a time-invariant speaker mixture for any conversation. We can see thisin Fig. 3(a) as we have only one node, located outside of therepetition plate. However, the actual distribution at each time
is time-varying because of “turn-taking,” i.e., the number of
speakers who produce speech at time is not constant. Suppose we have a conversation of three speakers, and at time
only one person speaks. The distribution at differs from thetime-invariant distribution, which suits three speakers’ observations. We illustrate this problem in Fig. 5.
Hereafter, we use the term “turn-taking” with a more generalmeaning. By turn-taking, we mean the changes of speaker proportions along the timeline caused by a speaker’s beginning andend of one utterance. These changes will be induced not only by
speaker changes (strict meaning of turn-taking), but also by anintermittent monologue that contains several switches between
silent and speech periods.
Next, we extend a simple GMM to a more powerful probabilistic model, namely LDA.
C. Latent Dirichlet Allocation
We can instead employ the Latent Dirichlet Allocation (LDA)
model [16], which is a famous BoW representation model originally designed for text data. In the LDA model, we have a set
latent mixture components called “topics.” Each topic isofcharacterized by its own word distribution. A document is represented as a mixture of topics, and each observed word in the
document is generated from one of the topics.
This LDA scheme is readily applicable to our speaker clustering problem. Topic components correspond to speakers represented as Gaussians with location parameters , time intervals

451

Fig. 6. Result of speaker cluster estimation by LDA model (ground truth: four
speakers). Each cluster (Gaussian) is assumed as a speaker. The horizontal axis
is the angle value (normalized from 0 5 to 0.5).

0:to documents, and samples
to words. Analogous to the
GMM model [(5)–(7)], we can formalize a speaker clusteringmodel with LDA as follows:
Dirichlet
Multinomial

(9)
(10)
(11)

The graphical model of LDA is shown in Fig. 3(b). The
main difference from a simple GMM is that the mixing ratiosof topics (clusters) are time-variant. The mixing ratio vector at
the th frame document is denoted asin (9) ().
is located inside of the -plate in
Fig. 3(b). This indicates that is variable over time; therefore,
we can freely represent time-varying speaker mixtures due toturn-taking. Equation (12) represents the way LDA models theparameter mixture behind the conversation observation:(12)
In the above equation, the mixture is defined by time-varying. This means that LDA allows the parameter
mixing ratio
mixture to change over time, and this feature enables LDA totreat turn-taking during conversations in a more reasonable andnatural way.
As in the discussion of GMM, we present the results of an experiment on LDA for elucidation. The same four-speaker dataset
was tested by LDA, and the result is presented in Fig. 6. As canbe seen, LDA performs better than GMM in terms of the es. However, it is also
timated number ofobvious that modeling with LDA still has a problem. This isindependence in (12). This can be
due to the assumption ofis sampled independent andobserved from (9) where every
identically distributed (i.i.d.) from the common Dirichlet. This
does not suit the speaker clustering task, since the speaker topic
are indeed correlated and similar to
mixtures at and ateach other. We illustrate the idea and the problem of LDA in
Fig. 7.
Originated in natural language processing, LDA usually assumes multinomial distribution for observed words. Please notethat we again adopt a Gaussian distribution for angle-words forthe same reason as in the GMM case.

452IEEE TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, VOL. 20, NO. 2, FEBRUARY 2012

Fig. 7. Illustration of LDA model for speaker clustering in diarization tasks.
Time-varying mixtures s are introduced, but they are completely independentof each other, even between successive frame documents.

G

IV. DYNAMIC LDA MODEL

Fig. 8. Illustration of dLDA model for speaker clustering in diarization tasks.
, enabling smooth speaker mixture
Markov dependency is assumed forchanges between frame documents to deal with turn-taking.

G

A. Idea of Dynamic LDA
From the above discussions, we conclude that we need a new
mixture model that can represent both of the following points:
1) time-varying speaker mixtures;
2) correlation between successive frame documents.
For that purpose, we adopt a new probabilistic model called
dynamic Latent Dirichlet Allocation (dLDA). dLDA was first
introduced in [15] to capture the time evolution of the data properties of documents, but it is applicable to many time-series
data in other domains. We found that the idea and formulation of dLDA is surprisingly well-suited to our speaker clustering problem in diarization tasks. This article is, to the bestof our knowledge, the first attempt to utilize dLDA in speakerdiarization.
Its idea is straightforward: dLDA sets a Markov property between the distributions of two consecutive time steps. This property is described as follows:
(13)
Beta

for

(14)
(15)

requirements for the mixture model to solve speaker clustering.
This idea is illustrated in Fig. 8.
: the curAlso, (16) gives us another interpretation ofrent speaker mixture in the parameter space is in fact a mixtureof independent innovation distributions. This interpretation is
helpful in solving the model.
It is worth noting that , an interpolation factor, is a probabilistic random variable sampled from Beta distribution (14) andallows the dLDA model to handle irregularthat regulating
distribution changes, ranging from small to significant changes.
This indicates that dLDA is extremely flexible in representing
the time-varying correlation between successive time steps.
It is beneficial to consider two extreme cases as follows. If wefor all , then allis equivalent toandset
the dLDA model is reduced to the simple GMM [(8)]. On thefor all makes all time steps indepenother hand, setting
dent of each other. In this case, we recover the latent Dirichlet
allocation (LDA) model [(12)]. Therefore, dLDA naturally generalizes GMM and LDA.
B. Generative Model of Dynamic LDA
The generative model of dLDA is shown as follows:
Dirichlet(16)
is a mixture of speaker paAs in the case of LDA [(12)],
rameters that reflect the actual utterances at time . The key point
is constructed as a linear interpolation of theof (15) is that
previous distribution
and a newly introduced “innova, which is responsible for the change intion distribution”
and .
is parameterthe sample distribution between,
ized by the time-varying mixing vector
whose sum of elements equals one. In this way, we can hold both

Beta

(17)
for

(18)
(19)

Multinomial
Multinomial
NormalGamma

(20)
(21)
(22)
(23)

The graphical model of the above dLDA is shown in Fig. 3(c).

ISHIGURO et al.: PROBABILISTIC SPEAKER DIARIZATION WITH BoW REPRESENTATIONS

Fig. 9. Pseudo-algorithm of a simple online VB inference of dLDA.
pre-defined constant, number of iterations.

N

is a

In (17), the probabilistic distribution of mixing ratio vector
for , notfor , is defined as a Dirichlet distribution.
is a hyperparameter for the Dirichlet. It is easier to work withfor inference than to consider the temporal mixing ratio
directly since
are conditionally independent of each other.
in (18) is defined as we have
Next, the interpolation factor
discussed.
are the hyperparameters for . Given , we(
) as in (19) to move from the past
computemixtures’ information to that of the current mixture.
, first we pick an
To generate the observed angle word
index,, meaning that the th innovation distribution
generates the angle wordbased on (20). Using, we, which means that the cluster
choose a cluster index
is responsible for generating. In otherwith parameterwords,
is an index of the speaker that produces the soundat time [(21)]. This model
heard from the direction of
can represent voice overlap situations naturally because
may take a different value for each independently. In (22) we
sample parameters for clusters from an appropriate prior disis sampled from the normal observation
tribution. Finally,
distribution with picked parameter . Note that we employ anormal distribution instead of the multinomial distribution usedin the original dLDA [15] because of the angle-word features.
As discussed, we assume each cluster corresponds to a
speaker, and its parameter represents the speaker location.
is important inThus, the choice of parameter priorobtaining good results. In this paper, we assume a one-dimensional Gaussian distribution for the actual angle words.
consists of meanand precision (inverse
Its parameter. Therefore, priors for these parameters areof variance)chosen from a Normal–Gamma distribution [29] to maintainare the hyperparameters. If we assume
conjugacy.
a multi-dimensional Gaussian for the observation distribution,then the parameter prior is chosen to be Normal–InverseWishart[29]. Because of conjugacy, the dLDA inference is fast andreliable as discussed in the next section.
C. Some Notes and Related Works
Since the dLDA model is fully formulated in a probabilistic
generative model, it can not only infer the number of clusters(speakers), but also flexibly estimate the time evolution of theparameter (sample) distributions from an observed sequence.

453

These elements are subject to the Dirichlet prior in (17). If we
set large enough, we will have an appropriate number of “effective” clusters that have a large mixing ratio, and the mixing
ratios of the other clusters will become negligibly small, i.e.,
the number of “effective” clusters (speakers) and their mixing
ratios are automatically estimated. Values of the parameters
are also fully probabilistically modeled, and are also optimized
automatically in a Bayesian manner.
There are two works closely related to dLDA. Both arestrongly formulated in a probabilistic (Bayesian) manner;
thus, it may be fruitful to discuss the difference between thesemodels.
dLDA can be extended to a dynamic Hierarchical Dirichlet, whereProcess (dHDP) [30] by taking the limit of
the finite Dirichlet prior is replaced by an infinite stick-breaking
prior. In their paper, dHDP is used for segmentation of a musicpiece. However, it is difficult to derive a VB inference algorithm
for dHDP, which is preferable for (future) real-time systems.
Another closely related model was recently proposed by Fox
et al. [10]. The model formulates an infinite number of speakers
and time-dependent turn-taking in a form of extended HMM. In
the aspect of model construction, their model differs from dLDA
in the assumption of dynamics in sequences. Since the HMM allows discrete state jumps, their model suits time-series sequencewith many abrupt structural changes. The dLDA model, on theother hand, prefers smoother changes able to be captured bylinear interpolation [(15)] with the dependencies controlled by[(14)].
One main difference in terms of practicality is that their model
does not allow overlapping of speakers, while dLDA does. Thisis because the hidden states of extended HMM correspond to“speaker A speaks,” “speaker B speaks,” and so forth. On theother hand, our dLDA model estimates the mixture of clusters
at each time frame. Thus, dLDA is able to represent overlappingof speakers in a more natural way than Fox’s model.
V. INFERENCE
A. Inference via Variational Bayes
There are two major approaches to solving the probabilistic
model. Gibbs sampling is accurate but slow in terms of convergence, while variational Bayes (VB) [29], [31] is fast but may
be trapped in locally optimal solutions. We prefer VB inference
for online speaker diarization.
In this paper, we develop an online and incremental inference by VB for dLDA. VB estimates the variational (approxof the hidden variables and parameters.
imated) posterior
Since a VB solution for a Bayes model is just an approximation,
it may be trapped by local optima. However, it provides fast iterative computation, which is important when applying a complex probabilistic model to real applications. The performanceof VB is better than that of point estimation methods such as
maximum-likelihood (ML) and maximum a posteriori (MAP)
solutions in many cases. In signal/sound processing research,[31], [32] are pioneer works on the use of VB.
We first describe the derivation of the VB solution for theare approximated posdLDA model [15]. VB posteriors
teriors in the sense that all posteriors are independent of other454

IEEE TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, VOL. 20, NO. 2, FEBRUARY 2012variables since the true posteriors are not. Because of the conjugacy incorporated in the generative model (17)–(23), we expect
the following forms of the variational posteriors:(37)

Dirichlet

(38)
(24)
(39)
Beta

(25)

Multinomial
Multinomial
NormalGamma

(26)
(27)
(28)

where
(29)
(30)

Note indicates the digamma function.
2) Phase II—Computing the Posterior Parameters: In phases ands to estimate the parameters ofII, we computevariational posteriors in (24)–(28).
It is obvious that these quantities indicate the average prob: picking the innovation measureand 2)
abilities of 1): picking cluster to sample
since these parameters
appear as they are in (26) and (27), respectively.
We can compute these quantities by the equations shown as
follows:

(31)
(32)
(33)

(40)

(34)

(41)

(35)
It is easy to derive these variational posteriors if you follow
the textbook [29]. The objective of the inference is to estimate allof the parameters
to represent the posteriors above.
1) Phase I—Computing Variational Expectations: The inference process consists of two phases that are iteratively computed
as in the standard EM algorithm. The first phase computes expectations of several hidden variables over the variational posteriors.
as the expectation of function over the
Let us denotevariational posterior of variable . Then, in phase I, we computethe following expectations:

(42)
(43)

B. Estimating the Mixing Ratios and the Number of Speakers
We describe how to estimate the mixing ratios and the numberof speakers. First, we estimate the expected number of samples
assigned to cluster (a speaker) at time as
(44)

(36)
where

to determine the posterior estimates ofSecond, we use
temporal mixing ratios
and global mixing ratio
as follows::(45)

ISHIGURO et al.: PROBABILISTIC SPEAKER DIARIZATION WITH BoW REPRESENTATIONS

We count the number of “effective clusters” (speakers) as those. In many
that have mixing ratio larger than chance levelcases, the mixing ratios of other minor clusters are negligible.

455

TABLE ISPECIFICATIONS OF REAL RECORDED DATA SETS

C. Techniques for Fast Online VB Processing
Inherently, the computational cost of the inference process
grows with the time step, since the number of variables to be estimated and the samples to be stored keep increasing. Therefore,
we developed several techniques to accelerate online dLDA inference. Estimation precision can be well balanced against computation costs.
1) Limiting Re-Estimated Variables: One of the most commonly used techniques for adaptive estimation is to re-estimate(update) only the variables of the most recent time steps. We expect that the hidden variables or parameters of older time steps
have converged to stable values because they have been repeatedly updated by VB.
Thus, we minimize computation cost by limiting the variables
to be re-estimated. Equations (36)–(43) are updated only for thesteps. This reduces thevariables of the most recent, where is the total number ofcomputational cost totime steps.
2) Limiting Data Used in Inferences: The above trick reduces the number of variables that need to be re-estimated. However, inference of each variable requires all currently available
observations and estimation results. Thus, the computational
cost basically increases with the time step.
We studied the dLDA model and found that this cost could
be reduced based on the model characteristics. As discussed,
has a time-dependency and the strength of the dependency
is controlled by [(18)]. will be close to 1 when the sample
distributions change drastically (e.g., speaker turn-taking).
Assuming that the most recent turn-taking is observed at,, and it is easy to see thatin (19).
This indicates that the posterior probability of, namely, will be close to zero for. In turn, the corresponding
computations for VB inference can be eliminated.
Therefore, we can forget those quantities, which greatly reduces the computation cost of each update equation. We retain
only the information (estimation results and observations) of themost recent time steps, where is the time index of
the oldest variables to be updated.
3) Limiting Cluster Updates: We expect that the mixing ratio. However, allclusters
concentrates on fewer clusters
and their parameters must be maintained during the inference
clusters.
process, including “negligible”
Thus, we introduce another trick for computational costsaving. At each VB iteration, we update the th cluster, which.
has the mixing ratio of , with the probability ofin several ways. Our experiments use
We can define
the following definition:(46)
This equation indicates that “effective” clusters that have larger
mixing ratios than chance levels
are updated at each time

step, while the other clusters are re-computed according to themixing ratio.
To summarize, we only update the estimation of variational
steps, and retain the estimation
posteriors for the most recent
steps. Re-comresults and observations of the most recent
putations of cluster parameters and likelihoods are further confined probabilistically. These techniques drastically reduce thecomputation cost of dLDA.
D. Inference of Hyperparameters
Thus far, we have discussed how to infer the hidden variables
in a Bayesian manner. In previous discussions, hyperparameterssuch aswere assumed to be constant. In several cases,
however, the initial values of these hyperparameters will affectthe estimation of hidden variables. This problem disturbs VB
inference because VB is a deterministic iterative procedure thatstops at local optima.
To overcome this problem, we infer the hyperparametersin the inference process simultaneously. Since VB is fully
Bayesian, we would like to estimate the hyperparameters in a
Bayesian manner as well, instead of resorting to heuristics oroffline model selection techniques.
One of the most common ways is to put vague priors on hyperparameters and estimate their posteriors by VB as well. Another approach [33] is to reparameterize and sample each hy, instead of precise posterior
perparameter in terms ofevaluations of the original hyperparameter. For example, if hyis assumed to be Gamma-distributed, we conperparametervert
to. Sampling can be achieved from. We estimate (unnormalized) posteriora uniform grid on
probability densities at several values and choose one to update the hyperparameter.
In this paper, we select the latter approach because it is moreconvenient to implement since it is applicable to non-conjugate
cases and prevents the VB solutions from being trapped at local
optima due to the jumps provided by the hyperparameter sampling.
VI. EXPERIMENTS
A. Data Sets
We collected four data sets of the real recordings gatheredin [13]. Their specifications are shown in Table I. We note thatthe recordings contain much turn-taking and voice-overlapping
periods; thus, diarization of these four data sets are challenging.
Each data set took the form of a 300-(s) recording that was sampled at 16 (kHz). The frame length was 64 (ms) and the frame
shift was 32 (ms).

456IEEE TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, VOL. 20, NO. 2, FEBRUARY 2012

We also employed the benchmark datasets provided by the
AMI project [34]. We used some samples from the IDIAP subset(“IS” meetings). The datasets are used in many studies, such as[35], [36]. Each recording involves four participants engaged
in a scenario-based meeting with different durations. Approximately 18% of speech in the recordings overlaps.
We computed angle-word representations, estimated speakerclusters, and inferred who spoke when for all speakers.

TABLE II
DER MEASURES (%) ACHIEVED IN DIARIZATION EXPERIMENTS. FOR THE
FIRST FOUR DATA, “CP2” IS USED AS A DEVELOPMENT SET. FOR THE ISDATASET, “IS1008D” IS USED AS A DEVELOPMENT SET. DEVELOPMENT SETS
ARE MARKED BY ASTERISKS. PARENTHETIC DER VALUES OF [8] ON THE
FIRST FOUR DATA SETS WERE COMPUTED BY OUR IMPLEMENTATION: THEY
MIGHT BE WORSE THAN THE ORIGINAL IMPLEMENTATION. WE EXCERPT THE
DER VALUES OF [8] ON THE IS DATASET FROM THE ONLINE APPENDIX OF [36]

B. Quantitative Evaluation: Speaker Diarization PrecisionFirst, we show quantitative performance comparisons. We
evaluated the classification performance to verify the entire diarization system (Fig. 1). We employ the diarization error rate(DER) measure [3] for the evaluations. DER is a percentage ofthe error time length against the total sound-recording length.
The error time consists of the following three errors.
1) MST (missed speaker time): the length of time intervals inwhich the system detected no utterances while a speaker(s)actually spoke.
2) FAT (false alarm speaker time): the length of time intervals
in which the system detected speaker utterances while no
speaker actually spoke.
3) SET (speaker error time): the length of time intervals overwhich the system correctly detected the speaker utterances,
but failed to associate the utterance events with the correct
speaker.
Given these three errors, DER is computed as follows:
MST FAT SET
DER
(47)
TOTAL
where TOTAL denotes the length of the total speech. The evaluation criteria also follow those provided by NIST [3], that is,
the speech segments were split with non-speech periods of morethan 300 (ms) in length, and the allowed tolerance for the difference between the system outputs and the correct labels was 250(ms). The correct speech onset and offset labels for the recorded
data were generated manually. All DER values are computed
using a script provided by NIST.
We test a simple classification rule based on the posterior ofto estimate each speaker’s diarization
sample assignments
activity (speaks or not). The rule isspeaks at

ifdoes not speak atotherwise

where
are predefined thresholds. This simple classification rule is based on the one used in [13].
We compare the performance of our proposed model withtwo previously studied techniques. The first is [13], which is a
non-probabilistic online speaker clustering model with the same
DOA features. We also evaluated the diarization performancewith a method proposed by ICSI [8], which is a state-of-the-art
model. The method is based on agglomerative clustering using, where each cluster is modeled with one GMM per feature stream and embedded into an ergodic HMM, and both aspeech feature (MFCC) and a location feature (TDOA) are utilized. Please note that [8] is a batch (offline) algorithm; thus, we
do not expect the proposed online (incremental) dLDA model tooutperform [8] in the DER scores.

Fig. 10. Turn-taking detection by the trajectory of estimated w values in simulation data (best viewed in color). Circles, rectangles, and triangles denote theturn-taking. (A): Ground truth of speaker utterances. (B): Variational expectation of w .

For each data collection, we choose one conversation
recording as a development set to tune the parameters of [13]and the proposed models. For the proposed models, we tuned
parameters related to preprocessing and postprocessing, i.e.,
parameters for the BoW feature extractions (length of merged
consecutive frames for a document, noise threshold for weakpower responses, strength of VAD, and map from to ) andparameters for the classification ( and ). Please recall thatthe model parameters (hidden variables and hyperparameters)of GMM, LDA and dLDA models are automatically estimatedby Variational Bayes updates.
Table II summarizes the computed DER measures. Alongwith [8], [13], we present the DER values of GMM and LDA
models. GMM and LDA models are simulated by setting allasor, respectively. As expected, we confirm that DERs of the GMM and the LDA models are muchworse than those of dLDA. This is reasonable since the assumptions made in GMM and LDA models are not realistic, ignoring
the natural properties of conversations. Compared to previous
research, the dLDA model presents comparable or better performance against the non-Bayesian online diarization system ofISHIGURO et al.: PROBABILISTIC SPEAKER DIARIZATION WITH BoW REPRESENTATIONS

457

Fig. 11. Clustering results on CP1 real record data set (four speakers). Vertical axis denotes the probability density, and horizontal axis denotes the normalized
angle (location). Left: clustering results by GMM. Center: clustering results by LDA. Right: clustering results by dLDA.

Fig. 12. Clustering results on CN real record data set (three speakers). Vertical axis denotes the probability density, and horizontal axis denotes the normalized
angle (location). Left: clustering results by GMM. Center: clustering results by LDA. Right: clustering results by dLDA.

[13] in many recordings. Concerning the comparisons with [8],the DER values of [8] in Table II for the first four data were computed by our implementation. Thus, these values could be worse
than the true performance of [8]. For the IS meeting dataset,
DER values were taken from the online appendix of [36]. As expected, [8], which is one of the best offline diarization methods,
exhibits the best DER values in many IS meeting recordings.
However, the proposed dLDA model also marks comparable orbetter results for a few IS recordings. Considering that dLDA istested as an online (incremental) model in the experiments, thisresult is quite promising.
These results provide more understanding about the dLDA
model. The main difference between the first four datasets and
IS corpus recordings is the duration of each speaker’s turn. In
IS recordings, each turn of utterance has a longer time duration. On the other hand, the first four datasets are characterizedwith much shorter durations of turns, and frequent turn-taking.
The generative process of turn-taking in dLDA [(14)–(16)] indicates that dLDA is able to estimate and track the changes of
speaker distributions for every frame document thanks to . At
the same time, the dLDA may overfit if the changes of speaker
distributions are not so frequent and drastic, as in the case of
IS recordings (with long durations of speech turns). We also
note that locations of speakers are not strictly fixed during therecordings for some IS recordings. The dLDA model does notassume the changes of the speaker locations; therefore, we think

this model mismatch is one reason for severely degraded performance of dLDA for some recordings (e.g., IS1008a).
C. Qualitative Evaluations
Next, we examine several qualitative results to elucidate thereasons for the good DER scores of dLDA.
1) Turn-Taking Detection by Interpolation Variable: One
distinct feature of dLDA is the notion of interpolation factorin (14) and (15). Larger
indicates that turn-taking is likely attime .
for the simulated recording
Fig. 10 presents estimateddata. The simulated data has 467 frames with 64 (ms) frame
shift intervals. Out of 467 frames, 45 frames were excludedin advance as no-speech periods based on VAD. We merged
five consecutive frames into a document. The data simulated
the conversation of three people with speaker overlapping, convolving English speech signals and measured impulse responses
in a room [37].
Colored plots in the upper panel, (A), are the actual speakerutterance periods. Circles, rectangles and triangles denote turntaking. The lower panel, (B), shows the expectation of variational posterior of . The trajectory is characterized by a few
spikes. These spikes indicate large values of , which imply theexistence of turn-taking. In fact, we find that most of the (actual)
turn-taking matches these spikes in . This result shows that

458

IEEE TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, VOL. 20, NO. 2, FEBRUARY 2012

dLDA is able to model and represent turn-taking in the recorded
conversation automatically.
2) Speaker Clustering Results: Finally, we tested the performance of dLDA with regard to the speaker clustering tasks, ina comparison against the GMM and the LDA models. After the
online clustering of the recorded data up to the last time , weexamined the resultant number of clusters and their parameters.
We present a part of the clustering results from the real
recorded data sets in Fig. 11 (CP1), and Fig. 12 (CN). Illustrated normal distributions are “effective” clusters in terms ofin (45).
estimateddLDA clustering yielded better results than either GMM or
LDA. We assume that this superiority comes from the ability
to model the intermittent changes of speaker (cluster) distributions. The dLDA model could not achieve perfect clusteringon the DC and CN data sets: the model produced extra clusters due to noise inputs. However, those “noise” clusters had
smaller mixing ratios than the “speaker” clusters. Therefore, we
can further improve the clustering of dLDA by carefully refining). Since speaker clustering is performed so
threshold (precisely, it is no surprise that dLDA achieved good DER values.
Based on these results, we conclude that the dLDA model
is able to infer the speaker clusters from speakers’ location information. The quantitative results show that the dLDA model
especially suits the conversations with frequent turn-taking andshorter speech turn durations.

(TDOA) between microphone pairquency slot:(48)
where denotes the complex conjugate, and is the index of thereference microphone that is arbitrarily selected from one of
microphones. We then calculate the direction of arrival (DOA)
information at each time–frequency slot using the TDOA in, which consists of theof all microformation
phone pairs:(49)
where is the propagation velocity of the signals, and denotes
the Moore–Penrose pseudo-inverse. is the microphone coor, where adinate information
represents the location of microphone . When
3-D vector
the source azimuth is and the elevation is , the DOA vector
can be written as

In this paper, we employ only azimuth for simplicity as the
DOA information.
from direction at time is calculated
The signal powerby adding the power of the time–frequency component:

VII. CONCLUSION
In this paper, we introduced a new probabilistic model
for speaker diarization tasks. We adopted the dynamic LDA(dLDA) model for speaker clustering. The dLDA model automatically infers the number of clusters and data partitioning,
and is able to handle time-varying cluster distributions. We
proposed a fully probabilistic model for speaker diarization in
combination with a bag-of-angle word representation of DOA
features. We developed an online inference algorithm based on
variational Bayes, and experimentally confirmed that dLDA isable to infer the number of speakers successfully and improves
DER performance of online diarization. We also found thatthe online dLDA recorded good DER values compared to thestate-of-the-art offline diarization system.
Some previous studies concerning online diarization proposed using additional cues. For example, [36], [38] proposeda multi-modal (visual + audio) setup for online diarization.
Including other features such as MFCC and visual cues is very
interesting. We think augmenting other feature models is apromising direction for improving the proposed method.

APPENDIX
This Appendix explains our methods of calculating the DOA.
feature and the power vector
be a signal observed by microphoneLet(. To calculate the DOA of a sound source
active at time , we first calculate the time difference of arrivalat each time–fre-(50)

ACKNOWLEDGMENT
The authors would like to thank D. Zoeke from the Universityof Erlangen-Nuremberg for helping to implement the system in[8] and calculate the diarization performance, Dr. T. Matsubayashi for computational support, and anonymous reviewers for
insightful comments and advice.
REFERENCES[1] J. Carletta, S. Ashby, S. Bourban, M. Flynn, M. Guillemot, T. Hain,
J. Kadlec, V. Karaiskos, W. Kraaij, M. Kronenthal, G. Lathoud, M.
Lincoln, A. Lisowska, I. McCowan, W. Post, D. Reidsma, and P.
Wellner, “The AMI meeting corpus: A pre-announcement,” in Proc.
Workshop Multimodal Interaction Related Mach. Learn. Algorithms(MLMI), 2006, vol. 3869, Lecture Notes in Compuer Science.
[2] D. Mostefa, N. Moreau, K. Choukri, G. Potamianos, S. M. Chu, A.
Tyagi, J. R. Casas, J. Turmo, L. Cristoforetti, F. Tobia, A. Pnevmatikakis, V. Mylonakis, F. Talantzis, S. Burger, R. Stiefelhagen, K.
Bernardin, and C. Rochet, “The CHIL audiovisual corpus for lecture
and meeting analysis inside smart rooms,” Lang. Resources Eval., vol.
41, no. 3-4, pp. 389–407, 2007.
[3] J. G. Fiscus, J. Ajot, and J. S. Garofolo, “The rich transcription 2007
meeting recognition evaluation,” in Proc. Int. Eval. Workshops CLEAR
2007 and RT 2007: Multimodal Technol. for Perception of Humans,
2008, vol. 4625, Lecture Notes in Computer Science, pp. 373–389.
[4] A. Waibel, M. Bett, F. Metze, K. Rise, T. Schaaf, T. Schultz, H. Soltau,
H. Yu, and K. Zechner, “Advances in automatic meeting record creation
and access,” in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process.
(ICASSP), 2001, pp. 597–600.

ISHIGURO et al.: PROBABILISTIC SPEAKER DIARIZATION WITH BoW REPRESENTATIONS[5] R. Cutler, Y. Rui, A. Gupta, J. J. Cadiz, I. Tashev, L.-W. He, A. Colburn,
Z. Zhang, Z. Liu, and S. Silverberg, “Distributed meetings: A meeting
capture and broadcasting system,” in Proc. ACM Int. Conf. Multimedia,
2002, pp. 503–512.
[6] Z. Yu, M. Ozeki, Y. Fujii, and Y. Nakamura, “Towards smart meeting:
Enabling technologies and a real-world application,” in Proc. ACM Int.
Conf. Multimodal Interface, 2007, pp. 86–93.
[7] S. Tranter and D. Reynolds, “An overview of automatic speaker diarization systems,” IEEE Trans. Audio, Speech, Lang. Process., vol.
14, no. 5, pp. 1557–1565, Sep. 2006.
[8] C. Wooters and M. Huijbregts, “The ICSI RT07s speaker diarization
systems,” in Proc. Int. Eval. Workshops CLEAR 2007 and RT 2007:
Multimodal Technol. for Perception of Humans, 2008, vol. 4625, Lecture Notes in Computer Science, pp. 509–519.
[9] J. Huang, E. Marcheret, K. Visweswariah, and G. Potamianos, “TheIBM RT07 evaluation systems for speaker diarization on lecture meetings,” in Proc. Int. Eval. Workshops CLEAR 2007 and RT 2007: Multimodal Technol. for Perception of Humans, 2008, vol. 4625, Lecture
Notes in Computer Science, pp. 497–508.
[10] E. Fox, E. Sudderth, M. Jordan, and A. Willsky, “An HDP-HMM forsystems with state persistence,” in Proc. 25th Int. Conf. Mach. Learn.
(ICML), Helsinki, Finland, Jul. 2008.
[11] G. Friedland and O. Vinyals, “Live speaker identification in conversations,” in Proc. 16th ACM Int. Conf. Multimedia, 2008.
[12] X. Anguera, C. Wooters, J. M. Pardo, and J. Hernando, “Automaticweighting for the combination of TDOA and acoustic features inspeaker diarization for meetings,” in Proc. IEEE Int. Conf. Acoust.,
Speech, Signal Process. (ICASSP), 2007, vol. 4, pp. 241–244.
[13] S. Araki, M. Fujimoto, K. Ishizuka, H. Sawada, and S. Makino, “A
DOA based speaker diarization system for real meetings,” in Proc. Joint
Workshop Hands-Free Speech Commun. Microphone Arrays, 2008, pp.
29–32.
[14] J. M. Pardo, X. Anguera, and C. Wooters, “Speaker diarization for
multi-microphone meetings using only between-channel differences,”in Proc. 3rd Joint Workshop Multimodal Interaction and Rel. Mach.
Learn. Algorithms, Washington, DC, 2006, pp. 257–264.
[15] I. Pruteanu-Malinici, L. Ren, J. Paisley, E. Wang, and L. Carin, “Hierarchical Bayesian modeling of topics in time-stamped documents,”
IEEE Trans. Pattern Anal. Mach. Intell., vol. 32, no. 6, pp. 996–1011,
Jun. 2010.
[16] D. M. Blei, A. Y. Ng, and M. I. Jordan, “Latent Dirichlet allocation,”
J. Mach. Learn. Res., vol. 3, pp. 993–1022, Jan. 2003.
[17] K. Ishiguro, T. Yamada, S. Araki, and T. Nakatani, “A probabilistic
speaker clustering for DOA-based diarization,” in Proc. IEEE Workshop Applicat. Signal Process. Audio Acoust. (WASPAA 2009), 2009,
pp. 241–244.
[18] M. Fujimoto, K. Ishizuka, and T. Nakatani, “A voice activity detection
based on adaptive integration of multiple speech features and a signal
decision scheme,” in Proc. IEEE Int. Conf. Acoust., Speech, Signal
Process. (ICASSP), Las Vegas, NV, Mar. 2008, pp. 4441–4444.
[19] S. Araki, H. Sawada, R. Mukai, and S. Makino, “DOA estimation for
multiple sparse sources with normalized observation vector clustering,”
in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process. (ICASSP),
Toulouse, France, May 2006, vol. 5, pp. 33–36.
[20] R. O. Duda, P. E. Hart, and D. G. Stork, Pattern Classification. New
York: Wiley, 2001.
[21] J. Sivic, B. C. Russell, A. Zisserman, W. T. Freeman, and A. A. Efros,“Unsupervised discovery of visual object class hierarchies,” in Proc.
IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognition (CVPR),
Anchorage, AK, Jun. 2008, pp. 1–8.
[22] E. Bart, I. Porteous, P. Perona, and M. Welling, “Unsupervised learningof visual taxonomies,” in Proc. IEEE Comput. Soc. Conf. Comput. Vis.
Pattern Recognition (CVPR), Anchorage, AK, Jun. 2008, pp. 1–8.
[23] T. Hospedales, S. Gong, and T. Xiang, “A Markov clustering topic
model for mining behaviour in video,” in Proc. IEEE 12th Int. Conf.
Comput. Vis. (ICCV), 2009, pp. 1165–1172.
[24] X. Zhang, Z. Li, L. Zhan, W.-Y. Ma, and H.-Y. Shum, “Efficient indexing for large scale visual search,” in Proc. IEEE 12th Int. Conf.
Comput. Vis. (ICCV), 2009, pp. 1103–1110.
[25] M. Rodriguez, S. Ali, and T. Kanade, “Tracking in unstructured
crowded scenes,” in Proc. IEEE 12th Int. Conf. Comput. Vis. (ICCV),
2009, pp. 1389–1396.

459[26] S. Kim, S. Narayanan, and S. Sundaram, “Acoustic topic model foraudio information retrieval,” in Proc. IEEE Workshop Applicat. Signal
Process. Audio Acoust. (WASPAA), 2009, pp. 37–40.
[27] D. G. Lowe, “Distinctive image features from scale-invariant keypoints,” Int. J. Comput. Vis., vol. 60, no. 2, pp. 91–110, Nov. 2004.
[28] S. Araki, T. Nakatani, H. Sawada, and S. Makino, “Blind sparsesource separation for unknown number of sources using Gaussian
mixture model fitting with Dirichlet prior,” in Proc. 34th IEEE Int.
Conf. Acoust., Speech, Signal Process. (ICASSP), Taipei, Taiwan, Apr.
2009, pp. 33–36.
[29] C. M. Bishop, Pattern Recognition and Machine Learning. New
York: Springer-Verlag, 2006.
[30] L. Ren, D. B. Dunson, and L. Carin, “The dynamic hierarchical
Dirichlet process,” in Proc. 25th Int. Conf. Mach. Learn. (ICML),
Helsinki, Finland, Jul. 2008, pp. 824–831.
[31] H. Attias, “A variational Bayesian framework for graphical models,” inAdvances in Neural Information Processing Systems (NIPS). Cambridge, MA: MIT Press, 2000, vol. 12, pp. 209–215.
[32] S. Watanabe, Y. Minami, A. Nakamura, and N. Ueda, “Variational
Bayesian estimation and clustering for speech recognition,” IEEE
Trans. Speech Audio Process., vol. 12, no. 4, pp. 365–381, Jul. 2004.
[33] P. D. Hoff, “Subset clustering of binary sequences, with an application to genomic abnormality data,” Biometrics, vol. 61, no. 4, pp.
1027–1036, 2005.
[34] “AMI Project,” in AMI Meeting Corpus [Online]. Available: http://
corpus.amiproject.org[35] K. Boakye, B. Trueba-Hornero, O. Vinyals, and G. Friedland, “Overlapped speech detection for improved speaker diarization in multiparty
meetings,” in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process.
(ICASSP), 2008, pp. 4353–4356.
[36] G. Friedland, C. Yeo, and H. Hung, “Dialocalization: Acoustic speakerdiarization and visual localization as joint optimization problem,” ACM
Trans. Multimedia Comput., Commun., Applicat. (TOMCCAP), vol. 6,
no. 4, pp. 27:1–27:18, 2010.
[37] S. Araki, H. Sawada, and S. Makino, “Blind speech separation in a
meeting situation with maximum SNR beamformers,” in Proc. IEEE
Int. Conf. Acoust., Speech, Signal Process. (ICASSP), 2007, vol. 1, pp.
41–44.
[38] A. Noulas and B. J. A. Krose, “On-line multi-modal speaker diarization,” in Proc. Int. Conf. Multimodal Interfaces (ICMI), 2007, pp.
350–357.

Katsuhiko Ishiguro (M’09) received the B.Eng.
and M.Inf. degrees from the University of Tokyo,
Tokyo, Japan, in 2000 and 2004, respectively, and
the Ph.D. degree from the University of Tsukuba,
Ibaraki, Japan, in 2010.
He has been a Researcher at the NTT Communication Science Laboratories, NTT Corporation,
Kyoto, Japan, since 2006. His research interests
include multimedia data modeling with Bayesian
approaches, probabilistic models for data mining,
and time series analysis.
Dr. Ishiguro is a member of the IEICE and IPSJ.

Takeshi Yamada (M’08) received the B.S. degree inmathematics from the University of Tokyo, Tokyo,
Japan, in 1988 and the Ph.D. degree in informatics
from Kyoto University, Kyoto, Japan, in 2003.
He is currently a Senior Research Scientist, Supervisor, and Leader of the Emergent Learning and Systems Research Group, NTT Communication Science
Laboratories, Kyoto, Japan. His research interests include data mining, statistical machine learning, graphvisualization, metaheuristics, and combinatorial optimization.
Dr. Yamada is a member of the IEICE.

460IEEE TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, VOL. 20, NO. 2, FEBRUARY 2012

Shoko Araki (M’01) received the B.E. and M.E. degrees from the University of Tokyo, Tokyo, Japan,
in 1998 and 2000, respectively, and the Ph.D. degreefrom Hokkaido University, Sapporo, Japan in 2007.
She is with NTT Communication Science Laboratories, NTT Corporation, Kyoto, Japan. Since
she joined NTT in 2000, she has been engaged inresearch on acoustic signal processing, array signal
processing, blind source separation (BSS) applied
to speech signals, meeting diarization, and auditory
scene analysis.
Dr. Araki is a member of the organizing committee of the ICA 2003, thefinance chair of IWAENC 2003, the co-chair of a special session on undetermined sparse audio source separation in EUSIPCO 2006, the registration chairof WASPAA 2007, and the evaluation co-chair of SiSEC2008, 2010 and 2011.
She received the 19th Awaya Prize from Acoustical Society of Japan (ASJ) in
2001, the Best Paper Award of the IWAENC in 2003, the TELECOM System
Technology Award from the Telecommunications Advancement Foundation in
2004, the Academic Encouraging Prize from the Institute of Electronics, Information, and Communication Engineers (IEICE) in 2006, and the Itakura Prize
Innovative Young Researcher Award from ASJ in 2008. She is a member of the
IEICE and ASJ.

Tomohiro Nakatani (M’03–SM’06) received the
B.E., M.E., and Ph.D. degrees from Kyoto University, Kyoto, Japan, in 1989, 1991, and 2002,
respectively.
He is a Senior Research Scientist at NTT Communication Science Labs, NTT Corporation, Kyoto.
Since he joined NTT Corporation in 1991, he has
been investigating speech enhancement technologies
for developing intelligent human–machine interfaces. From 2005 to 2006, he was a Visiting Scholar
at the Georgia Institute of Technology. Since 2007,
he has been a Visiting Associate Professor at Nagoya University.
Dr. Nakatani was honored to receive the 1997 JSAI Conference Best Paper
Award, and the 2005 IEICE Paper Award. He is a member of IEEE Signal
Processing Audio and Electroacoustics Technical Committee and IEEE CAS
Blind Signal Processing Technical Committee, an Associate Editor of the IEEE
TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, and a Technical Program Chair of IEEE WASPAA-2007. He is a Technical Program Committee Chair of the IEEE Kansai Section. He is a member of IEICE and ASJ.

Hiroshi Sawada (M’02–SM’04) received the B.E.,
M.E., and Ph.D. degrees in information science from
Kyoto University, Kyoto, Japan, in 1991, 1993, and2001, respectively.
He joined NTT Corporation, Kyoto, in 1993. He is
now the Group Leader of the Learning and Intelligent
Systems Research Group, NTT Communication Science Laboratories, Kyoto. His research interests include statistical signal processing, machine learning,
latent variable model, graph-based data structure, andcomputer architecture.
Dr. Sawada served as an Associate Editor of the IEEE TRANSACTIONS ON
AUDIO, SPEECH, AND LANGUAGE PROCESSING from 2006 to 2009. He is amember of the Audio and Acoustic Signal Processing Technical Committee of
the IEEE SP Society. He received the Ninth TELECOM System Technology
Award for Student from the Telecommunications Advancement Foundation in
1994, the Best Paper Award of the IEEE Circuit and System Society in 2000,
and the MLSP Data Analysis Competition Award in 2007. Dr. Sawada is amember of IEICE and ASJ.