Dirichlet Process
Yee Whye Teh, University College London
Related keywords: Bayesian nonparametrics, stochastic processes, clustering,
infinite mixture model, Blackwell-MacQueen urn scheme, Chinese restaurant
process.

Definition
The Dirichlet process is a stochastic proces used in Bayesian nonparametricmodels of data, particularly in Dirichlet process mixture models (also known as
infinite mixture models). It is a distribution over distributions, i.e. each draw
from a Dirichlet process is itself a distribution. It is called a Dirichlet process because it has Dirichlet distributed finite dimensional marginal distributions, justas the Gaussian process, another popular stochastic process used for Bayesiannonparametric regression, has Gaussian distributed finite dimensional marginal
distributions. Distributions drawn from a Dirichlet process are discrete, butcannot be described using a finite number of parameters, thus the classification
as a nonparametric model.

Motivation and Background
Probabilistic models are used throughout machine learning to model distributions over observed data. Traditional parametric models using a fixed and finite
number of parameters can suffer from over- or under-fitting of data when there
is a misfit between the complexity of the model (often expressed in terms of thenumber of parameters) and the amount of data available. As a result, model selection, or the choice of a model with the right complexity, is often an important
issue in parametric modeling. Unfortunately, model selection is an operationthat is fraught with difficulties, whether we use cross validation or marginal
probabilities as the basis for selection. The Bayesian nonparametric approach isan alternative to parametric modeling and selection. By using a model with anunbounded complexity, underfitting is mitigated, while the Bayesian approachof computing or approximating the full posterior over parameters mitigates overfitting. A general overview of Bayesian nonparametric modeling can be foundunder its entry in the encyclopedia [?].
Nonparametric models are also motivated philosophically by Bayesian modeling. Typically we assume that we have an underlying and unknown distribu-

1

tion which we wish to infer given some observed data. Say we observe x1 , . . . , xn ,with xi ∼ F independent and identical draws from the unknown distribution
F . A Bayesian would approach this problem by placing a prior over F thencomputing the posterior over F given data. Traditionally, this prior over distributions is given by a parametric family. But constraining distributions tolie within parametric families limits the scope and type of inferences that canbe made. The nonparametric approach instead uses a prior over distributions
with wide support, typically the support being the space of all distributions.
Given such a large space over which we make our inferences, it is importantthat posterior computations are tractable.
The Dirichlet process is currently one of the most popular Bayesian nonparametric models. It was first formalized in [1]1 for general Bayesian statisticalmodeling, as a prior over distributions with wide support yet tractable posteriors. Unfortunately the Dirichlet process is limited by the fact that draws fromit are discrete distributions, and generalizations to more general priors did nothave tractable posterior inference until the development of MCMC techniques[3, 4]. Since then there has been significant developments in terms of inference algorithms, extensions, theory and applications. In the machine learning
community work on Dirichlet processes date back to [5, 6].

TheoryThe Dirichlet process (DP) is a stochastic process whose sample paths are probability measures with probability one. Stochastic processes are distributions overfunction spaces, with sample paths being random functions drawn from the distribution. In the case of the DP, it is a distribution over probability measures,
which are functions with certain special properties which allow them to be interpreted as distributions over some probability space Θ. Thus draws from a DP
can be interpreted as random distributions. For a distribution over probability
measures to be a DP, its marginal distributions have to take on a specific form
which we shall give below. We assume that the user is familiar with a modicumof measure theory and Dirichlet distributions.
Before we proceed to the formal definition, we will first give an intuitive
explanation of the DP as an infinite dimensional generalization of Dirichlet
distributions. Consider a Bayesian mixture model consisting of K components:θk∗ |H ∼ Hαα,..., K)π|α ∼ Dir( Kzi |π ∼ Mult(π)

xi |zi , {θk∗ } ∼ F (θz∗i )(1)where π is the mixing proportion, α is the pseudocount hyperparameter ofthe Dirichlet prior, H is the prior distribution over component parameters θk∗ ,and F (θ) is the component distribution parametrized by θ. It can be shownthat for large K, because of the particular way we parametrized the Dirichletprior over π, the number of components typically used to model n data items
1 Note however that related models in population genetics date back to [2].

2

becomes independent of K and is approximately O(α log n). This implies thatthe mixture model stays well-defined as K → ∞, leading to what is known as
an infinite mixture model [5, 6]. This model was first proposed as a way tosidestep the difficult problem of determining the number of components in a
mixture, and as a nonparametric alternative to finite mixtures whose size cangrow naturally with the number of data items. The more modern definition ofthis model uses a DP and with the resulting model called a DP mixture model.
The DP itself
PK appears as the K → ∞ limit of the random discrete probability
measure k=1 πk δθk∗ , where δθ is a point mass centred at θ. We will return to
the DP mixture towards the end of this entry.
Dirichlet Process
For a random distribution G to be distributed according to a DP, its marginal
distributions have to be Dirichlet distributed [1]. Specifically, let H be a distribution over Θ and α be a positive real number. Then for any finite measurable
partition A1 , . . . , Ar of Θ the vector (G(A1 ), . . . , G(Ar )) is random since G is
random. We say G is Dirichlet process distributed with base distribution H andconcentration parameter α, written G ∼ DP(α, H), if(G(A1 ), . . . , G(Ar )) ∼ Dir(αH(A1 ), . . . , αH(Ar ))(2)

for every finite measurable partition A1 , . . . , Ar of Θ.
The parameters H and α play intuitive roles in the definition of the DP. Thebase distribution is basically the mean of the DP: for any measurable set A ⊂ Θ,
we have E[G(A)] = H(A). On the other hand, the concentration parameter canbe understood as an inverse variance: V [G(A)] = H(A)(1 − H(A))/(α + 1).
The larger α is, the smaller the variance, and the DP will concentrate moreof its mass around the mean. The concentration parameter is also called thestrength parameter, referring to the strength of the prior when using the DP
as a nonparametric prior over distributions in a Bayesian nonparametric model,
and the mass parameter, as this prior strength can be measured in units ofsample size (or mass) of observations. Also, notice that α and H only appear ase = αH,
their product in the definition (2) of the DP. Some authors thus treat H
eas the single (positive measure) parameter of the DP, writing DP(H) instead ofDP(α, H). This parametrization can be notationally convenient, but loses thedistinct roles α and H play in describing the DP.
Since α describes the concentration of mass around the mean of the DP, as
α → ∞, we will have G(A) → H(A) for any measurable A, that is G → H
weakly or pointwise. However this not equivalent to saying that G → H. As weshall see later, draws from a DP will be discrete distributions with probability
one, even if H is smooth. Thus G and H need not even be absolutely continuouswith respect to each other. This has not stopped some authors from using the
DP as a nonparametric relaxation of a parametric model given by H. However,
if smoothness is a concern, it is possible to extend the DP by convolving G withkernels so that the resulting random distribution has a density.

3

A related issue to the above is the coverage of the DP within the classof all distributions over Θ. We already noted that samples from the DP are
discrete, thus the set of distributions with positive probability under the DP issmall. However it turns out that this set is also large in a different sense: if the
topological support of H (the smallest closed set S in Θ with H(S) = 1) is allof Θ, then any distribution over Θ can be approximated arbitrarily accurately
in the weak or pointwise sense by a sequence of draws from DP(α, H). This
property has consequence in the consistency of DPs discussed later.
For all but the simplest probability spaces, the number of measurable partitions in the definition (2) of the DP can be uncountably large. The natural
question to ask here is whether objects satisfying such a large number of conditions as (2) can exist. There are a number of approaches to establish existence.
[1] noted that the conditions (2) are consistent with each other, and made useof Kolmogorov’s consistency theorem to show that a distribution over functions
from the measurable subsets of Θ to [0, 1] exists satisfying (2) for all finite
measurable partitions of Θ. However it turns out that this construction does
not necessarily guarantee a distribution over probability measures. [1] also provided a construction of the DP by normalizing a gamma process. In a later
section we will see that the predictive distributions of the DP are related to theBlackwell-MacQueen urn scheme. [7] made use of this, along with de Finetti’stheorem on exchangeable sequences, to prove existence of the DP. All the above
methods made use of powerful and general mathematical machinery to establish
existence, and often require regularity assumptions on H and Θ to apply thesemachinery. In a later section, we describe a stick-breaking construction of the
DP due to [8], which is a direct and elegant construction of the DP which need
not impose such regularity assumptions.
Posterior Distribution
Let G ∼ DP(α, H). Since G is a (random) distribution, we can in turn draw
samples from G itself. Let θ1 , . . . , θn be a sequence of independent draws fromG. Note that the θi ’s take values in Θ since G is a distribution over Θ. We areinterested in the posterior distribution of G given observed values of θ1 , . . . , θn .
Let A1 , . . . , Ar be a finite measurable partition of Θ, and let nk = #{i : θi ∈ Ak }be the number of observed values in Ak . By (2) and the conjugacy between the
Dirichlet and the multinomial distributions, we have:(G(A1 ), . . . , G(Ar ))|θ1 , . . . , θn ∼ Dir(αH(A1 ) + n1 , . . . , αH(Ar ) + nr )(3)

Since the above is true for all finite measurable partitions, the posterior distribution over G must be a DP as well. A little algebra shows that the posterior
DP
P
αH+ n
i=1 δθi,
has updated concentration parameter α + n and base distribution
α+nPn
where δi is a point mass located at θi and nk = i=1 δi (Ak ). In other words,
the DP provides a conjugate family of priors over distributions that is closed
under posterior updates given observations. Rewriting the posterior DP, we

4have:
Pnαni=1 δθiG|θ1 , . . . , θn ∼ DP α + n, α+nH + α+nn(4)Notice that the posterior base distribution is a weighted P
average between thenδθiprior base distribution H and the empirical distribution i=1. The weightnassociated with the prior base distribution is proportional to α, while the empirical distribution has weight proportional to the number of observations n.
Thus we can interpret α as the strength or mass associated with the prior. In
the next section we will see that the posterior base distribution is also the predictive distribution of θn+1 given θ1 , . . . , θn . Taking α → 0, the prior becomes
non-informative in the sense that the predictive distribution is just given by theempirical distribution. On the other hand, as the amount of observations grows
large, n  α, the posterior is simply dominated by the empirical distribution
which is in turn a close approximation of the true underlying distribution. This
gives a consistency property of the DP: the posterior DP approaches the true
underlying distribution.
Predictive Distribution and the Blackwell-MacQueen Urn Scheme
Consider again drawing G ∼ DP(α, H), and drawing an i.i.d. sequence θ1 , θ2 , . . . ∼G. Consider the predictive distribution for θn+1 , conditioned on θ1 , . . . , θn andwith G marginalized out. Since θn+1 |G, θ1 , . . . , θn ∼ G, for a measurable A ⊂ Θ,
we have
P (θn+1 ∈ A|θ1 , . . . , θn ) = E[G(A)|θ1 , . . . , θn ]1=α+nαH(A) +nX!
δθi (A)(5)i=1where the last step follows from the posterior base distribution of G given thefirst n observations. Thus with G marginalized out:
!
n
X
1
δθi
(6)
θn+1 |θ1 , . . . , θn ∼
αH +
α+n
i=1
Therefore the posterior base distribution given θ1 , . . . , θn is also the predictive
distribution of θn+1 .
The sequence of predictive distributions (6) for θ1 , θ2 , . . . is called the BlackwellMacQueen urn scheme [7]. The name stems from a metaphor useful in interpreting (6). Specifically, each value in Θ is a unique color, and draws θ ∼ G
are balls with the drawn value being the color of the ball. In addition we havean urn containing previously seen balls. In the beginning there are no balls in
the urn, and we pick a color drawn from H, i.e. draw θ1 ∼ H, paint a ball withthat color, and drop it into the urn. In subsequent steps, say the n + 1st, we
αwill either, with probability α+n, pick a new color (draw θn+1 ∼ H), paint anball with that color and drop the ball into the urn, or, with probability α+n,5reach into the urn to pick a random ball out (draw θn+1 from the empirical
distribution), paint a new ball with the same color and drop both balls backinto the urn.
The Blackwell-MacQueen urn scheme has been used to show the existenceof the DP [7]. Starting from (6), which are perfectly well-defined conditional
distributions regardless of the question of the existence of DPs, we can construct a distribution over sequences θ1 , θ2 , . . . by iteratively drawing each θigiven θ1 , . . . , θi−1 . For n ≥ 1 let
P (θ1 , . . . , θn ) =nY

P (θi |θ1 , . . . , θi−1 )

(7)

i=1

be the joint distribution over the first n observations, where the conditional
distributions are given by (6). It is straightforward to verify that this random
sequence is infinitely exchangeable. That is, for every n, the probability ofgenerating θ1 , . . . , θn using (6), in that order, is equal to the probability ofdrawing them in any alternative order. More precisely, given any permutation
σ on 1, . . . , n, we have
P (θ1 , . . . , θn ) = P (θσ(1) , . . . , θσ(n) )(8)

Now de Finetti’s theorem states that for any infinitely exchangeable sequenceθ1 , θ2 , . . . there is a random distribution G such that the sequence is composedof i.i.d. draws from it:
Z Yn
P (θ1 , . . . , θn ) =G(θi ) dP (G)(9)
i=1

In our setting, the prior over the random distribution P (G) is precisely the
Dirichlet process DP(α, H), thus establishing existence.
A salient property of the predictive distribution (6) is that it has point masses
located at the previous draws θ1 , . . . , θn . A first observation is that with positive
probability draws from G will take on the same value, regardless of smoothnessof H. This implies that the distribution G itself has point masses. A further
observation is that for a long enough sequence of draws from G, the value ofany draw will be repeated by another draw, implying that G is composed onlyof a weighted sum of point masses, i.e. it is a discrete distribution. We will see
two sections below that this is indeed the case, and give a simple construction
for G called the stick-breaking construction. Before that, we shall investigate
the clustering property of the DP.
Clustering, Partitions and the Chinese Restaurant Process
In addition to the discreteness property of draws from a DP, (6) also implies aclustering property. The discreteness and clustering properties of the DP play
crucial roles in the use of DPs for clustering via DP mixture models, describedin the application section. For now we assume that H is smooth, so that all6repeated values are due to the discreteness property of the DP and not due to∗
H itself2 . Since the values of draws are repeated, let θ1∗ , . . . , θmbe the unique∗values among θ1 , . . . , θn , and nk be the number of repeats of θk . The predictive
distribution can be equivalently written as:
!
m
X
1
αH +
(10)
nk δθk∗θn+1 |θ1 , . . . , θn ∼α+nk=1

Notice that value θk∗ will be repeated by θn+1 with probability proportional tonk , the number of times it has already been observed. The larger nk is, thehigher the probability that it will grow. This is a rich-gets-richer phenomenon,
where large clusters (a set of θi ’s with identical values θk∗ being considered acluster) grow larger faster.
We can delve further into the clustering property of the DP by looking atpartitions induced by the clustering. The unique values of θ1 , . . . , θn inducea partitioning of the set [n] = {1, . . . , n} into clusters such that within eachcluster, say cluster k, the θi ’s take on the same value θk∗ . Given that θ1 , . . . , θn
are random, this induces a random partition of [n]. This random partitionin fact encapsulates all the properties of the DP, and is a very well studied
mathematical object in its own right, predating even the DP itself [2, 9, 10].
To see how it encapsulates the DP, we simply invert the generative process.
Starting from the distribution over random partitions, we can reconstruct thejoint distribution (7) over θ1 , . . . , θn , by first drawing a random partition on [n],then for each cluster k in the partition draw a θk∗ ∼ H, and finally assign θi = θk∗for each i in cluster k. From the joint distribution (7) we can obtain the DP by
appealing to de Finetti’s theorem.
The distribution over partitions is called the Chinese restaurant process(CRP) due to a different metaphor3 . In this metaphor we have a Chinese
restaurant with an infinite number of tables, each of which can seat an infinite
number of customers. The first customer enters the restaurant and sits at thefirst table. The second customer enters and decides either to sit with the first
customer, or by herself at a new table. In general, the n + 1st customer either
joins an already occupied table k with probability proportional to the number
nk of customers already sitting there, or sits at a new table with probability
proportional to α. Identifying customers with integers 1, 2, . . . and tables asclusters, after n customers have sat down the tables define a partition of [n]with the distribution over partitions being the same as the one above. The
fact that most Chinese restaurants have round tables is an important aspect of
the CRP. This is because it does not just define a distribution over partitionsof [n], it also defines a distribution over permutations of [n], with each tablecorresponding to a cycle of the permutation. We do not need to explore this
aspect further and refer the interested reader to [9, 10].
This distribution over partitions first appeared in population genetics, where
it was found to be a robust distribution over alleles (clusters) among gametes
2 Similar conclusions can be drawn when H has atoms, there is just more bookkeeping.
3 The name was coined by Lester Dubins and Jim Pitman in the early 1980’s [9].

7(observations) under simplifying assumptions on the population, and is known
under the name of Ewens sampling formula [2]. Before moving on we shallconsider just one illuminating aspect, specifically the distribution of the numberof clusters among n observations. Notice that for i ≥ 1, the observation θiαtakes on a new value (thus incrementing m by one) with probability α+i−1
independently of the number of clusters among previous θ’s. Thus the numberof cluster m has mean and variance:nXα= α(ψ(α + n) − ψ(α))
α+i−1
i=1

n
' α log 1 +for N, α  0,αV [m|n] = α(ψ(α + n) − ψ(α)) + α2 (ψ 0 (α + n) − ψ 0 (α))
n' α log 1 +for n > α  0,αE[m|n] =

(11)

(12)

where ψ(·) is the digamma function. Note that the number of clusters grows only
logarithmically in the number of observations. This slow growth of the numberof clusters makes sense because of the rich-gets-richer phenomenon: we expect
there to be large clusters thus the number of clusters m has to be smaller than
the number of observations n. Notice that α controls the number of clusters
in a direct manner, with larger α implying a larger number of clusters a priori.
This intuition will help in the application of DPs to mixture models.
Stick-breaking Construction
We have already intuited that draws from a DP are composed of a weighted sumof point masses. [8] made this precise by providing a constructive definition of
the DP as such, called the stick-breaking construction. This construction is
also significantly more straightforward and general than previous proofs of theexistence of DPs. It is simply given as follows:θk∗ ∼ Hβk ∼ Beta(1, α)
π k = βkk−1
Y(1 − βk )

G=

l=1

∞
X

πk δθk∗(13)

k=1

Then G ∼ DP(α, H). The construction of π can be understood metaphoricallyas follows. Starting with a stick of length 1, we break it at β1 , assigning π1
to be the length of stick we just broke off. Now recursively break the otherportion to obtain π2 , π3 and so forth. The stick-breaking distribution over π issometimes written π ∼ GEM(α), where the letters stand for Griffiths, Engen
and McCloskey [10]. Because of its simplicity, the stick-breaking construction
has lead to a variety of extensions as well as novel inference techniques for the
Dirichlet process [11].

8Applications
Because of its simplicity, DPs are used across a wide variety of applicationsof Bayesian analysis in both statistics and machine learning. The simplest
and most prevalent applications include: Bayesian model validation, density
estimation and clustering via mixture models. We shall briefly describe the firsttwo classes before detailing DP mixture models.
How does one validate that a model gives a good fit to some observed data?
The Bayesian approach would usually involve computing the marginal probability of the observed data under the model, and comparing this marginal
probability to that for other models. If the marginal probability of the modelof interest is highest we may conclude that we have a good fit. The choice ofmodels to compare against is an issue in this approach, since it is desirable tocompare against as large a class of models as possible. The Bayesian nonparametric approach gives an answer to this question: use the space of all possible
distributions as our comparison class, with a prior over distributions. The DP
is a popular choice for this prior, due to its simplicity, wide coverage of the classof all distributions, and recent advances in computationally efficient inference in
DP models. The approach is usually to use the given parametric model as thebase distribution of the DP, with the DP serving as a nonparametric relaxation
around this parametric model. If the parametric model performs as well or better than the DP relaxed model, we have convincing evidence of the validity ofthe model.
Another application of DPs is in density estimation [12, 5, 3, 6]. Here weare interested in modeling the density from which a given set of observations
is drawn. To avoid limiting ourselves to any parametric class, we may again
use a nonparametric prior over all densities. Here again DPs are a popular.
However note that distributions drawn from a DP are discrete, thus do nothave densities. The solution is to smooth out draws from the DP with a kernel.
Let G ∼ DP(α, H) and let f (x|θ) be a family of densities (kernels) indexed by
θ. We use the following as our nonparametric density of x:Zp(x) = f (x|θ)G(θ) dθ
(14)Similarly, smoothing out DPs in this way is also useful in the nonparametricrelaxation setting above. As we see below, this way of smoothing out DPs
is equivalent to DP mixture models, if the data distributions F (θ) below aresmooth with densities given by f (x|θ).
Dirichlet Process Mixture Models
The most common application of the Dirichlet process is in clustering datausing mixture models [12, 5, 3, 6]. Here the nonparametric nature of the Dirichlet process translates to mixture models with a countably infinite number ofcomponents. We model a set of observations {x1 , . . . , xn } using a set of latentparameters {θ1 , . . . , θn }. Each θi is drawn independently and identically from9

G, while each xi has distribution F (θi ) parametrized by θi :xi |θi ∼ F (θi )θi |G ∼ G
G|α, H ∼ DP(α, H)(15)Because G is discrete, multiple θi ’s can take on the same value simultaneously,
and the above model can be seen as a mixture model, where xi ’s with the samevalue of θi belong to the same cluster. The mixture perspective can be mademore in agreement with the usual representation of mixture models using the
stick-breaking construction (13). Let zi be a cluster assignment variable, whichtakes on value k with probability πk . Then (15) can be equivalently expressedasπ|α ∼ GEM(α)θk∗ |H ∼ H

zi |π ∼ Mult(π)

xi |zi , {θk∗ } ∼ F (θz∗i )

(16)

P∞

with G = k=1 πk δθk∗ and θi = θz∗i . In mixture modeling terminology, π is the
mixing proportion, θk∗ are the cluster parameters, F (θk∗ ) is the distribution overdata in cluster k, and H the prior over cluster parameters.
The DP mixture model is an infinite mixture model—a mixture model witha countably infinite number of clusters. However, because the πk ’s decrease
exponentially quickly, only a small number of clusters will be used to model thedata a priori (in fact, as we saw previously, the expected number of components
used a priori is logarithmic in the number of observations). This is different
than a finite mixture model, which uses a fixed number of clusters to model thedata. In the DP mixture model, the actual number of clusters used to model
data is not fixed, and can be automatically inferred from data using the usual
Bayesian posterior inference framework (see [4] for a survey of MCMC inference
procedures for DP mixture models). The equivalent operation for finite mixture
models would be model averaging or model selection for the appropriate numberof components, an approach which is fraught with difficulties. Thus infinite
mixture models as exemplified by DP mixture models provide a compelling
alternative to the traditional finite mixture model paradigm.

Generalizations and Extensions
The DP is the canonical distribution over probability measures and a wide range
of generalizations have been proposed in the literature. First and foremost is
the Pitman-Yor process [13, 11], which has recently seen successful applications
modeling data exhibiting power-law properties [14, 15]. The Pitman-Yor process includes a third parameter d ∈ [0, 1), with d = 0 reducing to the DP.
The various representations of the DP, including the Chinese restaurant process and the stick-breaking construction, have analogues for the Pitman-Yor
process. Other generalizations of the DP are obtained by generalizing one of10its representations. These include Pólya trees, normalized random measure,
Poisson-Kingman models, species sampling models and stick-breaking priors.
The DP has also been used in more complex models involving more than
one random probability measure. For example, in nonparametric regression wemight have one probability measure for each value of a covariate, and in multitask settings each task might be associated with a probability measure withdependence across tasks implemented using a hierarchical Bayesian model. In
the first situation the class of models is typically called dependent Dirichlet processes [16], while in the second the appropriate model is a hierarchical Dirichlet
process [17].

Future Directions
The Dirichlet process, and Bayesian nonparametrics in general, is an active areaof research within both machine learning and statistics. Current research trends
span a number of directions. Firstly there is the issue of efficient inference in
DP models. [4] is an excellent survey of the state-of-the-art in 2000, with allalgorithms based on Gibbs sampling or small-step Metropolis-Hastings MCMC
sampling. Since then there has been much work, including split-and-merge andlarge-step auxiliary variable MCMC sampling, sequential Monte Carlo, expectation propagation, and variational methods. Secondly there has been interestin extending the DP, both in terms of new random distributions, as well as
novel classes of nonparametric objects inspired by the DP. Thirdly, theoreticalissues of convergence and consistency are being explored to provide frequentist
guarantees for Bayesian nonparametric models. Finally there are applicationsof such models, to clustering, transfer learning, relational learning, models ofcognition, sequence learning, and regression and classification among others.
We believe DPs and Bayesian nonparametrics will prove to be rich and fertile
grounds for research for years to come.

Cross References
Bayesian Methods, Prior Probabilities, Bayesian Nonparametrics.

Recommended Reading
In addition to the references embedded in the text above, we recommend thebook [18] on Bayesian nonparametrics.
[1] T. S. Ferguson. A Bayesian analysis of some nonparametric problems.
Annals of Statistics, 1(2):209–230, 1973.
[2] W. J. Ewens. The sampling theory of selectively neutral alleles. Theoretical
Population Biology, 3:87–112, 1972.

11

[3] M. D. Escobar and M. West. Bayesian density estimation and inference
using mixtures. Journal of the American Statistical Association, 90:577–
588, 1995.
[4] R. M. Neal. Markov chain sampling methods for Dirichlet process mixture
models. Journal of Computational and Graphical Statistics, 9:249–265,
2000.
[5] R. M. Neal. Bayesian mixture modeling. In Proceedings of the Workshop
on Maximum Entropy and Bayesian Methods of Statistical Analysis, volume 11, pages 197–211, 1992.
[6] C. E. Rasmussen. The infinite Gaussian mixture model. In Advances in
Neural Information Processing Systems, volume 12, 2000.
[7] D. Blackwell and J. B. MacQueen. Ferguson distributions via Pólya urn
schemes. Annals of Statistics, 1:353–355, 1973.
[8] J. Sethuraman. A constructive definition of Dirichlet priors. Statistica
Sinica, 4:639–650, 1994.
[9] D. Aldous. Exchangeability and related topics. In École d’Été de Probabilités de Saint-Flour XIII–1983, pages 1–198. Springer, Berlin, 1985.
[10] J. Pitman. Combinatorial stochastic processes. Technical Report 621, Department of Statistics, University of California at Berkeley, 2002. Lecturenotes for St. Flour Summer School.
[11] H. Ishwaran and L. F. James. Gibbs sampling methods for stick-breaking
priors. Journal of the American Statistical Association, 96(453):161–173,
2001.
[12] A.Y. Lo. On a class of bayesian nonparametric estimates: I. density estimates. Annals of Statistics, 12(1):351–357, 1984.
[13] J. Pitman and M. Yor. The two-parameter Poisson-Dirichlet distribution
derived from a stable subordinator. Annals of Probability, 25:855–900, 1997.
[14] S. Goldwater, T.L. Griffiths, and M. Johnson. Interpolating between types
and tokens by estimating power-law generators. In Advances in Neural
Information Processing Systems, volume 18, 2006.
[15] Y. W. Teh. A hierarchical Bayesian language model based on Pitman-Yor
processes. In Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics, pages 985–992, 2006.
[16] S. MacEachern. Dependent nonparametric processes. In Proceedings of theSection on Bayesian Statistical Science. American Statistical Association,
1999.

12

[17] Y. W. Teh, M. I. Jordan, M. J. Beal, and D. M. Blei. Hierarchical Dirichlet
processes. Journal of the American Statistical Association, 101(476):1566–
1581, 2006.
[18] N. Hjort, C. Holmes, P. Müller, and S. Walker, editors. Bayesian Nonparametrics. Number 28 in Cambridge Series in Statistical and Probabilistic
Mathematics. Cambridge University Press, 2010.

13