Hierarchical Dirichlet Processes AMS 241 Project , Fall 2010

This work is part review and part experimental investigation of the work done by Teh, Jordan, Beal, and Blei on Hierarchical Dirichlet Processes, first published as a technical report in 2004 [7]. In this report, we review the theory and motivation behind hierarchical Dirichlet processes (HDP), and we study HDP inference using simulated data and a Gibbs sampler we developed for this purpose. We assume the reader is already familiar with the theory behind (non-hierarchical) Dirichlet processes and Dirichlet process mixture models, and focus specifically on the hierarchical extension of those models. Hierarchical Dirichlet process models deal with the problem of modeling data that is divided into groups which share some common traits, e.g. data from counties in a given state. They are a flexible, non-parametric extension to the standard Bayesian parametric hierarchical models. Parametric (Bayesian) hierarchical models assume that the data distribution in each group j = 1, . , J has the same form, e.g. Normal(μ, σ), albeit it with different underlying parameters for each group, e.g. group dependent means N(μj , σ ). The sharing of information in such models can be achieved by giving the group level distributions a common variance parameter, which is itself a random variable with a prior distribtion. The flexibility in HDP models comes from the fact that data distribution in each group is driven by a group-specific non-parametric Dirichlet process prior, thus allowing each group’s distribution to take on a completely different form. The sharing of information across different groups comes about by making the group level distributions dependent on a common global measure, which is also driven by a Dirichlet process. We formalize this idea below. The application that motivates our study of hierarchical Dirichlet processes is in the field of information retrieval (IR) : modeling text documents as mixtures of topics, i.e. topic modeling. In this scenario, each document (in a fixed collection of documents, i.e. a corpus) is treated as a “bag-of-words” : all the words in a given document are independent and exchangeable (this is obviously a very strong assumption, but it is nevertheless a standard one in the IR domain). We assume that the words in a document are generated from a number of latent mixture components or topics, and each topic is typically taken to be a (multinomial) distribution over a set of words from a finite and given vocabulary [4]. The goal is to discover the latent topics in a given set of documents and subsequently use them to generate a compact representation of each document in the corpus; this representation can be used to discover and/or cluster related documents. Mapping this to our HDP model, each document is a group of observations (words), and we model it as a mixture of topics (distributions); furthermore, we are allowed to share (re-use) topics among different groups (documents). The report is organized as follows. Section 2.1 formally defines the HDP and our data model; section 3 discusses various representations of the HDP and examples of the prior distributions it induces; section 4 describes a Gibbs sampling procedure for inference in HDP models; section 5 describes our experimental setup and results; section 6 concludes the report.