CS598JHM: Advanced NLP (Spring ʼ10)

with the most likely topic assignments FIGURE 4. The analysis of a document from Science. Document similarity was computed using Eq. (4); topic words were computed using Eq. (3). the assignment of words to topics in the abstract of the article, and the top ten most similar articles. 3. POSTERIOR INFERENCE FOR LDA The central computational problem for topic modeling with LDA is approximating the posterior in Eq. (2). This distribution is the key to using LDA for both quantitative tasks, such as prediction and document generalization, and the qualitative exploratory tasks that we discuss here. Several approximation techniques have been developed for LDA, including mean field variational inference (Blei et al., 2003), collapsed variational inference (Teh et al., 2006), expectation propagation (Minka and Lafferty, 2002), CS598JHM: Advanced NLP What is the distribution of words defined by a topic? Instead of using term probabilities directly, downweight them by how likely they are to be generated by any topic: Visualizing a topic 19 6 D. M. BLEI AND J. D. LAFFERTY Visualizing a topic. Exploring a corpus through a topic model typically begins with visualizing the posterior topics through their per-topic term probabilities β. The simplest way to visualize a topic is to order the terms by their probability. However, we prefer the following score, (3) term-scorek,v = βk,v log   βk,v K j=1 β j,v 1 K   . This is inspired by the popular TFIDF term score of vocabulary terms used in information retrieval Baeza-Yates and Ribeiro-Neto (1999). The first expression is akin to the term frequency; the second expression is akin to the document frequency, down-weighting terms that have high probability under all the topics. Other methods of determining the difference between a topic and others can be found in (Tang and MacLennan, 2005). Visualizing a document. We use the posterior topic proportions θd,k and posterior topic assignmentszd,n,k to visualize the underlying topic decomposition of a document. Plotting the posterior topic proportions gives a sense of which topics the document is “about. ” These vectors can also be used to group articles that exhibit certain topics with high proportions. Note that, in contrast to traditional clustering models (Fraley and Raftery, 2002), articles contain multiple topics and thus can belong to multiple groups. Finally, examining the most likely topic assigned to each word gives a sense of how the topics are divided up within the document. Finding similar documents. We can further use the posterior topic proportions to define a topic-based similarity measure between documents. These vectors provide a low dimensional simplicial representation of each document, reducing their representation from the (V − 1)-simplex to the (K − 1)-simplex. One can use the Hellinger distance between documents as a similarity measure, (4) document-similarityd, f = K k=1 θd,k − θ f,k 2 . To illustrate the above three notions, we examined an approximation to the posterior distribution derived from the JSTOR archive of Science from 1980–2002. The corpus contains 21,434 documents comprising 16M words when we use the 10,000 terms chosen by TFIDF (see Section 3.2). The model was fixed to have 50 topics. We illustrate the analysis of a single article in Figure 4. The figure depicts the topic proportions, the top scoring words from the most prevalent topics, TOPIC MODELS 5 contractual employment female markets criminal expectation industrial men earnings discretion gain local women investors justice promises jobs see sec civil expectations employees sexual research process breach relations note structure federal enforcing unfair employer managers see supra agreement discrimination firm officer note economic harassment risk parole perform case gender large inmates FIGURE 3. Fiv topics from a 50-topic model fit to the Yale Law Journal from 1980–2003. of the properties of the Dir hlet, and repl ce these modeling choices with an alternative distribution over the simplex. 2.2. Exploring a corpus with the posterior distribution. LDA provides a joint distribution over the observed and hidden random variables. The hidden topic decomposition of a particular corpus arises from the corresponding poste ior distribution of the hidden variables given the D observed d cu s w1:D, p( θ1:D, z1:D,1:N , β1:K | w1:D,1:N , α, η) = (2) p( θ1:D, z1:D, β1:K | w1:D, α, η) β1:K θ1:D z p( θ1:D, z1:D, β1:K | w1:D, α, η) . Loosely, this posterior can be thought of the “reversal” of the generative process described above. Given the observed corpus, the posterior is a distribution of the hidden variables which generated it. As discussed in Blei et al. (2003), this distribution is intractable to compute because of the integral in the denominator. Before discus i g approximation m thods, however, we illustrate how the posterior distribution gives a decomposition of the corpus that can be used to better understand and organize its contents. The quantities needed for exploring a corpus are the posterior expectations of the hidden variables. These are the topic probability of a term βk,v = E[βk,v | w1:D,1:N ], the topic proportions of a document θd,k = E[θd,k | w1:D,1:N ], and the topic assignment of a word zd,n,k = E[Zd,n = k | w1:D,1:N ]. Note that each of these quantities is conditioned on the observed corpus. CS598JHM: Advanced NLP Visualizing a topic D. Blei and J. Lafferty. Topic Models. In A. Srivastava and M. Sahami, editors, Text Mining: Theory and Applications. Taylor and Francis, 2009.