Topic Modeling In NLP. With a focus on Latent Dirichlet... | by Arun Jagota | Towards Data Science

In natural language processing, the term topic means a set of words that “go together”. These are the words that come to mind when thinking of this topic. Take sports. Some such words are athlete, soccer, and stadium.
A topic model is one that automatically discovers topics occurring in a collection of documents. A trained model may then be used to discern which of these topics occur in new documents. The model can also pick out which portions of a document cover which topics.
Consider Wikipedia. It has millions of documents covering hundreds of thousands of topics. Wouldn’t it be great if these could be discovered automatically? Plus a finer map of which documents cover which topics. These would be useful adjuncts for people seeking to explore Wikipedia.
We could also discover emerging topics, as documents get written about them. In some settings (such as news) where new documents are constantly being produced and recency matters, this would help us detect trending topics.
This post covers a statistically powerful and widely used approach to this problem.
Latent Dirichlet Allocation
This approach involves building explicit statistical models of topics and documents.
A topic is modeled as a probability distribution over a fixed set of words (the lexicon). This formalizes “the set of words that come to mind when referring to this topic”. A document is modeled as a probability distribution over a fixed set of topics. This reveals the topics the document covers.
The aim of learning is to discover, from a corpus of documents, good word distributions of the various topics, as well as good topic proportions in the various documents. The number of topics is a parameter to this learning.
Generating A Document
At this stage, it will help to describe how to generate a synthetic document from a learned model. This will reveal key aspects of how this model operates that we haven’t delved into yet.
First, we’ll pick the topics this document will cover. One way to do this is to first pick a random document from our corpus, then set the new document’s topic proportions to those of the seed document.
Next, we’ll set the document length, call it n.
Next, we will repeat the following n times:
sample a topic from the document’s topic proportionssample a word from the chosen topic’s words-distribution
This will emit a sequence of n words. These words will come annotated with the topics they were sampled from.
The resulting document is gibberish. A bag of words sampled from a mix of topics. That’s not a problem — it wasn’t meant to be read. It does reveal which words were generated from which topics, which can be insightful.
Example
Lexicon: {athlete, football, soccer, tennis, computer, smartphone, laptop, printer,Intel, Apple, Google}Num Topics : 3Topic 1: {athlete, football, soccer, tennis}Topic 2: {computer, smartphone, laptop, printer}Topic 3: {Intel, Apple, Google}Topic proportions in a document: { 2 ⇒ 70%, 3 ⇒ 30% }
In the above, we’ve described a topic as a set of words. We interpret this as: all the words in the set are equiprobable; the remaining words in the lexicon have zero probability.
Let’s see a 4-word generated document.
Topic:  2      3       2          2Word: laptop Intel smartphone computer
Topic 3’s proportion in this document (25%) is close to its proportion (30%) in its sampling distribution.
Learning
As usual, this is where things get especially interesting.
First, let’s remind ourselves of the aim of learning. It is to discover, from a corpus of documents, the word distributions of the various topics, and the topic proportions in the various documents. In short, what words describe which topic, and which topics are covered in which document.
The algorithm we’ll describe is in wide use. It is also not hard to understand. It is a form of Gibbs Sampling.
This algorithm works by initially assigning the topics to the various words in the corpus somehow, then iteratively improving these assignments. During its operation, the algorithm keeps track of certain statistics on the current assignments. These statistics help the algorithm in its subsequent learning. When the algorithm terminates, it is easy to “read off” the per-topic word distributions and the per-document topic proportions from the final topic assignments.
Let’s start by describing the statistics mentioned in the previous paragraph. These take the form of two matrices of counts: topic_word and doc_topic. Both are derived from the current assignment of topics to the words in the corpus. topic_word(t,w) counts the number of occurrences of topic t for word w. doc_topic(d,t) counts the number of occurrences of topic t in document d.
Let’s see a numeric example to make sure we got it right. Below we see a two-document corpus along with an assignment of topics to its words. The lexicon is A, B, C.
Doc 1’s words:  A B A C A        Doc 2’s words:  B C C BDoc 1’s topics: 1 1 1 2 2        Doc 2’s topics: 2 2 2 2
Actually let’s first use this opportunity to muse about some peculiarities we see. In doc 1, notice that A is assigned sometimes to topic 1 and sometimes to topic 2. This is plausible if word A has a high probability in both topics. In doc 2, notice that B is consistently assigned to topic 2. This is plausible if Doc 2 covers only topic 2, and B has a positive probability in topic 2’s distribution.
Okay, now to the two matrices of counts.
topic_word:           doc_topic:  A B C                    1 21 2 1 0                 d1 3 22 1 2 3                 d2 0 4
We’ve bolded some entries that are a bit striking. Perhaps doc2 prefers topic 2. Perhaps topic 2 prefers word C.
Ok, let’s start explaining the learning. The first step is to label the words in the corpus with randomly-sampled topics. This sounds easy enough. Actually there is a bit more to it. Instead of hard-coding this random-sampling, it is better to sample off suitable prior distributions. This gives us a potentially powerful mechanism to inject domain knowledge or results from external text analyses.
This priors-based mechanism works as follows. First, we make copies of the two matrices we introduced earlier. Call them prior_topic_word and prior_doc_topic respectively. As before, the entries in these matrices are counts. These counts capture our prior beliefs.
These prior matrices influence the initial assignment of topics. As learning progresses, this influence diminishes, albeit not to zero.
How exactly do we sample the initial assignment of topics from these counts? First, we calculate
P(w|t) = prior_topic_word(t,w)/sum_w’ (prior_topic_word(t,w’))P(t|d) = prior_doc_topic(t,d)/sum_t’ (prior_doc_topic(t’,d)
P(w|t) is just the fraction of the assignments of topic t whose word is w. P(t|d) is just the fraction of the words in document d whose assigned topic is t.
Next, we sample the assignments from these. More specifically, we sample the topic for word w in document d from a distribution whose numerator is P(w|t)P(t|d).
This may be understood as follows. P(w|t)P(t|d) is exactly the probability of generating word w in document d in our generative model. Viewed as a function of t, it captures the likelihood that t was used during this process.
Now let’s discuss setting the values of these counts in the two prior matrices. For our purposes here, all we care about is that no topic be preferred over another. Such preferences would be unwanted biases. We can achieve this by setting all the counts in each matrix to the same positive number. 1 is the simplest choice. Occam’s razor reasoning.
prior_topic_word(t,w)=1 for every topic t and word wprior_doc_topic(d,t)=1 for every document d and topic t
Okay, so the topic assignments will be sampled from these counts and come out uniformly random.
Subsequent to this initial assignment, we will repeatedly do the following in the hopes of improving the assignment and consequently, our models learned from it:
1. Pick a word w from a document d in the corpus2. Sample a topic t’ from the distribution whose numerator is    Q(w|t)Q(t|d)3. Set w’s topic in d to t’.
What is Q(w|t)? It is our current belief in the likelihood of generating the word w from topic t. In fact, good values of Q(w|t) are what we seek. These will form the final topic-specific words-distributions.
Before starting learning, we captured whatever prior beliefs we had about this distribution into P(w|t). As learning progresses, P(w|t) starts getting revised into the posterior belief Q(w|t).
Q(t|d)’s explanation is similar. Our prior beliefs about the per-document topic proportions go into P(t|d). As learning progresses, these get revised into Q(t|d).
How are Q(w|t) and Q(t|d) computed? Consider the assignment of topics to the various words in the corpus at any point during learning. From this assignment of topics, we can calculate the counts in the topic_word and the doc_topic matrices. Next, we do the following
posterior_topic_word = topic_word + prior_topic_wordposterior_doc_topic  = doc_topic  + prior_doc_topic
Note that the ‘+’ is matrix addition. From the posterior versions of the count matrices, we may compute Q(w|t) and Q(t|d). Just as we computed P(w|t) and P(t|d) from the prior versions.
Some Intuition
How do we know that this iterative process actually improves the topic assignments? We won’t give a proof. Instead, some intuition.
First note that the overall quality of an assignment of topics to the words in the corpus may be obtained by multiplying out the various Q(w|t)Q(t|d) terms, over all word occurrences w in the corpus. Here d denotes the document in which w appears, and t the topic assigned to that occurrence.
Next, we’ll reveal that the score function has certain desirable characteristics.
This Score Function Favors Topic Specificity
By “topic specificity” we mean that Q(w|t) is concentrated on only a few topics. This is a desired property of those words that strongly correlate with specific topics. Let’s elaborate. Consider a diverse corpus such as Wikipedia. Say our aim is to discover the wide variety of topics it covers. Consider the word cat. Its topic specificity is high, i.e. it conjures up only a few of these topics. As it should. So the score function’s bias towards favoring topic specificity is a good thing.
That said, not every word should be topic-specific. For example the. Later we’ll discuss a separate mechanism that will counteract topic specificity in such cases. First, let’s explain the topic-specificity bias. We’ll call topic-specific words informative.
Consider n >> 1 occurrences of the same word w in the corpus, perhaps spread across many documents. Let T1 and T2 be two different topic assignments to these n occurrences. All the topics in T1 are distinct. Call this set {1,2,3,...,n}. All the topics in T2 are the same, the one that maximizes Q(w|t). Call this topic tmax. T1’s likelihood under Q(w|t) is Q(w|1)Q(w|2)*...*Q(w|n). T2’s likelihood under Q(w|t) is Q(w|tmax)^n. T2’s likelihood can be way higher than T1’s when w is informative and n is not too small.
This analysis may be summarized as
The score function encourages informative words to stay on topic
This Score Function Favors Document Specificity
By “document specificity” we mean that the document covers only a few topics. Documents tend to be specific. The question is whether the score function (and consequently the learning algorithm) exploits this tendency to do its job better. The answer is yes. As explained below.
Consider a document d on n words. Let T1 and T2 be two different topic assignments to its words. All the topics in T1 are distinct. Call this set {1,2,3,...,n}. All the topics in T2 are the same, this time the one with the highest proportion in d. Call it tmax. T1’s likelihood under Q(t|d) is Q(1|d)Q(2|d)*...*Q(n|d). T2’s likelihood under Q(t|d) is Q(tmax|d)^n. Clearly T2 can be way higher, especially for large documents.
This analysis may be summarized as
The score function encourages documents to stay on topic
These Influences Sometimes Compete!
Consider an especially common word: the. Reasonably assume it appears in almost all documents. Topic consistency favors all these occurrences being assigned the same topic. Document specificity protests, as this would force all these documents to cover this one topic.
Let’s imagine how this might play out. Topic consistency might back off its grip for such words. Their assigned topics might just “go with the flow”, taking on the identity of whichever topic is being covered in the neighborhood. Sure, the topic consistency component of the likelihood of these assignments might decrease. The document specificity portion, on the other hand, would increase.
More On The Learning Algorithm
We’ve seen that the score function is biased in good ways. This is only helpful if the learning algorithm exploits them well. So let’s pivot to discussing the algorithm a bit more.
We’ll start by noting that the algorithm works by locally optimizing the global topic assignment quality score. This alone suggests it is paying attention to the biases.
Next, we’ll run through an iteration or two in a simple example. This will help the reader get a better feel for its “local optimizing” behavior. It’s more nuanced than some might imagine.
Example
We’ll set the lexicon to {A, B}. Our corpus will have two documents, AAA (d1) and BBB (d2). (While these may sound silly, this exercise will be instructive.) We’ll aim to fit two topics to the corpus.
Let’s set all the prior counts to 0.0000001. It will still produce initial assignments that are uniformly random, though simplify our numeric calculations.
Say the following is the initial assignment
d1     d2AAA    BBB121    122
The learned model from this assignment is
Q(A|1)  = 2⁄3 , Q(B|1) = 1⁄3    // these sum to 1Q(A|2)  = 1⁄3 , Q(B|2) = 2⁄3    // these sum to 1Q(1|d1) = 2⁄3 , Q(2|d1)= 1⁄3    // these sum to 1Q(1|d2) = 1⁄3, Q(2|d2) = 2⁄3    // these sum to 1
Now imagine picking a word from d1 and resampling its topic. (We could have chosen d2 instead but the reasoning is similar.) There are three possible outcomes: 121 → 121, 121 → 111, 121 → 122 or 221. The third one is effectively where we started with: swap the topic names and rearrange the order. So the first and the third outcome effectively put us back in the same state. So let’s zoom in on the second one: 121 → 111. The probability of this transition is positive. (In fact quite high as the switch from 2 → 1 improves both the topic likelihood component and also the document likelihood component.) So if we keep repeating this process at some point all of d1 will be assigned the same topic t (=1 or 2).
Next, we reestimate the various parameters of the model from this assignment t to d1. Q(t|d1) is now 1. Q(A|t) may well be higher than Q(A|1) was when we began this process. We have arrived at a “happy state” from which it is difficult to escape.
Variants
Now let’s look at some variants of this algorithm. These involve modifying step 2 below.
1. Pick a word w from a document d in the corpus2. Sample a topic t’ from the distribution whose numerator is    Q(w|t)Q(t|d)3. Set w’s topic in d to t’.
We can replace step 2 by “set t’ to the topic that maximizes Q(w|t)Q(t|d)”. This is called maximum likelihood estimation. In our setting, this yields a greedy algorithm.
This variant is attractive in its simplicity and may converge quicker. However, it is more susceptible to getting trapped in suboptimal local optima.
Our next variant introduces a parameter called temperature, which may be varied to span behaviors ranging from the Gibbs sampler to the greedy algorithm. Therein lies this algorithm’s appeal. It does open up a new issue though: how to set the temperature.
Setting temperature aside, let’s see how it works. Consider the word w whose current topic t is being considered for re-assignment. For every topic t’ we evaluate
delta(t’) = — [ log Q(w|t’)Q(t’|d) — log Q(w|t)Q(t|d) ]
We won’t explain why the log here. We will note that delta(t’) being less than 0 corresponds to t’ being a better fit than t in this situation. A switch from t to t’ may be viewed as an energy-reducing (or hill-descending) move.
What happens next? We’ll describe it qualitatively. So now we know delta(t’) for the various values of t’, including t. From these delta-values, the algorithm defines a suitable probability distribution over the topics. This distribution is parameterized by temperature.
At high temperature, even “up-hill moves”, i.e. moves to topics whose delta-values are positive are permitted. Such moves, while regressing on the current topic assignment, can help escape local energy minima.
At low temperature, the distribution favors moves whose delta-values are negative. In the zero temperature extreme, this yields greedy behavior.
Richer Priors
To this point, we’ve been set prior counts to 1. Here we consider richer settings.
Previously we noted that it is desirable for certain words to be topic-specific. Can we find such words and capture their topic-specificity preference into the prior counts? This could speed up subsequent learning.
Here is a sensible way to do this. First, we’ll set all the prior counts to 1. Next, for each word w, we’ll compute n-n(w). Here n is the total number of documents in the corpus and n(w) the number of documents in which word w appears. Think of n-n(w) as a sort-of “inverse document frequency” of w. Next, we will do the following independently for each of the words. We will pick a topic t randomly from the universe of topics. We’ll add n-n(w) to the prior count word_topic(t,w).
The idea is to map uncommon words to specific topics. Since the topic choices are random, different words will likely map to different topics.
Is this prior overly biased? We can easily soften it should we feel that is the case. There are a number of possibilities. One is to replace n-n(w) by log n — log n(w). A second is to sample more than one topic (albeit not many) when amplifying a word’s prior count.
Other Modeling Enhancements
What are other areas to consider for improving the model? First, let’s spell out its assumptions:
1. The topics a document covers are sampled independently. They   are not checked for compatibility.2. Word proximity in a document is not considered.3. The hierarchical structure among topics is not modeled.
Let’s say a bit about each. Topic compatibility can be important. A document is more likely to cover tech companies & computers than tech companies & sports. Word proximity also matters. Two words that repeatedly occur near each other are more likely to be on the same topic than those that appear thousands of words apart. Topic hierarchies model documents better. A common writing pattern is for a document to cover a broad topic together with its various subtopics.
Below we describe how to address these issues. The relaxations may improve the accuracy of the model in some use cases, though at the potential cost of increasing its complexity or learnability. They also offer opportunities to inject domain knowledge. Plus in some ways even help with the learning!
Exactly how this plays out depends heavily on the use case.
Topic Compatibilities
This involves adding a Markov model to the mix. Its states are the topics. Its transitions model compatibility among topics in the corpus.
How is this model used when sampling a word’s topic in a document? We need to extend Q(t|d) suitably. This extension is easiest described by imagining sampling topics from it.
This sampling may be viewed as a random walk on the Markov model with perhaps occasional restarts. We start the walk by choosing a topic sampled from Q(t|d). Next, we do the following. Most of the time we walk a transition from this topic’s state with the probability that is on that transition. Occasionally we re-sample from Q(t|d), i.e., jump to a new topic.
This random walk induces a new distribution, call it Q’(t|d, M) which is influenced by both the Markov model M and Q(t|d). The Markov model is a corpus-level construct. The use of Q(t|d) customizes its behavior to the document d.
How are the parameters of the Markov model learned? This is easier to explain. Our learning algorithm to this point already works at the level of assigning topics to individual words in the documents of the corpus. The Markov model’s parameters are fully determined by this assignment.
Specifically, for every pair s, t of topic assignments in a document, we increment counts on the arcs s → t and t → s unless s equals t in which case only once.
So as topic assignments improve, the parameters of the topic-compatibility model also improve. The two act synergistically.
Let’s look at this synergy in a wider scope. As we see more and more documents on the same two topics, the Markov model starts learning that these topics are compatible. This improved learning leads to improved topic assignments elsewhere.
Word Proximity
This can be incorporated by extending Q(w|t) suitably. Specifically, let the choice of t be influenced not only from w but also from words near w.
Let’s formalize this. Let W(d) = w(-d),...,w,...,w(+d) denote the sequence of words of length 2d+1 centered at w. Here d is a nonnegative integer. Next, we extend Q(w|t) to Q(W(d)|t).
Does this complicate our model from the learning perspective? With a suitable assumption, fortunately not. We assume the words in W(d) are conditionally independent given t. Under this assumption we get
Q(W(d)|t) = Q(w(-d)|t)*...*Q(w|t)*...*Q(w(+d)|t)
The RHS terms involving t are all of the same forms as before. So learning needs no change!
The assumption notwithstanding, we get the benefits that come with using context. Chief among them is that the topic assigned to the central word w may be determined more accurately, as now we have more context. Another helpful characteristic, especially early on in the learning, is that we get some topic continuity. As we slide the window W(d) one word at a time, the assigned topic is less likely to change since the context hasn’t changed much. Compared to W(0).
Topics Hierarchy
This is an advanced topic. Our coverage here is partial.
Here is the key observation. Comparing the word distributions of two topics can help assess whether or not one is a descendant of the other. The descendant’s keywords will tend towards being a subset of the ancestor’s.
Using this notion, we can (re)learn a hierarchy on the topics any time we want during the learning process. Even a rough learned hierarchy can be better than none.
As we did for topic compatibility, we can extend our Q(t|d) models to take the learned hierarchy into account. As there, this extension is easiest described by imagining sampling topics for a specific document from it.
This sampling is a random walk on the hierarchy with perhaps occasional restarts. We start the walk by choosing a topic sampled from Q(t|d). Next, we do the following. Most of the time we walk to a child from this topic. Occasionally we resample from Q(t|d), i.e., jump to a topic somewhere else in the hierarchy.
This random walk induces a new distribution, call it Q’(t|d, H) which is influenced by both the topic hierarchy H and Q(t|d). The topic hierarchy is a corpus-level construct. The use of Q(t|d) customizes its behavior to the document d.
Enhancements That Improve Readability
The readability of a document is not the primary aim of LDA. Nonetheless, the modeling framework presents opportunities to inject mechanisms that improve it. So let’s take advantage of this.
Let’s start by surfacing the issues.
1. Topic continuity is not maintained. Neighboring words can jump    from topic to topic.2. Topic coherence is not maintained. A topic spews out its words    independently.
Topic continuity
We can maintain topic continuity by adding a Markov model with two states: continue and switch. Continue to continue the current topic, switch to switch to a new one. The arc from continue to itself should have a high probability. The arc from switch to continue should have a near-1 probability as we almost surely want to continue following a switch.
The probabilities on this model are easy to learn from the topic assignments. Buried in them are the continue and switch events. Such learning dovetails with the learning of the topic assignments, iteratively benefiting each other.
Leveraging word proximity in documents also helps maintain topic continuity. The reason we chose to cover the approach of this section as well is that it is far simpler to implement than word proximity.
This Also Benefits Learning
Readability aside, injecting the topic continuity mechanism also potentially improves the quality of the learned assignments. In effect, it acts as a smoothness regularizer, disfavoring outliers.
Here is an example, an extension of one we used earlier. Consider the word the, which we assumed occurs in almost all documents in the corpus. Previously we explained why we would like the topic assigned to a particular occurrence of the to “go with the flow” of the topic in its neighborhood. The topic continuity mechanism adds more weight to this preference, as it explicitly favors “go with the flow”.
Topic Coherence
We can maintain topic coherence by relaxing the bag-of-words assumption. Instead, use a first-order Markov model that models the influence of the current word on the next one. Each topic has its own Markov model.
The per-topic Markov models are easy to (re)learn from the current topic assignment in the corpus.
Further Reading
David M. Blei. Probabilistic Topic Models. Communications of the ACM. 2012 http://www.cs.columbia.edu/~blei/papers/Blei2012.pdf
David M. Blei, Andrew Ng, Michael Jordan. Latent Dirichlet allocation. JMLR (3) 2003 pp. 993–1022.
Tom Griffiths. Gibbs sampling in the generative model of Latent Dirichlet Allocation.https://people.cs.umass.edu/~wallach/courses/s11/cmpsci791ss/readings/griffiths02gibbs.pdf