What is perplexity in topic modeling?
Perplexity is a measure of how successfully a trained topic model predicts new data. In LDA topic modeling of text documents, perplexity is a decreasing function of the likelihood of new documents. In other words, as the likelihood of the words appearing in new documents increases, as assessed by the trained LDA model, the perplexity decreases. And vice-versa. The idea is that a low perplexity score implies a good topic model, ie. one that is good at predicting the words that appear in new documents. Although this makes intuitive sense, studies have shown that perplexity does not correlate with the human understanding of topics generated by topic models. Hence, while perplexity is a mathematically sound approach for evaluating topic models, it is not a good indicator of human-interpretable topics.