Is Automated Topic Model Evaluation Broken?: The Incoherence of Coherence

Abstract:
Topic model evaluation, like evaluation of other unsupervised methods, can be contentious. However, the field has coalesced around automated estimates of topic coherence, which rely on the frequency of word co-occurrences in a reference corpus. Contemporary neural topic models surpass classical ones according to these metrics. At the same time, topic model evaluation suffers from a validation gap: automated coherence, developed for classical models, has not been validated using human experimentation for neural models. In addition, a meta-analysis of topic modeling literature reveals a substantial standardization gap in automated topic modeling benchmarks. To address the validation gap, we compare automated coherence with the two most widely accepted human judgment tasks: topic rating and word intrusion. To address the standardization gap, we systematically evaluate a dominant classical model and two state-of-the-art neural models on two commonly used datasets. Automated evaluations declare a winning model when corresponding human evaluations do not, calling into question the validity of fully automatic evaluations independent of human judgments. 1 Revisiting Topic Model Evaluation Topic models are a machine learning technique widely used outside computer science, including political science (Grimmer and Stewart, 2013; Isoaho et al., 2021), social and cultural studies (Mohr and Bogdanov, 2013), digital humanities (Meeks and Weingart, 2012), and bioinformatics (Liu et al., 2016). Typically, topic model users are domain experts trying to identify global categories or themes present in a document collection (Boyd-Graber et al., 2017). This practice constitutes a computer-assisted form of content analysis (Krippendorff, 2004; Chuang et al., 2014), also related to distant reading in literary studies (Underwood, 2017). In general, topic models help humans understand large corpora.2 Evaluation of topic models has vacillated between automated and human-centered. While real-world users of topic models evaluate outputs based on their specific needs, topic model developers have gravitated toward generalized, automated proxies of human judgment to help inform rapid iteration of models (Doogan and Buntine, 2021). Initially, models were evaluated with held-out perplexity, but it disagrees with human interpretability (Chang et al., 2009). Consequently, the field adopted automated coherence metrics like normalized pointwise mutual information (NPMI), a measure of word relatedness that does correlate with topic interpretability (Section 2.2; Newman et al., 2010; Aletras and Stevenson, 2013; Lau et al., 2014). The balance shifted towards automated coherence. Human evaluations have been abandoned by topic model developers in the years since automated coherence metrics were adopted. In a thorough meta-analysis of contemporary topic model methods ∗Equal contribution Topic models are also used for other purposes, such as information retrieval or downstream document classification. However, the discovery and application of categories for human interpretation is their dominant use, and other computational applications have been largely eclipsed by modern neural approaches. 35th Conference on Neural Information Processing Systems (NeurIPS 2021). Classical Neural station album tropical tropical spore manhattan_project line band storm landfall basidia los_alamos_laboratory bridge music hurricane cyclone spores robert_oppenheimer railway song cyclone utc mycologist enrico_fermi trains released depression weakening hyphae physicist NPMI 0.274 0.285 0.394 0.446 0.456 0.470 Table 1: The first three columns are the highest-NPMI topics for a classical topic model (LDA estimated via Gibbs sampling using Mallet, McCallum, 2002; Griffiths and Steyvers, 2004). The next three are counterparts from a neural model (our D-VAE reimplementation, Burkhardt and Kramer, 2019). Models are trained on Wikitext (Merity et al., 2017) with fifty topics, and NPMI is estimated over the top five words in each topic using a 4.6M-document reference Wikipedia corpus. The mean top-five NPMI over all topics is 0.156 for the classical and 0.256 for the neural model. papers, none conduct systematic human evaluations (Section 3). Instead, they rely solely on automated metrics for model comparison.3 However, current neural topic models are a far cry from the classical models that substantiated the original correlations—manifestly, topics produced by neural models are often qualitatively distinct from those of classical models (e.g., Table 1).4 This validation gap raises the question of whether automated metrics are still consistent with human judgments of topic quality. Moreover, we should always be cautious when extrapolating outside the range of data that was used to establish a relationship between variables. As an example, a neural model in Hoyle et al. (2020) produces much larger NPMI values than those used to determine human correlations in the original Lau et al. (2014) study; the implicit assumption is that greater NPMI corresponds to more humaninterpretable topics. Finally, a myopic focus on a presumed proxy for human preferences can produce low-quality results (Stiennon et al., 2020). Does Goodharts’ law—“when a measure becomes a target, it ceases to be a good measure” (Strathern, 1997)—apply to automated metrics of topic models? Another challenge for automated evaluation, whether of classical or neural topic models, is widespread inconsistency (Section 3). Researchers frequently fail to specify the information needed to calculate automated metrics or diverge from the practices that underpin human correlations. Furthermore, evaluation datasets, preprocessing, and hyperparameter optimization vary dramatically, even within a given paper. This standardization gap likely limits the generalizability and reliability of topic model developers’ findings. We address the standardization and validation gaps in topic model evaluation: 1. We present a meta-analysis of neural topic model evaluation (Section 3); 2. we develop standardized, pre-processed versions of two widely-used English-language evaluation datasets, along with a transparent end-to-end code pipeline for reproduction of results (Section 4.1)5; 3. we optimize three topic models—one classical and two neural—using identical preprocessing, model selection criteria, and hyperparameter tuning (Section 4.2); 4. we evaluate these models using human ratings and word intrusion tasks (Section 5); and 5. we provide new evaluations of the correlation between automated and human evaluations (Section 6). Our findings challenge the validity of fully-automated evaluations as currently practiced: automated evaluation declares winners between models when the corresponding human evaluations cannot. Outside of the core method-development literature, human evaluations have been used to develop new metrics and improve understanding of existing model behavior (Bhatia et al., 2017; Morstatter and Liu, 2018; Lund et al., 2019; Alokaili et al., 2019, inter alia). We use “classical” to mean generative models defined by a chain of conjugate exponential family distributions optimized by Gibbs sampling or variational inference. github.com/ahoho/topics

heading: 1 Revisiting Topic Model Evaluation
text: Topic models are a machine learning technique widely used outside computer science, including political science (Grimmer and Stewart, 2013; Isoaho et al., 2021), social and cultural studies (Mohr and Bogdanov, 2013), digital humanities (Meeks and Weingart, 2012), and bioinformatics (Liu et al., 2016). Typically, topic model users are domain experts trying to identify global categories or themes present in a document collection (Boyd-Graber et al., 2017). This practice constitutes a computer-assisted form of content analysis (Krippendorff, 2004; Chuang et al., 2014), also related to distant reading in literary studies (Underwood, 2017). In general, topic models help humans understand large corpora.2
Evaluation of topic models has vacillated between automated and human-centered. While real-world users of topic models evaluate outputs based on their specific needs, topic model developers have gravitated toward generalized, automated proxies of human judgment to help inform rapid iteration of models (Doogan and Buntine, 2021). Initially, models were evaluated with held-out perplexity, but it disagrees with human interpretability (Chang et al., 2009). Consequently, the field adopted automated coherence metrics like normalized pointwise mutual information (NPMI), a measure of word relatedness that does correlate with topic interpretability (Section 2.2; Newman et al., 2010; Aletras and Stevenson, 2013; Lau et al., 2014). The balance shifted towards automated coherence.
Human evaluations have been abandoned by topic model developers in the years since automated coherence metrics were adopted. In a thorough meta-analysis of contemporary topic model methods
∗Equal contribution 2Topic models are also used for other purposes, such as information retrieval or downstream document classification. However, the discovery and application of categories for human interpretation is their dominant use, and other computational applications have been largely eclipsed by modern neural approaches.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
papers, none conduct systematic human evaluations (Section 3). Instead, they rely solely on automated metrics for model comparison.3 However, current neural topic models are a far cry from the classical models that substantiated the original correlations—manifestly, topics produced by neural models are often qualitatively distinct from those of classical models (e.g., Table 1).4 This validation gap raises the question of whether automated metrics are still consistent with human judgments of topic quality.
Moreover, we should always be cautious when extrapolating outside the range of data that was used to establish a relationship between variables. As an example, a neural model in Hoyle et al. (2020) produces much larger NPMI values than those used to determine human correlations in the original Lau et al. (2014) study; the implicit assumption is that greater NPMI corresponds to more humaninterpretable topics. Finally, a myopic focus on a presumed proxy for human preferences can produce low-quality results (Stiennon et al., 2020). Does Goodharts’ law—“when a measure becomes a target, it ceases to be a good measure” (Strathern, 1997)—apply to automated metrics of topic models?
Another challenge for automated evaluation, whether of classical or neural topic models, is widespread inconsistency (Section 3). Researchers frequently fail to specify the information needed to calculate automated metrics or diverge from the practices that underpin human correlations. Furthermore, evaluation datasets, preprocessing, and hyperparameter optimization vary dramatically, even within a given paper. This standardization gap likely limits the generalizability and reliability of topic model developers’ findings.
We address the standardization and validation gaps in topic model evaluation:
1. We present a meta-analysis of neural topic model evaluation (Section 3); 2. we develop standardized, pre-processed versions of two widely-used English-language
evaluation datasets, along with a transparent end-to-end code pipeline for reproduction of results (Section 4.1)5; 3. we optimize three topic models—one classical and two neural—using identical preprocessing, model selection criteria, and hyperparameter tuning (Section 4.2); 4. we evaluate these models using human ratings and word intrusion tasks (Section 5); and 5. we provide new evaluations of the correlation between automated and human evaluations
(Section 6).
Our findings challenge the validity of fully-automated evaluations as currently practiced: automated evaluation declares winners between models when the corresponding human evaluations cannot.
3Outside of the core method-development literature, human evaluations have been used to develop new metrics and improve understanding of existing model behavior (Bhatia et al., 2017; Morstatter and Liu, 2018; Lund et al., 2019; Alokaili et al., 2019, inter alia).
4We use “classical” to mean generative models defined by a chain of conjugate exponential family distributions optimized by Gibbs sampling or variational inference.
5github.com/ahoho/topics

heading: 2 Operationalizing Topic Coherence
text: A topic model is a probabilistic generative model of text that uses latent topics to summarize a larger collection of documents. The most influential variant, latent Dirichlet allocation (Blei et al., 2003, LDA), assumes that K latent topics are distributions over word types, βk, and that the documents D are admixtures over the topics, θd. Users often evaluate model outputs globally, focusing on the most probable N words of each topic, and locally, considering the most probable topics for each document.
While techniques for topic modeling have progressed from variational inference (Blei et al., 2003) to Gibbs sampling (Griffiths and Steyvers, 2004) to deep generative approaches (Srivastava and Sutton, 2017; Wang et al., 2020b), the core goal discussed in Section 1, obtaining human-understandable categories, remains central. The latest wave of methods, neural topic models (NTM), use continuous word representations and gradient optimization to fit parameters. These models claim to produce more interpretable topics than other prior methods, including LDA.
Those claims are supported by improvements on automated measures of topic coherence.

heading: 2.1 Human Metrics of Topic Coherence
text: Like the concept of interpretability, that of real-world coherence is “simultaneously important and slippery” (Lipton, 2018). We will not attempt to formalize it here—though see discussion in Section 7. For present purposes, the term has its roots in Latin cohaerere, “to stick together,” and we will think of coherence as an intangible sense, available to human readers, that a set of terms, when viewed together, enable human recognition of an identifiable category.6 We review two human ratings of topic quality: direct ratings and intrusion.
Rating Raters see a topic and then give the topic a quality score, conventionally on a three-point ordinal scale (Newman et al., 2010; Mimno et al., 2011; Aletras and Stevenson, 2013, inter alia).
Intrusion Chang et al. (2009) devise the word intrusion task as a behavioral way to assess topic coherence. The core idea is that when the top words in a topic identify a coherent latent category, it is easier to identify words that do not belong to that category. Operationally, each topic is represented as its top words plus one “intruder” word which has a low probability of belonging to that topic, but a high probability of belonging to a different topic. Topic coherence is then judged by how well human annotators detect the “intruder” word.

heading: 2.2 NPMI: The Standard Automated Topic Model Coherence Evaluation
text: Using the word intrusion task, Chang et al. (2009) showed that perplexity—the original topic model evaluation metric—negatively correlates with human evaluations of topic quality. This finding revealed a need for an automated measurement of topic coherence: an automated metric can measure model quality without expensive, time-consuming, and difficult-to-reproduce human experiments.
Lau et al. (2014) find some metrics that positively correlate with human intrusion and rating scores, particularly when aggregating scores over all topics from a given model. Because of that validation, the prevailing evaluation for model comparison is pairwise normalized pointwise mutual information. NPMI scores topics highly if the top N words—summed over all pairs wi and wj—have high joint probability P (wj , wi) compared to their marginal probability:7
N∑︂ j=2 j−1∑︂ i=1 log P (wj ,wi) P (wi)P (wj) −logP (wi, wj) . (1)
The probabilities are estimated using word co-occurrence counts from a reference corpus for a specific context window (which can range from ten words to the entire document). As a result, the choice of reference corpus determines the strength of human correlation (Lau et al., 2014; Röder et al., 2015).
6This perspective aligns with Propositions 2 and 3 of Doogan and Buntine (2021): “an interpretable topic is one that can be easily labeled,” and “has high agreement on labels.”
7Alternative metrics exist, but they typically also rely on either joint probability estimates or NPMI directly (e.g., Cv Röder et al., 2015).
A measurement is valid to the extent that it measures what it is intended to measure in the real world. Historically, automated coherence has been validated using human judgements from either crowdworkers (Newman et al., 2010; Aletras and Stevenson, 2013) or experts (Mimno et al., 2011). However, correlations based on classical models may not be applicable for NTMs. Our skepticism is motivated by theory, as neural word representations are intimately connected to NPMI, as explicitly used by Aletras and Stevenson (2013) and which produce similar NPMI scores as Lau et al. (2014). Levy and Goldberg (2014) show that multiple representations create factorizations of PMI matrices. Topic models that have access to these rich representations (e.g. Dieng et al., 2020, and others) could thus create topics with good NPMI scores without explaining the corpus well to a user. In contrast to classical topic models, no one has investigated the validity of NPMI evaluation for NTMs.
Given this lacuna, we conduct experiments aimed at validating that automated topic evaluations still correlate with human judgments of neural topic model quality. We compare against two common human evaluations of individual topic quality: direct rating and intrusion. Human evaluations, like automated topic modeling, lack standardization, which we address in Section 5.

heading: 3 A Meta-Analysis of Neural Topic Modeling
text: We survey the neural topic modeling (NTM) literature to assess the state of evaluation in contemporary topic model development. First, we take all references made by an existing, comprehensive survey of NTMs (Zhao et al., 2021b), from which we select (a) modeling papers which (b) mention topic interpretability and (c) compare models’ topics with an existing baseline. This yields forty models, which all claim superior topic coherence. We examine data processing steps, hyperparameter tuning, baseline selection, and automated coherence calculations. Table 2 summarizes our results and Appendix A.1 enumerates the papers.
Our analysis reveals variance in all areas. Preprocessing, which can significantly affect model quality and automated metrics, is often (30%) inconsistent across datasets within the same paper. When preprocessing is consistent, authors omit details necessary to fully replicate the pipeline. These issues imply that automated metrics for the same baselines and source datasets vary across papers. Compounding the problem, researchers often train their models on different datasets from those used to establish the relationships between human annotations and automated metrics; Doogan and Buntine (2021) find that the same metrics may not predict interpretability in new domains. Mirroring findings from Dodge et al. (2019), 40% of papers fail to clearly specify their model tuning procedure, often even the metric used for model selection.
Calculation of automated coherence metrics is equally fraught. As discussed in Section 2.2, a complete specification for NPMI involves several pieces of information, including the reference corpus used to estimate joint word probabilities, the co-occurrence window size, and the number of words selected from the head of the topic distribution. Three out of four papers fail to explicitly indicate the reference corpus; even when we can assume the input corpus is used (13 cases), it remains uncertain whether authors use, e.g., a held-out set or the training documents themselves. For the 61% that specify the implementation of their coherence metric (by pointing to a code repository or writing out the formula), some of these factors may still be in question. For instance, six authors
reference Lau et al. (2014) and the supporting code,8 but the implications are ambiguous: the original paper suggests a large corpus from the same source as the training data, but the repository script defaults to Wikipedia. In other cases, authors use bespoke implementations, which creates room for errors, or deviate from the settings used in human experiments. For example, several papers use a document-wide context window with NPMI, which has not been correlated with human judgments.
Last, even if automated evaluations are consistent, all claims of coherence improvement depend on the validity results in Lau et al. (2014) generalizing to neural topic models.

heading: 4 Closing the Standardization Gap for Topic Models
text: Our human evaluation of topic model outputs serves multiple purposes: (a) establishing whether NTMs show improved coherence over a classical baseline and (b) re-evaluating the efficacy and reliability of automated coherence metrics. In addition, a key goal is (c) to provide a standardized preprocessing pipeline to support head-to-head comparisons as new methods are developed.9
We identify two commonly-used datasets, which we in turn process using a standard pipeline. We then estimate topic models on each dataset following a computationally fair hyperparameter search. Our standardization efforts are similar to concurrent work by Terragni et al. (2021); the main differences are that we (a) mandate consistent preprocessing between training and reference corpora, (b) support multi-word expressions during vocabulary creation (see below), and (c) support distributed hyperparameter searches.

heading: 4.1 Datasets and Preprocessing
text: Following Chang et al. (2009), we use English articles from Wikipedia and the New York Times (Table 7). For Wikipedia, we use Wikitext-103 (WIKI, Merity et al., 2017), and for the Times, we subsample roughly 15% of documents from LDC2008T19 (NYT, Sandhaus, 2008), making it an order of magnitude larger than WIKI. To compute reference counts, we use a 4.6M document Wikipedia dump from September 2017 and the full 1.8M document LDC2008T19 set, processed identically to the training data.
We use SpaCy (Honnibal et al., 2020) to tokenize and identify entities in the text. We create new tokens for detected entities of the form New_York_City, per Krasnashchok and Jouili (2018). Schofield and Mimno (2016) find that lemmatization and word-stemming can hurt English topic interpretability, so we do not lemmatize. To maintain a roughly equal vocabulary size over datasets, we use a power-law relationship of corpus size (c.f. Zipf, 1949) to rule out tokens occurring in fewer than a given number of documents.10 In addition to a standard stopword list, we define corpus-specific stopwords as tokens appearing in more than 90% of documents. See Appendix A.2 for complete preprocessing details.

heading: 4.2 Models
text: We evaluate one venerable classical model and two newer neural models:
Gibbs-LDA As a strong classical baseline, we use the widely-loved Mallet (McCallum, 2002) implementation of Gibbs-sampling for LDA (Griffiths and Steyvers, 2004). Mallet produces topics of (qualitatively) competitive quality to neural models (Srivastava and Sutton, 2017).
Dirichlet-VAE We reimplement Dirichlet-VAE (Burkhardt and Kramer, 2019), a state-of-the-art NTM. For simplicitly, we use pathwise gradients for the Dirichlet (Jankowiak and Obermeyer, 2018), rather than the rejection sampling variational inference of the authors’ primary variant.11 DirichletVAE is a wholesale improvement on one of the first successful NTMs, the popular ProdLDA (Srivastava and Sutton, 2017), and is competitive against recent models on automated coherence. The generative
8github.com/jhlau/topic_interpretability 9Our preprocessing pipeline is agnostic to dataset and easily portable. github.com/ahoho/topics
10We target vocabularies approximating the number of words known by an adult English-speaker (Brysbaert et al., 2016): roughly 40k for WIKI and 35k for NYT.
11We replicate their NPMI and redundancy scores on 20 newsgroups. github.com/ahoho/dvae
model is simple and retains a broad similarity to LDA. The primary difference is that it does not constrain the estimated topic-word distributions to the simplex.
ETM Thanks to their improved flexiblity, many NTMs incorporate external word representations, on the premise that large-scale, general language knowledge improves topic quality (Bianchi et al., 2021; Hoyle et al., 2020). The Embedded Topic Model (Dieng et al., 2020) is a popular NTM that relies on word embeddings in its generative model.12
We maintain a fixed computational budget per model following the exhortation of Dodge et al. (2019) and use a random set of 164 hyperparameter settings across datasets for each model type.13 We train models for a variable number of steps (a hyperparameter); to calculate automated coherence for the model, we use the topics produced at the last step. For human evaluations, we select the models that maximize NPMI, estimated using the reference corpus with a ten-word window over the top ten topic words, per Lau et al. (2014). We follow the recommendation of Dieng et al. (2020) and learn skip-gram embeddings on the training corpus for ETM (experiments with external pretrained embeddings did not yield substantially different results). As in Hoyle et al. (2020), we eliminate models with highly redundant topics, a known degeneracy of NTMs (Burkhardt and Kramer, 2019): (a) models in which any of the top five words of one topic overlap with another and (b) models that have a topic uniqueness score (Nan et al., 2019) above 0.7. Ranges for hyperparameters and other details are in Appendix A.3.

heading: 5 Human Evaluations of Topic Quality
text: We use the ratings and word intrusion tasks from Section 2.2 as human evaluations of topic quality. We recruit crowdworkers using Prolific.co, an online panel provider and collect data with the Qualtrics survey platform. We pay workers 2.5 USD per ratings survey and 3 USD per word intrusion survey, equivalent to 15 USD/hour.
In order to draw meaningful conclusions from human annotations, we require an adequate number of participants to ensure acceptable statistical power. However, Card et al. (2020) show that many NLP experiments, including those relying on human evaluation, are insufficiently powered to detect model differences at reported levels. Adopting a straightforward generative model of annotations (Appendix A.5), we select enough crowdworkers per task to ensure sufficient statistical power (at least 1−β = 0.9) to obtain significance at α = 0.05, resulting in a minimum of fifteen crowdworkers per topic for both tasks. On this criterion, both Chang et al. (2009) and thus Lau et al. (2014), with eight annotators, are underpowered.
For each of our two datasets, we generate fifty topics each from the three models in Section 4.2. In the word intrusion task, we sample five of the top ten topic words plus one intruder; for the ratings task, we present the top ten words in order (Figure 4). We separate the datasets for each task and
12github.com/adjidieng/ETM 13While runtimes can vary drastically by model, this study is not concerned with implementation efficiency (al-
though efficiency matters, see Ethayarajh and Jurafsky, 2020).
randomly sample 40 of the 150 topics. In the ratings task, we include an additional sixteen synthetic poor-quality topics to help calibrate scores and filter out low-quality respondents.14
Phrasing of questions closely follows the wording used by Chang et al. (2009), and crowdworkers received detailed instructions with examples (Appendix A.4) before responding to items.15 As topics can be esoteric (e.g., last columns of Table 1), we ask crowdworkers about their familiarity with the words in each question. We speculate that this question can help protect against spurious low scores for otherwise coherent topics, as real-world users of topic models are usually familiar with domain-specific terminology (see further discussion in Section 7).

heading: 6 Human Judgment Differs From Automated Metrics
text: We compare human judgments to automated methods on topics estimated using our three models.

heading: 6.1 Human Assessment
text: To establish model differences using human ratings, we use pairwise significance tests: a proportion test for the intrusion scores, a U test (Mann and Whitney, 1947) for the ratings, and a t-test for automated metrics (Figure 2), using one-tailed tests for each pair in both directions. Although D-VAE fares better on the intrusion task, evaluation using ratings favors G-LDA.16
Our human evaluation results are consistent with past iterations of the ratings and word intrusion tasks for topic models. Mimno et al. (2011) report an average of 2.36 on the ratings task on a dataset of medical paper abstracts.17 Our ratings means are 2.5 to 2.8 across all variations (Figure 2). Our word intrusion means range from 0.7 to 0.8, which is comparable to the roughly 0.8 accuracy on the LDA model evaluated in Chang et al. (2009). Median time taken on the tasks was 8–9 minutes.
14For generating synthetic poor-quality topics, we use random high-probability words appearing in topics from other hyperparameter settings, but that have low probability among selected topics. Eight topics each are generated from the vocabularies of NYT and WIKI.
15Code to convert topic model output into deployable questionnaires is at github.com/ahoho/topics. 16These discrepancies among human tasks support the argument that standard coherence metrics alone may
be insufficient for automated model selection (Doogan and Buntine, 2021). 17Newman et al. (2010) and Lau et al. (2014) do not report an average.
Following Aletras and Stevenson (2013), we calculate inter-annotator agreement with the mean Spearman correlation between each respondent’s score per topic and the average of other respondent scores, obtaining a value of 0.75 (compare to their value of 0.7 on the NYT corpus). Additionally, we include synthetic poor-quality topics (footnote 14)—correctly identified by annotators—and we monitor the duration taken for the survey to hedge against insincere submissions.

heading: 6.2 Automated Metrics
text: NPMI declares D-VAE the unequivocal victor among the three models (with G-LDA a clear second), a very different story from the human judgments. To understand the relationship between automated metrics and human ratings, we estimate the Spearman correlation between the two sets of values for each task and dataset for metric variants (Table 3). Although previous studies have used mean human ratings over topics, this decision obscures the inherent variance of the human ratings and leads to overconfident estimates. We therefore construct 95% confidence intervals by resampling ratings, with replacement, equal to the number of annotators per task (Table 3). We estimate NPMI with the standard 10-word window and Cv (Röder et al., 2015) with the recommended 110-word window.18 The Wikipedia corpus appears to be best correlated with human judgments, even for the models trained on the NYT corpus—this contradicts Lau et al. (2014), where within-domain data have the highest correlations.
While all correlation coefficients are statistically significant, the strength of the correlation alone does not justify their use in model selection, as is standard in the NTM literature (Section 3). In particular, the inherent uncertainty of human judgments means that it is difficult to determine when an increase in a model’s mean automated coherence implies a significant improvement in the corresponding human scores.19
As noted above (Figure 2), automated metrics exaggerate model differences compared to human judgments. To help clarify the utility of automated metrics for model selection, we ask how often an automated metric incorrectly asserts that one model is superior to another. To do so, we generate a bootstrapped estimate of the false discovery rate of each model. First, for each dataset, we randomly sample two independent sets of K = 50 topics (without replacement) from the original pool of 150, along with their corresponding automated and human scores (resampled with replacement, as in Table 3). Treating the two sampled sets as outputs from two different models, we compute pairwise significance tests between each set for both the K automated metrics and K × M human scores (using a proportions z-test for the intrusion scores and t-tests for all other values). After repeating this process for N = 1000 iterations, we report the proportion of significant differences detected using
18We use gensim (Řehůřek and Sojka, 2010) to calculate coherence. We process the reference corpora identically to the training data, retaining only terms that exist in the training vocabulary. Other metrics, like CUCI (Newman et al., 2010) and CUMASS (Mimno et al., 2011), show low correlations.
19Better models of human scores could help quantify this relationship (e.g., GLMs, see Appendix A. 10).
the predicted scores despite equivalent human scores (after correcting for the probability of type I errors, α = 0.05).20 Even the best-performing automated metrics predict significant differences absent a meaningful human effect roughly one-fifth of the time (Table 4).
These results suggest that automated metrics alone may be inadequate for model comparison.

heading: 6.3 Explaining the discrepancy
text: One reason for the discrepancy between human judgments and automated metrics is that metrics favor more esoteric topics. Specifically, there is a significant negative correlation between a topic’s NPMI or Cv and the share of respondents reporting familiarity with topic words (Pearson’s ρ = −0.29). And while D-VAE achieves the highest automated metric scores of the three models, it produces topics with the fewest familiar words: respondents report familiarity with terms over 90% of the time on both tasks for G-LDA and ETM, but they do so only 70% of the time for D-VAE. This difference suggests that the topics selected by D-VAE are narrower in scope than those of the other models. As shown in Figure 3, removing item annotations where respondents indicate unfamiliarity causes both accuracy in the word intrusion task and the ratio of “Very related” terms in the ratings task for D-VAE to increase substantially.
Qualitatively, this result is apparent when examining topics with a high NPMI but low humans ratings. In Table 5, the top rows consists of financial terms that frequently appear together in NYT articles,
20Details on testing equivalence are in Section A.5.1.
and the second row contains rare terms about boating—arguably both are reasonable topics for their respective corpora. We can also see instances where words are qualitatively very related (bottom half of table), but that NPMI fails to score high—perhaps because these words, while related, may not frequently appear together within a ten-word sliding window (Equation 1).
Even for familiar words, some topics may be sensible in the context of the specific corpus, despite their component words lacking an immediately obvious semantic relationship. For example, the topic words in the third and fourth rows appear somewhat unrelated (e.g., “taxes” and “bedroom” in the third row), but they are in fact characteristic of common document types in the New York Times: real estate listings and obituaries. Topics like these render the word intrusion task more difficult: only 23% of crowdworkers identified the intruder for both topics.
Furthermore, using term familiarity as a proxy for domain expertise does not address the key problems with topic model evaluation: even after filtering out respondents who are not familiar with topic terms, automated metrics still overstate model differences (Appendix A.7). The problems with topic model evaluation may therefore extend to our choice of human evaluations as well.

heading: 7 So. . . is Automated Topic Modeling Evaluation Broken?
text: To the extent that our experimentation accurately represents current practice, our results do suggest that topic model evaluation—both automated and human—is overdue for a careful reconsideration. In this, we agree with Doogan and Buntine (2021), who write that “coherence measures designed for older models [. . . ] may be incompatible with newer models” and instead argue for evaluation paradigms centered on corpus exploration and labeling. The right starting point for this reassessment is the recognition that both automated and human evaluations are abstractions of a real-world problem. The familiar use of precision-at-10 in information retrieval, for example, corresponds to a user who is only willing to consider the top ten retrieved documents. In future work, we intend to explore automated metrics that better approximate the preferences of real-world topic model users.
One primary use of topic models is in computer-assisted content analysis. In that context, rather than taking a methods-driven approach to evaluation, it would make sense to take a needs-driven approach.21 Generic evaluation of topic models using domain-general corpora like NYT needs to be revisited, since there is no such thing as a “generic” corpus for content analysis, nor a generic analyst. Content analysis can be formulated in a broad way, as Krippendorff (2004) has shown, but its actual application is always in a domain, by people familiar with that domain. This fact stands in tension with the desirable practicalities of general corpora and crowdworker annotation, and the field will need to address this tension. We have identified “coherence” as calling out a latent concept in the mind of a reader. It follows that we must think about who the relevant human readers are and the conceptual spaces that matter to them.
21These needs also have a computational component: neural models usually have longer runtimes even when accelerated with GPUs, whereas many practitioners work in local, CPU-only, environments. See Appendix A.3 for additional details on runtimes.

heading: Acknowledgements
text: This material is based upon work supported by the National Science Foundation under Grants 2031736, 2008761, 1822494, ARLIS, and by an Amazon Research Award. We thank Sweta Agrawal for her suggestion to conduct a meta-analysis. We owe much appreciation to Dallas Card for his keen advice on power analyses. Thanks to Frank Fineis for help on several statistical questions, as well as Shuo Chen for his suggestions regarding the false discovery rate calculations. Finally, we thank Caitie Doogan for her helpful comments on the clarity of argumentation, as well as our anonymous reviewers.

Review summaries:
Topic model evaluation in terms of the quality of learned topics is an important problem in topic modelling, the validation of which has a profound impact on the model development and its application in cross disciplines. This paper reassesses the automatic topic model evaluation metrics, e.g., topic coherence scores like NPMI, that are commonly adopted in the topic modelling literature. It points out the potential flaws of applying those measures to neural topic models developed recently, via conducting a set of human evaluations. The paper identifies two gaps existing in the current evaluation paradigm used in almost all NMTs, the validation gap and the standardisation gap through analysing a large number of NTMs. Its findings could have wide implications in the practice.

This paper investigates automated evaluation metrics of topic models in terms of coherence such as NPMI. NPMI is widely used to evaluate the performance of topic models, but the authors tackle that it is less related to human evaluation metrics such as rating and intrusion. For experiments, the authors train three topic models, LDA, D-VAE, and ETM with two different corpora, Wikipedia and new york times. To evaluate the topics by humans, the authors ask at least fifteen crowdworkers per topic to avoid insufficient power of human evaluation problem. Experiments show that D-VAE outperforms other models in terms of NPMI, but there are no significant differences among models by human evaluations.   The main strength of this paper is claiming the usefulness of automated evaluations of topic models with deep considerations that follows recent concerns of NLP research evaluations. I really enjoy reading this paper since it can guide the way to compare the two or more different evaluation metrics in various domains including topic modeling. 

This paper questions the use of automatic metric, such as the ones based on NPMI, for evaluating the quality of topic models. It sets up a precise, reproducible framework (with code) that compare classic approaches (LDA) and recent ones based on neural networks (D-VAE). From my point of view, it is more a positioning paper that ca n help the community to change its habits than a novel solution to a given problem. However I think we need this type of papers which can lead to designing better models.

The paper addresses standardization gap in automatic metrics used for evaluation of topic modeling as well as a validation gap for the coherence of those automatic metrics with the human judgement, both revisiting the models for which such validation had been done (LDA) as well as the recent Neural Top. Models which have been purely evaluated using the automatic metrics.  The authors provide a deep investigation on the automatic metrics currently in widespread use within Topic Modeling as well as a historic overview on the last studies of the coherence of the automatic metrics (NPMI) and human judgements.  One of the main contributions of the paper is the rigorous study and comparison of the human judgements and the automatic metrics NPMI and C_v ensuring the statistical power of the test (involving large-scale evaluation by crowdsourced judges) making this work a steady reference point for future works.