Is Automated Topic Model Evaluation Broken? The Incoherence of Coherence

1 Revisiting Topic Model Evaluation . Topic models are a machine learning technique widely used outside computer science , including political science ( Grimmer and Stewart , 2013 ; Isoaho et al. , 2021 ) , social and cultural studies ( Mohr and Bogdanov , 2013 ) , digital humanities ( Meeks and Weingart , 2012 ) , and bioinformatics ( Liu et al. , 2016 ) . Typically , topic model users are domain experts trying to identify global categories or themes present in a document collection ( Boyd-Graber et al. , 2017 ) . This practice constitutes a computer-assisted form of content analysis ( Krippendorff , 2004 ; Chuang et al. , 2014 ) , also related to distant reading in literary studies ( Underwood , 2017 ) . In general , topic models help humans understand large corpora.2 Evaluation of topic models has vacillated between automated and human-centered . While real-world users of topic models evaluate outputs based on their specific needs , topic model developers have gravitated toward generalized , automated proxies of human judgment to help inform rapid iteration of models ( Doogan and Buntine , 2021 ) . Initially , models were evaluated with held-out perplexity , but it disagrees with human interpretability ( Chang et al. , 2009 ) . Consequently , the field adopted automated coherence metrics like normalized pointwise mutual information ( NPMI ) , a measure of word relatedness that does correlate with topic interpretability ( Section 2.2 ; Newman et al. , 2010 ; Aletras and Stevenson , 2013 ; Lau et al. , 2014 ) . The balance shifted towards automated coherence . Human evaluations have been abandoned by topic model developers in the years since automated coherence metrics were adopted . In a thorough meta-analysis of contemporary topic model methods ∗Equal contribution 2Topic models are also used for other purposes , such as information retrieval or downstream document classification . However , the discovery and application of categories for human interpretation is their dominant use , and other computational applications have been largely eclipsed by modern neural approaches . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . papers , none conduct systematic human evaluations ( Section 3 ) . Instead , they rely solely on automated metrics for model comparison.3 However , current neural topic models are a far cry from the classical models that substantiated the original correlations—manifestly , topics produced by neural models are often qualitatively distinct from those of classical models ( e.g. , Table 1 ) .4 This validation gap raises the question of whether automated metrics are still consistent with human judgments of topic quality . Moreover , we should always be cautious when extrapolating outside the range of data that was used to establish a relationship between variables . As an example , a neural model in Hoyle et al . ( 2020 ) produces much larger NPMI values than those used to determine human correlations in the original Lau et al . ( 2014 ) study ; the implicit assumption is that greater NPMI corresponds to more humaninterpretable topics . Finally , a myopic focus on a presumed proxy for human preferences can produce low-quality results ( Stiennon et al. , 2020 ) . Does Goodharts ’ law— “ when a measure becomes a target , it ceases to be a good measure ” ( Strathern , 1997 ) —apply to automated metrics of topic models ? Another challenge for automated evaluation , whether of classical or neural topic models , is widespread inconsistency ( Section 3 ) . Researchers frequently fail to specify the information needed to calculate automated metrics or diverge from the practices that underpin human correlations . Furthermore , evaluation datasets , preprocessing , and hyperparameter optimization vary dramatically , even within a given paper . This standardization gap likely limits the generalizability and reliability of topic model developers ’ findings . We address the standardization and validation gaps in topic model evaluation : 1 . We present a meta-analysis of neural topic model evaluation ( Section 3 ) ; 2. we develop standardized , pre-processed versions of two widely-used English-language evaluation datasets , along with a transparent end-to-end code pipeline for reproduction of results ( Section 4.1 ) 5 ; 3. we optimize three topic models—one classical and two neural—using identical preprocessing , model selection criteria , and hyperparameter tuning ( Section 4.2 ) ; 4. we evaluate these models using human ratings and word intrusion tasks ( Section 5 ) ; and 5. we provide new evaluations of the correlation between automated and human evaluations ( Section 6 ) . Our findings challenge the validity of fully-automated evaluations as currently practiced : automated evaluation declares winners between models when the corresponding human evaluations can not . 3Outside of the core method-development literature , human evaluations have been used to develop new metrics and improve understanding of existing model behavior ( Bhatia et al. , 2017 ; Morstatter and Liu , 2018 ; Lund et al. , 2019 ; Alokaili et al. , 2019 , inter alia ) . 4We use “ classical ” to mean generative models defined by a chain of conjugate exponential family distributions optimized by Gibbs sampling or variational inference . 5github.com/ahoho/topics 2 Operationalizing Topic Coherence . A topic model is a probabilistic generative model of text that uses latent topics to summarize a larger collection of documents . The most influential variant , latent Dirichlet allocation ( Blei et al. , 2003 , LDA ) , assumes that K latent topics are distributions over word types , βk , and that the documents D are admixtures over the topics , θd . Users often evaluate model outputs globally , focusing on the most probable N words of each topic , and locally , considering the most probable topics for each document . While techniques for topic modeling have progressed from variational inference ( Blei et al. , 2003 ) to Gibbs sampling ( Griffiths and Steyvers , 2004 ) to deep generative approaches ( Srivastava and Sutton , 2017 ; Wang et al. , 2020b ) , the core goal discussed in Section 1 , obtaining human-understandable categories , remains central . The latest wave of methods , neural topic models ( NTM ) , use continuous word representations and gradient optimization to fit parameters . These models claim to produce more interpretable topics than other prior methods , including LDA . Those claims are supported by improvements on automated measures of topic coherence . 2.1 Human Metrics of Topic Coherence . Like the concept of interpretability , that of real-world coherence is “ simultaneously important and slippery ” ( Lipton , 2018 ) . We will not attempt to formalize it here—though see discussion in Section 7 . For present purposes , the term has its roots in Latin cohaerere , “ to stick together , ” and we will think of coherence as an intangible sense , available to human readers , that a set of terms , when viewed together , enable human recognition of an identifiable category.6 We review two human ratings of topic quality : direct ratings and intrusion . Rating Raters see a topic and then give the topic a quality score , conventionally on a three-point ordinal scale ( Newman et al. , 2010 ; Mimno et al. , 2011 ; Aletras and Stevenson , 2013 , inter alia ) . Intrusion Chang et al . ( 2009 ) devise the word intrusion task as a behavioral way to assess topic coherence . The core idea is that when the top words in a topic identify a coherent latent category , it is easier to identify words that do not belong to that category . Operationally , each topic is represented as its top words plus one “ intruder ” word which has a low probability of belonging to that topic , but a high probability of belonging to a different topic . Topic coherence is then judged by how well human annotators detect the “ intruder ” word . 2.2 NPMI : The Standard Automated Topic Model Coherence Evaluation . Using the word intrusion task , Chang et al . ( 2009 ) showed that perplexity—the original topic model evaluation metric—negatively correlates with human evaluations of topic quality . This finding revealed a need for an automated measurement of topic coherence : an automated metric can measure model quality without expensive , time-consuming , and difficult-to-reproduce human experiments . Lau et al . ( 2014 ) find some metrics that positively correlate with human intrusion and rating scores , particularly when aggregating scores over all topics from a given model . Because of that validation , the prevailing evaluation for model comparison is pairwise normalized pointwise mutual information . NPMI scores topics highly if the top N words—summed over all pairs wi and wj—have high joint probability P ( wj , wi ) compared to their marginal probability:7 N∑︂ j=2 j−1∑︂ i=1 log P ( wj , wi ) P ( wi ) P ( wj ) −logP ( wi , wj ) . ( 1 ) The probabilities are estimated using word co-occurrence counts from a reference corpus for a specific context window ( which can range from ten words to the entire document ) . As a result , the choice of reference corpus determines the strength of human correlation ( Lau et al. , 2014 ; Röder et al. , 2015 ) . 6This perspective aligns with Propositions 2 and 3 of Doogan and Buntine ( 2021 ) : “ an interpretable topic is one that can be easily labeled , ” and “ has high agreement on labels. ” 7Alternative metrics exist , but they typically also rely on either joint probability estimates or NPMI directly ( e.g. , Cv Röder et al. , 2015 ) . A measurement is valid to the extent that it measures what it is intended to measure in the real world . Historically , automated coherence has been validated using human judgements from either crowdworkers ( Newman et al. , 2010 ; Aletras and Stevenson , 2013 ) or experts ( Mimno et al. , 2011 ) . However , correlations based on classical models may not be applicable for NTMs . Our skepticism is motivated by theory , as neural word representations are intimately connected to NPMI , as explicitly used by Aletras and Stevenson ( 2013 ) and which produce similar NPMI scores as Lau et al . ( 2014 ) . Levy and Goldberg ( 2014 ) show that multiple representations create factorizations of PMI matrices . Topic models that have access to these rich representations ( e.g . Dieng et al. , 2020 , and others ) could thus create topics with good NPMI scores without explaining the corpus well to a user . In contrast to classical topic models , no one has investigated the validity of NPMI evaluation for NTMs . Given this lacuna , we conduct experiments aimed at validating that automated topic evaluations still correlate with human judgments of neural topic model quality . We compare against two common human evaluations of individual topic quality : direct rating and intrusion . Human evaluations , like automated topic modeling , lack standardization , which we address in Section 5 . 3 A Meta-Analysis of Neural Topic Modeling . We survey the neural topic modeling ( NTM ) literature to assess the state of evaluation in contemporary topic model development . First , we take all references made by an existing , comprehensive survey of NTMs ( Zhao et al. , 2021b ) , from which we select ( a ) modeling papers which ( b ) mention topic interpretability and ( c ) compare models ’ topics with an existing baseline . This yields forty models , which all claim superior topic coherence . We examine data processing steps , hyperparameter tuning , baseline selection , and automated coherence calculations . Table 2 summarizes our results and Appendix A.1 enumerates the papers . Our analysis reveals variance in all areas . Preprocessing , which can significantly affect model quality and automated metrics , is often ( 30 % ) inconsistent across datasets within the same paper . When preprocessing is consistent , authors omit details necessary to fully replicate the pipeline . These issues imply that automated metrics for the same baselines and source datasets vary across papers . Compounding the problem , researchers often train their models on different datasets from those used to establish the relationships between human annotations and automated metrics ; Doogan and Buntine ( 2021 ) find that the same metrics may not predict interpretability in new domains . Mirroring findings from Dodge et al . ( 2019 ) , 40 % of papers fail to clearly specify their model tuning procedure , often even the metric used for model selection . Calculation of automated coherence metrics is equally fraught . As discussed in Section 2.2 , a complete specification for NPMI involves several pieces of information , including the reference corpus used to estimate joint word probabilities , the co-occurrence window size , and the number of words selected from the head of the topic distribution . Three out of four papers fail to explicitly indicate the reference corpus ; even when we can assume the input corpus is used ( 13 cases ) , it remains uncertain whether authors use , e.g. , a held-out set or the training documents themselves . For the 61 % that specify the implementation of their coherence metric ( by pointing to a code repository or writing out the formula ) , some of these factors may still be in question . For instance , six authors reference Lau et al . ( 2014 ) and the supporting code,8 but the implications are ambiguous : the original paper suggests a large corpus from the same source as the training data , but the repository script defaults to Wikipedia . In other cases , authors use bespoke implementations , which creates room for errors , or deviate from the settings used in human experiments . For example , several papers use a document-wide context window with NPMI , which has not been correlated with human judgments . Last , even if automated evaluations are consistent , all claims of coherence improvement depend on the validity results in Lau et al . ( 2014 ) generalizing to neural topic models .

Summary: This paper questions the use of automatic metric, such as the ones based on NPMI, for evaluating the quality of topic models. It sets up a precise, reproducible framework (with code) that compare classic approaches (LDA) and recent ones based on neural networks (D-VAE). From my point of view, it is more a positioning paper that ca n help the community to change its habits than a novel solution to a given problem. However I think we need this type of papers which can lead to designing better models.