Is Automated Topic Model Evaluation Broken? The Incoherence of Coherence

1 Revisiting Topic Model Evaluation . Topic models are a machine learning technique widely used outside computer science , including political science ( Grimmer and Stewart , 2013 ; Isoaho et al. , 2021 ) , social and cultural studies ( Mohr and Bogdanov , 2013 ) , digital humanities ( Meeks and Weingart , 2012 ) , and bioinformatics ( Liu et al. , 2016 ) . Typically , topic model users are domain experts trying to identify global categories or themes present in a document collection ( Boyd-Graber et al. , 2017 ) . This practice constitutes a computer-assisted form of content analysis ( Krippendorff , 2004 ; Chuang et al. , 2014 ) , also related to distant reading in literary studies ( Underwood , 2017 ) . In general , topic models help humans understand large corpora.2 Evaluation of topic models has vacillated between automated and human-centered . While real-world users of topic models evaluate outputs based on their specific needs , topic model developers have gravitated toward generalized , automated proxies of human judgment to help inform rapid iteration of models ( Doogan and Buntine , 2021 ) . Initially , models were evaluated with held-out perplexity , but it disagrees with human interpretability ( Chang et al. , 2009 ) . Consequently , the field adopted automated coherence metrics like normalized pointwise mutual information ( NPMI ) , a measure of word relatedness that does correlate with topic interpretability ( Section 2.2 ; Newman et al. , 2010 ; Aletras and Stevenson , 2013 ; Lau et al. , 2014 ) . The balance shifted towards automated coherence . Human evaluations have been abandoned by topic model developers in the years since automated coherence metrics were adopted . In a thorough meta-analysis of contemporary topic model methods ∗Equal contribution 2Topic models are also used for other purposes , such as information retrieval or downstream document classification . However , the discovery and application of categories for human interpretation is their dominant use , and other computational applications have been largely eclipsed by modern neural approaches . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . papers , none conduct systematic human evaluations ( Section 3 ) . Instead , they rely solely on automated metrics for model comparison.3 However , current neural topic models are a far cry from the classical models that substantiated the original correlations—manifestly , topics produced by neural models are often qualitatively distinct from those of classical models ( e.g. , Table 1 ) .4 This validation gap raises the question of whether automated metrics are still consistent with human judgments of topic quality . Moreover , we should always be cautious when extrapolating outside the range of data that was used to establish a relationship between variables . As an example , a neural model in Hoyle et al . ( 2020 ) produces much larger NPMI values than those used to determine human correlations in the original Lau et al . ( 2014 ) study ; the implicit assumption is that greater NPMI corresponds to more humaninterpretable topics . Finally , a myopic focus on a presumed proxy for human preferences can produce low-quality results ( Stiennon et al. , 2020 ) . Does Goodharts ’ law— “ when a measure becomes a target , it ceases to be a good measure ” ( Strathern , 1997 ) —apply to automated metrics of topic models ? Another challenge for automated evaluation , whether of classical or neural topic models , is widespread inconsistency ( Section 3 ) . Researchers frequently fail to specify the information needed to calculate automated metrics or diverge from the practices that underpin human correlations . Furthermore , evaluation datasets , preprocessing , and hyperparameter optimization vary dramatically , even within a given paper . This standardization gap likely limits the generalizability and reliability of topic model developers ’ findings . We address the standardization and validation gaps in topic model evaluation : 1 . We present a meta-analysis of neural topic model evaluation ( Section 3 ) ; 2. we develop standardized , pre-processed versions of two widely-used English-language evaluation datasets , along with a transparent end-to-end code pipeline for reproduction of results ( Section 4.1 ) 5 ; 3. we optimize three topic models—one classical and two neural—using identical preprocessing , model selection criteria , and hyperparameter tuning ( Section 4.2 ) ; 4. we evaluate these models using human ratings and word intrusion tasks ( Section 5 ) ; and 5. we provide new evaluations of the correlation between automated and human evaluations ( Section 6 ) . Our findings challenge the validity of fully-automated evaluations as currently practiced : automated evaluation declares winners between models when the corresponding human evaluations can not . 3Outside of the core method-development literature , human evaluations have been used to develop new metrics and improve understanding of existing model behavior ( Bhatia et al. , 2017 ; Morstatter and Liu , 2018 ; Lund et al. , 2019 ; Alokaili et al. , 2019 , inter alia ) . 4We use “ classical ” to mean generative models defined by a chain of conjugate exponential family distributions optimized by Gibbs sampling or variational inference . 5github.com/ahoho/topics 2 Operationalizing Topic Coherence . A topic model is a probabilistic generative model of text that uses latent topics to summarize a larger collection of documents . The most influential variant , latent Dirichlet allocation ( Blei et al. , 2003 , LDA ) , assumes that K latent topics are distributions over word types , βk , and that the documents D are admixtures over the topics , θd . Users often evaluate model outputs globally , focusing on the most probable N words of each topic , and locally , considering the most probable topics for each document . While techniques for topic modeling have progressed from variational inference ( Blei et al. , 2003 ) to Gibbs sampling ( Griffiths and Steyvers , 2004 ) to deep generative approaches ( Srivastava and Sutton , 2017 ; Wang et al. , 2020b ) , the core goal discussed in Section 1 , obtaining human-understandable categories , remains central . The latest wave of methods , neural topic models ( NTM ) , use continuous word representations and gradient optimization to fit parameters . These models claim to produce more interpretable topics than other prior methods , including LDA . Those claims are supported by improvements on automated measures of topic coherence . 2.1 Human Metrics of Topic Coherence . Like the concept of interpretability , that of real-world coherence is “ simultaneously important and slippery ” ( Lipton , 2018 ) . We will not attempt to formalize it here—though see discussion in Section 7 . For present purposes , the term has its roots in Latin cohaerere , “ to stick together , ” and we will think of coherence as an intangible sense , available to human readers , that a set of terms , when viewed together , enable human recognition of an identifiable category.6 We review two human ratings of topic quality : direct ratings and intrusion . Rating Raters see a topic and then give the topic a quality score , conventionally on a three-point ordinal scale ( Newman et al. , 2010 ; Mimno et al. , 2011 ; Aletras and Stevenson , 2013 , inter alia ) . Intrusion Chang et al . ( 2009 ) devise the word intrusion task as a behavioral way to assess topic coherence . The core idea is that when the top words in a topic identify a coherent latent category , it is easier to identify words that do not belong to that category . Operationally , each topic is represented as its top words plus one “ intruder ” word which has a low probability of belonging to that topic , but a high probability of belonging to a different topic . Topic coherence is then judged by how well human annotators detect the “ intruder ” word . 2.2 NPMI : The Standard Automated Topic Model Coherence Evaluation . Using the word intrusion task , Chang et al . ( 2009 ) showed that perplexity—the original topic model evaluation metric—negatively correlates with human evaluations of topic quality . This finding revealed a need for an automated measurement of topic coherence : an automated metric can measure model quality without expensive , time-consuming , and difficult-to-reproduce human experiments . Lau et al . ( 2014 ) find some metrics that positively correlate with human intrusion and rating scores , particularly when aggregating scores over all topics from a given model . Because of that validation , the prevailing evaluation for model comparison is pairwise normalized pointwise mutual information . NPMI scores topics highly if the top N words—summed over all pairs wi and wj—have high joint probability P ( wj , wi ) compared to their marginal probability:7 N∑︂ j=2 j−1∑︂ i=1 log P ( wj , wi ) P ( wi ) P ( wj ) −logP ( wi , wj ) . ( 1 ) The probabilities are estimated using word co-occurrence counts from a reference corpus for a specific context window ( which can range from ten words to the entire document ) . As a result , the choice of reference corpus determines the strength of human correlation ( Lau et al. , 2014 ; Röder et al. , 2015 ) . 6This perspective aligns with Propositions 2 and 3 of Doogan and Buntine ( 2021 ) : “ an interpretable topic is one that can be easily labeled , ” and “ has high agreement on labels. ” 7Alternative metrics exist , but they typically also rely on either joint probability estimates or NPMI directly ( e.g. , Cv Röder et al. , 2015 ) . A measurement is valid to the extent that it measures what it is intended to measure in the real world . Historically , automated coherence has been validated using human judgements from either crowdworkers ( Newman et al. , 2010 ; Aletras and Stevenson , 2013 ) or experts ( Mimno et al. , 2011 ) . However , correlations based on classical models may not be applicable for NTMs . Our skepticism is motivated by theory , as neural word representations are intimately connected to NPMI , as explicitly used by Aletras and Stevenson ( 2013 ) and which produce similar NPMI scores as Lau et al . ( 2014 ) . Levy and Goldberg ( 2014 ) show that multiple representations create factorizations of PMI matrices . Topic models that have access to these rich representations ( e.g . Dieng et al. , 2020 , and others ) could thus create topics with good NPMI scores without explaining the corpus well to a user . In contrast to classical topic models , no one has investigated the validity of NPMI evaluation for NTMs . Given this lacuna , we conduct experiments aimed at validating that automated topic evaluations still correlate with human judgments of neural topic model quality . We compare against two common human evaluations of individual topic quality : direct rating and intrusion . Human evaluations , like automated topic modeling , lack standardization , which we address in Section 5 . 3 A Meta-Analysis of Neural Topic Modeling . We survey the neural topic modeling ( NTM ) literature to assess the state of evaluation in contemporary topic model development . First , we take all references made by an existing , comprehensive survey of NTMs ( Zhao et al. , 2021b ) , from which we select ( a ) modeling papers which ( b ) mention topic interpretability and ( c ) compare models ’ topics with an existing baseline . This yields forty models , which all claim superior topic coherence . We examine data processing steps , hyperparameter tuning , baseline selection , and automated coherence calculations . Table 2 summarizes our results and Appendix A.1 enumerates the papers . Our analysis reveals variance in all areas . Preprocessing , which can significantly affect model quality and automated metrics , is often ( 30 % ) inconsistent across datasets within the same paper . When preprocessing is consistent , authors omit details necessary to fully replicate the pipeline . These issues imply that automated metrics for the same baselines and source datasets vary across papers . Compounding the problem , researchers often train their models on different datasets from those used to establish the relationships between human annotations and automated metrics ; Doogan and Buntine ( 2021 ) find that the same metrics may not predict interpretability in new domains . Mirroring findings from Dodge et al . ( 2019 ) , 40 % of papers fail to clearly specify their model tuning procedure , often even the metric used for model selection . Calculation of automated coherence metrics is equally fraught . As discussed in Section 2.2 , a complete specification for NPMI involves several pieces of information , including the reference corpus used to estimate joint word probabilities , the co-occurrence window size , and the number of words selected from the head of the topic distribution . Three out of four papers fail to explicitly indicate the reference corpus ; even when we can assume the input corpus is used ( 13 cases ) , it remains uncertain whether authors use , e.g. , a held-out set or the training documents themselves . For the 61 % that specify the implementation of their coherence metric ( by pointing to a code repository or writing out the formula ) , some of these factors may still be in question . For instance , six authors reference Lau et al . ( 2014 ) and the supporting code,8 but the implications are ambiguous : the original paper suggests a large corpus from the same source as the training data , but the repository script defaults to Wikipedia . In other cases , authors use bespoke implementations , which creates room for errors , or deviate from the settings used in human experiments . For example , several papers use a document-wide context window with NPMI , which has not been correlated with human judgments . Last , even if automated evaluations are consistent , all claims of coherence improvement depend on the validity results in Lau et al . ( 2014 ) generalizing to neural topic models .

Summary: The paper addresses standardization gap in automatic metrics used for evaluation of topic modeling as well as a validation gap for the coherence of those automatic metrics with the human judgement, both revisiting the models for which such validation had been done (LDA) as well as the recent Neural Top. Models which have been purely evaluated using the automatic metrics.  The authors provide a deep investigation on the automatic metrics currently in widespread use within Topic Modeling as well as a historic overview on the last studies of the coherence of the automatic metrics (NPMI) and human judgements.  One of the main contributions of the paper is the rigorous study and comparison of the human judgements and the automatic metrics NPMI and C_v ensuring the statistical power of the test (involving large-scale evaluation by crowdsourced judges) making this work a steady reference point for future works.