Is Automated Topic Model Evaluation Broken? The Incoherence of Coherence
Authors: Alexander Hoyle
Pranav Goel
Andrew Hian-Cheong
Denis Peskov
Jordan Lee Boyd-Graber
Philip Resnik
Venue: NeurIPS 2021 Spotlight (2021)
Keywords: topic model
topic model evaluation
npmi
topic coherence
human evaluation
metric
evaluation
validation
crowdsourcing
model comparison
automatic metric
automated metric

Abstract: Topic model evaluation, like evaluation of other unsupervised methods, can be contentious. However, the field has coalesced around automated estimates of topic coherence, which rely on the frequency of word co-occurrences in a reference corpus. Contemporary neural topic models surpass classical ones according to these metrics. At the same time, topic model evaluation suffers from a validation gap: automated coherence, developed for classical models, has not been validated using human experimentation for neural models. In addition, a meta-analysis of topic modeling literature reveals a substantial standardization gap in automated topic modeling benchmarks. To address the validation gap, we compare automated coherence with the two most widely accepted human judgment tasks: topic rating and word intrusion. To address the standardization gap, we systematically evaluate a dominant classical model and two state-of-the-art neural models on two commonly used datasets. Automated evaluations declare a winning model when corresponding human evaluations do not, calling into question the validity of fully automatic evaluations independent of human judgments.

Is Automated Topic Model Evaluation Broken?: The Incoherence of Coherence
Alexander Hoyle*
Pranav Goel
Denis Peskov*
Andrew Hian-Cheong
Computer Science
Jordan Boyd-Graber CS, iSchool, UMIACS, LSC
Philip Resnik
UMIACS, Lingusities
University of Maryland
(hoyle, pgoel 11 11, dpeskov, andrewhc, jbg, resnik @cs. und. edu
Abstract
Topic model evaluation, like evaluation of other unsupervised methods, can be contentious. However, field coalesced around automated estimates topic coherence, which rely on the frequency of word co-nceurrences in reference corpus. Contemporary neural topic models surpass classical ones according to these metries. At the same time, topic model evaluation suffers from validation gap: automated coherence, developed for classical models, has not been validated using human experimentation for neural models. In addition, meta-analysis of topic modeling literature reveals substantial standardization gap au- tomated topic modeling benchmarks. To address the validation gap, we compare automated coherence with the two most widely accepted human judgment tasks: topic rating and word intrusion. To address the standardization gap, systematically evaluate dominant classi- model and two state-of the-ar art neural models evaluations declare a winning model when corresponding human evaluations do not, calling into question the validity of fully automatic evaluations independent of human judgments.
commonly used datascts. Automated
Revisiting Topic Model Evaluation
Topic models are machine learning technique widely used outside computer science, including political science (Grimmer and Stewart 2013] Isoaho et al. | [202I). social and cultural studies (Mohr and Bogdanov 2013), digital humanities (Meeks and Weingart 2012). and bioinformatics (Liu et al. 2016). Typically, topic model users are domain experts trying to identify global categories themes present in document collection (Boyd-Graber et al. 2017). This practice constitutes
a computer-assisted form of content analysis (Krippendorfi) 2004 Chuang et al. (2014), also related
distant reading in literary studies (Underwood) 2017). In general, topic models help humans
understand large corpora 3
Evaluation of topic models has vacillated between automated and human-centered. While real-world users of topic models evaluate outputs based on their specific needs, topic model developers have gravitated toward generalized, automated proxies of human judgment to help inform rapid iteration of models (Doogan and Buntine 202T). Initially, models were evaluated with held-out perplexity, but disagrees with human interpretability (Chang et al., 2009). Consequently. the field adopted automated coherence metrics like normalized pointwise mutual information (NPMT), a measure of word relatedness that does correlate with topic interpretability (Section 2.2 Newman al.) 2010 Aletras and Stevenson 2013 Lau et al. 2014). The balance shifted towards automated coherence.
Human evaluations have been abandoned by topic model developers in the years since automated coherence metrics were adopted. thorough meta-analysis of contemporary topic model methods
Equal contribution
'Topic models are also used for other purposes, such as information retrieval or downstream document classification. However, the discovery and application of categories for human interpretation is their dominant use, and other computational applications have been largely eclipsed by modern neural approaches.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Classical
Neural
station
album tropical tropical
spore
manhattan project
line
band storm landfall eyclone
basidia
los alamos laboratory
bridge
music song relcased
hurricane cyclone
spores
robert oppenheimer enrico enrico_ferma fermi physicist
railway
ute
mycologist
trains
depression
weakening
hyphac
NPMI 0.274
0.285
0.394
0.446
0.456
0.470
Table I: The first three columns are the highest-NPMI topics for a classical topic model (LDA estimated via Gibbs sampling using Mallet, [McCallum] 2002 Griffiths and Steyvers (004). The next three are counterparts from a neural model (our D VAE reimplementation, Burkhardt Burg and Kramer 2019). Models are trained on Wikitext (Merity al. 2017 with fifty topies, and NPMI is estimated over the top five words in each topic using a 4.6M-document reference Wikipedia corpus. The mean top-five NPMI over all topies is 0.156 for the classical and 0.256 for the neural model.
papers, none conduct systematic human evaluations (Section 3). Instead, they rely solely on automated metrics for model comparison However, current neural topic models are far cry from the classical models that substantiated the original correlations mannoesssy.....y....... topics produced by neural models are often qualitatively distinct from those classical models (e.g.. Table I This validation gap raises the question of whether automated metries are still consistent with human judgments of topic quality.
Moreover, should always be cautious when extrapolating outside the range of data that was used to establish a relationship between variables. As an example, neural model in Hoyle et al. (2020) produces much larger NPMI values than those used determine human correlations in the origi- nal Lau et al. (2014) study; the implicit assumption is that greater NPMI corresponds to more human- interpretable topics. Finally, myopic focus on presumed proxy for human preferences can produce low-quality results (Stiennon et al. (020)) Does Goodharts' law-"when a measure becomes target, ceases be a good measure" (Strathern) 1997)-apply to automated metrics of topic models?
Another challenge for automated evaluation, whether of classical or neural topic models, is widespread inconsistency (Section 3). Researchers frequently fail to specify the information needed to calculate automated metrics or diverge from the practices that underpin human correlations. Furthermore, evaluation datasets, preprocessing, and hyperparameter optimization vary dramatically, even within given paper. This standardization gap likely limits the generalizability and reliability of topic model developers' findings.
We address the standardization and validation gaps in topic model evaluation:
1. We present a meta-analysis of neural topic model evaluation (Section
2. we develop standardized, pre-processed versions of two widely-used languuee evaluation datascts, along with a transparent end-to-end code pipeline for reproduction of results (Section 9
3. we optimize three topic models classical and two ing, model selection criteria, and hyperparameter tuning (Section 4.2)
identical preprocess-
4. we evaluate these models using human ratings and word intrusion tasks (Section and
5. we provide new evaluations of the correlation between automated and human evaluations ()..
Our findings challenge the validity of fully-automated evaluations as currently practiced: automated evaluation declares winners between models when the corresponding human evaluations cannot.
*Outside of the core method development literature, human evaluations have been used develop new metrics and improve understanding of existing model behavior (Bhatia et al. 2017 Morstatter and Liu 2018
Lund et al. 2019 Alokaili et al, 2019 inter alia).
use "classical" to mean generative models defined by chain of conjugate exponential family distribu- tions optimized by Gibbs sampling or variational inference.

---

2 Operationalizing Topic Coherence
A topic model is a probabilistic generative model of text that uses latent topics to summarize larger collection of documents. The most influential variant, latent Dirichlet allocation (Blei al. 2003 LDA), assumes that K latent topics are distributions over word types, Bg and that the documents D are admixtures over the topics, Og. Users often evaluate model outputs globally, focusing on the most probable N words of each topic, and locally, considering the most probable topics for each document.
While techniques for topic modeling have progressed from variational inference (Blei et al. 2003) to Gibbs sampling (Griffiths and Steyvers, 2004) to deep generative approaches (Srivastava and Sutton 2017 Wang al. (020b), the core goal discussed in Section [ obtaining human- buman-underssandabbe categories, remains central. The latest wave of methods, neural topic models (NTM), use continuous word representations and gradient optimization to fit parameters. These models claim produce more interpretable topics than other prior methods, including LDA.
Those claims are supported by improvements automated measures of topic coherence.
2.1 Human Metrics of Topic Coherence
Like the concept of interpretability, that of real-world coherence is "simultaneously important and slippery" (Lipton 2018). We will not attempt formalize it here-thougle though discussion in Section 7 For present purposes, the term has its roots in Latin cohaerere, "to stick together," and we will think of coherence together, enable human recognition of an identifiable category We review two human ratings of topic quality: direct ratings and intrusion.
intangible sense, available human readers, that set of terms, when viewed
Rating Raters see a topic and then give the topic a quality score, conventionally on three-point ordinal scale (Newman et al. 2010; Mimno al., 201 I Aletras and Stevenson 2013 inter alia).
Intrusion Chang et al. 2009) devise the word infrusion task as a behavioral way to assess topic coherence. The core idea is that when the top words in a topic identify a coherent latent category, it easier to identify words that do not belong to that category. Operationally, each topic is represented as its top words plus one "intruder" word which has a low probability of belonging to that topic, but a different topic. Topic coherence is then judged by how well human
high probability of belonging annotators detect the "intruder" word.
2.2 NPMI: The Standard Automated Topic Model Coherence Evaluation
Using the word intrusion task, Chang et al. (2009) showed that perplexity-the original topic model evaluation metric-negatively correlates with human evaluations of topic quality. This finding revealed need for automated measurement of topic coherence: automated metric can measure model quality without expensive, time-consumen consuminn,,,,, and difficult-o--reproduuee human experiments.
Lau et al. (2014) find some metrics that positively correlate with human intrusion and rating scores, particularly when aggregating scores over all topies from given model. Because of that validation, the prevailing evaluation for model comparison is pairwise normalized pointwise mutual information. NPMI scores topics highly if the top N words-summed over all pairs w; and have high joint probability P(w,, wx) compared their marginal probability
1-1
log P(an))((es
p(w),ws)
ΣΣ
log P(w;, wy)
The probabilities are estimated using word co-occurrence counts from a reference corpus for specific context window (which can range from ten words to the entire document), As result, the choice of reference corpus determines the strength of human correlation (Lau et al. 2014 Röder et al. 2015).
'This perspective aligns with Propositions and of Doogan and Buntine 2021 "an interpretable topic one that can easily labeled," and "has high agreement on labels." Alternative metrics exist, but they typically also rely on either joint probability estimates or NPMI di-
rectly (e.g.. C2 Röder al. 2015)).
3

---

Evaluation
Count
Experimentation
Count
Number of human evaluations
0
(0%)
Preprocessing
Automated Coherence
Inconsistent datasets
12
(30%)
Metric
Ambiguous preprocessing
(23%)
NPMI
26
(72%)
Model comparisons
Other
22
(61%)
All models tuned
5
(13%)
Explicit implementation
22
(61%)
Unclear h.param scarch
16
(40%)
Explicit ref. corpus
10
(28%)
Unclear LDA baseline, if used
(24%)
Perplexity w/o coherence
3
(8%)
Recent baseline (w/in yrs)
31
(78%)
Multiple runs / sig. testing
11
(28%)
Table 2:
Meta-analysis of forty neural topic modeling papers (denominator may change, as not all conditions are applicable). No recent neural topic modeling papers use human evaluations of coherence, and the metrics and models are difficult to replicate.
A measurement is valid to the extent that it measures what it is intended to measure in the real world. Historically, automated coherence has been validated using human judgements from either crowdworkers (Newman et al.) 2010 Aletras and Stevenson 2013) experts (Mimno et al. 2011). However, correlations based on classical models may not be applicable for NTMs. Our skepticism motivated by theory, neural word representations are intimately connected to NPMI, explicitly used by Aletras and Stevenson 2013) and which produce similar NPMI scores as Lau et al. (0014)
Levy and Goldberg (2014) show that multiple representations create factorizations of PMI matrices. Topic models that have access to these rich representations (e.g. Dieng et al.) 2020] and others) could thus create topics with good NPMI scores without explaining the corpus well to user. In contrast to classical topic models, no one has investigated the validity of NPMI evaluation for NTMS.
Given this lacuna, we conduct experiments aimed at validating that automated topic evaluations still correlate with human judgments of neural topic model quality. We compare against two common human evaluations of individual topic quality: direct rating and intrusion. Human evaluations, like automated topic modeling, lack standardization, which we address in Section S
3 A Meta-Analysis of Neural Topic Modeling
We survey the neural topic modeling (NTM) literature to assess the state of evaluation in contemporary topic model development. First, we take all references made by an existing, comprehensive survey of NTMS (Zhao et al. 2021b), from which we select (a) modeling papers which (b) mention topic interpretability and (c) compare models' topics with an existing baseline. This yields forty models, which all claim superior topic coherence. We examine data processing steps, hyperparameter tuning, baseline selection, and automated coherence calculations. Table 2 summarizes results and Appendix A enumerates the papers.
Our analysis reveals variance in all areas. Preprocessing, which can significantly affect model quality and automated metrics, is often (30%) inconsistent across datasets within the same paper. When preprocessing is consistent, authors omit details necessary to fully replicate the pipeline. These issues imply that automated metrics for the same baselines and source datasets vary across papers. Compounding the problem, researchers often train their models on different datasets from those used establish the relationships between human annotations and automated metries; Doogan and Buntine (202T) find that the same metrics may not predict interpretability in new domains. Mirroring findings from Dodge et al. (2019). 40% of papers fail to clearly specify their model tuning procedure, often even the metric used for model selection.
Calculation of automated coherence metrics is equally fraught. As discussed in Section 2 complete specification for NPMI involves several pieces of information, including the reference corpus used to estimate joint word probabilities, the co-occurrence window size, and the number of words selected from the head of the topic distribution. Three out of four papers fail to explicitly indicate the reference corpus; even when we can assume the input corpus used (13 cases), it remains uncertain whether authors use, e.g., a held-out set or the training documents themselves. For the 61% % that specify the implementation of their coherence metric (by pointing code repository or writing out the formula), some of these factors may still be in question. For instance, six authors

---

reference Lau et al. 2014) and the supporting code Tode but the implications ambiguous: the original paper suggests large corpus from the same source as the training data, but the repository script defaults Wikipedia. In other cases, authors use bespoke implementations, which creates room for errors, or deviate from the settings used in human experiments. For example, several papers use document-wide context window with NPMI, which has not been correlated with human judgments.
Last, even if automated evaluations are consistent, all claims of coherence improvement depend on the validity results in Lau et al. (2014) generalizing neural topic models.
4 Closing the Standardization Gap for Topic Models
Our human evaluation of topic model outputs serves multiple purposes: (a) establishing whether NTMS show improved coherence over classical baseline and (b) re-evaluating the efficacy and reliability of automated coherence metrics. In addition, key goal is (c) to provide standardized preprocessing pipeline to support head-to-head comparisons as new methods are developed
We identify two commonly-used datasets, which we in turn process using a standard pipeline. We then estimate topic models on each dataset following computationally fair hyperparameter search. Our standardization efforts are similar to concurrent work by Terragni et al. (2021); the main differences are that we (a) mandate consistent preprocessing between training and reference corpora, (b) support multi-word expressions during vocabulary creation (see below), and (c) support distributed hyperparameter searches.
4.1 Datasets and Preprocessing
Following Chang et al. 2009), we use English articles from Wikipedia and the New York Times (Table 7. For Wikipedia, we use Wikitext- 103 (WIKI, |Merity al. 2017). and for the Times, we subsample roughly 15% of documents from LDC2008T1 (NYT, Sandhaus 2008), making it an order of magnitude larger than WIKI. To compute reference counts, we use a 4.6M document Wikipedia dump from September 2017 and the full 1.8M document LDC2008T19 set, processed identically to the training data.
We use SpaCy (Honnibal et al. 2020) to tokenize and identify entities in the text. We create new tokens for detected entities of the form New_York_City per Krasnashchok and Jouili (2018). Schofield and Mimno (016)) find that lemmatization and word-stemmingg can hurt English topic interpretability, we do not lemmatize. To maintain roughly equal vocabulary size over datasets, we use a power-law relationship of corpus size (c.f. Zipi 1949) to rule out tokens occurring in fewer than given number of documents 1D In addition a standard stopword list, we define corpus-specific stopwords as tokens appearing in more than 90% of documents. See Appendix A.2 for complete preprocessing details.
4.2 Models
We evaluate one venerable classical model and two newer neural models:
Gibbs-LDA As strong classical baseline, we use the widely-loved Mallet (McCallum) 2002
implementation of Gibbs-sampling for LDA (Griffiths and Steyvers 2004). Mallet produces topics of (qualitatively) competitive quality to neural models (Srivastava and Sutton 2017).
Dirichlet-vAE We reimplement Dirichlet-vAr VAE (Burkhardt and Kramer, 2019), state-of-ther NTM. For simplicitly, we use pathwise gradients for the Dirichlet (Jankowiak and Obermeyer 2018), rather than the rejection sampling variational inference of the authors' primary variant Dirichlet- VAF a wholesale improvement on one of the first successful NTMs, the popular ProdI.DA (Srivastaval and Sutton 2017), and is competitive against recent models automated coherence. The generative
$ 'github. com/ tt/ wttpooopcciiiieettteaeeeee lity
9Our preprocessing pipeline is agnostic to dataset and easily portable. gi git.ub. com/ahono/topiccs
J0We target vocabularies approximating the number of words known by an adult English-speaker (Brysbaert
al. 20166 roughly 40k for WIKI and 35k for NYT.
We replicate their NPMI and redundancy scores 20 newsgroups, wwtthhu..co/aacchh//ooooo
5

---

Select which term is the least related to all other terms and your familiarity with the words
Terms
O painting
O paintings
casualties
painter
literary
O poems
Answer Confidence
Jmm familiar with most these terms.
am not familiar with most these terms, but can answer confidently.
am not familiar with most these terms, and so cannot answer confidently.
Figure 1: The word intrusion task presented crowdworkers (the ratings task is in Appendix A
model is simple and retains a broad similarity to LDA. The primary difference is that it does not constrain the estimated topic-word distributions to the simplex.
ETM Thanks to their improved flexiblity, many NTMs incorporate external word representations, on the premise that large-scale, general language knowledge improves topic quality (Bianchi et al.)
2021 Hoyle et al. 2020). The Embedded Topic Model (Dieng et al.) 2020) is popular NTM that relies on word embeddings in its generative model
12
We maintain fixed computational budget per model following the exhortation of Dodge al. (2019) and use random set of 164 hyperparameter settings across datasets for each model type 13 We train models for variable number of steps (a hyperparameter); to calculate automated coherence for the model, we use the topics produced at the last step. For human evaluations. we select the models that maximize NPML, estimated using the reference corpus with ten-word window over the top ten topic words, per Lau et al. (014). We follow the recommendation of Dieng et al. (2020) and learn skip-gram embeddings embeddings did not yield substantially different results). As in Hoyle et al. (2020), we eliminate models with highly redundant topics, a known degeneracy of NTMS (Burkhardt and Kramer (019)):
the training corpus for ETM (experiments with external pretrained
(a) models in which any of the top five words of one topic overlap with another and (b) models that have a topic uniqueness score (Nan al. 2019) above 0.7. Ranges for hyperparameters and other details are in Appendix A.3
5 Human Evaluations of Topic Quality
We use the ratings and word intrusion tasks from Section 2.2 as human evaluations of topic quality. We recruit crowdworkers using Prolific.co. an online panel provider and collect data with the Qualtries survey platform. We pay workers 2.5 USD per ratings survey and 3 USD per word intrusion survey, equivalent to 15 USD/hour.
In order to draw meaningful conclusions from human annotations, we require an adequate number of participants to ensure acceptable statistical power. However, Card et al. (2020) show that many NLP experiments, including those relying on human evaluation, are insufficiently powered to detect model differences reported levels. Adopting straightforward generative model of annotations (Appendix (A.5), we select enough crowdworkers per task to ensure sufficient statistical power (at least 1 - 3 = 0.9) obtain significance at 0.05, resulting in minimum of fifteen crowdworkers per topic for both tasks. On this criterion, both Chang et al. (2009) and thus Lau et al. 2014). with eight annotators, are underpowered.
For each of our two datasets, we generate fifty topies each from the three models in Section 4.2 In the word intrusion task, we sample five of the top ten topic words plus one intruder; for the ratings task, we present the top ten words in order (Figure 4) We separate the datasets for each task and
Nigtthb... ccm/ ad fidjiidingggg ETM
While runtimes can vary drastically by model, this study is not concerned with implementation efficiency (al- though efficiency matters, see Ethayarajh and Jurafsky (202)).

---

Automated
Intrusion
Ratings
0.5
1-
3-
New York Times
0.75
2.5
0.2
0.50
2-
0.25
1.5-
34
0.75
Wikipedia
0.50
2
0.1
0.25
1.5
G-LDA
D-VAE
ETM
G-LDA D-VAE
ETM
G-LDA D-VAE ETM
Figure 2: While automated evaluations (here, NPMI) suggest clear winner between models, human evaluation is more nuanced. Human judgments exhibit greater variability over smaller range of values. Colored circles correspond to pairwise one-tailed significance tests between model scores 0.05; for example, the rightmost orange circle at bottom right shows that human intrusion ratings for VAV are significantly higher than ETM for topics derived from Wikipedia.
randomly sample 40 of the 150 topics. In the ratings task, we include an additional sixteen synthetic poor-quality topics to help calibrate scores and filter low- fow-quality respondents
Phrasing of questions closely follows the wording used by Chang et al. (2009), and crowdworkers received detailed instructions with examples (Appendix A..4) before responding to items As topics can be esoteric (e.g., last columns of Table T. we ask crowdworkers about their familiarity with the words in each question. We speculate that this question
help protect against spurious low scores for otherwise coherent topics, as real-world users of topic models are usually familiar with domain-specific terminology (see further discussion in Section 7
Human Judgment Differs From Automated Metrics
compare human judgments to automated methods on topies estimated using our three models.
6.1 Human Assessment
To establish model differences using human ratings, pairwise significance tests: proportion test for the intrusion scores, test (Mann and Whitney 1947) for the ratings, and a t-test for automated metrics (Figure 2 using one-tailed tests for each pair both directions. Although D-VAE fares better on the intrusion task, evaluation using ratings favors G-LDA
Our human evaluation results are consistent with past iterations of the ratings and word intrusion tasks for topic models. Mimmo et al. (201 report an average of 2.36 on ratings task on a dataset of medical paper abstracts Our ratings means are 2.5 to 2.8 across all variations (Figure 2). Our word intrusion means range from 0.7 to 0.8, which is comparable to the roughly 0.8 accuracy on the LDA model evaluated in Chang al. (2009). Median time taken on the tasks was 8-9 minutes.
"For generating synthetic poor-quality topics, we use random high-probability words appearing in topics from other hyperparameter settings, but that have low probability among selected topics. Eight topics each generated from the vocabularies NYT and WIK1.
Coode convert topic model output into deployable questionnaires is at github. com/ ahoho ttchoho/toppt opiis 16These discrepancies among human tasks support the argument that standard coherence metries alone may insufficient for automated model selection (Doogan and Buntine 202T).
"Newman et al. 2010) and Lau et al. (2014) do not report average.
7

---

NPMI (10-token window)
CN
(110-token window)
Ref. Corpus > Train Corpus +
NYT
WIKI
Train
Val
NYT
WIkI
Train
Val
Intrusion
NYT
0.27
0.43
0.27
0.24
0.34
0.45
0.35
0.34
WIKI
0.34
0.36
0.39
0.17
0.32
0.34
0.34
0.20
Concatenated
0.29
0.40
0.32
0.17
0.32
0.40
0.35
0.24
Rating
NYT
0.37
0.48
0.37
0.39
0.41
0.46
0.44
0.45
WIKI
0.34
0.41
0.44
0.28
0.32
0.40
0.40
0.34
Concatenated
0.37
0.44
0.41
0.35
0.38
0.42
0.42
0.42
Table 3: Spearman correlation coefficients between mean human scores and automated metrics. Underlined values have overlapping bootstrapped 95% confidence intervals with that of the largest value in each row. "Concatenated" refers correlations computed on a concatenation of values for the NYT and WIKI items. "Val" is small held-out set of 15% of the training corpus. Using the more -ata-appropriate logistic and ordered probit regressions for word intrusion and ratings data leads to different conclusions about relative metric strength (Appendix Table I0). CIs are estimated using 1,000 samples.
Following Aletras and Stevenson 2013), calculate inter-annotator agreement with the mean Spearman correlation between each respondent's score per topic and the average of other respondent scores, obtaining value of 0.75 (compare their value of 0.7 the NYT corpus). Additionally, we include synthetic poor-quality topics (footnote corcerrettly identified by annotators- we monitor the duration taken for the survey hedge against insincere submissions.
6.2 Automated Metrics
NPMI declares D- D-E the unequivocal victor among the three models (with G-LDA a clear second), very different story from the human judgments. To understand the relationship between automated metrics and human ratings, we estimate the Spearman correlation between the two sets of values for each task and dataset for metric variants (3). Although previous studies have used mean human ratings over topies, this decision obscures the inherent variance of the human ratings and leads to overconfident estimates. We therefore construct 95% confidence intervals by resampling ratings, with replacement, equal to the number of annotators per task (Table 3). We estimate NPMI with the standard 10-word window and C, (Röder et al.) 2015) with the recommended 1 110-word window 18 The Wikipedia corpus appears to be best correlated with human judgments, even for the models trained the NYT corpus- this contradicts Lau et al. (2014), where within-domain data have the highest correlations.
While all correlation coefficients are statistically significant, the strength of the correlation alone does not justify their use in model selection, as is standard in the NTM literature (Section B). In particular, the inherent uncertainty of human judgments means that it is difficult to determine when an increase a model's 's mean automated coherence implies a significant improvement the corresponding human scores 19
As noted above (Figure 2 automated metrics exaggerate model differences compared human judgments, To help clarify the utility of automated metrics for model selection, we ask how often an automated metric incorrectly asserts that one model superior to another. To do so, we generate bootstrapped estimate of the false discovery rate of each model. First, for each dataset, we randomly sample two independent sets of K along with their corresponding automated and human scores (resampled with replacement, as in Table (3p. Treating the two sampled sets as outputs from two different models, we compute pairwise significance tests between each set for both the K automated metrics and K x M human scores (using proportions z-test for the intrusion scores and (-tests for all other values). After repeating this process for N = 1000 iterations, we report the proportion of significant differences detected using
topies (without replacement) from the original pool of 150,
We gensim [Rehåfek and Sojka, 2010 to calculate coherence. We process the reference corpora identically to the training data, retaining only terms that exist in the training vocabulary. Other metrics, like Cuci (Newman et al. (0000) and CUMASS (Mimno et al. 200TT, show low correlations.
Better models of human scores could help quantify this relationship (e.g.. GLMs, Appendix A. 10
8

---

NPMI (10-token window)
C, (110-token window)
Ref. Corpus + Train Corpus
NYT
WIKI
Train
NYT
WIKI
Train
Intrusion
NYT
46/53
34/48 48
48 /0
35 33/38
30
WIKI
74
33/78
33/77 75
45/48
38 /9
37 /4
Concatenated
67/67
40/66
41 41/66
36/46 46
31 44
30 45
Rating
NYT
45 50
45/51
417
27 29
26 26
21/26
WIKI
70/73
31
33/71
38 /0
31 /4/
28 28/3
Concatenated
/9/66
36/66
/2
31 31/38
28/
28/38
19/36
Table 4: False discovery rate (1 (1-precision, lower is better) and false omission rate of significant model differences when using automated metrics; automated metrics often overstate meaningful model differences. Bolded values are those with the lowest geometric mean of FDR and FOR. We sample two independent sets of 50 topics along with their human scores and automated metrics; these sets act as outputs of two "models". We then compute significance tests between sets (per Figure () on both the automated scores and human scores. A false positive occurs when one set has significantly larger automated scores despite no meaningful difference actual human scores. Estimates are over 1,000 samples.
Intrusion
Ratings
0.8
0.6
York
0.4
0.2
Familarity
All Fittaadd
0.6
2
Vilipedaa
0.4
1
0.2
G-LDA
D-VAE
ETM
G-LDA
D-VAE
ETM
Figure 3: Mean human evaluation on the ratings and word intrusion tasks, after filtering out respon- dents who reported lack of familiarity with the topic words. When filtering, D-VAE scores improve, highlighting its tendency to produce esoteric topics.
the predicted scores despite equivalent human scores (after correcting for the probability of type I errors, a = 0.05) 20 Even the best-performing automated metrics predict significant differences absent meaningful human effect roughly one-fifth of the time (Table 4
These results suggest that automated metrics alone may be inadequate for model comparison.
6.3 Explaining the discrepancy
One reason for the discrepancy between human judgments and automated metrics that metrics favor more esoteric topics. Specifically, there is significant negative correlation between topic's NPMI C, and the share of respondents reporting familiarity with topic words (Pearson's 's = -0.29). And while D- VAE achieves the highest automated metric scores of the three models. it produces topics with the fewest familiar words: respondents report familiarity with terms over 90% the time on both tasks for G-LDA and ETM, but they do only 70% of the time for D-VAE. This difference suggests that the topics selected by D-VAE are narrower in scope than those the other models. As shown in Figure 33, removing item annotations where respondents indicate unfamiliarity causes both accuracy in the word intrusion task and the ratio of "Very related" terms in the ratings task for D-VAE increase substantially.
Qualitatively, this result is apparent when examining topics with a high NPMI but low humans ratings. In Table 5 the top rows consists of financial terms that frequently appear together in NYT articles,
Details testing equivalence are in Section A.5.1

---

Data
Model
Topic
NPMI
Rat.
Int.
D-VAE
inc 6mo carss rev qir 9mo nyse outst dee waterline conning turrets boilers amidships aft knots armament guns mounts
0.56
1.60
0.77
WIKI
D-VAE
0.33
1.93
0.65
NYT
G-LDA
bedruom bath year market listed kitchen broker weeks
0.30
2.00
0.23
NYT
D-VAE
condolences mourns mourn board of directors heartfelt deepest esteemed
0.38
2.60
0.23
NYT
D-VAE
sharcholders earnings federated mei shares takeover new_york sar__orr__ssh stock exccange
0.18
3.00
0.81
WIKI
D-VAE
continental_ _arny expedition militia frigate musket frigales muskets skirmish
0.11
3.00
0.69
NYT
D-VAE
medicaid medicare hospituls welfare uninsured patients
0.13
2.80
0.96
NYT
G-LDA
city mayor state new__york new new_york _ook_ooroityyyo -iit officials county yesterday governor
0.09
2.53
1.00
Table 5: Topics with the largest human- NPMI discrepancies; top half are topics where NPMI is high and human preferences are low, bottom half is the reverse. NPMI favors esoteric and corpus-spectiss topics. NPMI is calculated with 10-token sliding window over the in-domain reference corpus, Rat. is the average 3-point rating for topic, and Int, refers to the percentage of annotators who identify the intruder word.
and the second row contains rare terms about boating- aagginby both are reasonable topics for their respective corpora. We can also see instances where words are qualitatively very related (bottom half of table), but that NPMI fails to score eigh-perhaps. because these words, while related, may not frequently appear together within a ten-word sliding window (Equation]) D.
Even for familiar words, some topics may be sensible in the context of the specific corpus, despite their component words lacking an immediately obvious semantic relationship. For example, the topic words in the third and fourth rows appear somewhat unrelated (e.g.. "taxes" and "bedroom" in the third row), but they are in fact characteristic of common document types in the New York Times: real estate listings and obituaries. Topics like these render the word intrusion task more difficult: only 23% of crowdworkers identified the intruder for both topics.
Furthermore, using term familiarity as proxy for domain expertise does not address the key problems with topic model evaluation: even after filtering out respondents who are not familiar with topic terms, automated metries still overstate model differences (A.7). The problems with topic model evaluation may therefore extend to our choice of human evaluations as well.
io Automated Topic Modeling Evaluation Broken?
To the extent that our experimentation accurately represents current practice, our results do suggest that topic model bvaluation--both automated and human-is overdue for a careful reconsideration. In this, we agree with Doogan and Buntine (2021). who write that "coherence measures designed for older models [...] may be incompatible with newer models" and instead argue for evaluation paradigms centered on corpus exploration and labeling. The right starting point for this reassessment the recognition that both automated and human evaluations are abstractions of real-world problem. The familiar use of precision-at- in information retrieval, for example, corresponds to a user who only willing to consider the top ten retrieved documents. In future work, we intend to explore automated metrics that better approximate the preferences of real-world topic model users.
One primary of topic models is in computer-assistedd content analysis. In that context, rather than taking a methods-driven approach to evaluation, it would make sense to take a needs-driven approach. [T Generic evaluation of topic models using domain-general corpora like NYT needs to be revisited, since there is no such thing as a "generic" corpus for content analysis, nor a generic analyst. Content analysis can be formulated in broad way, as Krippendorfi (2004) has shown, but its actual application is always in a domain, by people familiar with that domain. This fact stands in tension with the desirable practicalities of general corpora and crowdworker annotation, and the field will need to address this tension. We have identified "coherence" as calling out a latent concept in the mind of reader. It follows that we must think about who the relevant human readers are and the conceptual spaces that matter to them.
Thees needs also have a computational component: neural models usually have longer runtimes even when accelerated with GPUs, whereas many practitioners work in local, CPU-only, environments. See Appendix A.3 for additional details runtimes.
10

---

Acknowledgements
This material is based upon work supported by the National Science Foundation under Grants 2031736, 2008761, 1822494, ARLIS, and by an Amazon Research Award. We thank Sweta Agrawal for her suggestion to conduct a meta-analysis. We owe much appreciation to Dallas Card for his keen advice on power analyses. Thanks Frank Fineis for help on several statistical questions, as well as Shuo Chen for his suggestions regarding the false discovery rate calculations. Finally, we thank Caitie Doogan for her helpful comments on the clarity of argumentation, as well as our anonymous reviewers.
References
Nikolaos Aletras and Mark Stevenson. 2013. Evaluating topic coherence using distributional se- mantics In International Conference on Computational Semantics (IWCS). Association for Computational Linguistics.
Daniel Allington, Sarah Brouillette, and David Golumbia. 2016. Neoliberal tools (and archives): political history of digital humanities) In LA Review of Books.
Areej Alokaili, Nikolaos Aletras, and Mark Stevenson. 2019. Re-ranking words to improve in- terpretability of automatically generated topics In International Conference on Computational Semantics. Association for Computational Linguistics.
Shraey Bhatia, Jey Han Lau, and Timothy Baldwin. 2017. An automatic approach for document-level topic model evaluation In Conference on Computational Natural Language Learning, Vancouver, Canada. Association for Computational Linguistics.
Federico Bianchi, Silvia Terragni, and Dirk Hovy. 2021.
Pre-training is hot topic: Contextu alized document embeddings improve topic coherence In Proceedings of the Association for Computational Linguistics, Online. Association for Computational Linguistics.
David M. Blei, Andrew Ng. and Michael I. Jordan. 2003. Latent Dirichlet Allocation. Journal of Machine Learning Research, 3:993-1022.
Samuel R. Bowman, Luke Vilnis, Oriol Vinyals, Andrew Dai, Rafal Jozefowicz, and Samy Bengio. 2016. Generating sentences from continuous space In Conference Computational Natural Language Learning. Association for Computational Linguistics.
Jordan Boyd-Graber, Yuening Hu, and David Mimno. 2017. Applications of Topic Models. NOW Publishers.
Mare Brysbaert, Michaël Stevens, Pawel Mandera, and Emmanuel Keuleers. 2016. How many words do we know? Practical estimates of vocabulary size dependent on word definition, the degree of language input and the participant's 's age In Frontiers Psychology.
Sophie Burkhardt and Stefan Kramer. 2019. Variational Autoencoder Topic Model
Decoupling Sparsity and Smoothness in the Dirichlet In Journal of Machine Learning Research.
Dallas Card, Peter Henderson, Urvashi Khandelwal, Robin Jia, Kyle Mahowald. and Dan Jurafsky. 2020. With little power comes great responsibility. In Proceedings of Empirical Methods in Natural Language Processing. Association for Computational Linguistics.
Dallas Card, Chenhao Tan, and Noah A. Smith. 2018. Neural models for documents with metadata In Proceedings of the Association for Computational Linguistics. Association for Computational Linguistics.
Jonathan Chang, Jordan I.. Boyd-Graber, Sean Gerrish, Chong Wang, and David M. Blei. 2009. Reading tea leaves: How humans interpret topic models, In Proceedings of Advances Neural Information Processing Systems. Curran Associates, Inc

---

Jason Chuang. John D. Wilkerson, Rebecca Weiss, Dustin Tingley, Brandon M. Stewart, Margaret E. Roberts, Forough Poursabzi-Sangdeh, Justin Grimmer, Leah Findlater, Jordan Boyd-Graber and Jeff Heer. 2014. Computer-assisted content analysis Topic models for exploring multiple subjective interpretations. In Advances in Neural Information Processing Systems Workshop on Humman-Propelled Machine Learning.
Matthew J Denny and Arthur Spirling. 2018. Text preprocessing for unsupervised learning: Why it matters, when it misleads, and what to do about it. In Political Analysis. Cambridge University Press.
Adji B. Dieng, Francisco J. Ruiz, and David M. Blei. 2020. Topic modeling in embedding spaces Transactions of the Association for Computational Linguistics.
Ran Ding, Ramesh Nallapati, and Bing Xiang. 2018. Coherence-aware neural topic modeling In Proceedings of Empirical Methods in Natural Language Processing. Association for Computational
Linguistics.
Jesse Dodge, Suchin Gururangan, Dallas Card, Roy Schwartz, and Noah A. Smith. 2019. Show your work: Improved reporting of experimental results In Proceedings of Empirical Methods in Natural Language Processing. Association for Computational Linguistics.
Caitlin Doogan and Wray Buntine. 2021. Topic model or topic twaddle? Re-evaluating semantic interpretabillty measures] In Conference of the North American Chapter of the Association for Computational Linguistics. Association for Computational Linguistics.
Jacob Eisenstein, Amr Ahmed, and Eric P. Xing. 2011. Sparse additive generative models of text In Proceedings of the International Conference of Machine Learning. Omnipress.
Kawin Ethayarajh and Dan Jurafsky. 2020. Utility is in the eye of the user: A critique of NLP leaderboard design. In Proceedings of Empirical Methods in Natural Language Processing. Association for Computational Linguistics.
Alan H. Feiveson. 2002. Power by simulation In The Stata Journal.
Jiachun Feng, Zusheng Zhang. Cheng Ding, Yanghui Rao, and Haoran Xie. 2020. Context reinforced neural topic modeling over short texts. In ArXiv.
Thomas L Griffiths and Mark Steyvers. 2004. Finding scientific topics. In Proceedings of the National Academy of Sciences. National Academy of Sciences.
Justin Grimmer and Brandon M Stewart. 2013. Text as data: The promise and pitfalls of automatic content analysis methods for political texts. In Political Analysis. Cambridge University Press.
Lin Gui, Jia Leng, Gabriele Pergola, Yu Zhou, Ruifeng Xu, and Yulan He. 2019. Neural topic model with reinforcement learning In Proceedings of Empirical Methods in Natural Language Processing. Association for Computational Linguistics.
Pankaj Gupta, Yatin Chaudhary, F. Buettner, and Hinrich Schütze, 2019a. textTOvee: Deep con-
textualized neural autoregressive models of language with distributed compositional prior In Proceedings of the International Conference on Learning Representations.
Pankaj Gupta, Yatin Chaudhary, Florian Buettner, and Hinrich Schütze. 2019b. Document informed neural autoregressive topic models with distributional prior Association for the Advancement of Artificial Intelligence. AAAI Press.
Ruifang He, Xuefei Zhang, Di Jin, Longbiao Wang, Jianwu Dang, and Xiangang Li. 2018. Interaction aware topic model for microblog conversations through network embedding and user attention In International Conference on Computational Linguistics. Association for Computational Linguis-
tics.
Matthew Honnibal, Ines Montani, Sofie Van Landeghem, and Adriane Boyd. 2020.
spaCy: Industrial
strength Natural Language Processing in Python
12

---

Alexander Miserlis Hoyle, Pranav Goel, and Philip Resnik. 2020. [mprovvin Neural Topie Mod- els using Knowledge Distillation In Proceedings of Empirical Methods in Natural Language Processing. Association for Computational Linguisties.
Xuemeng Hu, Rui Wang. Deyu Zhou, and Yuxuan Xiong. 2020. Neural topic modeling with cycle--onsiisent adversarial training In Proceedings of Empirical Methods in Natural Language Processing. Association for Computational Linguistics.
Karoliina Isoaho, Daria Gritsenko, and Eetu Mäkelä. 2021. Topic modeling and text analysis for gualitative policy research In Policy Studies Journal.
Masaru Isonuma, Junichiro Mori, Danushka Bollegala, and Ichiro Sakata. 2020. Stre-Streetured
Neural Topic Model In Proceedings of the Association for Computational Linguistics. Association for Computational Linguistics.
Martin Jankowiak and Fritz Obermeyer. 2018. Pathwise derivatives beyond the reparameterizztioon trick, In Proceedings of the International Conference of Machine Learning. PMLR.
Weonyoung Joo. Wonsung Lee, Sungrae Park, and Il-Chul Moon. 2020. Dirichlet variational
autoencoder Pattern Recognition, 107:107514.
Namkyu Jung and Hyeong In Choi. 2017. Continuous semantic topic embedding model using variational autoencoder ArXiv.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization Proceedings of the International Conference on Learning Representations.
In
Katsiaryna Krasnashchok and Salim Jouili. 2018. Improving topic quality by promoting named
entities in topic modeling) In Proceedings of the Association for Computational Linguistics. Association for Computational Linguistics.
Klaus Krippendorff. 2004. Content Analysis: an Introduction its Methodology. SAGE
Jey Han Lau, David Newman, and Timothy Baldwin. 2014. Machine reading tea leaves: Automatically levaluating topic coherence and topic model quality] In Conference of the North American Chapter of the Association for Computational Linguistics. Association for Computational Linguistics.
Omer Levy and Yoav Goldberg. 2014, Neural word embedding as implicit matrix factorization In Proceedings of Advances Neural Information Processing Systems. Curran Associates, Inc.
Lihui Lin, Hongyu Jiang, and Yanghui Rao. 2020. Copula guided neural topic modelling for short texts] In Proceedings of the ACM SIGIR Conference on Research and Development in Information
Retrieval. ACM.
Tianyi Lin, Zhiyue Hu, and Xin Guo. 2019. Sparsemax and relaxed wasserstein for topic sparsity In International Conference on Web Search and Data Mining (WSDM). ACM.
Zachary C Lipton. 2018 The mythos of model interpretabilityy: In machine learning, the concept of interpretability is both important and slippery. In Queue. ACM.
Lin Liu, Lin Tang. Wen Dong, Shaowen Yao, and Wei Zhou. 2016. An overview of topic modeling and its current applications in bioinformatics. In SpringerPlus.
Luyang Liu, Heyan Huang, Yang Gao, Yongfeng Zhang, and Xiaochi Wei. 2019. Neural variational correlated topic modeling In Proceedings of the World Wide Web Conference. ACM.
Jeffrey Lund, Piper Armstrong, Wilson Feam, Stephen Cowley, Emily Hales, and Kevin Seppi. 2019. Cross-referencinn using fine- ain-gnaieed topic modeling In Proceedings of the Association for Computational Linguistics, Minncapolis, Minnesota. Association for Computational Linguistics.
Henry Berthold Mann and Donald Ransom Whitney. 1947. On Test of Whether one of Two Random Variables Stochastically Larger than the Other In The Annals of Mathematical Statistics. Institute of Mathematical Statistics.
13

---

Stephen Marche. 2012.
Literature is not data: Against digital humanities In LAA Review of Books.
Andrew Kachites McCallum. 2002. MALLET: A machine learning for language toolkit
Elijah Meeks and Scott B Weingart. 2012. The digital humanities contribution topic modeling. In Journal of Digital Humanilies.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2017. Pointer sentinel mixture
models In Proceedings of the International Conference on Learning Representations.
Yishu Miao, Edward Grefenstette, and Phil Blunsom, 2017. Discovering discrete latent topics with neural variational inference In Proceedings of the International Conference of Machine Learning. PMLR.
Yishu Miao, Lei Yu, and Phil Blunsom. 2016. Neural variational inference for text processing In Proceedings of the International Conference of Machine Learning. PMLR.
David Mimno, Hanna Wallach, Edmund Talley, Miriam Leenders, and Andrew McCallum. 2011
Optimizing semantic coherence in topie models] In Proceedings of Empirical Methods Natural Language Processing. Association for Computational Linguistics.
John W. Mohr and Petko Bogdanov. 2013. Introduction -topi models: What they are and why they
matter In Poetics.
Fred Morstatter and Huan Liu. 2018. In search of coherence and consensus: Measuring the inter pretability of statistical topics Journal of Machine Learning Research.
Feng Nan, Ran Ding, Ramesh Nallapati, and Bing Xiang. 2019. Toppi modeling with Wasserstein autoencoders In Proceedings of the Association for Computational Linguistics. Association for Computational Linguistics.
David Newman, Jey Han Lau, Karl Grieser, and Timothy Baldwin. 2010. Automatic evaluation of topic coherence In Conference of the North American Chapter of the Association for Computa- tional Linguisties. Association for Computational Linguistics.
Dat Quoe Nguyen, Richard Billingsley, Lan Du, and Mark Johnson. 2015. Improving topic models with latent feature word representations Transactions of the Association for Computational Lingwistics.
Xuefei Ning, Y. Zheng, Zhuxi Jiang, Y. Wang, H. Yang, and Huang. 2020. Nonparametric topic modeling with neural inference. In Neurocomputing.
Madhur Panwar, Shashank Shailabh, Milan Aggarwal, and Balaji Krishnamurthy. 2020. TAN- NTM: Topic attention networks for neural topic modeling. In Proceedings of the Association for Computational Linguistics.
Min Peng, Qianqian Xie, Yanchun Zhang, Hua Wang, Xiuzhen Zhang, Jimin Huang, and Gang Tian. 2018. |Neural sparse topical coding In Proceedings of the Association for Computational Linguistics. Association for Computational Linguistics.
Radim Rehúřck and Petr Sojka. 2010. Software Framework for Topic Modelling with Large Corpora. In Proceedings of the Language Resources and Evaluation Conference. ELRA.
Mehdi Rezaee and Francis Ferraro. 2020. A discrete variational recurrent topic model without the reparametriiztionn trick, In Proceedings of Advances in Neural Information Processing Systems. Curran Associates, Inc.
Michael Röder, Andreas Both, and Alexander Hinneburg. 2015. Exploring the space of topic coherence measures) In International Conference on Web Search and Data Mining (WSDM).
ACM.
Evan Sandhaus. 2008. The New York Times annotated corpus. In Linguistic Data Consortium.
14

---

Benjamin M Schmidt. 2012. Words alone: Dismantling topie models in the humanities In Journal of Digital Humanities.
Alexandra Schofield and David Mimno. 2016. Comparing apples to apple: The effects of stemmers topic models, Transactions of the Association for Computational Linguistics.
Donald J Schuirmann. 1987. A comparison of the two one-sided tests procedure and the power approach for assessing the equivalence of average bioavailability. In Journal of pharmacokinetics and biopharmaceutics. Springer.
Denys Silveira, André Carvalho, Marco Cristo, and Marie- Francine Moens. 2018. Topic Modeling us- ing Variational uuto--Enooders with Gumbel-Softmax and Logistic-Normal Mixture Distributions In International Joint Conference Neural Networks (IJCNN).
Akash Srivastava and Charles Sutton. 2017. Autoencoding variational inference for topic models In Proceedings of the International Conference on Learning Representations.
Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan J. Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. 2020. Learning to summarize from human feedback In Proceedings of Advances in Neural Information Processing Systems. Curran Associates, Inc.
Marilyn Strathem. 1997. Improving Ratings: Audit in the british university system In European Review. Cambridge University Press.
Silvia Terrggni, Elisabetta Fersini, Bruno Giovanni Galuzzi, Pietro Tropeano, and Antonio Candelieri. 2021. OCTIS: Comparing and optimizing topic models simple! In Conference of the North American Chapter of the Association for Computational Linguisties. Association for Computational Linguistics.
Laure Thompson and D. Mimno. 2020. Topic modeling with contextualized word representation clusters. In ArXiv.
Runzhi Tian, Yongyi Mao, and Richong Zhang. 2020. Learning VAE-LDA models with rounded reparameterization trick) In Proceedings of Empirical Methods Natural Language Processing. Association for Computational Linguistics.
William E Underwood. 2017. A genealogy of distant reading. In Digital Humanities Quarterly. Alliance of Digital Humanities Organisations.
Rui Wang, Xuemeng Hu, Deyu Zhou, Yulan He, Yuxuan Xiong, Chenchen Ye, and Haiyang Xu. 2020a. NNurral topic modeling with bidirectional adversarial training In Proceedings of the Association for Computational Linguistics. Association for Computational Linguistics.
Rui Wang, Deyu Zhou, and Yulan He. 2020b. ATM: Adversarial topic model. In Proceedings of the Association for Computational Linguistics.
Stefan Wellek. 2010. Testing Statistical Hypotheses of Equivalence and Noninferiority Chapman and Hall/CRC.
Jiemin Wu, Yanghui Rao, Zusheng Zhang, Haoran Xie, Qing Li, Fu Lee Wang, and Ziye Chen. 2020a. Neural mixed counting models for dispersed topic discovery In Proceedings of the Association for Computational Linguistics. Association for Computational Linguistics.
Xiaobao Wu, Chunping Li,
Yan Zhu, and Yishu Miao. 2020b. Short text topic modeling with topic distribution quantization and negative sampling decoder In Proceedings of Empirical Methods in Natural Language Processing. Association for Computational Linguistics.
Liang Yang, Fan Wu, Junhua Gu, Chuan Wang, Xiaochun Cao, Di Jin, and Yuanfang Guo. 2020. Graph attention topic modeling network Proceedings of the World Wide Web Conference. ACM.
15

---

Hao Zhang, Bo Chen, Dandan Guo, and Mingyuan Zhou. 2018. WHAT: weibull hybrid autoencoding inference for deep topic modeling In Proceedings of the International Conference on Learning Representations.
He Zhao, Lan Du, Wray L.. Buntine, and Mingyuan Zhou. 2018. Dirichlet belief networks for topic structure learning In Proceedings Advances in Neural Information Processing Systems. Curran Associates, Inc.
He Zhao, Dinh Phung, Viet Huynh, Trung Le, and Wray Buntine. 2021a. Neural topic model via optimal transport In Proceedings of the International Conference on Learning Representations.
He Zhao, Dinh Q. Phung, Viet Huynh, Y. Jin, Lan Du, and W. Buntine. 2021b. Topic modelling meets deep neural networks: A survey. In ArXiv.
Deyu Zhou, Xuemeng Hu, and Rui Wang. 2020. Neural topic modeling by incorporating document relationship graph In Proceedings of Empirical Methods in Natural Language Processing. Association for Computational Linguistics.
Qile Zhu, Zheng Feng, and Xiaolin Li. 2018. GraphBTMM Graph enhanced autoencoded variational inference for biterm topic model In Proceedings of Empirical Methods in Natural Language Processing. Association for Computational Linguistics.
George K. Zipf. 1949. Human Behaviour and the Principle of Least Effort. Aadison-Wesley,
16


Reviews:
[{"review_id": "luk8tXUb53r", "reviewer": "Reviewer_KzLn", "summary": "This paper questions the use of automatic metric, such as the ones based on NPMI, for evaluating the quality of topic models. It sets up a precise, reproducible framework (with code) that compare classic approaches (LDA) and recent ones based on neural networks (D-VAE). From my point of view, it is more a positioning paper that ca n help the community to change its habits than a novel solution to a given problem. However I think we need this type of papers which can lead to designing better models.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall I found that the paper is well motivated and well written (despite a couple of typos, see below). I agree with the main assumptions: the current setting for evaluating the topic models is flawed and we need a stronger framework. It's not really surprising that recent neural models that leverage word embedding achieve better results when we look at metrics such as NPMI on an external corpus... what the authors point out when they mention the Goodharts’ law (\"when a measure becomes a target, it ceases to be a good measure\").\n\nI'd add that judging the quality of topics should be based not only on the targeted task but also on the people who will use them. It seems clear to me that an expert will need more precise, focused topics than a non expert. We can find some discussion about this point in the paper (last paragraph of 6.1), but I think a novel evaluation setting may go beyond.\n\nI will vote for accepting this paper because I think we need this kind of work in the community in order to face the so many topic models published every year. However I have a couple of small concerns I'd like to raise:\n\n- I understand the choice to follow the original framework designed by the seminal papers (Chang et al., 2009; Lau et al., 2014). However, I think this work should have been the occasion to go a step beyond. For instance, the quality of topics may be studied at the light of the expertise / expectation of the user. Another possible improvement I saw in some previous work: it's not the same situation if the spurious word is taken from a close topic or a far topic.\n\n- I think the final discussion should include the practical complexity of the different approaches. Table 11 of appendix is not included but it clearly shows that the \"good old LDA\" is considerably faster than modern NTM. If it's at the price of some points in accuracy, most of the users won't opt for NTM.\n\nSome typos I found:\n- a straightforward a generative model =>  a straightforward generative model\n- uniquivocal => unequivocal\n- into quesetion => into question \n\n---\n\nI've read authors' response and I'm still ok for accepting this paper.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I don't see any negative impact of this work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2h"}, "extra_text": {}}, {"review_id": "i9sXur4XUx", "reviewer": "Reviewer_KzUd", "summary": "This paper investigates automated evaluation metrics of topic models in terms of coherence such as NPMI. NPMI is widely used to evaluate the performance of topic models, but the authors tackle that it is less related to human evaluation metrics such as rating and intrusion. For experiments, the authors train three topic models, LDA, D-VAE, and ETM with two different corpora, Wikipedia and new york times. To evaluate the topics by humans, the authors ask at least fifteen crowdworkers per topic to avoid insufficient power of human evaluation problem. Experiments show that D-VAE outperforms other models in terms of NPMI, but there are no significant differences among models by human evaluations. \n\nThe main strength of this paper is claiming the usefulness of automated evaluations of topic models with deep considerations that follows recent concerns of NLP research evaluations. I really enjoy reading this paper since it can guide the way to compare the two or more different evaluation metrics in various domains including topic modeling.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: Tasks, models, and metrics are not new, but the analysis methods are somewhat new.\n\nQuality: Experiments and methods are sound. Results are quite interesting but it would be better to show the reason for results that are not the same as previous work (i.e. why the Wikipedia corpus is best correlated with human scores even the training data is not the same)\n\nClarity: This paper is well-written, just some typos and hyperlinks need to be fixed. For example, ‘(Table 6)’ in page 6 will be ‘(Table 3)’. \n\nSignificance: The claim is important because we need to create better automated evaluation methods for topic models. And the final section is also good for discussion about how to create a better automated topic modeling evaluation metric.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "\nI have some questions and comments that I would like to hear from the authors.\n\n- How about rethinking about the human evaluation methods [1]? The authors claim the automated evaluation methods by comparing with the human judgments. What if the human evaluation methods are not good? I am curious and doubtful about the following: Can we capture the difference of relatedness of words by only three scales? What if we increase the numbers to five or ten (i.e. five Likert scales)? Can we possibly change the difficulty of the intrusion task? Of course, this is out of the scope of this paper, but I am curious about the author's opinions for better automated topic model evaluations.\n- Recent papers suggest various automated topic model evaluation methods that are better than NPMI [2,3]. It would be better to discuss these.\n- Why is the Wikipedia corpus best correlated with human scores even though the training data is not the same? It would be better to analyze the reasons such as showing the examples.\n\n[1] Clark E, August T, Serrano S, Haduong N, Gururangan S, Smith NA. All That's' Human'Is Not Gold: Evaluating Human Evaluation of Generated Text. In Proceedings of the ACL 2021\n\n[2] Lund J, Armstrong P, Fearn W, Cowley S, Byun C, Boyd-Graber J, Seppi K. Automatic Evaluation of Local Topic Quality. In Proceedings of the ACL 2019\n\n[3] Xing L, Paul M, Carenini G. Evaluating Topic Quality with Posterior Variability. In Proceedings of the EMNLP 2019\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10"}, "extra_text": {}}, {"review_id": "TzY3oDdP2p", "reviewer": "Reviewer_Ti42", "summary": "The paper addresses standardization gap in automatic metrics used for evaluation of topic modeling as well as a validation gap for the coherence of those automatic metrics with the human judgement, both revisiting the models for which such validation had been done (LDA) as well as the recent Neural Top. Models which have been purely evaluated using the automatic metrics.\n\nThe authors provide a deep investigation on the automatic metrics currently in widespread use within Topic Modeling as well as a historic overview on the last studies of the coherence of the automatic metrics (NPMI) and human judgements.\n\nOne of the main contributions of the paper is the rigorous study and comparison of the human judgements and the automatic metrics NPMI and C_v ensuring the statistical power of the test (involving large-scale evaluation by crowdsourced judges) making this work a steady reference point for future works.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### significance\n\nThe work belongs to the rare kind of \"sanity check\" works around the evaluation of the modern automated methods, in this, case, topic modeling. I believe it has a potential not only affect future work on topic modeling and how it's evaluated but also will inspire thoughts on careful analysis of the adequacy of automated metric used in other fields.\n\nThe work is valuable for the community. It also sets quite a high standard on the statistical rigor of experimentation.\n\n### originality\n\nIt is not (sufficiently) often that the research community undertakes an effort of verifying fundamentals, in this case of an automated metric used in topic modeling. I consider the paper sufficiently original.\n\n### clarity\n\nGiven the complex nature of experiments, the paper successfully explains the nature of the problem, the proposed approach, and the experimentation procedure. Additionally, a discussion on the consequences of the paper is included in Section 7.\n\nMinor observations:\n\nIn **4.2 Models**:\n\n\"estimated using the reference corpus (Table 6) \" - Table 6 is in Appendix and lists the referenced work, so not quite related to the reference corpus. Might be some confusion.\n\n\"...this study is not focused on efficiency of the implementation Ethayarajh and Jurafsky (though cf. 2020).\" -> reference not quite clear\n\nFootnote 9: should it also provide a GitHub link?\n\n\nIn **5 Human Evaluations of Topic Quality**:\n\nWithin main text body: \"we include an additional sixteen synthetic poor-quality topics\"\nThe footnote: \"Eight topics each are generated from the vocabularies of NYT and WIKI.\" - so, why only 8 topics out of 16 are explained?\n\n\n### quality\n\nA high quality writing, very few typos.\n\nLANGUAGE\n\nA few typos:\n\nIn **5 Human Evaluations of Topic Quality**:\n\n\"Adopting a straightforward a generative model \" -> extra \"a\"\n\n\"contents can be esoteric (e.g.,last columns of Table 1),\" -> I'd suggest replacing the word \"esoteric\" in general (more so when the indicated example is about nuclear physics research project).\n\nFigure 1 caption: \"orange circle\" -  colors are not transmitted at black-and-white printing. It's always a great courtesy to the reader to use grayscale in graphics.\n\nFigure 1 caption: \"topics derived from on Wikipedia.\" -> too many prepositions\n\nIn **6 Human Judgment Differs From Automated Metrics** :\n\n\"we compare and contrast model comparisons\" - sounds like tautology\n\nIn **6.1 Human Assessment**:\n\n\"However,D-VAE\" -> space\n\nIn **6.2 Automated Metrics**:\n\n\"per task (table 4).\" -> Table (and a few other references to tables, sections, and figures on the same page)\n\n\"in the the NTM literature\" -> extra \"the\"\n\n\nREFERENCES\n\n\n1. Aren't these 2 references the same paper?:\n\n* Jesse Dodge, Suchin Gururangan, D. Card, Roy Schwartz, and Noah A. Smith. 2019a. Show your work: Improved reporting of experimental results. ArXiv, abs/1909.03004.\n\n* Jesse Dodge, Suchin Gururangan, Dallas Card, Roy Schwartz, and Noah A. Smith. 2019b. Show your work: Improved reporting of experimental results. In Proceedings of Empirical Methods in Natural Language Processing. Association for Computational Linguistics.\n\n2. It's advisable to brush up the reference formatting: some first names are abbreviated, most are not; sometimes \".\" is omitted (e.g., \"David M Blei.\" -- after \"M\")", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "limitations addressed ; societal impact doesn't apply ", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "-IvKGf37we2", "reviewer": "Reviewer_8PGF", "summary": "Topic model evaluation in terms of the quality of learned topics is an important problem in topic modelling, the validation of which has a profound impact on the model development and its application in cross disciplines. This paper reassesses the automatic topic model evaluation metrics, e.g., topic coherence scores like NPMI, that are commonly adopted in the topic modelling literature. It points out the potential flaws of applying those measures to neural topic models developed recently, via conducting a set of human evaluations. The paper identifies two gaps existing in the current evaluation paradigm used in almost all NMTs, the validation gap and the standardisation gap through analysing a large number of NTMs. Its findings could have wide implications in the practice.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "* The paper is well motivated. The meta-analysis of the forty neural topic models is a substantial amount of work. The authors find, for example, inconsistency in document pre-processing, a lack of clarity in the calculation of topic coherence scores, and the ambiguity of comparison settings, which highlights potentially the problems for topic modelling practitioners. I like the argument on their skepticisim on the correlation found on classical models may not be applied for neural topic models, which seems to be supported by their human evaluation.\n* In regard to the human evaluation paradigm, there is nothing new but adopting similar evaluation approaches used by previous work on human evaluation of topic models, i.e., topic quality rating and word intrusion. The difference is that the authors applied human evaluation to neural topic models. \n* The authors carry out both human and automatic evaluation on topics generated by one classical model, Gibbs-LDA and two neural topic models, D-VAE and ETM, and find that “human judgment differs from automatic metrics”, which seems to support their skepticism on applying existing topic coherence measures to neural topic models.  However, to make the finding more convincing, should one consider more topic models in the experiment in order to draw such a conclusion? It would be good to see this finding is widely applicable to NTMs. \n* The paper is written pretty well, and the necessary details of the experiments are provided, including the screenshots of instructions, the hourly wage for the participants in either the main paper or the supplementary materials. Given that there are human subjects involved, one might wonder if the ethics are sought and approved.\n\nSome other minor comments\n* In section 6.1 human assessment, what is the definition of “sub-optimal topics”?\n* The discussion of Table 3, it seems that the words for “good topics” and “bad topics” are not reflected in the table. \n\n----After the author's response----\nI acknowledge that the authors' response to the review comments has been read and considered in the rating. I would like to keep my current rating.\n---", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The findings of this paper suggest that the automatic topic evaluation should be carefully reconsidered and reassessed for newer models, like those recently developed neural topic models. They are going to have an impact on the adaptation of topic models in analyzing corpora in different domains. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": ">5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviews recommended acceptance. The work was seen as somewhat novel and well motivated for an important task of topic model evaluation, and experiments were considered sound. Thus the work seems clearly valuable to the community. Some questions were raised about the experiments (analysis why the Wikipedia corpus is best correlated, alternatives to NPMI, use of only three-point scales in human evaluations, considering expertise of the user), but overall the work seems clearly acceptable.
Decision: Accept (Spotlight)