Large-Scale Correlation Analysis of Automated Metrics for Topic Models
Year: 2023

Abstract:
Automated coherence metrics constitute an important and popular way to evaluate topic models. Previous works present a mixed picture of their presumed correlation with human judgement. In this paper, we conduct a large-scale correlation analysis of coherence metrics. We propose a novel sampling approach to mine topics for the purpose of metric evaluation, and conduct the analysis via three large corpora showing that certain automated coherence metrics are correlated. Moreover, we extend the analysis to measure topical differences between corpora. Lastly, we examine the reliability of human judgement by conducting an extensive user study, which is designed as an amalgamation of different proxy tasks to derive a finer insight into the human decision-making processes. Our findings reveal some correlation between automated coherence metrics and human judgement, especially for generic corpora.

heading: 1 Introduction
text: Topic modelling is an important tool in the analysis and exploration of text corpora in terms of their salient topics (Blei et al., 2003). To evaluate the effectiveness of topic models, the preponderance of topic modeling literature rely on automated coherence metrics. A key benefit is convenience, allowing researchers to sidestep expensive and timeconsuming user studies. The basis for this reliance is the assumption that the coherence metrics correlate with human judgement (Mimno et al., 2011; Lau et al., 2014; Röder et al., 2015).
The presumed correlation with human judgement should not be taken for granted. There are recent works that challenge the assumption. Doogan and Buntine (2021) highlight the inconsistencies of automated coherence metrics via correlation analysis within each metric. In Hoyle et al. (2021), they claimed some disagreement between human judgement and automated coherence metrics.
We postulate that the reasons behind such a mixed picture could be the differences in the topic samples as well as the underlying corpora from which the statistics were derived, resulting in localised “biases” that affect the conclusions reached by respective studies. Given their importance, we seek to conduct an extended analysis of automated coherence metrics on a larger scale than anything previously attempted. This study includes orders of magnitudes greater than the number of topics typically analysed, covering three large corpora, employing a comprehensive user study with extensive labels, across most of the widely used metrics.
There is a strong motivation for quantity. Given a vocabulary, a combinatorially large number of possible topics exist. If each topic is a vector of its scores on different metrics, the resulting curse of dimensionality (Bellman and Kalaba, 1959) necessitates a larger sample size. We argue that evaluating thousands of topics might not be sufficient, and a larger sample size is required to approximate a diverse distribution, where sampled topics is representative of the corpus and the metrics.
We surmise that the previous practice of using topic models to generate topics could introduce a bias in the analysis. Firstly, topic models vary in performance, Hoyle et al. (2021) compiled a lengthy list. There is also emerging debate on the performance between traditional and neural topic models (Doogan and Buntine, 2021). Additionally, some neural models might be inconsistent, producing different topic sets in independent runs (Hoyle et al., 2022). Conversely, topic model might be too stable and generate similar topics (Xing and Paul, 2018). To objectively evaluate whether the coherence metrics are usable, we propose to generate candidate topics independently of topic models.
In this paper, our contributions are three-fold. First, we begin by analysing the inter-metric correlations (see Section 4). We propose a novel approach to sample “topics” for the purpose of
13874
evaluating automated coherence metrics (see Section 4.1). Compared to prior works, we sample these topics free from topic model bias, and in a meaningful diverse manner. Evaluated on three large corpora, we reaffirm that certain selected metrics do not contradict each other, and highlight the underestimated effects of ϵ (see Section 4.2).
Second, we extend our analysis to investigate inter-corpora correlations (see Section 5). We examine the understated differences of corpora statistics on the metrics by comparing the correlations across corpora. While such correlations do exist to some degree, the metrics are still dependent on each corpus. Thus, any expectation that these metrics would correlate uniformly with human judgement on all possible corpora may be misplaced.
Finally, pivotal to any interpretability research, we design and conduct a user study, which is the keystone of our work (see Section 6). Compared to prior work, its design is more complex as we seek to benchmark human judgement at a finer granularity across different random user study groups (see Section 6.1). We analyse the user study results via a few novel proxy measures, revealing that human judgement is nuanced and varies between individuals, metric correlation to human judgement is corpus-dependant, with the average participant being attuned to the generic corpora (see Section 6.2).
Our implementation and releasable resources can be found here1, and we hope that it will enable convenient coherence evaluation of topic models and to further advance interpretability research.

heading: 2 Related Work
text: Topic models. There are many approaches for topic modelling Blei et al. (2003), from non-neural based Zhao et al. (2017b); Hoffman et al. (2010), to many other neural-based methods, via autoencoders (Kingma and Welling, 2014) such as Miao et al. (2016); Srivastava and Sutton (2017); Dieng et al. (2020); Zhang and Lauw (2020); Bianchi et al. (2021), via graph neural networks (Yang et al., 2020; Shen et al., 2021; Zhang and Lauw, 2022), and hierarchical methods (Meng et al., 2020). A common factor is the use of automated coherence metrics to benchmark against baselines. We select several popular metrics for evaluation as listed in Section 3. Topic models are applied in downstream tasks (Lau et al., 2017; Wang et al., 2019, 2020).
User studies in metric evaluation. Mimno et al. 1https://github.com/PreferredAI/topic-metrics
(2011) utilize expert annotators to independently label 148 topics, using another 10 expert annotators to evaluate the same topics via intruder word detection tasks. Röder et al. (2015) benchmark topics against different permutations of metrics with the largest evaluation set containing 900 topics with human ratings aggregated from prior works (Aletras and Stevenson, 2013; Lau et al., 2014; Rosner et al., 2014). In Hoyle et al. (2021), a minimum of 15 crowdworkers were employed in simple rating and word intrusion tasks evaluating 40 topic-modelgenerated (Griffiths and Steyvers, 2004; Burkhardt and Kramer, 2019; Dieng et al., 2020) and 16 synthetic random topics. In Doogan and Buntine (2021), their largest user study required 4 subject matter experts creating 3,120 labels across 390 topics generated via topic models (Blei et al., 2003; Zhao et al., 2017a). In comparison, our study has both large quantities of topics and study participants, annotating 800 unbiased topics split between 40 study participants with at least an undergraduate level of education, generating 180K word-pair labels2. Our automated experiments deal with hundreds of thousands of unique topics.
Human involvement. There are many interesting research that examine linguistic problems via the human lens. Card et al. (2020) investigates the number of annotators required to achieve significant statistical power. Plank (2022) examines the variation in human labels. Ethayarajh and Jurafsky (2022) questions the authenticity of annotators. Clark et al. (2021) tests the human ability to learn how to differentiate between machine-generated and human-generated texts. Human-in-the-loop systems or processes, such as Li et al. (2022), are also being actively explored.

heading: 3 Preliminaries
text: In this section, we define the automated coherence metrics that we will be using, and describe the corpora we use to obtain the word probabilities.

heading: 3.1 Coherence Metrics
text: We follow the definition styles of Röder et al. (2015), where direct confirmation measure m is a function of a word-pair statistic. Direct coherence metrics is defined as a mean aggregation of m between word-pairs (Equation 1), where t is a topic which is a k-sized set of words. For our evaluations,
2Each question has 45 possible combinations of wordpairs, each label is binary, denoting coherence relations.
we set k = 10. Within t, the words are arranged based on P (w|t) in descending order. Since our approach does not produce P (w|t), we can locally optimize the word positions within a topic to obtain the best possible score for position-sensitive metrics CUMass and CP (See Appendix B). We use subscript s to denote alphabetical order and subscript o to denote optimized positions. Let p = |t|·|t−1|2 , which represents the number of word-pairs in a topic.
C(t,m) = 1
p
∑
wi∈t
∑
wj∈t i>j
m(wi, wj) (1)
CNPMI (Equation 2) is the mean aggregation of mnlr, defined as Normalised Pointwise Mutual Information (NPMI) (Bouma, 2009) value, between word-pair statistics in a topic. We exclude CUCI as it uses Point-wise Mutual Information (Church and Hanks, 1990; Lau et al., 2014), which is correlated to NPMI.
CNPMI(t) = 1
p
∑
wi∈t
∑
wj∈t i>j
mnlr(wi, wj) (2)
mnlr(wi, wj) = log
P (wi,wj)+ϵ P (wi)·P (wj)
− log(P (wi, wj) + ϵ) (3)
CUMass is the mean ordinal aggregation of mlc (Mimno et al., 2011), which measures the log conditional probability between ordered word-pair in a topic:
CUMass(t) = 1
p
∑
wi∈t
∑
wj∈t i>j
mlc(wi, wj) (4)
mlc(wi, wj) = log P (wi, wj) + ϵ
P (wj) (5)
CP is the mean ordinal aggregation of mf , Fitelson’s coherence (Fitelson, 2003), interpreted as the degree to which wi supports wj , between ordered word-pairs in a topic:
CP (t) = 1
p
∑
wi∈t
∑
wj∈t i>j
mf (wi, wj) (6)
mf (wi, wj) = P (wi|wj)− P (wi|¬wj) P (wi|wj) + P (wi|¬wj)
(7)
CV (Equation 8) is the final metric that we are using. CV is considered as an indirect coherence metric, as it uses word-group relations as opposed to word-pairs relations like aforementioned direct coherence metrics. Intuitively, it measures the mean cosine similarity (Equation 9) between each word’s feature vector and the topic’s feature vector represented as the sum of all of its words’ feature vectors (Equation 10).
CV (t, γ) =
∑ wi∈t scos(v(wi, t, γ), v̄(t, γ))
|t| (8)
scos(v⃗i, v⃗j) =
∑ v⃗i · v⃗j
||v⃗i||2 · ||v⃗j ||2 (9)
v̄(t, γ) = ∑
wj∈t v(wj , t, γ) (10)
v(w, t, γ) = {mnlr(w,wj)γ ∀wj ∈ t} (11)
For indirect confirmation measure m̃, instead of directly using word-word probabilities, it uses m to create a vector of features v (Aletras and Stevenson, 2013) that represent a word w from the topic t it belongs to, distorted by hyper-parameter γ (Equation 11). We will evaluate γ at 1 and 23.

heading: 3.2 Corpora
text: We use word co-occurrences statistics obtained from three large corpora:
ArXiv. We use ArXiv abstracts dataset4 where we consider each abstract as a document. These abstracts mainly comprise of research work related to non-medical science disciplines.
Pubmed. We use PubMed Central (PMC) Open Access Subset5 that contains journal articles and pre-prints related to medical research and information. We consider each article body as a document and we remove citations within it.
3Prior to version 0.1.4 (released Sep 21, 2022), Palmetto’s (Röder et al., 2015) γ was set to 2.
4Kaggle - Cornell-University/ArXiv 5ncbi.nlm.nih.gov/pmc/tools/openftlist
Wiki. We use the English-Wikipedia dump6 of August’22 processed using Attardi (2015). We consider the content of the article as a document. To check for correctness, we also use the popular benchmark Palmetto (Röder et al., 2015), which uses a subset of Wikipedia’11.
For each corpus, we apply processing steps suggested in Hoyle et al. (2021), retaining up to 40K frequently occurring words. Moreover, we generate a lemmatized (denoted with the suffix -lemma) and unlemmatized variant (original) for further analysis. More information on common vocabulary between corpora can be found in Table 14, Appendix C.

heading: 4 Examining Inter-Metric Correlations
text: Intuitively, if two different metrics are to correlate with human judgement, we would expect the scores of these metrics to correlate. However, it is claimed in Doogan and Buntine (2021) that these metrics do not correlate well. For reasons described in Section 1, we propose a new non-topic modelling approach to sample topics to evaluate these metrics.

heading: 4.1 Approach: Balanced Sampling
text: There are few tested methods to generate topics: from topic models (Aletras and Stevenson, 2013; Lau et al., 2014), beam search optimized on coherence (Rosner et al., 2014), random sampling of words (Hoyle et al., 2021). Considering only optimized topics, or completely random topics (mostly bad), would generate a skewed distribution. In contrast, we seek to mine topics that emulates a balanced distribution for a meaningful comparison. We also desire uniqueness among topics, which avoids repetition and is representative of the corpus. Figure 1 illustrates an overview of our approach.
Mining topics of k words can be framed as the classical k-clique listing problem (Chiba and Nishizeki, 1985; Danisch et al., 2018). To generate meaningful topics, we can map the corpus-level information as a graph, treating each word from its vocabulary set V as a vertex. Each word will share an edge with every other word. We choose mnlr to determine the value of the edges between two vertices as its normalised range is intuitive allowing us to easily identify the range of values for sub-graph generation. In contrast, using mlc and mf increases sampling’s complexity as they are order-dependant resulting in bi-directional edges in its sub-graph. Sampling using any m, not only
6dumps.wikimedia.org
mnlr, might introduce bias, which our approach seeks to mitigate.
The initial graph will be a complete graph of |V | vertices. A topic of k words would be a ksized sub-graph. Combinatorially, there are |V | choose k number of possible unique topics. It is practically infeasible and unnecessary to list all kcliques. For a more tractable approach, we modify the routine from Yuan et al. (2022) (pseudo-code in Appendix A) to include:
Sub-graphs of varying quality. This routine seeks to generate smaller graphs from the original complete graph to cover the spectrum of topic quality. We eliminate edges conditionally via their value, and the remaining edges and connected vertices constitute the new sub-graph. We generate three different kinds of sub-graphs, pos where edgevalues are above a given lower-bound, mid where edge-values are between threshold values, and neg where edges are below an upper-bound7.
Topic extraction. Inspired by Perozzi et al. (2014), instead of iterating through all the neighbouring nodes or searching for the next best node, we randomly select a neighbour, that has an edge with all explored nodes, to explore. We extract the explored k-path as our sampled topic.
Topic uniqueness. To attain a variety of topics, we remove all edges in a mined clique, making it impossible to sample a similar topic from the same sub-graph. Figure 2 illustrates this feature.
Balance distribution of topics. For a given corpus, we further introduce common topics sampled from a different corpora, which differ in its word distribution. We refer to this segment of external topics as ext. Lastly, random is a segment, comprising of groups of random words, included to represent topics that might not have been covered via the other segments. Table 2 shows the result from this mining approach. The total would thus be more balanced, comprising topics of varying scores along the spectrum.
7Hyper-parameters listed in Table 9, Appendix A
4.2 Evaluation: Metric Correlations Analysis
We evaluate the correlation (Pearson’s r8) between different automated metrics measured on Wiki (see Table 3), Pubmed, and ArXiv (see Table 10, Appendix C). We expect a high positive correlation score between metrics if they are both purportedly measuring for coherence. Our first inter-metric analysis (see Table 3a), with metrics calculated at ϵ = 1e−12, shows the poor correlation of CV metrics against other metrics. Theoretically, CV relies on mnlr as its features, and given
8Based on reasons provided in Doogan and Buntine (2021), with the main argument that datasets (scores) are continuous and have a bi-variate normal distribution.
an unrelated topic, where word-pair scored on mnlr with ϵ = 1e−12 produces similar mnlr vectors which scores highly on CV . This phenomenon of high cosine similarity between the equally negative mnlr vectors, results in contradicting scores between CV and other metrics.
Hence, for our second inter-metric analysis (see Table 3b) we evaluate the metrics at ϵ = 0, denoted with subscript ̸ ϵ. For the resulting undefined calculations, we default to 0. Intuitively, the purpose of setting ϵ = 1e−12 is to prevent and to penalise word-pairs that produces undefined calculation. In contrast, ϵ = 0 treats these word-pairs neutrally. Comparing the new results in Table 3b to the previous results in Table 3a, we note that correlation scores between CV metric and other automated coherence metrics improved greatly, suggesting alleviation of the contradicting factor. Additionally, we note that for CP and CUMass, ϵ is essential. We then examine these metrics with their better ϵ mode (see Table 4a), and most metrics (except CUMass) have a decent correlation with other metrics, implying that they do not contradict each other.
There could be a concern that the neg and random sampled sections would have an outsized influence in the previous analysis. In this ablation, we restrict the same analysis to only topics where CNPMI > 0. Comparing to the previous results (see Table 4a), we derive a similar interpretation from this constrained results (see Table 4b), suggesting that our balanced sampling approach is effective as the behaviour of the full set of data is similar to its smaller subset.

heading: 5 Examining Inter-Corpus Correlations
text: A natural extension after inter-metrics comparison, is to compare metrics measured on different corpora. It is a common expectation that research works would employ multiple corpora, with the differences between corpora quantified superficially (such as in Section 3.2). We propose an alternative approach to quantify the differences, at a topical level, using common topics measured using automated coherence metrics. If the corpora are thematically similar, we would expect a high correlation.
Analysis. Using the common topics from the paired corpora, we conduct a correlation analysis on the scores measured on each corpus per metric. Table 5 shows decent correlations between each corpus. However, even as they are positive, these correlations do not imply identical statistics in various corpora. Assuming that human judgement is constant for a given topic, we posit that variance in scores measured on different corpora could result in a lower correlation due to the missing themes
Corpus |T̄ | Cγ=1V,̸e C γ=2 V,̸e CNPMI,̸e CNPMI CP,o CUMass,o
ArXiv 80K 0.98 0.98 0.98 0.98 0.97 0.92 Pubmed 27K 0.94 0.97 0.94 0.92 0.93 0.94
Wiki 143K 0.99 0.99 0.99 0.98 0.96 0.95
(a) Comparison of scores from selected topics measured on both lemmatized and unlemmatized corpus.
Corpus |T̄ | Cγ=1V,̸e C γ=2 V,̸e CNPMI,̸e CNPMI CP,o CUMass,o
ArXiv 111K 0.97 0.98 0.95 0.94 0.94 0.95 Pubmed 60K 0.97 0.98 0.98 0.92 0.95 0.97
Wiki 150K 0.99 0.98 0.98 0.98 0.98 0.98
(b) Selected topics compared to its lemmatized variants, scores from both variants are measured on unlemmatized corpus.
Corpus |T̄ | Cγ=1V,̸e C γ=2 V,̸e CNPMI,̸e CNPMI CP,o CUMass,o
ArXiv 126K 0.94 0.95 0.92 0.84 0.85 0.88 Pubmed 68K 0.93 0.95 0.91 0.82 0.83 0.82
Wiki 245K 0.98 0.98 0.97 0.92 0.93 0.92
(c) Selected topics, measured on the unlemmatized corpus, are compared to its lemmatized variants, which are measured on the lemmatized corpus.
Table 6: Pearson’s r (mean from 5 independently sampled sets of size |T̄ |) of automated coherence metric measured on different scenarios. Each selected topic will have two variants that will produce two scores for each metric. We compare the correlation of the two set of scores for a set of topics. Error bars omitted as S.D ≤ 0.01. See Table 12 and Table 11, Appendix C for additional quantitative data on the topics.
within the shared vocabulary space in either corpus. We conduct a control analysis on pairs of similar corpus differing in lemmatization, originating from the same documents, in Table 6a. These corpora would be thematically similar whilst being superficially different. Our previous analysis in Table 5, comparing to the control analysis in Table 6a, shows lower correlation scores suggesting some topical difference between the various corpora. This difference highlights the metrics’ strong dependency on the corpus used, with a subset of common topics disagreeing on the scores, revealing that these metrics are not a one-size-fits-all solution for coherence evaluation.
Ablations. While we know how lemmatization affects topic modelling (Schofield and Mimno, 2016), its effect on evaluation is unclear. We carried out two additional ablations simulating lemmatizing topics post-training. For the first ablation, we shortlist topics that contain at least one unlemmatized word, where if lemmatized, the lemmatized word can be found in the same unlemmatized corpus. We compare the correlation of the original and lemmatized topic, with their scores measured on the same unlemmatized corpus. Their scores have a strong correlation (see Table 6b), suggesting that the difference between lemmatized topics and
unlemmatized topics is small. For the second ablation, the shortlisting process is similar, however, with lemmatized topics measured on the lemmatized corpus. Our results (see Table 6c) show a strong correlation across the various metrics and imply that post-processing topics for evaluation is a viable option.

heading: 6 User Study
text: Previous works measure human judgement through simple evaluation tasks such as rating the coherence of a topic on a few-point ordinal scale (Mimno et al., 2011; Aletras and Stevenson, 2013), identifying the intruder word that introduced into the topic (Chang et al., 2009), or both (Lau et al., 2014; Hoyle et al., 2021). For word intrusion, the detection of outliers signals the cohesiveness of the topic, which is similar to rating topics on an ordinal scale. However for both tasks, qualitative gaps might exist. In word intrusion, study participants are restricted to just one outlier per topic, assuming perfect coding, it results in exponential drop in scoring, i.e. 100% detection for a perfect topic, 50% for a topic with a clear outlier, and so forth. For topic ratings, topics of differing qualities might get the same score, i.e. a perfect topic and a topic with a clear outlier might both get the same scores.
Additionally, while the decisions between human annotators might be equivalent, it is not evident if their thought processes are similar. The key reason for this line of inquiry stems from the observation that everyone is different in some aspects, such as knowledge, culture, and experiences. Assuming our understanding of words is influenced by our prior beliefs, what and how we perceive similarity and coherence might differ from person to person.
For these reasons, we decide to design a user study that combines both word intrusion and topic rating tasks but measured at a finer granularity such that we can quantify the decision-making process. Users are tasked to cluster word-groups which indicate coherent and outlier word-groups. We then examine the relationships between automated coherence metrics and different proxy tasks derived from the user study.

heading: 6.1 User Study Design
text: For our study S, we recruit 8 user study groups U , S = {U1, . . . , U8}, 5 study participants per group. Majority of the participants recruited have
at least a graduate degree or are undergraduates. For each study group, we prepared 8 unique question sets Q = {T1, . . . , T8}, each containing 100 10-word topics, Ti = {t1,i, . . . , t100,i} and t = {w0,j,i, . . . , w10,j,i}. For each participant u ∈ Ui, we present each tj,i ∈ Ti individually sorted alphabetically. We ask participants to cluster words in tj,i that they deem similar to form coherent word groups g, where their response Ru,j,i to tj,i is a set of unique g. We constrain each word to only belong to one coherent word group to limit the task complexity. Additionally, a word considered to be unrelated may form its group of one. We use Likert matrix9 as the response format (see Figure 3), mandating a response for each word wk,j,i ∈ tj,i. Actual instructions are shown in Appendix E.
Topic selection. We construct an initial pool of 1000 topics. To achieve comparability between corpus, we randomly sample 400 common topics from Wiki, ArXiv, and Pubmed. To represent nonscientific topics, we randomly sample 200 topics from Wiki that do not appear in ArXiv/Pubmed. For ArXiv/Pubmed exclusive topics, we randomly sample 200 topics each, with these topics also appearing in Wiki. We sample in a 7:1:1:1 ratio of pos/mid/neg/random segments of the corpus, seeking to emulate a uniform score distribution. To account for word familiarity, we select lemmatized topics with words found in 20K most frequently used words10. For each user study, we randomly
9There is no scaling. 10Corpus of Contemporary American English
sampled 100 topics from the pool without replacement. For topics not found in ArXiv or Pubmed, we exclude them during evaluation of those corpus.
Proxy Tasks. Representing coherence as wordclusters allows us to derive a deeper insight into what we perceive as human judgement. From our user study task, we further decompose this study into a few proxy tasks, where we measure the correlation (Spearman’s ρ11) of its results to automated coherent metrics. We propose three topic-level human coherence measures. Using density of human agreement, we define P1 as the mean agreement of Ui on all possible word-pairs on any topic tj,i:
P1(tj,i) =
∑ u∈Ui ∑ g∈Ru |g|(|g| − 1)
|Ui| |tj,i|(|tj,i|−1)2 (12)
If tj,i has perfect agreement on coherence, we expect P1(tj,i) to have a value of 1, and for incoherence, a value of 0.
Subsequently, we consider the largest selected word group within tj,i, and define P2 as the mean of this measure amongst Ui:
P2(tj,i) = 1 |Ui| ∑
u∈Ui max({|g||g ∈ Ru}) (13)
A value of 1 will suggest that each word in tj,i have no relations to each other and a value of |tj,i| suggest perfect agreement on coherence.
Lastly, we define P3 as the mean number of annotated word groups amongst Ui:
P3(tj,i) = 1 |Ui| ∑
u∈Ui |Ru| (14)
The interpretation of P3 is the inverse of P2. While these group-wise measures might seem similar, they measure different nuances of humanannotated data. P1 evaluates the sizes of multiword groups, weighted towards larger groups. P2 only accounts for the largest word group, which ignores the properties of the other remaining group. P3 ignores group sizes to a certain extent and includes single-word "outlier" groups. We evaluate these measures’ correlation against various C(tj,i).

heading: 6.2 User Study Results
text: We find that the three different proxy tasks produce similar results12, shown in Table 7a, 7b, and 7c re-
11We use Spearman’s ρ instead of Pearson’s r, as we generally obtain a better r (than ρ shown) through distortion of scores. To ensure parity, we use ρ instead.
12We note that these results include outlier U3, whose negative results differ radically from other groups. Individual
ArXiv Pubmed Wiki Cγ=1V,̸e 0.319 ± 0.152 0.516 ± 0.067 0.651 ± 0.099 Cγ=2V,̸e 0.356 ± 0.146 0.510 ± 0.095 0.652 ± 0.119 CNPMI,̸e 0.366 ± 0.136 0.521 ± 0.064 0.664 ± 0.094 CNPMI 0.304 ± 0.169 0.428 ± 0.111 0.624 ± 0.087 CP,o 0.266 ± 0.178 0.459 ± 0.093 0.634 ± 0.091
CUMass,o 0.243 ± 0.176 0.183 ± 0.161 0.329 ± 0.066
(a) Proxy Task I: Density of agreement among study participants. Full Breakdown in Table 16, Appendix C.
ArXiv Pubmed Wiki Cγ=1V,̸e 0.316 ± 0.159 0.511 ± 0.053 0.643 ± 0.110 Cγ=2V,̸e 0.355 ± 0.153 0.507 ± 0.080 0.648 ± 0.130 CNPMI,̸e 0.369 ± 0.135 0.517 ± 0.049 0.654 ± 0.104 CNPMI 0.303 ± 0.175 0.421 ± 0.094 0.615 ± 0.090 CP,o 0.260 ± 0.182 0.454 ± 0.081 0.624 ± 0.103
CUMass,o 0.232 ± 0.182 0.170 ± 0.152 0.320 ± 0.060
(b) Proxy Task II: Mean of maximum coherent group between study participants. Full Breakdown in Table 17, Appendix C.
ArXiv Pubmed Wiki Cγ=1V,̸e -0.382 ± 0.164 -0.547 ± 0.109 -0.645 ± 0.085 Cγ=2V,̸e -0.415 ± 0.168 -0.541 ± 0.135 -0.648 ± 0.100 CNPMI,̸e -0.434 ± 0.171 -0.549 ± 0.118 -0.660 ± 0.084 CNPMI -0.342 ± 0.195 -0.453 ± 0.118 -0.627 ± 0.085 CP,o -0.320 ± 0.200 -0.484 ± 0.107 -0.631 ± 0.082
CUMass,o -0.277 ± 0.172 -0.202 ± 0.126 -0.354 ± 0.053
(c) Proxy Task III: Mean of coherent group counts between study participants. For this task, stronger negative score is better as a completely coherent topic gets P3(t) = 1 and an incoherent topic gets P3(t) = 10. Hence, this proxy measure is inversely related to the coherence metric score where a larger score indicates coherence. Full Breakdown in Table 18, Appendix C.
Table 7: Average Spearman’s ρ between automated coherence metrics and respective proxy measure. The values shown are the mean correlation scores from the 8 study groups with error bars. The lemmatized version of corpus are ommitted as its values are similar to the original. CUMass,s and CP,s ommited as they are almost identical to their o variant.
spectively, indicating correlations between human judgement and some automated coherence metrics. Since most of our study participants have some science-related background, we are surprised by ArXiv’s lower correlation scores relative to Wiki in each proxy task. These results imply that our perception of coherence might be biased towards the word distribution of a generic corpus such as Wiki. Lastly, in each proxy task, the higher variances in ArXiv’s and Pubmed’s correlation scores compared to Wiki’s might imply increased subjectivity.
Inter-rater reliability (IRR). There are many factors that wil affect the variation for IRR (Belur et al., 2021). For our user study, we attempted to mitigate some of these factors. In terms of fram-
results detailed in Appendix C.
ing and education, study participants were given a short introductory primer as well as some example questions prior to starting the tasks (Appendix E). To mitigate fatigue effect, we allowed the study participants a week to work on the task, pausing and continuing at their own pace. We were not concerned about learning effect, as our presented topics spans across a plethora of themes and the correctness of the task is subjective to their own personal preference. As our objective is to poll for their beliefs, with many possible valid answers, there is not a need to review and enforce consistency between study participants.
We use Krippendorf’s α (Krippendorff, 2011) , defining pair-wise rater similarity as Jaccard distance measuring common answers between raters. We treat each wk,j,i ∈ tj,i as a multi-classification question, comprising of other words (in tj,i) and "not related" as categories, producing boolean vector representations. The mean ᾱ is 0.366 with a standard deviation of 0.04, lowest α at 0.325 and highest α at 0.464 (see Table 15, Appendix C). A completely random study response will have an α of 0.12, being significantly less than the study’s ᾱ, giving us some confidence about the reliability of the responses. Overall, considering that there are many possible combinations for each topic response, the α reported suggests some degree of similarity between different responses.
User study ablations. We examine if positioning affects position-dependent automated coherence metrics via human pair-wise agreement proxy task P4. We detail our optimizing approach in Appendix B. We define P4 as the percentage of agreement between any word-pairs wa and wb from tj,i from Ti evaluated by its corresponding Ui:
P4(wa, wb) = 1 |Ui| ∑
u∈Ui
∑
g∈Ru wa ∈ g ∧ wb ∈ g
(15)
We measure the correlation of P4(wa, wb) in a group to its pair-wise automated coherence metric score via m(wa, wb) from different orderings. Our results in Table 8 show some non-significant differences in correlation on the pair-wise level. However, that difference disappears when we evaluate the topics as a group, with the sorted and optimized variant achieving similar correlations (see Table 7). Furthermore, this difference of coherence at the pair-wise and group-wise levels, suggests that the presence of other words in the topic has an influence on the human perception of word-pair coherence. Finally, we replicate most experiments with the corpus statistics from Palmetto (Röder et al., 2015), which produced similar correlation results to Wiki.

heading: 7 Conclusion
text: Our large-scale analysis reaffirms that these automated coherence metrics are still meaningful. We are confident in using these metrics measured on generic corpus such as Wiki, and specialised corpora, Arxiv and Pubmed, for nicher tasks. Our user study empirically supports this conclusion, as our participants’ collective response correlates well to metrics measured on Wiki, albeit weaker but meaningful correlation on the specialized corpora. This work shows that popular automated coherence metrics, CNPMI , CV , and CP , are alive and well, and works regardless of lemmatization. Furthermore, we stress that the selection of the reference corpus is just as important as the selection of the metric, with Wiki being the best reference corpus that correlates with human perception of coherence. Moving forward, when evaluating for coherence aligned towards human interpretability, we recommend future topic models to be evaluated against Wiki-variants. We also recommend calculating CV with ϵ = 0, to avoid the confusion from its contradiction of other metrics at ϵ = 1e−12.

heading: Acknowledgments
text: This research/project is supported by the National Research Foundation, Singapore under its AI Singapore Programme (AISG Award No: AISG2-RP2021-020). Hady W. Lauw gratefully acknowledges the support by the Lee Kong Chian Fellowship awarded by Singapore Management University. We extend our gratitude to our user study participants for their efforts, as well as, our reviewers for their kind feedback.
Limitations
User Study. Most, if not all, of the participants are pursuing or have obtained at least a university degree/bachelor’s. While we attempted to recruit widely, majority of our participants’ education background is science-related, with strong leanings towards technology. Furthermore, we assume that our participants are proficient in English from their education level and the fact that they are based in a city that uses English as the common language. It is possible that there are some unknown common bias such as culture or knowledge that might affect the results. The tie-breaking constrain in our study, where study participants are required to assign one word to its most coherent group, might affect the correlation scores for the user study.
Corpora. The selected corpora are constructed from documents that are formal in prose, with the purpose of being informative and instructional. We do not know if the user study results are applicable to a corpus with documents that are informal in prose, such as that of a conversational nature. However, one can always evaluate topics on a large external generic corpus to determine coherence relative to human judgement.
Ethics Statement
User Study. Prior to carrying out our user study, the survey methodology was reviewed and approved by our Institutional Review Board for ethical compliance. While unlikely, we examined each question for its appropriateness. To ensure participants’ anonymity, the responses are anonymized and aggregated, and it is extremely unlikely that a participant can be identified via their response. In terms of fair compensation, we paid S$15 for each complete response of 100 questions, assuming an hour’s worth of work, it is higher than our institution’s prevailing rate for undergraduate student work. To ensure their well-being, study participants are allowed up to a week to complete the tasks, at their own preferred pace and place.
Corpora. We select corpora that have open licensing agreements that allows for non-profit academic use, and the permissions allowing us to transform and re-distribute the processed corpora as word-pair counts.

heading: A Algorithm Pseudocode
text: Pre-processing steps to reduce complexity, Algorithm 1 and Algorithm 2, remain unchanged from Yuan et al. (2022). These steps can be skipped when the graph is large and dense, such as during neg sub-graphs generation. Our modification in Algorithm 3 and Algorithm 4 introduces randomness via permutations and early stopping, when a k-clique is found in Algorithm 3 and a desired number of k-cliques found in Algorithm 4. The subgraph reduction is implemented in Algorithm 3.
Algorithm 1 PRE-CORE(G, k) Prune vertices with less than k edges from G
Input: A graph G and a positive integer k Q← ∅, F ← ∅ for u ∈ G do
if du < k − 1 then Q.push(u) F ← F ∪ {u}
end if end for while Q ̸= ∅ do
u← Q.pop() for node v ∈ neighbours Nu do
dv ← dv − 1 if dv < k − 1 ∧ v /∈ F then
F ← F ∪ {v} Q.push(v)
end if end for
end while
Algorithm 2 PRE-LIST(G,k) Find exact k-cliques and remove them from G
for each connected components C ∈ G do mc ← |E(C)|, nc ← |V (C)| if mc = (nc − 1)nc then
remove C from G output k-cliques C
end if end for
A set of connected components refers to a set of nodes where each node shares an edge with all
Algorithm 3 SDegreeList(k, R, C, G⃗) for u ∈ Permutate(C) do
if |C| ≤ l − 2 then continue end if if k < 2 then
return ∅ end if Ĉ ← N+u ∩ C if k = 2 ∧ |Ĉ| > 0 then
O ← R ∪ {u} remove (ui, uj) from G⃗ ∀ui, uj ∈ O return O
end if if |Ĉ| > l − 2 then
return SDegreeList(k − 1, R, Ĉ, G⃗) end if
end for
other nodes in the set. Finding next connected components Ĉ, requires a set intersection operation between all possible neighbours of randomly selected node u, denoted N+u , and current connected components C.
Algorithm 4 Main(G, k, target) G← PRE-CORE(G, k) G← PRE-LIST(G, k) Generate DAG G⃗ O ← ∅ for u ∈ Permutate(G⃗) do
r ← SDegreeList(k − 1, {u}, N+u , G⃗) if |r| == k then
O = O ∪ {r} end if if target == |O| then
return O end if
end for
The main algorithm gets invoked once per subgraph, we can generate multiple sub-graphs by selecting a set of words that neighbours a randomly chosen word. We then truncate the edges that do not fulfill the edge-conditions.

heading: B Optimizing Position-Based Scoring
text: Given a set of k words as a topic, our goal is to optimize the position-based score. We can reduce this problem to a weighted activity selection prob-
lem, which is equivalent to finding a max-weight independent set in an interval graph and can be solved in polynomial time (Bar-Noy et al., 2001).
Consider a word w at the jth position, index starting from 0, we can visualize the ordering as having j incoming edges, indicating precedence of other words, and k − j + 1 outgoing edges, indicating w precedence to other ensuing words. An activity will be defined by its start-time (position) and its preceding and ensuing activities. Each activity has an equal interval and the weight of the activity is determined by the difference of outgoing and incoming edges to all other words scored via m. We can transform the activities into an interval graph, with |C lj | · |C ll−j+1| combinatorial number of possible instances for each word per time slot in the schedule.
Our transformation will result in an interval graph of k disjoint graphs. While the number of activities might seem to be combinatorially explosive, selecting the first activity at T = 0, only involves k activities, and upon selection prunes multiple branches, resulting in k − j choices at T = j. Hence, we are only required to select the best activity within each disjoint graph conditioned on availability (word not selected before).

heading: C Supplementary Tables
text: This section lists tables with quantitative supplementary information.
Table 10 details the results for ArXiv and Pubmed corpus for inter-metric correlation analysis in Section 4.2.
Table 11 provides additional information on the similarity between control and treated topics for the lemmatization effect ablation in Section 5.
Table 12 provides a detailed breakdown of subgraph segments that is shortlisted for the lemmatization effect ablation in Section 5.
Table 13 details the full complete results for intercorpus correlation analysis, its partial table can be found in Table 5, Section 5.
Table 14 has additional quantitative information regarding the quantity of common topics in corpuspairs used in the inter-corpus experiments of Section 5.
Table 15 has the individual Krippendorf’s α for each user study group U for the user study in Section 6.
Tables 16, 17, 18, and 19 has the individual correlation scores of each user study group U to the
various coherence metrics for Proxy Task I, II, III, and pair-wise ablation respectively. Its averages are tabled in Tables 7a, 7b, 7c, and 8 in Section 6.

heading: D Topic Examples (User Study)
text: This set of 100 topics belongs to T1, and were shown to U1:
1. ethic humanities intellectual interdisciplinary journal philosophical scientific social society sociology 2. automate behavior check computation correct fluid limitation numerical processing specify 3. behavioral differ differentiation extent furthermore interaction neural overlap similarity trait 4. accountant archdiocese citizenship compile cultivate enlarge ferry grab interim wield 5. care educate educational engage life pandemic participation preparedness social support 6. advent anatomy enhance harmless interfere mortality psychiatrist swallow terminate urine 7. agent buy buyer maximize profit risk sell seller social utility 8. benchmark effectiveness experiment extensive indoor outdoor performance real-world synthetic validate 9. bandwidth beam conversion generation laser photon pulse pump purity silicon
10. anxiety child depression distress illness mental parent parenting social stress 11. account activity audit employment fund provision public purpose resource security 12. acidity alcoholic biochemical compete fuse insulin pathological short-term smell spontaneous 13. access communication device hardware infrastructure management resource secure technology wireless 14. bladder blood cardiac cavity congenital gastrointestinal intestinal obstruction procedure surgical 15. assess assessment company industry maturity methodology organization quality research software 16. building conditional embryonic glacial hair multiplicity overly programming questionnaire renewable 17. adoption encryption insurance job minimal native nowadays predictor resilience visit 18. continued doubling feedback growing guideline hypothetical induction pad readiness worth 19. automated detect detection measurement observation optical radar real-time sensor spacecraft 20. advent bald deficiency household liquid museum parasite physique qualify rude 21. control evaluation framework implement level monitor optimal regulation response specific 22. dose gland hormone inflammation inject muscle secretion serum stimulate toxin 23. creative family handy lie mold rank residual semantic transmission weaken 24. broad hair irregular length longitudinal mature somewhat spore tooth yellow 25. appropriate behavioral combination condition define evaluate prescribe specify substance weight 26. astronomical binary celestial galactic gravitational orbital radiation stellar telescope velocity 27. cheese dish egg fruit layer leaf meat oven rice tray 28. appropriate authority case document guideline investigation legal necessary regulation submit 29. acid biological chromosome cluster determine interact observe similarity structure visualize 30. affect concern cost development environmental provision quality relate reproductive resource 31. care health licensed medical nurse provider qualified skilled specialty technician 32. binary decomposition infinite molecule parameter possible radiation ratio sphere unstable 33. attacker contract ensure identity malicious protect protection provider trust user 34. container functionality handle item lock normal optional slot thread type 35. acquire appraisal author baby device plentiful poor sandwich schizophrenia tailor 36. cancer cause genetic immune likely malignant occur patient syndrome viral 37. concern government information legal political public regard society technology topic 38. bubble gas interstellar medium outflow shock supersonic turbulence turbulent wind 39. cool heat load plate roof rotate stack tray underneath wrap 40. attempt collapse crush escape knock push save ultimate unable unconscious 41. automate benefit health human infrastructure life online public quality user 42. academic career degree graduate medicine nursing program science student university 43. amp award consultant deliver new radio scientist staff technology visual 44. abolish administer annex autonomy dominion mandate sovereign statute territorial treaty 45. duct ear genital insert lip muscle nerve nipple tissue vagina 46. align architecture benign command embryonic legal population strange superficial team 47. historical news perspective reader recommendation researcher science summarize summary try 48. barrel bolt flame knife metal needle rod rope thread wire 49. barbecue cuisine dish grill lamb meal pork potato spicy stew 50. adverse benefit decrease efficacy long-term prevent short-term stress surgery sustain 51. aftermath avalanche blast collapse damage earthquake explosion landslide massive tsunami 52. application component design different handle process quality technique typical use
53. aim automation community document effort expert goal language machine vision 54. bring challenge engineering functionality practical protection safety threat usage vulnerability 55. abdominal anemia condition disorder liver lung pain suffer syndrome ulcer 56. book brother child early finally fine originally piano queen sir 57. attacker choose client cost decision game maximize objective selfish strategy 58. accept associate book early inscription middle parish queen seven valuable 59. build business company engineering intelligence methodology practitioner predictive student tool 60. atmospheric barrier conventional electron interference internal layer noise radiation thermal 61. act allow ban discrimination government legislation permit prevent refusal removal 62. chassis conventional diesel driver fit gear manual maximum speed vehicle 63. accelerator advance advanced facility offer optic physics promise science versatile 64. abdominal abnormality blood cardiovascular diagnostic gastrointestinal pain respiratory surgery tissue 65. argument civilization critique emphasize idea knowledge linguistic phenomenon religious understanding 66. behavioral institutional intervention nurse occupational practitioner prevention provider rehabilitation specialist 67. apt bother bounce catalog excuse portrayal respectable royalty smoke strive 68. drug fever lung paralysis polio prevent recover recovery suffer victim 69. apologize honest quote remark respond sad smile surprised tell truth 70. adversary broadcast internet node protocol route send service traffic transmission 71. expert health participant peer people preference public receive share topic 72. design enable equipment output package provide quality tool validate verification 73. atom decomposition determine energy fluid mechanism observe phase ratio substrate 74. contain core critical date distinct effectively hard mercury method true 75. billion corporate equity finance financial invest investor portfolio retail telecom 76. liver lung medication metabolism reduce renal respiratory secretion toxic urine 77. amphitheater bog combustion construction install lowering parachute populous successive youthful 78. automate detection electronic equipment measurement optical retrieval scan signal spacecraft 79. bread fry meat menu onion pie pizza potato specialty vegetable 80. broad irregular measure slight specimen spherical spore texture tip typical 81. acknowledge astronomy baseline chapter climate economics explosion movement prize thing 82. definition french industrial micro percentage post purity spot superior supplement 83. advance communication computing development device industry platform promising sensor thing 84. clean drink flush fresh kitchen pipe recycle supply wash waste 85. algorithm bit detect fast feedback hardware implementation minimize mode slow 86. characteristic characterize chemical condition diagnostic essential organism plasma precise understanding 87. adverse brain complication induce muscle pain pregnancy sleep spontaneous surgical 88. aesthetic criticism interpretation introduction lecture philosophy psychology study theoretical thesis 89. algorithm arithmetic binary cpu logic manipulate output processing processor programmer 90. application autonomous capability computing delivery modern networking resource smart software 91. final finish goal injury preseason raider regular score season squad 92. application capability desktop enable encryption hardware networking software technology wireless 93. asleep bed morning notice sleep sneak wake walk watch worry 94. advance analysis clinical develop high-quality method objective patient provide tool 95. care health healthy nurse quarantine sanitary sanitation surgeon vaccination veterinary 96. affordable availability development device hardware internet mobile need platform software 97. application automate component display install integrate menu monitor server window 98. aspect auditory behaviour emotional interaction learner psychology relate researcher understand 99. advantage allow collaboration collaborative construction facilitate open opportunity platform sharing
100. advantage analog camera card compatible converter modular processor storage use

heading: E User Study Instructions
text: E.1 Primer on Task
Evaluating the relations between words from a computational lens serves to further the research and understanding of artificial intelligence linguistic research.
A group of words can be considered coherent if they share a similar theme. For example, the group "apples banana coconut durian" can be considered coherent as most people would identify "fruit", "food" or "tree" as the common theme or link.
However, some group of words might be more ambiguous and the common theme might not be as straightforward. For example, "trees ore corn hydrogen" might be considered incoherent to some, while others might identify the common theme as "resources".
Ultimately, it is up to one’s personal preferences and experiences to decide on whether a group of words are coherent.
E.2 Task Instructions
You will be presented with 10 English words. These words belongs to the 20,000 most frequently used words, so it is unlikely that you will encounter strange words. If you do encounter words that you have never seen before, you are free to use a dictionary or search engine (e.g. Google).
You will then be asked to assign each word to groups, where each group contains words that you think are coherent when grouped together.
Given an example: alcohol athlete breakfast drink eat habit intake meal obesity sleep Some might divide the words into two groups identifying Group 1 is "alcoholic"-themed and Group 2
is "healthy"-themed.
Group 1 Group 2 Group 3 Group 4 Not Related alcohol O athlete O
breakfast O drink O eat O
habit O intake O meal O
obesity O sleep O
In another example given: atom calcium component material reduction temperature titanium typical weight yield
Some might group most of the words as "chemistry"-themed.
Group 1 Group 2 Group 3 Group 4 Not Related atom O
calcium O component O
material O reduction O
temperature O titanium O typical O weight O yield O
If you believe that certain word(s) do not belong in any group, select the "Not Related" option in the last column. There can be multiple words that are not related to each other.
For example: animal bed carrot fungible great osmosis paradise star telcommunication water
Group 1 Group 2 Group 3 Group 4 Not Related animal O
bed O carrot O fungible O great O osmosis O paradise O
star O telcommunication O
water O
We want to emphasise that there are no right or wrong answers for the tasks, we wish to capture your beliefs on what you think is "correct". We understand that at times, you might encounter words that belong to multiple groups, however to simplify the tasks, we ask that you be the tiebreaker and assign it to the word-group with the strongest similarity.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Limitations
A2. Did you discuss any potential risks of your work? Not applicable. Left blank.
3 A3. Do the abstract and introduction summarize the paper’s main claims? Abstract, 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? 3, 4
3 B1. Did you cite the creators of artifacts you used? 3 - cited 4 - ours
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Ethics
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Ethics
3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Ethics
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? 3, Limitations, Ethics
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. 3, 4, 5, 6, Appendix C, D
C 3 Did you run computational experiments? 4, 5, 6
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Appendix A
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? 3, Appendix C
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? 4,5,6
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? 3
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? 6
3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Appendix E, Institute Review Board application withheld for anonymity
3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Ethics
3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Ethics
3 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Ethics
3 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Ethics

heading: Future Work
text: Moving forward, when evaluating for coherence aligned towards human interpretability, we recommend future topic models to be evaluated against Wiki-variants. We also recommend calculating CV with ϵ = 0, to avoid the confusion from its contradiction of other metrics at ϵ = 1e−12