SlowGuess/NAACL_2021_OCR huggingface dataset:
The SMEs were asked to rate on a scale of 1-3 how difficult it was to label each  $T_{dc}$ , where 1 was difficult, 3 was easy, and 0 was where a label could be assigned. This qualitative statistic is denoted  $Q_{dif}$ . The researchers then scored, on a scale of 1-5, the degree of alignment between topic labels and the labels assigned to their corresponding collections. A score of 5 indicated the labels were identical, and a score of 0 indicated the  $T_{ws}$  and/or  $T_{dc}$  was incoherent. This statistic is denoted  $Q_{aln}$ . Examples of these tasks are in Appendix A.
3.5 Statistical Tests
We measure the strength of the association between variables using Pearson's  $r$  correlation coefficient in evaluation 1 (see section 4.1) and Spearman's  $\rho$  correlation coefficient in evaluations 2-5 (see sections 4.2, 4.3, 4.4, and 4.5). Pearson's  $r$  is used in the few papers that evaluate coherence scores over the same datasets (Röder et al., 2015; Lau et al., 2014b). The practical reason for using Pearson's  $r$  for our evaluation of proposition 1 was to make valid comparisons with these studies. The statistical justification for using Pearson's  $r$  (rather than Spearman's  $\rho$ ) is that the datasets are continuous (neither is ordinal, as Spearman's  $\rho$  requires) and believed to have a bivariate normal distribution. Spearman's  $\rho$  is only appropriate when the relationship between variables is monotonic, which has not been consistently demonstrated for coherence (Röder et al., 2015; Bovens and Hartmann, 2004). Spearman's  $\rho$  is appropriate to assess the association between coherence scores and human judgments in evaluations 2-5. It is a preferred method
for such tasks(Aletras and Stevenson, 2013; Newman et al., 2010a) as it is unaffected by variability in the range for each dataset (Lau et al., 2014b).
Here we detail the results of our analysis of the five propositions about interpretability evaluation.
4.1 Evaluation 1: Coherence Measure Correlations
As per proposition 1, coherence measures should be robust and highly correlated. To test this proposition, we conducted a Pearson's correlation analysis of paired coherence measures calculated for  $K = 10 - 60$  for each model-dataset combination. Pooling the results for  $K$  and the three datasets, we calculate the  $\overline{x}_r$  for LDA and MetaLDA.
$C_{NPMI}$  and  $C_P$  scores were strongly correlated for all datasets. Ranging from  $\overline{x}_r = 0.779 - 0.902$  for LDA, and  $\overline{x}_r = 0.770 - 0.940$  for MetaLDA.  $C_{NPMI}$  and  $C_{NPMI-ABC}$  also showed a moderate-to-strong correlation for all datasets with LDA ranging from  $\overline{x}_r = 0.719 - 0.769$ , and MetaLDA from  $\overline{x}_r = 0.606 - 0.716$ .  $C_{NPMI-ABC}$  appears more sensitive to changes in  $K$  than  $C_P$ . No significant trends were seen between other coherence measures calculated for any dataset. These results are reported in Appendix B.
Methods to aggregate coherence scores may mask any differences in the models' behaviors as  $K$  increases. To test this, aggregate coherence measures, typical of the empirical evaluation of topic models, were calculated per value of  $K$ . These were the mean of all topics (Average), the mean for all topics weighted by the topic proportion (WeightedAverage), and the mean of the Top-N percent of ranked topics by coherence score (Top-Npcnt), where  $N = \{25, 50, 80\}$ .
Both models showed trends in aggregated coherence scores calculated on the AP dataset. As shown in Figure 1, the peak for each measure varies according to different values of  $K$  and between models. For instance, aggregates of both models  $C_{NPMI}$  and  $C_{NPMI-ABC}$  peak at 60 and 10 topics, respectively. However,  $C_V$  aggregate peaks are completely divergent between models,  $K = 200$  for MetaLDA and  $K = 50$  for LDA. Indeed, the two models favored different coherence measures and aggregate methods. Generally, MetaLDA exhibits superior performance across all aggregates for  $C_V$  and  $C_A$ , while LDA is superior for  $C_{Umass}$ . No
<sup>7</sup>We confirmed this with a Kolmogorov-Smirnov test for normality on the coherence scores. <sup>8</sup> Although Kendall's  $\tau$  has been used for similar evaluations (Rosner et al., 2013), it is unreliable when the range of each dataset varies significantly as in these experiments (Sanderson and Soboroff, 2007).
Figure 1: Comparison of LDA (triangle) and MetaLDA (circle) aggregated coherence scores for the AP dataset. Scores are shown on the y-axis, and  $K$  is shown on the x-axis. Individual points are averaged across five runs, where the typical sample standard is 0.005, but up to 0.010 for  $K = 20$ .
tably, MetaLDA shows superior  $C_{NPMI}$ ,  $C_{NPMI-ABC}$ ,  $C_{NPMI-AP}$  scores for Top20pcnt, Top50pcnt, and Top80pcnt aggregations, but is inferior when the full average of these scores is calculated. Other datasets are broadly similar and shown in Appendix B.
We also compare MetaLDA with LDA. Pooling the results for  $K = 10 - 200$  for each of the four datasets, we get a set of differences in the scores and compute the  $p$ -value for a one-sided student  $t$ -test to determine whether LDA has higher average coherence scores than MetaLDA. MetaLDA yields significantly higher  $C_{NPMI}$  scores calculated using the Top20pcnt ( $p < 0.01$ ) and Top50pcnt of topics ( $p < 0.05$ ). Conversely, LDA yields significantly higher  $C_{NPMI}$  scores for the other aggregates ( $p < 0.01$ ). Except for the full average, MetaLDA achieves significantly higher ( $p < 0.01$ )  $C_{NPMI-ABC}$ ,  $C_{NPMI-AP}$ , and  $C_V$  scores than LDA for the other aggregate methods.
Disturbingly, the "best" models, or optimal  $K$  varies depending on the coherence measure and the aggregate measure used to calculate it. This has implications for topic model selection in applied settings, where coherence is used to inform  $K$  (Kirilenko et al., 2021). When repeating the analysis using different  $K$ , a second trend emerges: Met-
aLDA significantly outperforms LDA in  $C_{NPMI}$  for smaller  $K$  on average but loses out for larger  $K$ . Results from our qualitative analysis confirmed this occurred because LDA had many less frequent topics (e.g., when  $K = 60$ , all topics occur about 1/60 of the time), unlike MetaLDA, which mixes more and less frequent topics.
4.2 Evaluation 2: Labeling Topic Words-sets
Proposition 2 states that if topics can be labeled they are interpretable. Coherence as a measure of interpretability should then be predictive of topics that can be labeled. To evaluate this proposition, a Spearman's  $\rho$  correlation coefficient was used to assess the relationship between coherence measures and the number of raters able to label the  $T_{ws}$ ,  $Q_{nbr}$ , for each of the 130 topics produced per model-dataset combination. These results are available in Appendix C. There was no significant correlation between any coherence measure and  $Q_{nbr}$ . Interestingly, the SMEs reported several topics they could not label despite their high coherence scores. For instance, the LDA modeled topic "red, wear, flag, blue, gold, black, tape, tie, green, iron" could not be labeled despite being the  $9^{\text{th}}/60$  highest ranked topic for  $C_{NPMI}$ .
4.3 Evaluation 3: Topic Label Agreement
Proposition 3 states an interpretable topic is one where there is high agreement between annotators on its label. As such, coherence should align to measures of consensus or agreement. To evaluate this proposition, we calculate the gold-standard ICR measures, Fleiss' kappa  $(\kappa)$  (Fleiss, 1971) and Krippendorff's alpha  $(\alpha)$  (Krippendorff, 2004). Both allow for multiple coders and produce a chance-corrected estimate of ICR but do not facilitate the isolation of low-agreement topics. For this, we also calculated the Percentage Agreement  $Q_{agr}$  for each topic, as shown in Appendix D.
Generally,  $\alpha$ ,  $\kappa$ , and  $Q_{agr}$  improved as  $K$  increased. As shown in Table 1, LDA consistently outperformed MetaLDA when  $K = 60$  across all three datasets and generally attained higher  $\alpha$ ,  $\kappa$ , and  $Q_{agr}$  scores than MetaLDA. There was a moderate-to-strong agreement between SMEs, a reliable result for an open labeling task (Landis and Koch, 1977). However, the performance of each model was notably affected by the datasets. LDA outperformed MetaLDA on the AP dataset across all three measures except for  $\kappa$  when  $K = 20$ , and for  $Q_{agr}$  when  $K = 10$ . Except for  $\kappa$  when  $K = 40$ ,
MetaLDA achieved higher or comparable scores to LDA on the AWH dataset when  $K = 20 - 40$ , but outperformed LDA only when  $K = 10 - 20$  on the AWM dataset.
Kripp. α	Fliess κ	Pcnt. Qagr
LDA	Meta	LDA	Meta	LDA	Meta
AP	0.584	0.486	0.578	0.485	0.503	0.492
AWH	0.512	0.498	0.527	0.515	0.439	0.411
AWM	0.513	0.447	0.535	0.492	0.428	0.369
Spearman's  $\rho$  was calculated to measure the strength of the relationship between  $Q_{agr}$  and the generated coherence measures. As shown in Appendix D, results were random with no significant correlations. As shown in Table 2, there was a statistically significant correlation between  $Q_{agr}$  and  $Q_{nbr}$  when  $K = 60$ .
Table 1: Krippendorff's  $\alpha$ , Fleiss'  $\kappa$ , and  $Q_{agr}$  ICR statistics for topic labeling when  $K = 60$ .
LDA	10	20	40	60
ρ	p	ρ	p	ρ	p	ρ	p
AP	0.240	0.504	0.548	0.012	0.431	<0.01	0.475	<0.01
AWH	0.000	0.000	0.458	0.042	0.561	<0.01	0.644	<0.01
AWM	0.506	0.136	0.345	0.136	0.490	<0.01	0.697	<0.01
Meta LDA	10	20	40	60
ρ	p	ρ	p	ρ	p	ρ	p
AP	0.544	0.104	0.147	0.535	0.445	<0.01	0.690	<0.01
AWH	0.532	0.113	0.478	0.033	0.147	0.366	0.629	<0.01
AWM	0.548	0.101	0.414	0.069	0.743	<0.01	0.700	<0.01
Table 2: The Spearman's  $\rho$  correlation coefficients for pairwise combinations of  ${Q}_{agr}$  and  ${Q}_{nbr}$  for all learned models.
Coherence measures did not correlate with  $Q_{agr}$ , and in some cases, were contradictory. For example,  $Q_{agr}$  generally increases with  $K$  (and our experts reported that labeling was often easier for smaller topics), but coherence measures such as  $C_A$  and  $C_{NPMI-ABC}$  tended to decrease (in Figure 1). These results show that the two models show different sensitivities to dataset preparation and the value of  $K$ .
4.4 Evaluation 4: Labeling Topic Document-collections
Proposition 4 states that topics that are interpretable have a  $T_{dc}$  that is easily labeled. To evaluate this proposition, a Spearman's  $\rho$  was used to assess the relationship between coherence measures and SME ratings of  $T_{dc}$  labeling difficulty,  $Q_{dif}$ . The full set, Top25pcnt, top50pcnt, and bottom 15% (Bot15pcnt) of ranked  $Q_{dif}$  scores were analyzed. The only notable correlation was between the Bot15pcnt of LDA  $T_{dc}$  for CNPMI-ABC  $(\rho = -0.817$ ,
$p = < 0.01)$  . Interestingly, when ranked by topic diagnostic  $D_{ew}$  , the Top25pcnt and Top50pcnt of  $T_{dc}s$  showed moderate correlation with  $Q_{dif}$  for MetaLDA  $(\rho = -0.764,p <   0.01;\rho = -0.630,p <   0.01)$
A repeat analysis with topic diagnostic  $D_{tp}$  did not yield any statistically significant results. However, we observed that for  $T_{dc}$ s produced by MetaLDA, the three largest and three smallest topics could not be labeled. By contrast, the LDA  $T_{dc}$ s that were not interpretable were from the smallest 20% of topics. We hypothesize that this distinction results from MetaLDA's broadly distributed  $D_{tp}$  (0.017±0.155), which features several very large and very small topics. By comparison, LDA  $D_{tp}$  is approximately uniformly distributed (0.017 ± 0.001).
4.5 Evaluation 5: Topic Label Alignment
Proposition 5 states that an interpretable topic is one that is descriptive of the  $T_{dc}$ . To test this proposition, we constructed an alignment score  $Q_{aln}$  which rate the similarity between the standardized topic label from  $T_{ws}$  and the label from  $T_{dc}$ . Similar to the evaluation of Proposition 4, we conducted a Spearman's  $\rho$  to test for a relationship between  $Q_{aln}$ , coherence measures, and diagnostic scores.
The following illustrates a high scoring, but poorly aligned topic with a  $C_{NPMI}$  of 0.073.  $T_{ws}$ : "law, bill, power, gun, democracy, control, freedom, rule, protect, legislation" was labeled "Gun control", but the  $T_{dc}$  was labeled "Foreign Interference Act". Appendix F contains additional examples.
LDA showed a strong relationship between  $Q_{\text{alin}}$  and  $C_{\text{NPMI-ABC}}$  for the Top25pcnt of topics ( $\rho = 0.825, p < 0.01$ ), but the relationship was weak for other coherence measures. No coherence measures were correlated with MetaLDA  $Q_{\text{alin}}$  scores.
As per section 4.4, we repeated the analysis by ranking topics by  $D_{ew}$ . MetaLDA showed a strong-to-moderate correlation between  $D_{ew}$  and  $Q_{aln}$  for the Top25pcnt ( $\rho = -0.776$ ,  $p = < 0.01$ ), Top50pcnt ( $\rho = -0.646$ ,  $p < 0.01$ ), and Bot15pcnt ( $\rho = 0.693$ ,  $p = 0.039$ ) of topics, making  $D_{ew}$  a potentially useful proxy for alignment for MetaLDA.
We repeated the work of Zhao et al. (2017), who demonstrated that when the top-ranked topics by  $C_{NPMI}$  are considered, MetaLDA produces higher  $C_{NPMI}$  scores than LDA. However, when  $C_{NPMI}$  was measured using alternative aggregate methods, we
discovered that LDA outperformed MetaLDA. This is likely to be because the smaller topics in MetaLDA can be effectively ignored or scrapped, while in LDA, all topics are of comparable size and are used by the model. Other non-parametric topic models are belived to behave similarly. While MetaLDA generated higher  $C_{NPMI-ABC}$  scores than LDA for all aggregates, it was highly dependent on dataset heterogeneity and the value of  $K$ . This should indicate that MetaLDA is more adaptive to specialized language, an effect expected in other topic models supported by word embeddings.
The comparative performance of coherence measures can vary significantly depending on the aggregate calculation method used and the way the data has been prepared. This latter point has been well established in the literature, most notably for Twitter data (Symeonidis et al., 2018), but is often overlooked when evaluating novel topic models. This is a cause for concern, given the growing reliance on coherence measures to select the optimal model or  $K$  in applied settings (Xue et al., 2020; Lyu and Luli, 2021).
Propositions 2 and 3 addressed  $T_{ws}$  interpretability. We have demonstrated the difference between comprehending a topic and providing a topic label that is both informative and reliable. However, coherence measures may not be informative of these qualities. Propositions 4 and 5 addressed  $T_{dc}$  interpretability. We have demonstrated that the ease of labeling a  $T_{dc}$  and the alignment between  $T_{ws}$  and  $T_{dc}$  does not correlate with coherence measures. Additionally, we identified several areas for future research into the use of diagnostic statistics in applied settings. We observed unexpected behaviors in the distributions of  $D_{ew}$  and  $D_{tp}$  after a comparative analysis between LDA and the non-parametric model MetaLDA, affecting the interpretability of both  $T_{ws}$  and  $T_{dc}$ . Correlations between  $Q_{dif} / Q_{aln}$  and  $D_{ew} / D_{tp}$  for MetaLDA, for example, indicate that these topic diagnostics could assist in evaluating  $T_{dc}$  interpretability.
We have shown that coherence measures can be unreliable for evaluating topic models for specialized collections like Twitter data. We claim this is because the target of "interpretability" is ambiguous, compromising the validity of both automatic and
human evaluation methods<sup>9</sup>.
Due to the advancements in topic models, coherence measures designed for older models and more general datasets may be incompatible with newer models and more specific datasets. Our experiments show that non-parametric models, such as MetaLDA, which employs embeddings to improve support for short-texts, behaves differently to LDA for these performance and diagnostic measures. This is critical because recent research has focused on sophisticated deep neural topic models (Zhao et al., 2021), which make tracing and predicting behaviors more challenging. Abstractly, we may compare the use of coherence measures in topic modeling to the use of BLEU in machine translation. Both lack the finesse necessary for a complete evaluation, as is now the case with BLEU (Song et al., 2013).
Additionally, our study demonstrated that an examination of the  $T_{dc}$  could provide greater insights into topic model behaviors and explained many of the observed problems. We argue for the representation of topics as a combination of thematically related  $T_{dc}$  and  $T_{ws}$ , and the further adoption of empirical evaluation using specialized datasets and consideration of  $T_{dc}$  interpretability. To date, few papers have attempted this combination (Korenčić et al., 2018).
However, we believe coherence measures and automated labeling techniques will continue to play a critical role in applied topic modeling. Contextually relevant measures like  $C_{NPMI-ABC}$  and topic diagnostics like  $D_{ew}$  can be key indicators of interpretability. Aside from the empirical evaluation of novel topic models, new automated labeling techniques, having proven themselves useful for labeling  $T_{tw}$ , should be extended for  $T_{dc}$ .
We thank Callum Waugh, Elliot Freeman and Elizabeth Daniels for conducting the expert annotations. We also thank Henry Linger for providing feedback on earlier drafts. The first author discloses the following financial support for the research: an Australian Government Research Training Program (RTP) Stipend and RTP Fee-Offset Scholarship, and an Australian Government Defence Science and Technology Group Research scholarship.
${}^{9}$  Specifically,construct validity,which confirms if the operational definition of a variable (interpretability) reflects the true theoretical meaning of a concept (O’Leary-Kelly and Vokurka, 1998).
Ethics and Impact
This project has been reviewed and approved by the Monash University Human Research Committee (Project ID: 18167), subject to abidance with legislated data use and protection protocols. In particular, the Twitter Inc. developers policy prohibits the further distribution of collected tweets and associated metadata by the authors group, with the exception of tweet IDs which may be distributed and re-hydrated. The subject matter of the tweets collected is Australian Politics. We have forgone the use of material included in the paper that would be offensive or problematic to marginalized groups in the Australian political context.
A. Agrawal, W. Fu, and T. Menzies. 2018. What is wrong with topic modeling? And how to fix it using search-based software engineering. Information and Software Technology, 98:74-88.
N. Aletras, T. Baldwin, J. H. Lau, and M. Stevenson. 2017. Evaluating topic representations for exploring document collections. Journal of the Association for Information Science and Technology, 68(1):154-167.
N. Aletras, J. H. Lau, T. Baldwin, and M. Stevenson. 2015. TM 2015-topic models: Post-processing and applications workshop. In Proc. of the 24th ACM Int. on Conf. on Information and Knowledge Management (CIKM), pages 1953-1954.
N. Aletras and M. Stevenson. 2013. Evaluating topic coherence using distributional semantics. In Proc. of the 10th Int. Conf. on Computational Semantics (IWCS), pages 13-22.
A. Alokaili, N. Aletras, and M. Stevenson. 2019. Re-ranking words to improve interpretability of automatically generated topics. In Proc. of the 13th Int. Conf. on Computational Semantics (IWCS), pages 43-54, Gothenburg, Sweden. ACL.
D. AlSumait, L.and Barbara and C. Domeniconi. 2008. On-line LDA: Adaptive topic models for mining text streams with applications to topic detection and tracking. In Proc. of the 8th IEEE Int. Conf. on Data Mining (ICDM)), pages 3-12. IEEE.
R.-K. Ando and L. Lee. 2001. Iterative residual rescaling. In Proc. of the 24th annual int. ACM SIGIR conf. on Research and development (SIGIR), SIGIR '01, pages 154-162, New York, NY, USA. ACM.
M. Andreotta, R. Nugroho, M. Hurlstone, F. Boschetti, S. Farrell, I. Walker, and C. Paris. 2019. Analyzing social media data: A mixed-methods framework combining computational and qualitative text analysis. Behavior research methods, 51(4):1766-1781.
C. Arnold, A. Oh, S. Chen, and W. Speier. 2016. Evaluating topic model interpretability from a primary care physician perspective. Computer Methods and Programs in Biomedicine, 124:67-75.
E. Baumer, D. Mimno, S. Guha, E. Quan, and G. Gay. 2017. Comparing grounded theory and topic modeling: Extreme divergence or unlikely convergence? Journal of the Association for Information Science and Technology, 68(6):1397-1410.
S. Bhatia, J. H. Lau, and T. Baldwin. 2017. An automatic approach for document-level topic model evaluation. In Proc. of the 21st Conf. on Computational Natural Language Learning (CoNLL), pages 206-215.
S. Bhatia, J. H. Lau, and T. Baldwin. 2018. Topic intrusion for automatic topic model evaluation. In Proc. of the 2018 Conf. on Empirical Methods in Natural Language Processing (EMNLP), pages 844-849.
S J. Blair, Y. Bi, and Maurice D Mulvenna. 2020. Aggregated topic models for increasing social media topic coherence. Applied Intelligence, 50(1):138-156.
D. Blei and J. Lafferty. 2007. A correlated topic model of science. The Annals of Applied Statistics, 1(1):17-35.
D. Blei, A. Ng, and M. Jordan. 2003. Latent Dirichlet allocation. Journal of Machine Learning Research, 3(Jan):993-1022.
L. Bovens and S. Hartmann. 2004. Bayesian Epistemology. Oxford University Press.
J. Boyd-Graber and D. Blei. 2007. PUTOP: Turning predominant senses into a topic model for word sense disambiguation. In 4th SemEval-2007, pages 277-281.
J. Boyd-Graber, D. Mimno, and D. Newman. 2015. Care and feeding of topic models: Problems, diagnostics, and improvements. In Edoardo M. Airoldi, David Blei, Elena A. Erosheva, and Stephen E. Fienberg, editors, Handbook of Mixed Membership Models and Their Applications. CRC Press Boca Raton, FL.
N. Brown. 2019. Methodological cyborg as black feminist technology: constructing the social self using computational digital autoethnography and social media. Cultural Studies  $\leftrightarrow$  Critical Methodologies, 19(1):55-67.
J. Chang, S. Gerrish, C. Wang, J. Boyd-Graber, and D. Blei. 2009. Reading tea leaves: How humans interpret topic models. In Proc. of the 23rd Annu. Conf. Neural Information Processing Systems (NeurIPS), pages 288-296.
W. Chen, J. Wang, Y. Zhang, H. Yan, and X. Li. 2015. User based aggregation for bitterm topic model. In Proc. of the 53rd Annu. Meeting of the Association