SlowGuess/NAACL_2021_OCR huggingface dataset:
for Computational Linguistics and the 7th Int. Joint Conf. on Natural Language (ACL-IJCNL), pages 489-494.
X. Cheng, X. Yan, Y. Lan, and J. Guo. 2014. BTM: Topic modeling over short texts. IEEE Transactions on Knowledge and Data Engineering, 26(12):2928-2941.
C. Doogan, W. Buntine, H. Linger, and S. Brunt. 2020. Public perceptions and attitudes toward COVID-19 nonpharmaceutical interventions across six countries: A topic modeling analysis of twitter data. Journal of Medical Internet Research, 22(9).
D. Dunning, K. Johnson, J. Ehrlinger, and J. Kruger. 2003. Why people fail to recognize their own incompetence. Current directions in psychological science, 12(3):83-87.
M. Eickhoff and R. Wieneke. 2018. Understanding topic models in context: A mixed-methods approach to the meaningful analysis of large document collections. In Proc. of the 51st Hawaii Int. Conf. on System Sciences (HICSS), pages 903-912.
J. Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5):378-382.
T. Hecking and L. Leydesdorff. 2019. Can topic models be used in research evaluations? reproducibility, validity, and reliability when compared with semantic maps. Research Evaluations, 28(3):263-272.
K. Hui. 2001. Automatic Topic Detection From News Stories. Ph.D. thesis, CUHK.
I. Kagashe, Z. Yan, and I. Suheryani. 2017. Enhancing seasonal influenza surveillance: topic analysis of widely used medicinal drugs using Twitter data. Journal of Medical Internet Research, 19(9):e315.
A. Karami, A. Dahl, G. Turner-McGrievy, H. Kharrazi, and G. Shaw Jr. 2018. Characterizing diabetes, diet, exercise, and obesity comments on Twitter. International Journal of Information Management, 38(1):1-6.
A. Karami, M. Lundy, F. Webb, and Y. Dwivedi. 2020. Twitter and research: a systematic literature review through text mining. IEEE Access, 8:67698-67717.
E. Kim, Y. Jeong, Y. Kim, K. Kang, and M. Song. 2016. Topic-based content and sentiment analysis of ebola virus on Twitter and in the news. Journal Information Science, 42(6):763-781.
A. P. Kirilenko, S. O. Stepchenkova, and X. Dai. 2021. Automated topic modeling of tourist reviews: Does the Anna Karenina principle apply? Tourism Management, 83:104241.
D. Korencic, S. Ristov, and J. Snajder. 2018. Document-based topic coherence measures for news media text. Expert Systems With Applications, 114:357-373.
K. Krippendorff. 2004. Measuring the reliability of qualitative text analysis data. *Quality and quantity*, 38:787-800.
J. Landis and G. Koch. 1977. The measurement of observer agreement for categorical data. Biometrics, pages 159-174.
J. H. Lau and T. Baldwin. 2016. The sensitivity of topic coherence evaluation to topic cardinality. In Proc. 17th Conf. North American Chapter of the Association for Computational Linguistics: Human Language Technologies (HLT-NAACL), pages 483-487.
J. H. Lau, P. Cook, D. McCarthy, S. Gella, and T. Baldwin. 2014a. Learning word sense distributions, detecting unattested senses and identifying novel senses using topic models. In Proc. of the 52nd Annual Meeting of the Association for Computational Linguistics (ACL), pages 259-270.
J. H. Lau, K. Grieser, D. Newman, and T. Baldwin. 2011. Automatic labelling of topic models. In Proc. of the 49th Conf. on Human Language Technologies (HLT), pages 1536-1545, Portland, Oregon, USA. ACL.
J. H. Lau, D. Newman, and T. Baldwin. 2014b. Machine reading tea leaves: Automatically evaluating topic coherence and topic model quality. In Proc. of the 9th Conf. European Chapter of the Association for Computational Linguistics (EACL), pages 530-539.
X. Li, C. Li, J. Chi, and J. Ouyang. 2018. Short text topic modeling by exploring original documents. Knowledge and Information Systems, 56(2):443-462.
Kar Wai Lim, Changyou Chen, and Wray L. Buntine. 2013. Twitter-Network Topic Model: A full Bayesian treatment for social network and text modeling. In Proc. of the 27th Annu. Conf. on Neural Information Processing Systems (NeurIPS): Topic Models Workshop, NeurIPS Workshop 2013, pages 1-5.
Z. C. Lipton. 2018. The mythos of model interpretability. Communications of the ACM, 61(10):36-43.
J. Lund, P. Armstrong, W. Fearn, S. Cowley, C. Byun, J. Boyd-Graber, and K. Seppi. 2019. Automatic evaluation of local topic quality. In Proc. of the 57th Annual Meeting of the Association for Computational Linguistics (ACL), pages 788-796.
J. Lyu and G. Luli. 2021. Understanding the public discussion about the centers for disease control and prevention during the covid-19 pandemic using twitter data: Text mining analysis study. Journal of Medical Internet Research, 23(2):e25108.
D. Maier, A. Waldherr, P. Miltner, G. Wiedemann, A. Niekler, A. Keinert, B. Pfetsch, G. Heyer, U. Reber, and T. Häussler. 2018. Applying LDA topic modeling in communication research: Toward a
valid and reliable methodology. Communication Methods and Measures, 12(2-3):93-118.
A. McCallum. 2002. Mallet: A machine learning for language toolkit.
D. Mimno, H. Wallach, E. Talley, M. Leenders, and A. McCallum. 2011. Optimizing semantic coherence in topic models. In Proc. of the 2011 Conf. on Empirical Methods in Natural Language Processing (EMNLP), EMNLP '11, pages 262-272, USA. ACL.
F. Morstatter and H. Liu. 2017. In search of coherence and consensus: Measuring the interpretability of statistical topics. Journal of Machine Learning Research, 18(1):6177-6208.
F. Morstatter, Y. Shao, A. Galstyan, and S. Karunasekera. 2018. From alt-right to alt-rechts: Twitter analysis of the 2017 German federal election. In Proc. of the 2018 World Wide Web Conference (TheWebConf), pages 621-628. IW3C2.
A. Nerghes and J. S. Lee. 2019. Narratives of the refugee crisis: A comparative study of mainstream media and Twitter. Media and Communication, 7(2):275-288.
D. Newman, E. Bonilla, and W. Buntine. 2011. Improving topic coherence with regularized topic models. In Proc. of the 25th Annu. Conf. on Neural Information Processing Systems (NeurIPS), pages 496-504.
D. Newman, J. H. Lau, K. Grieser, and T. Baldwin. 2010a. Automatic evaluation of topic coherence. In Proc. of the 11th Conf. North American Chapter of the Association for Computational Linguistics: Human Language Technologies (HLT-NAACL), pages 100-108. ACL.
D. Newman, Y. Noh, E. Talley, S. Karimi, and T. Baldwin. 2010b. Evaluating topic models for digital libraries. In Proc. of the 10th ACM/IEEE-CS Joint Conf. on Digital Libraries, (JCDL), pages 215-224.
D. O'Callaghan, D. Greene, J. Carthy, and P. Cunningham. 2015. An analysis of the coherence of descriptors in topic modeling. *Expert Systems With Applications*, 42(13):5645-5657.
S. O'Leary-Kelly and R. Vokurka. 1998. The empirical assessment of construct validity. Journal of Operations Management, 16(4):387-405.
J. Pennington, R. Socher, and C. Manning. 2014. GloVe: Global vectors for word representation. In Proc. of the 2014 Conf. on Empirical Methods in Natural Language Processing (EMNLP), pages 1532-1543, Doha, Qatar. ACL.
N. Ramrakhiyani, S. Pawar, S. Hingmire, and G. Palshikar. 2017. Measuring topic coherence through optimal word buckets. In Proc. of the 15th Conf. European Chapter of the Association for Computational Linguistics (EACL), pages 437-442.
M. Röder, A. Both, and A. Hinneburg. 2015. Exploring the space of topic coherence measures. In Proc. of the 8th ACM Int. Conf. on Web Search and Data Mining (WSDM), pages 399-408.
F. Rosner, A. Hinneburg, M. Roder, M. Nettling, and A. Both. 2013. Evaluating topic coherence measures. In Proc. 27th Annu. Conf. on Neural Information Processing Systems (NeurIPS), pages 1-4.
M. Sanderson and I. Soboroff. 2007. Problems with kendall's tau. In Proc. of the 30th annual int. ACM SIGIR conf. on Research and development (SIGIR), pages 839-840.
L. Sinnenberg, A. Buttenheim, K. Padrez, C. Mancheno, L. Ungar, and R. Merchant. 2017. Twitter as a tool for health research: a systematic review. American Journal of Public Health, 107(1):e1-e8.
X. Song, T. Cohn, and L. Specia. 2013. BLEU deconstructed: Designing a better MT evaluation metric. International Journal of Computational Intelligence and Applications, 4(2):29-44.
X. Sun, X. Liu, B. Li, Y. Duan, H. Yang, and J. Hu. 2016. Exploring topic models in software engineering data analysis: A survey. In Proc. of the 17th IEEE/ACIS Int. Conf. on Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing (SNPD), pages 357-362.
S. Symeonidis, D. Effrosynidis, and A. Arampatzis. 2018. A comparative evaluation of pre-processing techniques and their interactions for twitter sentiment analysis. Expert Systems with Applications, 110:298-310.
Y. Teh, M. Jordan, M. Beal, and D. Blei. 2006. Hierarchical Dirichlet processes. Journal of the American Statistical Association, 101(476):1566-1581.
A. Tornberg and P. Tornberg. 2016. Combining CDA and topic modeling: Analyzing discursive connections between Islamophobia and anti-feminism on an online forum. Discourse & Society, 27(4):401-422.
J. Wang, L. Chen, L. Qin, and X. Wu. 2018. Astm: An attentional segmentation based topic model for short texts. In Proc. of the 18th IEEE Int. Conf. on Data Mining (ICDM)), pages 577-586.
J. Xue, J. Chen, C. Chen, C. Zheng, S. Li, and T. Zhu. 2020. Public discourse and sentiment during the COVID 19 pandemic: Using latent dirichlet allocation for topic modeling on twitter. *PloS one*, 15(9):e0239441.
C. Zhang and H. W. Lauw. 2020. Topic modeling on document networks with adjacent-encoder. volume 34, pages 6737-6745.
H. Zhao, L. Du, W. Buntine, and G. Liu. 2017. Metalda: A topic model that efficiently incorporates meta information. In Proc. of the 17th IEEE Int. Conf. on Data Mining (ICDM), pages 635-644.
H. Zhao, Dinh P., v. Huynh, y. Jin, L. Du, and W. Buntine. 2021. Topic modelling meets deep neural networks: A survey. arXiv preprint arXiv:2103.00498.
W. Zhao, J. Jiang, J. Weng, J. He, E. Lim, H. Yan, and X. Li. 2011. Comparing twitter and traditional media using topic models. In Proc. of the 33rd European Conf. on Information Retrieval (ECIR), pages 338-349, Berlin, Heidelberg. Springer Berlin Heidelberg.
M. Zhou and L. Carin. 2015. Negative binomial process count and mixture modeling. IEEE Transactions on Pattern Analysis and Machine Intelligence, 37(2):307-320.
Y. Zuo, J. Zhao, and K. Xu. 2016. Word network topic model: a simple but general solution for short and imbalanced texts. Knowledge and Information Systems, 48(2):379-398.
APPENDIX: Topic Model or Topic Twaddle? Re-evaluating Semantic Interpretability Measures
Caitlin Doogan and Wray Buntine
A Examples of tasks for Qualitative Experiments
Topic word-set labeling
Figure 2: Example of topic word-set labeling task. Topic 40 from AP modeled on LDA.
Topic document-collection labeling
@ScottMorrisonMP @KellyODwyer #Budget2018 #Auspol have a read. No benefits to company tax cuts
...and not increasing executive salaries or share buybacks. #TaxCuts #auspol
Can anyone explain who RTPDS Aus Pty Ltd is and how out of $14,000,000,000 ($14B if you can't keep up with the zeros) taxable income they paid only $6,375 in corporate tax? Is that even possible?? #auspol
@SwannyQLD why should multinationals and corporations that pay ZERO tax get a tax cut? #Insiders #AskingForAFriend #auspol
Corp tax cuts, just another gift, nothing to do with stimulating the economy, especially when so many don't even pay tax as it is. #auspol
#Fukushima #auspol AVOID #thorium #nuclear SHILLS John Quakes Quade Mallows Gus Rawles Leo Sutton Marcelina Thomas Hypermetropia
#auspol Profits up 25% wages up 0% ... give my corporation a tax cut!!! via @watoday
So Treasurer, tell us again how a tax cut for corporations who don't pay tax anyway will mean they will increase salaries. #auspol
Cut out the corporate tax cuts and do not even give them trickle #auspol
Well how else could they afford to pay him so much if they had to pay tax and workers as well!! #auspol
Please provide a topic label that best describes the collection of tweets above. Your label should describe a central theme, concept, event, or other unifying feature of the collection. If one or more tweets do not align with a label that is suitable to describe all other tweets, then you may disregard them. You may click on the tweet to see it in context. If you cannot identify an appropriate label, please enter 'NA'.
0% 100%
Figure 3: Example of topic document-collection labeling task. Only the top 10 tweets have been shown for brevity.
Difficulty of topic document-collection labeling
Figure 4: Example question asking SME to rate how difficult it was to label a topic document-collection.
Topic word-set and topic document-collection label alignment
Please rate how aligned the following topic-wordset label to the topic-document collection label?
Identical: Labels are nearly identical or synonymous. Example: Dual citizenship crisis AND Section 44 crisis.
Closely Aligned: Labels are related but describe different aspects of the matter. Example: Change-the-date campaign AND Australia Day debate.
Somewhat Aligned: The topic label is very general acting as a category the collection label would fit under. Example: Political scandal AND Politicians extramarital affairs.
Loosely Aligned: Labels are related, but the topic labeldoes not accurately describe the document collection. Example: Adani coal mine AND Anti-fracking protests.
Not Aligned: The labels are not aligned. Example: Racial Villification AND Climate Change
Note that common words or synonyms which are contextually dissimilar do not align topics.
Not Aligned	Loosely
Aligned	Somewhat
Aligned	Closely
Aligned	Identical
Election AND Opinion Poll results	○	○	○	○	○
Climate Science AND Arctic warming	○	○	○	○	○
Media AND Investigative Journalism	○	○	○	○	○
Industrial Relations AND Train driver strikes	○	○	○	○	○
National Identity AND Republican movement	○	○	○	○	○
Addiction services AND Cannabis decriminalisation	○	○	○	○	○
0%	100%
Figure 5: Example question asking SME to rate how aligned a topic word-set label was to topic document-collection label.
B Evaluation 1: Coherence Measures
Pearson's correlation coefficients between paired coherence measures
LDA
AP	10	20	40	60
ρ	p	ρ	p	ρ	p	ρ	p
CNPMI and CNPMI-ABC	0.671	0.034	0.749	< 0.01	0.762	< 0.01	0.727	< 0.01
CNPMI and CUnmass	0.661	0.037	0.752	< 0.01	0.579	< 0.01	0.447	< 0.01
CNPMI and CNPMI-AP	0.383	0.275	0.394	0.086	0.258	0.108	0.256	0.048
CNPMI and CP	0.897	< 0.01	0.912	< 0.01	0.919	< 0.01	0.878	< 0.01
CNPMI and CA	0.692	0.027	0.523	0.018	0.547	< 0.01	0.556	< 0.01
CNPMI and CV	0.127	0.726	0.117	0.624	-0.071	0.663	0.089	0.498
CNPMI-ABC and CNPMI-AP	0.538	0.109	0.541	0.014	0.343	0.030	0.375	0.003
MetaLDA
AP	10	20	40	60
ρ	p	ρ	p	ρ	p	rp	p
CNPMI and CNPMI-ABC	0.778	0.008	0.783	< 0.01	0.625	< 0.01	0.651	< 0.01
CNPMI and CUnmass	0.834	0.003	0.738	< 0.01	0.715	< 0.01	0.590	< 0.01
CNPMI and CNPMI-AP	0.523	0.121	0.096	0.686	-0.084	0.606	0.005	0.972
CNPMI and CP	0.929	< 0.01	0.959	< 0.01	0.949	< 0.01	0.923	< 0.01
CNPMI and CA	0.788	0.007	0.435	0.055	0.623	< 0.01	0.505	< 0.01
CNPMI and CV	-0.471	0.170	-0.078	0.742	-0.284	0.075	-0.145	0.269
CNPMI-ABC and CNPMI-AP	0.100	0.784	0.190	0.423	0.125	0.443	0.172	0.188
Aggregate mean for coherence measures
Table 3: Pearson's  $r$  and  $p$ -values reported for the analysis of correlations between coherence measures for the AP dataset
LDA	AP	AWH	AWM
CNPMI and CNPMI-ABC	0.727±0.040	0.719±0.050	0.769±0.115
CNPMI and CUnmass	0.601±0.129	0.395±0.406	0.419±0.088
CNPMI and CNPMI-AP	0.323±0.076	0.507±0.189	0.387±0.123
CNPMI and CP	0.902±0.018	0.779±0.101	0.855±0.043
CNPMI and CA	0.578±0.076	0.391±0.094	0.626±0.069
CNPMI and CV	0.066±0.092	-0.108±0.053	0.253±0.118
CNPMI-ABC and CNPMI-AP	0.449±0.105	0.565±0.056	0.423±0.073
MetaLDA	AP	AWH	AWM
CNPMI and CNPMI-ABC	0.709±0.083	0.606±0.126	0.716±0.104
CNPMI and CUnmass	0.719±0.100	0.539±0.086	0.272±0.249
CNPMI and CNPMI-AP	0.135±0.269	0.258±0.267	0.181±0.153
CNPMI and CP	0.940±0.017	0.770±0.149	0.884±0.037
CNPMI and CA	0.588±0.154	0.360±0.183	0.285±0.187
CNPMI and CV	-0.245±0.174	-0.138±0.273	0.087±0.217
CNPMI-ABC and CNPMI-AP	0.147±0.042	0.362±0.223	0.390±0.159
Table 6: The aggregate mean Pearson's correlation coefficient for LDA and MetaLDA across all topics.
LDA
AWH	10	20	40	60
ρ	p	ρ	p	ρ	p	ρ	p
CNPMI and CNPMI-ABC	0.794	0.006	0.702	0.01	0.693	< 0.01	0.688	< 0.01
CNPMI and CUMass	0.714	0.020	-0.193	0.414	0.454	0.003	0.606	< 0.01
CNPMI and CNPMI-AP	0.746	0.013	0.545	0.013	0.438	0.005	0.297	0.021
CNPMI and CP	0.628	0.052	0.816	< 0.01	0.846	< 0.01	0.826	< 0.01
CNPMI and CA	0.315	0.375	0.316	0.174	0.421	0.007	0.511	< 0.01
CNPMI and CV	-0.093	0.798	-0.176	0.459	-0.112	0.490	-0.049	0.709
CNPMI-ABC and CNPMI-AP	0.586	0.075	0.619	0.004	0.567	< 0.01	0.488	< 0.01
MetaLDA
AWH	10	20	40	60
ρ	p	ρ	p	ρ	p	rp	p
CNPMI and CNPMI-ABC	0.719	0.019	0.613	0.004	0.428	0.006	0.663	< 0.01
CNPMI and CUnmass	0.593	0.071	0.459	0.042	0.473	0.002	0.631	< 0.01
CNPMI and CNPMI-AP	0.450	0.192	0.476	0.034	0.204	0.208	-0.098	0.458
CNPMI and CP	0.547	0.102	0.844	< 0.01	0.855	< 0.01	0.835	< 0.01
CNPMI and CA	0.089	0.808	0.407	0.075	0.469	0.002	0.476	< 0.01
CNPMI and CV	0.034	0.925	-0.491	0.028	0.115	0.479	-0.209	0.108
CNPMI-ABC and CNPMI-AP	0.489	0.151	0.580	0.007	0.303	0.058	0.076	0.565
Table 4: Pearson's  $r$  and  $p$  -values reported for the analysis of coherence measures correlations for the AWH dataset
Graphs of aggregate coherence measures for LDA vs MetaLDA
Figure 6: Comparison of LDA (triangle) and MetaLDA (circle) aggregated coherence scores for the AWH dataset. Scores are shown on the y-axis and  $K$  is shown on the x-axis.
Average
WeightedAverage
Top80pcnt
Top50pont
Top25pont
LDA
AWM	10	20	40	60
ρ	p	ρ	p	ρ	p	rp	p
CNPMI and CNPMI-ABC	0.894	< 0.01	0.821	< 0.01	0.734	< 0.01	0.627	< 0.01
CNPMI and CUmass	0.512	0.131	0.450	0.047	0.303	0.058	0.410	< 0.01
CNPMI and CNPMI-AP	0.294	0.409	0.389	0.090	0.560	< 0.01	0.304	0.018
CNPMI and CP	0.898	< 0.01	0.886	< 0.01	0.816	< 0.01	0.821	< 0.01
CNPMI and CA	0.725	0.018	0.584	0.007	0.620	< 0.01	0.576	< 0.01
CNPMI and CV	0.341	0.335	0.105	0.660	0.354	0.025	0.210	0.108
CNPMI-ABC and CNPMI-AP	0.364	0.301	0.391	0.088	0.528	< 0.01	0.407	< 0.01
MetaLDA
AWM	10	20	40	60
ρ	p	ρ	p	ρ	p	ρ	p
CNPMI and CNPMI-ABC	0.866	< 0.01	0.687	0.001	0.684	< 0.01	0.625	< 0.01
CNPMI and CUnmass	0.302	0.396	-0.080	0.736	0.500	0.001	0.366	< 0.01
CNPMI and CNPMI-AP	0.289	0.418	0.133	0.576	0.315	0.048	-0.015	0.912
CNPMI and CP	0.939	< 0.01	0.860	< 0.01	0.861	< 0.01	0.875	< 0.01
CNPMI and CA	0.080	0.825	0.333	0.151	0.207	0.201	0.518	< 0.01
CNPMI and CV	-0.109	0.765	0.384	0.094	-0.033	0.839	0.105	0.425
CNPMI-ABC and CNPMI-AP	0.417	0.231	0.516	0.020	0.469	0.002	0.160	0.222
Table 5: Pearson's  $r$  and  $p$ -values reported for the analysis of correlations between coherence measures for the AWM dataset
Figure 7: Comparison of LDA (triangle) and MetaLDA (circle) aggregated coherence scores for the AWM dataset. Scores are shown on the y-axis and  $K$  is shown on the x-axis.
Average WeightedAverage Top80pcnt Top50pcnt Top25pcnt
Figure 8: Comparison of LDA (triangle) and MetaLDA (circle) aggregated coherence scores for the AWHM dataset. Scores are shown on the y-axis and  $K$  is shown on the x-axis.
Average WeightedAverage Top80pcnt Top50pcnt Top25pcnt
C Evaluation 2: Labeling Topics
The Spearman's  $\rho$  correlation coefficients for pairwise combinations of  $Q(nbr)$  and coherence measures for all learned models.
LDA
AP	10	20	40	60
ρ	p	ρ	p	ρ	p	ρ	p
CNPMI	0.058	0.873	0.264	0.261	0.144	0.374	-0.165	0.209
CNPMI-ABC	0.522	0.122	0.379	0.099	0.245	0.128	-0.038	0.774
CUmass	0.522	0.122	0.104	0.663	-0.022	0.892	-0.384	< 0.01
CNPMI-AP	0.406	0.244	0.626	0.003	0.638	< 0.01	0.217	0.096
CP	0.174	0.631	0.355	0.125	0.245	0.128	-0.062	0.636
CA	0.406	0.244	0.409	0.073	0.383	0.015	0.246	0.058
CV	0.058	0.873	0.138	0.562	0.254	0.114	0.472	< 0.01
Dtp	-0.174	0.631	0.046	0.846	0.017	0.919	0.056	0.671
Dew	-0.522	0.122	-0.814	< 0.01	-0.535	< 0.01	-0.223	0.087
MetaLDA
AP	10	20	40	60
ρ	p	ρ	p	ρ	p	ρ	p
CNPMI	0.623	0.054	-0.069	0.774	0.253	0.115	0.336	< 0.01
CNPMI-ABC	0.337	0.340	0.096	0.686	0.098	0.547	0.403	< 0.01
CUmass	0.623	0.054	-0.302	0.195	0.099	0.544	0.348	< 0.01
CNPMI-AP	0.450	0.192	0.054	0.822	0.220	0.172	0.329	0.010
CP	0.623	0.054	-0.088	0.714	0.339	0.032	0.307	0.017
CA	0.701	0.024	-0.022	0.927	0.600	< 0.01	0.304	0.018
CV	-0.545	0.103	0.320	0.169	0.172	0.289	0.118	0.371
Dtp	0.078	0.831	-0.250	0.288	0.069	0.670	0.230	0.077
Dew	-0.017	0.962	-0.211	0.372	-0.082	0.616	-0.099	0.45
Table 7: Spearman's  $\rho$  and  $p$  -values reported for the analysis of correlations between coherence measures and  ${Q}_{\left( nbr\right) }$  for the AP dataset
LDA
AWH	10	20	40	60
ρ	p	ρ	p	ρ	p	ρ	p
CNPMI
hline CNPMI-ABC	0.000	nan	0.371	0.107	-0.045	0.784	0.101	0.441
0.000	nan	0.328	0.158	0.160	0.325	0.181	0.165
CUmass	0.000	nan	-0.280	0.232	-0.078	0.634	-0.003	0.980
CNPMI-AP	0.000	nan	0.182	0.443	0.355	0.025	0.216	0.097
CP	0.000	nan	0.375	0.104	-0.016	0.921	0.066	0.615
CA	0.000	nan	0.103	0.665	0.179	0.270	0.319	0.013
CV	0.000	nan	0.043	0.857	0.192	0.235	0.235	0.071
Dtp	0.000	nan	-0.131	0.581	0.024	0.884	0.193	0.140
Dew	0.000	nan	-0.589	< 0.01	-0.345	0.029	-0.157	0.230
MetaLDA
AWH	10	20	40	60
ρ	p	ρ	p	ρ	p	ρ	p
CNPMI	-0.522	0.122	0.074	0.758	-0.051	0.753	0.394	< 0.01
CNPMI-ABC	-0.406	0.244	0.217	0.357	0.401	0.010	0.305	0.018
CUmass	-0.290	0.416	0.127	0.594	-0.039	0.813	0.297	0.021
CNPMI-AP	-0.174	0.631	0.676	< 0.01	0.490	< 0.01	0.011	0.934
CP	-0.522	0.122	-0.031	0.897	-0.087	0.595	0.392	< 0.01
CA	-0.406	0.244	0.322	0.166	0.240	0.135	0.404	< 0.01
CV	0.290	0.416	-0.088	0.713	0.338	0.033	-0.013	0.920
Dtp	0.174	0.631	0.200	0.399	0.132	0.417	0.321	0.0120
Dew	0.058	0.873	-0.255	0.279	-0.374	0.017	-0.049	0.708
Table 8: Spearman's  $\rho$  and  $p$  -values reported for the analysis of correlations between coherence measures and  ${Q}_{\left( nbr\right) }$  for the AWH dataset
LDA
AWM	10	20	40	60
ρ,p	ρ,p	ρ,p	ρ,p	ρ,p	ρ,p	ρ,p	ρ,p
CNPMI	0.142	0.696	-0.039	0.871	0.092	0.573	0.014	0.914
CNPMI-ABC	0.321	0.366	0.063	0.792	0.141	0.386	0.207	0.112
CUmass	-0.142	0.696	0.015	0.952	0.083	0.609	-0.314	0.015
CNPMI-AP	0.321	0.366	0.285	0.223	0.323	0.042	0.295	0.022
CP	0.306	0.390	-0.119	0.619	-0.007	0.968	-0.045	0.730
CA	0.350	0.321	-0.056	0.816	0.237	0.140	0.286	0.027
CV	0.634	0.049	-0.099	0.677	0.357	0.228	0.080
Dtp	0.007	0.984	0.312	0.180	0.059	0.719	0.432	< 0.01
Dew	-0.500	0.141	-0.508	0.022	-0.408	0.009	-0.203	0.120
MetaLDA
AWM	10	20	40	60
ρ	p	ρ	p	ρ	p	ρ	p
CNPMI	0.151	0.678	0.187	0.429	0.492	< 0.01	0.407	< 0.01
CNPMI-ABC	0.243	0.499	-0.002	0.993	0.228	0.158	0.159	0.224
CUnmass	0.125	0.732	-0.289	0.216	-0.067	0.680	-0.015	0.908
CNPMI-AP	0.321	0.365	-0.099	0.678	-0.055	0.734	0.047	0.721
CP	0.282	0.430	0.153	0.520	0.527	< 0.01	0.481	< 0.01
CA	-0.164	0.651	-0.085	0.722	0.053	0.747	0.267	0.039
CV	-0.164	0.651	0.332	0.153	0.053	0.745	0.220	0.091
Dtp	0.085	0.815	0.004	0.986	0.594	< 0.01	0.372	< 0.01
Dew	-0.125	0.732	-0.040	0.868	0.274	0.087	0.224	0.085
Table 9: Spearman's  $\rho$  and  $p$  -values reported for the analysis of correlations between coherence measures and  $Q\left( {nbr}\right)$  ,for the AWM dataset
D Evaluation 3: Topic Label Agreement
Inter-coder Reliability results
Kripp.
alpha	10	20	40	60
LDA	Meta	LDA	Meta	LDA	Meta	LDA	Meta
AP	0.398	0.363	0.250	0.361	0.402	0.361	0.584	0.486
AWH	0.283	0.391	0.294	0.327	0.368	0.405	0.512	0.498
AWM	0.267	0.344	0.323	0.322	0.366	0.361	0.513	0.447
Fleiss
kappa	10	20	40	60
LDA	Meta	LDA	Meta	LDA	Meta	LDA	Meta
AP	0.387	0.363	0.156	0.332	0.411	0.344	0.578	0.485
AWH	0.265	0.406	0.290	0.305	0.381	0.371	0.527	0.515
AWM	0.258	0.394	0.321	0.353	0.362	0.356	0.535	0.492
Qagr	10	20	40	60
LDA	Meta	LDA	Meta	LDA	Meta	LDA	Meta
AP	0.417	0.433	0.167	0.292	0.342	0.283	0.503	0.492
AWH	0.283	0.400	0.258	0.258	0.286	0.296	0.439	0.411
AWM	0.217	0.250	0.275	0.292	0.296	0.279	0.428	0.369
E Evaluation 4: Ease of Labeling Collections
Difficulty labeling documentcollections
This section presents all the correlations with  $Q_{aln}$  and  $Q_{dif}$ .