SlowGuess/ACL_2020_OCR huggingface dataset:
The primary goal of our solution is to effectively perform hierarchical topic modeling so that more coherent topics can be extracted. To evaluate topic model coherence, we consider 12 real-world datasets as reference. All of them were obtained from previous works in the literature. For all datasets, we performed stopwords removal (using the standard SMART list) and removed words such as adverbs, using the VADER lexicon dictionary (Hutto and Gilbert, 2014), as the vast majority of the essential words for identifying topics are nouns and verbs. These procedures improved both the efficiency and effectiveness of all analyzed strategies. Table 1 provides a summary of the reference datasets, reporting the number of features (words) and documents, as well as the mean number of words per document (density) and the corresponding references.
Table 1: Dataset characteristics
Dataset	#Feat	#Doc	Density
Angrybirds	1,903	1,428	7.135
Dropbox	2,430	1,909	9.501
Evernote	6,307	8,273	11.002
InfoVis-Vast3	6,104	909	86.215
Pinterest	2,174	3,168	4.478
TripAdvisor	3,152	2,816	8.532
Tweets	8,029	12,030	4.450
WhatsApp	1,777	2,956	3.103
20NewsGroup4	29,842	15,411	76.408
ACM	16,811	22,384	30.428
Uber	5,517	11,541	7.868
Facebook	5,168	12,297	6.427
We compare the HTM strategies using representative topic quality metrics in the literature (Nikolenko, 2016; Nikolenko et al., 2017). We consider three classes of topic quality metrics based on three criteria: (a) coherence, (b) mutual information, and (c) semantic representation. In this paper, we focus on these three criteria since they are the most used metrics in the literature (Shi et al., 2018). We consider three topic lengths (5, 10 and 20 words) for each parameter in our evaluation, since different lengths may bring different challenges.
Regarding the metrics, coherence captures easiness of interpretation by co-occurrence. Words that frequently co-occur in similar contexts in a corpus are easier to correlate since they usually define a more well-defined "concept" or "topic". We employ an improved version of regular coherence (Nikolenko, 2016), called Coherence, defined as
$$
c (t, W _ {t}) = \sum_ {w _ {1}, w _ {2} \in W _ {t}} \log \frac {d \left(w _ {1} , w _ {2}\right) + \varepsilon}{d \left(w _ {1}\right)}, \tag {2}
$$
where  $d(w1)$  denotes the number of occurrences of  $w1$ ,  $d(w1, w2)$  is the number of documents that contain both  $w1$  and  $w2$  together, and  $\varepsilon$  is a smoothing factor used for preventing  $\log(0)$ .
Another class of topic quality metrics is based on the notion of pairwise pointwise mutual information (PMI) between the top words in a topic. It captures how much one "gains" in the information given the occurrence of the other word, taking dependencies between words into consideration. Following a recent work (Nikolenko, 2016), we here compute a normalized version of PMI (NPMI) where, for a given ordered set of top words  $W_{t} = (w_{1},\dots,w_{N})$  in a topic:
$$
N P M I _ {t} = \sum_ {i <   j} \frac {\log \frac {p \left(w _ {i} , w _ {j}\right)}{\overline {{p \left(w _ {i}\right)}} p \left(w _ {j}\right)}}{- \log p \left(w _ {i} , w _ {j}\right)}. \tag {3}
$$
Finally, the third class of metrics is based on the distributed word representations introduced in (Nikolenko, 2016). The intuition is that, in a well-defined topic, the words should be semantically similar, or at least related, to be easily interpreted by humans. In a  $d$ -dimensional vector space model in which every vocabulary word  $w \in W$  has been assigned to a vector  $v_w \in R^d$ , the vectors corresponding to the top words in a topic should be close to each other. In (Nikolenko, 2016), the authors define topic quality as the average distance between the top words in the topic, as follows:
<sup>3</sup>https://www.cc.gatech.edu/gvu/ii/jigsaw/datafiles.html
<sup>4</sup>http://qwone.com/~jason/20Newsgroups/
8143
$$
W 2 V - L 1 = \frac {1}{\left| W _ {t} \right| \left(\left| W _ {t} \right| - 1\right)} \sum_ {w _ {1} \neq w _ {2} \in W _ {t}} d _ {\cos} \left(v _ {w _ {1}}, v _ {w _ {2}}\right). \tag {4}
$$
Generally speaking, let  $d(w_{1},w_{2})$  be a distance function in  $R^d$ . In this case, larger  $d(w_{1},w_{2})$  corresponds to worse topics (with words not as localized as in topics with smaller average distances). In (Nikolenko, 2016), the authors suggest four different distance metrics, with cosine distance achieving the best results. We here also employ the cosine distance, defined as  $d_{cos}(x,y) = 1 - x^{T}y$ .
We compare our approach described in Section 4, with seven hierarchical topic model strategies marked in bold in Section 2. For the input parameters of CluHTM (Algorithm 1), we set  $\mathcal{K}_{min} = 5$ ,  $\mathcal{K}_{max} = 25$ ,  $\mathcal{R} = 10$  and  $\mathcal{D}_{max} = 3$ . We define  $\mathcal{K}_{min}$  through empirical experiments, and the  $\mathcal{K}_{max}$  was defined according to the number of topics exploited in (Viegas et al., 2019). For the baseline methods, we adopt the parameters suggested by their own works. We assess the statistical significance of our results employing a paired t-test with  $95\%$  confidence and Holm-Bonferroni correction to account for multiple tests.
We start by comparing CluHTM against four state-of-the-art uHTM baselines considering the twelve reference datasets. Three hierarchical levels for each strategy are used in this comparison. In Figures 2, 4 and 3 we contrast the results of our proposed CluHTM and the reference strategies, considering the NPMI, W2V-L1, and Coherence metrics.
Figure 2: uHTM Comparative Results (NPMI).
Figure 3: uHTM Comparative Results (Coherence)
Figure 4: uHTM Comparative Results (W2V-L1)
Note that each strategy extracted a different number of topics in its hierarchical structure. Considering NPMI, the most important metric to evaluate the quality of topics (Nikolenko, 2016), we can see in Figure 2 that our strategy outperforms all baselines in all datasets by large margins, with gains over  $500\%$  against some of the strongest ones. Some of these results are the highest in terms of NMPI ever reported for several of these datasets. Considering the Coherence scores (Figure 3), our strategy achieves the single best results in 2 out of 12 datasets, with gains up to  $58\%$  and  $92\%$  against the most robust baseline (hPAM), tying in 8 out 12 and losing two times for hLDA and hPAM. Similar results can be observed for the W2V-L1 metric (Figure 4) - CluHTM ties in 10 out of 12 results, with one win and one loss for KHTM. As we will see, even with very few losses in these metrics, our method proves to be
8144
Dataset	CluHTM	SLDA	SNLDA	HSLDA
Coherence
20News	-62.6898 ± 21.0606▲	-403.3413 ± 90.2313	-410.0020 ± 71.2366	-309.9041 ± 132.5511
ACM	-32.3371 ± 29.5853▲	-539.6660 ± 115.2125	-507.4476 ± 108.6966	-486.4835 ± 104.9369
W2V-L1
20News	1.1863 ± 0.1176▼	0.3093 ± 0.2006	0.3456 ± 0.2051	0.0952 ± 0.1094
ACM	1.0489 ± 0.6506●	0.6347 ± 0.2617	0.6803 ± 0.2243	0.2816 ± 0.1567
NPMI
20News	0.9351 ± 0.0365▲	0.2714 ± 0.1157	0.2205 ± 0.0752	0.4383 ± 0.2162
ACM	0.9641 ± 0.0416▲	0.2071 ± 0.0579	0.2064 ± 0.0529	0.2761 ± 0.0978
Table 2: Comparing the results achieved by each supervised HTM strategy for Coherence, W2V-L1 and NPMI.
Table 3: Number of times each strategy was the top performer. CluHTM is the best performer in most cases.
Method	Metric
NPMI	W2V-L1	Coherence	Σ (Sum)
CluHTM	12	11	10	33
hPAM	0	9	8	17
hLDA	0	2	9	11
HSOC	0	9	0	9
KHTM	0	6	2	8
SNLDA	0	2	0	2
HSLDA	0	2	0	2
textbfSLDA	0	1	0	1
more consistent than the baselines.
We now turn our attention to the effectiveness of our proposal when compared to the supervised HTM strategies. We consider the 20News and ACM datasets for which have a ground truth for supervised strategies. Table 2 presents the results considering Coherence, W2V-L1, and NPMI. The statistical significance tests ensure that the best results, marked in  $\triangle$ , are superior to others. The statistically equivalent results are marked in  $\bullet$  while statistically significant losses are marked in  $\nabla$ . Once again, in Table 2, our proposed strategy achieves the best results in 4 out of 6 cases, tying with SNLDA and HSLDA in ACM and loosing only to SLDA in 20News, both considering the W2V-L1 metric. It is important to remind that, differently from these supervised baselines, our method does not use any privileged class information to build the hierarchical structure nor to extract topics.
We provide a comparative table with all experimental results<sup>5</sup>, including the results for each extracted level of the hierarchical structure. We summarize our findings regarding the behavior of all analyzed strategies in the 12 datasets, counting the number of times each strategy figured out as a top performer<sup>6</sup>. The summarized results can be seen in Table 3. Our proposal is in considerable advantage over the other explored baselines, being
the strategy of choice in the vast majority of cases. Overall, considering a universe of 36 experimental results (the combination of 3 evaluation metrics over 12 datasets), we obtained the best results (33 best performances), with the most robust baseline - hPAM - coming far away, with just 17 top performances. Another interesting observation is that, in terms of NPMI, CluHTM wins in all cases. Details of this analysis are summarized in the Appendix.
5.3 Impact of the Factors
One important open question remains to be answered: To what extent the characteristics of the dataset impact the quality of the topics generated by our strategy? To answer this question, we provide a quantitative analysis regarding the hierarchical topic modeling effectiveness, measured by the NPMI score.
We start our analysis by quantifying the effects of the parameters of interest (i.e., factors). Those factors might affect the performance of the system under study, while also determining whether the observed variations are due to significant effects (e.g., measurement errors, the inherent variability of the process being analyzed (Jain, 1991)). To this end, we adopt a full factorial design, which uses all the possible combinations of the levels of the factors in each complete experiment. The first factor is the dataset. The idea is to analyze the impact of textual properties such as dataset size, density, dimensionality, etc. Thus, each level of this factor is a dataset in Table 1. The second factor is the HTM strategies evaluated in the previous Section. In this factor, we intend to assess the impact of the extracted topics, as well as the hierarchical structure. Each level of this factor is an evaluated HTM strategy. All the possible combination between these two factors will be measured by the average of NMPI among topics of the hierarchical structure.
Results are shown in Table 4. In the Table, we highlight the average NPMI and the effects of each
5see Appendix, Section Supplementary Results for detailed results
If two approaches are statistically tied as top performers in the same dataset, both will be counted.
8145
Dataset-Algorithm	CluHTM	hLDA	hPAM	HSOC	KHTM	Row Sum	Row Mean	Row Effect
Angry Birds	0.8934	0.5593	0.3604	0.2120	0.4940	2.5191	0.5038	0.0507
Dropbox	0.9002	0.5806	0.2529	0.1703	0.4022	2.3062	0.4612	0.0082
Evernote	0.9374	0.5668	0.1534	0.1222	0.4426	2.2224	0.4445	-0.0086
Facebook	0.8686	0.5998	0.1517	0.1128	0.4791	2.2120	0.4424	-0.0107
InfoVis-Vast	0.9935	0.1650	0.1191	0.1632	0.1459	1.5867	0.3173	-0.1357
Pinterest	0.8482	0.5614	0.3028	0.1865	0.3912	2.2901	0.4580	0.0049
Trip Advisor	0.9265	0.5769	0.2745	0.1477	0.5007	2.4263	0.4853	0.0322
Tweets	0.8950	0.5966	0.2130	0.1928	0.4759	2.3733	0.4747	0.0216
Uber	0.9116	0.5829	0.1403	0.1006	0.4168	2.1522	0.4304	-0.0226
Whatsapp	0.8594	0.5881	0.3976	0.2031	0.5172	2.5654	0.5131	0.0600
Col Sum	9.0338	5.3774	2.3657	1.6112	4.2656	22.6537	-	-
Col Mean	0.9034	0.5377	0.2366	0.1611	0.4266	-	0.4531	-
Col effect	0.4503	0.0847	-0.2165	-0.2920	-0.0265	-	-	-
Table 4: Overview of the factorial desgin
Component	Sum of Degrees	% Variation	Degrees of Freedom	Mean Square	F-Computed	F-Table (0.99)
y	14.0670	-	50	-	-	-
y..	10.2638	-	1	-	-	-
y - y..	3.8032	100.00%	49	-	-	-
A	3.4276	90.12%	4	0.8569	127.9197	3.8903
B	0.1345	3.92%	9	0.0149	2.2303	2.9461
e	0.2412	6.34%	36	0.0067	-	-
Table 5: ANOVA Test with  ${99}\%$  confidence to measure the impact of each factor.
factor. From the effects, we can observe that the CluHTM impact in the NPMI value is  $99.38\%$  higher than the overall average. We can also see that hLDA has an NPMI score higher than the overall average  $(18.67\%)$  and HSOC has an NPMI score of approximately  $64.44\%$  smaller than overall NMPI. Concerning the datasets' effects, the full factorial design experiment tells us that they have a small impact on the variation concerning the obtained average NPMI scores. We can also observe that the dataset with the most variation of NPMI is InfoVis-Vast, with a score of  $29.97\%$  smaller than the overall NPMI.
We perform a ANOVA test to assess whether the studied factors are indeed statistically significant and conclude, with  $99\%$  confidence according to the F-test, that the choice of algorithm (factor B) explains approximately  $90\%$  of the obtained NPMI values. We can also conclude that the investigated properties of the textual data (factor A), as well as the experimental errors, have a small influence on the experimental results. Summarizing, we can conclude that the characteristics of the datasets have a lower impact on the results and that the impact of CluHTM is consistent across all of them. The ANOVA test details are presented in Table 5.
We advanced the state-of-the-art in hierarchical topic modeling (HTM) by designing, implementing and evaluation a novel unsupervised nonprobabilistic method - CluHTM. Our new method exploits a more elaborate (global) semantic data
representation – CluWords – as well as an original application of a stability measure to define the “shape” of the hierarchy. CLUHTM excelled in terms of effectiveness, being around two times more effective than the strongest state-of-the-art baselines, considering all tested datasets and evaluation metrics. The overall gains over some of these strongest baselines are higher than  $500\%$  in some datasets. We also showed that CluHTM results are consistent across most datasets, independently of the data characteristics and idiosyncrasies. As future work, we intend to apply CluHTM in other representative applications on the Web, such as hierarchical classification by devising a supervised version of CluHTM. We also intend to incorporate some type of attention mechanism into our methods to better understand which Cluwords are more important to define certain topics.
This work is partially supported by CAPES, CNPq, Finep, Fapemig, Mundiale, Astrein, projects InWeb and MASWeb.
David M Blei, Andrew Y Ng, and Michael I Jordan. 2003. Latent dirichlet allocation. Journal of machine Learning research, 3(Jan):993-1022.
Derek Greene, Derek O'Callaghan, and Pádraig Cunningham. 2014. How many topics? stability analysis for topic models. CoRR.
Thomas L Griffiths, Michael I Jordan, Joshua B Tenenbaum, and David M Blei. 2004. Hierarchical topic
8146
models and the nested chinese restaurant process. In Advances in neural information processing systems, pages 17-24.
Clayton J. Hutto and Eric Gilbert. 2014. VADER: A parsimonious rule-based model for sentiment analysis of social media text. In ICWSM'14.
Raj Jain. 1991. The Art of Computer Systems Performance Analysis: Techniques for Experimental Design, Measurement, Simulation, and Modeling.
Suin Kim, Jianwen Zhang, Zheng Chen, Alice Oh, and Shixia Liu. 2013. A hierarchical aspect-sentiment model for online reviews. In Twenty-Seventh AAAI Conference on Artificial Intelligence.
Harold W. Kuhn. 2010. The hungarian method for the assignment problem. In 50 Years of Integer Programming.
Daniel D Lee and H Sebastian Seung. 2001. Algorithms for non-negative matrix factorization. In Advances in neural information processing systems.
Wei Li and Andrew McCallum. 2006. Pachinko allocation: Dag-structured mixture models of topic correlations. In Proceedings of the 23rd international conference on Machine learning, pages 577-584. ACM.
Rui Liu, Xingguang Wang, Deqing Wang, Yuan Zuo, He Zhang, and Xianzhu Zheng. 2018. Topic splitting: a hierarchical topic model based on nonnegative matrix factorization. Journal of Systems Science and Systems Engineering, 27(4):479-496.
Jon D Mcauliffe and David M Blei. 2008. Supervised topic models. In Advances in neural information processing systems, pages 121-128.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space.
Tomas Mikolov, Edouard Grave, Piotr Bojanowski, Christian Puhrsch, and Armand Joulin. 2018. Advances in pre-training distributed word representations. In LREC'18.
David Mimno, Wei Li, and Andrew McCallum. 2007. Mixtures of hierarchical topics with pachinko allocation. In Proceedings of the 24th ICML, pages 633-640. ACM.
Zhao-Yan Ming, Kai Wang, and Tat-Seng Chua. 2010. Prototype hierarchy based clustering for the categorization and navigation of web collections. In Proceedings of the 33rd ACM SIGIR, pages 2-9. ACM.
Sergey I Nikolenko. 2016. Topic quality metrics based on distributed word representations. In SIGIR'16.
Sergey I Nikolenko, Sergei Koltcov, and Olessia Koltsova. 2017. Topic modelling for qualitative studies. Journal of Information Science.
John Paisley, Chong Wang, David M Blei, and Michael I Jordan. 2014. Nested hierarchical dirichlet processes. IEEE Transactions on Pattern Analysis and Machine Intelligence, 37(2):256-270.
Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. In EMNLP.
Adler J Perotte, Frank Wood, Noemie Elhadad, and Nicholas Bartlett. 2011. Hierarchically supervised latent dirichlet allocation. In Advances in neural information processing systems, pages 2609-2617.
Philip Resnik, William Armstrong, Leonardo Claudino, Thang Nguyen, Viet-An Nguyen, and Jordan Boyd-Graber. 2015. Beyond lda: exploring supervised topic modeling for depression-related language in twitter. In Proc. of the 2nd Workshop on Computational Linguistics and Clinical Psychology: From Linguistic Signal to Clinical Reality, pages 99-107.
Tian Shi, Kyeongpil Kang, Jaegul Choo, and Chandan K. Reddy. 2018. Short-text topic modeling via non-negative matrix factorization enriched with local word-context correlations. In WWW '18, pages 1105-1114.
Jian Tang, Zhaoshi Meng, XuanLong Nguyen, Qiaozhu Mei, and Ming Zhang. 2014. Understanding the limiting factors of topic modeling via posterior contraction analysis. In Proceedings of the 31st ICML'14, pages I-190-I-198. JMLR.org.
Yee Whye Teh, Michael I Jordan, Matthew J Beal, and David M Blei. 2006. Hierarchical dirichlet processes. Journal of the American Statistical Association, 101(476):1566-1581.
Felipe Viegas, Sérgio Canuto, Christian Gomes, Washington Luiz, Thierson Rosa, Sabir Ribas, Leonardo Rocha, and Marcos André Gonçalves. 2019. Cluwords: Exploiting semantic word clustering representation for enhanced topic modeling. In Proceedings of WSDM '19, pages 753-761.
Chi Wang, Jialu Liu, Nihit Desai, Marina Danilevsky, and Jiawei Han. 2015. Constructing topical hierarchies in heterogeneous information networks. Knowledge and Information Systems, 44(3):529-558.
Yueshen Xu, Jianwei Yin, Jianbin Huang, and Yuyu Yin. 2018. Hierarchical topic modeling with automatic knowledge mining. Expert Systems with Applications, 103:106-117.
8147
Supplementary Results
The Tables below expand on the results of Section 5.
Datasets	CluNMF	SLDA	SNLDA	HSLDA
20News	-62.68 ± 21.06	-403.34 ± 90.23	-410.00 ± 71.23	-309.90 ± 132.55
ACM	-32.33 ± 29.58	-539.66 ± 115.21	-507.44 ± 108.69	-486.48 ± 104.93
Table 6: Overall Coherence results compared with supervised HTM strategies.
Datasets	CluNMF	SLDA	SNLDA	HSLDA
20News	0.9351 ± 0.0365	0.2714 ± 0.1157	0.2205 ± 0.0752	0.4383 ± 0.2162
ACM	0.9641 ± 0.0416	0.2071 ± 0.0579	0.2064 ± 0.0529	0.2761 ± 0.0978
Table 7: Overall NPMI results compared with supervised HTM strategies.
Datasets	CluNMF	SLDA	SNLDA	HSLDA
20News	1.1863 ± 0.1176	0.3093 ± 0.2006	0.3456 ± 0.2051	0.0952 ± 0.1094
ACM	1.0489 ± 0.6506	0.6347 ± 0.2617	0.6803 ± 0.2243	0.2816 ± 0.1567
Table 8: Overall W2V-L1 results compared with supervised HTM strategies.
Datasets	Level	CluNMF	SLDA	SNLDA	HSLDA
20News	1 Level	-12.65 ± 0.00	-403.34 ± 90.23	-428.59 ± 0.00	-317.08 ± 0.00
2 Level	-45.37 ± 22.72	-	-426.88 ± 103.53	-194.91 ± 0.00
3 Level	-39.03 ± 22.94	-	-403.75 ± 56.71	-313.75 ± 135.41
ACM	1 Level	-34.15 ± 16.73	-539.66 ± 115.21	-451.24 ± 0.00	-594.07 ± 0.00
2 Level	-34.15 ± 16.7377	-	-431.38 ± 55.64	-467.37 ± 0.00
3 Level	-27.26 ± 6.48	-	-516.92 ± 110.93	-483.32 ± 106.59
Table 9: Coherence results by level of hierarchy compared with supervised HTM strategies.
Datasets	CluNMF	hLDA	hPAM	HSOC	KHTM
20News	-62.68 ± 21.06	-339.45 ± 186.20	-62.56 ± 9.81	-393.40 ± 76.70	-397.19 ± 143.59
ACM	-32.33 ± 29.58	-219.85 ± 159.25	-77.20 ± 9.92	-577.08 ± 102.33	-341.39 ± 98.87
AngryBirds	-77.39 ± 41.17	-107.70 ± 65.08	-46.80 ± 16.44	-492.40 ± 33.12	-148.41 ± 94.66
Dropbox	-69.56 ± 33.86	-119.76 ± 103.54	-65.52 ± 12.96	-529.34 ± 28.97	-285.62 ± 88.38
Evernote	-54.45 ± 32.81	-190.48 ± 123.60	-87.33 ± 8.80	-634.04 ± 51.01	-292.90 ± 93.97
Facebook	-110.57 ± 51.62	-151.63 ± 130.89	-98.11 ± 12.53	-689.19 ± 51.92	-297.75 ± 96.46
InfoVis-Vast	-4.46 ± 11.36	-407.05 ± 74.89	-61.9381 ± 10.1260	-416.49 ± 42.25	-464.38 ± 60.96
Pinterest	-111.90 ± 48.77	-106.95 ± 87.06	-58.00 ± 16.46	-533.79 ± 42.19	-267.71 ± 84.46
TripAdvisor	-55.60 ± 27.54	-126.96 ± 90.43	-68.95 ± 15.60	-584.30 ± 29.14	-147.53 ± 97.18
Tweets	-93.11 ± 31.38	-114.24 ± 71.03	-92.52 ± 9.71	-665.23 ± 59.39	-266.22 ± 77.54
Uber	-75.25 ± 39.94	-185.50 ± 136.32	-94.72 ± 9.24	-681.78 ± 55.40	-391.70 ± 108.40
WhatsApp	-105.28 ± 38.64	-60.81 ± 62.04	-48.48 ± 15.55	-552.77 ± 45.50	-204.83 ± 87.73
Table 10: Overall Coherence results compared with uHTM strategies.
Datasets	CluNMF	hLDA	hPAM	HSOC	KHTM
20News	0.9351 ± 0.0365	0.4603 ± 0.1498	0.2176 ± 0.0622	0.2875 ± 0.0782	0.4433 ± 0.1223
ACM	0.9641 ± 0.0416	0.5781 ± 0.1021	0.1758 ± 0.0432	0.1889 ± 0.0490	0.4631 ± 0.0769
AngryBirds	0.8934 ± 0.0514	0.5593 ± 0.0565	0.3604 ± 0.1005	0.2120 ± 0.0306	0.4940 ± 0.0711
Dropbox	0.9002 ± 0.0454	0.5806 ± 0.0864	0.2529 ± 0.0877	0.1703 ± 0.0325	0.4022 ± 0.0615
Evernote	0.9374 ± 0.0334	0.5668 ± 0.0819	0.1534 ± 0.0564	0.1222 ± 0.0232	0.4426 ± 0.0763
Facebook	0.8686 ± 0.0531	0.5998 ± 0.0734	0.1517 ± 0.0765	0.1128 ± 0.0458	0.4791 ± 0.0744
InfoVis-Vast	0.9935 ± 0.0190	0.1650 ± 0.0732	0.1191 ± 0.0533	0.1632 ± 0.0504	0.1459 ± 0.0793
Pinterest	0.8482 ± 0.0535	0.5614 ± 0.0664	0.3028 ± 0.0988	0.1865 ± 0.0414	0.3912 ± 0.0559
TripAdvisor	0.9265 ± 0.0344	0.5769 ± 0.0745	0.2745 ± 0.0906	0.1477 ± 0.0306	0.5007 ± 0.0744
Tweets	0.8950 ± 0.0323	0.5966 ± 0.0381	0.2130 ± 0.0453	0.1928 ± 0.0534	0.4759 ± 0.0472
Uber	0.9116 ± 0.0424	0.5829 ± 0.0863	0.1403 ± 0.0582	0.1006 ± 0.0305	0.4168 ± 0.0861
WhatsApp	0.8594 ± 0.0456	0.5881 ± 0.0326	0.3976 ± 0.0750	0.2031 ± 0.0385	0.5172 ± 0.0634
Table 11: Overall NPMI results compared with uHTM strategies.
8148
Datasets	CluNMF	hLDA	hPAM	HSOC	KHTM
20News	1.1863 ± 0.1176	1.4423 ± 0.1412	1.1318 ± 0.0860	0.3201 ± 0.2085	0.2153 ± 0.1757
ACM	1.0489 ± 0.6506	1.4741 ± 0.0915	1.1296 ± 0.0987	0.6408 ± 0.2712	0.1544 ± 0.1522
AngryBirds	1.1489 ± 0.1157	1.3236 ± 0.0327	1.2286 ± 0.0779	1.1816 ± 0.0388	1.2603 ± 0.0285
Dropbox	1.1454 ± 0.0918	1.3388 ± 0.0402	1.1794 ± 0.0873	1.1687 ± 0.0417	1.3032 ± 0.0421
Evernote	1.0999 ± 0.1247	1.3828 ± 0.0524	1.1447 ± 0.0643	1.1825 ± 0.0453	1.3272 ± 0.0525
Facebook	1.1909 ± 0.1224	1.4152 ± 0.0411	1.1598 ± 0.0541	1.1767 ± 0.0561	1.3008 ± 0.0390
InfoVis-Vast	1.1047 ± 0.0867	1.1939 ± 0.0717	1.1651 ± 0.0651	1.1919 ± 0.0509	1.1720 ± 0.0510
Pinterest	1.2101 ± 0.0963	1.2912 ± 0.0263	1.2147 ± 0.0712	1.1760 ± 0.0495	1.2255 ± 0.0257
TripAdvisor	1.1081 ± 0.1082	1.3686 ± 0.0464	1.1814 ± 0.0685	1.1470 ± 0.0318	1.3161 ± 0.0327
Tweets	1.0493 ± 0.1086	1.4315 ± 0.0314	1.2142 ± 0.0654	1.2242 ± 0.0687	1.3285 ± 0.0372
Uber	1.1323 ± 0.1328	1.3758 ± 0.0419	1.1370 ± 0.0664	1.1677 ± 0.0381	1.3018 ± 0.0518
WhatsApp	1.1239 ± 0.1087	1.2254 ± 0.0141	1.2162 ± 0.0656	1.1732 ± 0.0423	1.2952 ± 0.0268