SlowGuess/NAACL_2021_OCR huggingface dataset:
Caitlin Doogan
Faculty of Information Technology
Monash University, Australia
caitlin.doogan@dmonash.edu
Wray Buntine
wray.buntine@dmonash.edu
When developing topic models, a critical question that should be asked is: How well will this model work in an applied setting? Because standard performance evaluation of topic interpretability uses automated measures modeled on human evaluation tests that are dissimilar to applied usage, these models' generalizability remains in question. In this paper, we probe the issue of validity in topic model evaluation and assess how informative coherence measures are for specialized collections used in an applied setting. Informed by the literature, we propose four understandings of interpretability. We evaluate these using a novel experimental framework reflective of varied applied settings, including human evaluations using open labeling, typical of applied research. These evaluations show that for some specialized collections, standard coherence measures may not inform the most appropriate topic model or the optimal number of topics, and current interpretability performance validation methods are challenged as a means to confirm model quality in the absence of ground truth data.
Topic modeling has become a popular tool for applied research such as social media analysis, as it facilitates the exploration of large documentcollections and yields insights that would not be accessible by manual methods (Sinnenberg et al., 2017; Karami et al., 2020). However, social media data can be challenging to model as it is both sparse and noisy (Zhao et al., 2011). This has resulted in increased demand for short-text topic models that can handle these challenges (Lim et al., 2013; Zuo et al., 2016; Chen et al., 2015).
Topic word-sets, denoted  $T_{ws}$ , are considered to be semantically related words that represent the latent component of the underlying topic's document-collection, denoted  $T_{dc}$ . Meaning is derived from these topics through the interpretation of either the
$T_{ws}$  (Nerghes and Lee, 2019), the corresponding  $T_{dc}$  (Maier et al., 2018), or both (Tornberg and Tornberg, 2016). Since meaning requires topics to be interpretable to humans, empirical assurance is needed to confirm a novel topic models' capacity to generate "semantically interpretable" topics, as well as a method to guide model selection and other parameters such as the number of topics,  $K$ . This is often achieved by calculating the coherence scores for  $T_{ws}$  (Lau and Baldwin, 2016)
Recent literature contradicts previous evaluations of some short-text topic models that claim superior interpretability (Li et al., 2018; Eickhoff and Wieneke, 2018; Bhatia et al., 2017). Such rethinking flows from the fact there is no agreement on the best measure of interpretability (Lau et al., 2014b; Morstatter and Liu, 2017) and is compounded by the unclear relationship between human evaluation methodologies and automated coherence scores (Lau et al., 2014b). Finally, despite assurances of generalizability and applicability, topic model evaluations in machine learning are conducted in experimental settings that are not representative of typical applied use. This raises questions of whether coherence measures are suitably robust to measure topic interpretability and inform model selection in applied settings, particularly with challenging datasets like that of social media.
Advances in topic modeling for static documentcollections have produced non-parametric approaches such as HDP-LDA, which employ sophisticated hierarchical priors that allow for different prior proportions (Teh et al., 2006). Non-negative matrix factorization (Zhou and Carin, 2015), the use of word embeddings, and neural network methods (Zhao et al., 2021) are a few of these other innovations.
To support these advances, it is crucial to establish the robustness of topic modeling interpretability measures, especially given the growing trend towards evaluating topic models using coherence
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3824-3848 June 6-11, 2021. ©2021 Association for Computational Linguistics
measures, often in the absence of perplexity or other predictive scores (?). Additionally, increasingly sophisticated methods for automatic topic labeling have been developed. Beginning with Lau et al. (2011), this research relies on models which generate interpretable topics. While these advances enhance the technologies available to conduct applied research, they do not address the underlying question of whether topic interpretability can be adequately assessed using coherence measures.
In this paper, we demonstrate a research gap in topic model evaluation methods in light of their growing use in specialized settings. Previously declared state-of-the-art models are under-performing in applied settings (Li et al., 2018; Arnold et al., 2016), and little work has been done to improve application relevance (Hecking and Leydesdorff, 2019). Following the work of (Lau and Baldwin, 2016; Bhatia et al., 2017; Hecking and Leydesdorff, 2019), this study examines whether coherence is a valid predictor of topic model interpretability when interpretability is defined as more than just the ability to label a  $T_{ws}$ , and as the diversity of topic models, datasets and application tasks increases.
Earlier research has established a correlation between novel coherence measures and human ranking of interpretability, as measured by qualitative tests (Cheng et al., 2014; Newman et al., 2010a). However, since bounded experimental settings constrain these tests, they are unlikely to reliably and consistently indicate topic quality in applied research settings. As a result, we ask the following question: To what extent can we rely on current coherence measures as proxies for topic model interpretability in applied settings?
This work has significant practical implications. It signals the need to re-develop interpretability measures and reappraise best-practice for validating and evaluating topic models and their applications. Our research contributes the following:
1. Introduces a novel human-centered qualitative framework for evaluating interpretability in model development that mimics those processes seen in applied settings.
2. Demonstrates that the ranking of topic quality using state-of-the-art coherence measures is inconsistent with those produced through validation tasks performed in an applied setting.
3. Systematically quantifies the impact of model behavior, dataset composition, and other pre
viously reported factors (Morstatter and Liu, 2017; Lau and Baldwin, 2016), on coherence measures for many topics across four variant datasets and two topic models.
4. Provide evidence to show that interpretability measures for evaluating  $T_{ws}$  and  $T_{dc}$  for applied work in specialized contexts (e.g., Twitter) are ill-suited and may hinder model development and topic selection.
The remainder of this paper is organized as follows. Section 2 provides a review of related work around the interpretability of topic models. Section 3 describes five propositions that have informed the design of interpretable topic models and their evaluation measures. This is followed by a description of the experimental framework we designed to test these propositions. Section 4 provides the results of these evaluations and Section 5 contains a discussion of findings.
This section provides a brief overview of work related to interpretability evaluation, followed by a review of the challenges associated with coherence optimization for specialized contexts.
2.1 Topic Model Interpretability
Topic model interpretability is a nebulous concept (Lipton, 2018) related to other topic model qualities, but without an agreed-upon definition. Measures of semantic coherence influence how easily understood the top- $N$ $T_{ws}$  are (Morstatter and Liu, 2017; Lund et al., 2019; Newman et al., 2010a; Lau et al., 2014b). This is also referred to as topic understandability (Röder et al., 2015; Aletras et al., 2015).
A coherent topic is said to be one that can be easily labeled and thus interpreted (Newman et al., 2011; Morstatter and Liu, 2017), but only if the label is meaningful (Hui, 2001; Newman et al., 2010b,a). Some have modeled coherence measures based on topic meaningfulness (Lau et al., 2014a); others state that a meaningful topic is not necessarily a useful one (Boyd-Graber et al., 2015). Indeed, the literature remains divided over whether usefulness is a property of an interpretable topic (Röder et al., 2015), or if interpretability is a property of a useful topic (Aletras and Stevenson, 2013; Newman et al., 2010b). Such terminological disagreement suggests that there are challenges to the progression of this area of research.
The ease of labeling a topic is assumed to be an expression of how coherent that topic is and thus its degree of interpretability. This assumption is challenged when annotators provide different labels for a topic. Morstatter and Liu (2017) presented interpretability from the perspective of both coherence and consensus, where consensus is a measure of annotator agreement about a topics' representation in its  $T_{dc}$ . Alignment is how representative a topic is of its  $T_{dc}$  and is another understanding of interpretability (Ando and Lee, 2001; Chang et al., 2009; Mimno et al., 2011; Bhatia et al., 2017; Alokaili et al., 2019; Morstatter and Liu, 2017; Lund et al., 2019). However, the probabilistic nature of topic models impede this measure. The ambiguity of interpretability as a performance target raises questions about how topic models are used and evaluated.
Following the seminal work of Chang et al. (2009), the development of coherence measures and the human evaluation tasks that guide their design has been actively pursued (Newman et al., 2010a; Bhatia et al., 2017, 2018; Morstatter and Liu, 2017; Lau and Baldwin, 2016; Lund et al., 2019; Alokaili et al., 2019). Newman et al. (2010a) showed that human ratings of topic coherence (observed coherence) correlated with their coherence measure when the aggregate Pointwise Mutual Information (PMI) pairwise scores were calculated over the top- $N T_{ws}$ . In addition to the word intrusion task (Chang et al., 2009), Mimno et al. (2011) validated their coherence measure for modeling domain-specific corpora using expert ratings of topic quality. The measure takes the order of the top- $N T_{ws}$  into account using a smoothed conditional probability derived from document co-occurrence counts. This performance was further improved by substituting PMI for Normalized PMI ( $C_{NPMI}$ ) (Aletras and Stevenson, 2013; Lau et al., 2014b). Aletras and Stevenson (2013) used crowdsourced ratings of topic usefulness to evaluate distributional semantic similarity methods for automated topic coherence. Röder et al. (2015) conducted an exhaustive study evaluating prior work and developing several improved coherence measures.
Similarly, Ramrakhiyani et al. (2017) made use of the same datasets and evaluations and presented a coherence measure which is approximated with the size of the largest cluster produced from embed
dings of the top- $N$ $T_{ws}$ . Human evaluation tasks have also been created to measure how representative a topic model is of the underlying  $T_{dc}$  (Chang et al., 2009; Bhatia et al., 2017; Morstatter and Liu, 2017; Alokaili et al., 2019; Lund et al., 2019).
2.3 Practical Applications
Within computer science, topic modeling has been used for tasks such as word-sense disambiguation (Boyd-Graber and Blei, 2007), hierarchical information retrieval (Blei et al., 2003), topic correlation (Blei and Lafferty, 2007), trend tracking (AlSumait and Domeniconi, 2008), and handling shorttexts (Wang et al., 2018). Outside of computer science, topic modeling is predominantly used to guide exploration of large datasets (Agrawal et al., 2018), often with a human-in-the-loop approach. Here topics are generated before some form of qualitative method is used to gain insights into the data. These methods include exploratory content analysis (Korencic et al., 2018), critical discourse analysis (Tornberg and Tornberg, 2016), digital autoethnography (Brown, 2019), grounded theory (Baumer et al., 2017), and thematic analysis (Doogan et al., 2020; Andreotta et al., 2019).
Qualitative techniques make use of topics in different ways. "Open labeling" of topics by Subject Matter Experts (SME) is followed by a descriptive analysis of that topic (Kim et al., 2016; Morstatter et al., 2018; Karami et al., 2018). However, this method is subjective and may fail to produce the depth of insight required. Supplementing a topic analysis with samples from the  $T_{dc}$  increases the depth of insight (Eickhoff and Wieneke, 2018; Kagashe et al., 2017; Nerghes and Lee, 2019). Alternatively, the  $T_{dc}$  alone can be used for in-depth analysis (Tornberg and Tornberg, 2016). However, human evaluation tasks that require open labeling are not generally used to validate new coherence measures (O'Callaghan et al., 2015; Korencic et al., 2018).
3 Evaluating Interpretability
We have generated five propositions about the relationship between coherence scores, human evaluation of topic models, and the different views of interpretability to explore the research question. We conduct five experiments to interrogate these propositions and re-evaluate how informative coherence measures are for topic interpretability. Because we are evaluating existing coherence measures, we
do not employ automatic topic labeling techniques. Instead, we make use of human evaluation tasks that reflect those conducted in applied settings.
Proposition 1. If coherence scores are robust, they should correlate. The battery of coherence measures for evaluating novel topic models and automated labeling approaches are inconsistent across the literature. Each new measure claims superior alignment to topic model interpretability. As these measures are evolutionary (Röder et al., 2015), and there is no convention for which measure should be used, particularly as a standard measure of qualitative performance (Zuo et al., 2016; Zhao et al., 2017; Zhang and Lauw, 2020), they are considered notionally interchangeable. Thus, we would expect that there would be some correlation between these measures. However, previous studies have not considered the impact that the data type or model has on the coherence scores. Particularly for non-parametric models, these issues may be compounded by how coherence measures are presented as an aggregate, e.g., The presentation of the top-N models. Indeed, studies reporting multiple coherence measures have demonstrated inconsistencies at the model-level that are obscured during reporting (Blair et al., 2020).
Proposition 2. An interpretable topic is one that can be easily labeled. How easily a topic could be labeled has been evaluated on an ordinal scale where humans determined if they could hypothetically give a topic a label (Mimno et al., 2011; Morstatter and Liu, 2017). However, humans are notoriously poor at estimating their performance, particularly when they are untrained and do not have feedback on their performance (Dunning et al., 2003; Morstatter and Liu, 2017). Thus, a rater's perception of whether they could complete a task is actually less informative than having them complete the task.
Proposition 3. An interpretable topic has high agreement on labels. Agreement on a topic label is considered a feature of interpretability by Morstatter and Liu (2017), who propose "consensus" as a measure of interpretability. A high level of agreement on topic labels, particularly in crowdsourcing tasks, is seen as a means to infer that a  $T_{ws}$  is interpretable. However, in applied tasks, a topic is described in a sense-making process resulting in one coherent label. Thus, the consensus task is not necessarily a reasonable means to infer inter
pretability. A robust way to measure agreement on a topic label is needed. Inter-coder reliability (ICR) measures are an appropriate means to achieve this.
Proposition 4. An interpretable topic is one where the document-collection is easily labeled. The investigation of topic document-collections is an emerging trend in the applied topic modeling literature. In these studies, authors have either used a topics "top documents" to validate or inform the labels assigned to  $T_{ws}$  (Kirilenko et al., 2021), or have ignored the  $T_{ws}$  in favor of qualitative analysis of the richer  $T_{dc}$  (Doogan et al., 2020). The use of topic modeling for the exploration of documentcollections requires a  $T_{dc}$  to be coherent enough that a reader can identify intertextual links between the documents. The label or description given to the  $T_{dc}$  results from the readers' interpretation of individual documents relative to the other documents in the collection.  $T_{dc}$  that have a high degree of similarity between their documents will be easiest to interpret and therefore label. The ease of labeling a  $T_{dc}$  decreases as the documents become more dissimilar.
Proposition 5. An interpretable topic word-set is descriptive of its topic document-collection. The alignment of  $T_{ws}$  to  $T_{dc}$  is an expected property of a "good" topic (Chang et al., 2009), which human evaluation tasks have been developed to assess. Typically these tasks ask annotators to choose the most and/or least aligned  $T_{ws}$  to a given document (Morstatter and Liu, 2017; Lund et al., 2019; Alokaili et al., 2019; Bhatia et al., 2018), identify an intruder topic (Chang et al., 2009; Morstatter and Liu, 2017), rate their confidence in a topic-document pair (Bhatia et al., 2017), or select appropriate documents given a category label (Aletras et al., 2017). However, none of these methods address the need for the topic document-collection to be evaluated and labeled. Furthermore, they generally use one document and/or are not comparable to applied tasks.
The Auspol-18 dataset was constructed from 1,830,423 tweets containing the hashtag #Auspol, an established Twitter forum for the discussion of Australian politics. The diminutives, slang, and domain-specific content provide a realistic example of a specialized context. Four versions of the dataset were constructed from a subset of 123,629 tweets; AWH (contains the 30 most frequent hash
tags), AWM (contains the 30 most frequent mentions of verified accounts), AWMH (contains the 30 most frequent hashtags and 30 most frequent mentions of verified accounts), and AP (contains neither hashtags nor mentions). Pre-processing included stopwords removal, POS-tagging, lemmatization, exclusion of non-English tweets, duplicate removal, removal of tokens with a frequency  $n < 10$ , and removal of tweets with  $n < 5$  tokens, and standardization of slang, abbreviations (Agrawal et al., 2018; Doogan et al., 2020) $^{1}$ .
3.2 Models and Parameters
To investigate interpretability in an applied setting, we compare LDA to MetaLDA (Zhao et al., 2017), a recent non-parametric topic model designed to improve short-text topic modeling by leveraging the incorporation of the document and word meta-information using word embeddings as well as non-parametrically learning topic proportions. Despite the many extensions to LDA, the vanilla model maintains popularity among applied researchers (Sun et al., 2016), and as the baseline model, it is necessary to compare LDA with a model purpose-built for short-text applications. MetaLDA is one reasonable representative of such models and has demonstrated effectiveness on Twitter data for applied work (Doogan et al., 2020). The extensive effort of human labeling in our experiments (see Section 3.4) precludes us from adding more models. LDA and MetaLDA are available in the MetaLDA package<sup>2</sup>, which is implemented on top of Mallet (McCallum, 2002).
Default parameter settings were used for both LDA and MetaLDA. We use Glove2Vec embeddings trained on the Wikipedia corpus (Pennington et al., 2014) for MetaLDA. We constructed topic sets with the number of topics  $K = \{10, 40, 20, 60, 80, 100, 150, 200\}$ .
3.3 Coherence Measures
Several coherence measures were evaluated. These were  $C_{Umass}$  (Mimno et al., 2011),  $C_V$ ,  $C_P$  (Röder et al., 2015),  $C_A$  and  $C_{NPMI}$  (Aletras and Stevenson, 2013). These were calculated for each topic using the Palmetto package $^3$  using the top ten most frequent words. Along with the default  $C_{NPMI}$ , which is calculated using Wikipedia, we introduced
$C_{NPMI-ABC}$ , which is calculated using a collection of 760k Australian Broadcasting Company (ABC) news articles<sup>4</sup> with 150 million words (enough to make the  $C_{NPMI}$  scores stable), and  $C_{NPMI-AP}$  calculated using the AP dataset and is used to test  $C_{NPMI}$  but with statistics drawn from the training data. We report the average scores and the standard deviations over five random runs.
3.4 Qualitative Experiments
A primary concern in machine learning research is the need to establish model performance. Following the recent trend to analyze  $T_{dc}$ , we devised qualitative tests for the assessment of whether the  $T_{ws}$  and  $T_{dc}$  were adequately aligned and whether current performance measures are informative of this alignment. We also tested to see if there is a relationship between topic alignment and the topic diagnostic statistics; effective number of words<sup>5</sup>, and topic proportion, denoted  $D_{ew}$  and  $D_{tp}$ , respectively.
Topic Word-sets: Four SMEs were recruited from a multidisciplinary pool of researchers who were representative of the political-ideological spectrum and who were Australian English speakers. They were shown the same topics consisting of the top-10 words ranked by term frequency that were generated by LDA and MetaLDA on AP, AWH, and AWM for  $K = 10 - 60$  topics $^6$ , producing a total of 3,120 labels (780 for each SME) generated for the 390 topics (130 per model-dataset combination). Their task was to provide a descriptive label for each  $T_{ws}$  and to use 'NA' if they were unable to provide a label. Appendix A provides an example of this task. Two measures were constructed from these labels. The first was the number of raters able to label the topic, a count between 0-4 denoted  $Q_{nbr}$ . The second was a simple ICR measure, Percentage Agreement denoted  $Q_{agr}$ , which calculated as the number of times a set of annotators agree on a label, divided by the total number of annotations, as a percentage.
Topic Document-collections: Two SMEs analyzed the  $T_{dc}$ s of the 60 topics each modeled by LDA and MetaLDA on the AP dataset, referred to hereafter as the qualitative set. Samples of  $T_{dc}$  generated by each model (K=10–60) were reviewed, and those generated from both models 60-topic sets
1Tweet IDs and pre-processing details are available at: https://github.com/wboutine/auspoldata 2https://github.com/ethanhezhao/MetaLDA 3http://aksw.org/Projects/Palmetto.html
4http://www.abc.net.au/news/archive 5For word proportion vector  $\vec{p}$  this is  $e^{-Entropy(\vec{p})}$  6The AWMH dataset was not included.
were found to be of equal or higher quality than those produced by other values of  $K$ .
The SMEs reviewed the top-30 tweets representative of a topic and provided a label for each tweet. They then inductively determined a label or phrase describing that  $T_{dc}$ . They noted any key phrases, names, or other terms that were consistent across the collection. The SMEs were experienced at annotating such datasets and were familiar with the online #Auspol community. The SMEs then discussed the results together and agreed on a final label for each  $T_{dc}$ .