SlowGuess/ACL_2020_OCR huggingface dataset:
Felipe Viegas<sup>1</sup>, Washington Cunha<sup>1</sup>, Christian Gomes<sup>1</sup>, Antonio Pereira<sup>2</sup>, Leonardo Rocha<sup>2</sup>, Marcos Andre Gonçalves<sup>1</sup>
<sup>1</sup> Universidade Federal de Minas Gerais - Brazil  
<sup>2</sup> Universidade Federal de São João del Rei - Brazil  
{frviegas, washingtoncunha}@dcc.ufmg.br  
{christianreis, mgoncalv}@dcc.ufmg.br  
{antoniopereira, lcrocha}@ufsj.edu.br
Hierarchical Topic modeling (HTM) exploits latent topics and relationships among them as a powerful tool for data analysis and exploration. Despite advantages over traditional topic modeling, HTM poses its own challenges, such as (1) topic incoherence, (2) unreasonable (hierarchical) structure, and (3) issues related to the definition of the "ideal" number of topics and depth of the hierarchy. In this paper, we advance the state-of-the-art on HTM by means of the design and evaluation of CluHTM, a novel nonprobabilistic hierarchical matrix factorization aimed at solving the specific issues of HTM. CluHTM's novel contributions include: (i) the exploration of richer text representation that encapsulates both, global (dataset level) and local semantic information – when combined, these pieces of information help to solve the topic incoherence problem as well as issues related to the unreasonable structure; (ii) the exploitation of a stability analysis metric for defining the number of topics and the "shape" the hierarchical structure. In our evaluation, considering twelve datasets and seven state-of-the-art baselines, CluHTM outperformed the baselines in the vast majority of the cases, with gains of around  $500\%$  over the strongest state-of-the-art baselines. We also provide qualitative and quantitative statistical analyses of why our solution works so well.
Topic Modeling (TM) is the task of automatically extracting latent topics (e.g., a concept or a theme) from a collection of textual documents. Such topics are usually defined as a probability distribution over a fixed vocabulary (a set of words) that refers to some subject and describes the latent topic as a whole. Topics might be related to each other, and if they are defined at different semantic granularity levels (more general or more specific), this naturally induces a hierarchical structure. Although
traditional TM strategies are of great importance to extract latent topics, the relationships among them are also extremely valuable for data analysis and exploration. In this context, Hierarchical Topic Modeling (HTM) aims to achieve – to induce latent topics from text data while preserving the inherent hierarchical structure (Teh et al., 2006). Relevant scenarios have been shown to enjoy the usefulness of HTM, such as (i) hierarchical categorization of Web pages (Ming et al., 2010), (ii) extracting aspects hierarchies in reviews (Kim et al., 2013) and (iii) discovering research topics hierarchies in academic repositories (Paisley et al., 2014).
Despite its practical importance and potential advantages over traditional TM, HTM poses its own challenges, the main ones being: (i) topic incoherence and (ii) unreasonable hierarchical structure. Topic Incoherence has to do with the need to learn meaningful topics. That is, the top words that represent a topic have to be semantically consistent with each other. Unreasonable structure is related to the extracted hierarchical topic structure. Topics near the root should be more general, while topics close to the leaves should be more specific. Furthermore, child topics must be coherent with their corresponding parent topics, guaranteeing a reasonable hierarchical structure. Finally, (iii) the number of topics in each hierarchy level is usually unknown and cannot be previously set to a predefined value since it directly depends on the latent topical distribution of the data.
Both supervised and unsupervised approaches have been applied to HTM. Supervised methods use prior knowledge to build the hierarchical tree structure, such as labeled data or linking relationships among documents (Wang et al., 2015). Those strategies are unfeasible when there is no explicit taxonomy or hierarchical scheme to associate with documents or when such an association (a.k.a., labeling) is very cumbersome or costly to obtain. Unsupervised HTM (uHTM)
8138
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8138-8150 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
deals with such limitations. uHTM methods do not rely on previous knowledge (such as taxonomies or labeled hierarchies), having the additional challenge of discovering the hierarchy of topics based solely on the data at hand.
HTM solutions can also be roughly grouped into non-probabilistic and probabilistic models. In probabilistic strategies, textual data is considered to be "ruled" by an unknown probability distribution that governs the relationships between documents and topics, hierarchically. The major drawback in this type of approach has to do with the number of parameters in the model, which rapidly grows with the number of documents. This leads to learning inefficiencies and proneness to over-fitting, mainly for short textual data (Tang et al., 2014). To overcome these drawbacks, non-probabilistic models aim at extracting hierarchical topic models through matrix factorization techniques instead of learning probability distributions. Such strategies also pose challenges. They are usually limited to just local information (i.e., data limitation) as they go deeper into the hierarchy when extracting the latent topics. That is, as one moves more in-depth in the hierarchical structure representing the latent topics, the available data rapidly reduces in size, directly impacting the quality of extracted topics (in terms of both coherence and structure reasonableness). Probabilistic models mitigate this phenomenon as they rely on global information when handling the probability distributions(Xu et al., 2018). Because of that, the current main HTM methods are built based on probabilistic methods (Griffiths et al., 2004; Mimno et al., 2007).
In this paper, we aim at exploring the best properties of both non-probabilistic and probabilistic strategies while mitigating their main drawbacks. Up to our knowledge, the only work to explore this research venue is (Liu et al., 2018). In that work, the authors explore NMF for solving HTM tasks by enforcing three optimization constraints during matrix factorization: global independence, local independence, and information consistency. Those constraints allow their strategy, named HSOC, to produce hierarchical topics that somehow preserve topic coherence and reasonable hierarchical structures. However, as we shall see in our experiments, HSOC is still not capable of extracting coherent topics when applied to short text data, which is currently prominent on the Web, especially on social network environments.
We here propose a distinct approach, taking a data engineering perspective, instead of focusing on the optimization process. More specifically, we explore a matrix factorization solution properly designed to explore global information (akin to probabilistic models) when learning hierarchical topics while ensuring proper topic coherence and structure reasonableness. This strategy allows us to build a data-efficient HTM strategy, less prone to over-fitting that also enjoys the desired properties of topic coherence and reasonable (hierarchical) structure. We do so by applying a matrix factorization method over a richer text representation that encapsulates both, global and semantic information when extracting the hierarchical topics.
Recent non-probabilistic methods (Shi et al., 2018; Viegas et al., 2019) have produced top-notch results on traditional TM tasks by taking advantage of semantic similarities obtained from distances between words within an embedding space (Mikolov et al., 2013; Pennington et al., 2014). Our critical insight for HTM was to note that the richer (semantic) representation offered by distributional word embeddings can be readily explored as a global source of information in more profound levels of the hierarchical structure of topics. This insight gives us an essential building block to overcome the challenges of matrix factorization strategies for HTM without the need for additional optimization constraints.
In (Viegas et al., 2019), the authors exploit the nearest words of a given "pre-trained" word embedding to generate "meta-words", aka Cluwords, able of expanding and enhancing the document representation in terms of syntactic and semantic information. Such an improved representation is capable of mitigating the drawbacks of using the projected space of word embeddings as well as extracting cohesive topics when applying non-negative matrix factorization for topic modeling.
Motivated by this finding, we here advance the state-of-the-art in HTM, by designing, developing and evaluating an unsupervised non-probabilistic HTM method that exploits CluWords as a key building block for TM when capturing the latent hierarchical structure of topics. We focus on the NMF method for uncovering the latent hierarchy as it is the most effective matrix factorization method for our purposes. Finally, the last aspect needed
<sup>1</sup>Distances in the embeddings space are global as they do consider the whole vocabulary and interactions among words in specific contexts.
8139
to be addressed for the successful use of NMF for HTM is the definition of the appropriate number of topics  $k$  to be extracted. Choosing just a few topics will produce overly broad results while choosing too many will result in over-clustering the data into many redundant, highly-similar topics. Thus, our proposed method uses a stability analysis concept to automatically select the best number of topics for each level of the hierarchy.
As we shall see, our approach outperforms HSOC and hLDA (current state-of-the-art) for both small and large text datasets, often by large margins. To summarize, our main contributions are: (i) a novel non-probabilistic HTM strategy - CluHTM - based on NMF and CluWords that excels on HTM tasks (in both short and large text data) while ensuring topic coherence and reasonable topic hierarchies; (ii) the exploitation in an original way of a cross-level stability analysis metric for defining the number of topics and ultimately 'the shape' of the hierarchical structure; as far as we know this metric has never been applied with this goal; (iii) an extensive empirical analysis of our proposal considering twelve datasets and seven state-of-the-art baselines. In our experimental evaluation, CluHTM outperformed the baselines in the vast majority of the cases (In case of NPMI, in all cases), with gains of  $500\%$  when compared to hLDA and  $549\%$  when compared to HSOC, some of the strongest baselines; and finally, (iv) qualitative and quantitative statistical analyses of the individual components of our solution.
Hierarchical Topic Modeling (HTM) can be roughly grouped into supervised and unsupervised methods. Considering the supervised HTM strategies, we here highlight some relevant supervised extensions to the traditional Latent Dirichlet Allocation (LDA) (Blei et al., 2003), a widely used strategy for the topic modeling (TM). LDA assumes a Dirichlet probability distribution over textual data to estimate the probabilities of words for each topic. In (Mcauliffe and Blei, 2008), the authors propose SLDA, a supervised extension of LDA that provides a statistical model for labeled documents. SLDA allows connecting each document to a regression variable to find latent topics that will best predict the response variables for future unlabeled documents. Based on SLDA, Hierarchical Supervised LDA (HSLDA) (Perotte
et al., 2011) incorporates the hierarchy of multi-label and pre-labeled data into a single model, thus providing extended prediction capabilities w.r.t., the latent hierarchical topics. The Supervised Nested LDA (SNLDA) (Resnik et al., 2015), also based on SLDA, implements a generative probabilistic strategy where topics are sampled from a probability distribution. SNLDA extends SLDA by assuming that the topics are organized into a tree structure. Although our focus is on unsupervised solutions, we include SLDA, HSLDA and SNLDA as baselines in our experimental evaluation.
We now turn our attention to unsupervised HTM strategies, in which a hierarchical structure is learned during topic extraction. In (Mimno et al., 2007) the authors propose Hierarchical Pachinko Allocation Model (hPAM), an extension of Pachinko Allocation (PAM) (Li and McCallum, 2006). In PAM, documents are a mix of distributions over an individual topic set, using a directed acyclic graph to represent the co-occurrences of topics. Each node in such a graph represents a Dirichlet distribution. At the highest level of PAM, there is only a single node, where the lowest levels represent a distribution between nodes of the next higher level. In hPAM, each node is associated with a distribution over the vocabulary of documents.
In (Griffiths et al., 2004), the authors propose the hLDA algorithm, which is also an expansion of LDA, being considered state-of-the-art in HTM. In hLDA, in addition to using the text Dirichlet distribution, the nested Chinese Restaurant Process (nCRP) is used to generate a hierarchical tree. NCRP needs two parameters: the tree level and a  $\gamma$  parameter. At each node of the tree, a document can belong to a path or create a new tree path with probability controlled by  $\gamma$ . More recently, in (Xu et al., 2018), the authors propose the unsupervised HTM strategy named a knowledge-based hierarchical topic model (KHTM). This method is based on hLDA and, as such, models a generative process whose parameter estimation strategy is based on Gibbs sampling. KHTM is able to uncover prior knowledge (such as the semantic correlation among words), organizing them into a hierarchy, consisting of knowledge sets (k-sets). More specifically, the method first generates, through hLDA, an initial set of topics. After comparing pairs of topics, those topics with similarity higher than  $\alpha$  (a.k.a., k-sets) are then filtered so that the first 20 words of each topic are kept, and the remaining are just discarded. Those
extracted k-sets are then used as an extra weight when extracting the final topics. All these methods are used as baselines in our experimentation.
Probably the most similar work to ours is the HSOC strategy, proposed in (Liu et al., 2018), which proposes to use NMF for solving HTM tasks. In order to mitigate the main drawbacks of NMF in the HTM setting $^{2}$ , HSOC relies on three optimization constraints to properly drive the matrix factorization operations when uncovering the hierarchical topic structure. Such constraints are global independence, local independence, and information consistency, and allow HSOC to derive hierarchical topics that somehow preserve topic coherence and reasonable hierarchical structures.
As it can be observed, almost all models, supervised or unsupervised, are based on LDA. As discussed in Section 1, though matrix factorization strategies normally present better results than Dirichlet strategies in TM tasks, for HTM, the situation is quite different. In fact, matrix factorization methods face difficult challenges in HTM, mainly regarding data size as ones go deeper into the hierarchy. More specifically, at every hierarchical level, a matrix factorization needs to be applied to increasingly smaller data sets, ultimately leading to insufficient data at lower hierarchy levels. These approaches also do not exploit semantics nor any external enrichment, relying only on the statistical information extracted from the dataset. Contrarily, here we propose a new HTM approach, called CluHTM, which exploits externally built word embedding models to incorporate global semantic information into the hierarchical topic tree creation. This brings some important advantages to our proposal in terms of effectiveness, topic coherence, and hierarchy reasonableness altogether.
3.1 CluWords Representation
Cluwords (Viegas et al., 2019) combine the traditional Bag of Words (BoW) statistical representation with semantic information related to the words present in the documents. The semantic context is obtained employing a "pre-trained" word representation, such as Fasttext (Mikolov et al., 2018). Figure 1 presents the process of transforming each original word into a Cluword
(cluster of words) representation. First, the strategy uses the information about the dataset, as well as pre-trained word embedding (i.e. Fasttext) to build semantic relationships between a word and its neighbors (described in Section 3.1.1). Next, statistical information on words (e.g., term frequency, document frequency) is extracted from the dataset. Then, both semantic and statistical information are combined to measure the importance of each Cluword as explained in Section 3.1.2. Cluwords enjoy the best of "two worlds": it conjugates statistical information on the dataset, which has demonstrated to be very effective, efficient and robust in text applications, enriched with semantic contextual information captured by distributional word embeddings adapted to the dataset by the clusterization process described next.
Figure 1: Diagram showing the steps for building the CluWords representation.
3.1.1 Cluwords Generation
Let  $\mathcal{W}$  be the set of vectors representing each word  $t$  in the dataset vocabulary (represented as  $\nu$ ). Each word  $t \in \mathcal{V}$  has a corresponding vector  $u \in \mathcal{W}$ . The CluWords representation is defined as in Figure 1. The semantic matrix in the Figure 1 is defined as  $C \in \mathbb{R}^{|\mathcal{V}| \times |\mathcal{V}|}$ , where each dimension has the size of the vocabulary  $(|\mathcal{V}|)$ ,  $t'$  represents the rows of  $C$  while  $t$  represents the columns. Finally, each index  $C_{t', t}$  is computed according to Eq. 1.
$$
C _ {t ^ {\prime}, t} = \left\{ \begin{array}{l l} \omega \left(u _ {t ^ {\prime}}, u _ {t}\right) & \text {i f} \omega \left(u _ {t ^ {\prime}}, u _ {t}\right) \geq \alpha \\ 0 & \text {o t h e r w i s e ,} \end{array} \right. \tag {1}
$$
where  $\omega (u_{t'},u_t)$  is the cosine similarity and  $\alpha$  is a similarity threshold that acts as a regularizer for the representation. Larger values of  $\alpha$  lead sparser representations. In this notation each column  $t$  of the semantic matrix  $C$  will be forming a CluWord  $t$  and each value of the matrix  $C_{t',t}$  may receive the cosine similarity between the vectors  $u_{t'}$  and  $u_{t}$  in the embedding space  $\mathcal{W}$  if it is greater than
<sup>2</sup>Namely, the incoherence of topics and unreasonable hierarchical structure caused by the lack of a learned probability distribution that governs the document/topics relationships
8141
or equal to  $\alpha$ . Otherwise, the  $C_{t', t}$  receives zero, according to the Eq. 1.
3.1.2 TFIDF Weight for CluWords
In Figure 1, the CluWords representation is defined as the product between the statistical matrix (a.k.a. term-frequency matrix) and semantic matrix  $C$ . The statistical matrix  $(TF)$  can be represented as a  $TF \in \mathbb{R}^{|\mathcal{D}| \times |\mathcal{V}|}$ , where each position  $TF_{d,t}$  relates to the frequency of a word  $t$  in document  $d$ . Thus, given a CluWord (CW)  $t$  for a document  $d$ , its data representation corresponds to  $\mathrm{CW}_{d,t} = \overrightarrow{TF_d} \times \overrightarrow{C_{,t}}$ , where  $\overrightarrow{TF_d}$  has the term-frequencies of document  $d$ , and  $\overrightarrow{C_{,t}}$  is the semantic scores for the CluWord  $t$ , according to Eq. 1.
The TFIDF weighting for a CluWord  $t$  in a document  $d$  is defined as  $CW_{d,t} = CW_{d,t}\times idf_t$ . The IDF component is defined as  $idf_t = \log \left(\frac{|\mathcal{D}|}{\sum_{1\leq d\leq |\mathcal{D}|}\mu_{t,d}}\right)$ , where  $\mathcal{D}$  is the number of documents and  $\mu_{t,d}$  is the average of semantic weights of the semantic matrix  $C$  for the CluWord  $t$  ( $\overrightarrow{C}_{,t}$ ) that occurs in the vocabulary  $\nu_{d}$ . The average  $\mu_{t,d}$  is defined as  $\mu_{t,d} = \frac{1}{|\nu_{d,t'}|}\cdot \sum_{t'\in (\nu_d\cap \overrightarrow{C}_{,t})}C_{t',t}$ .
3.2 Stability Measure
The Stability measure is motivated by the term-centering approach generally taken in topic modeling strategies, where topics are usually summarized as a truncated set of top words (Greene et al., 2014).
The intuition behind this strategy is, given some  $K$  topics, to measure whether running multiple random samplings for a topic modeling strategy results in Stability, in terms of  $p$  top words extracted from the topics. Given a range of topics  $[\mathcal{K}_{min},\mathcal{K}_{max}]$ , and some topic modeling strategy (on our case, Non-negative Factorization Matrix method), the strategy proceeds as follows. First, it learns a topic model considering the complete data set representation  $\mathcal{D}$ , which will be used as a reference point  $(\mathcal{W}_D)$  for analyzing the Stability afforded by the  $K$  topics. Note that the  $p$  top words represent each topic. Subsequently,  $\mathcal{S}$  samples of the data are randomly drawn from  $\mathcal{D}$  without replacement, forming a subset of  $\mathcal{D}'$  documents. Then,  $|\mathcal{S}|$  topic models are generated, one for each subsampling  $(\mathcal{W}_{\mathcal{S}_i})$ .
To measure the quality of  $K$  topics, the Stability computes the mean agreement among each pair of  $(\mathcal{W}_{\mathcal{D}},\mathcal{W}_{S_i})$ . The goal is to find the best match between the  $p$  top words of the compared topics. The agreement is defined as agree  $(\mathcal{W}_x,\mathcal{W}_y) = \frac{1}{p}\sum_{i = 1}^{p}AJ(w_{xi},\rho (w_{xi}))$ , where  $AJ(\cdot)$  is the av
erage Jaccard coefficient used to compare the similarity among the words  $w$  and  $\rho(\cdot)$  is the optimal permutation of the words in  $\mathcal{W}_{S_i}$  that can be found in  $\mathbb{O}(p^3)$  time by solving the minimal weight bipartite matching problem using the Hungarian method (Kuhn, 2010).
4 Proposed Solution
CluHTM is an iterative method able to automatically define the best number of topics in each hierarchy, given a range of possible number of topics  $\left[\mathcal{K}_{min},\mathcal{K}_{max}\right]$ . CluHTM explores Cluwords and Non-negative Matrix Factorization (NMF) (Lee and Seung, 2001), one of the main non-probabilistic strategies. Finally, the Stability method (described in Section 3) is used to select NMF  $k$  parameters (a.k.a number of topics).
CluHTM has five inputs (Algorithm 1), (i)  $\mathcal{D}_{max}$  corresponds to the depth down to which we want to extract the hierarchical structure. (ii)  $\kappa_{min}$  and  $\kappa_{max}$  control the range of some topics, such range will be used in all levels of the hierarchy; (iii)  $\mathcal{T}$  is the input text data; and (iv)  $\mathcal{W}$  is the "pre-trained" word embedding vector space used in the CluWords generation. The output is the hierarchical structure  $\mathcal{H}$  of  $p$  top words for each topic.
The method starts by getting the root topic (line 2-3 of Algorithm 1), which is composed of all documents in  $\mathcal{T}$ . Since the method is iterative, each iteration is controlled by a queue schema to build a hierarchical structure. Thus, at each iteration (line 3), the algorithm produces the
8142
CluWords representation for the documents  $\in \mathcal{T}'$  (line 5), chooses the number of topics, exploiting the Stability measure (line 6), and runs the NMF method (line 7) to extract the  $p$  words for each topic in  $\mathcal{O}$  (line 8). Then, in the loop of line 9, each topic is stored in the queue, as well as the respective documents of each topic.
Summarizing, our solution exploits global semantic information (captured by CluWords) within local factorizations, limited by a stability criterion that defines the 'shape' of the hierarchical structure. Though simple (and original), the combination of these ideas is extremely powerful for solving the HTM task, as we will see next.