Scale-Invariant Infinite Hierarchical Topic Model
Year: 2023

Abstract:
Hierarchical topic models have been employed to organize a large number of diverse topics from corpora into a latent tree structure. However, existing models yield fragmented topics with overlapping themes whose expected probability becomes exponentially smaller along the depth of the tree. To solve this intrinsic problem, we propose a scale-invariant infinite hierarchical topic model (ihLDA). The ihLDA adaptively adjusts the topic creation to make the expected topic probability decay considerably slower than that in existing models. Thus, it facilitates the estimation of deeper topic structures encompassing diverse topics in a corpus. Furthermore, the ihLDA extends a widely used tree-structured prior (Adams et al., 2010) in a hierarchical Bayesian way, which enables drawing an infinite topic tree from the base tree while efficiently sampling the topic assignments for the words. Experiments demonstrate that the ihLDA has better topic uniqueness and hierarchical diversity than existing approaches, including state-of-the-art neural models.

heading: 1 Introduction
text: Topic models (Blei et al., 2003b; Blei and Lafferty, 2006; Chang and Blei, 2010; Roberts et al., 2016) have been used to summarize, annotate, and categorize documents. Recent advances in large-scale topic models have enabled the estimation of thousands of topics to accommodate various concepts in a large corpus (Li et al., 2014; Yu et al., 2015; Yuan et al., 2015; Chen et al., 2016), requiring users to interpret numerous topics.
Hierarchical topic models have been proposed to improve the topic organization by learning the latent topic hierarchy (Blei et al., 2003a, 2010; Adams et al., 2010; Kim et al., 2012; Paisley et al., 2015; Isonuma et al., 2020; Chen et al., 2021). However, these hierarchical topic models will create a fragmented tree structure with the probabilities of a substantial number of topics becoming
exponentially smaller. These topics typically have few assigned words and similar word distributions. Recent hierarchical topic models with neural architectures (Isonuma et al., 2020; Chen et al., 2021) have the same issue of topic fragmentation and use a fixed number of layers for all documents (Duan et al., 2021).
The reason for the topic fragmentation is that the stick-breaking process (Sethuraman, 1994) used in existing models creates topics whose expected probability decays along the depth of the tree. Existing models alleviate the issue by restricting the tree structure, for example by truncating the depth to three levels. Isonuma et al. (2020) also introduced a topic-diversity regularizer and a heuristic rule to update topics, whereas Chen et al. (2021) truncated topics based on their corpus coverage.
To address this intrinsic issue of topic probabilities, we propose a scale-invariant hierarchical infinite topic model (ihLDA) and make three main contributions. First, the ihLDA adjusts the probability scale of the stick-breaking process at each level by considering the size of the parent topic to avoid fragmented topic structures. The expected topic probability of the ihLDA decays considerably slower than that of the existing models, thereby reflecting the diversity of topics in a corpus by using a flexible depth and width. The existing probabilistic and neural models that leverage the stick-breaking process can also benefit from our model.
Table 1 compares the top words of several topics from different topic models: the ihLDA estimates topics with general words in the shallower levels (L1 and L2), and topics with more specific words at a deeper level (L3). In contrast, nCRP and TSNTM will create topics with overlapping themes. The columns of nCRP and TSNTM show that most topics share the top words, even at the third level because of the issue described above.
Second, the ihLDA extends the tree-structured stick-breaking process (TSSB; Adams et al., 2010),
11731
a prior for a latent hierarchy that is also employed in recent neural models and various applications (Deshwar et al., 2015; Chien, 2016; Nassar et al., 2019). The ihLDA enables drawing an infinite topic tree for each document from a base infinite tree in a hierarchical Bayesian fashion.
Finally, we implement an efficient algorithm that can draw the topics and hierarchical structures from the tree-structured prior without enumerating all possible candidates.
We empirically show that the ihLDA performs better in topic quality using two measures and crowdsourced evaluation. Moreover, the number of estimated topics by the ihLDA is comparable to that by existing models, even when a tree is deeper than three levels.

heading: 2 Background: Tree-Structured Stick-Breaking Process
text: A tree-structured stick-breaking process (TSSB) (Adams et al., 2010) is a prior for constructing a topic tree of theoretically unbounded depth and width, comprising two types of stick-breaking processes (Sethuraman, 1994). Figure 1 illustrates a draw from the TSSB, where each blue interval represents a topic, whereas the square brackets denote the path to reach it.1 Hierarchical topic models
1This path notation is adapted from Isonuma et al. (2020) and is different from that in Adams et al. (2010).
assign a latent topic to each word in a document. As an equivalent representation of the Dirichlet process (see Appendix A for details), a stickbreaking process repeatedly breaks a stick of length 1, where each broken stick corresponds to a topic with the length equal to its probability. Appendix B provides a formal definition and illustration of the stick-breaking process.
Here, we introduce notation to formalize the TSSB. Topic ϵ at the level |ϵ| in a tree has its ancestors and children. Let κ≺ ϵ indicate that κ is an ancestor of ϵ: in Figure 1, topic [1 1 1] has two ancestors, {κ : κ≺ [1 1 1]} = {[1], [1 1]}. Specifically, we use a prime symbol ′ to denote the parent topic, i.e., [1 1 1]′ = [1 1]. The child topics of ϵ are {ϵk : k ∈ 1, 2, 3, . . .}. For example, topic [1 2] in Figure 1 has children [1 2 1], [1 2 2], · · · .
Given this setup, the probability assigned to a topic ϵ under TSSB can be expressed as a product of stick-breaking processes:
πϵ = νϵ ∏
κ≺ϵ (1−νκ) ·
∏ κ⪯ϵ ϕκ , (1)
where ϕϵk = ψϵk ∏k−1
j=1(1−ψϵj). The first term in Equation (1) is the probability of stopping at the topic ϵ vertically. The next product terms refer to passing ancestors of ϵ while horizontally stopping at ϵ and its ancestors. These vertical and horizontal probabilities of stopping follow Beta distributions:
νϵ ∼ Be(1, α0) , ψϵ ∼ Be(1, γ0) . (2) Appendix C presents an example of this process.
We also introduce a scaling factor λ used in Adams et al. (2010) and set α0 at each level, αϵ=α|ϵ| ·λ|ϵ|−1, 0≤λ ≤ 1, instead of α0 in Equation (2). This parametrization makes a word more likely to stop as |ϵ| becomes larger, i.e., deeper in the tree. Hereafter, we do not use subscript ϵ and denote αϵ as α for simplicity.

heading: 3 Scale-Invariant TSSB
text: Although the TSSB constitutes a crucial building block of recent hierarchical topic models (Isonuma et al., 2020; Chen et al., 2021), the expected probability of each topic in the TSSB decays exponentially along the depth of the topic hierarchy. Figure 2(a) shows two topic trees drawn from the original TSSB: topics in the third and the fourth levels have extremely small probabilities compared to the topics in the higher levels, resulting in a topic fragmentation in the tree.
This property of the TSSB is attributed to the probability of a horizontal stop, ψϵ, having the same expectation regardless of the level. As shown in Appendix D, the expected probability of a horizontal break at the level ℓ is E[ϕ |ℓ]≈1/(2γ+1)ℓ, where the level appears in the exponent of the denominator. The dotted line in Figure 3 depicts this exponential decay with ℓ.
To avoid this exponential decay, we rescale γ0 in Equation (2):
ψϵ ∼ Be(1, ϕϵ′γ0), (3)
where we set ϕϵ′ = 1 when ϵ is the root topic. Hereafter, we denote γ=γϵ=ϕϵ′γ0 for simplicity.
The key idea in Equation (3) is to use the horizontal breaking proportion of a parent topic, ϕϵ′ , to draw a relative stick length for its child topic, ψϵ, creating a larger break if the stick to break is already short. As presented in Appendix D, this new parametrization yields the average stick length E[ϕ |ℓ]≈1/(2γ + 1/E[ϕ |ℓ−1]) for ℓ ≥ 2, which achieves an invariant partitioning scale by not decaying exponentially with ℓ. The solid lines in Figure 3 depict the effect of this new parametrization.
Figure 2(b) shows our scale-invariant TSSB with the same hyperparameters as in (a). The probability of the topics is less likely to decrease at the deeper levels in (b).

heading: 4 Scale-Invariant Infinite Hierarchical Topic Model
text: We employ our scale-invariant TSSB to model the hierarchical latent topics in a document. Topic models consist of two types of distributions: documenttopic distributions for topic composition and topicword distributions for word emission. The ihLDA leverages the scale-invariant TSSB in Section 3 to construct the former and employs a hierarchical Pitman-Yor process (Teh, 2006) for the latter. Combining both distributions will embed the topics into an infinite tree, which we call the ihLDA, scale-invariant infinite hierarchical LDA.

heading: 4.1 Document-Topic Distribution
text: The topic composition of each document differs for each document, but the topics must be shared across all the documents. In this regard, we generalize the scale-invariant TSSB to a hierarchical treestructured stick-breaking process (HTSSB). The HTSSB generates document-specific topic probabilities while making these topics shared by all documents.
Specifically, we hierarchically generate a child TSSB for a document from the base TSSB, as shown in Figure 4. It applies the hierarchical Dirichlet process (HDP; Teh et al., 2006) sepa-
rately to the vertical and horizontal probabilities that constitute the TSSB in Equations (2) and (3). In this regard, the HTSSB is an infinite product of the HDPs in terms of its component probabilities. Appendix E provides a formal explanation of the HDP in a topic model context.
We use the tilde symbol (˜) to denote a corresponding topic in the base TSSB. When ϵ is a topic (say, ϵ=[1 1 4]) in a child TSSB, ϵ̃ represents the same topic ([1 1 4]) in the base TSSB. We can determine the probabilities for vertical stopping at node ϵ as follows, based on the theory of the HDP (Teh et al., 2006): νϵ∼Be(aτϵ̃, a(1− ∑ κ⪯ϵ̃ τκ)) where
τϵ = νϵ ∏
κ≺ϵ(1−νκ). Similarly, the probability for horizontal stopping at the k’th child of ϵ is ψϵk∼Be(bϕϵ̃k, b(1− ∑k j=1 ϕϵ̃j)). We can draw a topic tree, π, for each document with these vertical and horizontal probabilities by using Equation (1). Note that the topic assignments in each document affect the base TSSB because each TSSB shares the same topics across the documents in the HTSSB.

heading: 4.2 Topic-Word Distributions
text: The hierarchical Pitman-Yor process (HPY; Teh, 2006) provides the semantic similarity between a
parent topic and its children while increasing the specificity of the topics as the tree deepens. Let Hϵ be the probability distribution over words for topic ϵ. We use a Pitman-Yor process (Pitman and Yor, 1997; Goldwater et al., 2005) as a prior for Hϵ: Hϵ ∼ PY(d|ϵ|, θ|ϵ|, Hϵ′). We repeat this process until it reaches the root of the topic tree where we use H0 as a prior: H[1] ∼ PY(d0, θ0, H0). If the size of the lexicon is V , we set H0=1/V for all words in the corpus. The tree structure of the topic-word distribution is the same as that of the base TSSB. Thus, all topics in a document have a corresponding topic-word distribution because a child TSSB is drawn from the base TSSB for each document.

heading: 4.3 Data Generation Process
text: We summarize the generation process of the documents in the ihLDA as follows: Let π(d) specify a TSSB for a document d.
1. Draw a base TSSB π̃. 2. Draw topic-word distributions Hϵ from the
HPY for each topic in π̃. 3. Draw a document-topic distribution for each
document d, π(d) ∼ HTSSB(π̃). 4. For each word position i in a document d,
• draw a topic, zdi ∼ π(d), and • draw a word, wdi ∼ Hzdi .

heading: 5 Inference

heading: 5.1 Vertical and Horizontal Probabilities
text: The vertical and horizontal probabilities in Equations (2) and (3) determine the topic hierarchy. We employ the Chinese restaurant district process (CDP; Paisley and Carin, 2009, see Appendix F) representation of the Dirichlet process for each document-specific topic structure (child TSSB) and a shared topic structure (base TSSB).
We count the number of words that have stopped at a topic ϵ as n0(ϵ) for a vertical stop and m0(ϵ) for a horizontal stop, along with the number of words that have passed ϵ as n1(ϵ) for a vertical pass and m1(ϵ) for a horizontal pass. In addition, we define n(ϵ)=n0(ϵ)+n1(ϵ) and m(ϵ)= m0(ϵ)+m1(ϵ). After conditioning on the observed data and the rest of the probabilities, we can obtain the expectation of the posterior of the vertical and horizontal probabilities as: ν̂ϵ = E[νϵ |rest] = (1+n0(ϵ))/(1+α+n(ϵ)) and ψ̂ϵ = E[ψϵ |rest] = (1 +m0(ϵ))/(1 + γ +m(ϵ)). Finally, using Equation (1), we can compute the expectation of the
posterior πϵ as E[πϵ |rest] = ν̂ϵ ∏ κ≺ϵ ( 1 − ν̂κ ) ·∏
κ⪯ϵ ϕ̂ϵ where ϕ̂ϵk = ψ̂ϵk ∏k−1
j=1(1− ψ̂ϵj). Following the same idea, we apply a hierarchical CDP to the HTSSB. More specifically, when a word w stops vertically at a topic ϵ in a child TSSB for a document, we probabilistically update the counts of the corresponding topic ϵ̃ in the base TSSB with a probability proportional to aνϵ̃/(a+n(ϵ)). We update the count only in the child TSSB with a probability proportional to n(ϵ)/(a+n(ϵ)). Horizontal probabilities have the same count update process: a probability proportional to bψϵ̃/(b+m(ϵ)) is used to update the base TSSB and m(ϵ)/(b+m(ϵ)) for the child TSSB. The expectation of the posterior vertical and horizontal probabilities in a document d is similar to that shown above,
E [ ν (d) ϵ |rest ] =
aτϵ̃ + n0(ϵ)
a(1−∑κ≺ϵ̃ τκ) + n(ϵ) ,
E [ ψ (d) ϵk |rest ] =
bϕϵ̃k +m0(ϵk)
b(1−∑k−1j=1 ϕϵ̃′j) +m(ϵk) .
We employ slice sampling (Neal, 2003)2 to estimate all hyperparameters in our model, that is, {α|ϵ|, γ0, λ, a, b}.

heading: 5.2 Topic Assignments
text: The ihLDA has an infinite number of topics, and all possible topics in a tree cannot be enumerated. Our Gibbs sampling strategy implements a combination of retrospective sampling (Papaspiliopoulos and Roberts., 2008) and binary search, which follows the original approach used in the TSSB (Adams et al., 2010). The key observation is that each topic in a tree takes a certain share of a stick of length 1 (see Figure 1). Therefore, we draw a uniform random variable, u∼Unif [0, 1), to find a random topic that corresponds to u. Algorithm 1 outlines the Gibbs sampling process of topic assignment for each word. The function does not need to enumerate all the topics, because it only compares the new likelihood q with the slice variable ρ. Algorithm 2 is a function for finding a topic that corresponds to a value in [0, 1). This function rescales u as it goes down the tree.

heading: 5.3 Other Parameters
text: Parameters in the HPY are also updated during the topic sampling in the HTSSB. Appendix B of Teh
2Specifically, we used the unbounded slice sampling (Mochihashi, 2020) to sample from [0,∞) effectively.
Algorithm 1 Gibbs sampling of a topic function sample_assignment(ϵ) a = 0; b = 1; ρ = Unif [0, 1) · p(ϵ) while True do
u = Unif [a, b) ϵ′ = find_topic(u, ϵroot) q = p(ϵ′) if q > ρ then return ϵ′ else if ϵ′ < ϵ then b = u else a = u
end if end while end function
Algorithm 2 Finding a topic function find_topic(u, ϵ) if u < νϵ then
return ϵ else
u = (u−νϵ)/(1−νϵ); k = 1 while True do
if u < 1−∏kj=1(1−ψϵj) then break else
k += 1 Create ϵk if necessary
end if end while u = (u−1)(1−ψϵk) + ∏k j=1(1−ψϵj)
ψϵk · ∏k
j=1(1−ψϵj) return find_topic(u, ϵk)
end if end function
(2006) provides an inference strategy for sampling θ and d used in the ihLDA.

heading: 6 Experiments

heading: 6.1 Data
text: In our experiments, we used the BBC News corpus (Greene and Cunningham, 2006), the 20News corpus (Lang, 1995), and the original Wikipedia corpus. The BBC News corpus contains 2,225 documents in five topic areas from the BBC news website, the 20News corpus is a collection of 18,828 posts from 20 USENET newsgroups, and the Wikipedia corpus comprises 50,153 English articles randomly sampled from ten main categories3 and their subcategories. We selected 80% of the data randomly for training.

heading: 6.2 Experimental Setup
text: We compared the ihLDA against two probabilistic and two neural topic models, namely, the nested
3Art, engineering, computer science, food, humanities, medicine, nature, social science, sports, and statistics
Chinese restaurant process (nCRP; Blei et al., 2003a), the recursive Chinese restaurant process (rCRP; Kim et al., 2012), the tree-structured neural topic model (TSNTM; Isonuma et al., 2020), and the nonparametric tree-structured neural topic model (nTSNTM; Chen et al., 2021). The publicly available replication codes for the rCRP, TSNTM, and nTSNTM were used along with a package for the nCRP (Lee, 2021). We used the default parameter values. As both neural models internally truncate the topics, the results were based on topics with at least 100 assigned words for a fair comparison. The maximum level of the ihLDA was six for BBC News and 20News even when we made the model unbounded but we truncated the tree at four for Wikipedia. All the experiments were conducted on a cluster computer with a Python 3 environment (Intel Xeon CPU 2.2-2.3 GHz and 10 GB RAM). We do not report the results of nCRP on Wikipedia and rCRP on 20News and Wikipedia, because they required more than two weeks to complete 10,000 iterations.

heading: 6.3 Numerical Evaluation
text: We employed two measures (TU and AO) from the existing literature and developed a new measure (TD) to compare the performance of the ihLDA with those of the existing approaches.
First, the topic uniqueness (TU) calculates the uniqueness of all topics (Nan et al., 2019; Masson and Montariol, 2020; Chen et al., 2021). A higher TU implies that the topics represent unique themes. Second, the average overlap (AO) measures the average repetition rate of the top u words between the parent topic and its children (Chen et al., 2021). A lower AO indicates that less overlap occurs between the top words from a parent and those from
its children. Although this measure was used in Chen et al. (2021), parent and child topics need some overlapping words to have semantic coherence; thus, a smaller AO does not always mean better interpretability. Appendix G provides formal definitions of these two measures.
Finally, the tree diversity (TD) is a new measure for assessing child topics as being unique, while considering the importance of the parent topics. Let T be a set of topics in the estimated tree, C(ϵ) be a set of topics that are the children of a topic ϵ, D(ϵ) be a set of topics that are descendants of a topic ϵ, and VN be a set of unique words that are used for the top u words of a set of topics N . We define TD as follows:
TD = ∑
ϵ∈T wϵ
|VC(ϵ)| u|C(ϵ)| ; wϵ = |D(ϵ)|∑ κ∈T |D(κ)| .
The fraction in TD is the proportion of unique words among the top words of the children of ϵ. Then it takes the sum of the fraction weighted by the normalized importance of each topic, that is, the proportion of descendants of ϵ. A higher TD is better because it implies that the top words in child topics contain more unique words.
Table 2 summarizes the results and the estimated number of topics. All metrics are calculated with different numbers of top words (u=5, 10, and 15), and we report their average. The ihLDA performs better than the existing models in terms of the TD and TU. Existing probabilistic models (nCRP and rCRP) create too many topics in comparison with the ihLDA, even though they truncate the topic tree at three levels. The ihLDA shows a reasonable number of topics even when it has a deeper tree without truncation, as shown in the parentheses. The two neural models, TSNTM and nTSNTM,
find fewer topics than the ihLDA but have lower performance in the BBC News and 20 News corpora and have a lot of redundancy as shown in Table 1. With the Wikipedia corpus, the ihLDA estimates 17 topics when the depth is fixed at three, which is a reasonable number given that the Wikipedia corpus is sampled from ten categories and their subcategories (see footnote 3).

heading: 6.4 Crowdsourced Evaluation
text: We devised three human evaluation tasks to assess both the interpretability and the hierarchical structure of the topics. An interpretable hierarchical topic model should show similarity between parent and child topics, while each child topic is coherent and distinctive from others.
The first task, Word Intrusion, is a slight alteration from the methods in Chang et al. (2009) and Ying et al. (2022). To measure the coherence of the estimated topics, crowdsourced workers observed four different word sets (each word set consists of four words). Three word sets were randomly selected from the top words of one topic, whereas the other set (the intruder) was randomly selected from those of a different topic that did not share the parent topic with the three word sets. The “correct” answer means that a worker identified the intruder word set.
The second task, Vertical, is an original task to measure the hierarchical structure. The workers observed four items and categorized them into two groups. We represented the items and groups with four words randomly chosen from the top words, where each item was a child topic of one of the groups. The “correct” answer means that a worker categorized a child topic into its parent topic.
The third task, Horizontal, is also an original task to measure horizontal distinctiveness. The workers grouped four items represented by four words randomly selected from the top words of topics that had the same parent topic. The same topic could appear in multiple items. The “correct” answer means that a worker categorized items from the same topic into the same group. If a model estimates overlapping topics, a worker cannot provide the correct answer.
We used the outputs from the BBC News corpus because news articles are accessible and familiar to crowdsourced workers from Amazon Mechanical Turk (Ying et al., 2022). We dropped workers who failed to pass our quality check questions and those
who spent too little (bottom 10%) or too much (top 10%) time4. Appendix H describes more details of crowdsourced experiments.
Figure 5 illustrates the proportion of correct answers, weighted to represent each level equally. The performance of the proposed model is at least statistically indistinguishable from the best existing model and better than the worst existing model in all tasks. nCRP exhibits competitive performance, but this is because it creates numerous specific topics even for a small corpus as shown in Table 2.

heading: 6.5 Estimated Tree Structure
text: Figure 6 displays π̃, the estimated global tree prior in the ihLDA. Both (a) and (b) show that topics do not decay significantly even if the maximum level is six.
Figure 7 presents the top five words for some topics that facilitate comparison between models. The ihLDA estimates topics with general words in the first and the second levels, and topics become more specific at lower levels. Figure 8 supports this topic specificity: proper nouns have higher
4The total number of observations was 535 (Word Intrusion), 900 (Vertical), and 620 (Horizontal).
probabilities at the bottom of the tree, whereas general terms appear more frequently at the top levels.

heading: 7 Related Work
text: The existing models have heuristically addressed the issue of topic fragmentation by truncating topics at a certain threshold and truncating the tree structure to a small number of levels. Adams et al. (2010) constructed a hierarchical document model with the TSSB5 and truncated the topics with less than 50 assigned documents when the corpus had 1740 documents. The nTSNTM (Chen et al., 2021) sequentially selected the topics until the sum of probabilities in the corpus exceeded 95%. Existing probabilistic approaches (Blei et al., 2003a, 2010; Kim et al., 2012; Paisley et al., 2015) only consider three levels. Isonuma et al. (2020) introduced neural architectures but fixed the number of levels to three with an initial number of three branches for both the second and third levels.
5The topic model in Adams et al. (2010) differs from our setting. In their experiments, each node has a unique topic distribution.
Another advantage of the ihLDA is that it employs a hierarchical Bayesian extension of the TSSB to draw a child TSSB from the base TSSB (see Figure 4). Unlike some probabilistic models that restrict a document-topic distribution to a single or multiple topic-path on a tree (Blei et al., 2003a, 2010; Paisley et al., 2015), ihLDA does not limit topics that can appear in a document.
Hierarchical topic models have a wide range of extensions (Mao et al., 2012; Yang and Hsu, 2016; Shin and Moon, 2017; Xu et al., 2018; Zou et al., 2019; Isonuma et al., 2021), and the ihLDA is orthogonal to them and useful for these extensions.

heading: 8 Conclusion
text: Existing hierarchical topic models yield topics with exponentially smaller probabilities. To address this intrinsic issue, we propose the ihLDA, a nonparametric Bayesian model that learns a latent topic hierarchy with arbitrary depth and width. Our model adjusts topic creation to achieve the expected topic probability without dependence on its depth, which can also improve other models that use the stickbreaking process. As a topic model, the ihLDA is a hierarchical extension of the TSSB and draws topic assignments efficiently without enumerating all possible candidates. Our experiments on standard document datasets confirm that the ihLDA outperforms the existing methods, including the latest neural models, and extracts meaningful topic structures with better hierarchical diversity and uniqueness.
Limitations
Although the ihLDA shows better performance than existing models in multiple experiments, there are three limitations that we did not fully address in this paper.
First, the Gibbs sampling is slower than other approaches such as autoencoding variational Bayes (Kingma and Welling, 2014), which limits data scalability. We can incorporate the literature on distributed algorithms for topic modeling (Newman et al., 2009; Yu et al., 2015; Karras et al., 2022) and variational inference (Wang and Blei, 2009; Wang et al., 2011; Bryant and Sudderth, 2012; Hughes et al., 2015) in future research.
Second, crowdsourced evaluation limits a corpus choice because we should not expect workers to have any prior knowledge (Ying et al., 2022). Our crowdsourced evaluation only used BBC News, the most accessible documents among the three corpora. Future research can thoroughly validate the performance of the crowdsourced workers and trained coders. Existing literature (Buhrmester et al., 2016; Kees et al., 2017) found that MTurk had a comparable quality against traditional survey panels, but they did not use MTurk for evaluating outputs from a machine learning model.
Third, an estimated hierarchical structure does not necessarily match the semantic hierarchy human readers expect. This mismatch is not surprising because unsupervised models do not directly incorporate information about a tree structure. Existing papers improved the interpretability of flat topic models by providing topic-specific sets of keywords (Jagarlamudi et al., 2012; Harandizadeh et al., 2022) and labels (Mcauliffe and Blei, 2007; Ramage et al., 2009), which is a future direction for a hierarchical topic model.

heading: Acknowledgements
text: We thank Adam Breuer, Dean Knox, Tomoya Sasaki, Yuki Shiraito, Soichiro Yamauchi, and members of the Imai Research Group at Harvard University for helpful discussions and comments on this project. We would also like to acknowledge anonymous reviewers for their constructive feedback.

heading: A Dirichlet Process
text: The Dirichlet process (DP) is the foundation of nonparametric Bayesian models. As the ihLDA is an infinite mixture model (i.e., an infinite number of topics can exist), we draw the topics using the DP.
Formally, G is a DP with a base distribution G0 and a concentration parameter c:
G ∼ DP(c,G0). (4)
The DP has three representations: the stickbreaking process, the Chinese restaurant process, and the Chinese restaurant district process.

heading: B Stick-Breaking Process
text: Formally, the stick-breaking representation of a DP with a base distribution G0 and a concentration parameter c, G∼DP(c,G0), is
G = ∞∑
k=1
δηkπk, πk = vk
k−1∏
j=1
(1− vj),
vk ∼ Be(1, c), ηk ∼ G0,
where ηk takes a distinct value to indicate a single category. Figure 9 depicts this process.

heading: C An Example of TSSB
text: In a hierarchical topic model, we consider both the vertical and horizontal placements of each word in a tree. To assign a topic for each word in the corpus, we first determine whether the word uses the root topic [1] (i.e., the word stops at [1]) or not (i.e., the word passes [1]) according to a vertical probability ν[1] ∼ Be(1, α0). The probability of stopping at [1] is thus π[1]=ν[1]. If the word passes [1], it goes down to the next level. Each solid line in Figure 1 connects a parent topic to its children in the next level. At the second level, horizontal stopping probabilities, ψ[1 1], ψ[1 2], ψ[1 3], · · · , determine the child topic to descend. Subsequently, the vertical probabilities ν[1 1], ν[1 2], ν[1 3], · · · , decide whether the word stops or proceeds further down the tree. We repeat this process until the word stops both vertically and horizontally. For example, the probability of using the topic [1 2 2] can be computed as follows:
π[1 2 2] = ( 1− ν[1] )
× ( 1− ν[1 2] ) · ψ[1 2] · ( 1− ψ[1 1] ) × ν[1 2 2] · ψ[1 2 2] · ( 1− ψ[1 2 1] ) .

heading: D The Expected Probability of Topics in Hierarchical Stick-Breaking Process
text: We consider the following stick-breaking process:
ϕk = vk
k−1∏
j=1
(1− vj), vk ∼ Be(1, γ).
The expectation of parameter vk is E[vk]=1/(1+ γ); hence, the expected probability of the kth broken stick is,
E[ϕk] = 1
1 + γ
( γ
1 + γ
)k−1 = 1
γ
( γ
1 + γ
)k .
Next, we consider the expected probability of a
topic in the stick-breaking process,
E[ϕ] = ∞∑
k=1
E[ϕk] · ϕk
≈ ∞∑
k=1
E[ϕk]2
= ∞∑
k=1
( γ
γ + 1 · 1 γ
)2
= 1
γ2
∞∑
k=1
( 1
( 1 + 1γ
)2 )k
= 1 γ2 · 1(
1 + 1γ )2 − 1
= 1
2γ + 1 ,
where the expectation of ϕk is used for the approximation. Using the standard stick-breaking process, the expected probability of the topic at the ℓth level is
E[ϕ |ℓ] = E[ϕ |ℓ− 1] · E[ϕ] ≈ 1 (2γ + 1)ℓ ,
where the first equality means that the (ℓ−1)th level stick is broken at the ℓth level. Because of the modification described in Section 3, the expectation becomes
E[ϕ |ℓ] = E[ϕ |ℓ− 1] · E[ϕ]
≈ E[ϕ |ℓ− 1] · 1 2(γ · E[ϕ |ℓ− 1]) + 1
= 1
2γ + 1/E[ϕ |ℓ−1] for ℓ ≥ 2 .
If ℓ = 1 (the root level), then E[ϕ |ℓ = 1] = 1/(2γ+1). The expected probability of the topic does not become exponentially smaller even when proceeding down the tree.

heading: E Hierarchical Dirichlet Process
text: Suppose that the global distribution of topics G is distributed as a DP with the concentration parameter c: G∼DP(c,G0). The actual distribution over the topics in the dth document, Gd, follows another DP, Gd∼DP(c0, G); hence, the distribution of Gd varies around G. Given Gd, we can draw a topic assignment for each word in the dth document.

heading: F Chinese Restaurant District Process
text: As shown in Figure 10, the CDP uses the counts of n words to determine a category z for the next
word:
p(zn+1 ≥ k |z1:n) = 1 +
∑∞ j=k+1 nj
1 + α+ ∑∞ j=k nj ,
where α is the concentration parameter corresponding to Equation (2). In the CDP terminology, a word using the kth category is referred to as “stopping at k” and that using the jth (j > k) category is referred to as “passing k”. Each word passes through categories until it stops; hence, we keep track of the number of data points that stopped and passed at each category.
To compute the vertical and horizontal probabilities νϵ and ψϵ, we count the number of words that have stopped at a topic ϵ as n0(ϵ) for a vertical stop and m0(ϵ) for a horizontal stop, as well as the number of words that have passed ϵ as n1(ϵ) for a vertical pass and m1(ϵ) for a horizontal pass.
Suppose that the first word stops at [1 3] in Figure 11. For this to occur, the word passes the root topic, [1], goes down to the next level, and passes two child topics, [1 1] and [1 2]. Hence, n0([1])=0 and n1([1]) = 1 when passing the root topic, and m0([1 1])=m0([1 2])=0, m1([1 1])=m1([1 2])= 1, m0([1 3]) = 1, and m1([1 3]) = 0 when horizontally stopping at the third child of the root topic. As the word vertically stops at the topic [1 3], the vertical count becomes n0([1 3]) = 1 and n1([1 3]) = 0. If the word vertically passes the topic [1 3] and further goes down the topic tree, then n0([1 3]) = 0 and n1([1 3]) = 1. In addition, we define n(ϵ) = n0(ϵ) + n1(ϵ) and m(ϵ)=m0(ϵ)+m1(ϵ).
Using pass and stop counts from the CDP, we can obtain the posterior distribution of the vertical and horizontal probabilities, νϵ and ψϵ, because the construction of πϵ is the result of choosing “stop” or “pass” on the way to reach ϵ:
νϵ |rest ∼ Be(1 + n0(ϵ), α+ n1(ϵ)) (5) ψϵ |rest ∼ Be(1 +m0(ϵ), γ +m1(ϵ)) (6)
Note that each probability is conditioned on the observed data and rest of the probabilities. By taking the expectations of Equations (5) and (6), we obtain
ν̂ϵ = E[νϵ |rest] = 1 + n0(ϵ)
1 + α+ n(ϵ) ,
ψ̂ϵ = E[ψϵ |rest] = 1 +m0(ϵ)
1 + γ +m(ϵ) .
(7)

heading: G Details of Evaluation Measures
text: G.1 Topic Uniqueness
Topic uniqueness (TU) calculates the uniqueness of all topics (Nan et al., 2019; Masson and Montariol, 2020; Chen et al., 2021). Let T be a set of topics in the estimated tree. We define TU as follows:
TU = 1 |T | ∑
ϵ∈T
( 1
u
u∑
u′=1
1
n(u′, ϵ)
) ,
where n(u′, ϵ) is the total number of times that the u′th top word in topic ϵ appears in the top u words across all topics. A higher TU implies that the topics represent unique themes.
G.2 Average Overlap
Average overlap (AO) measures the average repetition rate of the top u words between the parent topic and its children (Chen et al., 2021),
AO = 1 |T | ∑
ϵ∈T
|Vϵ ∩ Vϵ′ | u ,
where Vϵ is a set of unique words that appear in the top u words of a node ϵ. A lower AO indicates that less overlap occurs between the top words from a parent and those from its children. Although this measure was used in Chen et al. (2021), parent and child topics need some overlapping words to have semantic coherence; thus, less overlap does not necessarily mean better interpretability.

heading: H Additional Information for the Crowdsourced Evaluation
text: H.1 Design
We recruited participants via Amazon Mechanical Truk and used Qualtrics to prepare our evaluation tasks. Once participants agreed on the consent form (Figure 12), they read the instruction before conducting five tasks (Figure 13). We compensated the participants through payment ($0.5 to $0.55 per participant). The amount of compensation is determined to match the federal minimum wage in the United States.
H.2 Quality Check
The last task was the same as the one the participants saw in the instruction. We did not include the participants who failed to answer this quality check question, because we expected that careful crowdsourced workers could answer the question explained in the instruction. Additionally, dropped those who spent too little (bottom 10%) or too much (top 10%) time to complete the tasks.
The total number of observations was 535 (Word Intrusion), 900 (Vertical), and 620 (Horizontal).
H.3 An Ethics Review An institutional review board of an author’s institution reviewed our experimental design.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Yes, we discussed the limitations of our work in Section 9 (after the conclusion section). The limitations are particularly related to the scope of our claims.
A2. Did you discuss any potential risks of your work? Not applicable. Our paper makes contributions to the machine learning theory and does not meet any of the criteria mentioned in the checklist.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Section 1 is the introduction. The abstract comes before the introduction.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 7 Did you use or create scientific artifacts? Left blank.
B1. Did you cite the creators of artifacts you used? Not applicable. Left blank.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Not applicable. Left blank.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Not applicable. Left blank.
C 3 Did you run computational experiments? Section 6 compares our model against existing models.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
7 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Since our model is not a neural model, our theory section (Section 4) enumerates all parameters; hence, the number of parameters is evident from the paper. Also, we did not use GPU at all. Section 6.2 describes the cluster computer (only CPU) we used.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Section 6.2 explains our experimental setup.

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 6 describes our results. Figure 5 illustrates the result with error bars. Table 2 reports the means of evaluation metrics for three different numbers of top words. Section 6.3 explains how we took the means.

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? We used the existing implementations of machine learning models for evaluation in Section 6. We listed the packages we used in Section 6.2 and described the parameter settings (we used the default parameter values).
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? We used crowdworkers to evaluate models in Section 6.4.

heading: 3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots,
text: disclaimers of any risks to participants or annotators, etc.? Appendix H has a screenshot of the consent form and an example task. Our experiment did not show offensive content.

heading: 3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students)
text: and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? We mentioned the crowdsourcing platform (Amazon Mechanical Turk) in Section 6.4 and Appendix H. We mentioned the compensation in Appendix H.

heading: 3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to
text: crowdworkers explain how the data would be used? We included a screenshot of the consent form in Appendix H.

heading: 3 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board?
text: Sections 6.4 and H.3
7 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? We could not retrieve such information from Amazon Mechanical Turk.

heading: Future Work
text: , 2015) in future research. Second, crowdsourced evaluation limits a corpus choice because we should not expect workers to have any prior knowledge (Ying et al. , 2022). Our crowdsourced evaluation only used BBC News, the most accessible documents among the three corpora. Future research can thoroughly validate the performance of the crowdsourced workers and trained coders. Existing literature (Buhrmester et al. , 2016; Kees et al. , 2017) found that MTurk had a comparable quality against traditional survey panels, but they did not use MTurk for evaluating outputs from a machine learning model. Third, an estimated hierarchical structure does not necessarily match the semantic hierarchy human readers expect. This mismatch is not surprising because unsupervised models do not directly incorporate information about a tree structure. Existing papers improved the interpretability of flat topic models by providing topic-specific sets of keywords (Jagarlamudi et al. , 2012; Harandizadeh et al. , 2022) and labels (Mcauliffe and Blei, 2007; Ramage et al. , 2009), which is a future direction for a hierarchical topic model