Knowledge-Aware Bayesian Deep Topic Model
Authors: Dongsheng Wang
Yi.shi Xu
Miaoge Li
Zhibin Duan
Chaojie Wang
Bo Chen
Mingyuan Zhou
Venue: NeurIPS 2022 Accept (2022)
Keywords: Topic modeling
hierarchical document representation
knowledge graph
WordNet

Abstract: We propose a Bayesian generative model for incorporating prior domain knowledge into hierarchical topic modeling. Although embedded topic models (ETMs) and its variants have gained promising performance in text analysis, they mainly focus on mining word co-occurrence patterns, ignoring potentially easy-to-obtain prior topic hierarchies that could help enhance topic coherence. While several knowledge-based topic models have recently been proposed, they are either only applicable to shallow hierarchies or sensitive to the quality of the provided prior knowledge. To this end, we develop a novel deep ETM that jointly models the documents and the given prior knowledge by embedding the words and topics into the same space. Guided by the provided domain knowledge, the proposed model tends to discover topic hierarchies that are organized into interpretable taxonomies. Moreover, with a technique for adapting a given graph, our extended version allows the structure of the prior knowledge to be fine-tuned to match the target corpus. Extensive experiments show that our proposed model efficiently integrates the prior knowledge and improves both hierarchical topic discovery and document representation.

Knowledge-Aware Bayesian Deep Topic Model
Dongsheng Wang, Yishi Xu, Miaoge Li, Zhibin Duan, Chaojie Wang, Bo Chen' National Laboratory of Radar Signal Processing
Xidian University, Xn'an, Shanxi 710071, China
(wds, xuyishi, limimcaa aoge, zhibinduan @stu. xidian. edu cn xd siilyy1611.... com, behendmaill xidian. edu.en
Mingyuan Zhou
McCombs School of Business
The University of Texas at Austin, Austin, TX 78712, USA mingyuan. z www.
www.us.eeecco..... edu
Abstract
We propose a Bayesian generative model for incorporating prior domain knowledge into hierarchical topic modeling. Although embedded topic models (ETMs) and its variants have gained promising performance text analysis, they mainly focus on mining word co-occurrence patterns, ignoring potentially easy-to-obtain prior topic hierarchies that could help enhance topic coherence. While several knowledge- based topic models have recently been proposed, they either only applicable to shallow hierarchies or sensitive to the quality of the provided prior knowledge. To this end, develop novel deep ETM that jointly models the documents and the given prior knowledge by embedding the words and topics into the same space. Guided by the provided domain knowledge, the proposed model tends to discover topic hierarchies that are organized into interpretable taxonomies. Moreover, with technique for adapting a given graph, our extended version allows the structure of the prior knowledge to be fine-tuned to match the target corpus. Extensive experiments show that our proposed model efficiently integrates the prior knowledge and improves both hierarchical topic discovery and document representation.
1 Introduction
Topic models (TMs) such as latent Dirichlet allocation (LDA) have enjoyed success text mining and analysis in unsupervised manner [4. Typically, the goal of TMs is to infer per-document topic proportions and a set of latent topics from the target corpus using word co-occurrences within each document. The extracted topics widely used in various machine learning tasks (20 19 33 39). However, the objective function of most TMs is to maximize the likelihood of the observed data, which causes over-concentration be assigned to irrelevant topics due to the lack of side information. Those two drawbacks could lead to commann-nfrrendlyy topies that fail to make sense to end users in practice. This issue is further exacerbated in hierarchical cases where a large number of topics and their relevance need to be modeled 22 12].
high-frequency words. Moreover, the infrequent words might
In many cases, users with prior domain knowledge are concerned with a specific topic structure, as shown in Fig. [ Such topic hierarchy provides semantic common sense among topics and words, which can guide high-quality topic discovery. Therefore, several researches have attempted to exploit
*Correspodiing author
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

ROOT
ROOT
ball
windows
basketball
football software
hardware
ball basketball football baseball
windows
vahicle
software hardware manitar
Target Corpus
bike
gas helmets
valleyball
game
player
install
basketball basketball nba
football football soccer cup game
softwaree software install windows computer
hardwaare hardware keyboard mouse display
The definition of basketball:
car
A game played on a court by two opposing teams...
benw
game
player
Figure 1: Motivation of the proposed model. We use the pre-specified topic hierarchy as the knowledge graph (left), where topics (colored nodes) and words (gray nodes) organized into tree-like taxonomy, and each topic is associated with brief definition. Given topic tree the target corpus, our proposed model aims to I) retrieve a set of coherent words for each topic node (right); 2) explore new topic hierarchies that are missed in the provided topic tree, e.g., the dashed boxes (right).
various prior knowledge into topic modeling improve topic representation. For example, word correlation-basee topic models 11 30 use must-links and cannot-links to improve the interpretabilityy of learned topics. Word semantic methods [8 allow users to predefine a set of seed words that could appear together in certain topic to improve topic modeling. Furthermore, knowledge graph- based topic models [7 34 combine LDA with entity vectors inject multi-relational knowledge, achieving better semantic coherence. Although the above mentioned methods improve topic coherence in different ways, they only work with shallow topic models and ignore the relationships between topics.
More recently, embedded topic models (ETMs) and their hierarchical variants [10 38 32 12] employ
distributed embedding vectors to encode semantic relationship, which have gained growing research interest due to their effectiveness and better flexibility. Viewing words and topics as embeddings in shared latent space makes it possible for ETMs to integrate prior knowledge in the forms of pretrained word embeddings or semantic regularities [I0 27 [3]. The main idea behind those models is to constrain the topic embeddings under the predefined topic structure. Those models however are built on a strong assumption that the given topic hierarchy is well defined and matched the target corpus, which is often unavailable in practice. Such mismatched structure might hinder the learning process and lead to sub-optimal results.
To address the above shortcomings, in this paper, we first propose TopicKG, knowledge graph- guided deep ETM that views topics and words nodes in the predefined topic tree. As a Bayesian probabilistic model, TopicKG aims to model the document Bag-of- (BoW) vector and the given topic tree jointly by sharing the embedding vectors. Specifically, we first adopt a deep document generation model to discover multi-layer document representations and topics. To incorporate the human knowledge in the topic tree and guide the learning of topic hierarchy, graph generative model is employed, which refines the node embeddings by exploring the belonging relations hidden in topic tree, resulting in more interpretable topics. Besides, address the mismatch between the given topic tree and target corpus, we further extend TopicKG to TopicKGA, which adopts the graph Adaptive technique to explore the missing links in topic tree from the target corpora, achieving better document representation. The final revised topic tree is obtained by combining the prior structure of the given topic tree with the added structures learned from the current corpus. Thus the topic tree in our TopicKGA has the ability to reinforce itself constantly according to the given corpus. Finally, both the proposed models are based on variational autoencoders (VAEs) 211 and can be trained by maximizing the evidence lower bound (ELBO), enjoying promising flexibility and scalability.
The main contributions of this paper can be summarized as follows: 1), novel knowledge-aware deep ETM named TopicKG is proposed to incorporate prior domain knowledge into hierarchical topic modeling by accounting for the document and the given topic tree in Bayesian generative framework. 2). To overcome the drawback of the mismatch issue between the prior structure of the provided topic tree and the current corpus, TopicKG is further extended to TopicKGA, which allows to revise the prior structure to better represent the current corpus. 3), Besides achieving promising performance on hierarchical topic discovery and document classification task, the proposed models give good flcxibility and efficiency, providing strong bascline for knowledge- based deep TMs.

---

2 Related Work
Embedded Topic Models: The recent developed ETM is new topic-modeling framework in embedding space. Dieng et al. [10) first proposed ETM with the goal of marrying traditional topic models with word embeddings. In particular, ETM regards words and topics as continue embedding vectors and the topic's distribution over words is calculated using the inner product of the topic's embedding and each word's embedding, which provides natural way incorporate word meanings into TMs. In constrast to LDA, ETM employs the logistic-normal distribution to estimate the posterior of per-document topic proportion, making is easier to reparameterization in the inference step. All parameters in ETM are optimized by maximizing its evidence lower bound (ELBO). To explore hierarchical document topical representations, ETM is extended to SawFTM [2 with the gamma belief network 42 SawETM designs the sawtooth connection (SC) to capture the relations between topics at two adjacent layers the embedding space, resulting in more efficient algorithm for hierarchical topic mining. Note that our proposed TopicKG is built upon SawETM, but different from SawETM that only considers the BoW representation, which may fail to discover high quality topic hierarchies when a large of topies and their relations need to be modeled 13), TopicKG introduces a novel Bayesian generative model where both document and the pre-specific domain knowledge considered to integrate human knowledge into NTMs, resulting in a new knowledge- topic modeling framework.
Topic Models With Various Knowledge: Pre-trained language models such as BERT 9 and
GPT [6] have been successful in various natural language processing tasks. Pre-trained on huge text, such models can serve as powerful encoder that outputs contextual semantic embeddings. Behind this idea, combining a TM with BERT via knowledge distillation [[6] treats BERT as the teacher model to improve the representation of topic models. CombinedTM 3 concatenates the Bag-of-Word (BoW) vector with the BERT contextual embedding as the final imput of TM, and performs a consistent increase in topic coherence. Despite their improvement, those BERT-based TMs focus on the document latent representation, they ignore the relationship between words and topics. Moreover they require large amount of memory to load the pre-trained BERT. On the other hand, incorporating knowledge graph (e.g., WordNet (28)) to improve existing NTMs becomes an interesting direction recently. For example, (77 use the category tree (usually with simple and shallow structure) as the supervised information and preserve such relative hierarchical structure in the spherical embedding space. explore user interested topic structure. [13 view words and topics as the Gaussian distributions and employ the asymmetrical Kullback-Leibler (KL) divergence to measure the directional similarity in the given topic hierarchy. However, computing such KL divergence for each pair is time-consuming that limits its application to large-scale knowledge. Moreover, those knowledge-based models ignore the mismatch problem between the given topic tree and the current corpus, leading to sub-optimal results in practice.
3 The Proposed Model
To incorporate human knowledge into deep TMs, we first propose TopicKG that generates the target corpus and the given topic tree in the Bayesian manner. We further extend TopicKG to its adaptive version TopicKGA that allows the topic structure to be refined according to the current corpus, resulting in a better document representation. Below we introduce the details of our proposed model.
3.1 Problem Formulation
Consider a corpus containing J documents with vocabulary of V unique tokens. Each document represented by a BoW vector æ E RY, where I,, denotes the number of occurrences of the th word. Unlike other TMs that are purely data-driven, TopicKG intends to inject the knowledge graph to improve topic quality. Specifically, we introduce a topic tree T as our prior knowledge, where each node e, T denotes word topic. For each topic node, there is corresponding definition as shown in Fig. D. Suppose there are L + 1 layers in the topic tree with the bottom word layer and L topic layers, and there are K₁ nodes Ti {e(") ,.... (; at the I-th layer, where Ko V.
(1)
Generally, for the k-th topic node at the I-th layer 6x we use S(e( to denote the set of its child (1)
(1)
nodes, and use ((e()) (1) to denote its key words extracted from the definition sentence. Mathematically,
adopt the binary matrix 50() {0, 1 1 and (0)) {0, VXKi, I = 1, ..., L. to denote
3

---

the belonging relations between two adjacent layers and the links between topics and their concept words, respectively. Guided by this pre-specified topic tree, TopicKG expects to retrieve hierarchical topics from the target corpus that provide a clear taxonomy for the end users.
3.2 Knowledge-Aware Bayesian Deep Topic Model
Given the corpus D and topic tree T, we aim to discover hierarchical document representations and topic hierarchy by jointly modeling the BoW vector S and C(O), / I... L in a Bayesian framework. Specifically, we employ the gamma belief network (GBN) of 40 to model the count-
value vector æ, and use the semantic similarity between two node embeddings to generate the topic tree T. The whole generative model with L layers can be expressed as:
0 0 ~ Gam(y: (,,ilyyy11)) 1/cftt1),,
{0))) ~ Gam c (1 (+
.1/e9t1)
1-1
1=1
Ki-i.Kibb
Ij Pois($10)))
{ 1 = Softmax (1)) (1)
1 kikk
= (1-1)
(1)
1=11"
cka
Cka
==
( k2 Berm(o(e)
(t-1)
(1-1"
W)! k 2a
Ki-nKKiL
{ cill (ern(a(e)))
(0)
(1)
V.Ki,
ek
1=1,-=1,111
where zj is generated as in Poisson factor analysis 41], in which xy is factorized as the product of the factor loading matrix (topies) + (1) E R and the gamma distributed factor scores (topic proportions) 0. I under the Poisson likelihood; Then the GBN 42 is applied explore multi-layer document representations by stacking hierarchical prior in the topic proportions 0,. The topics at each layer & (1)) = 1, .... is calculated by the semantic similarity (e.g., the inner product) between corresponding topic embeddings, followed by a Softmax layer to satisfy the probability simplex: Σ (t) k 1. e E Rd is the embedding vector of k-th topic at I-th layer (k-th
node at [-th layer ce in T), d is the embedding dimension. ( is the embedding vector for v-th word in vocabulary. a--) is the sigmoid function and Bern(-) denotes the Bernoulli distribution that employed to model the two edge types ((() and C(()) As suggested in 27 and [3], we use asymmetry to capture the directed relations in s(!), while use symmetry to capture the semantic similarity between topics and their concept words. W is a learnable parameter to guarantee the directed structure. Under the Bernoulli likelihood, our TopicKG has the ability to incorporate the topic hicrarchical relations in T into topic modeling by sharing word and topic embeddings with 2. In particular, for node pair in T, the embedding semantics of the source and destination nodes need to be similar enough to determine whether there is an edge, which will guide the learning of & resulting in more coherent topics and some appealing model properties as described below.
The Flexibility of Human Knowledge Incorporation: As mentioned above, guided by the pre- defined topic tree, To models x, s(()) and (()), ( 1, jointly, making it possible for the learned hierarchical topics that not only satisfy the interpretable taxonomy structure but also prevent incoherent issues with the help of side information. Besides, the incorporation of topic tree T is simple and flexible. Firstly, as shown in Eq. 1 and Sec. 4 the plug-and-play module of ( and (ike) can be applied to most of ETMs without changing their model structures, which provides a convenient alternative for introducing side information to TMs. Secondly, the pre-defined T can be constructed in various ways to encode our beliefs about the graph structure, e.g., the knowledge graph, taxonomy, and hierarchical label tree [34 17 27].
The Relationship Between { H/ and {/s The topic distribution matrix & ( and the symmetry matrix g()) are related but play different roles. The sparse structure matrix s!!) is constructed from the topic tree which will be used to guide the learning of the hierarchy of topics, while the learned topics (!) contains more specific knowledge extracted from the text corpus, and can be viewed as the dense version of (()) In other words, the topics learned from TopicKG retains the semantic structure of ((l) and enriches itself by the current corpora.

---

E
1
GCN
E
11
0
(b)
(2)
E E
4
0
()
GCN
S
T
T
adu + A
Figure 2: Overview of the proposed framework. It jointly models the topic tree and document via the shared node embeddings E(T) The proposed two models TopicKG (a) and TopicKGA (b) share the same document generative module (left part) while with different graph modelings.
3.3 Inference Network of TopicKG
The inference network of the proposed model is built around two main components: Weibull upward- downward variational encoder and the GCN-based topic aggregation module. The former aims to infer the topic proportions given the document, and the latter updates the node embedding by aggregating the neighbor information via the GCN layer.
Weibull Hpward-Downwer
To approximate the posterior of the topic proportions {o()), }. like most of VAE-based methods, we define the variational distribution q(0)|æz)) which can be further factorized as 31::
Variational Encoder:
L-1 g(0,l=,) = 9 D II q 10 ,x,))
In practice, it first obtains the latent feature by feeding the input æ; into a residual upward neural
networks: h !1) h ((-1) + f!() (1) (h) (11))) ) where 1, .... L, h (0)
æy. f(()) 1 (-) layer
fully connected network parameterized by ( To complete the variational distribution, we adopt the Weibull downward stochastic path [3]]
q(0)
(1+1)
h (+11) Weibull( ! (2())
(0)) Softplus (()
(++)
k )),
((++)) 1 Softplus( ((() ( (1+1) g + x(r)))
(2)
(1) Relu W h +b))
X (!) Relu(W (0i (1)
k
where () denotes the concatenation topic dimension, and we use the Softplus to make sure the positive Weibull shape and scale parameters. f is single layer fully connected network, correspondingly. The Weibull distribution is chosen mainly because it is reparameterizable and the KL divergence from the gamma to Weibull distributions has an analytic expression 15]
GCN-Based Topic Aggregation: Attracted by the excellent ability of graph neural network (GCN) 22 2 in propagating graph structure information, and the predefined topic tree can be viewed as a directed graph. We thus construct a deterministic aggregating module for node embeddings with GCN:
E GCN(A. A, E E(1-1)), 1,...T
33)
5

---

where T is the number of GCN layer, E. E RdxN is the node embedding matrix, and N Eio Kj A D AD- is the normalized adjacent matrix with degree matrix D, e.g., Dü Ej=1 Aij. A E (0, 10,1N is the adjacent matrix of the topic tree 'T Unlike previous ETMs that view embeddings as independent learnable parameters, the GCN module in TopicKG updates the node embeddings by considering their child-level topies and related words, resulting in more meaningful topic embeddings. We calculate (T()) [=1 via the updated node embeddings:
e (T) (1-)²² (T)())
Softmax
ek
(4)
3.4 TopicKG With Adaptive Structure
One of the main assumptions in the previous section is that the pre-defined topic tree is helpful for the current corpus and the corresponding edges are highly reliable. However, this is generally unrealistic in practical upplications, T may be (i) noisy, (ii) built the topic discovering task [14] 8]. Consequently, we further propose TopicKGA that overcomes the above mismatching issue and revises the structure of the given topic tree based the corpus at hand.
ad hoc basis, (iii) not closely related to
In detail, TopicKGA first randomly initializes a learnable node embedding dictionaries E E R for all nodes in T. Then an adaptive graph A ada generated based on Σ using the certain kernel function & Rd Rd - R:
A ada
(oft Ea))
(5)
EAE.
Here, we choose the consine similarity to define our kernel function: E(EA, EA)
T EAT and
Softmax function is used to normalize the adaptive matrix. Note that, instead of generating A ada we directly generate its normalized version avoid unnecessary calculations (2) During training. E will be updated automatically to learn the hidden dependencies which are ignored by A. Following
[36). the final revised adjacency matrix is formed as (here we still use A to denote the revised graph for convenience):
À = A + A
(6)
which will enhance the GCN in Eq. by replacing the adjacency matrix with the revised one.
4 Training
As deep ETM, TopicKG intends to learn the latent document-topic distribution and the deterministic hierarchical topic embeddings. Like other ETMs, the posterior of the topie proportions and topic embeddings are intractable to compute. We thus derive efficient algorithm for approximating the posterior with amortized variational inference (5), which makes the proposed model flexible for downstream task. The resulting algorithm either use pre- pre-trained word embeddings, train them from scratch. The ELBO of the proposed model can be expressed as:
Ki-1 logn( (1)
L
D e U 2 (logp(.r ny + 0
Σ
(1el-1) (1) 11 1e ebkz
L
+ logo(
(1) (
skg
(000)
e
(1))
ΣΣ
(L((0())) MI o (1+1) ( 0
1=1
consists of three main parts: the expected log- likelihood of æ (first term), the concept structure log-likelihood (the two middle terms), and the KI. divergence from prior ( to g ((())) The graph weight 3 denotes the belief about the predefined topic tree. Notably, the two middle terms distinguish the proposed models from previous deep ETMs. On the one hand, it acts
regularization that constrains the embedding vectors to conform to the provided prior structure, and on the other hand, provides alternative for ETM to introduce side information to improve its interpretability.
Annealed Training for ToA: In TopieKGA, the structure of the revised graph changes during the training. To incorporate the new edges that are inferred from the target corpus, develop an

---

annealed training algorithm for TopicKGA. Specifically, we update 5 and C in Eq. every M iterations:
$10
11,
if
A Aika
8;
C ok2
f1. 0.
ifA
uk2 8;
[.. else
clse
where is the threshold, and A is the corresponding sub-block in A. As A becomes stable, the structure of the graph will converge to a good blueprint that balances the prior graph and the current corpus. We summarize all training algorithm in Appendix.
5 Experiment
In this section, we conduct extensive experiments on several benchmark text datasets to evaluate the performance of the proposed models against other knowledge-based TMs. in terms of topic interpretabillity and document representations. The code is available at http:///sttbbbooo/////222////779
5.1 Corpora
Our experiments are conducted on four widely used benchmark text datasets, varying in scale. The datasets include 20 Newsgroups (20NG) [24], Reuters extracted from Reuters-21578 dataset, R8, and Reuters Corpus Volume 2 (RCV2) [25] R8 is subset of Reuters that collected from 8 different classes. For the multi-labe, RCV2 dataset, we follow previous works [29] in which documents with single label at the second level topics are left, resulting in O.IM documents totally. Both R8 and RCV2 are already pre-processed. For other two datasets, we tokenize and clean text by excluding standard stop words and low-free wwwffrequeccy words appearing less than 20 times [35]. The statistics of the preprocessed datasets are summarized Appendix.
5.2 Baselines
To demonstrate the effectiveness of incorporating human knowledge into deep TMs, we consider several baselines for a fair comparison, including representative ETMs and recent knowledge-bard topic models, described as follows: 1). ETM [T0], the first neural embedded topic model that assumes topics and words live in the same embedding space. The topic distributions ETM are calculated by the inner product of the word embedding matrix and the topic embedding. 2). Combined B a BERT-based neural topic model that uses the pre-trained BERT
contextual encoder. We use it as our BERT baseline. 3), SawETM [12), a hierarchical ETM that employs the Poisson and gamma distributions to model the BoW vector and the latent representation, respectively. 4), JoSH [27], knowledge-based hierarchical topic mining method that uses category hierarchy as the side information and employs the EM algorithm learn the spherical tree and text embedding. 5), TopicNet [33,, semantic graph guided topic model that views the words and topics as the Gaussian distributions and uses the KL divergence to regularize the structure of the pre-defined tree. For all baselines, employ their official codes and default settings obtained from their release repositories.
5.3 External Knowledge
Since the learning of TopicKG involves pre-defined topic tree, some generie knowledge graphs such as WordNet (88 provide a convenient way to bring in external knowledge. WordNet is large lexical database that groups semantically similar words into synonym sets, these sets are further linked by the hyponymy relation. For example, in WordNet the category furniture includes bed, which in turn includes bunkbed. With these relations, the topic structure can be easily defined according certain heuristic rules. Specifically, for each dataset we first get the word intersection of the vocabulary and WordNet, the chosen words are then considered leaf nodes of the topic hierarchy. Afterwards, each leaf node can continuously find its ancestor nodes based on hyponymy relations and finally all leaf nodes converge the same root node, resulting in an adaptive topic structure.
5.4 Settings
For experiments, we set the embedding dimension as 50, the knowledge confidence hyper- parameter as 8 = 50.0, the threshold 0.4. Empirically, find that the above settings work
7

---

2INOCTC)
(a(rC)
ROVATTC)
Restera/TC)
0.13
0.1
0.0
0.29
0.00
0.35
20HG/TD)
0.5
0.3
0.1
0.6
0.4
0.4
0.2
0.3
0.4
0.3
0.30
0.32
0.25
125
0.2
02
0.2
0.55
0.15
0.1
-031
TepuRtA
Figure 3: Topic coherence (TC. row), topic diversity (TD. middle row), and word embedding coherence (WE, bottom row) results for various deep topic models four datasets. In each subfigure. the horizontal axis indicates the layer index topies. metrics, higher better.
well for all datasets, we refer readers to the appendix for more detailed analysis. We initialize the node embedding from the Gaussian distribution (0, 0.02). We set the batch size as 200 and use the Adam W (26 optimizer with learning rate 0.01. We generate a 7-layer topic hierarchy for each dataset with the method described in Sec. 5.3 From the top to bottom, the number of topics at each layer
are: 20NG: 185,2,77770 R8: [1,2,11,77,287, 17,5071,5477 501,547]; RCV2: 1,,211,68,, 24,,2663,6995 469,662]]; For the single-layer methods (ETM and CombinedTM) we set the number of topics as the same of the first layer of deep models. For all methods, we run the algorithms in comparison five times by modifying only the random seeds and calculate the mean and standard deviation. All experiments are performed on an Nvidia RTX 3090-Ti GPU our proposed models implemented with Py Torch.
Reuters: 11,11,,83300,52665553
Table 1: Micro FI and Macro Fl score of different models on three datasets. The digits in brackets indicate the number of layers. Micro F1 /Marro Fl.
Model
20NG
RB
RCV2
ETM
50.25 +0004///7.444 +0.21
88. 10
59.67 +0.24
68.63 +0,15 /4.40 +0.11
CombinedTM
10.44440.11450 54.95 #0.11
93.69 10.09 /84.14 +0.10
84.85 20,11 +0.21
Sawtooth(3)
52.41 #2.088 15..0.0//11.3--- 10.10
90.04 =0.15/78884 =0.21
82.54 #0.11 /49.25 +0.10
TopicNet(3)
35.16:40.22244.77
+0.34
=0.17 64.15 +0.16
84.15 +0.25/55..77 +0.22
TopicKG(3)
55.73 +0.15/54.44 +0.08
93.6 / 83.32 +0.07
84.75 +0.16/ +0.41
TopicKGA(3)
18.33:40.55/57.999 10.10
93.70 =0.52 84.50 10.11
85.34 #0.14/4/2522 +1.10
ETM
47.9-40.12/44.199 +1.01
86.54 / +1.11
63.77 +0.14/21.44 +1.04
CombinedTM
58.16 18.110/558110001.
93.50 10.13 /84.84 pm0.11
82.91 #0.11 / =0.05
Sawtooth(7) (7)
53.71 +4..11/53.02 +0.47
92.86 +0.07 82.54 =0.41
82.46 =4.15/49.34 +0.34
TopicNet(7)
56.13 1...-40.1//5...4 +0.39
90.65 +0.00 66.57
+0.00
82.81 +0.00 +0.00
(GT
56.32 12..1/////73..5 +0004
+0.12 85.04 +0.11
82.48 +0.11 +0.09
TopicKGA(7)
60.04 +0.34 59.12 10.13
94.10 #0.08 85.50 10.10
83.08 +0.23 50.50 +0.08
5.5 Topic Interpretability
Generally, topic models are evaluated based perplexity. However, perplexity on the held-out test not an appropriate measure of the topic quality and sometimes can even contrary to human judgements 7 34 To this end, we instead adopt three common metrics, including Topic coherence (TC), Topic diversity (TD) and word embedding topic coherence (WE), to evaluate the learned topics
from various aspects 10 3). TC measures the average Normalized Pointwise Mutual Information

---

(NPMI) over the top 10 words of each topic, and a higher score indicates more interpretable topics. TD denotes the percentage of unique words in the top 25 words of the selected topies. WE, as its name implies, provides an embedded measure of how similar the words in topic are. WE is calculated by the average pairwise cosine similarity of the word embeddings of the top-10 words in a topic. We report the topic quality results at the first five layer in Fig. 3 We here focus on topic hierarchy and ignore the single layer methods. We find that: 1), Knowledge-basee CombinedTM, TopicNet and our proposed models are generally better than other likelihood- -badee ETMs, which illustrates the benefit of incorporating side information: 2). Our proposed TopicKG and TopicKGA achieve higher performance than other knowledge-basse based models in most cases, especially on TC and WE tasks, which means our proposed model prefer to mine more coherent topics while achieving comparable TD. We attribute this to the joint topic tree likelihood that provides an efficient alternative to integrate human knowledge into ETMs. 3). Compared to TopicKG, its adaptive version TopicKGA discovers better topics at higher layers. It is not surprise that TopicKGA gives greater robustness and flexibility by refining the topic structure according to the current corpus.
methods including JoSH,
matter
matter wates ak lin ot world
6 Layer
wates als Si, oll world
2th Layer
1 Layer
il health tealth, medical study cold. dieess, pobients
vehicle
ill health
vehidle vohicle, spenecratt lounch roceet joads, milss
free topic e1
walidde oar launch, hike bus melhoryte spacecsafi,
health, rict deetons eanove medics dieesse patients
safe burbo, driver mil madh cans cup
ilines health noc. cancec disessn tolpttl
infection tealth, es wids medicll pasinn Insement
Whereltec vishicle bron cne bun mnsite miks turbo driving
Fiee topic nik heart cat doctors disess body. ditt I9c
ilness health, déesse. Hospial Pnsineee
infection
rocker apacoouft, bureh shurte station aetmoomy sky
ufreeled viehide hile rat can, with mics huttin ot dridig
free topic #2
spacecraft shutte, rodket technral atofitc sky
heath diease Ni. mecioal perent beaement
hards
meads hmm
cars salt
font drving. battery
(a)
(b)
Figure 4: The learned hierarchical topic structure from TopicKG (a) and TopicKGA(b) on 20NG. Each topic box contains its concept name and the corresponding keywords. Topics in dashed box (free topics) are newly added by TopicKGA from the target corpus. The thickness of the arrow represents the relation weight and different colors denote different layers.
5.6 Document Classification
Besides the topic quality evaluation, we also use document classification to compare the extrinsic predictive performance. In detail, we collect the inferred topic proportions, e.g., o(1), and than apply logistic regression to predict the document label. We report the Micro F1 and Macro F1 score on 20NG, R8 and RCV2 datasets in Table. [ Overall, the proposed models outperform the baselines on all corpora, which confirms the effectiveness of our innovation of combining human knowledge and ETMs in improving document latent representations. Moreover, with the revised topic structure fine-tuned to the current corpus, TopicKGA surpasses other knowledge graph fixation methods with significant gaps. This conclusion is consistent with one of our motivations that the predefined concept tree may not match the target corpus, leading to suboptimal document representations.
5.7 Qualitative Analysis
We visualize the learned topic hierarchies of TopicKG and TopicKGA 20NG in Fig. 4 a)),), respectively. Each topic box consists of the pre-specified concept name on the top bar and its keywords listed in the bottom content. We can observe that 1), The mined keywords are highly relevant to the corresponding topics, providing a clean description of their concepts. 2), Guided by human knowledge, the connections between topics at two adjacent layers are highly interpretable. resulting in friendff topic taxonomies. 3). More interestingly, to further verify the adaptive ability of our proposed TopicKGA, we manually added several free topics (dashed boxes in Fig. (()) to each layer of the given topic tree, which can be viewed as the missing topics in the predefined
We can't 't report the results of JoSH on RCV2 as JoSH requires the sequential text, but only the BoW form is available for RCV2.

---

knowledge and need to be learned from the target corpus. We find that our TopicKGA indeed has the ability capture the missing concepts, and revise the prior graph match the current corpus.
5.8 Time Efficiency
To demonstrate the time efficiency of incorporating human knowledge into TMs, we run TopicKG, TopicKGA and TopicNet (as they are implemented in PyTorch, JoSH is in C) on RCV2 and 20NG dataset with 7-layer topic tree. Fig. 5 shows the Negative log- log-likelhhood of documents 2, which shows that the proposed models not only have faster learning speed than TopicNet, but also achieve better reconstruction performance on both small and large corpus. This result illustrates the efficiency of the introduced graph likelihood which is important real-tima applications.
RCV2
500
Topichet TopicKG TopicKOA
450
400
350
500
1000
1500
2000
2500
Time(s)
20NG
360
Conclusion
We develop efficient Bayesian probabilistic model that integrates pre-specified domain knowledge into hierarchi- cal ETMs. The core idea is the joint document and topic tree modeling with the shared word and topic embeddings. Besides, with the graph adaptive technique, TopicKGA has the ability to revise the given prior structure according the target corpus, enjoying robustness and flexibility in practice. Extensive experiments show that the proposed models outperform competitive methods in term of both topic quality and document classification task. Moreover, thanks to the efficient knowledge incor- poration algorithm, our proposed models achieve faster learning speed than other knowledge-base models.
200
Time(s)
Figure 5: Negative log likelihhoood (NLL) curves of various methods on RCV2 and 20NG datasets.
Acknowledgements
This work was supported in part by the National Natural Science Foundation of China under Grant U21B2006; in part by Shaanxi Youth Innovation Team Project; in part by the 111 Project under Grant B18039; part by the Fundamental Research Funds for the Central Universities QTZX22160.
References
[1] D. Andrzejewski, X. Zhu, and M. Craven. Incorporating domain knowledge into topic modeling via dirichlet forest priors. In Proceedings of the 26th annual international conference on machine learning, pages 25-32, 2009.
[2] L. Bai, L. Yao, Li, X. Wang, and C. Wang. Adaptive graph convolutional recurrent network for traffic forecasting, In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
[3] F. Bianchi, S. Terragni, and D. Hovy. Pre-training is a hot topic: Contextualized document embeddings improve topic coherence. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/JJCNLP 2021, (Volume 2: Short Papers), Virtual Event, August 1-6, 2021, pages 766.
[4] D. M. Blei, A. Y. Ng. and M. I. Jordan. Latent dirichlet allocation. the Journal of machine Learning research, 3:993 19999-1022, 2003.
[5] D. M. Blei, A. Kucukelbir, and J. D. McAuliffe. Variational inference: A review for statisticians. Journal of the American statistical Association, 112(518):55-887 2017.
10

---

[6] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, Shyam, G. Sastry, A. Askell, Agarwal, A. Herbert- Voss, G. Krueger. T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Lan- guage models are few-shot learners. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual.
[7] Chang. S. Gerrish, C. Wang, L.. Boyd-Graber, and D. M. Blei. Reading tea leaves: How humans interpret topic models. In Advances in neural information processing systems, pages 288-296, 2009.
[8] Y. Chen, L. Wu, and M. Zaki. Iterative deep graph learning for graph neural networks: Better and robust node embeddings. Advances Neural Information Processing Systems, 33, 2020.
[9] J. Devlin, M. Chang, K. Lee, and K. Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume (Long and Short Pupers), pages 4171-4186.
[10] A. B. Dieng, Ruiz, and D. M. Blei. Topic modeling in embedding spaces. Transactions of the Association for Computational Linguistics, 8:439 453, 2020.
[11] Doshi-Velez. B. Wallace, and R. Adams. Graph-sparse Ida: topic model with structured sparsity. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 29, 2015.
[12] Z. Duan, D. Wang, B. Chen, C. Wang, W. Chen, Y. Li, Ren, and M. Zhou. Sawtooth factorial topic embeddings guided gamma belief network. In International Conference on Machine Learning, pages 2903-2913. PMLR, 2021.
[13] Z. Duan, Y. Xu, B. Chen, dongsheng wang, C. Wang, and M. Zhou. Topicnet: Semantic graph-guided topic discovery. In Advances in Neural Information Processing Systems, 2021.
[14] P. Elinas, E. V. Bonilla, and L. C. Tiao. Variational inference for graph convolutional networks in the absence of graph data and adversarial settings. In Advances in Neural Information Processing Systems 33: Anmual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6- 6-1 2020, virtual, 2020.
[15] X. Fan, S. Zhang, B. Chen, and M. Zhou. Bayesian attention modules. In Advances in Neural Information Processing Systems 33: Annual Conference Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual.
[16] A. M. Hoyle, P. Goel, and P. Resnik. Improving neural topic models using knowledge distillation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing. EMNLP 2020, Online, November 16-20, 2020, pages 1752-1771.
[17] Z. Hu, G. Luo, M. Sachan, E. P. Xing, and Z. Nie. Grounding topic models with knowledge bases. In IJCAL, volume 16, pages 1578-1584 2016.
[18] Jagarlamudi, H. Daumé III, and R. Udupa. Incorporating lexical priors into topic mod- els. In Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics, pages 204-213, 2012.
[19] M. Jin, X. Luo, H. Zhu, and H. H. Zhuo Combining deep learning and topic modeling for review understanding in context-aware recommendation. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume I (Long Papers), pages 1605-1614, 2018.
[20] S. S. Kataria, K. S. Kumar, R. R. Rastogi, P. Sen, and S. H. Sengamedu. Entity disambiguation with hierarchical topic models. In Proceedings of the 17th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 1037-1045, 201
11

---

[21] D. P. Kingma and M. Welling. Auto-encoding variational bayes. In 2nd International Conference on Learning Representations. ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings.
[22] T. N. Kipf and M. Welling. Semi-supervised classification with graph convolutional networks. arXiv preprini (XXiv:660900200) 2016.
[23] T. N. Kipf and M. Welling. Variational graph auto-encoders, arXiv preprint arXiv:
2016.
[24] K. Lang. Newsweeder: Learning to filter netnews. In Machine Learning Proceedings 1995, pages 331-339. Elsevier, 1995.
[25] D. D. Lewis, Y. Yang, T. G. Rose, and F. Li. Revl: A new benchmark collection for text categorization research. Journal of Machine Learning Research, 5:361-397, 2004.
[26] Loshchilov and F. Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.
[27] Y. Meng, Y. Zhang, J. Huang, Y. Zhang, C. Zhang, and J. Han. Hierarchical topic mining via joint spherical tree and text embedding. Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2020.
[28] G. A. Miller. Wordnet: lexical database for english. Communications of the ACM, 38(11): 39-41, 1995.
[29] T. Miyato, A. M. Dai, and I. J. Goodfellow. Adversarial training methods for semi-supervised text classification. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings.
[30] D. Newman, E. V. Bonilla, and W. Buntine. Improving topic coherence with regularized topic models. Advances in neural information processing systems, 24:496-504, 2011.
[31] A. Vahdat and J. Kautz. NVAE: A deep hierarchical variational autoencoder. In Advances Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6- 6-, 2020. virtual.
[32] D. Wang, D. Guo, H. Zhao, H. Zheng. K. Tanwisuth, B. Chen, and M. Zhou. Representing mixtures of word embeddings with mixtures of topic embeddings. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtall Event, April 25-29, 2022.
[33] H. Xu, W. Wang. W. Liu, and L. Carin. Distilled wasserstein learning for word embedding and topic modeling. In Advances in Neural Information Processing Systems 31: Annual Conference on Newral Information Processing Systems 2018, NeurlPS 2018, December 3-8, 2018, Montréal, Canada, pages 1732.
[34] L. Yao, Y. Zhang, B. Wei, Z. Jin, R. Zhang, Zhang, and Q. Chen. Incorporating knowledge graph embeddings into topic modeling. In Thirty-first AAAI conference artificial intelligence, 2017.
[35] L. Yao, C. Mao, and Y. Luo. Graph convolutional networks for text classification. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 7370-7377, 2019.
[36] D. Yu, R. Zhang, Z. Jiang, Y. Wu, and Y. Yang. Graph-revised convolutional network. In Machine Learning and Knowledge Discovery in Databases European Conference, ECML PKDD 2020, Ghent, Belgium, September 14-18, 2020, Proceedings, Part III, volume 12459 of Lecture Notes in Computer Science, pages 378-393.
[37] H. Zhang, B. Chen, D. Guo, and M. Zhou. WHAI: weibull hybrid autoencoding inference for deep topic modeling. In 6th International Conference on Leurning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018 Conference Track Proceedings. OpenRe- net..net.
12

---

[38] H. Zhao, D. Phung, V. Huynh, T. Le, and W. L. Buntine. Neural topic model via optimal transport. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021.
[39] H. Zhao, L. Du, W. Buntine, and M. Zhou. Dirichlet belief networks for topic structure learning. Advances in neural information processing systems, 31, 2018.
[40] M. Zhou, Y. Cong, and B. Chen. The poisson gamma belief network. Advances in Neural Information Processing Systems 28: Annual Conference Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pages 3043-3051.
[41] M. Zhou, L. Hannah, D. Dunson, and L. Carin. Beta-negative binomial process and poisson factor analysis. In Artificial Intelligncce and Statistics, pages 1462-1471, PMLR, 2012.
[42] M. Zhou, Y. Cong, and B. Chen. The poisson gamma belief network, Advances in Neural Information Processing Systems, 28, 2015.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] Please refer to the appendix
(c) Did you discuss any potential negative societal impacts of your work? [Yes] Please refer the appendix
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes] The code and data are included the supplementary material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you citep the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "UzdvUY5Thiz", "reviewer": "Reviewer_1Lsj", "summary": "The goal of this paper is to learn topic hierarchies by deriving knowledge from a pre-existing hierarchy of topics. This pre-existing hierarchy does not have topical words in the nodes.\n\nTo this end, the authors develop a Bayesian deep topic model. In the topic model, the Bayesian component is the topic tree that is used as a piece of prior knowledge of the model. The deep part is derived from the Gamma Belief Network framework.  Later, the authors used GCN-based methods to model the hierarchies. Parameter estimation is then presented using amortized variational inference model. Experimental results demonstrate that the model improves upon existing methods.", "questions": "Can the authors mention some real-world applications where this work could be useful? How and why this model can be applied to real-world tasks? Section 5.8 only uses the 20NG dataset that is very small.", "limitations": "The key issue is that the model relies on an external knowledge base. One might need knowledge bases in different domains to use this model.", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The key strength lies in modelling topic hierarchies using side information. To this end, a hierarchy is used as prior knowledge and words are generated in each hierarchy.\n\nThe main weakness of the work is that it relies upon a knowledge base. While WordNet is \"relatively smaller\" in size compared to knowledge bases such as Wikidata, where the model will struggle is when the knowledge bases are huge, e.g., several gigabytes in size. It will be good for the authors to mention which hierarchies the model can work and which it cannot.\n\nWhile the authors have presented their detailed results, the question is why this work is important? The issue is that the model relies on a hierarchy that it supplied manually. There are already unsupervised topic models such as hLDA that can learn hierarchies, of course, not in the way that the authors have presented their work. However, it is still unclear how this work is a step-change in the topic modelling literature.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "A9g14ztJkhV", "reviewer": "Reviewer_3oGT", "summary": "The paper proposes a topic modeling approach that extends previous work on Gamma Belief Network [1] to jointly model a knowledge graph provided as input to guide the topics. The graph structure and the keywords representing each graph node is integrated into the generative process. The graph node embedding is modelled using a graph neural networks, which is jointly modelling during the inference process. This allows data driven changes to the knowledge graph, as illustrated as the addition of new topics in the experimental results.\n\n[1] Sawtooth Factorial Topic Embeddings Guided Gamma Belief Network. Duan et al. ICML 2021\n", "questions": "Other than addition of new nodes, does the graph structure evolve? e.g., are edges added or removed?", "limitations": "- as rightly pointed out in the appendix, it will be interesting to investigate how the knowledge in large language models can be used in such topic modeling works", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n- the proposed approach of guiding a topic model with a knowledge graph seems more natural than previous work such as [1], and in their evaluation, was shown to achieve better evaluation results compared to previous work.\n\n- the proposed model allowed the graph structure to evolve according to the data in a variant called TopicKGA, which is the best performing model\n\n- they showed that their approach converges better than TopicNet [1]\n\n\nWeaknesses\n- more ablation could be done on how GNN contributes in the model. The dynamic structure of the topic graph is only shown in the addition of new topics. Was there any changes in the graph structure other than addition of nodes? \n\n- using wordnet as the knowledge graph seems too general a way to guide a topic model. Knowledge graphs such as those derived from Wikidata or KG more specifically customized to user needs could lead to more practical usage.\n\n\n[1] TopicNet: Semantic Graph-Guided Topic Discovery. Duan et al. 2021.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "8-6CHNPKHQP", "reviewer": "Reviewer_o1DX", "summary": "This paper proposes an improved embedded topic model that considers an existing knowledge about the topic hierarchy that can potentially alleviate biased and long-tail topics and words. It considers such hierarchy information as a matrix representing the interactions of topics following a Bernoulli distribution. The experiments include various aspects of the approach. They use four datasets and compare the proposed approach with five baselines. Also, the interpretability and the coherence of the topic is empirically shown and examples are provided.", "questions": "I appreciate the authors for the efforts to incorporate prior knowledge to topic modeling. While there is some evidence of the improved performance, it was not sufficiently explained in general. Please refer to W1-W3 to address these questions. Especially, deep diving why this modeling is better than the baselines is very important to understand where the strength is coming from. Also, clarifying what comes from an existing approach, and what is novel can improve the readability of the paper greatly, and that way, this paper doesn't need to try to explain everything.\n\nAdditional comments/questions follow.\n- Q1. A GCN was used to consider the structure of the topic tree and update the node weights. How was the neural network training and the Bayesian model optimization done together?\n\n- Typos: EBLO -> ELBO. consists -> consists of. widely-used -> widely used. two adjacent layer -> two adjacent layers.", "limitations": "I don't see a negative societal impact of this work.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths.\n- S1: The proposed approach provides a way to incorporate the topic hierarchy from an external knowledge source.\n- S2: The approach was extensively evaluated with five baselines and four datasets.\n- S3: The examples and interpretability of the derived topics and their hierarchy are provided to understand the outcome of the approach.\n\nWeaknesses.\n- W1: The presentation can be improved. There is no overview of the approach to explain the components, and a few components and concepts appear without much prior context. For example, \"encoder\" appears without where it is exactly being used. Same for \"topic aggregation\". \"count vector\" was used only once without definition.\n- W2: This paper does not provide the related work description for the existing ETM this work is based on in the related work section. It also doesn't distinguish that in the method section. Thus, it is hard to evaluate the novel contribution of this paper compared to the existing approach. For example, much of the framework comes from the SawTooth paper, but this paper failed to include or summarize the overall structure.\n- W3: Many parts of the paper are not justified or explained enough. For example, how modeling the relation with a Bernoulli distribution complement Phi modeling the relations between layers in SawTooth, or why the proposed approach is better was not explained. How is the adaptive structure different from just fine-tuning the parameters? The paper quickly attributes the improved performance to the joint tree likelihood, but how exactly is it better than the other approaches such as SawTooth or TopicNet? What information does it capture that SawTooth and TopicNet do not?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept