TopicNet: Semantic Graph-Guided Topic Discovery
Authors: Zhibin Duan
Yi.shi Xu
Bo Chen
dongsheng wang
Chaojie Wang
Mingyuan Zhou
Venue: NeurIPS 2021 Poster (2021)
Keywords: Order Embeddings
Gaussian Embeddings
Knowledge Graph
Hierarchical Topic Models
Gamma Belief Network

Abstract: Existing deep hierarchical topic models are able to extract semantically meaningful topics from a text corpus  in an unsupervised manner and automatically organize them into a topic hierarchy.  However, it is unclear how to incorporate prior belief such as knowledge graph to guide the learning of the topic hierarchy. To address this issue, we introduce TopicNet as a deep hierarchical topic model that can inject prior structural knowledge as inductive bias to influence the learning. TopicNet represents each topic as a Gaussian-distributed embedding vector, projects the topics of all layers into a shared embedding space, and explores both the symmetric and asymmetric similarities between Gaussian embedding vectors to incorporate prior semantic hierarchies. With a variational auto-encoding inference network,  the model parameters are optimized by minimizing the evidence lower bound and supervised loss via stochastic gradient descent. Experiments on widely used benchmark show that TopicNet outperforms related deep topic models on discovering deeper interpretable topics and mining better document representations. 

TopicNet: Semantic Graph-Guided Topic Discovery
Zhibin Duan, Yishi Xu, Bo Chen', Dongsheng Wang, Chaojje Wang National Laboratory Radar Signal Processing, Xidian University, Xi'an, China xd_ wwwiii@163....mmmm com, bchen@mail xidian wwddin..ed.....
Mingyuan Zhou
McCombs School of Business, The University of Texas at Austin mingyuan.. zhou@mccombs. utwxas.eeccomm edu
Abstract
Existing deep hierarchical topic models are able to extract semantically meaningful topics from text corpus an unsupervised manner and automatically organize them into topic hierarchy. However, it is unclear how to incorporate prior belief such as knowledge graph to guide the learning of the topic hicrarchy. To address this issue, we introduce TopicNet prior structural knowledge as an inductive bias to influence the learning. TopicNet represents each topic topics of all layers into a shared embedding space, and explores both the symmetric and asymmetric similarities between Gaussian embedding vectors to incorporate prior semantic hierarchies. With an auto-encoding variational inference network, the model parameters optimized by minimizing the evidence lower bound and a regularization term via stochastic gradient descent. Experiments on widely used benchmarks show that TopicNet outperforms related deep topic models on discov- ering deeper interpretable topics and mining better document representations.
deep hierarchical topic model that can inject
Gaussian -istrrrbbedttteeeeddd embedding vector, projects the
Introduction
Topic models, which have the ability to uncover the hidden semantic structure in a text corpus, have been widely applied text analysis. Generally, topic model is designed to discover a set of semantically-mannine
latent topics from collection of documents, each of which captures word co-occurrence patterns commonly observed in document. While vanilla topic models, such as latent Dirichlet allocation (LDA) [] and Poisson factor analysis (PFA) (2), are able to achieve this goal, series of their hierarchical extensions [3 have been developed in the hope of exploring multi-layer document representations and mining meaningful topic taxonomies. Commonly, these hierarchical topic models aim to learn a hicrarchy, in which the latent topics exhibit an increasing level of abstraction when moving towards deeper hidden layer, as shown in Fig. Ia). Consequently, it provides users with an intuitive and interpretable way to better understand textual data.
Despite their attractive performance, many existing hierarchical topic models are purely data-driven and incorporate no prior domain knowledge, which may result some learned topics failing to describe a semantically coherent concept [13,, especially for those at a deeper hidden layer [14] Furthermore, the inflexibility of adding prior knowledge also somewhat limits the applicability of hierarchical topic models, since it is common that a user is interested in specific topic structure and only focused on information related [55]. To address this issue, we assume a structured prior in the form of predefined topic hierarchy, where each topic is described by semantic concept, with the concepts at adjacent layers following the hypernym relations, as illustrated in Fig. [b). Such a
"Corresponding Author.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

home school way road bike student
vetice
bibe, ship. rocket, Jrt, bost
vehice
Inndwehidg
aireratt
watercratt
landhveiide
airrsft
untercraft
road bike walk
school (wll home
stusienc home way
Car bike Van bus
sliprochett Airplane
Bipione let Raft
ship
pucket
road
bike
Walk
school
home
studiend
bike
rockett
joc
FARE
(a)
(b)
(c)
Figure 1: Illustration of (a) topics discovered by hierarchical topic model, (b) semantic graph constructed by
prior knowledge, and (c) topics learned by TopicNet, the proposed knowledge-basee hierarchical topic model.
hierarchy can be easily constructed, either by some generic knowledge graph like WordNet [66 or according to the user's customization. However, there are two main challenges: one is how to model this semantic hierarchy, and the other is how to combine it with topic models.
For the first challenge, a general and powerful approach is to learn a Distance-preseevvie mapping, which maps semantically similar topics nearby points in the embedding space, with symmetric distance measure (e.g- Euclidean or cosine distance) typically being used. However, the embeddings learned by such a scheme can not perfectly reflect the entailment between concepts. To mitigate this issue, Vendrov et al. [18] exploit the partial order structure of the semantic hierarchy to learn order embeddings that respect this partial order structure. On the other hand, considering that probability distributions are better at capturing uncertainties of concepts than point vectors, Athiwaratkun and Wilson [99 introduce density order embeddings as an improvement to order embeddings. Density order embeddings allows the entailment relationship to be expressed naturally, with general words such as "entity" corresponding to broad distributions that encompass more specific words such as "animal." Through encapsulation of probability densities, it can intuitively reflect the hierarchical structure of semantic concepts, thus offering us an effective way to model the topie hierarchy.
As for the second challenge of combining the constructed topic hierarchy with topic models, there are two major difficulties. For one thing, the structured prior requires modeling the relationship between topics across different layers directly, while hierarchical topic models often assume that the topics different layers are independently drawn in the prior [6]. which makes the fusion of the two bit far-fetched. For another, since the constructed topic hierarchy can be very deep for large-scal scagle text corpora, an equally deep topic model with multiple latent layers needed to incorporate with it. However, most existing deep neural topic models are hard to go very deep due to the problem of posterior collapse [1 25). Fortunately, SawETM [4,, a recently proposed neural topic model, not only builds the dependency between topics at different layers in shared embedding space, but also alleviates the issue of posterior collapse to some extent through a novel network module referred to as Sawtooth Connection (SC), and hence overcoming both difficulties.
In this paper, novel knowledge-baadd based hierarchical topic model has been proposed, the core idea of which is to represent each topic as a Gaussian embedding, and then project the topics of all layers into a shared embedding space to facilitate the injection of structured prior knowledge into the model. In particular, we first introduce the variant of SawETM with a stochastic decoder called Gaussian SawETM. To incorporate prior belief into Gaussian SawETM, we then constrain the topics at adjacent layers to encourage them to capture the concepts satisfying the predefined hypernym relations. With auto-encoding variational inference, the entire model is learned in an end-to-end manner by minimizing the evidence lower bound and regularization term. Extensive experiments demonstrate our model has competitive performance and better interpretability in comparison to most existing hierarchical topic models.
embeddings to the SawETM, resulting in
The main contributions of the paper can be summarized as follows:
To capture semantic uncertainties of words and topics. we propose a novel probabilistic generative model with stochastic decoder, referred to as Sawtooth Factorial Gaussian Topic Embeddings guided gamma belief network (Gaussian SawFTM).
To incorporate the structured prior knowledge from the real world, we propose TopicNet, novel knowledge-base based hierarchical topic model based on Gaussian SawETM.
In addition to detailed theoretical analysis, we conduct extensive experiments to verify the effectiveness of the above two models. One of the most appealing properties of TopicNet is

---

its interpretability. We conduct extensive qualitative evaluation on the quality of the topics discovered by TopicNet, including word embedding, topic embedding, and topic hierarchies.
2
Gaussian SawETM
In this section, we elaborate on the construction and inference of Gaussian SawETM, a deep hierar- chical topic model that represents words and topics as
embeddings [26].
2.1 Symmetric similarity: expected likelihood kernel
The distance measure plays a key role in quantifying the similarities between embeddings. A simple choice is taking the inner product between the means of two Gaussians, which, however, does not take the advantage of the semantic uncertainties brought by we employ the expected likelihood kernel
embeddings. Here as our similarity measure, which defined as
19 26
27
| Jre MERR
E (a, a2)
N (x; M, E N (z; M1 Ej)dx = (0: u. My E, + +2)
(1)
where ae, is Gaussian distribution with mean μ4, and diagonal covariance matrix Σ, As symmetric similarity function, this kernel considers the impact of semantic uncertainties brought by covariances.
2.2 Document decoder with sawtooth factorization and Gaussian embeddings
The generative network (also known as decoder) is one of the core components of topic models. As discussed in Section to build dependency between topics two adjacent layers and learn deep topic hierarchy, we draw experience from the Sawtooth Connection (SC) in SawETM. We develop a stochastic decoder by introducing -eussian-distributee embeddings to better represent the topics in SawETM. Formally, the generative model with T latent layers can be expressed as
Pois( (1) (( {0f") Gam 4 (4+1) gftt)) 1/0 T-1 1=1 T ~ -am 1/c, (T+1)),
④b (+) Softmax (lgg (E()) (a (1-1) (0) {)))
1= fal ~ Nax; (( }
(e)
In the above formula, 3ÿ E ZV denotes the word count vector of the jth document, which is factorized as the product of the factor loading matrix ((1) E <R\ and gamma distributed factor score ( under the Poisson likelihood; and to obtain a multi-layer document representation, the hidden units o E R of the 1th layer are further factorized into the product of the factor loading E
T (c+1)
K2 Ke11 and hidden units of the next layer. The top-layer hidden units (T) are sampled !
from prior distribution. In addition, the 4thh topic o at layer / follows a Gaussian distribution with mean fu
(t) and covariance {) where the mean vector describes a semantic concept and the covariance matrix reflects its level of abstraction. Note that the bottom layer a (0) represents the distributed embeddings of words. And & used to capture the relationships between topics at two adjacent layers, is calculated based on the topic representations the shared embedding space, instead of being sampled from Dirichlet distribution. In particular, in the equation (-)(() refers to the symmetric similarity function defined in Eq.
2.3 Document encoder: Weibull upward and downward encoder networks
As presented in Section 2.2 instead of using Gaussian latent variables like most of neural topic models (28). our generative model employs the gamma distributed latent variables that are more suitable for modeling sparse and non-negative document representations. While in sampling based inference, the gamma distribution is commonly used to represent the conditional posterior of these latent variables, the difficulty of reparameterizing gamma distributed random variable makes it difficult to apply it to an inference network 22 30 For this reason, we utilize Weibull upward- downward variational encoder inspired by the work in Zhang et al. [25] 31]. We let
-) = (eibull(k (0) A, (0)
(3)
3

---

object
organism
living thing
organism
living thing
animal
Person
0/3
iramal
insect
animal
plant
person
temale
g/
X
mammal
insect
plant
male
female
(a)
(b)
(c)
Figure 2: Overviews of (a) generative model of Gaussian SawETM, (b) TopicTrree constructed by WordNet, and (c) Density order embeddings where specific entities correspond concentrated distributions encapsulated broader distributions general entities.
where, the parameters k.D...' ,/
K, are deterministically transformed from both the observed
document features and the information from the stochastic up-down path 0 (++1) In detail, the inference network can be expressed as
h (0) = (e + 1)
n/ = h2 (1-1) ReLU (00 h +b()), 1= 11.... .,
h h 10) ( ((t))
0=0..... -.. h (T) h
(TT)
(4)
b (2) Softplus( (W
! h2)) + b())
n Softplus( w/
(1) (0) by
h
t= t=
3.T where {b. Ji=1,1=1 E RK (w! 3,7 Ji=1,t=1 E K. and { (1) N.T
Jj=1,==1 Rk @ denotes the concatenation in feature dimension, ReLU(.) max(0, is the nonlinear activation function, and Softplus(-) applies In + exp(-)] to each element to ensure positive shape and scale parameters of the Weibull distribution. To reduce the risk of posterior collapse, a skip-connected deterministic upward
path [4] is used to obtain the hidden representations {h, N.T N.T
f==1,(=1 of the input xz..
2.4 Inference and estimation
Similar to variational auto-encoders (VAEs) 32 33 the training objective of Gaussian SawETM is to maximize the Evidence Lower Bound (ELBO):
q00%
E Inp(z, (1) ]
LELBO
I
(5)
( ((++1) o
J=1
where --) is the variational Weibull distribution in Eq. B. and p(0 is the prior gamma distribution in Eq. (2). The first term is the expected log-likelihood or reconstruction error, while the second term is the Kullback-Leibler (KI.) divergence that constrains q(O()) be close to its prior p()!)) the generative model. The analytic KI. expression and simple reparameterization of the Weibull distribution make simple to estimate the gradient of the ELBO, with respect to { T o of Gaussian embeddings and other parameters in the inference network.
and {Σ (1) [T
3 TopicNet: Semantic graph-guided topic discovery
In Section 2 novel hierarchical topic model called Gaussian SawET'M is proposed to discover meaningful topies organized into a hierarchy. Below we describe our solution to incorporate prior belief, e.g., knowledge graph, into Gaussian SawETM to guide the learning of the topic hierarchy.
3.1 TopicTree:: Semantic graph constructed by prior knowledge
Finding an appropriate way to encode prior knowledge especially important for realizing the incorporation of prior belief into topic models. Due to the hicrarchical structure of deep topic models,

---

correspondingly, we represent prior knowledge in the form of a hierarchical semantic graph and we call it TopicTree. Particularly, in the TopicTree each node is described by a semantic concept and the concepts of nodes at two adjacent layers follow the entailment relations, which results in a bottom-up process of gradual abstraction, as shown in Fig. 2(b) Besides, it is worth mentioning that some generic knowledge graphs such as WordNet [T6] provide us a convenient way to build the TopicTrree, since the hypernyms or hyponyms of any given word can be easily found in them.
3.2 Partial order: Asymmetric entailment relationship
As described in Section 3.1 we organize prior knowledge into a hierarchical semantic graph so that it can be incorporated into Gaussian SawETM naturally. Specifically, for constructed Topic Tree with depth T. we build a Gaussian STWETM with T latent layers corresponding to it, then we match the topics of each layer in the Gaussian SawETM with the semantic concepts of the corresponding layer in the Tree i.e., each topic has corresponding semantic concept to describe it. Most significantly, the relationships between semantic concepts in the TopicTree should also be injected to guide the learning of the topic hierarchy. Since the Gaussian SawETM projects the topics of all layers to a shared embedding space, such a goal can be achieved by imposing constraints on the topic representations in the shared embedding space.
Inspired by Vendrov et al. 18), the semantic hierarchy (akin to the hypernym relation between words) of the Topic'Tree can be seen as special case of the partial order structure. In detail, a partial order over set of points X is binary relation V such that for a, b, E X, the following properties hold: 1) a I (reflexively): 2) if a I 6 and b < a then a = 6 (anti-symmetry); 3) if a < b and 6 < then 3 e (transitivity). Vendrov et al. [18) propose learning such asymmetric relationships with order embeddings: they represent semantic concepts as vectors of non-negative coordinates, and stipulate that smaller coordinates imply higher position in the partial order and correspond to a more general concept. For example, the following expression defines an ordered pair (z, y) of vectors in R.:.
x I if and only if A x. 2 y::
Since the order embedding condition defined by Eq. 66 is too restrictive satisfy by imposing hard constraint, Vendrov et al. [18] relax the condition and turn to seek an approximate order-embedding In particular, for an ordered pair of points, they apply a penalty that is formulated as:
E(x, y) = || max (0, ==
Note that this imposes a strong prior on the embeddings, which could encourage the learned relations to satisfy the partial order properties of transitivity and antisymmetry.
Although Gaussian-distribute embeddings are used to represent topics in our Gaussian SawETM, we find that they are also natural at capturing the partial order structure (or the semantic hierarchy of the TopicTree) [19]. For instance, for two topics following the hypernym relation, their Gaussian- distributed embeddings a; and a form an ordered pair. Assuming that a; represents the specific topic and a represents the general one, then we have a I azj meanwhile we expect that aj corresponds to broad distribution that encompasses a; To meet this expectation, the KL divergence between as and Oj provides us a ready-made tool:
/ BER"
E (as, a.)) DEL (N: [1)
N (æ: ps. Ex) (N (w; p4, E.)/N (æ; M4. 2,)) dæ
2 (tr E.) (m, (iu-M)) ,,)' (m. 11)) d log (det (E,) (Σ.)))
However, using E (a,, a,) directly as penalty for violating the partial order is undesirable. Since the KI divergence has a property that E (ar;, ax)) if and only if a; aj, which means the penalty is zero only if a azj. but we expect the penalty should also be zero when a, o aj and a; 3 ajj- For this reason, following Athiwaratkun and Wilson [99, we consider using a thresholded divergence our penalty for order violation:
d, (ass = max (a.,
(8)
So this penalty can be zero if or is properly encapsulated in o.,. Nevertheless, note that we no longer have the strict partial order with such a penalty. In other words, the transitivity and anti-symmetry are not guaranteed. Therefore, actually our learned relations also respect approximate order structure.
5

---

3.3 Objective: ELBO with prior regularity
Essentially, TopicNet is a topic model, so the ELBO still plays a leading role the training objective. At the same time, regularization term is also necessary for injecting hierarchical prior knowledge to guide the topic discovery. Therefore, the final objective of TopicNet can be written as:
LopicNet = LELBO + P
where /3 is a hyper-parameter used to balance the impact of these two terms. Particularly, for the ELBO term LELBO it is the same as defined by Eq. (5). As for the regularization term prior- cannot only constrain topics that follow the hypernym relation between two adjacent layers, but the topics that do not follow the hypernym relation should also be considered. This is consistent with the notion of positive and negative sample pairs in contrastive learning. Based on this idea, we use max- margin loss to encourage the largest penalty between positive sample pairs be below the smallest penalty between negative sample pairs, which is defined as:
L (++1 max (0. m max (d, (i,j))iED + min {d,, (1.5)
(10)
This represents the penalty for the jth topic at (t + 1)th layer, and the set of hyponym topics of topic j. For i E D. (i.j) forms a positive pair, for i D. (i, forms negative pair. Parameter m is the margin that determines the degree to which the positive and negative pairs are separated. Considering the penalties for all topics each layer, we obtain the prior regularization term.
L Lerior >'' KOI
D (1+1)
(11)
With the final objective. TopieNet learns the topics influenced by both the word co-occurrence information and prior semantic knowledge, thus discovering more interpretable topics. Note that, in experiments, the hyper- parametrss are m 10.0 and B 1.0.
3.4 Model Properties
In this part, we recap several good properties of TopicNet: Uncertainty, Interpretability, and Flexibility.
embeddings: Mapping a topic to a distribution instead of a vector brings many advantages, including: on one hand, better capturing uncertainty about representation, where the mean indicates the semantics of a topic and the covariance matrix reflects its level of abstraction; on the other hand, expressing asymmetric relationships more naturally than point vectors, which is beneficial to modeling the partial order structure of semantic hierarchy [19 26
Effectiveness of
Hierarchical interpretable topics: In conventional hierarchical topic models, the semantics of a topic is reflected by its distributed words, and we have summarize semantically coherent concept manually by post-hoc visualization. In some cases, the topic is too abstract to get a semantically coherent concept, which makes hard to interpret. Such cases are common deeper hidden layers. While in TopicNet, we know in advance what semantic concept a certain topic corresponds to, since it is encoded from prior knowledge and used guide the topic discovery process of the model as a kind of supervision information. Therefore, once the model is trained, the words distributed under each topic are semantically related the concept that describes it, shown in Fig. From this perspective, our TopicNet has better interpretabilityy
Flexibility to incorporate prior belief: Traditional topic models learn topics only by capturing word co-occurrence information from data, which do not consider the possible prior domain knowl- edge (e.g., Knowledge Graph like WordNet (34).. However, TopicNet provides a flexible framework incorporate the structured prior knowledge into hierarchical topic models.
Experiments
We conducted extensive experiments to verify the effectiveness Gaussian SawETM and TopieNet. Our code is available at htttts://ggthub.. http://hwww.ooooo/occeeeerrr/r/p

---

Table 1: Comparison of per- per-heldout-worf perplexity on three different datasets.
Method
Depth
20NG
RCV1
Wiki
Method
Depth
20NG
RCV1
Wiki
LDA I
735
942
1553
DNTM-WHAI
25
1
762
952
1657
AVITM [38
784
968
1703
DNTM-W WHAI
25
726
906
1595
ETM 35
742
951
1581
DNTM WHAI
25
15
724
902
1.922
DLDA-Gibbs
36
702
14
1
718
908
1536
DLDD--GibE Gibbs
16
678
DNTM- FawETM
14
5
688
873
1503
DLDA-Gibbs
36
670
DNTM-SuwETM
04
15
684
864
1492
DIDA- ASGGR
7
714
912
1455
DNTM GaussSowFTM 1
714
904
1.529
DLDA-TLASGR
684
877
1432
DNTM-Gaue GaussSawETM
5
685
866
1477
DLSA-TTLASCR
842
DNTM GaussSawFTM
15
678
857
1452
Baseline methods and their settings: We compare Gaussian SawETM with the state-of-the-art topic models: 1. LDA Group, including: latent Dirichlet allocation(LDA)) [. which is a basic probabilistic topic model; LDA with Products of Experts (AVITM) (28). which replaces the mixture model in LDA with a product of experts and uses the AVI for training; Embedding Topic Model (ETM) (35. a generative model that incorporates word embeddings into traditional topic model. 2. DLDA Group, including: Deep LDA inferred by Gibbs sampling (DLDA-Gibbs) (36] and by TLASGR-MCMC (DLDA -LASGR) 7 Deep Neural Topic Model (DNTM) Group, including Weibull Hybrid Autoencoding Inference model (WHAI) (25], which employs deep variational encoder to infer hierarchical document representations, and Sawtooth Factorial Embedded Topic Model (SawETM) [14]. where topics are modeled as learnable deterministic vectors. For all baselines, we use their official default parameters with best reported settings.
Datasets Our experiments are conducted four widely-used benchmark datasets, including 20Newsgroups (20NG), Reuters Corpus Volume (RCV1), Wikipedia (Wiki), and a subset of the Reuters-21578 dataset (R8), varying in scale and document length. 20NG, with a vocabulary of 2,000 words, has 20 classes and was split into 11,3 11,314 training and 7 7,532 test documents. RCV1 consists of 804,414 documents with vocabulary size of 8,000. Wiki, with vocabulary size of 10,000, consists of 3 million documents randomly downloaded from Wikipedia using the script provided by Hoffman al. (37]. R8, with vocabulary size of 10,000, has 8 classes and was split into 5,485 training and 2, 189 test documents. The summary statistics of these datasets and other implementation details (such as dataset preprocessing and length statisties) are described in the Appendix.
4.1 Unsupervised learning for document representation
Per-heldout-word perplexity: To measure predictive quality of the proposed model, we calculate the per-heldout-won word perplexity (PPI.) (7]. on three regular document datasets, e.g., 20NG, RCV1, and Wiki. For deep topic models in the DLDA and DNTM groups, we report the PPL. with 3 different stochastic layers
E {1, 5, 15). Specifically, for a 15-layer model, the topic size from bottom to top is set as K = [256, 224, 192, 160, 128, 112, 96, 80, 64, 56, 48, 40, 32, 16, 8). and the detailed description can be found in the Appendix. For each corpus, we randomly select 80% of the word tokens from each document to form a training matrix X, holding out the remaining 20% to form testing matrix Y. We use X to train the model and calculate the per-held- word perplexity as
V Den In Σ A (( g(l)> D. EV D 1 10
1
Σ
Σ
where S is the total number of collected samples and y. = =V u= -m= Ben- As shown in Tab. overall, ALDA-Gibbs perform best in terms of PPL. which is not surprising as they can sample from the true posteriors given a sufficiently large number of Gibbs sampling iterations. mini-batch based algorithm that is much more scalable in training than DLDA-Gibbs, at the expense of slightly degraded performance in out-of sample prediction. Apart from these deep probabilistic topic models, Gaussian SawETM outperforms other neural topic models in all datasets. In particular, compared with Gaussian based single layer neural topic model, WHAI performs better, which can be attributed to two aspects, the first is the effectiveness of Weibull distribution in modeling sparse and non-negative document latent representations, and the second is aggregating the information in both the higher (prior) layer and that upward propagated to the current layer via inference network. Benefiting from the embedding decoder and the Sawtooth Connection that builds the dependency
is
7

---

Table 2: Comparison of document classification and clustering performance.
Methods
Depth
Classification 20NG
Clustering
R8
20NG
R8
ACC
NMI
ACC
NM1
LDA M
72.6.10.3
88.4.0.5
46.5.10.2
45.11.0.4
51.4.00.
40.3.10.3
NVITM 33
71.510.4
87.310.2
48.310.5
46.410.3
52,510.2
41.2110.4
ETM 35
71940.2
87.940.4
49.840.6
48.420.5
55.340.4
42.34:022
PGBN 6
74.2+005
90.8+0.5
46.6+0.5
45.4+0.6
51.7+004
40.7+0.4
PGBN
76.0110.3
91.610.44
48.24:0.2
46.520.5
54.41004
41.2400.3
WHAI 25
72.8+0.3
85.3+0.2
49,4+0.5
46.5+0.4
57.940.5
42.3+0044
WHAT 25
73.7+00.5
88.4+0.3
49.5++044
47.0++0.3
60.4+044
44.1+0.2
SawETM
70.5+0.3
88.6+0.2
50.2+0.2
48.7=0.3
61.240.44
43,4+:04
SawETM
71.1+0.2
90.5+0.3
51.2+003
50.7+0.2
63.8+0.3
45.9+005
Gaussian SawETM
72.8+0.3
89.8+0.44
51.3+004
50.6+
61.5++0.3
44.1++0.2
Gaussian STwETM
75.210..4
91.0.10.3
53.0.0.3
$1.310.2
65.2.10.4
47,0.1..3
TopicNet
71.6+0.2
88.6+0.3
49.5++0.3
46.3++0.4
61.0++022
43.2+0.3
TopicNet
74.310..
89.5
50.8.t 0.3
47.8:10.4
64.2:10.3
46.5.:0.2
of hierarchical topics, SawETM performs better than WHAI, especially deeper layers. However, modeling topics with point vectors, SawETM cannot capture the uncertainties of topics. Gaussian SawETM represents the topies with dautrian-distributed embeddings to capture the uncertainties, and achieves promising improvements over SawETM, especially for modeling complex and big corpora such as Wiki. Note that although the methods in DLDA group get better performance, they require iteratively sampling to infer latent document representations in the test stage, which limits their application [25]. By contrast, Gaussian SawETM can infer latent representations via direct projection, which makes it both scalable to large corpora and fast in out-of-sample prediction.
Document Classificatton & Clustering Document-topic distributions can be viewed as unsuper- vised document representations, to evaluate their quality, we perform document classification and clustering tasks. In detail, we first use the trained topic models to extract the latent representations of the testing documents and then use logistic regression to predict the label and use k-means predict the clusters. we measure the classification performance by classification accuracy, and clustering performance by accuracy (ACC) and normalized mutual information (NMI), both of which are the higher the better. we test the model performance 20NG and R8, where the document labels are considered. Different from the perplexity experiments, a larger vocabulary of 20,000 words used for 20News to achieve better performance. The network structure is defined by TopicTreee, which introduced Section 3.1 with the details deferred to the Appendix. The comparison results are summarized in Tab. 2 Benefiting from the use of Gaussian distributions topics, Gaussian SawETM outperforms the other neural topic models on not only document classification but also clustering, demonstrating its effectiveness of extracting document latent representations. Note that Gaussian SawETM acquires promising improvements compared to SawETM the 20NG dataset, showing the potential for modeling more complex data. With the predefined knowledge prior, TopieNet does perform better, possibly because the introduced prior information does not well align with the end task. We leave the refinement of prior information for specific tasks future study.
4.2 Interpretable topic discovery
Apart from excelling at document representation learning, another appealing characteristic of Top- icNet that it can discover interpretable hierarchical topics. In this section, we perform the topic discovery experiment with 7-layer TopicNet trained 20NG. The network structure is constructed corresponding to the structure of TopicTree, which is set as [411, 316, 231, 130, 46. 11, 2].
We show six example topics and their corre- sponding children nodes in Tab. 3 which illustrates how a particular concept is being represented the 20 newsgroups. For example, the "military_action" concept, which has "assault", "defense", and "war" as its children nodes, has become tightly connected to the Waco Siege heavily discussed in this dataset; the "macromolecule" concept, which has "oil" as single child node, has been represented by topic related to vehicle, which is interesting the 20newsgroups dataset has the
Prior knowledge and corresponding learned topic:
8

---

Table 3: Semantic graph and corresponding learned topic on the bottom layer.
Topic
Concept
Children word nodes
Topic words
gun fbi guns koresh batf
11
militury_action
ussault defense war
waco assault children compound weapons
address shareware software unix windows
windows dos driver microsoft
coding_system system
drivers running applications unix using network
car bike cars engine
178
macromolecule
oil saturn ford riding miles road
govermment law rights state police right foderal shall crime court laws
186
polity
government
space launch nasa gov satellite carth mission shuttle hst orbit sky
atmosphere
sky
black blue brown color green red
color green blue red black brown led subject showed lines
color
following two newsgropus- "rec.autos" and "rec. frec.motorcycles" not newsgroup clearly related biology and chemistry; and the "Atmosphere" concept, which has "Sky" its single child node, has been represented by topic related to NASA, reflecting the fact that "sci.space" is one of the 20 news groups.
Topic quality: Two metrics are considered here to evaluate the topic quality. The first metric is topic coherence, which is computed by taking the average Normalized Pointwise Mutual Information (NPMI) of the top 10 words each topic [39]. It provides quantitative measure of the semantic coherence of a topic. The second metric is topic diversity [35,, which suggests the percentage of unique words in the top 25 words of all topics. Diversity close to 1 means more sundry topics. Following Dieng et al. B5), we define topic quality as the product of topic coherence and topic diversity. Note that topie quality is affected by the topic size, so it makes sense to compare different models on the same layer. As shown in Fig. 3 Gaussian SawETM performs better compared to SawETM, which can be attributed to the Gaussian-distributee distrband embeddings that capture the semantic uncertainties of topics. Incorporating the hierarchical prior knowledge TopicNet further achieves higher score than Gaussian SawETM. However, gets a lower score compared to DLDA-Gibl in the first two layers, the layer goes deeper, TopicNet performs better. This is easy to understand since information from data reduces slightly the shallow layers but severely in the deep layers.
0.6
Visualisation of embedding space: First of all. use 3-layer Gaussian SawETM with 2-dim embedding trained on 20NG for the Gaussian-distribute embedding visualization.
DLDA-Gibbs
DEDA-TLASGR
0.5
WHLA
SaWETM
Tanssian-SwWETM
0.4
TopicNet
The top words from the t6 topic at th layer visualized in Fig. 4[a] As see, the words under the same topic are closer in the embedding space, which demonstrates the learned embeddings are semantically sim- ilar. Apart from the semantics of words and topics, the shadow range describes the uncertain- ties and reflects the abstraction levels. Secondly, we visualize the means of
Topic Quality
0.3
0.2
0.1
embeddings learned by TopicNet with T-SNE 44). As shown in Fig. 4.2 in the shared embed- ding space, the words with similar semantics are close to each other, meanwhile they all locate near topic that has similar semantics them, which means the TopicNet effectively captures the prior semantic information and discovers more interpretable topics.
Number of Layer
Figure 3: Topic quality as measured by normalized
product of topic coherence and topic diversity (the higher the better) with the varied number of layers. Each layer's topic quality is influenced by its layer size, which is set as [411, 316, 231, 130, 46, 11, 2] from the bottom layer the top layer.

---

equipment
50
seeling
file
mac
bbs
sale
gell
evindows dos
price shipping
1.40: saftware
space
gun
2-165: sheet
microsoft
white weapons
guns
crime
administration,
rights
american
4-128: morality,
jesus
chrigtinn
law
americans
218: celestial body
goverimeny
Junar
laws
Clinton
4 inhabitant
feedorg
earth
church
god say
believe
1-172: moond
christians
217 god
Saith
christian
moon
religion
3-119: good
-1
(a) Words Embedding
(b) Topic Embedding
Figure 4: (a) 2-dim Gaussian embedding in Gaussian SawETM, which we choose the top four words for the topic and (b) T-SNE visualisation of the mean of Gaussian distribution embedding in TopicNet. (The Topic: t-j xx denotes the jth topic at t layer and xx represent the concept about the topic.)
yd physical univeriity people Toxas nows years
ya Layer: thanks reliion good
6t Layer: sadio gas uniersity splanss
layer: agent cors drive perfo way buy right sing
Layen: prychological Chrizzan belleve falth murality distribution
layer: posting wittes com privacy say mall
2d layar: code windows dos file microsoft Mes software running
2 leyer: equipment mat modem bos tax phone demon tonyy internal mail data
2n layer: Compound water ofl ai best service radio hou's gas high small cold
layer: spiritual being god say beleeve evil tatth othelst mean grace things
2a serviceman General group community socizty groups present term
software windows dos file Marosult
address widhoss
Sanslite Information control
electronic equipment modem lés tonv hours gas
Binary compound
organic compound writes subject
angel Michael mike folth
god belleee
milltary officer general information
enlisted person teele
support gystems
diplay leat using color
socicty present
cotions
data
years
Figure 5: An example of hierarchical topics learned by 7-layer TopicNet, only show example topics at the top two layers and bottom two layers. Boldface indicates the concept of each topic.
Hierarchical structure TopicNet: Distinct from many existing topic models that rely on post-
hoc visualization to understand topics, we can casily understand the semantic of the whole stochastic network via the predefined TopicTreee Specially, the hicrarchical structure
TopicNet is shown in Fig. 6 The semantic of each topic can be represented by the concept in the Topic which is more interpretable compared traditional topic models. This result further confirms our motivation described in Fig.
Conclusion
To equip topic models with the flexibility of incorporating prior knowledge, this paper makes meaningful attempt and proposes a knowledge-basee based hierarchical topic model TopicNet. This model first explores the use of Gaussian-distributee embeddings to represent topics, and then develops a regularization term that guides the model to discover topics by a predefined semantic graph. Compared with other neural topic models, TopicNet achieves competitive results on standard text analysis benchmark tasks, which illustrate a promising direction for the future development text analysis models toward better interpretability and practicability.
10

---

Acknowledgments
Bo Chen acknowledges the support of NSFC (U21B2006 and 61771361), Shaanxi Youth Innovation Team Project, the 111 Project (No. B1 B18039) and the Program for Oversea Talent by Chinese Central Govermment.
References
[1] David M Blei, Andrew Y Ng, and Michael I Jordan. Latent Dirichlet Allocation. Journal of machine Learning research, 30Jan)993-1022, 2003.
[2] Mingyuan Zhou, Lauren Hannah, David Dunson, and Lawrence Carin. Beta-negative binomial process and Poisson factor analysis. In Artificial Intelligence and Statistics, pages 1462-1471, 2012.
[3] David M Blei, Thomas I. Griffiths, and Michael I Jordan. The nested Chinese restaurant process and bayesian nonparametric inference of topic hierarchies. Journal of the ACM (JACM), 57(2): 1-30, 2010.
[4] John Paisley, Chong Wang. David M Blei, and Michael Jordan. Nested hierarchical Dirichlet processes. IEEE Transactions on Pattern Analysis and Machine Intelligence, 37(2):256-22 270, 2014.
[5] Zhe Gan, Changyou Chen, Ricardo Henao, David Carlson, and Lawrence Carin. Scalable deep Poisson factor analysis for topic modeling. In International Conference on Machine Learning, pages 1823-1832, 2015.
[6] Mingyuan Zhou, Yulai Cong, and Bo Chen. Augmentable gamma belief networks. The Journal of Machine Learning Research, 1 17(1)56556-569, 2016.
[7] Yulai Cong, Bo Chen, Hongwei Liu, and Mingyuan Zhou. Deep latent Dirichlet Allocation with lopic-layer-udaptive stochastic gradient riemannian memc. arXiv preprint arXiv:1706.01724 2017.
[8] He Zhao, Lan Du, Wray Buntine, and Mingyuan Zhou Dirichlet belief networks for topic structure learning. arXiv preprint 1rXiv:/811.007/11 1.00717, 2018.
[9] Dandan Guo, Bo Chen, Hao Zhang, and Mingyuan Zhou. Deep Poisson gamma dynamical systems. arXiv preprint arXiv: 1810. 1209, 2018
[10] Chaojie Wang, Bo Chen, and Mingyuan Zhou Multimodal Poisson gamma belief network. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
[11) Chaojie Wang, Bo Chen, Sucheng Xiao, and Mingyuan Zhou. Convolutional Poisson gamma belief network. In International conference on machine learning, pages 6515-6525. PMLR, 2019.
[12] Chaojic Wang. Hao Zhang, Bo Chen, Dongsheng Wang. Zhengjue Wang, and Mingyuan Zhou. Deep relational topic modeling via graph Poisson gamma belief network. Advances in Neural Information Processing Systems, 33, 2020.
[13] Jonathan Chang, Jordan Boyd-Graber, Chong Wang, Sean Gerrish, and David M Blei. Reading tea leaves: How humans interpret topic models. In Neural information processing systems, volume 22, pages 288-296. Citeseer, 2009.
[14] Zhibin Duan, Dongsheng Wang. Bo Chen, Chaojie Wang. Wenchao Chen, Yewen Li, Jie Ren, and Mingyuan Zhou. Sawtooth factorial topic embeddings guided gamma belief network. In ICML 2021: International Conference on Machine Learning. July 2021.
[15] Yu Meng, Yunyi Zhang, Jiaxin Huang, Yu Zhang, Chao Zhang, and Jiawei Han. Hierarchical topic mining via joint spherical tree and text embedding. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pages 1908-1917, 2020.

---

[16] George A Miller. Wordnet: lexical database for english. Communications of the ACM, 38(11): 39-41, 1995.
[17] Sumit Chopra, Raia Hadsell, and Yann LeCun. Learning a similarity metric discriminatively, with application to face verification. In 2005 IEEE Computer Society Conference Computer Vision and Puttern Recognition (CVPR'0S) (0)), volume 1, pages 539-546. IEEE, 2005.
[18] Ivan Vendrov, Ryan Kiros, Sanja Fidler, and Raquel Urtasun. Order-embeddings of images and language. arXiv preprint FrXiv:1511.06688 2015.
[19] Ben Athiwaratkun and Andrew Gordon Wilson. Hierarchical density order embeddings. arXiv preprint 180404434 2018
[20] X. Chen, D. P. Kingma, T. Salimans, Y. Duan, P. Dhariwal, J. Schulman, I. Sutskever, and
P. Abbeel. Variational lossy autoencoder. 2016.
[21] I. V. Serban, A. Sordoni, R. Lowe, L. Charlin, J. Pineau, A. Courville, and Y. Bengio. A hierarchical latent variable encoder-decoder model for generating dialogues. 2016.
[22] Avd Oord, O. Vinyals, and K. Kavukcuoglu. Neural discrete representation learning. 2017.
[23] S. Zhao, J. Song, and S. Ermon. Towards deeper understanding of variational autoencoding models. 2017.
[24] Casper Kaae Sønderby, Tapani Raiko, Lars Maaløe, Søren Kaae Sønderby, and Ole Winther. Ladder variational autoencoders. In Advances in neural information processing systems, pages 2016.
[25] Hao Zhang, Bo Chen, Dandan Guo, and Mingyuan Zhou. WHAI: Weibull hybrid autoencoding inference for deep topic modeling. In International Conference on Learning Representations, 2018.
[26] Luke Vilnis and Andrew McCallum. Word representations via Gaussian embedding. arXiv preprint arXiv: 1412.6623, 2014.
[27] Tony Jebara, Risi Kondor, and Andrew Howard. Probability product kernels. The Journal of Machine Learning Research, 5:819-844, 2004.
[28] Akash Srivastava and Charles Sutton. Autoencoding variational inference for topic models. arXiv preprint arXiv: I 1703.01488, 2017.
[29] Xinjie Fan, Shujian Zhang, Bo Chen, and Mingyuan Zhou Bayesian attention modules. In NeurlPS 2020: Advances in Neural Information Processing Systems, Dec. 2020. URL
https://arxix: org/ abs 10604 (the first two authors contributed equally).
[30] Shujian Zhang, Xinjie Fan, Bo Chen, and Mingyuan Zhou. Bayesian attention belief networks.
In ICML. 2021: International Conference on Machine Learning, July 2021. URL https
//arxiv. www.oorggg//bbeeeeeeeoee abs////s..././/.//////a 05251 (the first two authors contributed equally).
[31] Hao Zhang, Bo Chen, Yulai Cong, Dandan Guo, Hongwei Liu, and Mingyuan Zhou. Deep autoencoding topic model with scalable hybrid Bayesian inference. To appear in IEEE TPAMI, 2020. URL URL hhtt:://aarxti
org/ wabp/2006..o. 08804
[32] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint 1rXiv:1312.6114, 2013.
[33] Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and variational inference in deep latent Gaussian models. In International Conference on Machine Learning, volume 2, 2014.
[34] Christiane Fellbaum. Wordnet. In Theory and applications of ontology: computer applications, pages 231-243. Springer, 2010.
[35] Adji B Dieng, Francisco JR Ruiz, and David M Blei. Topic modeling in embedding spaces. Transactions of the Association for Computational Linguistics, 8:439-453, 2020.
12

---

[36] Mingyuan Zhou, Yulai Cong, and Bo Chen. The Poisson gamma belief network. Advances in Neural Information Processing Systems, 28:3043-30511 2015.
[37] Matthew Hoffman, Francis R Bach, and David M Blei. Online learning for latent Dirichlet Allocation. In advances in neural information processing systems, pages 836-864. Citeseer, 2010.
[38] Guangxu Xun, Vishrawas Gopalakrishnan, Fenglong Ma, Yaliang Li, Jing Gao, and Aidong Zhang. Topic discovery for short texts using word embeddings. In 2016 IEEE 16th international conference on data mining (ICDM), pages 1299-1304. IEEE, 2016.
[39] Nikolaos Aletras and Mark Stevenson. Evaluating topic coherence using distributional semantics. In Proceedings of the 10th International Conference on Computational Semantics (/WCS 2013) Long Papers, pages 13 22, 2013.
[40] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, (11)), 2008
[41] Jacob Devlin, Ming-Wor Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXtv:1810.04805. 2018.
[42] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl blog, 18))99 2019.
[43] Diederík P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[44] Joseph Redmon and Ali Farhadi. Yolo9000: better, faster, stronger. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7263-7271, 2017.
13


Reviews:
[{"review_id": "u_ZKxOCqXmN", "reviewer": "Reviewer_s9bs", "summary": "(Summary)\nThis paper proposes a generative model that can learn hierarchical topics that coherently incorporate a structured prior knowledge graph. By introducing Gaussian embeddings to an existing SawETM, a neural topic model where topics in different levels can relate each other, the authors first propose the Gaussian SawETM. Then their TopicNet based on Gaussian SawETM can learn hierarchical topics coherently to an input knowledge-graph. TopicNet is not only capable of learning hypernym-induced topics as hierarchies, but also equipped with rich interpretability.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "(Originality) \nInstead of using Gaussian variables as latent codes like other neural topic models based on VAE, the authors combine an idea of using gamma-distributed latent variables from Weibull hybrid inference (by Zhang et al) and SawETM (by Duan et al) for dependencies in two consecutive layers. Therefore the real novelty comes more from an idea of defining topic template as a topic tree and guiding the topics of each layer learned from the Gaussian SawETM by the corresponding semantic layer in the topic tree through imposing constraints on the topic embeddings. Due to such joining nature, one could equally argue that the contribution in modeling is mostly incremental.\n\n\n(Clarity)\nThe writing is generally clear, but readers might need to look at previous papers to understand the details of Weibull method and SawETM. Though the previous work doesn’t have to be completely rephrased, providing some degree of background will benefit future readers.\n\n\n(Significance)\nThe contribution of the overall paper seems to be more on applicational side as the proposed models rely on combining two successful neural networks. However, the experiments are limited to using either only small vocabulary datasets or too large vocabularies comparing to the number of documents, possibly weakening statistical robustness. Without demonstrating high performance on more realistic datasets (i.e., the number of topics << the number of words << the number of documents), it may be hard to convince potential users.\n\n\n(Major Comments and Questions)\n1) How is the vocabulary curated? Users in topic modeling typically put non-negligible amounts of efforts in vocabulary curation for properly removing both too frequent and too rare terms, which affects final topic quality. It is unclear how the authors curate the vocabulary for their experiments.\n\n2) As you learn a generative model for unsupervised learning, having a semi-synthetic experiment will be useful for verifying true performance. Note that it is a better practice to first learn realistic topic distributions from real text corpora, later sampling various sizes of documents by using the known topic distributions and randomly sampled multinomial topic proportions. Given ground-truth topic distributions and proportions, you will be able to firmly complement the clustering study.\n\n3) Performance of the most standard LDA model could be changed by a big margin based on the choice of hyperparameters as well as inference method such as Collapsed Gibbs Sampling (CGS) or Collapsed Variational Inference (CVI). How would the baseline LDA perform if using learned hyperparameters with CGS? Does the proposed model still significantly outperform LDA when the LDA is optimized as we use as a part of standard protocol?\n\n4) In Section 4.1, the authors said that “SawETM cannot capture the uncertainties of topics”. What is exactly the uncertainty and how is it mathematically defined? Do you have any real example where SawETM fails to capture these uncertainties, whereas Gaussian SawETm captures successfully? Providing at least one qualitative example will benefit the readers even if it is anecdotal.\n\n5) Being explainable and having topic interpretability are not identical in general category. In many cases, Explainable AI (XAI) tries to understand reasoning or inference path where users cannot transparently see how the forecast comes out. Is the topic discovery explainable in an agreeable sense?\n\n6) Topic diversity is named more frequently as inter-topic similarity or dissimilarity. Read some papers by David Mimno will be helpful. In addition, topic specificity (how far from the unigram corpus distribution) can often help quality measurement jointly with those two metrics.\n\n7) WordNet’s hierarchy semantic is mostly limited only to nouns. What happen if some topics are more about actions than entities, thereby being described more by verbs than nouns in their top words?\n\n\n(Minor Comments)\n1) Line 252, from  form\n\n\n(After Author-Feedback)\nHave updated my previous evaluation.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No problem has been found.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3 hours"}, "extra_text": {}}, {"review_id": "Xoi2C0hm8VP", "reviewer": "Reviewer_93MZ", "summary": "The paper builds on the recently proposed SawETM hierarchical topic model and combines it with a knowledge graph so that each node in the knowledge graph corresponds to a topic. They change it so that every topic is embedded with a Gaussian distribution. This enables to enforce a hierarchical ordering constraint on the topics. Experiments show that the model discovers deeper, more interpretable topics and finds better document representations.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper presents an interesting idea of incorporating knowledge graphs with hierarchical topic models. My criticism is limited to explanation and inconsistency in experiments. The article proposes two novel methods for Hierarchical Topic Modeling. Gaussian SawETM - A probabilistic generative model to capture semantic uncertainties of words and topics. TopicNet - A knowledge-based hierarchical topic model which incorporates prior knowledge (belief) of the real world. TopicNet employs Gaussian SawETM and learns the hierarchies based on prior beliefs such as knowledge graphs. The idea is theoretically sound and has a mathematical\nexplanation. The structure and quality of explanation can be improved in some\nsections. More consistent experiments and comparisons would have validated\nwhat the authors claim, the absence of which can leave a reader curious. \n\nThe quality of the explanation is inconsistent throughout the article. While the\nIntroduction section is not properly structured, it does a good job of explaining\nthe content it provides with the help of diagrams. Whereas, Section 2 and 3,\nwhich form the proposed model, lack quality of explanation.\nIt has the following explanation problems:\n\n- No proper figure is given for the proposed TopicNet model.\n- The figure provided for Gaussian SawETM is a general Gamma Belief Net-\nwork, it has no representation of upward and downward encoder networks\nas described in Section 2.3.\n- The authors claim to have reduced mode collapse by their use of Gaussian\nSawETM, but provide inconsistent results in Table 1 and Table 2. In perplexity representation depth till 15 are accounted for but in classification\ndepth of only 1 and 4 are accounted for. The depth inconsistency further\ncontinues in Figure 4, where the embedding visualization is done for Gaussian SawETM with a depth of 3. Depth should be made consistent while\ncomparison in all tables as well as figures, since claims based on different\nlayers, coincide with results of different layers.\n- While presenting the Topic Quality, the authors themselves hint at the\nreason behind the exceptional performance of TopicNet on severe infor-\nmation reduction from data in the deep layers. This specific case will lead\nto more diversity in words, hence Topic Quality can be a slightly skewed\nmetric here for the judgement. Results of Topic Coherence based on NPMI\ncan also be presented to clear the doubt.\n- Only selected hierarchies are presented. Human evaluation measures such\nas word intrusions can be utilized to evaluate the hierarchies.\n\n\nTwo further questions I have for the authors are:\n\n- How can we be sure that the knowledge graph has anything to do with the dataset that we use? What if additional topics are present or the topics in the knowledge graph are more detailed than the dataset calls for? \n- How are the semantic concepts themselves injected into the topic model? I only see how the relationships between the concepts are injected. But how do you make sure that the topic tree node \"animal\" actually corresponds to a topic with words related to animals?\n\n\nMinor comments:\n- Related work section and introduction are mixed together, I would suggest to separate them into two separate parts.\n- The heading of Section 2 should maybe better be changed to \"Gaussian SawETM\"\n- In some cases such as [7], the arXiv version of a paper is cited even though it is published in a peer-reviewed conference.\n- The references are very inconsistent, e.g. compare [25] and [26]\n- The example in Fig. 1a) could be read as being sexist, as mother, son and home are in one topic, but father is not, so only mothers walk their son to school? \n- the example in Fig. 1a) seems not very meaningful, for example the topic \"road read bike walk\", makes not much sense, as read has nothing to do with road.\n- line 128 BOund-> bound\n- Figure. 5 -> Fig. 5\n- Figure. 1 -> Fig. 1\n- tpoic -> topic\n- 7-ayer -> 7-layer\n- negetive -> negative\n\n\n\n*************************\nmy comments were sufficiently addressed and I am increasing my rating", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "One main limitation from my point of view is the fact that we do not know if the knowledge graph is appropriate for the dataset, as I wrote in the review:\nHow can we be sure that the knowledge graph has anything to do with the dataset that we use? What if additional topics are present or the topics in the knowledge graph are more detailed than the dataset calls for? \n\nSo the negative impact could be that topics in the dataset are not found because they are not present in the prior knowledge. Limitations are not really addressed in the paper.", "review_text": "", "extra_scores": {"needs_ethics_review": "Yes", "ethics_review_area": ["Discrimination / Bias / Fairness Concerns"], "time_spent_reviewing": "4"}, "extra_text": {"ethical_concerns": "As I wrote in the main review:\nThe example in Fig. 1a) could be read as being sexist, as mother, son and home are in one topic, but father is not. I would choose a different example or explain how this sort of bias can be prevented using the prior knowledge possibly."}}, {"review_id": "I7GWNpbypwr", "reviewer": "Reviewer_Thqc", "summary": "The authors propose a topic model that aim at embedding words and topics into a latent space via Gaussian embeddings. Additional constraints in the training procedure allows to incorporate prior knowledge on the topics, so that they are able to include semantic topic relations.\n\nThe model is evaluated on a large set of public datasets and on a number of tasks. It achieves better results with respect to baselines on clustering tasks, and comparable on classification tasks. However, the main advantage is the increase interpretability of topics and topic hierarchy.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is sometimes quite difficult to read. However, the authors made a clear effort to be exhaustive in explaining mathematical details and experiment settings.\n\nI am not sure I understand how it is possible to incorporate additional knowledge on the topic semantics. The authors mention that \"Since the Gaussian SawETM projects the topics of all layers to a shared embedding space, such a goal can be achieved by imposing constraints on the topic representations in the shared embedding space.\". However, as each topic is a collection of words, and the topic is embedded in a latent space, the semantic of the collection of words is lost. How the topic semantic (if the topic is automatically discovered by the procedure) can be mapped to the external knowledge graph between the entities?\n\nThe authors do not mention the scalability of the model. One main limitation of standard topic model is their scalability when increasing the vocabulary size, as for each topic all of the vocabulary is needed to be sampled to compute the probability distribution. From the proposed approach, it seems that the method aim at embedding topics into a latent space. It would be nice to assess the scalability in terms of words or topics.\n\nIn terms of experiments, the authors mention that they evaluated the model also in a classification task. However, it's not clear to me what the classification task is as the dataset they describe are textual documents. What labels have the document in the datasets? Each document has a \"category\" as a label?\n\nHighlighting the best results in Table 2 would make the table easier to read. It seems that in general the proposed method is better than the baselines on the various clustering tasks, and slightly worse for classification tasks. When adding prior knowledge, the method has actually worse performance. The authors mention that \"TopicNet does not perform better maybe because the prior information is irrelevant with the document classification and clustering tasks\". However, the additional information on topic relation should better guide the model, and I am a bit surprise this is happening. As I haven't fully understood how to include the prior information in the model I think this is an important point to explore better.\n\nMinor:\ntypo: negetive (page 6)", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "None", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5"}, "extra_text": {}}, {"review_id": "2WHDCvsDGgL", "reviewer": "Reviewer_aZKr", "summary": "This paper mainly proposes a new hierarchical topic discovery framework that is guided by prior (tree-structured) knowledge. The construction of the framework is broken down into two steps: Modeling semantic hierarchy with a Gaussian-embedding-based topic model (Gaussian SawETM), and injecting the prior semantic knowledge by regularizing via entailment relationship. The model is evaluated on a set of benchmarks with perplexity, topic quality, and clustering/classification protocols. Qualitative studies are also shown to demonstrate the interpretability of the model.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "[Originality]\n\nOverall, the novelty is at the borderline of a NeurIPS paper. In Sec. 2, the Gaussian SawETM model is essentially a combination of SawETM and Gaussian embeddings, so the ideas are not new there. In Sec. 3, the injection of prior knowledge is implemented as a set of regularization over the entailment relationship in the semantic graph (like WordNet), and the modeling of the transitivity and antisymmetry is commonly seen in hierarchy modeling studies (the line of hyperbolic embeddings), thus the design is reasonable but not particularly providing new insights. However, I like the overall motivation of this paper to guide topic discovery with prior (graph-structured) knowledge, which is not commonly seen in the topic modeling literature.\n\n[Quality]\n\nThe Gaussian SawETM model borrows the idea from a recent topic modeling paper SawETM to alleviate the model collapse issue for deep hierarchical topic models, and further leverages Gaussian embeddings to model semantic uncertainties of words. The model inference follows standard VAE-based neural topic models for optimizing the ELBO. The TopicNet framework, built upon the Gaussian SawETM model, incorporates prior tree-structured knowledge by regularizing the Gaussian embeddings to align with the partial order of semantic concepts as a new objective added to the ELBO objective. The overall design of the model is principled and neat. \n\nThe empirical evaluation is conducted on a set of benchmarks with metrics commonly used in topic modeling evaluation (perplexity, topic quality, and clustering/classification). The proposed models (Gaussian SawETM & TopicNet) demonstrate comparable or better performance than existing neural topic models. So the evaluation also seems convincing to me.\n\n[Clarity]\n\nThe paper is overall clearly written and well-organized. The method sections clearly draw connections with related studies and motivate each component. The experiment section clearly states the setup/evaluation protocol and presents detailed analyses.\n\n[Significance]\n\nThe significance of this paper is probably at an okay level--although the proposed models outperform baselines in some tasks (clustering/classification), they are not better under perplexity evaluation. Also, the guided version (TopicNet) underperforms the unsupervised version (Gaussian SawETM) for clustering/classification, which is inconsistent with the major motivation of the paper. However, I appreciate the overall direction of incorporating prior semantic knowledge into topic discovery, and feel that the paper has made an interesting exploration overall.\n\n[Questions]\n\nI hope the authors could clarify the following points:\n* Is WordNet used as the semantic graph in all experiments/datasets? Could you use other prior structures that better fit the specific domains/datasets? It is mentioned in the document clustering/classification experiments that TopicNet does not perform better than Gaussian SawETM because the prior information is irrelevant with the document classification and clustering tasks. This observation largely weakens the motivation of this paper that some prior guidance can help the topic discovery process.\nI'm curious whether a manually curated topic hierarchy based on the dataset will help improve the clustering/classification performance--even that will make the comparisons with baselines unfair, but at least that will confirm the motivation and demonstrate the effectiveness of TopicNet on modeling the given semantic graph.\n* It seems to me that the current setup of TopicNet can only model tree structures. If this is true, calling the model \"Semantic Graph-Guided\" might be a bit inappropriate, as the tree structure is only a subset of graph structures.\n* It is mentioned in the perplexity evaluation that the DLDA group requires iteratively sampling to infer latent document representations in the test stage, despite getting better performance. I feel that some concrete run time reports are necessary to support this argument (are they really that costly compared to the DNTM group?).\n\n[Summary]\n\nPros:\n* The paper is clearly written and well-organized; the incorporation of prior semantic knowledge is well-motivated.\n* The overall framework is reasonable and principled.\n* The evaluation is comprehensive.\n\nCons:\n* The novelty is not that high, as many essential components are borrowed from previous work.\n* The empirical evidence that TopicNet underperforms Gaussian SawETM in applications raises concerns about the basic assumption/motivation of the paper.\n\n################\n\nUpdates after rebuttal:\nI would like to thank the authors for conducting new experiments and providing clarifications that addressed my concerns. I increased my rating from 5 to 6.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I feel that the authors have addressed the limitations in a reasonable way.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper develops TopicNet, a generative model that can learn hierarchical topics by incorporating a pre-defined semantic graph. The main contribution of the paper seems to be on the application side, and all reviewers agreed the paper is above the acceptance threshold.

I would like to encourage the authors to incorporate the reviewers' feedback in their revised version. In particular:
+ Incorporate the new results from the response to reviewer aZKr.
+ Add the metrics of inter-topic (dis)similarity and topic specificity.
+ Add an experiment to showcase that the method can benefit from prior knowledge in terms of both interpretability and performance on downstream tasks.
+ Add a more detailed figure to illustrate the Gaussian SawETM and TopicNet models.
+ Clarify the other points raised in the reviews.
Decision: Accept (Poster)