HyperMiner: Topic Taxonomy Mining with Hyperbolic Embedding
Authors: Yi.shi Xu
Dongsheng Wang
Bo Chen
Ruiying Lu
Zhibin Duan
Mingyuan Zhou
Venue: NeurIPS 2022 Accept (2022)
Keywords: hierarchical topic modeling
hyperbolic embedding
knowledge incorporation

Abstract: Embedded topic models are able to learn interpretable topics even with large and heavy-tailed vocabularies. However, they generally hold the Euclidean embedding space assumption, leading to a basic limitation in capturing hierarchical relations. To this end, we present a novel framework that introduces hyperbolic embeddings to represent words and topics. With the tree-likeness property of hyperbolic space, the underlying semantic hierarchy among words and topics can be better exploited to mine more interpretable topics. Furthermore, due to the superiority of hyperbolic geometry in representing hierarchical data, tree-structure knowledge can also be naturally injected to guide the learning of a topic hierarchy. Therefore, we further develop a regularization term based on the idea of contrastive learning to inject prior structural knowledge efficiently. Experiments on both topic taxonomy discovery and document representation demonstrate that the proposed framework achieves improved performance against existing embedded topic models.

HyperMiner: Topic Taxonomy Mining with Hyperbolic Embedding
Yishi Xu, Dongsheng Wang, Bo Chen', Ruiying Lu, Zhibin Duan National Laboratory of Radar Signal Processing, Xidian University, Xi'an, China xuyishi@stu.. xidian. edu. en, behen@mail xxiiian edu.cc
Mingyuan Zhou
McCombs School of Business, The University of Texas at Austin, USA mingyuan. www..ccccoooo....oooooooooooooo. wtw.aa.eee........ edu
Abstract
Embedded topic models are able to learn interpretable topics even with large and heavy-tailed vocabularies. However, they generally hold the Euclidean embedding space assumption, leading to a basic limitation in capturing hicrarchical relations. To this end, present novel framework that introduces hyperbolic embeddings to represent words and topics. With the tree-likeness property of hyperbolic space, the underlying semantic hierarchy among words and topics can be better exploited to mine more interpretable topics. Furthermore, due to the superiority of hyperbolic geometry in representing hierarchical data, tree-structure knowledge naturally injected to guide the learning of a topic hierarchy. Therefore, we further develop regularization term based the idea of contrastive learning to inject prior structural knowledge efficiently. Experiments on both topic taxonomy discovery and document representation demonstrate that the proposed framework achieves improved performance against existing embedded topic models.
also be
Introduction
With a long track record of success in variety of applications [1-6], topic models have emerged as one of the most powerful tools for automatic text analysis. Typpcally, given a collection of documents, topic model aims identify group of salient topics by capturing common word co-occurrence patterns. Despite their popularity, traditional topic models such as Latent Dirichlet Allocation (LDA) [7] and its variants [8-12] are plagued by complicated posterior inference, presenting challenge to create deeper and more expressive models of text. Fortunately, recent developments of Variational AutoEncoders (VAEs) and Autoencoding Variational Inference (AVI) [13, 14] have shed light this problem, resulting in the proposal of a series of Neural Topic Models (NTMs) [15-18]. With better flexibility and scalability, NTMs have gained increasing research interest over the past few years.
Parallel to neural topic modeling, the idea of bringing word embeddings [19, 20] into topic models has also attracted much attention. Considering the large performance degradation to limited word co-occurrence information, some early works [21-23] exploit word embeddings as complementary metadata and incorporate them into the generative process of topic models. Recently, more flexible ways [24, 25] of combining word embeddings have been explored thanks to the development of NTMs. For example, Bianchi et al. [26] word embeddings directly as part of the encoder's input. In particular, a novel one called Embedded Topic Model (ETM) [27] stands out for its performance as well as the elegant way integrates word embeddings. Specifically, by
short texts due
Corresponding author
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

JALU
Bost
C exisjonce
com Sapaon
ccom
aoftware
Subjoct Times
Chrlobion
people' god goujeuuehhuehh
legal lure peliove
fuct
don
monitor mac
show windows
ROW
pcol drive
new
dirive
uSR
play hockey
dows
j.am
like
2
ion
States
people
aame Toad qun
rights
church christian
Figure I: Visualization of 21D Fuclidean emhedding space (left) and 2D hyperbolic embedding space (right) learned by ETM. Red points denote topic embeddings, hlue points represent word embeddings.
representing topics as points in the word embedding space. ETM assigns probabilities to words based on their (inner product) distances from each topic embedding. As result, semantically related words tend to fall around the same topic, thus facilitating the discovery of more interpretable topics.
Under the inspiration of ETM. Duan et al. [28] have extended a similar idea to hierarchical topic modeling and proposed SawETM. In addition to mapping words and hierarchical topics into a shared embedding space, SawETM has also developed unique Sawtooth Connection module capture the dependencies between the topics at different layers. which, on the other side, empowers it to support deep network structure. While achieving promising results, both ETM and SawETM hold the Euclidean embedding space assumption, leading to fundamental limitation that their ability to model complex patterns (akin to social networks, knowledge graphs, and taxonomies) inherently bounded by the dimensionality of the embedding space [0]. As a consequence, the underlying semantic hierarchy among the words and topics can hardly be expressed adequately in a relatively low-dimensional embedding space, as illustrated on the left side of Figure 1.
Apart from the difficulty in capturing the implicit semantic hierarchy, another concomitant problem is the dilemma of incorporating explicit structural knowledge. Assuming we have a prior taxonomy of concepts and wish to use it to guide the learning of hierarchical topics, is challenging to preserve the structure between concepts in Euclidean space by constraining the word and topic embeddings. embeddings as substitute for the vector embeddings to represent words and topics. As such, the prior knowledge of hypernym relations between concepts could be naturally injected via the encapsulation of probability densities. However, maintaining the semantic hierarchy in such an embedding space still suffers from certain degree of distortion, as poses challenge to the optimization of KL. divergence between distributions. Furthermore, the introduction of Graussin-ddisribnn limiting its potential scalability to large vocabularies and high-dimensional embedding spaces.
To cope with this issue, TopicNet [31] employs the
embeddings entails great demand on memory,
To overcome the above shortcomings brought by Euclidean embedding space, we propose to compute embeddings in hyperbolic space. Distinguished by the rree-likeness properties [32, 33], hyperbolic space has been consistently shown to be superior in modeling hierarchical data compared to Euclidean space [34-37]. By measuring the distance between words and topics in hyperbolic embedding space, the model is encouraged to better capture the underlying semantic hierarchy among words. As shown on the right side of Figure 1, some general words such as "new" and "does" fall around the center, they stay close to all other points because they often co-occur with other words. While more specific words like "moon" and "nasa" fall near the boundary and are only close to the nearby points. Moreover, hyperbolic space also provides better platform to inject prior structural knowledge, since hierarchical relations can be effectively preserved by imposing constraints on the distance between word and topic embeddings. In a nutshell, the main contributions of this paper are as follows:
We propose to compute the distance between topics and words in hyperbolic embedding space on top of existing embedded topic models, which is beneficial to both the mining of implicit semantic hierarchy and the incorporation of explicit structural knowledge.
We design a node-level graph representation learning scheme that can inject prior structural knowledge to effectively guide the learning of a meaningful topic taxonomy.

---

Extensive experiments on topic quality and document representation demonstrate that the proposed approach achieves competitive performance against baseline methods.
2 Background
2.1 Embedded topic model
ETM [27] neural topic model that builds on two main techniques: LDA [7] and word embeddings [19, 20J. To marry the probabilistic topic modeling of LDA with the contextual information brought by word embeddings, ETM maintains vector representations of both words and topics and uses them to derive the per-topic distribution over the vocabulary. Specifically, consider a corpus with V distinct terms comprising the vocabulary,
denote the word embedding matrix p R RPXV, where D is the dimensionality of the embedding space. For each topic, there is also an embedding representation ag RP, then ETM defines the per-topic distribution BR E RV over the vocabulary
Bx Softmax (pTas))
(1)
With the above definition, ETM specifies generative process analogous to LDA. Let wijn E {1,.., V} denote the th word in the jth document, the generative process is as follows.
1. Draw topic proportions 0j ~ LN (),
2. For each word n in the document:
(a) Draw topic assignment Zjn ~ Cat (0z).
(b) Draw word Wyn Cat (B.....
Where (-(()) in step 1 denotes the Dogistic-normal distribution [38], which transforms a standard Gaussian random variable to the simplex. By taking the inner product of the word embedding matrix P and the topic embedding Og derive fg, the intuition behind ETM that semantically related words will be assigned to similar topics. With this property, ETM has been demonstrated to improve the quality of the learned topics, especially in the presence of large vocabularies. Like most NTMs, ETM is fitted via efficient amortized variational inference algorithm.
2.2 Hyperbolic geometry
In this part, briefly review some key concepts on hyperbolic geometry. A comprehensive and in-depth description can be found in Lee [39] and Nickel and Kiela [40]. Under the mathematical framework of Riemannian geometry, hyperbolic geometry specializes in the case of constant negative curvatures. Intuitively, the hyperbolic space can be understood as a continuous version of trees: the volume of ball expands exponentially with its radius, just how the number of nodes in binary tree grows exponentially with its depth. Mathematically. there exist multiple equivalent models for hyperbolic space with different definitions and metries. Here, we consider two representative ones in light of optimization simplicity and stability: Poincaré ball model [29] and the Lorentz model [40].
Poincaré ball model The Poincaré ball model of an n-diimensional hyperbolic space with curvature CC < 0) is defined by the Riemannian manifold pd (B",
(B"h.p)) spp). where 8 = (x E R² ||>||| < 1/VICI} is the open re-dimensional ball with radius 1/ I/V and gp is the metric tensor that can be converted from the Euclidean metric tensor Ge I as
2
2
9p (x)
ge
1 ||
Lorentz model The Lorentz model (also named Hyperboloid model) of an n-dimensional hyper- bolic space with curvature C' (C < 0) is defined by the Riemannian manifold _ = (H").r)) where H. = {x E R=+1 (xx, x) c = 1/C} and gi diag (1-1, (r-)) denote the Lorentzian inner product. Let y E R=+1, the Lorentz inner product induced by is calculated as
(x. v)
9:y TT
x
Tollo +
intuitive illustration of the equivalence between the Poincaré ball model and the Lorentz model and some other related operations in hyperbolic space will be introduced in Appendix C.
3

---

3 Topic Taxonomy Mining with Hyperbolic Embedding
In this section, we elaborate on how the introduced hyperbolic embeddings facilitate the mining of implicit semantic hierarchy and the incorporation of explicit tree-structure knowledge, both of which encourage the model find more interpretable topics'.
3.1 Hierarchical topic modeling in hy hyperbolic space
Theoretically, the idea of representing words and topics in hyperbolic space is orthogonal to a wide range of topic models employing the word embeddings technique. To provide the foundation for the subsequent injection of structural knowledge, we here apply our method to a hierarchical embedded topic model SawETM [28]. SawETM utilizes the adapted Poisson gamma belief network (PGBN) [41] as its generative module (decoder) and decomposes the topic matrices into the inner product of topic embeddings at adjacent layers. The novelty of our method lies in that the hierarchical relations can be better reflected by the distances between embeddings in hyperbolic space. Mathematically, the generative model with L latent layers formulated
L) Gam (7,e)ltt)) 000
Gam ( (441)) ( .efttt)
0 (1) Gam x 0000 (32)) .. Pois
(4)
+ Softmax (s (at (t-1, .aa))
The above formula clearly describes how the multi- layer document representation is generated via top-down process. Specifically, the latent representation of the top layer 0 03/() is sampled from fixed gamma prior distribution, then each intermediate layer / the latent units 0 .R are factorized into the product of the factor loading matrix ((+1) E K and latent units 0f 0f(+1) R of the above layer. Until the bottom layer, the observation of word count vector I, E ZV is modeled as the Poisson distribution. Note that the subscript denotes the document index and some other variables Y, e, (1+1) ,. .e)
are hyperparameters. Especially, the factor loading matrix (()) of layer / is derived based on the distance between the topic embeddings two adjacent layers, i.c.. (!-1) - and (!) RK Note that ( V represents the word embeddings. Since all embeddings are projected into the hyperbolic space to fully explore the underlying semantic hierarchy among the words and topies, we design our similarity score function
vici arcosh (1
2C - y/
8 (xx -dp (x,, v)
(1+ (1+
(5)
S (x. 3) -dc(e,
arcosh (C(xx,))
VICI
Where dp(+,-) and .c(,,.) are the distance functions of the Poincaré ball model and the Lorentz model, respectively. As the two models hyperbolic space are mathematically equivalent, we take the Poincaré ball as an example for analysis. Eq. (5) shows that the distance changes smoothly with respect the norm of and y. This locality plays crucial role in learning continuous embeddings of hierarchies. For instance, the origin of 1 has zero norm, it would have relatively small distance to all other points, which exactly corresponds to the root node of tree. On the other hand, those points close to the boundary of the ball have norm close to one, so the distance between them grows quickly, which properly reflects the relationships between the leaf nodes of tree.
3.2 Knowledge guided topic taxonomy discovery
Hicrarchical structures are ubiquitous in knowledge representation and reasoning [40]. Particularly, mining a set of meaningful topics organized into a hierarchy from massive text corpora intuitively appealing, as it allows users to casily access the information of their interest. However, most existing hierarchical topic models struggle to realize this goal without any supervision, and some appropriate guidance with prior structural knowledge proves to be helpful for mitigating this issue [31, 42].
Our code is available https:/ /////ggttuuut http:::///gtuu..ooooooeeeeeeeeee wwtp://wwr..ettttteetttttltiiaaa

---

Positive peirs Negative pains
Tood
R
meet
drink
vegetables
beef pork milk yogurt lettuce broccoli
(a)
(b)
(c)
Figure 2: Overview of knowledge guided topic laxonomy discovery. (a) The prior concept laxonomy constructed from vocabulary: (b) Illustration the strategy for picking positive pairs and negative pairs the hyperbolic embedding space; (c) The hicrarchical generative model whose factor loading matrices are derived based on the hyperbolic distances between topic embeddings at two adjacent layers.
Form of prior knowledge We assume the prior knowledge takes the form of a concept taxonomy, which is compatible with the deep structure of the proposed hierarchical topic model in Section 3.1. In detail, the taxonomy exhibits a top-down reification process and concepts between two adjacent layers with connections follow the hypernym relation, as shown in Figure 2(a). Meanwhile, to keep the taxonomy consistent with the corresponding dataset, we construct it by traversing each word in the vocabulary find its ancestors along the hypernym paths provided by WordNet [43].
Hyperbolic contrastive loss Although the paradigm of contrastive learning has been successfully applied to graph representation learning in Euclidean space [44-47], those developed contrastive algorithms are not directly applicable in our case. On the one hand, we focus more on learning node representations while not destroying their prior hierarchical structures. On the other hand, hyperbolic space possesses distinctive properties (e.g., hierarchical awareness and spacious room) compared to its Euclidean counterpart. Consequently, to accommodate the two differences, design node-level hyperbolic contrastive loss such that the prior knowledge can be effectively injected as an inductive bias to influence the learning of topic taxonomy.
Specifically, we set the number topics at each layer of the hierarchical topic model to be the same as the number of concepts at each layer of the taxonomy, then each topic is assigned a corresponding concept as its semantic prior. Since concepts are connected in the taxonomy, such relations can be transfered accordingly between topics, which provide the basis for picking positive and negative pairs among the topic and word embeddings. Let T (ax()) denote the set of all embeddings, then
each embedding a RRP is associated with two groups of embeddings as the positive samples and the negative samples, respectively. Then the average hyperbolic contrastive loss is defined as (we omit the superscript for simplicity of notation)
exp (S(ar,);))/1)
LContra = EajeT -log
(6)
exp (S(a,, at)//)) Ea, EQ(a))))) exp (S(a,, a, a/T))
where ((,),) is the similarity score function defined in Eq. (5) and is the temperature parameter. Note that oz is positive sample drawn from P(aa;) and Q(a:; is the set of negative samples.
Sampling strategy Inspired by the homophily property (i.c., similar actors tend associate with each other) in many graph networks [29], we take one-hop neighbors of each anchor, i.e., its parent node and its child nodes as positive samples maintain the hierarchical semantic information. For the negative samples, we select m embeddings from the nin-firstoorder neighbors that have the highest similarity scores with the anchor embedding.
3.3 Training objective
As most existing NTMs can be viewed as the extensions of the framework of VAES [13, 14], they generally develop similar training objective to VAEs, which is to maximize the Evidence Lower BOund (ELBO). For our generative model, the ELBO of each document can be derived as
LELBO
Σ
DeL. (1(0 x (+1) (4t),00t2)) 0 + wcof") [ (0) (1,00)]
(7)
5

---

Algorithm 1 Knowledge-Guided Topic Taxonomy Mining
Input: mini-batch size B, number of layers T, adjacent matrix A built from concept taxonomy. Initialize the variational network parameters Q2 and the word and topic embeddings {a(") Nz while not converged do
1. Randomly draw batch of samples }=
2. Infer variational posteriors for the latent variables of different layers {0))) B,LL
)==1,
& 3. Derive factor loading matrices ( using (a(") based on Eq. (4);
4. Compute the ELBO on the joint marginal likelihood of {xr 1 based on Eq. (7);
5. Compute the hyperbolic contrastive loss using (a(")) -T= and A according to Eq. (6);
6. Update 52 and {a Ji-o using gradients Va S2, (a (M M=o: (z));=1)
end while
where the first term is the Kullback-Leeihl divergence that constrains the approximate posterior ((0[)) to be close to the prior p(O))) ). and the second term denotes the expected log-likelihood or recónstruction error. Considering that our generative model employs the gamma- distr--dtt latent variables, it brings the difficulty of reparameterizing gamma-distributed random variable when we design sampling-based inference network. Therefore, we instead utilize Weibull distribution to approximate the conditional posterior inspired by Zhang et al. [17]. as the analytic KI. expression and efficient reparameterization make easy to estimate the gradient of ELBO with respect to network parameters. The implementation details of our variational encoder is described in Appendix B.
Furthermore, to inject the prior knowledge guide the learning of topic laxonomy, we train the ELBO jointly with a regularization term specified by the proposed contrastive loss in Section 3.2
LELBO CC
where À is the hyper-parameer used to control the impact of the regularization term, whose detailed effect is investigated in Appendix D. We summarize our complete learning procedure Algorithm 1.
Table 1: Statistics of the datasets
Number of docs
Vocabulary size
Total number of words
Categories
20NG
18,846
8,581
1,446,490
20
TMN
32,597
13,368
592,973
7
WIKI
28,472
20,000
3,719,096
N/A
RCV2
804,414
7,282
60,209,009
N/A
Experiments
4.1 Experimental setup
Datasets We conduct our experiments four benchmark datasets with various sizes and document lengths, including 20Newsgroups (20NG) [48], Tag My News (TMN) [49], WikiText-103 (WIKT) [50], and Reuters Corpus Volume I/ (RCV2) [51]. The statistics of these datasets presented in Table 1. In particular, TMN is a short text corpus with an average of about 20 words per document; 20NG and TMN are the two corpora that are associated with document labels.
Baseline methods As baselines, we choose several exemplary ones from the state-of-the-art topic models, including: 1) LDA [7], one of the most widely used topic models; 2) ProdLDA [16], an
NTM which replaces the mixture model in LDA with product of experts; 3) ETM [27], an NTM that marries conventional topic models with word embeddings: 4) WHAI [17], a hierarchical NTM
which develops a deep Weibull variational encoder based on PGBN [41]: 5) SawETM [28], which proposes a Sawtooth Connection module to build the dependencies between topics at different layers; 6) TopicNet [31], a knowledge- hierarchical NTM that guides topic discovery through prior semantic graph. All baselines are implemented meticulously according to their official code.

---

Hundre
Teakcer
Hypore wi-00
2INGITC]
TMNTTG)
ROVATTC)
2:26
6:15
0:15
2:15
WHOOTD)
RCVSITD)
0.4
02
2
Figure 3: The performance comparison of different models on the topic quality. The top row shows the topic coherence score, i.e., NPMI, and the bottom row displays the topic diversity score. The horizontal axis represents the index of the layers and we set up 5 layers for all the hierarchical topic models. The result of TopicNet on TMN and WIKI is missing because of memoor overflow for large vocabulary size.
Evaluation metries We aim to evaluate our model's performance terms of both topic quality and document representation. For topic quality, we adopt topic coherence (TC) and topic diversity (TD) as performance metrics. Given reference corpus, TC measures the interpretability of each topic by computing the semantic coherence of the most significant words [52]. Precisely, we apply the widely used Normalized Pointwise Mutual Information (NPMI) [53] and compute it over the top 10 words of each topic, with the original document collections of each dataset serving as the reference corpus. Note that the value of NPMI ranges from -1 to 1, and higher values indicate better interpretability. TD, as the name suggests, measures how diverse the discovered topics are. Following Dieng et al. [27], we define TD to be the percentage of unique words in the top 25 words of all learned topics. TD close to O means redundant topics and TD close to I implies more varied topics.
On the other hand, since per-document topic proportions can be viewed as unsupervised document representations, we intend to evaluate the quality of such representations by performing document clustering tasks. We report the purity and Normalized Mutual Information (NMI) [54] on two datasets providing the document labels, i.e., 20NG and TMN. Concretely, with the default training/test split of each dataset, we first train a topic model on the training set, and then the trained model is used to extract features O of all test documents. Subsequently, we apply the KMeans algorithm on 0 and calculate the purity and NMI of the KMeans clusters (denoted by km-Purity and Nm-NMI), Note that both of the two metries range from 0 to 1, and higher scores indicate better performance. For the hierarchical topic models, we take latent units (1) of the first layer as the document feature.
4.2 Experimental results
Topic quality Considering that not all discovered topics are interpretable [55], we select the top 50% topics with the highest NPMI values and report the average score over those selected topics to evaluate the topic quality comprehensively. Figure 3 exhibits the performance comparison results of different models. Note that HyperMinnr is the variant of SawETM that replaces the inner product between Fuclidean embeddings with the distance between hyperbolic embeddings, corresponding to the model proposed in Section 3.1. While HyperMiner- is an advanced version of HyperMiner that guides the learning of topic taxonomy by external structural knowledge, as introduced in Section 3.2. From what is shown in Figure 3. HyperMiner achieves consistent performance gains on all datasets compared to SawETM, in regard of both TC and TD, which demonstrates the superiority of hyperbolic geometry in uncovering the latent hierarchies among topies and words. In addition, as knowledge- guided topic models, both TopicNet and HyperMiner-KC KG get better performance than those without any supervision, indicating the positive role of prior knowledge helping to mine more interpretable and diverse topics. However, HyperMiner-KC KG still performs slightly better than TopicNet while consuming less memory. We attribute this result to our naturally-designed framework of injecting the tree-structure knowledge in contrastive manner.
7

---

Table km-Purity and km-NMI for document clustering. The best and second best scores of each dataset are highlighted in boldface and with an underline, respectively. The embedding dimension for embedded topic models is set 50.
20NG
TMN
Method
km-Purity
km-NMI
km-Purity
km-NMI
LDA [7]
38.43 = 0.52
35.98 1 0.39
48.17 1 0.69
30.96 + 0.78
ProdLDA [16]
39.21 = 0.63
36.52 + 0.51
55.28 + 0.67
35.57 = 0.72
ETM [27]
42.68 0.71
37.72 1 0.64
59.35 1 0.74
38.75 1 0.86
WHAT [17]
40.89 0.35
38.90 + 0.27
58.06 + 0.45
37.34 0.48
SawETM [28]
43.36 = 0.48
41.59 + 0.62
61.13 + 0.56
40.78 + 0.63
TopieNet [31]
42.94 0.41
40.76 + 0.53
60.52 + 0.50
40.09 0.54
HyperETM
43.63 = 0.51
39.06 + 0.64
61.22 + 0.62
40,52 + 0.71
HyperMiner
44.37 = 0.38
42.83 1 0.45
62.96 1 0.48
41.93 0.52
HyperMiner-KG
45.16 = 0.35
43.65 + 0.39
63.84 + 0.43
42.81 + 0.47
Table 3: Accuracy for document classification on 20NG, with different embedding dimensions for embedded topic models.
D 2
D 5
D
50
ETM [27]
19.87 + 0.81
33.64 +: 0.69
39.06 + 0.54
42.13 + 0.47
43.85 + 0.42
HyperETM
24.33 + 0.76
36.57 0.65
40.92 + 0.56
43.04 + 0.43
44.38 0.40
SawETM [28]
16.74 + 0.78
27.05 + 0.66
31.68 + 0.51
34.06 + 0.42
35.42 + 0.37
HyperMiner
20.16 1 0.80
29.73 0.63
33.04 1 0.49
34.98 1 0.41
36.01 0.36
TopicNet [31]
20.29 + 0.58
31,26 0.51
34.57 0.45
36.84 + 0.39
38.02 0.36
HyperMiner-KG
22.83 1
33.15 0.50
36.28 1 0.43
38.11 0.40
39.46 0.34
Document representation Table 2 shows the clustering performance of different models. We run all the models in comparison five times with different random seeds and report the mean and standard deviation. From the results presented above, have the following remarks: For all the evaluation metrics, our proposed improved variants perform consistently better than their prototypical models (refer HyperETM versus ETM, and HyperMinnr versus SawETM), which demonstrates that the introduced hyperbolic embeddings are beneficial to both the discovery of high-quality topics and the learning of good document representations. ii) As a nooweedee-guided topic model, HyperMiner-Ke KG achieves a significant improvement over the base model SawETM, while TopicNet suffers slight performance degradation compared to SawETM, which also serves as base model the original paper. This observation shows that with the hyperbolic contrastive loss, our model not only injects the knowledge successfully into the learning of hierarchical topics, but also achieves a better balance among the comprehensive metrics of topic modeling. iil) The superior performance of our model on TMN also suggests its potential for short text topic modeling.
To further investigate the effectiveness of our method under different dimensional settings, we proceed compare the extrinsic predictive performance of document representations through classification tasks. Consistent with the practice in clustering tasks, we first collect the features of training set 0,r and test set A.c separately, which are inferred by a well-trained topic model. Then we train an SVM classifier using 0ur and their corresponding labels. Finally, we use the trained classifier to predict the labels of θtc and compute the accuracy. Table illustrates the classification results of different embedded topic models. From the table we can that the improved variants with our method surpass their base counterparts various dimensionality settings. Especially, the performance gap between them has been further widened in the low-dimensional embedding space, confirming the natural advantage of hyperbolic distance metric learning useful document representations.
Visualization of embedding space and topics As our proposed HyperMiner-Ke imposes a prior hierarchical constraint (i.c.. concept taxonomy) on the embedding space, the topic embeddings and

---

groupnndt
fine
attribuse,
peychologsa Setturg
me
altradioon
bwy, mnterrt
piraada
ebject...
cou dgentadd1
selturee
ogent
(b)
Figure 4: Visualization of 2D hyperbolic embedding space learned by HyperMiner-KG. (a) concept hicrarchy: topic embeddings and their corresponding prior semantic concepts, where different coloured points represent topic embeddings at different layers. (b) lexical hierarchy: word embeddings and their corresponding meanings.
fitribute.n.02
group.n.01
objjectin.01
idam chain chs
trail stress
predition st
health
harardhy
state.n.02
socill group. 01
land.n.04
shapa.n.02
whole.n.02
pekygon
staft trai
problem
health
consenuus
cids
electron
freedom.n.01
society.n001
feeling.n.01
round shapa.n.01
living thing
landmass.n.01
artifact.n.01
candida bacieria bull Vis
Dbelly
rilliration
teelings unmender
auailitble dic
anta africs con continent
indezend
world
male
rond
odstence
servic
socal diristiee
digitty
tavour
files
hill
penol
liberty
coding system
emotion
civi- lization
religious
organism
disk
district
Insitta mentality
continent
liquid
linence
windias
anger paric
ruhure dilloaiinn
prior enriciin relgicus
candids
team game
curope aca
ell
Lbeety
softwane
borteraa
radar
statts
frecware
world
hookay
battory
afric
bike
escape
Frux
histery
players
contioon
Figure 5: Illustration of the topic taxonomy learned by 5-layer HyperMiner-KG 20Newsgroups, we show the contents of some example topics, which are learned under the guidance prior semantic concepts, Note that the example topics are selected from the bottom layers, with different colored boxes to distinguish them.
word embeddings are learned to maintain this structure as much as possible. Therefore, to verify the effectiveness of our proposed regularization term for injecting structural knowledge, we visualize the two-dimensional hyperbolic embedding space learned by HyperMiner-Ke Figure 4(a) exhibits the learned topic embeddings and the concept hierarchy used to guide them, in which we can see that the distribution of topic embeddings well preserves semantic structure of prior knowledge. Specifically, for topics guided by higher-level and more general concepts (e.g., physical pttysiclleenttttyn____lllal entity.n.Ol), their embeddings tend to locate in the center of the disc. While for those led by slightly more certain concepts (e.g., substance.n.077 their embeddings prefer to scatter around the boundary. Figure 4(b) presents the distribution of learned word embeddings, which also reflects the underlying lexical hierarchy of the corpus. The words that co-occur more frequently with different terms (e.g., organization, subject) tend to fall around the center of the disc, so as to maintain a small distance from arbitrary words. In contrast, those words with precise meanings fall near the edge area with spacious room and keep small distance only from words with similar semantics.
as displayed in Figure
Furthermore, to qualitatively demonstrate the crucial role of prior structural knowledge in helping to discover more interpretable topics, we show the contents (i.e., top words) of some learned topics by HyperMiner- KG, illustrated in Figure 5. From it observe that in majority of cases the

---

prior concepts can successfully guide the topics to learn semantically consistent contents (c.g., the topics guided by aodinng_yster and instrumentality). Moreover, the contents of topics at different layers are also semantically related due to the concepts guiding them. For instance, the content of the topic guided by whole.n. covers the contents of topics led by fring_thing.n011 and artifact.n. 01, respectively. Another interesting phenomenon is that the topic led by round_shaped only words related to shapes, but also some other words such as files and fip. The reason could be one of its child concepts disk often co-occurs with those words in the given corpus, suggesting the topic learning is co-influenced by both data likelihood and knowledge regularization.
involves not
5 Related Work
Historically, many attempts have been made to develop hierarchical topic models. Expanding their flat counterparts, hierarchical topic models aim to generate an intuitive topic hierarchy by capturing the correlations among topics. Due to the inflexibility of requiring accurate posterior inference, early works [8, 9, 12, 41] primarily focused on learning potential topic hierarchies purely from data, with some additional probabilistic assumptions being imposed. Also, there is a small body of work that tries to integrate domain knowledge into the process of discovering topic hierarchies. For example, anchored CorEx [56] takes user-provided seed words and learns informative topics by maximizing total correlation while preserving anchor words related information. More recently, JoSH [57] adopts more effective strategy that takes a category hierarchy as the guidance and models category-word semantic correlation via joint spherical text and tree embedding. Different from anchored CorEx and JoSH, which deviate from the conventional topic modeling framework, our approach still follows the regular probabilistic generative process. In addition, we use concept taxonomy to guide the topic learning so that more fine-graid grained topics can be mined. this regard, our proposed HyperMiner-KGG is much more related to TopicNet [31], yet it is more efficient with smaller storage footprint.
Conclusion
This paper presents a novel framework that introduces hyperbolic embeddings to represent words and topics top of existing embedded topic models. By using the hyperbolic distance to measure the semantic similarity between words and topics, the model can better explore the underlying semantic hierarchy to find more interpretable topics. Besides, a hyperbolic contrastive loss has been further proposed, which effectively injects prior structural knowledge into hierarchical topic models to guide learning meaningful topic taxonomy. Our method shows appealing properties that overcome several shortcomings of existing embedded topic models. Extensive experiments have been carried out, demonstrating that our method achieves consistent performance improvements in discovering high-quality topics and deriving useful document representations.
Acknowledgments and Disclosure of Funding
This work was supported in part by the National Natural Science Foundation of China under Grant U21B2006; in part by Shaanxi Youth Innovation Team Project; in part by the 111 Project under Grant B1 B18039; part by the Fundamental Research Funds for the Central Universities QTZX22160.
References
[1] Xuerui Wang, Andrew McCallum, and Xing Wei. Topical n-grams: Phrase and topic discovery, with an application to information retrieval. In Seventh IEEE international conference on data mining (ICDM 2007), pages 697-702. IEER, 2007.
[2] David Mimno, Hanna Wallach, Jason Naradowsky, David Smith, and Andrew McCallum. Polylingual topic models. In Proceedings of the 2009 conference on empirical methods in natural language processing, pages 880-889 889, 2009.
[3] Timothy Rubin, America Chambers, Padhraic Smyth, and Mark Steyvers. Statistical topic models for multi-label document classification. Machine learning, 88(1):157-208, 2012.
10

---

[4] Shuhui Jiang, Xueming Qian, Jialie Shen, Yun Fu, and Tao Mei. Author topic model- based collaborative filtering for personalized POI recommendations. IEEE transactions on multimedia, 17(6):977-91,, 2015.
[5] Wenlin Wang, Zhe Gan, Wenqi Wang, Dinghan Shen, Jiaji Huang, Wei Ping. Sanjeev Satheesh, and Lawrence Carin. Topic compositional neural language model. In International Conference on Artificial Intelligence and Statistics, pages 356-365. PMLR, 2018.
[6] Hamed Jelodar, Yongli Wang, Rita Orji, and Shucheng Huang. Deep sentiment classification and topic discovery novel coronavirus covid-19 online discussions: NIp using Istm recurrent neural network approach. IEEE Journal of Biomedical and Health Informatics, 24 (10):2733-27422 2020.
[7] David Blei, Andrew Y Ng, and Michael Jordan. Latent Dirichlet allocation. Journal of machine Learning research, 3(Jan):993-102) 2003.
[8] Thomas Griffiths, Michael Jordan, Joshua Tenenbaum, and David Blei. Hierarchical topic models and the nested Chinese restaurant process. Advances in neural information processing systems, 16, 2003.
[9] David Blei and John Lafferty. Correlated topic models. Advances in neural information processing systems, 18:147, 2006.
[10] Jon Mcauliffe and David Blei. Supervised topic models. Advances in neural information processing systems, 20, 2007.
[11] Mingyuan Zhou, Lauren Hannah, David Dunson, and Lawrence Carin. Beta-negative binomial process and Poisson factor analysis. In Artificial Intelligence and Statisticy, pages 1462-1471. PMLR, 2012.
[12] John Paisley, Chong Wang, David M Blei, and Michael Jordan. Nested hierarchical Dirichlet processes. IEEE transactions on pattern analysis and machine intelligence, 37(2):256227,, 2014.
[13] Diederik P Kingma and Max Welling. Auto-encoding variational Bayes. arXiv preprint 13Xiv:1312.61 14, 2013.
[14] Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference deep generative models. In International conference on machine learning, pages 1278-1286. PMLR, 2014.
[15] Yishu Miao, Lei Yu, and Phil Blunsom. Neural variational inference for text processing. In International conference machine learning, pages 1727-1736. PMLR, 2016.
[16] Akash Srivastava and Charles Sutton. Autoencoding variational inference for topic models. arXiv preprint arXiv: 1703.01488 2017.
[17] Hao Zhang, Bo Chen, Dandan Guo, and Mingyuan Zhou. Whai: Weibull hybrid autoencoding inference for deep topic modeling. arXiv preprint 1rX3vv1803.0122. 2018.
[18) Feng Nan, Ran Ding, Ramesh Nallapati, and Bing Xiang. Topic modeling with Wasserstein autoencoders. arXiv preprint arXiv: /2374, 2019.
[19] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3788 2013.
[20] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed repre- sentations of words and phrases and their compositionality. Advances in neural information processing systems, 26, 2013.
[21] James Petterson, Wray Buntine, Shravan Narayanamurthy, Tibério Caetano, and Alex Smola. Word features for latent Dirichlet allocation. Advances Neural Information Processing Systems, 23, 2010.
11

---

[22] Dat Quoe Nguyen, Richard Billingsley, Lan Du, and Mark Johnson. Improving topic models with latent feature word representations. Transactions of the Association for Computational Linguistics, 3:299-313, 2015.
[23] Chenliang Li, Haoran Wang. Zhiqian Zhang, Aixin Sun, and Zongyang Ma. Topic modeling for short texts with auxiliary word embeddings. In Proceedings of the 39th International ACM SIGIR conference Research and Development in Information Retrieval, pages 165-174, 2016.
[24] He Zhao, Lan Du, Wray Buntine, and Mingyuan Zhou. Inter and intra topic structure learning with word embeddings. In International Conference Machine Learning, pages 5892-5901. PMLR, 2018.
[25] Dongsheng Wang, Dandan Guo, He Zhao, Huangjie Zheng, Korawat Tanwisuth, Bo Chen, and Mingyuan Zhou. Representing mixtures of word embeddings with mixtures of topic embeddings. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022.
[26] Federico Bianchi, Silvia Terragni, and Dirk Hovy. Pre-training is a hot topic: Contextualized document embeddings improve topic coherence. arXiv preprint arXiv:2004.03974 2020.
[27] Adji B Dieng, Francisco JR Ruiz, and David M Blei. Topic modeling in embedding spaces. Transactions of the Association for Computational Linguistics, 8:439-453, 2020.
[28] Zhibin Duan, Dongsheng Wang. Bo Chen, Chaojie Wang. Wenchao Chen, Yewen Li, Jie Ren, and Mingyuan Zhou. Sawtooth factorial topic embeddings guided gamma belief network. In International Conference on Machine Learning, pages 2903-2913. PMLR, 2021.
[29] Maximillian Nickel and Douwe Kiela. Poincaré embeddings for learning hierarchical represen- tations. Advances in neural information processing systems, 30, 2017.
[30] Maximilian Nickel, Xueyan Jiang, and Volker Tresp. Reducing the rank in relational factor- ization models by including observable patterns. Advances in Neural Information Processing Systems, 27, 2014.
[31] Zhibin Duan, Yishi Xu, Bo Chen, Chaojie Wang, Mingyuan Zhou, et al. Topicnet: Semantic topic discovery. Advances in Neural Information Processing Systems, 34, 2021.
[32] Mikhael Gromov. Hyperbolic groups. In Essays group theory, pages 75-263. Springer, 1987.
[33] Matthias Hamann. On the tree-likeness of hyperbolic spaces. In Mathematical proceedings of the cambridge philosophical society, volume 164, pages 345-361. Cambridge University Press, 2018.
[34] Octavian Ganea, Gary Bécigneul, and Thomas Hofmann. Hyperbolic entailment cones for learning hierarchical embeddings. In International Conference Machine Learning, pages 1646-1655, PMLR, 2018.
[35] Frederic Sala, Chris De Sa, Albert Gu, and Christopher Ré. Representation tradeoffs for hyperbolic embeddings. In International conference on machine learning, pages 4460 4469. PMLR, 2018.
[36] Alexandru Tifrea, Gary Bécigneul, and Octavian-Eugen Ganea. Poincaré glove: Hyperbolic word embeddings. arXiv preprint arXiv: 06546, 2018.
[37] Hyunghoon Cho, Benjamin DeMeo, Jian Peng. and Bonnie Berger. Large-margin classification in hyperbolic space. In The 22nd international conference artificial intelligence and statistics, pages 1832-1840. PMLR, 2019.
[38] J Atchison and Sheng M Shen. Logistic-normal distributions: Some properties and uses. Biometrika, 67(2):261-277 1980.
[39] John M Lee. Smooth manifolds. In Introduction to Smooth Manifolds, pages 1-31. Springer, 2013.
12

---

[40] Maximillian Nickel and Douwe Kiela. Learning continuous hierarchies in the lorentz model of hyperbolic geometry. In International Conference on Machine Learning. pages 3779-3788. PMLR, 2018.
[41] Mingyuan Zhou, Yulai Cong, and Bo Chen. Augmentable gamma belief networks. The Journal of Machine Learning Research, 17(1):5656-500 2016.
[42] Dongsheng Wang, Yishi Xu, Miaoge Li, Zhibin Duan, Chaojie Wang, Bo Chen, and Mingyuan Zhou. Knowledge-aware Bayesian deep topic model. arXiv preprint 22X.::2009.12228, 2022.
[43] George A Miller. Wordnet: lexical database for english. Communications of the ACM, 38 (11):39-41, 1995.
[44] Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. Graph contrastive learning with augmentations. Advances Neural Information Processing Systems, 33:5812-5555 5823, 2020.
[45] Kaveh Hassani and Amir Hosein Khasahmadi. Contrastive multi-viae view representation learning on graphs. In International Conference on Machine Learning, pages 4126. PMLR, 2020.
[46] Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, and Liang Wang. Deep graph contrastive representation learning. arXiv preprint arXiv:2006. 04131, 2020.
[47] Zhen Peng, Wenbing Huang, Minnan Luo, Qinghua Zheng, Yu Rong, Tingyang Xu, and Junzhou Huang. Graph representation learning via graphical mutual information maximization. In Proceedings of The Web Conference 2020, pages 259 270, 2020.
[48] Ken Lang. Newsweeder: Learning to filter netnews. Machine Learning Proceedings 1995, pages 331-339. Elsevier, 1995.
[49] Daniele Vitale, Paolo Ferragina, and Ugo Scaiella. Classification of short texts by deploying topical annotations. In European Conference on Information Retrieval, pages 376-387, Springer, 2012.
[50] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint arXiv: 1OXX::660900844 2016
[51] David D Lewis, Yiming Yang, Tony Russell-Re and Fan Li. RCVI: A new benchmark collection for text categorization research. Journal of machine learning research, 5(Apr): 361-397, 2004.
[52] David Mimno, Hanna Wallach, Edmund Talley, Miriam Leenders, and Andrew McCallum. Optimizing semantic coherence in topic models. Proceedings of the 2011 conference on empirical methods in natural language processing, pages 262-272, 2011.
[53] Nikolaos Aletras and Mark Stevenson. Evaluating topic coherence using distributional semantics. In Proceedings of the 10th International Conference on Computational Semantics (/WCS 2013)- Long Papers, pages 13 22, 2013.
[54] Christopher Manning, Prabhakar Raghavan, and Hinrich Schütze. Introduction to information retrieval. Natural Language Engineering, 16(1):100-103, 2010.
[55] He Zhao, Lan Du, Wray Buntine, and Mingyuan Zhou. Dirichlet belief networks for topic structure learning. Advances neural information processing systems, 31, 2018.
[56] Ryan Gallagher, Kyle Reing, David Kale, and Greg Ver Steeg. Anchored correlation expla- nation: Topic modeling with minimal domain knowledge. Transactions of the Association for Computational Linguistics, 54229-54,, 2017.
[57] Yu Meng, Yunyi Zhang, Jiaxin Huang. Yu Zhang, Chao Zhang, and Jiawei Han. Hierarchical topic mining via joint spherical tree and text embedding. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pages 1908-1917, 2020.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] We describe the limitations of our word in the appendix
(c) Did you discuss any potential negative societal impacts of your work? [Yes] We discuss the negative societal impacts in the appendix
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full of assumptions all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] They will be included in the supplementary material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes]
4. If you using existing assets (e.g., code, data, models) or curating/releasing assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any assets either in the supplemental material URL? [No]
(d) Did you discuss whether and how consent obtained from people whose data you's 're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "p5H_zxNobIj", "reviewer": "Reviewer_VQec", "summary": "This paper improves embedded topic modeling by using a hyperbolic space. The proposed approach particularly helps considering hierarchical structure of words and topics. In addition, the authors propose a revised contrastive loss to inject prior knowledge. The evaluation compares the proposed approach with 6 existing embedding and non-neural-embedding-based topic modeling approaches on 4 datasets. The proposed approach is shown to outperform the existing approaches in most configurations. Some analyses including the 2D visualization show the proposed approach performs as intended and encode the hierarchical information.", "questions": "The paper is quite straightforward and I appreciate it. Please take a look at W1-W3 for suggestions to further improve the presentation.", "limitations": "I cannot spot any negative societal impact of this work.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- S1. The proposed approach encodes hierarchical information, and can incorporate the existing knowledge by adopting the hyperbolic space into an existing neural topic modeling approach.\n- S2. The paper has good presentation and readability in general.\n- S3. The solid empirical results show the performance of the constructed topic model and the analysis shows that the approach works as intended.\n\nWeaknesses\n- W1. The variants HyperETM, HyperMiner are HyperMiner-KG are not explained, and it thus reduces the reproducibility of the experiments.\n- W2. The topic taxonomy (Figure 4b) is hard to relate without document examples and/or summary of the topics.\n- W3. Discussion of side-effect is missing. As the hyperbolic space is enforced, the freedom of representing a topic decreases. As such, there can be side-effects and a discussion of desired properties in topic modeling and side-effects from this approach would be beneficial. For example, is the tree-structure the best or can it be a rather limiting factor?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "dY2NVL2AS24", "reviewer": "Reviewer_BaAH", "summary": "This paper studies the hierarchical topic modeling problem. Existing studies generally learn topic representations in the Euclidean space, which might lead to some limitations in capturing hierarchical relations. To be more specific, the ability to model complex patterns is inherently bounded by the dimensionality of the embedding space. On the other hand, side information such as taxonomy of concepts is sometimes available to guide the learning of hierarchical topics, which might be challenging to preserve in the Euclidean space. As a consequence, this paper proposes a novel framework that introduces learning word and topic representations in the hyperbolic space. Experiments on three public text datasets and auxiliary ablation/case studies demonstrate the effectiveness of the proposed framework.", "questions": "1. If the ability of Euclidean space is bounded by its dimensionality, then what interests the reviewer is that if the dimensionality of one hyperbolic space is d, then what the dimension would be for a Euclidean space to achieve comparable performance in the topic modeling task (e.g., $k * d$ where $k$ is a fixed number)?", "limitations": "As mentioned in the appendix, the main limitation of this work is the mismatch problem between the given structural knowledge and the target corpus. To provide proper guidance for mining an interpretable topic taxonomy, the prior structural knowledge should be well matched with the corresponding dataset.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. This paper studies an important task. Hierarchical topic modeling is a well-studied yet important problem that has the potential of benefiting a wide spectrum of downstream applications such as classification, named entity recognition, etc.\n2. The motivation of the solution is very clear. As is well known, Euclidean space falls short when it comes to modeling hierarchical structures. In contrast, hyperbolic space has nice properties of hierarchical awareness and spacious room which benefits hierarchical relationship modeling.\n3. The experiments are thorough. They are sufficient to justify the superiority of hyperbolic space under the problem set. An ablation study between HyperMiner and HyperMiner-KG validates the modeling of hierarchical external knowledge. Meanwhile,  the experiments also show that the proposed method is extendible to all kinds of ETM models.\n4. The paper is carefully written and well organized.\n\nWeaknesses:\n1. More related works could be potentially included and discussed [1].\n\n[1] Hierarchical Topic Mining via Joint Spherical Tree and Text Embedding\n\nMinor issues:\n- Typos\nIn line 137, \"Sine\" should be \"Since\"\nIn Appendix line 93, \"wth\" should be \"with\"", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "_UAM9yI8Asb", "reviewer": "Reviewer_FP6y", "summary": "This paper presents a neural hierarchical topic model that uses\nhyperbolic embeddings. The paper uses a Gamma-Poisson factorization to\nfacilitate optimizing a variational objective. They augment this\nobjective with a taxonomy-aware contrastive loss as a way of\naddressing hyperbolic manifold learning. Across four datasets, they\ncompare against six baselines. Averaged across multiple runs, they\ndemonstrate small but consistent improvements in their metrics (all\nvariances are fairly small).", "questions": "Q1. What is the impact of changing aspects of the taxonomy (more or less, or regrouping concepts)?\n\nQ2. What is the impact of the positive and negative selection in the contrastive loss?\n\nQ3. Please address my confusion regarding the Lorentz vs. Poincare approaches.", "limitations": "No: see taxonomy experiments/Q1.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Update post-response: I thank the authors for their answers to my questions. \n\nI understand there's a limited amount that can be done during the author response. That said, while the additional taxonomy experiments are encouraging, though for inclusion in a paper they would need additional discussion. \n\nFor whatever revision is next, I suggest that while the proposed clarifications of Poincare vs. Lorentz may be helpful for some readers, the main question is, \"is this level of detail necessary for the main paper?\" If the answer is yes, then this point needs to be made more immediately obvious. \n\n\n=========================\n\nOverall, this paper has a clear mathematical overview. For the\nsingular main contribution (hyperbolic hierarchical topic modeling)\nthis paper presents a decent amount of experimental evidence that the\nproposed approach is effective on purely quantitative measures. While\nthis paper does not address the impact of the proposed approach on the\ntopic model as a language model (perplexity), the classification\nresults give some evidence as to the effect on document modeling.\n\nHowever, lack of examples and experimentation when it comes to the\ntaxonomy is noticable. This is especially the case given the tied\nnature of the model to the taxonomy (L167-169). This is important not\njust for overall completeness, but to better understand the results\nthat have been presented. The gains, e.g., in Figure 3, are generally\nsmall but consistent. That alone is not a negative. However, those\nsmaller gains are less grounded without additional examples or\nexperimentation.\n\nThis paper has some reorganization and other presentation\nissues. While these issues could arguably be viewed as \"easily\naddressable,\" they do impact the ability of a reader to udnerstand\nwhat the paper's take-aways are.\n\n* The current organization of the paper suggests that the authors\n  apply hyperbolic embeddings to a Poisson Gamma belief network\n  only. However, as the results show, this is not the case. I\n  recommend reorganizing so that the main aspect of \"hyperbolic\n  embeddings applied to (hierarchical) topic models\" stands out.\n\n* Since Figure 3 only makes distinctions based on color, it is a bit\n  difficult to quickly interpret.\n\n* The mathematical introduction of both the Lorentz and Poincare\n  similarities is confusing and, given the amount of space dedicated\n  to the equivalence explanation (L140), too nuanced. Indeed, even\n  from the appendix this equivalence is not clear (am I missing\n  something obvious)?\n\n* While Fig. 4b is helpful, 4a is too busy. The concept hierarchy is\n  clear (but that's a given), while the lexical hierarchy, except at a\n  very coarse granularity, is not.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept