TopicNet: Semantic Graph-Guided Topic Discovery

1 Introduction . Topic models , which have the ability to uncover the hidden semantic structure in a text corpus , have been widely applied to text analysis . Generally , a topic model is designed to discover a set of semantically-meaningful latent topics from a collection of documents , each of which captures word co-occurrence patterns commonly observed in a document . While vanilla topic models , such as latent Dirichlet allocation ( LDA ) [ 1 ] and Poisson factor analysis ( PFA ) [ 2 ] , are able to achieve this goal , a series of their hierarchical extensions [ 3–12 ] have been developed in the hope of exploring multi-layer document representations and mining meaningful topic taxonomies . Commonly , these hierarchical topic models aim to learn a hierarchy , in which the latent topics exhibit an increasing level of abstraction when moving towards a deeper hidden layer , as shown in Fig . 1 ( a ) . Consequently , it provides users with an intuitive and interpretable way to better understand textual data . Despite their attractive performance , many existing hierarchical topic models are purely data-driven and incorporate no prior domain knowledge , which may result in some learned topics failing to describe a semantically coherent concept [ 13 ] , especially for those at a deeper hidden layer [ 14 ] . Furthermore , the inflexibility of adding prior knowledge also somewhat limits the applicability of hierarchical topic models , since it is common that a user is interested in a specific topic structure and only focused on information related to it [ 15 ] . To address this issue , we assume a structured prior in the form of a predefined topic hierarchy , where each topic is described by a semantic concept , with the concepts at adjacent layers following the hypernym relations , as illustrated in Fig . 1 ( b ) . Such a ∗Corresponding Author . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . hierarchy can be easily constructed , either by some generic knowledge graph like WordNet [ 16 ] or according to the user ’ s customization . However , there are two main challenges : one is how to model this semantic hierarchy , and the other is how to combine it with topic models . For the first challenge , a general and powerful approach [ 17 ] is to learn a distance-preserving mapping , which maps semantically similar topics to nearby points in the embedding space , with a symmetric distance measure ( e.g. , Euclidean or cosine distance ) typically being used . However , the embeddings learned by such a scheme can not perfectly reflect the entailment between concepts . To mitigate this issue , Vendrov et al . [ 18 ] exploit the partial order structure of the semantic hierarchy to learn order embeddings that respect this partial order structure . On the other hand , considering that probability distributions are better at capturing uncertainties of concepts than point vectors , Athiwaratkun and Wilson [ 19 ] introduce density order embeddings as an improvement to order embeddings . Density order embeddings allows the entailment relationship to be expressed naturally , with general words such as “ entity ” corresponding to broad distributions that encompass more specific words such as “ animal. ” Through encapsulation of probability densities , it can intuitively reflect the hierarchical structure of semantic concepts , thus offering us an effective way to model the topic hierarchy . As for the second challenge of combining the constructed topic hierarchy with topic models , there are two major difficulties . For one thing , the structured prior requires modeling the relationship between topics across different layers directly , while hierarchical topic models often assume that the topics at different layers are independently drawn in the prior [ 6 ] , which makes the fusion of the two a bit far-fetched . For another , since the constructed topic hierarchy can be very deep for large-scale text corpora , an equally deep topic model with multiple latent layers is needed to incorporate with it . However , most existing deep neural topic models are hard to go very deep due to the problem of posterior collapse [ 14 , 20–25 ] . Fortunately , SawETM [ 14 ] , a recently proposed neural topic model , not only builds the dependency between topics at different layers in a shared embedding space , but also alleviates the issue of posterior collapse to some extent through a novel network module referred to as Sawtooth Connection ( SC ) , and hence overcoming both difficulties . In this paper , a novel knowledge-based hierarchical topic model has been proposed , the core idea of which is to represent each topic as a Gaussian embedding , and then project the topics of all layers into a shared embedding space to facilitate the injection of structured prior knowledge into the model . In particular , we first introduce the Gaussian-distributed embeddings to the SawETM , resulting in a variant of SawETM with a stochastic decoder called Gaussian SawETM . To incorporate prior belief into Gaussian SawETM , we then constrain the topics at adjacent layers to encourage them to capture the concepts satisfying the predefined hypernym relations . With auto-encoding variational inference , the entire model is learned in an end-to-end manner by minimizing the evidence lower bound and a regularization term . Extensive experiments demonstrate our model has competitive performance and better interpretability in comparison to most existing hierarchical topic models . The main contributions of the paper can be summarized as follows : • To capture semantic uncertainties of words and topics , we propose a novel probabilistic generative model with a stochastic decoder , referred to as Sawtooth Factorial Gaussian Topic Embeddings guided gamma belief network ( Gaussian SawETM ) . • To incorporate the structured prior knowledge from the real world , we propose TopicNet , a novel knowledge-based hierarchical topic model based on Gaussian SawETM . • In addition to detailed theoretical analysis , we conduct extensive experiments to verify the effectiveness of the above two models . One of the most appealing properties of TopicNet is its interpretability . We conduct extensive qualitative evaluation on the quality of the topics discovered by TopicNet , including word embedding , topic embedding , and topic hierarchies . 2 Gaussian SawETM . In this section , we elaborate on the construction and inference of Gaussian SawETM , a deep hierarchical topic model that represents words and topics as Gaussian-distributed embeddings [ 26 ] . 2.1 Symmetric similarity : expected likelihood kernel . The distance measure plays a key role in quantifying the similarities between embeddings . A simple choice is taking the inner product between the means of two Gaussians , which , however , does not take the advantage of the semantic uncertainties brought by Gaussian-distributed embeddings . Here we employ the expected likelihood kernel [ 19 , 26 , 27 ] as our similarity measure , which is defined as E ( s ) ( αi , αj ) = ∫ x∈Rn N ( x ; µi , Σi ) N ( x ; µj , Σj ) dx = N ( 0 ; µi − µj , Σi+Σj ) , ( 1 ) whereαi is a Gaussian distribution with mean µi and diagonal covariance matrix Σi . As a symmetric similarity function , this kernel considers the impact of semantic uncertainties brought by covariances . 2.2 Document decoder with sawtooth factorization and Gaussian embeddings . The generative network ( also known as decoder ) is one of the core components of topic models . As discussed in Section 1 , to build dependency between topics at two adjacent layers and learn a deep topic hierarchy , we draw experience from the Sawtooth Connection ( SC ) in SawETM . We develop a stochastic decoder by introducing Gaussian-distributed embeddings to better represent the topics in SawETM . Formally , the generative model with T latent layers can be expressed as x ( 1 ) j ∼ Pois ( Φ ( 1 ) θ ( 1 ) j ) , { θ ( t ) j ∼ Gam ( Φ ( t+1 ) θ ( t+1 ) j , 1/c ( t+1 ) j ) } T−1 t=1 , θ ( T ) j ∼ Gam ( γ , 1/c ( T+1 ) j ) , { Φ ( t ) k = Softmax ( log ( E ( s ) ( α ( t−1 ) , α ( t ) ) k ) ) } T t=1 , { α ( t ) k ∼ N ( x ; µ ( t ) k , Σ ( t ) k ) } T t=0 , ( 2 ) In the above formula , xj ∈ ZV denotes the word count vector of the jth document , which is factorized as the product of the factor loading matrix Φ ( 1 ) ∈ RV×K1+ and gamma distributed factor score θ ( 1 ) j under the Poisson likelihood ; and to obtain a multi-layer document representation , the hidden units θ ( t ) j ∈ R Kt + of the t th layer are further factorized into the product of the factor loading Φ ( t+1 ) ∈ RKt×Kt+1+ and hidden units of the next layer . The top-layer hidden units θ ( T ) j are sampled from a prior distribution . In addition , the kth topic α ( t ) k at layer t follows a Gaussian distribution with mean µ ( t ) k and covariance Σ ( t ) k , where the mean vector describes a semantic concept and the covariance matrix reflects its level of abstraction . Note that in the bottom layer α ( 0 ) represents the distributed embeddings of words . And Φ ( t ) k , used to capture the relationships between topics at two adjacent layers , is calculated based on the topic representations in the shared embedding space , instead of being sampled from a Dirichlet distribution . In particular , in the equation E ( s ) ( · ) refers to the symmetric similarity function defined in Eq . ( 1 ) . 2.3 Document encoder : Weibull upward and downward encoder networks . As presented in Section 2.2 , instead of using Gaussian latent variables like most of neural topic models [ 28 ] , our generative model employs the gamma distributed latent variables that are more suitable for modeling sparse and non-negative document representations . While in sampling based inference , the gamma distribution is commonly used to represent the conditional posterior of these latent variables , the difficulty of reparameterizing a gamma distributed random variable makes it difficult to apply it to an inference network [ 29 , 30 ] . For this reason , we utilize a Weibull upwarddownward variational encoder inspired by the work in Zhang et al . [ 25 , 31 ] . We let q ( θ ( t ) j | − ) = Weibull ( k ( t ) j , λ ( t ) j ) , ( 3 ) where , the parameters k ( t ) j , λ ( t ) j ∈ RKt are deterministically transformed from both the observed document features and the information from the stochastic up-down path θ ( t+1 ) j . In detail , the inference network can be expressed as h ′ ( 0 ) j = ReLU ( W ( 0 ) 1 xj + b ( 0 ) 1 ) ) h ′ ( t ) j = h ′ ( t−1 ) j + ReLU ( W ( t ) 1 h ′ ( t−1 ) j + b ( t ) 1 ) , t = 1 , · · · , T , h ( t ) j = h ′ ( t ) j ⊕Φ ( t+1 ) θ ( t+1 ) j , t = 0 , · · · , T − 1 , h ( T ) j = h ′ ( T ) j , k ( t ) j = Softplus ( W ( t ) 2 h ( t ) j + b ( t ) 2 ) , λ ( t ) j = Softplus ( W ( t ) 3 h ( t ) j + b ( t ) 3 ) , t = 0 , · · · , T , ( 4 ) where { b ( t ) i } 3 , T i=1 , t=1 ∈ RKt , { W ( t ) i } 3 , T i=1 , t=1 ∈ RKt×Kt−1 , and { h ( t ) j } N , T j=1 , t=1 ∈ RKt , ⊕ denotes the concatenation in feature dimension , ReLU ( · ) = max ( 0 , · ) is the nonlinear activation function , and Softplus ( · ) applies ln [ 1+ exp ( · ) ] to each element to ensure positive shape and scale parameters of the Weibull distribution . To reduce the risk of posterior collapse , a skip-connected deterministic upward path [ 14 ] is used to obtain the hidden representations { h ′ ( t ) j } N , T j=1 , t=1 of the input xj .

Summary: This paper mainly proposes a new hierarchical topic discovery framework that is guided by prior (tree-structured) knowledge. The construction of the framework is broken down into two steps: Modeling semantic hierarchy with a Gaussian-embedding-based topic model (Gaussian SawETM), and injecting the prior semantic knowledge by regularizing via entailment relationship. The model is evaluated on a set of benchmarks with perplexity, topic quality, and clustering/classification protocols. Qualitative studies are also shown to demonstrate the interpretability of the model.