TopicNet: Semantic Graph-Guided Topic Discovery

1 Introduction . Topic models , which have the ability to uncover the hidden semantic structure in a text corpus , have been widely applied to text analysis . Generally , a topic model is designed to discover a set of semantically-meaningful latent topics from a collection of documents , each of which captures word co-occurrence patterns commonly observed in a document . While vanilla topic models , such as latent Dirichlet allocation ( LDA ) [ 1 ] and Poisson factor analysis ( PFA ) [ 2 ] , are able to achieve this goal , a series of their hierarchical extensions [ 3–12 ] have been developed in the hope of exploring multi-layer document representations and mining meaningful topic taxonomies . Commonly , these hierarchical topic models aim to learn a hierarchy , in which the latent topics exhibit an increasing level of abstraction when moving towards a deeper hidden layer , as shown in Fig . 1 ( a ) . Consequently , it provides users with an intuitive and interpretable way to better understand textual data . Despite their attractive performance , many existing hierarchical topic models are purely data-driven and incorporate no prior domain knowledge , which may result in some learned topics failing to describe a semantically coherent concept [ 13 ] , especially for those at a deeper hidden layer [ 14 ] . Furthermore , the inflexibility of adding prior knowledge also somewhat limits the applicability of hierarchical topic models , since it is common that a user is interested in a specific topic structure and only focused on information related to it [ 15 ] . To address this issue , we assume a structured prior in the form of a predefined topic hierarchy , where each topic is described by a semantic concept , with the concepts at adjacent layers following the hypernym relations , as illustrated in Fig . 1 ( b ) . Such a ∗Corresponding Author . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . hierarchy can be easily constructed , either by some generic knowledge graph like WordNet [ 16 ] or according to the user ’ s customization . However , there are two main challenges : one is how to model this semantic hierarchy , and the other is how to combine it with topic models . For the first challenge , a general and powerful approach [ 17 ] is to learn a distance-preserving mapping , which maps semantically similar topics to nearby points in the embedding space , with a symmetric distance measure ( e.g. , Euclidean or cosine distance ) typically being used . However , the embeddings learned by such a scheme can not perfectly reflect the entailment between concepts . To mitigate this issue , Vendrov et al . [ 18 ] exploit the partial order structure of the semantic hierarchy to learn order embeddings that respect this partial order structure . On the other hand , considering that probability distributions are better at capturing uncertainties of concepts than point vectors , Athiwaratkun and Wilson [ 19 ] introduce density order embeddings as an improvement to order embeddings . Density order embeddings allows the entailment relationship to be expressed naturally , with general words such as “ entity ” corresponding to broad distributions that encompass more specific words such as “ animal. ” Through encapsulation of probability densities , it can intuitively reflect the hierarchical structure of semantic concepts , thus offering us an effective way to model the topic hierarchy . As for the second challenge of combining the constructed topic hierarchy with topic models , there are two major difficulties . For one thing , the structured prior requires modeling the relationship between topics across different layers directly , while hierarchical topic models often assume that the topics at different layers are independently drawn in the prior [ 6 ] , which makes the fusion of the two a bit far-fetched . For another , since the constructed topic hierarchy can be very deep for large-scale text corpora , an equally deep topic model with multiple latent layers is needed to incorporate with it . However , most existing deep neural topic models are hard to go very deep due to the problem of posterior collapse [ 14 , 20–25 ] . Fortunately , SawETM [ 14 ] , a recently proposed neural topic model , not only builds the dependency between topics at different layers in a shared embedding space , but also alleviates the issue of posterior collapse to some extent through a novel network module referred to as Sawtooth Connection ( SC ) , and hence overcoming both difficulties . In this paper , a novel knowledge-based hierarchical topic model has been proposed , the core idea of which is to represent each topic as a Gaussian embedding , and then project the topics of all layers into a shared embedding space to facilitate the injection of structured prior knowledge into the model . In particular , we first introduce the Gaussian-distributed embeddings to the SawETM , resulting in a variant of SawETM with a stochastic decoder called Gaussian SawETM . To incorporate prior belief into Gaussian SawETM , we then constrain the topics at adjacent layers to encourage them to capture the concepts satisfying the predefined hypernym relations . With auto-encoding variational inference , the entire model is learned in an end-to-end manner by minimizing the evidence lower bound and a regularization term . Extensive experiments demonstrate our model has competitive performance and better interpretability in comparison to most existing hierarchical topic models . The main contributions of the paper can be summarized as follows : • To capture semantic uncertainties of words and topics , we propose a novel probabilistic generative model with a stochastic decoder , referred to as Sawtooth Factorial Gaussian Topic Embeddings guided gamma belief network ( Gaussian SawETM ) . • To incorporate the structured prior knowledge from the real world , we propose TopicNet , a novel knowledge-based hierarchical topic model based on Gaussian SawETM . • In addition to detailed theoretical analysis , we conduct extensive experiments to verify the effectiveness of the above two models . One of the most appealing properties of TopicNet is its interpretability . We conduct extensive qualitative evaluation on the quality of the topics discovered by TopicNet , including word embedding , topic embedding , and topic hierarchies . 2 Gaussian SawETM . In this section , we elaborate on the construction and inference of Gaussian SawETM , a deep hierarchical topic model that represents words and topics as Gaussian-distributed embeddings [ 26 ] . 2.1 Symmetric similarity : expected likelihood kernel . The distance measure plays a key role in quantifying the similarities between embeddings . A simple choice is taking the inner product between the means of two Gaussians , which , however , does not take the advantage of the semantic uncertainties brought by Gaussian-distributed embeddings . Here we employ the expected likelihood kernel [ 19 , 26 , 27 ] as our similarity measure , which is defined as E ( s ) ( αi , αj ) = ∫ x∈Rn N ( x ; µi , Σi ) N ( x ; µj , Σj ) dx = N ( 0 ; µi − µj , Σi+Σj ) , ( 1 ) whereαi is a Gaussian distribution with mean µi and diagonal covariance matrix Σi . As a symmetric similarity function , this kernel considers the impact of semantic uncertainties brought by covariances . 2.2 Document decoder with sawtooth factorization and Gaussian embeddings . The generative network ( also known as decoder ) is one of the core components of topic models . As discussed in Section 1 , to build dependency between topics at two adjacent layers and learn a deep topic hierarchy , we draw experience from the Sawtooth Connection ( SC ) in SawETM . We develop a stochastic decoder by introducing Gaussian-distributed embeddings to better represent the topics in SawETM . Formally , the generative model with T latent layers can be expressed as x ( 1 ) j ∼ Pois ( Φ ( 1 ) θ ( 1 ) j ) , { θ ( t ) j ∼ Gam ( Φ ( t+1 ) θ ( t+1 ) j , 1/c ( t+1 ) j ) } T−1 t=1 , θ ( T ) j ∼ Gam ( γ , 1/c ( T+1 ) j ) , { Φ ( t ) k = Softmax ( log ( E ( s ) ( α ( t−1 ) , α ( t ) ) k ) ) } T t=1 , { α ( t ) k ∼ N ( x ; µ ( t ) k , Σ ( t ) k ) } T t=0 , ( 2 ) In the above formula , xj ∈ ZV denotes the word count vector of the jth document , which is factorized as the product of the factor loading matrix Φ ( 1 ) ∈ RV×K1+ and gamma distributed factor score θ ( 1 ) j under the Poisson likelihood ; and to obtain a multi-layer document representation , the hidden units θ ( t ) j ∈ R Kt + of the t th layer are further factorized into the product of the factor loading Φ ( t+1 ) ∈ RKt×Kt+1+ and hidden units of the next layer . The top-layer hidden units θ ( T ) j are sampled from a prior distribution . In addition , the kth topic α ( t ) k at layer t follows a Gaussian distribution with mean µ ( t ) k and covariance Σ ( t ) k , where the mean vector describes a semantic concept and the covariance matrix reflects its level of abstraction . Note that in the bottom layer α ( 0 ) represents the distributed embeddings of words . And Φ ( t ) k , used to capture the relationships between topics at two adjacent layers , is calculated based on the topic representations in the shared embedding space , instead of being sampled from a Dirichlet distribution . In particular , in the equation E ( s ) ( · ) refers to the symmetric similarity function defined in Eq . ( 1 ) . 2.3 Document encoder : Weibull upward and downward encoder networks . As presented in Section 2.2 , instead of using Gaussian latent variables like most of neural topic models [ 28 ] , our generative model employs the gamma distributed latent variables that are more suitable for modeling sparse and non-negative document representations . While in sampling based inference , the gamma distribution is commonly used to represent the conditional posterior of these latent variables , the difficulty of reparameterizing a gamma distributed random variable makes it difficult to apply it to an inference network [ 29 , 30 ] . For this reason , we utilize a Weibull upwarddownward variational encoder inspired by the work in Zhang et al . [ 25 , 31 ] . We let q ( θ ( t ) j | − ) = Weibull ( k ( t ) j , λ ( t ) j ) , ( 3 ) where , the parameters k ( t ) j , λ ( t ) j ∈ RKt are deterministically transformed from both the observed document features and the information from the stochastic up-down path θ ( t+1 ) j . In detail , the inference network can be expressed as h ′ ( 0 ) j = ReLU ( W ( 0 ) 1 xj + b ( 0 ) 1 ) ) h ′ ( t ) j = h ′ ( t−1 ) j + ReLU ( W ( t ) 1 h ′ ( t−1 ) j + b ( t ) 1 ) , t = 1 , · · · , T , h ( t ) j = h ′ ( t ) j ⊕Φ ( t+1 ) θ ( t+1 ) j , t = 0 , · · · , T − 1 , h ( T ) j = h ′ ( T ) j , k ( t ) j = Softplus ( W ( t ) 2 h ( t ) j + b ( t ) 2 ) , λ ( t ) j = Softplus ( W ( t ) 3 h ( t ) j + b ( t ) 3 ) , t = 0 , · · · , T , ( 4 ) where { b ( t ) i } 3 , T i=1 , t=1 ∈ RKt , { W ( t ) i } 3 , T i=1 , t=1 ∈ RKt×Kt−1 , and { h ( t ) j } N , T j=1 , t=1 ∈ RKt , ⊕ denotes the concatenation in feature dimension , ReLU ( · ) = max ( 0 , · ) is the nonlinear activation function , and Softplus ( · ) applies ln [ 1+ exp ( · ) ] to each element to ensure positive shape and scale parameters of the Weibull distribution . To reduce the risk of posterior collapse , a skip-connected deterministic upward path [ 14 ] is used to obtain the hidden representations { h ′ ( t ) j } N , T j=1 , t=1 of the input xj .

Summary: The authors propose a topic model that aim at embedding words and topics into a latent space via Gaussian embeddings. Additional constraints in the training procedure allows to incorporate prior knowledge on the topics, so that they are able to include semantic topic relations.  The model is evaluated on a large set of public datasets and on a number of tasks. It achieves better results with respect to baselines on clustering tasks, and comparable on classification tasks. However, the main advantage is the increase interpretability of topics and topic hierarchy.