Topic Modeling Revisited: A Document Graph-based Neural Network Perspective

1 Introduction . As one of the most widely-used techniques for document analysis , topic modeling aims to learn a set of latent topics from the observed document collection , each of which describes an interpretable semantic concept . In particular , Latent Dirichlet Allocation ( LDA ) [ 5 ] and its extensions [ 3 , 9 , 44 , 60 ] have achieved great success in various application scenarios over the past decades . These approaches usually specify a probabilistic generative model to draw the document data with a structure of latent variables sampled from prior distributions and word distribution on the vocabulary as topics . Recently , with the development of Variational Autoencoder ( VAE ) [ 24 ] , a Neural Variational Inference algorithm ( NVI ) with neural networks for topic modeling , namely Neural Topic Model , has attracted great attention [ 35 , 34 , 11 , 12 , 42 , 49 , 61 ] , due to its appealing flexibility and scalability . However , it is well-known that traditional topic modeling usually has the bag-of-word assumption , where the words in a document are assumed to be “ exchangeable ” , which also arouses two typical challenges behind most topic models . On one hand , given the topic distribution of the document , the words are conditionally independent . This implicates that the dependency relationship among words is totally ignored , while words are dependent on each other according to linguistic knowledge [ 30 ] . ∗ * Corresponding authors . This work was accomplished when Dazhong Shen was an intern in Baidu supervised by Chuan Qin and Hengshu Zhu . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . On the other hand , the concept of topics is introduced as multinomial distributions on vocabulary without modeling the dependency relation among words , which produces more ambiguous terms in topics . For example , LDA discovers word such as “ network ” in a topic which does not seem to be that insightful . Instead , we can convey more interpretability to readers by mining the strong word dependency between “ neural ” and “ network ” [ 19 ] . Along this line , many extensions have been proposed to address the above two issues by dealing with the dependency relation among words . One thread of these works focus on the sequential dependency by taking the document as a sequence of words [ 12 , 15 , 16 , 28 , 32 , 42 , 56 , 52 , 55 , 60 ] . However , most of them prefer to enhance language modeling with better performance on language modeling metric , such as Perplexity , rather than the state-of-the-art quality of learned topics . It may be due to the limitation that they only handle the linear dependency of words [ 30 ] and push models to capture the local syntax information instead of global semantic information . Specifically , to mitigate the first problem , most of them assume that the topic assignment or generative possibility of a word is dependent on the proceeding words [ 12 , 15 , 16 , 28 , 42 , 56 ] . As for the second problem , related literature is restricted to exploring n-gram terms by integrating few neighboring words as one term with Markov Chain based generative story [ 52 , 55 , 32 ] . Nonetheless , words may be mutually dependent on each other in a much more complex and nonlinear manner . Therefore , several studies explore to model the graph representation of documents , where nodes denote words and edges represent the relationships among words , such as syntax or semantic relation constructed by dependency parsing [ 30 , 54 ] or justified by relative position among words [ 9 , 61 ] . They either consider the graph structure as side information to constrain the relation of topic assignments of words [ 30 , 54 ] , or replacing the word sets with edge sets as the instances to generative [ 9 , 61 ] . Most of them only trickle with the first problem without efforts to enhance the interpretability of topics . To this end , in this paper , we represent documents as directed semantic graphs by introducing word dependency as edges between word nodes , and develop a Graph Neural Topic Modeling ( GNTM ) method . The key contributions of our model can be categorized into four parts : 1 ) we formulate a welldefined probabilistic generative story of the document graph with the novel generative probabilistic functions for both graph structure and word sets ; 2 ) we propose a new concept of topics , consisting of multinomial distributions on both the vocabulary , like conventional topic models , and word dependency edges ; 3 ) a Neural Variational Inference ( NVI ) approach is designed to infer our model with graph neural network to encode the document graphs ; 4 ) we also demonstrate that LDA can be derived from GNTM as a special case with similar objective functions . Finally , extensive experiments over four benchmark datasets have clearly demonstrated the effectiveness and interpretability of GNTM compared with state-of-the-art baselines . 2 Preliminary . 2.1 Topic Model . Topic modeling is one of the most important text mining techniques and has been extensively studied for a variety of applications in recent two decades [ 2 , 21 , 31 , 48 , 47 ] . Among the most representative models , Latent Dirichlet Allocation ( LDA ) [ 5 ] formulates each topic k as the distribution βvk on vocabulary V and draws each word wd , n in a document d from one of K topics stochastically . The well-known generative process of LDA can be described as three steps : 1 ) sampling document-topic proportions θd ∼ Dir ( α ) , where α is hype-parameter for the Dirichlet distribution prior ; 2 ) drawing individual topic assignments zd , n ∼ Multi ( θd ) ; 3 ) generating each word wd , n ∼ Multi ( βvzd , n ) . Then , the marginal likelihood for a document d is : p ( d|α , β ) = ∫ θd p ( θd|α ) ∏ n ∑ zd , n p ( wd , n|βvzd , n ) p ( zd , n|θd ) dθd . ( 1 ) Conventional approaches for inferring topic models , such as LDA , are either Markov Chain Monte Carlo [ 7 ] or probabilistic variational inference [ 22 ] , which are both widely applied [ 58 , 14 , 46 ] . However , they either require careful selection of distribution for each latent variable or tolerate arduous and customized mathematical derivation case by case , which limits the flexibility and scalability of model design [ 35 ] . Recently , with the successes of Variational Autoencoder ( VAE ) [ 24 ] , which applies the variational distributions parameterized by neural networks to approximate the posterior of latent variables , the use of deep learning on inferring topic models has attracted more and more attention [ 57 ] . Most of those neural topic models ( NTMs ) [ 35 , 34 , 49 , 11 ] can be regarded as extensions of LDA , but often integrate out the discrete latent variable zd , n . Therefore , the variational lower bound ( ELBO ) for the log-likelihood of the document d can be derived as : Ld = Eqφ ( θ|d ) [ ∑ n log p ( wd , n|θd , βv ) ] −DKL [ qφ ( θd|d ) ||p ( θd ) ] , ( 2 ) where qφ ( θ|d ) is the variational distribution of θd parametrized by a neural network φ . To obtain a differentiable estimator of Ld , the reparameterization trick ( RT ) has been applied for θ . In particular , due to the limitation of RT on Dirichlet distribution [ 49 ] , those approaches usually stand on Gaussianbased posterior qφ ( θd|d ) . 2.2 Document Graph Construction . Most existing methods in topic models commonly represent a document as an unordered collection of words , known as bag-of-word ( BOW ) representation , which totally ignores the relationship among words in the document . To represent documents with word dependency relations , we discuss the representation of documents in a graph manner , where the graph-based models has been widely explored and applied recently extensively [ 25 , 50 , 13 ] . Formally , given one document d with word sequences [ wd , t ] Ndt=1 , we construct a directed edge from wd , t1 to wd , t2 if 0 < t2 − t1 ≤ s , where s is the hyper-parameter of window size . The hidden logic is that the semantic is consistent and interdependent among neighboring words , which can be traced back to n-gram [ 8 ] , BTM [ 9 ] and Word2vec [ 36 ] . As the example in Figure 1 ( a ) shows ( stop words are filtered ) , we can denote a document d as a graph Gd = ( Vd , Ed ) , where Vd = { wd , n } Ndn=1 ( words as nodes , e.g. , 7 words in Figure 1 ( a ) ) and Ed are sets of nodes and word edges ( e.g. , ( “ artificial ” , “ neural ” ) ) , respectively . Note that , for simplification in analysis , we distinguish the same words on different positions in Ed and Vd . We also denote E as the unique word edge sets in the whole document collection with nodes in the vocabulary V = { wi } |V |i=1 ( unique word set ) . For convention , we further denoteGod = ( V o d , E o d ) as the graph structure ofGd with only the placeholders V od = { 1 , 2 , ... , Nd } ( e.g. , { 1 , 2 , .. , 7 } in Figure 1 ( a ) ) , not the specific words Vd , where |V od | = |Vd| , and the placeholder edge set Eod is the edge set on the placeholder set V o d ( e.g. , ( 1 , 2 ) ) , distinguishing from the word edge set Ed . 3 Graph Neural Topic Model . With the definition of the document graph Gd , we turn to introduce the technical detail of GNTM . Specifically , we first formulate the generative story for both the graph structure God and word sets Vd . Then , an NVI inference algorithm will be proposed to learn our model . 3.1 Generative Story . Figure 1 ( b ) shows our directed graphical model . In contrast to three-step generative process in LDA , four steps are involved in the generative story . The main differences lie in the conditional generation for the observed data given all latent variables ( steps 3 and 4 in the following ) . To be specific , 1 ) the generative story of a document d begins with topic proportion θ drawn from the Dirichlet prior Dir ( α ) . Then , 2 ) each word wn ( or placeholder n ) has corresponding topic assignment zn drawn from Multi ( θ ) . Finally we generate the observed data with two steps : 3 ) drawing the graph structure God conditioning on the topic assignment set Zd , i.e. , G o d ∼ p ( God|Zd ) ; 4 ) generating the word set Vd based on both topic assignment set Zd and graph structure God , i.e. , Vd ∼ p ( Vd|God , Zd ) . Based on the generative process , the joint model of Gd and all latent variables can be decomposed as : p ( Gd , θd , Zd ; α ) = p ( Vd|Zd , God ) p ( God|Zd ) Nd∏ n=1 p ( zd , n|θd ) p ( θ|α ) . ( 3 ) In the following , we turn to formulate the formal definition of p ( God|Zd ) and p ( Vd|Zd , God ) with additional parameters , i.e. , topic dependency matrix M and topic set β , which is the most important contribution of our model .

Summary: Going beyond the bag-of-word assumption in topic modelling is an interesting problem, which has been studied since the introduction of the vanilla LDA, for instance, the syntactic topic model. This paper proposes to represent each document as a directed word graph, which is constructed via moving a window along the word sequence. The main technical contributions of this paper can be 1) the graph neural topic model, where words are generated conditioned on both the topic assignments and the reconstructed word graph; 2) the VAE framework used to learn the posteriors.