Topic Modeling Revisited: A Document Graph-based Neural Network Perspective
Year: 2021

Abstract:
Most topic modeling approaches are based on the bag-of-words assumption, where each word is required to be conditionally independent in the same document. As a result, both of the generative story and the topic formulation have totally ignored the semantic dependency among words, which is important for improving the semantic comprehension and model interpretability. To this end, in this paper, we revisit the task of topic modeling by transforming each document into a directed graph with word dependency as edges between word nodes, and develop a novel approach, namely Graph Neural Topic Model (GNTM). Specifically, in GNTM, a well-defined probabilistic generative story is designed to model both the graph structure and word sets with multinomial distributions on the vocabulary and word dependency edge set as the topics. Meanwhile, a Neural Variational Inference (NVI) approach is proposed to learn our model with graph neural networks to encode the document graphs. Besides, we theoretically demonstrate that Latent Dirichlet Allocation (LDA) can be derived from GNTM as a special case with similar objective functions. Finally, extensive experiments on four benchmark datasets have clearly demonstrated the effectiveness and interpretability of GNTM compared with state-of-the-art baselines.

heading: 1 Introduction
text: As one of the most widely-used techniques for document analysis, topic modeling aims to learn a set of latent topics from the observed document collection, each of which describes an interpretable semantic concept. In particular, Latent Dirichlet Allocation (LDA) [5] and its extensions [3, 9, 44, 60] have achieved great success in various application scenarios over the past decades. These approaches usually specify a probabilistic generative model to draw the document data with a structure of latent variables sampled from prior distributions and word distribution on the vocabulary as topics. Recently, with the development of Variational Autoencoder (VAE) [24], a Neural Variational Inference algorithm (NVI) with neural networks for topic modeling, namely Neural Topic Model, has attracted great attention [35, 34, 11, 12, 42, 49, 61] , due to its appealing flexibility and scalability.
However, it is well-known that traditional topic modeling usually has the bag-of-word assumption, where the words in a document are assumed to be “exchangeable”, which also arouses two typical challenges behind most topic models. On one hand, given the topic distribution of the document, the words are conditionally independent. This implicates that the dependency relationship among words is totally ignored, while words are dependent on each other according to linguistic knowledge [30].
∗*Corresponding authors. This work was accomplished when Dazhong Shen was an intern in Baidu supervised by Chuan Qin and Hengshu Zhu.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
On the other hand, the concept of topics is introduced as multinomial distributions on vocabulary without modeling the dependency relation among words, which produces more ambiguous terms in topics. For example, LDA discovers word such as “network” in a topic which does not seem to be that insightful. Instead, we can convey more interpretability to readers by mining the strong word dependency between “neural” and “network” [19].
Along this line, many extensions have been proposed to address the above two issues by dealing with the dependency relation among words. One thread of these works focus on the sequential dependency by taking the document as a sequence of words [12, 15, 16, 28, 32, 42, 56, 52, 55, 60]. However, most of them prefer to enhance language modeling with better performance on language modeling metric, such as Perplexity, rather than the state-of-the-art quality of learned topics. It may be due to the limitation that they only handle the linear dependency of words [30] and push models to capture the local syntax information instead of global semantic information. Specifically, to mitigate the first problem, most of them assume that the topic assignment or generative possibility of a word is dependent on the proceeding words [12, 15, 16, 28, 42, 56]. As for the second problem, related literature is restricted to exploring n-gram terms by integrating few neighboring words as one term with Markov Chain based generative story [52, 55, 32].
Nonetheless, words may be mutually dependent on each other in a much more complex and nonlinear manner. Therefore, several studies explore to model the graph representation of documents, where nodes denote words and edges represent the relationships among words, such as syntax or semantic relation constructed by dependency parsing [30, 54] or justified by relative position among words [9, 61]. They either consider the graph structure as side information to constrain the relation of topic assignments of words [30, 54], or replacing the word sets with edge sets as the instances to generative [9, 61]. Most of them only trickle with the first problem without efforts to enhance the interpretability of topics.
To this end, in this paper, we represent documents as directed semantic graphs by introducing word dependency as edges between word nodes, and develop a Graph Neural Topic Modeling (GNTM) method. The key contributions of our model can be categorized into four parts: 1) we formulate a welldefined probabilistic generative story of the document graph with the novel generative probabilistic functions for both graph structure and word sets; 2) we propose a new concept of topics, consisting of multinomial distributions on both the vocabulary, like conventional topic models, and word dependency edges; 3) a Neural Variational Inference (NVI) approach is designed to infer our model with graph neural network to encode the document graphs; 4) we also demonstrate that LDA can be derived from GNTM as a special case with similar objective functions. Finally, extensive experiments over four benchmark datasets have clearly demonstrated the effectiveness and interpretability of GNTM compared with state-of-the-art baselines.

heading: 2 Preliminary

heading: 2.1 Topic Model
text: Topic modeling is one of the most important text mining techniques and has been extensively studied for a variety of applications in recent two decades [2, 21, 31, 48, 47]. Among the most representative models, Latent Dirichlet Allocation (LDA) [5] formulates each topic k as the distribution βvk on vocabulary V and draws each word wd,n in a document d from one of K topics stochastically. The well-known generative process of LDA can be described as three steps: 1) sampling document-topic proportions θd ∼ Dir(α), where α is hype-parameter for the Dirichlet distribution prior; 2) drawing individual topic assignments zd,n ∼ Multi(θd); 3) generating each word wd,n ∼ Multi(βvzd,n). Then, the marginal likelihood for a document d is:
p(d|α, β) = ∫ θd p(θd|α) ∏ n ∑ zd,n p(wd,n|βvzd,n)p(zd,n|θd)dθd. (1)
Conventional approaches for inferring topic models, such as LDA, are either Markov Chain Monte Carlo [7] or probabilistic variational inference [22], which are both widely applied [58, 14, 46]. However, they either require careful selection of distribution for each latent variable or tolerate arduous and customized mathematical derivation case by case, which limits the flexibility and scalability of model design [35]. Recently, with the successes of Variational Autoencoder (VAE) [24],
which applies the variational distributions parameterized by neural networks to approximate the posterior of latent variables, the use of deep learning on inferring topic models has attracted more and more attention [57]. Most of those neural topic models (NTMs) [35, 34, 49, 11] can be regarded as extensions of LDA, but often integrate out the discrete latent variable zd,n. Therefore, the variational lower bound (ELBO) for the log-likelihood of the document d can be derived as:
Ld = Eqφ(θ|d)[ ∑ n log p(wd,n|θd, βv)]−DKL[qφ(θd|d)||p(θd)], (2)
where qφ(θ|d) is the variational distribution of θd parametrized by a neural network φ. To obtain a differentiable estimator of Ld, the reparameterization trick (RT) has been applied for θ. In particular, due to the limitation of RT on Dirichlet distribution [49], those approaches usually stand on Gaussianbased posterior qφ(θd|d).

heading: 2.2 Document Graph Construction
text: Most existing methods in topic models commonly represent a document as an unordered collection of words, known as bag-of-word (BOW) representation, which totally ignores the relationship among words in the document. To represent documents with word dependency relations, we discuss the representation of documents in a graph manner, where the graph-based models has been widely explored and applied recently extensively [25, 50, 13].
Formally, given one document d with word sequences [wd,t]Ndt=1, we construct a directed edge from wd,t1 to wd,t2 if 0 < t2 − t1 ≤ s, where s is the hyper-parameter of window size. The hidden logic is that the semantic is consistent and interdependent among neighboring words, which can be traced back to n-gram [8], BTM [9] and Word2vec [36]. As the example in Figure 1(a) shows (stop words are filtered), we can denote a document d as a graph Gd = (Vd, Ed), where Vd = {wd,n}Ndn=1 (words as nodes, e.g., 7 words in Figure 1(a)) and Ed are sets of nodes and word edges (e.g., (“artificial”, “neural”)), respectively. Note that, for simplification in analysis, we distinguish the same words on different positions in Ed and Vd. We also denote E as the unique word edge sets in the whole document collection with nodes in the vocabulary V = {wi}|V |i=1 (unique word set). For convention, we further denoteGod = (V o d , E o d) as the graph structure ofGd with only the placeholders V od = {1, 2, ..., Nd} (e.g., {1, 2, .., 7} in Figure 1(a)), not the specific words Vd, where |V od | = |Vd|, and the placeholder edge set Eod is the edge set on the placeholder set V o d (e.g., (1, 2)), distinguishing from the word edge set Ed.

heading: 3 Graph Neural Topic Model
text: With the definition of the document graph Gd, we turn to introduce the technical detail of GNTM. Specifically, we first formulate the generative story for both the graph structure God and word sets Vd. Then, an NVI inference algorithm will be proposed to learn our model.

heading: 3.1 Generative Story
text: Figure 1(b) shows our directed graphical model. In contrast to three-step generative process in LDA, four steps are involved in the generative story. The main differences lie in the conditional generation for the observed data given all latent variables (steps 3 and 4 in the following). To be specific, 1) the generative story of a document d begins with topic proportion θ drawn from the Dirichlet prior Dir(α). Then, 2) each word wn (or placeholder n) has corresponding topic assignment zn drawn from Multi(θ). Finally we generate the observed data with two steps: 3) drawing the graph structure
God conditioning on the topic assignment set Zd, i.e., G o d ∼ p(God|Zd); 4) generating the word set Vd based on both topic assignment set Zd and graph structure God, i.e., Vd ∼ p(Vd|God, Zd). Based on the generative process, the joint model of Gd and all latent variables can be decomposed as:
p(Gd, θd, Zd;α) = p(Vd|Zd, God)p(God|Zd) Nd∏ n=1 p(zd,n|θd)p(θ|α). (3)
In the following, we turn to formulate the formal definition of p(God|Zd) and p(Vd|Zd, God) with additional parameters, i.e., topic dependency matrix M and topic set β, which is the most important contribution of our model.

heading: 3.1.1 Generation for Graph Structure
text: Here, we aim to construct the graph structure God based solely on the topic assignment zd,n on each placeholder wod,n. The hidden motivation is to explain the word dependency as the semantic dependency among their topics, which can encourage our model to find the relation among different topics in turn. To be specific, we denote M ∈ RK×K as the dependency matrix among topics, where each element mi,j ∈ [0, 1] represents the possibility that there exists a dependency edge between two placeholders with topic i and j, respectively. Then, we can formulate p(God|Zd) as :
p(God|Zd;M) = ∏
(n,n′)∈Eo d
mzd,n,zd,n′ ∏
(n,n′)/∈Eo d
(1−mzd,n,zd,n′ ), (4)
where M is a learnable parameter without additional assumptions.

heading: 3.1.2 Generation for Word Set
text: Contrast to LDA, once a document is represented as a graph, the word sampling should be cast as the sampling of this graph. In other words, we should consider dependency between words when generating word set Vd. To provide a well-defined word sampling process, we first re-define the concept of topic. Different from the conventional concept of topic in LDA, which is represented by the multinomial distribution βvk on the vocabulary, we further enhance each topic k by introducing one distribution on word dependency edge with parameters βek, i.e., βk = (β v k , β
e k), where:∑
w∈V
βvk,w = 1, ∑
w,w′∈V
βvk,wβ v k,w′β e k,(w,w′) = 1, β v k,w ≥ 0, βek,(w,w′) ≥ 0. (5)
where the product βvk,wβ v k,w′β e k,(w,w′) denotes the possibility of the directed edge from word w to w
′
under the topic k. Note that, in practice, only parameters βek,(w,w′) corresponding to edge (w.w ′) ∈ E would be used in model learning. In other words, we can reduce the space complexity of βek into O(|E|) easily by assuming that βek,(w,w′) = 0,∀(w,w ′) /∈ E.
Then, based on the new concept of the topics, we propose a well-defined probability measure for generating each word set Vd with the document graph structure God and topic assignment sets Zd. The probability of p(Vd|Zd, God) is modeled by :
p(Vd|Zd, God;β) = K∏ k=1 p(Vd,k|God,k, βk),
p(Vd,k|God,k;βk) = 1 |Ed,k| ∏
w∈Vd,k
βvk,w ∑
(w,w′)∈Ed,k
βek,(w,w′),
(6)
where God,k is the graph structure consisting of all placeholders with the topic assignment k, similarly, Vd,k is the set of words assigned by topic k and Ed,k is the word dependency edge among them. In other words, we split the document graph Gd into K dependent sub-graphs Gd,k based on topic assignment set Zd and then define the generative possibility of each topical sub-graph Gd,k with the word dependency edges Ed,k. If Ed,k = ∅, we set ∑ (w,w′)∈Ed,k β e k,(w,w′)/|Ed,k| = 1. Mathematically, we have the following theorem:
Theorem 1 Given the topic set β in Equation 5 and the document graph structureGod, the probability function p(Vd|Zd, God;β) in Equation 6 is a legal probability measure on the vocabulary V .
The proof can be found in Appendix A.1 which is mainly inspired by [30].

heading: 3.2 NVI Inference
text: Here, we infer our model by applying a NVI inference algorithm, which is well-know of flexibility and scalability [24]. Based on the mean field assumption, we approximate the posterior of latent variables θd and Zd for each document d with the following variational family:
q(θd, Zd|Gd) = qφ(θd|Gd;µd, δd) Nd∏ n=1 qφ(zd,n|Gd, wd,n;ϕd,n), (7)
where qφ(zd,n|Gd, wd,n) is defined as one multinomial distribution with parameters ϕd,n and qφ(θd|Gd) is the approximation for the true Dirichlet posterior p(θ|Gd). Both of them are parameterized by neural networks φ. However, directly approaching the Dirichlet distributions would make the reparameterization trick hard to apply. Here, we follow [49, 17] and resolve this issue by constructing a Laplace approximation to the Dirichlet posterior with the Logistic Normal distribution parameterized by (µd, δd), that is,
θd = softmax(N (µd, δ2d)). (8)
Then, we can derive the training objective for each document d, i.e., ELBO:
Ld = Eq(Zd|Gd)[log p(G o d|Zd;M) + log p(Vd|Zd, God;β)]
−KL[q(θd|Gd)||p(θd)]− Eq(θd|Gd)[ Nd∑ n=1 KL[q(zd,n|Gd, wd,n)||p(zd,n|θd)]], (9)
where the first line aims to reconstruct the observed data Gd, and the second line consists of two KL divergence that attempts to close the distance between the posterior and prior of latent variables θd and Zd. Fortunately, KL[q(θd|Gd)|p(θd)] and KL[q(zd,n|Gd, wd,n)||p(zd,n|θd)] are both analytic (see Appendix A.2). Then, we approximate the expectation of the second KL term over θ with Monte Carlo samples by applying the reparameterization trick on q(θd|G0d)2. As for the deduction of the first line, we will discuss in Section 3.2.3.
After the probabilistic definition of the generative story and the corresponding variational family, we turn to introduce the detailed network structure to implement our model. As the overview shown in Figure 2, the neural network structure of GNTM can be broadly split into three parts: 1) the topic representation, including the topic set β and the dependency relation matrix M ; 2) an encoder, which maps the observed data to the variational parameters; 3) a decoder, which computes the possibility of the observed data given the latent variables. We describe each term as below.
2Although, the expectation of the second KL term can also be computed analytically under Dirichlet distribution, in which our model is conjugate, we still use the sampling strategy here, which is more general and can also be applied for more flexible situations without conjugate distributions, such as [4, 53]

heading: 3.2.1 Topic Representation
text: Inspired by [34], we introduce word vectors r ∈ R|V |×H , topic-word vector uv ∈ RK×H , topic-edge vector ue ∈ RK×2H , and generate βvk and βek by:
βvk,w = exp(rw · (uvk)T )∑
w′∈V exp(rw′ · (uvk)T ) , βek,(w1,w2) = exp(r̂w1,w2 · (uek)T )∑ (w,w′)∈E β v k,wβ v k,w′exp(r̂w,w′ · (uek)T ) , (10)
where, r̂w1,w2 = rw1 ⊕ rw2 . (·) and ⊕ denote dot product and concatenation operations, respectively. Therefore, each topic k can be represented by the vector uk = uvk ⊕ uek. Meanwhile, βvk and βek can be explained as the semantic similarity between the topic and words or edges (the concatenation of word vectors as the edge vector). Along this line, we further define each element of the topic dependency matrix M as the semantic similarity among topics with additional learnable parameter matrix W ∈ R3H×2Y :
ak ⊕ bk = uk ·W, mi,j = σ(g(ai · bTj )), ∀ i, j, k, i 6= j, (11)
where σ(·) is the sigmod function. ak, bk ∈ RY are two transitional vectors for the topic k to compute the possibility of directed edges among different topics. g(·) denotes a vector similarity metric and is set as the consine similarity g(a, b) = C(a · b/(||a||||b||)) here, in which C is a hyper-parameter to re-scale the range of σ(g(·)) to approach [0, 1].

heading: 3.2.2 Encoder
text: Here, we aim to model the mapping from the observed data Gd to the variational parameters for latent variables θd and Zd. To be specific, we denote X ∈ R|V |×L as the trained and fixed word embeddings for all words in the vocabulary V . For the document d, Xd ∈ R|Vd|×L is the word embedding set for word set Vd. Then, we encode the local information of each node and the global information of the whole graph Gd with one layer of graph neural network, such as GraphConv [38]:
hd = tanh(GNN(G o d, Xd)),
hGd = tanh( Nd∑ n=1 σ(f1(hd,n ⊕ xd,n)) tanh(f2(hd,n ⊕ xd,n))), (12)
where hd ∈ R|Vd|×L is the node-level representation vectors, hGd ∈ RL is the graph-level representation vector, and f∗(·) is a full connected layer. Self-loops have been added to God for preserving the current word feature. The second project network is following [29].
Then, the parameters of variational marginals q(θd|Gd) and q(zd,n|Gd, wd,n) can be specified as:
µd = fµ(h G d ), log δ 2 d = fδ(h G d ),
ϕd,n = fϕ(h G d ⊕ hd,n ⊕ xd,n).
(13)

heading: 3.2.3 Decoder
text: We denote the reconstruction part in Equation 9 (the first line) as LRd , which can be deduced as: LRd = ∑
(n,n′)∈Eo d
ϕTd,n · (logM) · ϕd,n′ + ∑
(n,n′)/∈Eo d
ϕTd,n · log(1−M) · ϕd,n′
+ Nd∑ n=1 ϕTd,n · log βv·,wn + K∑ k=1 Eq(Zd|Gd)[log
∑ (n,n′)∈Eo
d zkd,nz k d,n′β e k,(wd,n,wd,n′ )∑
(n,n′)∈Eo d zkd,nz k d,n′
],
(14)
where zkd,n = 1 if zd,n = k otherwise 0. Indeed, the first two terms equal to Eq(Zd|Gd)[log p(G o d|Zd)] and the other items are derived from Eq(Zd|Gd)[log p(Vd|Zd, God)] (see Appendix A.2) . In practice, the graph God is often very sparse which causes the unbalance between the first two terms. Here, we follow [26] and re-weight the second terms with |Eod |( ∑ (n,n′)/∈Eod 1)−1.
Now, the only challenge to compute the ELBO loss Ld is at how to compute the last term in Eqaution 14, which is the expectation of a log-sum function and can not be computed analytically. Here, we estimate this expectation with Monte Carlo samples from q(Zd|Gd). Meanwhile, to ensure the differentiability, we use Straight-Through Gumbel-Softmax (STGS) estimator [20] to approximate
the undifferentiable multinomial sampling, which outputs the discrete samples as the approximated multinomial samples in the neural forward pass, but applies a continuous approximation in the backward pass.
We conclude this section with an observation to show that the objective function in LDA can be derived from that of GNTM as a special case. This demonstrates the generalization of our proposed model and the contribution to model the word dependency by representing documents as graphs. The proof can be found in Appendix A.3.
Corollary 1 If the word dependency edge Ed of the document d is one empty set, the objective function defined in Equation 9 would reduce into the ELBO in LDA with the Laplace approximation of the Dirichlet distribution.

heading: 4 Experiments
text: Here, we conduct extensive experiments on four benchmark text datasets to evaluate the effectiveness and interpretability of our model. Our code and data are available at https://github.com/ SmilesDZgk/GNTM.

heading: 4.1 Experimental Setup
text: Datasets: Our experiments are conducted on four benchmark datasets with varying sizes, including 20 News Groups (20NG) [27], Tag My News (TMN) [51], the British National Corpus (BNC) [10], Reuters extracted from the Reuters-21578 dataset. In particular, 20NG and TMN are single-labeled datasets. We follow [16] and only use the title text in TMN to represent the short documents. We also filter out the stop words, and words and dependency edges with low frequency among the whole collection to reduce noise. The statistics and links of these datasets are shown in Appendix A.4.
Evaluation Metrics: Here, we focus on measuring the quality of learned topics. Traditionally, perplexity has been used to measure the quality of topic models, but it has been repeatedly shown that perplexity is not a good metric for the qualitative evaluation of topics [39]. Therefore, we adopt another two metrics: topic coherence (TC) and topic diversity (TD). Given the reference corpus, TC measures the interpretability of a topic by computing the semantic coherence in the most significant words [37]. With the suggestion in [43], we selected two TC measurements to provide a robust evaluation, i.e, Normalized Pointwise Mutual Information (NPMI) [1] and Cv [43]. Specifically, both NPMI and Cv are computed on the top words of each topic with the original document corpus of each dataset as the reference documents. Higher value indicates better interpretability. TD aims to measure how diverse the discovered topics are. We follow [11] and defined TD as the percentage of unique words in the top words. TD closes to 0 indicates redundant topics. TD closes to 1 indicates more varied topics. In addition, we also evaluate our model in a document clustering task on labeled datasets, i.e., 20NG and TMN, to evaluate the quality of the learned topic propitiation. Specifically, following [40], we assign every document the topic with the highest probability as the clustering label and compute Purity and Normalized Mutual Information (NMI) [45] as metrics. Both of them always range from 0.0 to 1.0, and higher scores reflect better clustering performance.
Benchmark Models: We select several state-of-the art-topic model as the baselines, including: 1) LDA [5]; 2) GSM [34], an NTM model which replaces the Dirichlet-Multinomial parametrization in LDA with Gaussian Softmax; 3) ProdLDA [49], an NTM model which keeps the DirichletMultinomial parametrization with a Laplace approximation; 4) ETM [11], an NTM mdoel which incorporates word embedding to model topics; 5) GraphBTM [61], an NTM model with graph-based word dependency; 6) iDocNADE and iDocNADEe [16], neural autoregressive topic models with sentence-based word dependency, where iDocNADEe incorporates word embeddings. Note that iDocNADEe constrains the equality of the dimension of word embedding and the number of topics. We can only set the number of topics as 50 and 100, due to the lack of pretraining word-embedding with other dimensions in public. 7)TopicRNN [12], a neural autoregressive topic model with word sequence-based dependency. All baselines are implemented carefully with the guidance of their official code.
Experiment Settings: In practice, we construct document graphs with window size s = 53. In particular, we mix together with the same words in different positions as one node with word frequency to compute the correct likelihood. In other words, we implicitly assume that the same words in one document share the topic assignment, which is widely used in [5, 34, 42, 54]. We set the Dirichlet prior parameter α = 1. Following [49], batch-normalization [18] has been used on the inference network for the posterior parameters of θ. We utilized 300-dimensional GloVe word embeddings [41] to fix X (i.e., L = 300) in our model and word vectors in ETM. We also set the size of word vectors µv as 300, i.e., H = 300. The size of transitional vector ak and bk were set as Y = 64. For the optimization, We follow [34] and alternately update the decoder parameters with topic representation and the encoder parameters. Only one sample is used in neural variational inference for θd and Zd if needed. For the optimization, Adam [23] optimizer has been used with the initial learning rate of 0.001 and the linear learning rate decay trick to find the optimal. All quantitative results are based on 5 runs with different random seeds and the average performance is reported without special description. (More experimental details can be found in Appendix A.5.)

heading: 4.2 Experimental Results
text: We follow [11] and select the top 10 words with the highest probability in each topic as the representative word set to compute both TC and TD metrics. The results are displayed in Figure 3 with the varying number of topics K from 20 to 100 (see Appendix A.7 for more results based on the different number of top words). We find that our proposed model outperforms other baselines in terms of both Cv and NPMI score with high topic diversity on all datasets. In particular, GSM also achieves the competitive performance on TC metrics, but its TD score is lower than most models. It may be because that, in contrast to Dirichlet prior used in other models, the Gaussian-Softmax distribution in GSM leads to lower sparsity among topic proportions, which causes redundant topics in turn. Meanwhile, although some other baselines may have higher TD score than our model on several datasets with several topic numbers, they cannot achieve high topic coherence. In addition, despite
3When s = 5, the degree of one word is 10, i.e., 5 in-degrees and 5 out-degrees, which is similar to that in Skip-gram with the size of text window as 5 that performs better in empirical experiments [36]. We show the sensitivity analysis of the windows size s in Appendix A.5.
considering word dependency, GraphBTM fails to capture the state-of-the-art performance. The reason may be that it construct each document graph on the whole vocabulary and include words that are not in the document as noise. In terms of document clustering with results in Table 1, our model performs the best with a significant gap over other models on Purity metric, while GNTM is also comparative on NMI metric. Those results demonstrates that our model captures both interpretable topics with better quality and good document representations for clustering. We also note that the GraphBTM fails to cluster documents. Especially, the NMI metric has vanished to 0 on the TMN dataset, which indicates that all documents were clustered to one category and GraphBTM cannot capture significant topics with similar semantics to the document labels.
Visualization: Since GNTM models topics as multinomial distributions on both the vocabulary and the word dependency edge sets, we can represent each topic k as a graph on the vocabulary V with the edge set E and adjacency matrix Ak = (βvk · (βvk)T ) βek. We show two topics learned by a run of GNTM on News20 with K = 20 as examples in the left of Figure 4. First, we can find that the top words in each topic are highly semantically consistent, where topic #12 is about space science and topic #13 is related to computer technology. Second, we observe that the top edges are also highly interpretable with semantic dependency between two word nodes, such as “space station” and “satellite orbit” in topic #12, or “floppy dives” and “floppy disk’ in topic #13, which also helps us to reduce the possible ambiguity and understand the specific meaning of words in each specific topic. Third, the graph representation of topics also provides a new possibility to further separate the topic semantic by clustering the word nodes. For example, we utilize the fast unfolding algorithm [6], a widely-used community detection algorithm, to cluster nodes and color different categories. We find that, interestingly, the top words of topic #12 are split into three sub-clusters with different sub-topics related to space research (violet), celestial bodies (green), and space exploration (orange), respectively. Similarly, the top words of the topic #13 are separated into two categories: one is about software or hardware (orange); the other may be about programming development (violet).
In addition, we also show the topic dependency matrixM in the right of Figure 4, where each element mi,j represents the possibility of existing a directed edge from topic i to j. We have the following interesting observations. First, there always exist high possibility values between topic #0 with any other topics. It is actually reasonable because topic #0 is represented with several general words, commonly used in any scenario. Second, the diagonal tend to have a high value. It indicates that the words with the same topic should be linked, which is consistent with the widely-used assumption that interdependent (or neighboring) words prefer to have the same topic assignment [15, 30, 32, 59, 60]. Third, we note that the topic dependency matrix also mines the dependency relation between different topics. For example, in our topic sets, the top words of the topic #9 is full of sport-related words, while the topic #8 focuses on the hockey games with keywords related to team or game name, such as “bos” (Boston Bruins), “det” (Detroit Red Wings) and “nhl” (National Hockey League). Considering the strong relation between topic #8 and #9, the value m8,9 and m9,8 is high.

heading: 5 Conclusion
text: In this paper, we revisited the topic modeling techniques by representing documents as semantic graphs and proposed a Graph Neural Topic Model (GNTM) method. Specifically, we first formulated a well-defined probabilistic generative story to model both the graph structure and word sets with a new concept of topics, i.e., multinomial distributions on both the vocabulary and word dependency edge. Then, a Neural Variational Inference (NVI) approach was proposed to learn our model with graph neural networks to encode the document graphs. Besides, we also have demonstrated that LDA can be derived from GNTM as a special case with similar objective functions. Finally, extensive experiments on four benchmark datasets have clearly validated the effectiveness and interpretability of GNTM compared with state-of-the-art baselines.
Limitation and Future Work: Here, we discover word dependency only by linking words in a small sliding window on word sequence. Several other approaches can also construct document graphs, such as dependency parse [33]. An interesting direction of future work is to consider multiple types of word dependency relation together by representing documents as multi-relational graphs.

heading: 6 Acknowledgement
text: This research was supported by grants from the National Natural Science Foundation of China (Grant No. 91746301 and 61836013).

Review summaries:
In this paper, the authors provide a novel generative model to learn topics using graph modeling based on the word network, following a (now classic) variational encoding approach. The documents are encoded as graphs by modeling an edge between two words if they appear in the same window, which is similar to the embedding methods with the distributional hypothesis. The graph structure is then part of the generative process. They demonstrate that the well-known LDA model is a special case of their approach if we ignore edges. Most of the time, the model outperforms existing topic models (classic such as LDA and neural TM) on 4 English written datasets in term of Cv, NPMI and Topic Diversity. It also outperforms prior works in term of clustering accuracy.  

This work is based on neural topic modeling where a document is represented by a graph of word notes with edges representing the semantic dependency and the generative process of learning is a function of both the document-graph structure (i.e., word-dependency edges) as well as word sets (i.e., document vocabulary). On top, the AVI approach is introduced to perform learning over the proposed Graph Neural Topic Model (GNTM).   Several past works in topic modeling have investigated dependency relations among words: modeling sequential dependency and syntactic relationships [1] using dependency parse trees. Another thread of research [2, 3] incorporates both: BoW and contextualized information in the generative process of topic modeling, where the contextualized representations introduced modeling word-dependency relations in the generative process of neural topic models. This line of research is missing in quantitative comparison with the proposed topic model. 

Going beyond the bag-of-word assumption in topic modelling is an interesting problem, which has been studied since the introduction of the vanilla LDA, for instance, the syntactic topic model. This paper proposes to represent each document as a directed word graph, which is constructed via moving a window along the word sequence. The main technical contributions of this paper can be 1) the graph neural topic model, where words are generated conditioned on both the topic assignments and the reconstructed word graph; 2) the VAE framework used to learn the posteriors. 

The authors propose a topic model that explicitly captures the semantic dependency among words in the same document, by representing each word as a node in a graph and connecting words appearing closely in the document through an edge in the graph. This approach overcome the limitation of standard topic models as they can only consider word as independent. in this way, inferred topics may be more interpretable and semantically more meaningful. This is validated in the experiments as the proposed model achieves better topic coherence.