Topic Modeling Revisited: A Document Graph-based Neural Network Perspective
Authors: Dazhong Shen
Chuan Qin
Chao Wang
Zheng Dong
Hengshu Zhu
Hui Xiong
Venue: NeurIPS 2021 Poster (2021)
Keywords: Topic Model
Neural Variation Inference
Graph Neural Network

Abstract: Most topic modeling approaches are based on the bag-of-words assumption, where each word is required to be conditionally independent in the same document. As a result, both of the generative story and the topic formulation have totally ignored the semantic dependency among words, which is important for improving the semantic comprehension and model interpretability. To this end, in this paper, we revisit the task of topic modeling by transforming each document into a directed graph with word dependency as edges between word nodes, and develop a novel approach, namely Graph Neural Topic Model (GNTM). Specifically, in GNTM, a well-defined probabilistic generative story is designed to model both the graph structure and word sets with multinomial distributions on the vocabulary and word dependency edge set as the topics. Meanwhile, a Neural Variational Inference (NVI) approach is proposed to learn our model with graph neural networks to encode the document graphs. Besides, we theoretically demonstrate that Latent Dirichlet Allocation (LDA) can be derived from GNTM as a special case with similar objective functions. Finally, extensive experiments on four benchmark datasets have clearly demonstrated the effectiveness and interpretability of GNTM compared with state-of-the-art baselines.

Topic Modeling Revisited: A Document Graph-based Neural Network Perspective
Dazhong Shen 1.2 Chuan Qin', Chao Wang 1.2 Zheng Dong", Hengshu Zhu Hui Xiong
School of Computer Science and Technology, University of Science and Technology of China Baidu Talent Intelligence Center, Baidu Inc.
Artificial Intelligence Thrust, The Hong Kong University of Science and Technology sdz @ mail.ustc. www.luuut.cc...om chuanqin0426e wawt:/mmaloooooc wdyx2012 @ @mail. mail.ustc........ www..uudu..cc..... zhdong0@ outlook. com, zhuhengshu gmail.com, xionghui @ @ust.kk
Abstract
Most topic modeling approaches are based on the bag-of- of--odf-ordss assumption, where each word is required be conditionally independent in the same document. As result, both of the generative story and the topie formulation have totally ignored the semantic dependency among words, which is important for improving the semantic comprehension and model interpretability. To this end, in this paper, we revisit the task of topic modeling by transforming each document into directed graph with word dependency as edges between word nodes, and develop a novel approach, namely Graph Neural Topic Model (GNTM). Specifically, GNTM, well-defined probabilistic generative story is designed to model both the graph structure and word sets with multinomial distributions on the vocabulary and word dependency edge set as the topics. Mcanwhile, Neural Variational Inference (NVI) approach is proposed to learn our model with graph neural networks encode the document graphs. Besides, we theoretically demonstrate that Latent Dirichlet Allocation (LDA) can be derived from GNTM as special case with similar objective functions. Finally, extensive experiments on four benchmark datasets have clearly demonstrated the effectiveness and interpretability GNTM compared with state-of-the-an baselines.
1 Introduction
As one of the most widely-used techniques for document analysis, topic modeling aims to learn set of latent topics from the observed document collection, each of which describes an interpretable semantic concept. In particular, Latent Dirichlet Allocation (LDA) (5) and its extensions [ 44 60)
have achieved great success various application scenarios over the past decades. These approaches usually specify probabilistic generative model draw the document data with structure of latent variables sampled from prior distributions and word distribution on the vocabulary as topics. Recently, with the development of Variational Autoencoder (VAE) [24), Neural Variational Inference algorithm (NVI) with neural networks for topic modeling, namely Neural Topic Model, has attracted due its appealing flexibility and scalability.
great attention 35 34
12
42
49 61
However, well-known that traditional topic modeling usually has the bag-of-word assumption, where the words in document are assumed to be "exchangeable" which also arouses two typical challenges behind most topic models. On one hand, given the topic distribution the document, the words are conditionally independent. This implicates that the dependency relationship among words totally ignored, while words are dependent on each other according to linguistic knowledge [30].
**orresponddin authors. This work was accomplished when Dazhong Shen was intern in Baidu supervised by Chuan Qin and Hengshu Zhu.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

On the other hand, the concept of topics is introduced as multinomial distributions on vocabulary without modeling the dependency relation among words, which produces more ambiguous terms in topics. For example, LDA discovers word such as "network" in a topic which does not seem to be that insightful. Instead, we can convey more interpretability to readers by mining the strong word dependency between "neural" and "network" [19].
Along this line, many extensions have been proposed address the above two issues by dealing with the dependency relation among words. One thread of these works focus on the sequential dependency
by taking the document as a sequence of words I2 most of them prefer to enhance language modeling with better performance on language modeling metric, such as Perplexity, rather than the sfate-of-the-an quality of learned topics. may be due to the limitation that they only handle the linear dependency of words [30] and push models to capture the local syntax information instead of global semantic information. Specifically, to mitigate the first problem, most of them assume that the topic assignment or generative possibility of a word
15 I6
56 52
55
60]. However,
As for the second problem, related literature is restricted to exploring n-gram terms by integrating few neighboring words as one term
dependent on the proceeding words [2 1.5 16
28
42
56
with Markov Chain based generative story [52 55 32).
Nonetheless, words may be mutually dependent on each other in a much more complex and non- linear manner. Therefore, several studies explore model the graph representation of documents, where nodes denote words and edges represent the relationships among words, such as syntax or semantic relation constructed by dependency parsing 30 (44] or justified by relative position among words [9 61. They either consider the graph structure as side information to constrain the relation of topic assignments of words [30 (4), or replacing the word sets with edge sets as the instances to generative (9 [1]. Most of them only trickle with the first problem without efforts to enhance the
interpretability of topics.
To this end, in this paper, we represent documents as directed semantic graphs by introducing word dependency as edges between word nodes, and develop Graph Neural Topic Modeling (GNTM) method. The key contributions of our model can be categorized into four parts: 1) we formulate a well- defined probabilistic generative story of the document graph with the novel generative probabilistic functions for both graph structure and word sets; 2) we propose a new concept of topics, consisting of multinomial distributions on both the vocabulary, like conventional topic models, and word dependency edges; 3) a Neural Variational Inference (NVI) approach is designed to infer our model with graph neural network to encode the document graphs; 4) we also demonstrate that LDA can be derived from GNTM as special case with similar objective functions. Finally, extensive experiments over four benchmark datasets have clearly demonstrated the effectiveness and interpretability of GNTM compared with state-of-the-art baselines.
2
Preliminary
2.1 Topic Model
Topic modeling is one of the most important text mining techniques and has been extensively studied for variety of applications in recent two decades (2 21 31
48 47 Among the most representative models, Latent Dirichlet Allocation (LDA) [5] formulates each topic & as the distribution 3y on vocabulary V and draws each word Wd,n in a document d from one of K topics stochastically. The well-known generative process of LDA can be described as three steps: 1) sampling document-topic proportions f₂ ~ Dir(a)), where a is hype sepe-parameter for the Dirichlet distribution prior; 2) drawing individual topic assignments 2d,n ~ Multi(f.); 3) generating each word Wd,n Multi( Then, the marginal likelihood for document d is:
p(dla, B) = p(Oala) Ua
II p(ua.ll" )((zd, 1a))da. =d.n
Conventional approaches for inferring topic models, such as LDA, are either Markov Chain Monte Carlo 7 or probabilistic variational inference [22], which are both widely applied [58] 14 46).
However, they either require careful selection of distribution for each latent variable or tolerate arduous and customized mathematical derivation case by case, which limits the flexibility and
scalability of model design (35). Recently, with the successes of Variational Autoencoder (VAE) 24

---

artificial network
group
Go
M
neurons
Oa
neural - interconnected computational
B
artiicaal neuni network is en eteccoraceed group of computationll neurors.
(a) Document graph example with = 2
(b) Directed probabilistic graph
Figure 1: The document graph example and the directed probabilistic graph of our model.
which applies the variational distributions parameterized by neural networks to approximate the posterior of latent variables, the use deep learning on inferring topic models has attracted more and more attention [57]. Most of those neural topic models (NTMs) [35] 34 49 [1 can be regarded as extensions of LDA, but often integrate out the discrete latent variable 2d,n- Therefore, the variational lower bound (ELBO) for the log- ligleiill of the document d can be derived as:
L (ola)
log p( B")] - Dx (x..(
where 9 (0)d) is the variational distribution of 0g parametrized by neural network c. To obtain differentiable estimator of La. the reparametcrization trick (RT) has been applied for 0. In particular, due to the limitation of RT on Dirichlet distribution [49], those approaches usually stand on Gaussian- based posterior g (ogld))
2.2 Document Graph Construction
Most existing methods in topic models commonly represent document as unordered collection of words, known as bag-of-word (BOW) representation, which totally ignores the relationship among words in the document. To represent documents with word dependency relations, we discuss the representation documents a graph manner, where the graph-based models has been widely explored and applied recently extensively [25 50 [3].
Formally, given one document d with word sequences [wa,e]=-1 No we construct directed edge from wa.t, Wa,t, if < t2 t1 S s, where the typer-parameter of window size. The hidden logic is that the semantic is consistent and interdependent among neighboring words, which can be traced back to n-gram words are filtered), we can denote document d as a graph G. = (Va, Ea), where Va {wamn Na (words as nodes, e.g., 7 words in Figure [(a) and Ed are sets of nodes and word edges (e.g.. ("artificial", "neural")); respectively. Note that, for simplification
[8] BTM and Word2vec (36]. As the example in Figure [[a))shww (stop
analysis, we distinguish the same words on different positions in Eg and Vaa We also denote E as the unique word edge sets in
the whole document collection with nodes in the vocabulary V {w; | (unique word set). For convention, we further denote Go = (VP. Eg] as the graph structure of G with only the placeholders
Vi = 21, 2, ..., Na} (e.g., {1, 2, ... 7) Figure I(a).. not the specific words Va, where [V/] = [Val.
and the placeholder edge set E% is the edge set on the placeholder set Vi (e.g., (1, distinguishing from the word edge set Ea..
3 Graph Neural Topic Model
With the definition of the document graph Ga. we turn to introduce the technical detail of GNTM. Specifically, we first formulate the generative story for both the graph structure G2 and word sets Va- Then, an NVI inference algorithm will be proposed learn our model.
3.1 Generative Story
Figure [[b]]sow shows our directed graphical model. In contrast three- step generative process LDA, four steps are involved in the generative story. The main differences lie in the conditional generation for the observed data given all latent variables (steps and in the following). the generative story of a document d begins with topic proportion @ drawn from the Dirichlet prior Dir(a). Then, 2) each word w,, (or placeholder n) has corresponding topic assignment 2n drawn from Multi(@). Finally generate the observed data with two steps: 3) drawing the graph structure
be specific, 1)

---

G2 conditioning on the topic assignment set Za. i.e., G2 p(G 4) generating the word set Va based on both topic assignment set Za and graph structure Gā. i.e. Va p(V, Za). Based on the generative process, the joint model of Ga and all latent variables can be decomposed as:
Na I
p(Ga, 02. aa;a) = pl Za. G.) G2
(3)
In the following, we turn to formulate the formal definition of PG2 and p pValZaa G2) with additional parameters, i.e., topic dependency matrix M and topic 3, which is the most important contribution of our model.
3.1.1 Generation for Graph Structure
Here, we aim to construct the graph structure G2 based solely on the topic assignment 2d,n on each placeholder wa,,, The hidden motivation is to explain the word dependency as the semantic dependency among their topics, which can o our model to find the relation among different topics in turn. To be specific, we denote A E I x the dependency matrix among topics, where each element m E [0. 1] represents the possibility that there exists a dependency edge between two formulate p(G²) Za) as
placeholders with topic i and j. respectively. Then,
Za
Za: M)
II (m,n")EPP P
II
(4)
(m,n Rg
where M is a learnable parameter without additional assumptions.
3.1.2 Generation for Word Set
Contrast to LDA, once document is represented graph, the word sampling should be cast as the sampling of this graph. In other words, we should consider dependency between words when generating word set Va. To provide well-defined word sampling process, we first re-define the concept of topic. Different from the conventional concept of topic in LDA, which is represented by the multinomial distribution 3% the vocabulary, further enhance each topic by introducing one distribution on word dependency edge with parameters S6. i.., By = (0). (8), where:
Σ 1. weV
Σ Serno% (i,(m,,") PT. 0, (m, 2 O. MN
(5)
3% where the product 3% 5 w 3 (w,,w)) denotes the possibility of the directed edge from word w to w' under the topic k. Note that, in practice, only parameters 2 (k,(w,,rr) corresponding to edge (w. (w.w") E E would be used model learning. In other words, we can reduce the space complexity of 38 into O(E easily by assuming that 3 "k = 0, V(w, w') & E.
Then, based on the new concept of the topics, we propose a well-defined probability measure for generating each word set Va with the document graph structure G2 and topic assignment sets Zz- The probability of pVVal a Za. Gg) is modeled by:
P(Val Za, G3 B)
II PVVa..la,,
(6)
P(V.. G3,):
Σ (k
V
(w.
where G2 is the graph structure consisting of all placeholders with the topic assignment k. similarly, Va, is the set of words assigned by topic k and Ea,t is the word dependency edge among them. In other words, we split the document graph Ga into K dependent sub-graphs Ga based on topic assignment set Zu and then define the generative possibility of each topical sub- -ob-graph Ga.k 3n with the word dependency edges E, If E, = O, we set E(u,w")E Pk.(w, |E = 1. Mathematically, we have the following theorem:
Theorem Given the topic B in Equation and the document graph structure Go2 the probability function PVVal Za, Go (3) in Equation is legal probability measure on the vocabulary V.
The proof can be found in Appendix A.l which is mainly inspired by [30].

---

Topic Dependency
Local Info
fo
92
NND
Word Emizedding
Word Distribution Param
Ruy
Viv
Document Graph
Generate Word Set
Info
Logistic Normal
Edge Distribution Param
Encoder
Decoder
Topic Rep
Figure
The overivew of network structures infer our model.
3.2 NVI Inference
Here, we infer our model by applying a NVI inference algorithm, which is well-know of flexibility and scalability [24). Based on the mean field assumption, we approximate the posterior of latent variables 0g and Za for each document d with the following variational family:
Na
(00, (0, pa. 8a) II (a(za,n|aa wans pp.,),
where do (2d.n (Zddn]/Ce Ga, wa,n) is defined as one multinomial distribution with parameters Pd,n and Gaa) is the approximation for the true Dirichlet posterior pa)) Both of them pa- rameterized by neural networks Φ. However, directly approaching the Dirichlet distributions would make the reparameterization trick hard to apply. Here, follow 49 17 and resolve this issue by constructing a Laplace approximation to the Dirichlet posterior with the Logistic Normal distribution parameterized by (Ha, 84), that
softmax(N((aa 03))-
(8)
Then, can derive the training objective for each document d, i.e., ELBO:
La Ey(2a C2) Poog M) log G2; 8)]
Na
(9)
-KL1(O0[G..)l1(0... E LL(g(aam Ga w.,, ) (0)]]]
where the first line aims to reconstruct the observed data G.. and the second line consists two KL divergence that attempts to close the distance between the posterior and prior of latent variables 0g and Za. Fortunately, K L[g(O. (aa)pp((O)) and K L[q(z, Ga, Wa.a }pppppppppppppp z...n [0a] are both analytic (see Appendix A.2). Then, we approximate the expectation of the second KL term over 0 with Monte Carlo samples by applying the reparameterization trick on q(a G2 As for the deduction of the first line, we will discuss Section 3.2.3
After the probabilistic definition of the generative story and the corresponding variational family, we turn introduce the detailed network structure implement model. As the overview shown in Figure [ the neural network structure of GNTM can be broadly split into three parts: 1) the topic representation, including the topic set 3 and the dependency relation matrix M; 2) an encoder, which maps the observed data to the variational parameters: 3) a decoder, which computes the possibility of the observed data given the latent variables. We describe each term below.
Although, the expectation of the second KL term can also be computed analytically under Dirichlet distribution, in which our model is conjugate, we still use the sampling strategy here, which is more general and also be applied for more flexible situations without conjugate distributions, such [3
5

---

3.2.1 Topic Representation
we introduce word vectors E IR V topic-word vector ut" E RRKHH tooic-edgee vector u² Rx22 and generate 3% and 3g by:
Inspired by 34
exp(ru (u)) Kk. Dw'ev exp (ug)T) Sk (222)
cappfun,.uz (u))²)
(10)
E (wwww)) 3 k. W. expplee, (U))T)
where, fwr, = Twn, ① Twy (.) and @ denote dot product and concatenation operations, respectively.
Therefore, each topic k can be represented by the vector u4 = = @ ug. Meanwhile, gg and 8 can be explained as the semantic similarity between the topic and words or edges (the concatenation of word vectors the edge vector). Along this line, we further define each element of the topic dependency matrix M as the semantic similarity among topics with additional Icarnable parameter matrix W E 13
a bh W. ma. = a b))), Vijjk, i
(11)
where o(-) is the sigmod function. ag. by E R are two transitional vectors for the topic & compute the possibility of directed edges among different topics. g(-) denotes a vector similarity metric and is set as the consine similarity g(a,b) = C(a b/[[aa]]]b|]) here, in which C a hyper-parameter to re-scale the range of (gg() to approach [o, 1].
3.2.2 Encoder
Here, we aim model the mapping from the observed data Ga to the variational parameters for latent variables 0₂ and Za- To be specific, we denote X R f. the trained and fixed word embeddings for all words in the vocabulary V. For the document d, Xa R is the word embedding set for word set Va. Then, we encode the local information of each node and the global information of the whole graph Ga with one layer of graph neural network, such GraphConv [38]:
ha
tanh Xa)),
No
(12)
h = tanl Σ
(fii (ha, @ xa,))) @ tanhi/aa(aa 2 ra,"))),
where ha E RIVa(xL & is the node-level representation vectors, l E R2 is the graph-level representa- tion vector, and (-) is a full connected layer. Self-loops have been added to Go for preserving the current word feature. The second project network is following (29].
Then, the parameters of variational marginals (a) and g(za,n [aa, wan) can be specified as: Pa = fa (h log 53- = fa(h
(13)
Pd,n = (h + ha. t xam)).
3.2.3 Decoder
We denote the reconstruction part in Equation (the first line) as L which can be deduced as:
R
Σ
pd (log M) Pa.n'
Σ
o log(1 M) "dd..
(n
(n.""
(14)
Pd,n log Sr
Bo( llog
Σ E(nnoyegg 24, 2/,n'
n=1
where =/,, = if 2d,n = k otherwise O. Indeed, the first two terms equal to (g(z. log (Ogg Za)] and the other items are derived from Eq(2./Ga) [log p( Za, C%) (see Appendix A.2). In practice, the graph G is often very sparse which causes the unbalance between the first two terms. Here, we follow 26 and re-weight the second terms with EEgl (E ((m,,")) 1)
Now, the only challenge to compute the ELBO loss L. at how to compute the last term in Eqaution [4] which is the expectation of a log-sum function and can not be computed analytically. Here, we estimate this expectation with Monte Carlo samples from (Za(Ga). Meanwhile, to ensure the differentiability, we use Thrroghh Gumbel-Softmax (STGS) estimator [20] to approximate

---

the undifferentiable multinomial sampling, which outputs the discrete samples as the approximated multinomial samples in the neural forward pass, but applies a continuous approximation in the backward pass.
We conclude this section with observation to show that the objective function in LDA can be derived from that of GNTM as special case. This demonstrates the generalization of our proposed model and the contribution model the word dependency by representing documents as graphs. The proof can be found in Appendix A.3.
Corollary If the word dependency edge Ed of the document d is one empty set, the objective function defined in Equation would reduce into the ELBO in LDA with the Laplace approximation of the Dirichlet distribution.
4 Experiments
Here, we conduct extensive experiments on four benchmark text datasets to evaluate the effectiveness and interpretability of our model. Our code and data are available at bttps://githhb. com/ SmilesDZgk/GNTT4
4.1 Experimental Setup
Datasets:
Our experiments conducted on four benchmark datasets with varying sizes, including 20 News Groups (20N)) (27). Tag My News (TMN) [5T]. the British National Corpus (BNC) [00] Reuters extracted from the Reuters-21578 dataset. In particular, 20NG and TMN are single-labeled datasets. We follow [16] and only use the title text in TMN to represent the short documents. We also filter out the stop words, and words and dependency edges with low frequency among the whole collection to reduce noise.
The statistics and links of these datasets are shown in Appendix A.4.
Evaluation Metries:
Here, we focus on measuring the quality of learned topics. Traditionally, perplexity has been used to measure the quality of topic models, but has been repeatedly shown adopt
that perplexity is not good metric for the qualitative evaluation of topics [9]. Therefore, another two metrics: topic coherence (TC) and topic diversity (TD). Given the reference corpus, TC measures the interpretability of a topic by computing the semantic coherence in the most significant words (37] With the suggestion in [43. we selected two TC measurements to provide a robust evaluation, i.e, Normalized Pointwise Mutual Information (NPMI) [1 and C, 4 Specifically, both NPMI and C. are computed on the top words of each topic with the original document corpus of each dataset as the reference documents. Higher value indicates better interpretability. TD aims to measure how diverse the discovered topics are. We follow (II) and defined TD as the percentage of unique words in the top words. TD closes to 0 indicates redundant topics. TD closes indicates more varied topics. In addition, we also evaluate our model in a document clustering task on labeled datasets, i.e., 20NG and TMN, to evaluate the quality of the learned topic propitiation. Specifically, following [40]. we assign every document the topic with the highest probability as the clustering label and compute Purity and Normalized Mutual Information (NMI) 45 as metrics. Both of them always range from 0.0 to 1.0, and higher scores reflect better clustering performance.
Benchmark Models: We select several state-of-the art-topic model as the baselines, including: 1) LDA [5]: 2) GSM (34], an NTM model which replaces the Dirichlet-Multinoiial parametrization in LDA with Gaussian Softmax: 3) ProdLDA [49]. an NTM model which keeps the Dirichlet- Multinomial parametrization with Laplace approximation; 4) ETM (I]. an NTM mdoel which incorporates word embedding to model topics: 5) GraphBTM [T], an NTM model with graph-based word dependency; 6) iDocNADE and iDocNADEe [6), neural autoregressive topic models with cistence-based word dependency, where iDocNADEe incorporates word embeddings. Note that iDocNADEe constrains the equality of the dimension of word embedding and the number of topics. We can only set the number of topics as 50 and 100, due to the lack of pretraining word-embedding with other dimensions in public. 7)TopicRNN 122 a neural autoregressive topic model with word sequence- saguee dependency. All baselines are implemented carefully with the guidance of their official code.
7

---

GNTM
104
GRM
GrspriTT
310
0.30
-010
3.044
-028
0.40
0.22
Nannal
Num
Tarm
0.70
0.0
0331
0.9
6
0,90
6023
042
6422
Eopess
lsprss
Tarn ol BOOK
04
044
Mamuul
Num
Thrm
Num
(3) 2ONG
(b) TMN
(0) BNC
(00 Reuters
Figure 3: The performance on the topic quality. The first two rows show the topic coherence score, i.e. NPMI and Cr, with corresponding topic diversity in the third row, where error bars in each point denote the standard deviation of results on 5 runs. The result of iDeNADE, iDocNADEe, and TopicRNN has been omitted due to the uncompetitiveness. Completed results can be found in Appendix A.7.
Experiment Settings: In practice, we construct document graphs with window size particular, we mix together with the same words in different positions frequency to compute the correct likelihood. In other words, we implicitly assume that the same words in one document share the topic assignment, which is widely used in 55 34 544 We set the Dirichlet prior parameter a = 1. Following 4 batch-normalizatioo 18 has been used on the inference network for the posterior parameters of f. We utilized 300-dimensional Glo Ve word embeddings [41] to fix X (i.c., L 300) in our model and word vectors in ETM. We also set the size of word vectors n² as 300, i.e., H 300. The size of transitional vector aş and by were set as Y = 64. For the optimization, We follow 34 and alternately update the decoder parameters with topic representation and the encoder parameters. Only one sample is used in neural variational inference for & and Za if needed. For the optimization, Adam [23] optimizer has been used with the initial learning rate of 0.001 and the linear learning rate decay trick to find the optimal. All quantitative results are based on 5 runs with different random seeds and the average performance is reported without special description. (More experimental details can be found in Appendix A.5.)
In node with word
4.2 Experimental Results
We follow [ and select the top 10 words with the highest probability in each topic as the represen- tative word set compute both TC and TD metries. The results are displayed in Figure 3 with the varying number of topics K from 20 to 100 (see Appendix A.7 for more results based on the different number of top words). We find that our proposed model outperforms other baselines in terms of both C. and NPMI score with high topic diversity on all datasets. In particular, GSM also achieves the competitive performance on TC metrics, but its TD score lower than most models. It may be because that, in contrast to Dirichlet prior used in other models, the Gaussian-Softmar distribution in GSM leads to lower sparsity among topic proportions, which causes redundant topics turn. Meanwhile, although some other baselines may have higher TD score than our model on several datasets with several topic numbers, they cannot achieve high topic coherence. In addition, despite
'When s = 5. the degree of one word is 10, i.e., 5 in in-degrees and 5 out-degrees, which is similar to that in Skip-gre gram with the size of text window as that performs better in empirical experiments (36). We show the sensitivity analysis of the windows size in Appendix A.5.

---

Table 1: The performance on document cluster. We highlight the best and second scores in boldface and with an underline, respectively. The standard deviation can be found in Appendix A.7.
Purity
NMI
20
30
50
70
100
20
30
50
70
100
LDA
0.2980
0.3340
0.3375
0.3510
0.3740
0.2908
0.3013
0.2852
0.2878
0.2858
GSM
0.4133
0.4379
0,4629
0,4429
0.4210
0.4394
0.4369
0.4433
0,4449
0,4412
ProdLDA
0.3306
0,3450
0,3641
0,3638
0,3807
0.3405
0.3345
0.3350
0.3298
0.3343
20NG
ETM
0.3496
0.4154
0.4380
0.4510
0.4616
0.3842
0.4227
0.4296
0.4297
0.4356
GraphBTM
0.1448
0.1210
0.1630
0.1068
0.0992
0.1552
0.1108
0.1807
0.0816
0.0707
iDocNADE
0.2175
0.2844
0,3064
0,3187
0,3371
0.1128
0.1723
0.1802
0.1901
0.1990
IDocNADEe
0.1300
0.1507
0.1185
0.1228
TopicRNN
0.0728
0.0761
0,0865
0.0920
0.1001
0.0109
0.0141
0.0229
0.0303
0.0400
GNTM
0,4500
0,4882
0.5089
0.5090
0.5021
0.443
0.4419
0.4416
0.4362
0.4371
LDA
0.3509
0.3692
0.3725
0.4031
0.4228
0.0622
0.0665
0.0754
0.0901
O. 0.1064
GSM
0.5933
0.6054
0.6184
0.5934
0.2632
0.2848
0.2787
0.2996
0.3246
0.0204
ProdLDA
0.3141
0,2808
0.2505
0.2535
0.2438
0.0508
0.0334
0.0056
0.0053
0.0000
TMN
ETM
0,5841
0,5967
0.6347
0.6358
0.6420
0.2764
0.2705
0.2829
0.2784
0.2767
GraphBTM
0.2438
0.2438
0.2438
0.2438
0.2438
0,0000
0.0000
0,0000
0.0000
0,0000
iDocNADE
0.2712
0.3116
0.3314
0.3512
0,3730
0,1074
0.1436
0.1488
0.1585
0.1701
iDocNADEe
0,2623
0.4171
0.0332
0.1728
TopieRNN
0,2493
0.2537
0,2586
0,2684
0,2775
0.0048
0.0071
0.0108
0.0154
0.0221
GNTM
0.6150
0.6252
0.6472
0.6656
0.6773
0,2851
0.2801
0.2789
0.2821
0,2870
considering word dependency, GraphBTM fails to capture the state-of-the- performance. The reason may be that it construct each document graph the whole vocabulary and include words that are not in the document as noise. In terms of document clustering with results in Table [ our model performs the best with significant gap over other models on Purity metric, while GNTM is also comparative on NMI metric. Those results demonstrates that our model captures both interpretable topics with better quality and good document representations for clustering. We also note that the GraphBTM fails to cluster documents. Especially, the NMI metric has vanished to O on the TMN dataset, which indicates that all documents were clustered one category and GraphBTM cannot capture significant topics with similar semantics to the document labels.
Since GNTM models topics multinomial distributions on both the vocabulary and the word dependency edge sets, we can represent each topic A as a graph on the vocabulary V with the edge set E and adjacency matrix A&- = (9g (Bg)T) B8. We show two topics learned by of GNTM on News20 with K = 20 as examples in the left of Figure 4 First, we can find that the top words in each topic highly semantically consistent, where topic # 12 is about space science and topic # 13 is related to computer technology. Second, we observe that the top edges are also highly interpretable with semantic dependency between two word nodes, such as "space station" and "satellite orbit" in topic #12, or "floppy dives" and "floppy disk' in topic # 13, which also helps us to reduce the possible ambiguity and understand the specific meaning of words in each specific topic. Third, the graph representation of topics also provides a new possibility to further separate the topic semantic by clustering the word nodes. For example, we utilize the fast unfolding algorithm (6). a widely-used community detection algorithm, to cluster nodes and color different categories. We find that, interestingly, the top words of topic # 12 are split into three sub-clusters with different sub-topics related to space research (violet), celestial bodies (green), and space exploration (orange), respectively. Similarly, the top words of the topic # 13 are separated into two categories: one about software or hardware (orange); the other may be about programming development (violet).
Visualization:
In addition, we also show the topic dependency matrix M in the right of Figure 4 where each element myy represents the possibility of existing a directed edge from topic i to j. We have the following interesting observations. First, there always exist high possibility values between topic #0 with any other topics. It is actually reasonable because topic #0 is represented with several general words, commonly used in any scenario. Second, the diagonal tend to have high value. It indicates that the words with the same topic should be linked, which is consistent with the widely-used assumption that interdependent (or neighboring) words prefer to have the same topic assignment [[5] 30 Third, we note that the topic dependency matrix also mines the dependency relation between different topics. For example, our topic sets, the top words of the topic #9 is full of sport-related words, while the topic #8 focuses on the hockey games with keywords related to team game name, such as "bos" (Boston Bruins), "det" (Detroit Red Wings) and "nhl" (National Hockey League). Considering the strong relation between topic #8 and #9, the value ma.9 and m2g8 is high.
32 59 60].
9

---

#12
#13 card
#0
Aθ game
spece
(space station)
Moоpy drives)
people know
hockey win
(moon orbit)
(Roрpy
data
(satellic arbit)
disk
(Rpppy drive)
think
nhi bas
teamn season
available
(moon space)
hard
(loppy disk)
time
barth
(sateliite space)
dos
hhad drives)
use
players
progham
50
into
newd
card
space
tech
drive
myddam
cotfee
stcch
Dirggtary
right
Figure 4: The case study. The left shows topics learned by GNTM with top words, and top edges and corresponding topic graph representation the top 30 words, where the sizes of nodes and edges
are positively correlated to gE and A₁ (gE (8g)²) ") @ St. respectively. Nodes are colored by their
cluster labels detected by fast unfloding algorithm [6] The right displays the heatmap of the topic
dependency matrix M, with several topics represented by the top 5 words.
5 Conclusion
In this paper, we revisited the topie modeling techniques by representing documents as semantic graphs and proposed Graph Neural Topic Model (GNTM) method. Specifically, first formulated a well-defined probabilistic generative story to model both the graph structure and word sets with new concept of topics, i.c., multinomial distributions both the vocabulary and word dependency edge. Then, a Neural Variational Inference (NVT) approach was proposed to learn our model with graph neural networks encode the document graphs. Besides, also have demonstrated that LDA can be derived from GNTM as a special case with similar objective functions. Finally, extensive experiments four benchmark datasets have clearly validated the effectiveness and interpretability of GNTM compared with state-of-the-arr baselines.
Limitation and Future Work: Here, we discover word dependency only by linking words in a small sliding window on word sequence. Several other approaches can also construct document graphs, such dependency parse (33). An interesting direction of future work is consider multiple types of word dependency relation together by representing documents as multi-relational graphs.
Acknowledgement
This research was supported by grants from the National Natural Science Foundation of China (Grant No. 91746301 and 61836013).
References
[1] Nikolaos Aletras and Mark Stevenson. Evaluating topic coherence using distributional semantics. Proceedings of the 10th International Conference on Computational Semantics (/WCS 2013)- Long Papers, pages 13-22, 2013.
[2] David Blei. Probabilistic topic models. Communications of the ACM, 55(4):77-84, 2012.
[3] David M Blei and John D Lafferty. Dynamic topic models. In Proceedings of the 23rd international conference on Machine learning, pages 13--120, 2006.
[4] David M Blei, John D Lafferty, et al. A correlated topic model of seience. The annals of applied statistics, 1(1):17-35, 2007.
[5] David Blei, Andrew Y Ng, and Michael Jordan. Latent dirichlet allocation. the Journal of machine Learning research, 3:993-1022 1022, 2003.
10

---

[6] Vincent D Blondel, Loup Guillaume, Renaud Lambiotte, and Etienne Lefebvre. Fast unfolding of communities in large networks. Journal of statistical mechanics: theory and experiment, 2008(10):P10008, 2008.
[7] Chain Monte Carlo. Markov chain monte carlo and gibbs sampling. Lecture notes for EEB, 581, 2004.
[8] William B Cavnar, John M Trenkle, al. N-gram-based text categorization. In Proceedings of SDAIR-94, 3rd annual symposium on document analysis and information retrieval, volume 161175. Citeseer, 1994.
[9] Xueqi Cheng, Xiaohui Yan, Yanyan Lan, and Jiafeng Guo. Btm: Topic modeling over short texts. IEEE Transactions on Knowledge and Data Engineering, 26(12):2928-2944 2014.
[10] BNC Consortium et al. British national corpus. Oxford Text Archive Core Collection, 2007.
[11] Adji B Dieng, Francisco JR Ruiz, and David M Blei. Topic modeling in embedding spaces. Trunsuctions of the Association for Computational Linguistics, 8:439-453, 2020.
[12] Adji B Dieng, Chong Wang. Jianfeng Gao, and John Paisley. Topicrnn: A recurrent neural network with long-range semantic dependency. arXiv preprint 1100:1661..01122.220 2016.
[13] Santo Fortunato. Community detection in graphs. Physics reports, 486(3-5)755-1744 2010.
[14] Alex Graves. Practical variational inference for neural networks. Advances in neural information processing systems, 24, 2011.
[15] Amit Gruber, Yair Weiss, and Michal Rosen-Zvi. Hidden topic markov models. In Artificial intelligence and statistics, pages 163-170. PMLR, 2007.
[16] Pankaj Gupta, Yatin Chaudhary, Florian Buettner, and Hinrich Schütze. Document informed neural autoregressive topic models with distributional prior. In Proceedings of the AAAI Conference Artificial Intelligence, volume 33, pages 6505-6512, 2019.
[17] Philipp Hennig, David Stern, Ralf Herbrich, and Thore Graepel. Kernel topic models. In Artificial intelligence and statistics, pages 511-519, PMLR, 2012.
[18] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pages 448-456 PMLR, 2015.
[19] Shoaib Jameel and Wai Lam. A nonparametric n-gram topic model with interpretable latent topics. Asia Information Retrieval Symposium, pages 74-85. Springer, 2013.
[20] Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with numbel-softmax. arXiv preprint arXiv: 1611.01144.4 2016.
[21] Hamed Jelodar, Yongli Wang, Chi Yuan, Xia Feng, Xiahui Jiang, Yanchao Li, and Liang Zhao. Latent dirichlet allocation (Ida) and topic modeling: models, applications, survey. Multimedia Tools and Applications, 78(11):15199-1521, 2019.
[22] Michael I Jordan, Zoubin Ghahramani, Tommi Jaakkola, and Lawrence K Saul. An in- troduction to variational methods for graphical models. Machine learning, 37(2):183-233, 1999.
[23] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1rXiv:1412.6980, 2014.
[24] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv:3312.6114, 2013.
arXiv preprint
[25] Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional 2016.
networks. arXiv preprint

---

[26] Thomas N Kipf and Max Welling. arXiv:1611.07308, 2016.
Variational graph auto-encoders. arXiv preprint
[27] Ken Lang. Newsweeder: Learning to filter netnews. Machine Learning Proceedings 1995, pages 331-339. Elsevier, 1995.
(28] Hugo Larochelle and Stanislas Lauly. A neural autoregressive topic model. Advances in Neural Information Processing Systems, 01:2708-27166 2012.
[29] Yujia Li, Daniel Tarlow, Mare Brockschmidt, and Richard Zemel. Gated graph sequence neural networks, arXiv preprint arXiv: 1.54993, 2015.
[30] Zhixing Li, Siqiang Wen, Juanzi Li, Peng Zhang, and Jie Tang. On modelling non-linear topical dependencies. In International Conference on Machine Learning, pages 458-466. PMLR, 2014.
[31] Hao Lin, Hengshu Zhu, Yuan Zuo, Chen Zhu, Junjie Wu, and Hui Xiong, Collaborative company profiling: Insights from an employee's perspective. In Thirty- AAAI Conference on Artificial Intelligence, 2017.
[32] Robert Lindsey, William Headden, and Michael Stipicevic. A phrase-discovering topic model using hierarchical pitman-yor processes. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 214-222 2012.
[33] Ryan McDonald and Joakim Nivre. Analyzing and integrating dependency parsers. Computa- tional Linguistics, 37(1):197-233 2011.
[34] Yishu Miao, Edward Grefenstette, and Phil Blunsom. Discovering discrete latent topics with neural variational inference. In International Conference on Machine Learning, pages 2410- 2419. PMLR, 2017.
[35] Yishu Miao, Lei Yu, and Phil Blunsom. Neural variational inference for text processing. In International conference machine learning, pages 1727-1736. PMLR, 2016.
[36] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. arXiv preprint 1rXiv:1301.378/1 2013.
[37] David Mimno, Hanna Wallach, Edmund Talley, Miriam Leenders, and Andrew McCallum. Optimizing semantic coherence in topic models. Proceedingy of the 2011 conference on empirical methods in natural language processing, pages 262-272,21 2011.
[38] Christopher Morris, Martin Ritzert, Matthias Fey, William L Hamilton, Jan Eric Lenssen, Gaurav Rattan, and Martin Grohe Weisfeiler and leman go neural: Higher-order graph neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 4602-4609, 2019.
[39] David Newman, Jey Han Lau, Karl Grieser, and Timothy Baldwin. Automatic evaluation of topic coherence. In Human language technologies: The 2010 annual conference of the North American chapter of the association for computational linguistics, pages 100-108, 2010.
[40] Dat Quoe Nguyen, Richard Billingsley, Lan Du, and Mark Johnson. Improving topic models with latent feature word representations. Transactions of the Association for Computational Linguistics, 3:299-313, 2015.
[41] Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532 1543, 2014.
[42] Mchdi Rezace and Francis Ferraro. A discrete variational recurrent topic model without the reparametrization trick. arXiv preprint WrXiv:2010.1205: 2020.
[43] Michael Röder, Andreas Both, and Alexander Hinneburg. Exploring the space of topic coherence measures. Proceedings of the eighth ACM international conference on Web search and data mining, pages 399-408. 2015.
12

---

[44] Michal Rosen-Zyy Zvi, Thomas Griffiths, Mark Steyvers, and Pachraic Smyth. The author-topic model for authors and documents. arXiv preprint arXiv: 2012.
[45] Hinrich Schütze, Christopher D Manning, and Prabhakar Raghavan. Introduction to information retrieval, volume 39. Cambridge University Press Cambridge, 2008
[46] Dazhong Shen, Chuan Qin, Chao Wang, Hengshu Zhu, Enhong Chen, and Hui Xiong. Regular- izing variational autoencoder with diversity and uncertainty awareness. In Proceedings of the International Joint Conference Artificial Intelligence, 2021.
[47] Dazhong Shen, Chuan Qin, Hengshu Zhu, Tong Xu, Enhong Chen, and Hui Xiong. Joint repre- sentation learning with relation-enhanced topic models for intelligent job interview assessment. ACM Transactions on Information Systems (TOIS), 40(1):1-36, 2021.
[48] Dazhong Shen, Hengshu Zhu, Chen Zhu, Tong Xu, Chao Ma, and Hui Xiong. A joint learning approach to intelligent job interview assessment. In IJCAI, volume 18, pages 3542-3548, 2018.
[49] Akash Srivastava and Charles Sutton. Autoencoding variational inference for topic models. arXiv preprint arxiv: 1703.0118888 2017.
[50] Fangshuang Tang, Qi Liu, Hengshu Zhu, Enhong Chen, and Feida Zhu. Diversified social influence maximization. In 2014 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM 2014), pages 455-459, IEEE, 2014.
[51] Daniele Vitale, Paolo Ferragina, and Ugo Scaiella. Classification of short texts by deploying topical annotations. In European Conference on Information Retrieval, pages 376-33 376-387. Springer, 2012.
[52] Hanna M Wallach. Topic modeling: beyond bag-of-words. In Proceedings of the 23rd international conference on Machine learning, pages 977-984, 2006.
[53] Chong Wang and David M Blei. Variational inference in nonconjugate models. Journal of Machine Learning Research, 14(4), 2013.
[54] Tong Wang, Vish Viswanath, and Ping Chen. Extended topic model for word dependency. In Proceedings of the 53rd Anmual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing ( (Volume 2: Short Papers), pages 506-510, 2015.
[55] Xuerui Wang, Andrew McCallum, and Xing Wei. Topical n n-grams: Phrase and topic discovery. with an application to information retrieval. In Seventh IEEE international conference on data mining (ICDM 2007), pages 697-702. IEEE, 2007.
[56] Manzil Zaheer, Amr Ahmed, and Alexander J Smola. Latent Istm allocation: and non non-linear dynamic modeling of sequence data. In International Conference on Machine Learning, pages 3967-3976. PMLR, 2017.
Joint clustering
[57] He Zhao, Dinh Phung, Viet Huynh, Yuan Jin, Lan Du, and Wray Buntine. Topic modelling meets deep neural networks: A survey, arXiv preprint 1rXiv:2103.00493, 2021.
[58] Hongke Zhao, Qi Liu, Hengshu Zhu, Yong Ge, Enhong Chen, Yan Zhu, and Junping Du sequential approach market state modeling and analysis in online p2p lending. IEEE Transactions on Systems, Man, and Cybernetics: Systems, 48(1):21-33 2017.
[59] Wayne Xin Zhao, Jing Jiang, Jianshu Weng. Jing He, Fe-Peng Lim, Hongfei Yan, and Xiaoming Li. Comparing twitter and traditional media using topic models. In European conference on information retrieval, pages 338-349. Springer, 2011.
[60] Jun Zhu and Eric P Xing. Conditional topic random fields. 2010.
[61] Qile Zhu, Zheng Feng, and Xiaolin Li. Graphbtm: Graph enhanced autoencoded variational inference for biterm topic model. In Proceedings of the 2018 conference on empirical methods in natural language processing, pages 4663-4672, 2018
13


Reviews:
[{"review_id": "bV04QEFKTVp", "reviewer": "Reviewer_7TJJ", "summary": "This work is based on neural topic modeling where a document is represented by a graph of word notes with edges representing the semantic dependency and the generative process of learning is a function of both the document-graph structure (i.e., word-dependency edges) as well as word sets (i.e., document vocabulary). On top, the AVI approach is introduced to perform learning over the proposed Graph Neural Topic Model (GNTM). \n\nSeveral past works in topic modeling have investigated dependency relations among words: modeling sequential dependency and syntactic relationships [1] using dependency parse trees. Another thread of research [2, 3] incorporates both: BoW and contextualized information in the generative process of topic modeling, where the contextualized representations introduced modeling word-dependency relations in the generative process of neural topic models. This line of research is missing in quantitative comparison with the proposed topic model. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n----------------------------------------\n+ paper is well formulated and easy to read/follow \n+ the proposed methodology is well detailed and described\n+ novel approach of introducing document-graph structure in the generative process of topic modeling \n+ novel AVI approach of learning over GNTM model\n \n\nWeaknesses:\n----------------------------------------\n- Unclear contribution since several existing works in topic modeling have already investigated incorporating word-dependency relations in the generative process of neural topic modeling \n- comparison to the baselines methods used is not fair and incomplete\n- the baseline methods do not use pretrained word embeddings (e.g. word2vec), however the proposed model GNTM utilizes external knowledge via pre-trained embeddings word2vec. Therefore, comparison is unfair. The topic models with word embeddings such as [4 , 5, 6] that are based on BoW and have also incorporated pretrained word embeddings should be used in quantitative analysis for fair comparison. \n- missing comparison with the topic models that incorporates word dependency relations such as [1, 2, 3]\n- experimental setup is unclear; how are the scores of baseline methods obtained? \n- Additional experiments required to evaluate the quality of document representations such as text classification and unsupervised IR.  \n- DocNADE and its variants such as iDocNADE/iDocNADEe have shown to outperform [3] ProdLDA; however the score of iDocNADE are low. Please provide experimental details and reasoning on the deviation of scores from original works?\n- The proposed GNTM model incorporates word dependency relations, while the baseline methods do not and only assumes bow. Therefore, additional comparisons/experiments with the topic models variants such as [1, 2, 3] that incorporate (sequential/syntactic/contextualized) word-dependency relations are required for completeness. \n\nReferences:\n\n[1] Extended Topic Model for Word Dependency. ACL 2015.\n\n[2] TopicRNN: A Recurrent Neural Network with Long-Range Semantic Dependency. ICLR 2017.\n\n[3] Texttovec: Deep Contextualized Neural autoregressive Topic Models of Language with Distributed Compositional Prior. ICLR 2019.\n\n[4] Gaussian LDA for Topic Models with Word Embeddings. ACL 2015.\n\n[5] Improving topic models with latent feature word representations. TACL 2015.\n\n[6] Document Informed Neural Autoregressive Topic Models with Distributional Prior. AAAI 2019.\n\n\n<< --------------------------------------- Update -------------------------------------- >>\n\nThanks for clarification and additional evaluation in the baseline table. \n\nBased on the response, I increase my score by +1. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"ethical_concerns": "No", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "bIAJR9CCh8r", "reviewer": "Reviewer_hog6", "summary": "Going beyond the bag-of-word assumption in topic modelling is an interesting problem, which has been studied since the introduction of the vanilla LDA, for instance, the syntactic topic model. This paper proposes to represent each document as a directed word graph, which is constructed via moving a window along the word sequence. The main technical contributions of this paper can be 1) the graph neural topic model, where words are generated conditioned on both the topic assignments and the reconstructed word graph; 2) the VAE framework used to learn the posteriors.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "* Representing each document as a directed word graph is interesting, which could potentially capture complex and nonlinear relationships among words. The idea of integrating GCN with the topic model can also be found in [1-3]. The difference is that the reconstruction of the directed graph given a set of sampled latent topic assignments and the word generation given the reconstructed graph and topic assignment, which can be the strength and could be also the weakness. The strength resides in that it could potentially learn the semantic correlation between words. However, my concern is also about the graph reconstruction. I am a bit confused by G_d and G^o_d (the placeholder) in regards to the generative process. It would be good to explain it with the example in figure 1 (a). For instance, is it assumed that the relative position of words is fixed in G^o_d, the reconstruction is just about deciding if there is a directed edge between two words. However, eq (4) seems to say E^o_d is observed, and line 107 indicates it is different from E_d. Further clarity is needed here. \n* In the proposed model, each topic is also a distribution over edges between pairs of words. Presumably, there is a one to one making between a topic as a distribution over words and that as a distribution over edges.   I wonder if the topic assignment to edges is independent of the topic assigned to a word, or they are dependent.\n* Both topic coherence scores and downstream clustering tasks are used in the experiment, and the proposed GNTM is compared with several neural topic models, showing promising results.  Here are my comments:\n   * In regards to the topic coherence scores, it is not unknown that consistent document preprocessing can significantly affect the topic coherence scores, reporting how documents are preprocessed is necessary. \n   * There is a lack of oblation study on the impact of representing topics as a distribution over edges. \n   * Besides GraphBTM, it might also be good to consider those GCN based topic models, like [1-3]. \n   * it is good to show how sensitive the model is to the window size used in construction G_d.\n\nReferences\n[1] Xie et al 2021 Graph Topic Neural Network for Document Representation\n[2] Long et al 2020, Graph Structural-topic Neural Network\n[3] Yang et al 2020, Graph Attention Topic Modeling Network ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have briefly discussed the limitation of the proposed work. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": ">5"}, "extra_text": {}}, {"review_id": "M-BulZuVFY", "reviewer": "Reviewer_zRWm", "summary": "In this paper, the authors provide a novel generative model to learn topics using graph modeling based on the word network, following a (now classic) variational encoding approach. The documents are encoded as graphs by modeling an edge between two words if they appear in the same window, which is similar to the embedding methods with the distributional hypothesis. The graph structure is then part of the generative process. They demonstrate that the well-known LDA model is a special case of their approach if we ignore edges. Most of the time, the model outperforms existing topic models (classic such as LDA and neural TM) on 4 English written datasets in term of Cv, NPMI and Topic Diversity. It also outperforms prior works in term of clustering accuracy.  ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n\nThe model is quite clearly defined. It is novel and interesting. All modeling choices are motivated by previous works, and the additional construction of a document graph using word neighborhood is highly relevant: the BOW hypothesis is one of the main weaknesses of previous topic models. Even if incorporating the sequential information was already studied (e.g., HTMM), there is still room for improvement. \nThe graph that one can construct at the end when the parameters are learned (Figure 4, left) is particularly interesting for interpretability. It seems that it can really improve the understanding of the topic content. \nThe approach seems to outperform (recent) prior works in term of topic coherence, diversity, and clustering.\n\nCons:\n\nNot a rejection criterion per se, but your approach is beaten in few configurations, e.g. NMI on TMN and 20NG with a high number of topic for clustering and TD score for 20NG, BNC and Reuters, again with high number of topics. Do you have an intuition on why your method performs slightly worst when the number of topics grows?\nEven if you provide the effect of the window size in supp.mat. on the 20NG dataset, it would be interesting to provide further experiments regarding this parameter, on more datasets. Similarly, you should provide more experiments to test the effect of the word embedding used on the performances of your model.   \nL35: “For example, LDA discovers word such as “station” in a topic which does not seem to be that insightful. » You do not introduce this example. Where does it come from? You cannot drop examples like that and expect that the reader will understand your point. \nL 108: “We also filter out the stop words, and words and dependency edges with low frequency among the whole collection to reduce noise.”: it should be in the experimental section, and you should add the parameters you use (which stop words, the threshold for selection, etc…). All your experiments must be reproduceable.\nThe paper is not well written. There is a lot of grammatical errors, typos, and some sentences are not clear.\nAdditional remarks:\n2.2 : there should be a way to simplify the notation, this subsection is hard to follow.\nIn the abstract and in the experiment section, you should use present tense, not past tense.\nWhy using AVI rather than the usual expression (VAE)?\nL 100: “ we can donate a document” this is not clear, donate does not fit here\nL 104: “with nodes on the vocabulary” : should be “in the vocabulary”\nL 128:  just based  based solely on the topic assignment ?\nL 136 : should be cast into  cast as\nL 210 : “Now, the only challenge to compute the ELBO loss Ld is at the last term in Eqaution: “ this sentence is not grammatically correct and Eqaution should be Equation\nL 58 : Graphic Neural Topic Modeling  Graph Neural Topic Modeling. This is the name of your method! \n\n---\n\nI've read the authors' response and the other reviews. The authors answered most of my remarks and I choose to keep my score of 7. I'd be interested to read the reactions of the other reviewers before, but they never came...\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "We can read \"N/A\" in the form. I guess developing a new topic modeling approach let the authors ignore this point.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2h"}, "extra_text": {}}, {"review_id": "JNlDotd6RZK", "reviewer": "Reviewer_EBBL", "summary": "The authors propose a topic model that explicitly captures the semantic dependency among words in the same document, by representing each word as a node in a graph and connecting words appearing closely in the document through an edge in the graph.\nThis approach overcome the limitation of standard topic models as they can only consider word as independent. in this way, inferred topics may be more interpretable and semantically more meaningful. This is validated in the experiments as the proposed model achieves better topic coherence.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "One limitation of the approach is that the documents are split into windows to construct the graphs. This means that the model would disregard the co-occurrences of terms in the same document but in different positions of the document. This means that the model may link together words close-by, but lose more distant co-occurrences. It would be nice to see an experiment where this trade off is compared.\n\nAlso, the experiments are tested with a window of 5. I believe this is to limit the complexity of the method, as considering a bigger window size would require building a large number of subgraph. How much is the model scalable in terms of the window size? One limitation of standard topic models is that they require limited size of the vocabulary. By adding the complexity of creating subgraphs for each part of a document, I believe the model would require a limited vocabulary, . However, I could not find a comment on the practical scalability in the paper.\n\nIn terms of topic diversity, the results show that the proposed model performs worse when increasing the number of topics. Vice versa, a simple LDA is able to consistency achieving good or even better results with the increase of the number of topics. Additional comment on the results would be beneficial to understand this.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviews have positive comments about the family of topic models explored here, which uses graph-neural-networks to remove the word independence assumptions in classical models like LDA.  Some reviewers raised questions about experimental comparisons to related work, that were reasonably addressed by the author response; please be sure to include these comparisons in future revisions.  The manuscript text also needs to be polished to improve readability, and clarify several details raised in reviews.
Decision: Accept (Poster)