Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This work is based on neural topic modeling where a document is represented by a graph of word notes with edges representing the semantic dependency and the generative process of learning is a function of both the document-graph structure (i.e., word-dependency edges) as well as word sets (i.e., document vocabulary). On top, the AVI approach is introduced to perform learning over the proposed Graph Neural Topic Model (GNTM). 

Several past works in topic modeling have investigated dependency relations among words: modeling sequential dependency and syntactic relationships [1] using dependency parse trees. Another thread of research [2, 3] incorporates both: BoW and contextualized information in the generative process of topic modeling, where the contextualized representations introduced modeling word-dependency relations in the generative process of neural topic models. This line of research is missing in quantitative comparison with the proposed topic model. 
main_review: Strengths:
----------------------------------------
+ paper is well formulated and easy to read/follow 
+ the proposed methodology is well detailed and described
+ novel approach of introducing document-graph structure in the generative process of topic modeling 
+ novel AVI approach of learning over GNTM model
 

Weaknesses:
----------------------------------------
- Unclear contribution since several existing works in topic modeling have already investigated incorporating word-dependency relations in the generative process of neural topic modeling 
- comparison to the baselines methods used is not fair and incomplete
- the baseline methods do not use pretrained word embeddings (e.g. word2vec), however the proposed model GNTM utilizes external knowledge via pre-trained embeddings word2vec. Therefore, comparison is unfair. The topic models with word embeddings such as [4 , 5, 6] that are based on BoW and have also incorporated pretrained word embeddings should be used in quantitative analysis for fair comparison. 
- missing comparison with the topic models that incorporates word dependency relations such as [1, 2, 3]
- experimental setup is unclear; how are the scores of baseline methods obtained? 
- Additional experiments required to evaluate the quality of document representations such as text classification and unsupervised IR.  
- DocNADE and its variants such as iDocNADE/iDocNADEe have shown to outperform [3] ProdLDA; however the score of iDocNADE are low. Please provide experimental details and reasoning on the deviation of scores from original works?
- The proposed GNTM model incorporates word dependency relations, while the baseline methods do not and only assumes bow. Therefore, additional comparisons/experiments with the topic models variants such as [1, 2, 3] that incorporate (sequential/syntactic/contextualized) word-dependency relations are required for completeness. 

References:

[1] Extended Topic Model for Word Dependency. ACL 2015.

[2] TopicRNN: A Recurrent Neural Network with Long-Range Semantic Dependency. ICLR 2017.

[3] Texttovec: Deep Contextualized Neural autoregressive Topic Models of Language with Distributed Compositional Prior. ICLR 2019.

[4] Gaussian LDA for Topic Models with Word Embeddings. ACL 2015.

[5] Improving topic models with latent feature word representations. TACL 2015.

[6] Document Informed Neural Autoregressive Topic Models with Distributional Prior. AAAI 2019.


<< --------------------------------------- Update -------------------------------------- >>

Thanks for clarification and additional evaluation in the baseline table. 

Based on the response, I increase my score by +1. 

limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: Going beyond the bag-of-word assumption in topic modelling is an interesting problem, which has been studied since the introduction of the vanilla LDA, for instance, the syntactic topic model. This paper proposes to represent each document as a directed word graph, which is constructed via moving a window along the word sequence. The main technical contributions of this paper can be 1) the graph neural topic model, where words are generated conditioned on both the topic assignments and the reconstructed word graph; 2) the VAE framework used to learn the posteriors.

main_review: * Representing each document as a directed word graph is interesting, which could potentially capture complex and nonlinear relationships among words. The idea of integrating GCN with the topic model can also be found in [1-3]. The difference is that the reconstruction of the directed graph given a set of sampled latent topic assignments and the word generation given the reconstructed graph and topic assignment, which can be the strength and could be also the weakness. The strength resides in that it could potentially learn the semantic correlation between words. However, my concern is also about the graph reconstruction. I am a bit confused by G_d and G^o_d (the placeholder) in regards to the generative process. It would be good to explain it with the example in figure 1 (a). For instance, is it assumed that the relative position of words is fixed in G^o_d, the reconstruction is just about deciding if there is a directed edge between two words. However, eq (4) seems to say E^o_d is observed, and line 107 indicates it is different from E_d. Further clarity is needed here. 
* In the proposed model, each topic is also a distribution over edges between pairs of words. Presumably, there is a one to one making between a topic as a distribution over words and that as a distribution over edges.   I wonder if the topic assignment to edges is independent of the topic assigned to a word, or they are dependent.
* Both topic coherence scores and downstream clustering tasks are used in the experiment, and the proposed GNTM is compared with several neural topic models, showing promising results.  Here are my comments:
   * In regards to the topic coherence scores, it is not unknown that consistent document preprocessing can significantly affect the topic coherence scores, reporting how documents are preprocessed is necessary. 
   * There is a lack of oblation study on the impact of representing topics as a distribution over edges. 
   * Besides GraphBTM, it might also be good to consider those GCN based topic models, like [1-3]. 
   * it is good to show how sensitive the model is to the window size used in construction G_d.

References
[1] Xie et al 2021 Graph Topic Neural Network for Document Representation
[2] Long et al 2020, Graph Structural-topic Neural Network
[3] Yang et al 2020, Graph Attention Topic Modeling Network 
limitations_and_societal_impact: The authors have briefly discussed the limitation of the proposed work.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: In this paper, the authors provide a novel generative model to learn topics using graph modeling based on the word network, following a (now classic) variational encoding approach. The documents are encoded as graphs by modeling an edge between two words if they appear in the same window, which is similar to the embedding methods with the distributional hypothesis. The graph structure is then part of the generative process. They demonstrate that the well-known LDA model is a special case of their approach if we ignore edges. Most of the time, the model outperforms existing topic models (classic such as LDA and neural TM) on 4 English written datasets in term of Cv, NPMI and Topic Diversity. It also outperforms prior works in term of clustering accuracy.  
main_review: Pros:

The model is quite clearly defined. It is novel and interesting. All modeling choices are motivated by previous works, and the additional construction of a document graph using word neighborhood is highly relevant: the BOW hypothesis is one of the main weaknesses of previous topic models. Even if incorporating the sequential information was already studied (e.g., HTMM), there is still room for improvement. 
The graph that one can construct at the end when the parameters are learned (Figure 4, left) is particularly interesting for interpretability. It seems that it can really improve the understanding of the topic content. 
The approach seems to outperform (recent) prior works in term of topic coherence, diversity, and clustering.

Cons:

Not a rejection criterion per se, but your approach is beaten in few configurations, e.g. NMI on TMN and 20NG with a high number of topic for clustering and TD score for 20NG, BNC and Reuters, again with high number of topics. Do you have an intuition on why your method performs slightly worst when the number of topics grows?
Even if you provide the effect of the window size in supp.mat. on the 20NG dataset, it would be interesting to provide further experiments regarding this parameter, on more datasets. Similarly, you should provide more experiments to test the effect of the word embedding used on the performances of your model.   
L35: “For example, LDA discovers word such as “station” in a topic which does not seem to be that insightful. » You do not introduce this example. Where does it come from? You cannot drop examples like that and expect that the reader will understand your point. 
L 108: “We also filter out the stop words, and words and dependency edges with low frequency among the whole collection to reduce noise.”: it should be in the experimental section, and you should add the parameters you use (which stop words, the threshold for selection, etc…). All your experiments must be reproduceable.
The paper is not well written. There is a lot of grammatical errors, typos, and some sentences are not clear.
Additional remarks:
2.2 : there should be a way to simplify the notation, this subsection is hard to follow.
In the abstract and in the experiment section, you should use present tense, not past tense.
Why using AVI rather than the usual expression (VAE)?
L 100: “ we can donate a document” this is not clear, donate does not fit here
L 104: “with nodes on the vocabulary” : should be “in the vocabulary”
L 128:  just based  based solely on the topic assignment ?
L 136 : should be cast into  cast as
L 210 : “Now, the only challenge to compute the ELBO loss Ld is at the last term in Eqaution: “ this sentence is not grammatically correct and Eqaution should be Equation
L 58 : Graphic Neural Topic Modeling  Graph Neural Topic Modeling. This is the name of your method! 

---

I've read the authors' response and the other reviews. The authors answered most of my remarks and I choose to keep my score of 7. I'd be interested to read the reactions of the other reviewers before, but they never came...

limitations_and_societal_impact: We can read "N/A" in the form. I guess developing a new topic modeling approach let the authors ignore this point.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The authors propose a topic model that explicitly captures the semantic dependency among words in the same document, by representing each word as a node in a graph and connecting words appearing closely in the document through an edge in the graph.
This approach overcome the limitation of standard topic models as they can only consider word as independent. in this way, inferred topics may be more interpretable and semantically more meaningful. This is validated in the experiments as the proposed model achieves better topic coherence.
main_review: One limitation of the approach is that the documents are split into windows to construct the graphs. This means that the model would disregard the co-occurrences of terms in the same document but in different positions of the document. This means that the model may link together words close-by, but lose more distant co-occurrences. It would be nice to see an experiment where this trade off is compared.

Also, the experiments are tested with a window of 5. I believe this is to limit the complexity of the method, as considering a bigger window size would require building a large number of subgraph. How much is the model scalable in terms of the window size? One limitation of standard topic models is that they require limited size of the vocabulary. By adding the complexity of creating subgraphs for each part of a document, I believe the model would require a limited vocabulary, . However, I could not find a comment on the practical scalability in the paper.

In terms of topic diversity, the results show that the proposed model performs worse when increasing the number of topics. Vice versa, a simple LDA is able to consistency achieving good or even better results with the increase of the number of topics. Additional comment on the results would be beneficial to understand this.

limitations_and_societal_impact: Yes
review_title: null

Meta-review: Reviews have positive comments about the family of topic models explored here, which uses graph-neural-networks to remove the word independence assumptions in classical models like LDA.  Some reviewers raised questions about experimental comparisons to related work, that were reasonably addressed by the author response; please be sure to include these comparisons in future revisions.  The manuscript text also needs to be polished to improve readability, and clarify several details raised in reviews.
Decision: Accept (Poster)