Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: (Summary)
This paper proposes a generative model that can learn hierarchical topics that coherently incorporate a structured prior knowledge graph. By introducing Gaussian embeddings to an existing SawETM, a neural topic model where topics in different levels can relate each other, the authors first propose the Gaussian SawETM. Then their TopicNet based on Gaussian SawETM can learn hierarchical topics coherently to an input knowledge-graph. TopicNet is not only capable of learning hypernym-induced topics as hierarchies, but also equipped with rich interpretability.

main_review: (Originality) 
Instead of using Gaussian variables as latent codes like other neural topic models based on VAE, the authors combine an idea of using gamma-distributed latent variables from Weibull hybrid inference (by Zhang et al) and SawETM (by Duan et al) for dependencies in two consecutive layers. Therefore the real novelty comes more from an idea of defining topic template as a topic tree and guiding the topics of each layer learned from the Gaussian SawETM by the corresponding semantic layer in the topic tree through imposing constraints on the topic embeddings. Due to such joining nature, one could equally argue that the contribution in modeling is mostly incremental.


(Clarity)
The writing is generally clear, but readers might need to look at previous papers to understand the details of Weibull method and SawETM. Though the previous work doesn’t have to be completely rephrased, providing some degree of background will benefit future readers.


(Significance)
The contribution of the overall paper seems to be more on applicational side as the proposed models rely on combining two successful neural networks. However, the experiments are limited to using either only small vocabulary datasets or too large vocabularies comparing to the number of documents, possibly weakening statistical robustness. Without demonstrating high performance on more realistic datasets (i.e., the number of topics << the number of words << the number of documents), it may be hard to convince potential users.


(Major Comments and Questions)
1) How is the vocabulary curated? Users in topic modeling typically put non-negligible amounts of efforts in vocabulary curation for properly removing both too frequent and too rare terms, which affects final topic quality. It is unclear how the authors curate the vocabulary for their experiments.

2) As you learn a generative model for unsupervised learning, having a semi-synthetic experiment will be useful for verifying true performance. Note that it is a better practice to first learn realistic topic distributions from real text corpora, later sampling various sizes of documents by using the known topic distributions and randomly sampled multinomial topic proportions. Given ground-truth topic distributions and proportions, you will be able to firmly complement the clustering study.

3) Performance of the most standard LDA model could be changed by a big margin based on the choice of hyperparameters as well as inference method such as Collapsed Gibbs Sampling (CGS) or Collapsed Variational Inference (CVI). How would the baseline LDA perform if using learned hyperparameters with CGS? Does the proposed model still significantly outperform LDA when the LDA is optimized as we use as a part of standard protocol?

4) In Section 4.1, the authors said that “SawETM cannot capture the uncertainties of topics”. What is exactly the uncertainty and how is it mathematically defined? Do you have any real example where SawETM fails to capture these uncertainties, whereas Gaussian SawETm captures successfully? Providing at least one qualitative example will benefit the readers even if it is anecdotal.

5) Being explainable and having topic interpretability are not identical in general category. In many cases, Explainable AI (XAI) tries to understand reasoning or inference path where users cannot transparently see how the forecast comes out. Is the topic discovery explainable in an agreeable sense?

6) Topic diversity is named more frequently as inter-topic similarity or dissimilarity. Read some papers by David Mimno will be helpful. In addition, topic specificity (how far from the unigram corpus distribution) can often help quality measurement jointly with those two metrics.

7) WordNet’s hierarchy semantic is mostly limited only to nouns. What happen if some topics are more about actions than entities, thereby being described more by verbs than nouns in their top words?


(Minor Comments)
1) Line 252, from  form


(After Author-Feedback)
Have updated my previous evaluation.

limitations_and_societal_impact: No problem has been found.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper builds on the recently proposed SawETM hierarchical topic model and combines it with a knowledge graph so that each node in the knowledge graph corresponds to a topic. They change it so that every topic is embedded with a Gaussian distribution. This enables to enforce a hierarchical ordering constraint on the topics. Experiments show that the model discovers deeper, more interpretable topics and finds better document representations.
main_review: This paper presents an interesting idea of incorporating knowledge graphs with hierarchical topic models. My criticism is limited to explanation and inconsistency in experiments. The article proposes two novel methods for Hierarchical Topic Modeling. Gaussian SawETM - A probabilistic generative model to capture semantic uncertainties of words and topics. TopicNet - A knowledge-based hierarchical topic model which incorporates prior knowledge (belief) of the real world. TopicNet employs Gaussian SawETM and learns the hierarchies based on prior beliefs such as knowledge graphs. The idea is theoretically sound and has a mathematical
explanation. The structure and quality of explanation can be improved in some
sections. More consistent experiments and comparisons would have validated
what the authors claim, the absence of which can leave a reader curious. 

The quality of the explanation is inconsistent throughout the article. While the
Introduction section is not properly structured, it does a good job of explaining
the content it provides with the help of diagrams. Whereas, Section 2 and 3,
which form the proposed model, lack quality of explanation.
It has the following explanation problems:

- No proper figure is given for the proposed TopicNet model.
- The figure provided for Gaussian SawETM is a general Gamma Belief Net-
work, it has no representation of upward and downward encoder networks
as described in Section 2.3.
- The authors claim to have reduced mode collapse by their use of Gaussian
SawETM, but provide inconsistent results in Table 1 and Table 2. In perplexity representation depth till 15 are accounted for but in classification
depth of only 1 and 4 are accounted for. The depth inconsistency further
continues in Figure 4, where the embedding visualization is done for Gaussian SawETM with a depth of 3. Depth should be made consistent while
comparison in all tables as well as figures, since claims based on different
layers, coincide with results of different layers.
- While presenting the Topic Quality, the authors themselves hint at the
reason behind the exceptional performance of TopicNet on severe infor-
mation reduction from data in the deep layers. This specific case will lead
to more diversity in words, hence Topic Quality can be a slightly skewed
metric here for the judgement. Results of Topic Coherence based on NPMI
can also be presented to clear the doubt.
- Only selected hierarchies are presented. Human evaluation measures such
as word intrusions can be utilized to evaluate the hierarchies.


Two further questions I have for the authors are:

- How can we be sure that the knowledge graph has anything to do with the dataset that we use? What if additional topics are present or the topics in the knowledge graph are more detailed than the dataset calls for? 
- How are the semantic concepts themselves injected into the topic model? I only see how the relationships between the concepts are injected. But how do you make sure that the topic tree node "animal" actually corresponds to a topic with words related to animals?


Minor comments:
- Related work section and introduction are mixed together, I would suggest to separate them into two separate parts.
- The heading of Section 2 should maybe better be changed to "Gaussian SawETM"
- In some cases such as [7], the arXiv version of a paper is cited even though it is published in a peer-reviewed conference.
- The references are very inconsistent, e.g. compare [25] and [26]
- The example in Fig. 1a) could be read as being sexist, as mother, son and home are in one topic, but father is not, so only mothers walk their son to school? 
- the example in Fig. 1a) seems not very meaningful, for example the topic "road read bike walk", makes not much sense, as read has nothing to do with road.
- line 128 BOund-> bound
- Figure. 5 -> Fig. 5
- Figure. 1 -> Fig. 1
- tpoic -> topic
- 7-ayer -> 7-layer
- negetive -> negative



*************************
my comments were sufficiently addressed and I am increasing my rating
limitations_and_societal_impact: One main limitation from my point of view is the fact that we do not know if the knowledge graph is appropriate for the dataset, as I wrote in the review:
How can we be sure that the knowledge graph has anything to do with the dataset that we use? What if additional topics are present or the topics in the knowledge graph are more detailed than the dataset calls for? 

So the negative impact could be that topics in the dataset are not found because they are not present in the prior knowledge. Limitations are not really addressed in the paper.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper mainly proposes a new hierarchical topic discovery framework that is guided by prior (tree-structured) knowledge. The construction of the framework is broken down into two steps: Modeling semantic hierarchy with a Gaussian-embedding-based topic model (Gaussian SawETM), and injecting the prior semantic knowledge by regularizing via entailment relationship. The model is evaluated on a set of benchmarks with perplexity, topic quality, and clustering/classification protocols. Qualitative studies are also shown to demonstrate the interpretability of the model.
main_review: [Originality]

Overall, the novelty is at the borderline of a NeurIPS paper. In Sec. 2, the Gaussian SawETM model is essentially a combination of SawETM and Gaussian embeddings, so the ideas are not new there. In Sec. 3, the injection of prior knowledge is implemented as a set of regularization over the entailment relationship in the semantic graph (like WordNet), and the modeling of the transitivity and antisymmetry is commonly seen in hierarchy modeling studies (the line of hyperbolic embeddings), thus the design is reasonable but not particularly providing new insights. However, I like the overall motivation of this paper to guide topic discovery with prior (graph-structured) knowledge, which is not commonly seen in the topic modeling literature.

[Quality]

The Gaussian SawETM model borrows the idea from a recent topic modeling paper SawETM to alleviate the model collapse issue for deep hierarchical topic models, and further leverages Gaussian embeddings to model semantic uncertainties of words. The model inference follows standard VAE-based neural topic models for optimizing the ELBO. The TopicNet framework, built upon the Gaussian SawETM model, incorporates prior tree-structured knowledge by regularizing the Gaussian embeddings to align with the partial order of semantic concepts as a new objective added to the ELBO objective. The overall design of the model is principled and neat. 

The empirical evaluation is conducted on a set of benchmarks with metrics commonly used in topic modeling evaluation (perplexity, topic quality, and clustering/classification). The proposed models (Gaussian SawETM & TopicNet) demonstrate comparable or better performance than existing neural topic models. So the evaluation also seems convincing to me.

[Clarity]

The paper is overall clearly written and well-organized. The method sections clearly draw connections with related studies and motivate each component. The experiment section clearly states the setup/evaluation protocol and presents detailed analyses.

[Significance]

The significance of this paper is probably at an okay level--although the proposed models outperform baselines in some tasks (clustering/classification), they are not better under perplexity evaluation. Also, the guided version (TopicNet) underperforms the unsupervised version (Gaussian SawETM) for clustering/classification, which is inconsistent with the major motivation of the paper. However, I appreciate the overall direction of incorporating prior semantic knowledge into topic discovery, and feel that the paper has made an interesting exploration overall.

[Questions]

I hope the authors could clarify the following points:
* Is WordNet used as the semantic graph in all experiments/datasets? Could you use other prior structures that better fit the specific domains/datasets? It is mentioned in the document clustering/classification experiments that TopicNet does not perform better than Gaussian SawETM because the prior information is irrelevant with the document classification and clustering tasks. This observation largely weakens the motivation of this paper that some prior guidance can help the topic discovery process.
I'm curious whether a manually curated topic hierarchy based on the dataset will help improve the clustering/classification performance--even that will make the comparisons with baselines unfair, but at least that will confirm the motivation and demonstrate the effectiveness of TopicNet on modeling the given semantic graph.
* It seems to me that the current setup of TopicNet can only model tree structures. If this is true, calling the model "Semantic Graph-Guided" might be a bit inappropriate, as the tree structure is only a subset of graph structures.
* It is mentioned in the perplexity evaluation that the DLDA group requires iteratively sampling to infer latent document representations in the test stage, despite getting better performance. I feel that some concrete run time reports are necessary to support this argument (are they really that costly compared to the DNTM group?).

[Summary]

Pros:
* The paper is clearly written and well-organized; the incorporation of prior semantic knowledge is well-motivated.
* The overall framework is reasonable and principled.
* The evaluation is comprehensive.

Cons:
* The novelty is not that high, as many essential components are borrowed from previous work.
* The empirical evidence that TopicNet underperforms Gaussian SawETM in applications raises concerns about the basic assumption/motivation of the paper.

################

Updates after rebuttal:
I would like to thank the authors for conducting new experiments and providing clarifications that addressed my concerns. I increased my rating from 5 to 6.
limitations_and_societal_impact: I feel that the authors have addressed the limitations in a reasonable way.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors propose a topic model that aim at embedding words and topics into a latent space via Gaussian embeddings. Additional constraints in the training procedure allows to incorporate prior knowledge on the topics, so that they are able to include semantic topic relations.

The model is evaluated on a large set of public datasets and on a number of tasks. It achieves better results with respect to baselines on clustering tasks, and comparable on classification tasks. However, the main advantage is the increase interpretability of topics and topic hierarchy.
main_review: The paper is sometimes quite difficult to read. However, the authors made a clear effort to be exhaustive in explaining mathematical details and experiment settings.

I am not sure I understand how it is possible to incorporate additional knowledge on the topic semantics. The authors mention that "Since the Gaussian SawETM projects the topics of all layers to a shared embedding space, such a goal can be achieved by imposing constraints on the topic representations in the shared embedding space.". However, as each topic is a collection of words, and the topic is embedded in a latent space, the semantic of the collection of words is lost. How the topic semantic (if the topic is automatically discovered by the procedure) can be mapped to the external knowledge graph between the entities?

The authors do not mention the scalability of the model. One main limitation of standard topic model is their scalability when increasing the vocabulary size, as for each topic all of the vocabulary is needed to be sampled to compute the probability distribution. From the proposed approach, it seems that the method aim at embedding topics into a latent space. It would be nice to assess the scalability in terms of words or topics.

In terms of experiments, the authors mention that they evaluated the model also in a classification task. However, it's not clear to me what the classification task is as the dataset they describe are textual documents. What labels have the document in the datasets? Each document has a "category" as a label?

Highlighting the best results in Table 2 would make the table easier to read. It seems that in general the proposed method is better than the baselines on the various clustering tasks, and slightly worse for classification tasks. When adding prior knowledge, the method has actually worse performance. The authors mention that "TopicNet does not perform better maybe because the prior information is irrelevant with the document classification and clustering tasks". However, the additional information on topic relation should better guide the model, and I am a bit surprise this is happening. As I haven't fully understood how to include the prior information in the model I think this is an important point to explore better.

Minor:
typo: negetive (page 6)
limitations_and_societal_impact: None
review_title: null

Meta-review: This paper develops TopicNet, a generative model that can learn hierarchical topics by incorporating a pre-defined semantic graph. The main contribution of the paper seems to be on the application side, and all reviewers agreed the paper is above the acceptance threshold.

I would like to encourage the authors to incorporate the reviewers' feedback in their revised version. In particular:
+ Incorporate the new results from the response to reviewer aZKr.
+ Add the metrics of inter-topic (dis)similarity and topic specificity.
+ Add an experiment to showcase that the method can benefit from prior knowledge in terms of both interpretability and performance on downstream tasks.
+ Add a more detailed figure to illustrate the Gaussian SawETM and TopicNet models.
+ Clarify the other points raised in the reviews.
Decision: Accept (Poster)