Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper proposes an improved embedded topic model that considers an existing knowledge about the topic hierarchy that can potentially alleviate biased and long-tail topics and words. It considers such hierarchy information as a matrix representing the interactions of topics following a Bernoulli distribution. The experiments include various aspects of the approach. They use four datasets and compare the proposed approach with five baselines. Also, the interpretability and the coherence of the topic is empirically shown and examples are provided.
strengths_and_weaknesses: Strengths.
- S1: The proposed approach provides a way to incorporate the topic hierarchy from an external knowledge source.
- S2: The approach was extensively evaluated with five baselines and four datasets.
- S3: The examples and interpretability of the derived topics and their hierarchy are provided to understand the outcome of the approach.

Weaknesses.
- W1: The presentation can be improved. There is no overview of the approach to explain the components, and a few components and concepts appear without much prior context. For example, "encoder" appears without where it is exactly being used. Same for "topic aggregation". "count vector" was used only once without definition.
- W2: This paper does not provide the related work description for the existing ETM this work is based on in the related work section. It also doesn't distinguish that in the method section. Thus, it is hard to evaluate the novel contribution of this paper compared to the existing approach. For example, much of the framework comes from the SawTooth paper, but this paper failed to include or summarize the overall structure.
- W3: Many parts of the paper are not justified or explained enough. For example, how modeling the relation with a Bernoulli distribution complement Phi modeling the relations between layers in SawTooth, or why the proposed approach is better was not explained. How is the adaptive structure different from just fine-tuning the parameters? The paper quickly attributes the improved performance to the joint tree likelihood, but how exactly is it better than the other approaches such as SawTooth or TopicNet? What information does it capture that SawTooth and TopicNet do not?

questions: I appreciate the authors for the efforts to incorporate prior knowledge to topic modeling. While there is some evidence of the improved performance, it was not sufficiently explained in general. Please refer to W1-W3 to address these questions. Especially, deep diving why this modeling is better than the baselines is very important to understand where the strength is coming from. Also, clarifying what comes from an existing approach, and what is novel can improve the readability of the paper greatly, and that way, this paper doesn't need to try to explain everything.

Additional comments/questions follow.
- Q1. A GCN was used to consider the structure of the topic tree and update the node weights. How was the neural network training and the Bayesian model optimization done together?

- Typos: EBLO -> ELBO. consists -> consists of. widely-used -> widely used. two adjacent layer -> two adjacent layers.
limitations: I don't see a negative societal impact of this work.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The goal of this paper is to learn topic hierarchies by deriving knowledge from a pre-existing hierarchy of topics. This pre-existing hierarchy does not have topical words in the nodes.

To this end, the authors develop a Bayesian deep topic model. In the topic model, the Bayesian component is the topic tree that is used as a piece of prior knowledge of the model. The deep part is derived from the Gamma Belief Network framework.  Later, the authors used GCN-based methods to model the hierarchies. Parameter estimation is then presented using amortized variational inference model. Experimental results demonstrate that the model improves upon existing methods.
strengths_and_weaknesses: The key strength lies in modelling topic hierarchies using side information. To this end, a hierarchy is used as prior knowledge and words are generated in each hierarchy.

The main weakness of the work is that it relies upon a knowledge base. While WordNet is "relatively smaller" in size compared to knowledge bases such as Wikidata, where the model will struggle is when the knowledge bases are huge, e.g., several gigabytes in size. It will be good for the authors to mention which hierarchies the model can work and which it cannot.

While the authors have presented their detailed results, the question is why this work is important? The issue is that the model relies on a hierarchy that it supplied manually. There are already unsupervised topic models such as hLDA that can learn hierarchies, of course, not in the way that the authors have presented their work. However, it is still unclear how this work is a step-change in the topic modelling literature.
questions: Can the authors mention some real-world applications where this work could be useful? How and why this model can be applied to real-world tasks? Section 5.8 only uses the 20NG dataset that is very small.
limitations: The key issue is that the model relies on an external knowledge base. One might need knowledge bases in different domains to use this model.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes a topic modeling approach that extends previous work on Gamma Belief Network [1] to jointly model a knowledge graph provided as input to guide the topics. The graph structure and the keywords representing each graph node is integrated into the generative process. The graph node embedding is modelled using a graph neural networks, which is jointly modelling during the inference process. This allows data driven changes to the knowledge graph, as illustrated as the addition of new topics in the experimental results.

[1] Sawtooth Factorial Topic Embeddings Guided Gamma Belief Network. Duan et al. ICML 2021

strengths_and_weaknesses: Strength
- the proposed approach of guiding a topic model with a knowledge graph seems more natural than previous work such as [1], and in their evaluation, was shown to achieve better evaluation results compared to previous work.

- the proposed model allowed the graph structure to evolve according to the data in a variant called TopicKGA, which is the best performing model

- they showed that their approach converges better than TopicNet [1]


Weaknesses
- more ablation could be done on how GNN contributes in the model. The dynamic structure of the topic graph is only shown in the addition of new topics. Was there any changes in the graph structure other than addition of nodes? 

- using wordnet as the knowledge graph seems too general a way to guide a topic model. Knowledge graphs such as those derived from Wikidata or KG more specifically customized to user needs could lead to more practical usage.


[1] TopicNet: Semantic Graph-Guided Topic Discovery. Duan et al. 2021.
questions: Other than addition of new nodes, does the graph structure evolve? e.g., are edges added or removed?
limitations: - as rightly pointed out in the appendix, it will be interesting to investigate how the knowledge in large language models can be used in such topic modeling works
review_title: null

Meta-review: metareview: Learning hierarchies of themes has been a challenging field that saw considerable work about ten years ago but has largely been abandoned due to poor results and inefficient algorithms. I'm interested to see this and other work reviving interest. My prior for hierarchical topics is that they are difficult to construct because the search space is large and doing a good job requires prioritization and abstraction at a high level of AI. I like the semi-supervised approach here as a heuristic to guide the construction of hierarchies. WordNet is often difficult to apply in practice, so I share the reviewers interest in alternatives that would scale.

Reviewer complaints largely centered around clarity. The authors provided extensive responses which the reviewers found compelling and satisfactory. There is generally solid to strong support for the paper.
confidence: Certain
Decision: Accept