Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a meta-complement topic model (MCTM) to improve the quality of discovered topics in short texts. The key idea is to transfer the semantic knowledge learned on long documents to short texts via meta-learning. Some previous studies rely on external data, such as pre-trained word embeddings and document citation networks, to mitigate the scarcity of word co-occurrences in short texts. In contrast, this work proposes to solve this problem within the corpus itself. The authors also show how to generalize their framework when word embeddings or citation networks are available. Experiments on 4 datasets demonstrate the effectiveness of MCTM in text classification, link prediction, topic coherence, and perplexity.
strengths_and_weaknesses: Strengths:
+ The idea of using meta-learning to deal with short texts in topic modeling is novel and intuitive from my perspective. According to Section 3.4, it is also straightforward to extend the model when external signals are available, indicating the flexibility of MCTM.

+ The empirical study is comprehensive. Besides the comparison of common topic modeling metrics (i.e., NPMI and perplexity), the authors also explore downstream tasks such as document classification and link prediction (given that the model can be extended to utilize document links).

+ Ablation studies demonstrate the contribution of different components in model design.

Overall, this is a solid submission in my view. However, I still need to explain the following minor concerns.

Weaknesses:
- Significance tests are missing in all result tables. It is unclear whether your improvement over baselines/ablation versions is statistically significant or not. In fact, in Tables 3 and 7, some gaps are quite subtle. Please conduct t-tests and report p-values. (Also, please order all tables according to their number. For example, Table 1 should be presented before Table 2.)

- For the evaluation of topic modeling, the adopted metrics are NPMI and perplexity. However, a recent study [1] shows that such metrics may not align well with human judgment. I wonder if more human annotations can be included (e.g., the intrusion test) when evaluating topic qualities.

[1] Is Automated Topic Model Evaluation Broken? The Incoherence of Coherence. NeurIPS'21.
questions: - Could you conduct t-tests in your result tables and report whether your improvement over baselines/ablation versions is statistically significant or not?
limitations: The authors mention the technical limitations in Section 6.

They also mention the potential negative societal impact of their work in Section 6.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper focuses on improving topic modeling performance on short documents. Short documents have fewer word co-occurrences that results in worse topic quality. The paper assumes a corpus of documents of various lengths and leverage the longer document to complement the semantics of the short text. They intend to improve the performance on short text without hurting the performance on the longer text. Their model can also work with auxiliary data such as pre-trained word embeddings and the document connectivity structure. Experiments demonstrate improvement of classification accuracy on Cora-ML and Cora-PL, with shorter documents benefit more. Topic coherence and perplexity is evaluated and reported on Cora-ML, Cora-PL, HEP-TH, and Web. The proposed models outperform other methods on Cora-ML and Cora-PL and are similar to baseline methods on HEP-TH and Web.
strengths_and_weaknesses: The paper addresses a focused problem: how to improve the performance on short document when doing topic modeling, which relies on the co-occurrence of words. The semantics transfer and complement seems to have good improvement on the performance.
questions: 1. If I read correctly, in Table 2, it seems MCTM with plain texts is performing better than MCTM with auxiliary word embeddings. I'm wondering if there is any explanation of why auxiliary information (word embeddings) is not helping. (The presentation of Table 2 could be improved by adding the description of each group (Topic models with plain text, etc.)

2. I'm wondering if Table 6 can be re-organized and adding the number of without both semantic complement and meta-learning. It seems the effect of semantic complement is bigger.

3. I wonder if semantic complement could be viewed as a prediction/generation problem (given z_d', predict m_d') and utilize the pre-trained large language model for such prediction.
limitations: I'm assuming that if the length of the document is not uniform across the topics, for example, certain topics only contains short documents, it might be a problem for the proposed method. I'm wondering if the author has done analysis regarding this.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: In this paper, the authors address the limitation of previous topic models in modeling short texts. Their analysis demonstrates a decreasing classification accuracy with decreasing text lengths, due to the lack of word co-occurrences in short texts. They propose to complement the semantics of short text, by training a "missing semantic prediction" model on long text. Specifically, a proportion of a long text is masked and the missing semantic prediction model is expected to output a complementary representation ("missing semantics"), which adds to the representation of masked long text to reconstruct the representation of the full long text. The authors propose two configurations of the missing semantic prediction model, which are based on GNN and clustering respectively, and GNN seems to be a slightly better choice. It can also be extended to leverage external data, such as pre-trained word embeddings or document networks for further improvement. Experimental results show significant improvement in document classification, topic coherence, perplexity, and link prediction, for both long and short texts.
strengths_and_weaknesses: Strengths:
1. The paper is well motivated with quantitative analysis to demonstrate the disadvantage of previous topic models in handling short texts.
2. The method is novel in complementing the semantics of short text to overcome the lack of word co-occurrences in short texts.
3. Overall, the paper is well written with a comprehensive explanation of the method and formulations.
4. The experimental results show significant improvement, and the ablation studies are sufficient to justify the design choices.

Weaknesses:
1. Some important hyperparameter details are missing, such as the threshold for defining short text and long text, the percentage of masking long text, negative sampling rate, etc. 
2. The explanation of the meta-learning part is unclear. How do you optimize formulas 1 and 2? Also, it might be better to merge the paragraphs on meta-learning at the end of Section 2 and Section 3.3 for easier understanding.
questions: 1. The authors choose to complement document representations. However, it might be worth exploring a textual level complement, such as using a language model to generate extra sentences on short texts for richer semantics.
2. Is the constraint in formula (10) met? In other words, will the model still complement relatively long texts, or keep them unchanged?
3. It is quite surprising to observe a large improvement in long texts (0-25%) classification as well. What could be the reason for improvement? Will there be such an improvement if we take a look at more extremely long texts (0-5%)?
limitations: The authors did not include a discussion on limitations or ethical impact.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: A number of issues this study are handled are listed as follows.
1. Most of the previous topic models treated the document with various length uniformly.
2. Short documents may only have few words co-occurrences which will make the model hard to identify the corresponding topics.
3. Previous solutions dealt with the short documents require external auxiliary data.
strengths_and_weaknesses: Pros
1. The proposed method showed good performances in their evaluation.
2. Ablation study was conducted.

Cons
1. Content organization was not well adjusted. Figure only showed the limited information and there is no Algorithm. Therefore, it was quite hard for the readers to understand the learning process.
2. The methodology was not well-explained:
  (1) In the contrastive loss, “What is the meaning of contrastive in the loss function?, because there are no positive negative pairs” and “What is the intuition of using cosine similiraity as an input for softmax function?”
  (2) In the generative loss, " what is the function of m?", It just described as number of negative samples, but what is the role of m in the equation?
3. The proposed method was not evaluated on the dataset commonly used in the mentioned previous works, which is 20newsgroup. Therefore, it’s hard to say that the proposed method could outperform the previous solutions.
questions: The proposed method has clear motivation that it wants to treat short documents and the long documents uniquely. Therefore, more semantics can be extracted from the short documents to increase model performance in classification and topic coherence. Meta-learning is also introduced to deal with the limited data. However, the learning criterions used in this work is unclear and the organization is not well adjusted. The figure only delivers limited information. Since the learning process is quite complicated, it is suggested to provide Algorithm.
limitations: The novelty is not sufficiently justified.
review_title: null

Meta-review: metareview: Most of the reviewers were content with the paper and the interactions with the authors.   I was personally convinced by the response given to the fourth reviewer, primarily about the 20NG dataset, and hence suggesting acceptance.
confidence: Certain
Decision: Accept