Neural Topic Modeling with Large Language Models in the Loop
Authors: Xiaohao Yang, He Zhao, Weijie Xu, Yuanyuan Qi, Jueqing Lu, Dinh Phung, Lan Du
Accepted at: acl25


Reviewer paper summaries: The work proposes a novel topic modeling strategy (LLM-ITL) based on the integration of LLM with (neural) topic modeling approaches. Briefly, the topic modeling task involves creating descriptions of document collections where each topic has a unique theme and documents can be represented as a combination of topics. 

The authors begin the text by highlighting the importance and practical applications of the studied subject. Then, they introduce the limitations and characteristics of current approaches, especially regarding (i) the challenge of covering global topics, often faced by LLMs that focus on individual documents; and (ii) the challenge of considering multiple topics in long documents. The traditional and neural approaches are indeed challenging to manage this characteristic.

Given this limitation, the authors introduce LLM-ITL, a framework that incorporates LLMs into NTMs to improve the quality and interpretability of learned topics. This is achieved by learning global topics and document representations through neural topic modeling techniques. Simultaneously, an LLM fine-tunes these topics utilizing an Optimal Transport alignment. The authors argue that their proposed method surpasses the performance of existing methods in topic modeling. 

In summary, this research exhibits nice goals and innovative ideas, demonstrating notable potential. With minor changes addressed, I recommend accepting this paper.

After rebuttal: 

Thank you for your detailed responses and for incorporating additional experiments based on my suggestions.

I appreciate the inclusion of NMF as a non-neural baseline and the commitment to incorporating CluWords in the revised version. 
Regarding the evaluation metrics, I acknowledge the inclusion of Word2Vec-based coherence measures. 

Overall, these additions address my concerns and contribute to a more comprehensive assessment of the proposed method. 
I'm looking forward to seeing the revised version.
Summary of strengths: S1: The paper is well-written, concise, and straightforward. The authors clearly define the evaluated objectives, motivation, and contributions.

S2: The implementation details and method-specific hyperparameters were defined. Thus, the paper is (possibly) reproducible.

S3: The considered datasets are well-known and widely used in the topic modeling literature (20Newsgroup, Reuters, DBpedia, and AGNews).

S4: The authors discuss recent applications and related research on LLMs for Topic Modeling, including Neural Topic Modeling approaches.

S5: In terms of baselines, the proposed method was compared with 8 traditional and recent approaches, including (LDA, BERTopic, TopicGPT). The baselines were very well explained in a simple and straightforward way.

S6: The authors present a comprehensive set of experiments. Moreover, all experiments were conducted with multiple runs (five times) to assess the proposed approach's generalizability.
Summary of weaknesses: In general, the paper has a good idea and a good novelty factor. However, I have some suggestions for improvements listed as follows:

W1: Although the authors evaluate their proposal by comparing it with 8 topic modeling strategies, I missed the comparison with some non-neural approaches, which are still considered state-of-the-art for topic modeling [1]. To enhance the paper's findings, it would be valuable to include models such as NMF [2] and CluWords [3].

[1] Churchill, R., & Singh, L. (2022). The evolution of topic modeling. ACM Computing Surveys, 54(10s), 1-35.
[2] Wang, Y. X., & Zhang, Y. J. (2012). Nonnegative matrix factorization: A comprehensive review. IEEE Transactions on Knowledge and data engineering, 25(6), 1336-1353.
[3] Viegas, F., Canuto, S., Gomes, C., Luiz, W., Rosa, T., Ribas, S., ... & Gonçalves, M. A. (2019, January). CluWords: exploiting semantic word clustering representation for enhanced topic modeling. In Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining (pp. 753-761).

W2: Lack of Comparison with Robust Metrics: While CV and PN (average between Purity and NMI) are presented as definitive evaluation measures, comparing their performance against established metrics, such as TF-IDF coherence, and Word2vec metrics [4], would be valuable.

[4] Nikolenko, S. I. (2016, July). Topic quality metrics based on distributed word representations. In Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval (pp. 1029-1032).
Comments, suggestions and typos: See above.
Overall assessment scores: 4.5
Metareview score: 4
Metareview: This paper proposes a method called LLM-ITL to improve topic models by leveraging LLMs. Specifically, automatically-inferred topics are further refined by topic word suggestions by LLM, where the topic alignments are computed through Optimal Transport metric. All the reviewers agreed with the novelty of this method, but note that the absolute performance of topic coherence and alignment basically stays comparative with the vanilla LDA, while better than neural topic models.
Summary of reasons to publish: - As all the reviewers note, this paper proposes a clear metric of OT-based topic alignment for LLM intervention.
- This paper conducts extensive experiments over many baselines, and further reinforced by additional experiments in the author rebuttal.