Revisiting Automated Topic Model Evaluation with Large Language Models
Authors: Dominik Stammbach
Vilém Zouhar
Alexander Hoyle
Mrinmaya Sachan
Elliott Ash
Venue: EMNLP 2023 Main (2023)
Keywords: topic model evaluation
interpretability
large language models
text clustering
TL;DR: Automatically evaluating topic model output has been a longstanding challenge; we find that large language models turn out to be very good at this task.

Abstract: Topic models help us make sense of large text collections. Automatically evaluating their output and determining the optimal number of topics are both longstanding challenges, with no effective automated solutions to date. This paper proposes using large language models (LLMs) for these tasks. We find that LLMs appropriately assess the resulting topics, correlating more strongly with human judgments than existing automated metrics. However, the setup of the evaluation task is crucial — LLMs perform better on coherence ratings of word sets than on intrustion detection. We find that LLMs can also assist us in guiding us towards a reasonable number of topics. In actual applications, topic models are typically used to answer a research question related to a collection of texts. We can incorporate this research question in the prompt to the LLM, which helps estimating the optimal number of topics.

Revisiting Automated Topic Model Evaluation with Large Language Models
Dominik Stammbach
Vilém Zouhar Alexander Hoyle"
Mrinmaya Sachan
Elliott Ash
ETTH Zürich
"University of Maryland
(dominsta, vzouhar, ashe, msachan @ethz. ch
hoyle und. edu
Abstract
in the social sciences, digital humanities, and other disciplines to analyze large corpora (Talley et al., 2011; Grimmer and Stewart, 2013; Antoniak et al.,
Topic models help make sense of large text col- lections. Automatically evaluating their output and determining the optimal number of topics are both longstanding challenges, with no ef- fective automated solutions to date. This paper evaluates the effectiveness of large language models (LL.Ms) for these tasks. We find that LLMs appropriately assess the resulting top- ics, correlating more strongly with human judg- ments than existing automated metries. How- ever, the type of evaluation task matters LLMs correlate better with coherence ratings of word sets than find that LLMs can also guide users toward reasonable number of topics. In actual applica- tions, topic models are typically used to answer a research question related to a collection of texts. We can incorporate this research ques- tion in the prompt to the LLM, which helps estimate the optimal number of topics.
2019; Karami et al., 2020, inter alia).
Interpretability makes topic models useful, but human interpretation is complex and notoriously difficult to approximate (Lipton, 2018). Automated topic coherence metrics do not correlate well with human judgments, often overstating differences be- tween models (Hoyle et al., 2021; Doogan and Bun- tine, 2021). Without the guidance of an automated metric, the number of topics, an important hyperpa- rameter, is usually derived manually: Practitioners fit various topic models, inspect the resulting top- ics, and select the configuration which works best for the intended case (Hoyle et al., 2021). This is non-replicable and time-consuming process, requiring expensive expert labor.
word intrusion task. We
Recent NLP research explores whether large lan- guage models (LLMs) can perform automatic anno- tations; e.g., to assess text quality (Fu et al., 2023; Faggioli et al., 2023; Huang et al., 2023, inter alia). Here, we investigate whether LLMSs can automati- cally assess the coherence of topic modeling output and conclude that:
gåthub. com/ coom/domini www.nikkiinsseeee.o.......imiiie and/ eva wnstttnnnttttt c-model -uutuut.
1 Introduction
Topic models are, loosely put, an unsupervised dimensionality reduction technique that help orga- nize document collections (Blei et al., 2003). topic model summarizes a document collection with small number of topics. A topic is proba- bility distribution over words or phrases. A topic T is interpretable through representative set of words phrases defining the topic, denoted W7. Each document can, in turn, be represented distribution over topics. For each topic, we can retrieve representative document collection by sorting documents across topic distributions. We denote this set of documents for topic T as Dr. Because of their ability to organize large collections of texts, topic models are widely used
(1) LLMs can accurately judge topic coherence, (2) L.LMs can assist in automatically determining reasonable numbers of topics.
We use LLMs for two established topic coher- ence evaluation tasks and find that their judgment strongly correlates with humans on one these tasks. Similar to recent findings, we find that coher- ent topic word sets Wy do not necessarily imply an optimal categorization of the document collection (Doogan and Buntine, 2021). Instead, we automati- cally assign label to each document in a Dr and choose the configuration with the purest assigned labels. This solution correlates well with under- lying ground truth. Thus, LLMs can help find good numbers of topics for a text collection, as we show in three case studies.
'We think of "words" as an atomic unit in document, which can also he an n-gram or phrase. E.g., Waza = (litiga- tion, attorne--cliem privilege, intellectual property.....

---

2 Topic Model Evaluation
work by Rahimi et al. (2023) is carried contempo- rancously. LLMs have already been used to rank machine translations and generated text (Zhang et al., 2020; Fu et al., 2023; Koemi and Federmann, 2023) and have also been shown to perform on par with crowdworkers (Gilardi et al., 2023).
Most topic model evaluations focus the coher- ence of Wr, the most probable words from the topic-word distribution (Röder et al., 2015). Co- herence itself can be thought of as whether the top words elicit a distinct concept in the reader (Hoyle et al., 2021). To complicate matters, human evaluation of topic models can be done in diverse ways. E.g., we can ask humans to directly rate topic coherence, for example, man et al., 2010a; Mimno et al., 201 I; Aletras and Stevenson, 2013, inter alia). We can also add an unrelated intruder word to the list of top words, which human annotators are asked to identify. The intuition is that intruder words are easily identi- fied within coherent and self-contained topics, but hard to identify for incoherent or not self-contained topies (Chang et al., 2009). High human accuracy on this task is thus good proxy for high topic coherence. See both in Example 1.
Task
Dataset
NPMI
Cy
LLM
Ceiling
NYT
0.43
0.451
0.37
0.67
Intrusion
Wiki
0.391
0.34
0.35
0.60
1-3 scale (New-
Both
0.401
0.40
0.36
0.64
NYT
0.48
0.40
0.64*
0.72
Rating
Wiki
0.44
0.40
0.57*
0.56
Both
0.44
0.42
0.59*
0.65
Table Spearman correlation between human scores and automated metrics. All results use 1000 bootstrap- ping episodes - re-sampling human annotations and I.LM scores, and averaging the correlations. Marked if significantly better than second best (<0.05), otherwise f. Ceiling shows batched inter-annotator agreement.
LLM and Coherence
Intrusion Detection Task
water area
river
park
miles
game
First, we show that large language models can as- sess the quality of topics generated by different topic modeling algorithms. We use existing topic modeling output annotated by humans (Hoyle et al., 2021). This data consists of 300 topics, produced by three different topic modeling algorithms datasets: NYtimes (Sandhaus, 2008) and Wikitext (Merity et al., 2017). For each of the 300 topics, there are 15 individual human annotations for the topic word relatedness (on 1-3 scale), and 26 in- dividual annotations for whether a crowd-worker correctly detected an intruder word. We replicate both tasks, prompting LLMs instead of human an- notators. See Prompt for prompt excerpts, and Appendix for full details.
horses
horse breed
hindu coins
silver
Rating Task
health
hospital
medicare
welfare insure
3
horses
zurich
race
dog
canal
Example 1:
Two examples of intrusion detection (select
outlier) and topic rating tasks (rate overall coherence). Each example is on separate row.
Although many automated metrics exist (Wal- lach et al., 2009; Newman et al., 2010b; Mimno et al., 2011; Aletras and Stevenson, 2013), normal- ized pointwise mutual information (NPMI, Bouma, 2009) is the most prevalent when evaluating novel methods (Hoyle et al., 2021). Informally, NPMI is larger two words co-oceur together regularly in a reference corpus. Another popular metric, Cv, is a combination of NPMI and other measures and also popular (Röder et al., 2015). See the formula definitions in Appendix D. Despite their popular use, these metrics corre- late poorly with human evaluations (Hoyle al., 2021: Doogan and Buntine, 2021). In this work. we let LLMs perform the rating and intrusion detec- tion tasks for topic model evaluation and propose LLM scores as a novel automated metric. Similar
Intruder detection prompt.
System prompt: [--] Select which word is the least related to all other woods. most of place. [...]
multiple words do not fit, User prompt: water, area, river, park, miles, game
Rating Task prompt.
System prompt: [..]] Please mite related following are each other on a scale from I to 3 ("1" not very related, "2" moderately related, "3" very related), L..]
User prompt: lake, park, river, land, years, feet, ice, miles, water, area
Prompt 1: LLM prompts for assessing topic coherence.
We compute the Spearman correlation between the LLM answer and the human assessment of the topics and show results in Table 1.
We use ChatGPT as the main I.I (chat.openai.com) We list ablation results using other LLMS in Appendix B.
Models: Gibbs-I.DA (McCallum, 2002), Dirichlet-VAF (Burkhardt and Kramer, 2019), and ETM (Dieng et al., 2020).

---

Baseline metrics. For NPMI and Cv. we report the best correlation by Hoyle al. (2021). These metries depend on the reference corpus and other hyperparameters and we always report the best value. Hoyle et al. (2021) find no single best setting for these automated metrics and therefore this com- parison makes the baseline inadequately strong.
signments with greater label purity correlate with the ground-truth in three case studies.
Topics of interest might be a few broad topics such as politics or healthcare, or many specific topics, like municipal elections and maternity care. Following recent efforts that use research questions to guide LLM-baset text analysis (Zhong et al., 2023), incorporate this desideratum in the LLM prompt. We run collapsed Gibbs- sampled LDA (in MALLET: McCallum, 2002) on two text col- lections, with different numbers of topics (k 20 to 400), yielding 20 models per collection. To compare topic model estimates and ground-truth partitions, we experiment with a legislative Bill summary dataset (from Hoyle et al., 2022) and Wikitext (Merity et al., 2017), both annotated with ground-truth topic labels different granularities.
Intrusion detection task. The accuracies for de- tecting intruder words in the evaluated topics are almost identical humans correctly detect 71.2% of the intruder words, LLMs identify intruders in 72.2% of the cases. However, humans and LLMs differ for which topics these intruder words are identified. This results in overall strong correla- tions within human judgement, but not higher cor- relations than NPMI and Cv (in their best setting).
Coherence rating task. The LLM rating of the WT top word coherence correlates more strongly with human evaluations than all other automated metries in any setting. This difference statis- tically significant, and the correlation between LLM ratings and human assessment approaches the inter-annotator agreement ceiling. Appendix Appendix B shows additional results with different prompts and LLMs.
4.1 Proposed Metries
Ratings algorithm. For each of the 20 models, randomly sample Wr for some topics and let the LLM rate these Wr. The prompt is similar to the ratings prompt shown in Prompt 1, see Ap- pendix for full details. We then average ratings for each configuration. Intuitively, the model yield- ing the most coherent W,, should be the one with the optimal topic count. However, this procedure does not correlate with ground-truth labels.
Both findings support using LLMs for evaluating coherence of Wr in practice as they correlate highly with human judgements.
Recommendation.
Text labeling algorithm. Doogan and Buntine (2021) propose that domain experts assign labels to each document in Dr instead. A good topic should have a coherent Dr: The same label as- signed most documents. Hence, good configu- rations have high purity of assigned labels within each topic. We proceed analogously. For each of the 20 models, we randomly sample Dr for various topics. We retrieve the 10 most probable documents and then use the LLM to assign a label to these documents. We use the system prompt [...] Annotate the document with broadInarrow label [...], Appendix E for full details. We compute the purity of the assigned labels and average puri- ties and we select the configuration with the most pure topies. In both procedures, we smooth the LLM outputs using a rolling window average to re- duce noise (the final average goodness is computed as moving average of window of size 3).
4 Determining the Number of Topics
Topic models require specifying the number of top- ies. Practitioners usually run models multiple times with different numbers of topics (denoted by k). Af- ter manual inspection, the model which seems most suited for research question is chosen. Doogan al. (2023) review 189 articles about topic model- ing and find that common use cases are exploratory and descriptive studies for which no single best number of topics exists. However, the most preva- lent use case is to isolate semantically similar doc- uments belonging to topics of interest. For this, Doogan and Buntine (2021) challenge the focus on only evaluating Wr. and suggest an analysis interested in organizing
of Dr as well. If collection, then we would expect the top docu- ments in DT receive the same topic labels. We provide an L.LM -based strategy to determine good number of topics for this use case: We let an LLM assign labels to documents, and find that topic as-
4.2 Evaluation
We need human-derived metric to compare with the purity metric proposed above. We measure

---

BillSum, broad, = 0.59 Pw = 0.61
Wikitext, broad, 0.72 Pw= 0.33 Wikitext, specific, Pa
0.72 Pw
0.02
e Word LLM ARI e Doe LLM
aooa
oog
6
0.4
0.7
Document
0.6
0
140 200
260
320
380
20
80
140
200
260
320
380
20
200 260 320 380
Number of topics
Number of topics
Number of topics
Figure 1: (I) ARI for topic assignment and ground-truth topic labels, (2) LLM word set coherence, (3) LLM document set purity, obtained by our algorithm. ARI correlates with LLM document set purity, but not with LLM word set coherence. The ground-truth number of topics are: 21 topics in the BillSum dataset, 45 broad topics in Wikitext and 279 specific topies in Wikitext. PD and pw are document-LLLM and word-LL.M correlations with ARI.
the alignment between a topic model's predicted topic assignments and the ground-truie truth labels for a document collection (Hoyle et al., 2022).
strategy incorporates different research questions. For Bills, our rating algorithm suggests to use a topic model with =0100 topics. In Appendix G, we show corresponding word sets. The resulting Wr seem interpretable, although the ground-truth as- signments using document-topic estimates are not correlated with the ground-truth labels. The purity- based approach instead suggests to use k=20 top- ics, the same as indicated by the ARI. We show ground-trut labels and LLM-obtained text labels in Appendix G. We further manually evaluate 180 assigned LLM-labels and find that 94% of these labels are reasonable. Appendix F shows further evaluation of these label assignments.
We choose the Adjusted Rand Index (ARI) which compares two clusterings (Hubert and Arabie, 1985) and is high when there is strong overlap. The predicted topic assignment for each document is its most probable topic. Recall that there exist many different optimal topic models for single collec- tion. If we want topies to contain semantically similar documents, each ground-trute truth assignment reflects one possible set of topics of interests.
If our LLM-guided procedure and the ARI corre- late, this indicates that we discovered a reasonable value for the number of topics. In our case, the various ground truth labels are assigned with dif- ferent research questions in mind. We incorporate such constraints in the LLM prompt: We specify whether we are interested in broad or specific top- ies, and we enumerate some example ground-trute truth categories in prompt. Practitioners usually have priors about topics of interest before running topic models, thus we believe this setup be realistic.
Discussion
In this work, we revisit automated topic model evaluation with the help of large language mod- els. Many automated evaluation metrics for topic models exist, however these metrics seem to not correlate strongly with human judgment on word- set analysis (Hoyle et al., 2021). Instcad, we find that an LLM-based metric of coherent topic words correlates with human preferences, outperforming other metrics the rating task.
In Figure 1 show LLM scores and ARI for broad topics in the Bills dataset. We used this dataset find suitable prompt, hence this could be considered the "training set",, We plot coherence ratings of word sets in blue purity of document labels in (red), and the ARI between topic model and ground- fround-truth truth assignments in (green). The pu- rity of LLM-assigned D, labels correlate with the ARI, whereas the Wr coherence scores do not. The argmax of the purity-based approach leads to sim- ilar numbers of topics as suggested by the ARI argmax (although not always the same).
Second, the number of topics k has to be defined before running a topic model, practitioners multiple models with different k. We investigate whether LLMs can guide us towards reasonable k for a collection and research question. We first note that the term optimal number of topics is vague and that such quantity does not exist without additional context. If our goal is to find configuration which would result in coherent document sets for topics, our study supports evaluating Dr instead of Wr, as this correlates more strongly with the overlap between topic model and ground-truth assignment. This finding supports arguments made in Doogan and Buntine (2021) who challenge the focus on Wy in topic model evaluation.
For Wikitext, we evaluate the same 20 topic mod- els, but measure ARI between topic model assign- ment and two different ground-truth label sets. The LLM scores differ only because of different prompt- ing strategies. The distributions indicate that this

---

Limitations
Ethics Statement
Choice of LLM. Apart from ChatGPT, we also used open-source LLMs, such as FLAN-T5 (Chung et al., 2022), and still obtained reasonable, al- beit worse than ChatGPT, coherence correlations. Given the rapid advances, future iterations of open- source LLMs will likely become better at this task.
Using blackbox models in NLP. Statistically sig- nificant positive results are a sufficient proof of models' capabilities, assuming that the training data is not part of the training set. This data leak- age problem with closed-source LLMs is part of a bigger and unresolved discussion. In our case, we believe data leakage unlikely. Admittedly, the data used for our coherence experiments has been publicly available. However, the data is available in large JSON file where the topic words and anno- tated labels are stored disjointly. For our case stud- ies in Section 4, the topic modeling was constructed as part of this work and there is no ground-truth which could leak the language model.
Number of topies. The optimal number of topics is vague concept, dependent a practitioner's goals and the data under study. At the same time, it is required hyperparameter of topic models. Based on Doogan et al. (2023), we use an existing document categorization as one possible ground truth. While content analysis is the most popular application of topic models (Hoyle et al., 2022), it remains an open question how they compare to alternative clustering algorithms for this use (e.g., k-means over document embeddings).
Negative results with LLMs. In case of negative results, we cannot conclude that a model can not be used for a particular task. The negative results can be caused by inadequate prompting strategies and may even be resolved by advances in LLMs.
Interpretability. LLM label assignment and in- truder detection remain opaque. This hinders the understanding of the evaluation decisions.
LLMs and biases. LLMs are known to be bi- ased (Abid et al., 2021; ; Lucy and Bamman, 2021) and their usage in this application may potentially perpetuate these biases.
Topic modeling algorithm. In Section 3, we evaluate three topic modeling algorithms: Gibbs- LDA, Dirichlet-VAE and ETM (see Hoyle et al., 2021). In Section 4, we use only Gibbs-LDA and expansion to further models is left for future work.
Data privacy. All data used in this study has been collected part of other work. We find no potential violations of data privacy. Thus, we feel comfortable re-using the data in this work.
Future work.
Evaluation of clustering algorithms with LLMs
(e.g., k-means).
More rigorous evaluation of open-source LLMs. Formalization, implementation and release of an LLM-guided algorithm for automatically finding optimal numbers of topics for text collection and a research question.
Misuse potential. We urge practicioners to not blindly apply our method on their topic modeling output, but still manually validate that the topic out- puts would be suitable to answer a given research question.
References
David M. Blei, Andrew Y. Ng. and Michael I. Jordan. 2003. Latent dirichlet allocation. Mach, Learn. Res., 3(nul)::933110) 10222
Abubakar Abid, Maheen Farooqi, and James Zou. 2021, Persistent anti- muti-muslim bias large language models. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pages 298-306.
Gerlof J. Bouma. 2009. Normalized (pointwise) mutual information in collocation extraction.
Sophie Burkhardt and Stefan Kramer. 2019. Decoupling sparsity and smoothness in the dirichlet variational autoencoder topic model. Journal of Machine Learn- ing Research, 20(131):1-27.
Nikolaos Aletras and Mark Stevenson. 2013. Evaluat- ing topic coherence using distributional semantics. In Proceedings of the 10th international conference on computational semantics (IWCS 2013)-Long Papers, pages 13-22.
Jonathan Chang, Sean Gerrish, Chong Wang, Jordan Boyd-graber, and David Blei. 2009. Reading tea leaves:
Maria Antoniak, David Mimno, and Karen Levy. 2019, Narrative paths negotiation of power in birth sto- ries. Proc. ACM Hum Commut. Interact, 3(CSCW).
How humans interpret topic models. In Ad- vances Neural Information Processing Systems. volume 22, Curran Associates, Inc.

---

Yew Ken Chia, Pengfei Hong, Lidong Bing, and Sou- janya Poria. 2023. INSTRUCTEVAL: Towards holis- tic evaluation of instruction-tuned large language models.
Fan Huang, Haewoon Kwak, and Jisun An. 2023, Is chatgpt better than human annotators? potential and limitations of chatgpt in explaining implicit hate speech. arXiv preprint arXiv:3302.07736.
Hyung Won Chung, Le Hou, Shayne Longpre. Barret Zoph. Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Al- hert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdh- ery, Alex Castro-Ros Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu. Vincent Zhao, Yanping Huang. Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Ja- cob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. 2022. Scaling instruction-fincuuned language models.
Lawrence Hubert and Phipps Arabie. 1985. Comparing partitions. Journal of classification, 2:193-218.
Amir Karami, Morgan Lundy, Frank Webb, and Yo- gesh K. Dwivedi. 2020. Twitter and research: A sys- tematic literature review through text mining. IEEE Access, 8:67698-67717.
Tom Koemi and Christian Federmann. 2023, Large language models are state-of-the-art evaluators of translation quality.
Zachary C. Lipton. 2018. The mythos of model in- terpretability: In machine learning, the concept of interpretability is both important and slippery. Queue, 16(3):31-57.
Adji B. Dieng, Francisco J. R. Ruiz, and David M. Blei. 2020. Topic modeling in embedding spaces. Trans- actions of the Association for Computational Linguis-
tics, 8:439-453.
Li Lucy and David Bamman. 2021. Gender and rep- resentation bias in GPT-3 generated stories. In Pro- ceedings of the Third Workshop on Narrative Under- standing, pages 48-55.
Caitlin Doogan and Wray Buntine, 2021. Topic model topic twaddle? Re-evaluating semantic inter- pretability measures. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3824-3848, Online. Association for Computational Linguistics.
Andrew Kachites McCallum. 2002.
MALLET: A MAachine Learning for LanguagE Toolkit. http: / //mallet.cs.
umass. edu.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2017. Pointer sentinel mixture mod- els. In 5th International Conference on Learning Representations, ICLR 2017 Toulon, France, April 24-26, 2017, Conference Track Proceedings. Open- Review.net.
Caitlin Doogan, Wray Buntine, and Henry Linger. 2023. A systematic review of the use of topic models for short text social media analysis. Artificial Intelli- gence Review, pages 1-33.
Guglielmo Faggioli, Laura Dietz, Charles Clarke, Gi- anluca Demartini, Matthias Hagen, Claudia Hauff, Noriko Kando, Evangelos Kanoulas, Martin Potthast, Benno Stein, and Henning Wachsmuth, 2023. Per- spectives on large language models for relevance judgment.
David Mimno, Hanna Wallach, Edmund Talley, Miriam Leenders, and Andrew McCallum. 2011. Optimizing semantic coherence in topic models. In Proceed- ings of the 2011 conference empirical methods in natural language processing, pages 262-272.
Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. 2023. GPTScore: Evaluate as you desire.
David Newman, Jey Han Lau, Karl Grieser, and Timo- thy Baldwin. 2010a. Automatic evaluation of topic coherence. In Human language technologies: The 2010 annual conference of the North American chap- ter of the association for computational linguistics. pages 100-108.
Fabrizio Gilardi, Mcysam Alizadch, and Maël Kubli. 2023. ChaiGPT outperforms wrokd-workers for text- annotation tasks.
Justin Grimmer and Brandon M. Stewart. 2013. Text as data: The promise and pitfalls of automatic con- tent analysis methods for political texts. Political Analysis, 2772-7777
David Newman, Youn Noh. Edmund Talley, Sarvnaz Karimi, and Timothy Baldwin. 2010b. Evaluating topic models for digital libraries. In Proceedings of the 10th annual joint conference on Digital libraries, pages 215-224.
Alexander Hoyle, Pranav Goel, Denis Peskov, An- drew Hian-Cheong, Jordan Boyd-Graber, and Philip Resnik. 2021. Is automated topic model evaluation broken?: The incoherence of coherence.
Hamed Rahimi, Jacob Louis Hoover, David Mimno, Hu- bert Naacke, Camelia Constantin, and Bernd Amann. 2023. Contextualized topic coherence metrics.
Alexander Miserlis Hoyle, Rupak Sarkar, Pranav Goel, and Philip Resnik. 2022. Are neural topic models broken? In Findings of the Association for Computa- rional Linguistics: EMNLP 2022, pages 5321-5344, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Michael Röder, Andreas Both, and Alexander Hinneb- urg. 2015. Exploring the space of topic coherence measures. In Proceedings of the eighth ACM inter- national conference on Web search and data mining. pages 399-408.

---

Evan Sandhaus. 2008. The new york times annotated corpus.
and David Mimno. 2009. Evaluation methods for topic models. In Proceedings of the 26th annual in- ternational conference on machine learning, pages 1105-1112.
Edmund M Talley. David Newman, David Mimno, Bruce W Herr 2nd, Hanna M Wallach, Gully A P Burns, A G Miriam Leenders, and Andrew McCal- lum. 2011. Database of NIH grants using machine- learned categories and graphical clustering. Nat. Methods, 866):433-44.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q- Weinberger, and Yoav Artzi. 2020. BcRTS Evaluating text generation with BERT. In 8th In- ternational Conference on Learning Representations, ICLR 2020 Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Nguyen Xuan Vinh, Julien Epps. and James Bailey. 2010. Information theoretic measures for cluster- ings comparison: Variants, properties, normalization and correction for chance. J. Mach. Learn. Res.. 11:2837-2854.
Ruiqi Zhong, Peter Zhang, Steve Li, Jinwoo Ahn, Dan Klein, and Jacob Steinhardt. 2023. Goal driven dis- covery of distributional differences via language de- scriptions.
Hanna M Wallach, lain Murray, Ruslan Salakhutdinox,
A Language Model Prompts
We include a task and dataset description which leads to Prompt 4. The minimal prompt without the dataset description is shown in Prompt 5.
In this section, we show the used LLM prompts. The task descriptions are borrowed from (Hoyle et al., 2021) and mimic crowd We use a temperature of for LLMs, and the topic words are shuffled before being prompted. Both in- troduce additional variation within the results, sim- ilar to how some variation is introduced if different crowd-workers are asked to perform the same task.
instructions.
System prompt: You are a helpful assistant evalnating the top words of a topic midel output for a given topic. Please rate related following words are to each other on scale from I to 3 ("I"= not very related, "2" = moderately related, "3" very related).
The topic modeling is bused on the Wikipedia corpus. Wikipediu is an oaline encyclopedia covering huge range of topics. Articles can include biographies ("George Washington ") scientific phenomena (' ("Solar Belipse"), ), art pieces ("La Danse") I, music ( Ammziin Grace") ) transportation ("U,S, Route 131"), sports ("1952 winter olympics") historical events or periods ('Tang Dynusty") modia and pop culture (" Thee Simpsons Movic"), places ('Yosemite National Park") ) plants and animals ("koala"), and warfare ("USS Nevada (DB 36)"), among others. Reply with a single number, indicating the overall appropriateness of the topic.
Intruder detection task. Analogous to the hu- man experiment, we randomly sample (a) five word from the top 10 topic words and (b) additional intruder word from different topic which does not occur in the top 50 words of the current topic. We then shuffle these six words. We show the final prompt with an example topic in Prompt 2. We also construct a prompt without the datasct description (see Prompt and results in Table 2).
User prompt: lake, park, river, land, years, fect, ice, miles, water, area
Prompt 4: Rating Task. Topic terms are shuffled.
System prompt: You are a helpful assistant evaluating the top words of a topic model corput for given topic. Please rate how related the following words are each other scale
to 3 ("I" not very re related, "2" maderately related, "3" very related).
Reply single number, indicating the overall approriatenes of topic. User prompt: lake, park, river, land, years, feet, ice, miles, water, area
Prompt 5: Rating Task without dataset description. Topic terms are shuffled.
System prompt: You are helpful assistant evaluaring the top words of topic is least related words. If mulliple words do not fit, choose the word that is must out of place. corpus
model ourput for a given topic. Select
B Additional: Topic Model Outputs
The topic modeling based consists of articles from 1987 to 2007. Sections from typical paper include Intersaioonal, National, New York Regional, Business, Tochnology. and Sports news, features on topics such as Dining, Movies, Travel, Fashion; there are also obituaries and opinion pieces. Reply with a single word. User prompt: water. area, river, park, miles, game
The New York Times corpus.
Minimal prompt. Even without the dataset de- scription in the prompt, the results remain similar.
All human ratings. In our main results, we dis- card human annotations with low annotator con- fidence in the rating. We now consider all rat- ings, even the non-confident ones. The results are slightly better than with the filtering.
Prompt 2: Intruder Detection Task (the intruder word in this topic is game). We show the task description for the New York Times dataset in the prompt for the rating task (the dataset descriptions are kept the same).
System prompt: You are a helpful assistant ovaluaring the top words of topic model output for given topic. Select which word is the least related all other words. If multiple words do not lit,, choose the word that is most out of place.
Different LLM. We also evaluate both tasks with FLAN-TS XL (Chung et al., 2022), which is instruction finetuned across a range of tasks. This model performs well in zero-shot setting, and com- pares to recent state-of-the-arr (Chia et al., 2023). Although does not reach ChatGPT, the corre- lation with human annotators are all statistically
Reply with single word.
User prompt: water, aren, river, park, miles, game
Prompt 3: Intruder Detection Task. The intruder word in this topic is game.
Rating Task. Similar to the human experiment,
we retrieve the top 10 topic words and shuffle them.

---

Task
Dataset
NPMI
Cy
LLM (main)
LLM (min.)
LLM (all ann.)
FLAN-TS
Ceiling
NYT
0.43
0.45
0.37
0.41
0.39
0.37
0.67
Intrusion
Wiki
0.39
0.34
0.35
0.27
0.36
0.18
0.60
Both
0.40
0.40
0.36
0.34
0.38
0.28
0.64
NYT
0.48
0.40
0.64
0.64
0.65
0.31
0.72
Rating
Wiki
0.44
0.40
0.57
0.51
0.56
0.17
0.56
Both
0.44
0.42
0.59
0.57
0.61
0.25
0.65
Table 2: Additional experiments reporting Spearman correlation between mean human scores and automated metrics.
LLM (main) repeats our main results Table 1 for reference. LLM (min.) results using minimal prompt without dataset descriptions. LLM (all ann) no discarding low-confidence annotations. FLAN-TS FLAN-TS XL instead ChatGPT. All numbers are the average result of 1000 bootstrapping episodes re-sampling human annotations and LLM scores. Ceiling shows batched inter-annotate agreement.
significant. For the NYT and concatenated exper- iments, the resulting correlation are statistically indistinguishable from the best reported automated metrics NPMI and Cy in (Hoyle et al., 2021). We also our experiments with Alpaca-7B and Falcon-7B, with largely negative results.
Dataset
Topics
ARI
AMI
Compl.
Homog.
Bills Words
Broad
0.61
0.74
-0.58
Wiki Words
Broad
-0.38
-0.38
-0.38
0.38
Wiki Words
Specific
0.03
-0.24
-0.19
0.17
Bills Does
Broad 0.59
0.36
0.57
-0.58
Wiki Does
Broad
0.72
0.72
0.72
-0.70
Wiki Does
Specific 0.72
0.66
0.20
-0.20
Table 3: Spearman correlation coefficients between our language-model based scores and various popular rics for assessing the overlap between the topic model assignment and the underlying ground -ruthh Compl. = Completeness, Homog. Homogenity.
C Alternative Clustering Metrics
In our main results, we show correlations between LLM scores and the adjusted Rand Index, ARI, which measures the overlap between ground-truth clustering and topic model assignments. There other cluster metrics, such Adjusted Mutual In- formation, AMI (Vinh al., 2010), completeness, homogeneity. In Table 3, we show Spearman correlation statistics for these metrics. Our corre- lations are robust to the choice of metric used to measure the fit between the topic model assignment and the ground-truths in our case studies.
E Optimal Number of Topics Prompts
We now show the prompts for the optimal number of topies. We incorporate research questions two ways: (I) we specify whether we are looking for broad or narrow topics, and (2) we prompt 5 example categories. We believe this is realistic operationalization. If our goal is a reasonable parti- tioning of collection, usually have some priors about what categories want the collection to be partitioned into.
D Definitions
Prompt 6 shows the prompt for rating Tas by models run with different numbers of topics. The task description and user prompt identical to the prompt used in our prior experiments, displayed in e.g., Prompt 4. However, the dataset descrip- tion is different and allows for some variation. In Prompt 7, we show the prompt for automatically assigning labels to a document from Tac- To au- tomatically find the optimal number of topics for topic model, we prompt an LLM to provide a con- cise label a document from the topic document collection, the most likely documents assigned by a topic model to a topic (see Prompt 7).
See Bouma (2009) for justification of the NPMI for- mula. p(w;) and p(w; wi) are unigram and joint probabilities, respectively.
PMI( wi, wr) log p(w))10))
p(m))p(w))
NPMI(w;, wy) -log p(w,, myy) log wi, wr)
The Cy metric (Röder et al., 2015) more complex and includes, among others, the combina- tion NPMI and cosine similarity for top words.

---

You are belpfill assistant evalusting the top words of topic model output for given topic, Please how related the following words are ench other scale from to 3 ("1"= not very related. "2" moderately related, "3" very refated). The topic modeling based on logislativee Bill summary dataset, We are interested coherent nanee topics. Typical topics in dataset include "uopic 1", "Ippic 2", "topke 3". "upic 4" and "topic 5*. Reply with a single number, indicating overall uppropriateness the lopic. User prompt: lake, park, river, land, years, focr, lec, miles, water, arca
germany or tropical cyclones: atlantic. Recall that the prompted labels correspond the five most prevalent ground-truth categories of the ground- truth annotation. We find that the assigned label makes sense in 93.9% of examined labels. In the 11 errors spotted, the assigned label does not meet the granularity in 6 cases, is no adequate description of the document in 3 cases, and is a summary of the document instead of a label in 2 cases.
Prompt 6: Rating Task without dataset description. Topic terms are shuffled. We apply this prompt to 2 different datasets and 2 different research goals (broad and narrow topies), and would set this part of the prompt accordingly. Also, we set as topic I to topic 5 the 5 most prevalent ground-truth labels from a dataset.
Automated Metries. Given that we have ground- truth labels for each document, we can compute cluster metries between the assigned labels by the LLM and the ground-trutt -tuut labels (see Table 4). These values refer to comparing all labels assigned during our case study to their ground-truth label (1000 assigned datapoints per dataset).
System prompt: You are helpful research assistant with lots of knowledge about topic models. You given document assigned to a topic by topic model. Annocate
document with bradlaarmn label, for example "topie I". "uppcc2 2", "Ispic 3", 'topic 4" and "lopic 5".
phrase, indicating the label the document. User prompt: National Black Clergy for the Elimination of HIVAIDS Act of 2011 Authorizes the Director of the Office of Minority Health of Department of Health and Humam Services (III1S) make grants to public health agenctes and fuith-based organizations to conduct HIVIAIDS prevention, sesting, and related outreach activities...
Roply with single
Assigning label to document belonging to the top document collection of a topic. The label pro- vided in this example health. We apply this prompt to 2 different datasets and 2 different research goals (broad and narrow topics). and would set this part of the prompt accordingly. Also, we topic J to topic 5 the 5 most prevalent ground -ruth labels from a dataset.
Prompt 7:
Dataset Ground- -ruth Labels
ARI
AMI
Bills Wiki Wiki
Broad
19
43
Broad
52
57
Narrow
49
34
Table 4: Accuracy of the label assignment task. We find that the assigned labels clustering overlaps with the ground-truth labels.
F Additional: Document Labeling
In our study, we automatically label the top 10 doc- uments for five randomly sampled topics. The ARI between-topic model partitioning and ground-truth labels correlates if we were to only examine these top 10 documents or all documents in the collec- tion. The correlation between these two in the Bills dataset is 0.96, indicating that analyzing only the top 10 documents in topic is decent proxy for the whole collection.
On average, assign 10 times many unique labels to documents than there are ground-truth labels (we assign 172 different labels in the Bills dataset, 348 labels in the broad Wikitext dataset and 515 labels in the narrow Wikitext dataset). Nev- ertheless, the automated metrics indicate a decent overlap between woound-truth and assigned labels. Thus, the LLM often assigns the same label to doc- uments with the same ground-trun -ruth label.
Next, we evaluate the LLM-based label assigne- ment to a document. Our documents are usually long, up to 2000 words. We only consider the first 50 words in a document as input to the LLM. For Wikipedia, this is reasonable, because the first 2-3 sentences define the article and give a good summary of the topic of an article For Bills, we manually confirm that the topic of article is in- troduced at the beginning of a document.
G Qualitative Results
In this section, we show qualitative results of our automated investigation of numbers of topics. In Table 5, we show three randomly sampled topics from the preferred topic model in our experiments. We contrast these with three randomly sampled top- ics from the topic model configuration which our procedures indicate as least suitable.
Human evaluation. From each case study, we randomly sample 60 documents and assigned labels (3 examples for each of the twenty topic models), resulting in 180 examples in total. We then evalu- ate whether the assigned label reasonably captures the document content given the specification in the input prompt (e.g., a broad label such as health or defense, or narrow label such as warships of
In Table 6, we show true labels and LLM- assigned labels for three randomly sampled topics from the preferred topic model, contrasting it with true and L.-M-ssieeeed labels from topics in the least suitable configuration. We find that indeed, the assigned labels and the ground-truth label often match and that the purity of the -asaigeeee labels reflects the purity of the ground-truth label.

---

Bills (broad categories)
veterans, secretary, veteran, assistance, service, disability, benefits, educational, compensation, veterans affairs (3)
Most
land, forest, management, lands, act, usda, projects, secretary, restoration, federal (3)
suitable
mental, health, services, treatment, abuse, programs, substance, grants, prevention, program (3)
gas, sccretary, lease, oil, leasing, act, way, federal, production, environmental (2)
Least
covered, eriminal, history, act, restitution, child, background, amends, checks, victim (2)
suitable
information, heneficial, value, study, ownership, united_ states, act, area, secretary, new_york (I)
Wikitext (broad categories)
episode, star, trek, enterprise, series, season, generation, ship, episodes (3)
Most
series, episodes, season, episode, television, cast, production, second, viewers, pilot (3)
suitable
car, vehicle, vehicles, engine, model, models, production, cars, design, rear (3)
episode, series, doctor, season, character, time, story, trek, set
Least
stage, tour, ride, park, concert, dance, train, coaster, new, roller (1)
suitable
said, like, character, time, life, love, relationship. later, peuple, way (1)
Wikitext (specifie categories)
episode, star, trek, enterprise, series, season, crew, generation, ship, episodes (3)
Most
car, vehicle, vehicles, engine, model. models, production, cars, design, rear (3)
suitahle
world, record, meter, time, won, freestyle, gold, championships, relay, seconds (2)
fossil, fossils, found, specimens, years, evolution, modern, million, eddie, like (2)
Least
match, event, impact, joe, team, angle, episode, styles, championship, tag (1)
suitable
brown, rihanna, usher, love, girl, loud, yeah, wrote, bow, bad (1)
Table 5: Most and least suitable topics according to our LLM- based assessment on different datasets and use cases. brackets the L.M rating the coherence this topic.
Bills
Wikitext (broad)
Wikitext (specific)
LLM-label
True label
LLM-label
True label
LLM-label
True label
Most suitable
health
Health
amusement
Recreation
politician
Historical figures: politicians
park ride amusement park ride amusement park ride amusement
elder abuse
Social
Wel-
Recreation
politician
Historical figures: politicians
prevention
fare
health
Health
Recreation
american civil war lawyer and politician historical newspaper
Historical figures: politicians
health
Health
Recreation
Historical figures: other
park ride amusement
health
Health
Recreation
Joumalism and newspupers
park ride
Least suitable
public land
Public Lands
warship and naval unit
Armies and mil- itary units
classical
Poetry
greck poctry hinduism
public land
Public Lands
warship and naval unit warship and naval unit warship and naval unit war poetry
Armies and mil- itary units Military people
Religious doctrines, teachings, texts, events, and symbols Religious doctrines, teachings, texts, events, and symbols Philosophical doctrines, teach- ings, texts, events, symbols Philosophical doctrines, teach- ings, texts, events, symbols
public land
Environment
hinduism
indigenous
Government
Military people
philosophy
affair
Operations
indigenous
Government
Language and literature
philosophy
affair
Operations
Table 6: Assigned LLM labels and ground-truth labels for a given topic from the most and the least suitable cluster configuration according to our algorithm. The purity higher the most suitable configuration LLM labels and ground-truth labels.


Reviews:
[{"review_id": "hrnisXOJSb", "reviewer": "Reviewer_sobM", "summary": "", "questions": "- Can you give a brief explanation of how to retrieve the 10 most probable documents in the text labeling algorithm for investigating the number of topics? \n", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks a lot for the helpful review which is greatly appreciated!\n\n1. In the revised version, we will include a short discussion in the related work section about previous endeavors in nonparametric Bayesian models for estimating the number of topics [e.g. 1].\n2. The 10 most probable documents are the documents with the highest estimated probability of belonging to a document. LDA estimates a $k$-length distribution over topics $\\theta_d$ for each document, which we compose into a $N \\times K$ matrix for all $N$ documents. To get the most probable documents per topic, we take an argsort over each matrix column. We will make this more explicit in the final version of the paper. More informally, we retrieve the top ten documents having assigned the highest probability for a given topic by the topic model.\n\n[1] Teh, Yee Whye et al. [“Hierarchical Dirichlet Processes.”](http://www.gatsby.ucl.ac.uk/~ywteh/research/npbayes/jasa2006-print.pdf) Journal of the American Statistical Association 101 (2006): 1566 - 1581.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Achieving both objectives of evaluating topic models and determining the optimal number of topics, this paper stands as the first to employ LLM. \n\n- The paper presents a substantial body of insights that can greatly assist the community in leveraging LLM for these specific objectives. \n", "reasons_to_reject": "- It would enhance the discussion if the authors acknowledged previous endeavors in nonparametric Bayesian models designed to estimate the number of topics and estimate latent topics.\n", "questions_to_authors": "- Can you give a brief explanation of how to retrieve the 10 most probable documents in the text labeling algorithm for investigating the number of topics? \n", "paper_topic_and_main_contributions": "- This paper delves into the feasibility of utilizing LLM for automated topic model evaluation, taking into account prior efforts in evaluation metrics like NPMI, as well as previous research indicating that NPMI does not exhibit a strong correlation with human evaluation.\n\n- The paper further explores the utilization of LLM to automatically determine the ideal number of topics. \n", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "This paper is well written. "}, "extra_text": {}}, {"review_id": "KD7G40WvMe", "reviewer": "Reviewer_aBQ5", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review which is greatly appreciated! \n\n1. We have incorporated your and other suggestions regarding the paper writing.\n2. In all cases we use Spearman correlation because we are interested in the *ranking* (evaluation) of outputs instead of the absolute values. We specified this in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Extensive evaluation\n- Interesting read, the proposed approach can be relevant for future research and practitioners", "reasons_to_reject": "- The paper is hard to read in some parts, see examples below.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper tackles the problem of automatically evaluating topic models by leveraging LLMs.\nIn addition, the authors also attempt to use KKNs to automatically determine the optimal number of topics.\nThe authors find that LLMs can indeed accurately assess topic coherence, and \"can assist\" in the second task.\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- I'd remove the mathematical notations and definitions from the first paragraph of the intro, to make it more readable and appealing.\n- I'd specify the correlation type measured in Figure 1."}, "extra_text": {}}, {"review_id": "OeV7NjujlJ", "reviewer": "Reviewer_2esg", "summary": "", "questions": "Large language models might focus on common language patterns rather than the specific nuances of certain topics. How do you solve the Overfitting issues  to General Language Patterns in broad topics such as  politics or healthcare?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks a lot for the helpful review which is greatly appreciated!\n\n1. We agree with your comment about “Relying on an opaque model to evaluate another model's output might introduce additional uncertainty and lack of transparency.” We already have a dedicated section about this in the limitation section of our paper (L321, p. 5). In the final version, we will include this discussion in the main paper because it is important and timely. At the same time, we think there is value in exploring whether and how we can use large language models to tackle long-standing issues in NLP, such as automated evaluation of topic models. This paper contributes to such efforts.\n2. We explore the overfitting issue in future work where we will explore datasets requiring more in-depth domain knowledge and using large language models specialised in these domains (e.g., the biomedical or legal domain). It is true that there could be an interaction between the evaluating models’ primary domain and its performance on evaluating texts outside of this domain (similarly to quality estimation [1]). However, since this concerns a much larger research question than the scope of this paper, we leave it to future investigations, which are enabled by our present work.\n\n[1] Sharami, Javad Pourmostafa Roshan, et al. [\"Tailoring Domain Adaptation for Machine Translation Quality Estimation.\"](https://arxiv.org/abs/2304.08891) arXiv preprint arXiv:2304.08891 (2023).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Automated Topic Model Evaluation with LLM Assistance.", "reasons_to_reject": "Large language models can be challenging especially if the topics requires domain expertise. Relying on an opaque model to evaluate another model's output might introduce additional uncertainty and lack of transparency.", "questions_to_authors": "Large language models might focus on common language patterns rather than the specific nuances of certain topics. How do you solve the Overfitting issues  to General Language Patterns in broad topics such as  politics or healthcare?", "paper_topic_and_main_contributions": "The authors proposed to use LLM to evaluate the output of the topic model. The authors have made several claims 1.LLMs can accurately judge topic coherence, 2.LLMs can assist in automatically determining reasonable numbers of topics.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers all found this paper sound, with extensive evaluation and a solid experimental setup, and most found it exciting. They noted that the insights from the paper will be beneficial to the community and that the paper is well-grounded in the existing literature on the topic while building upon it. The main reasons to reject had to do with style, formatting, and discussions that could be expanded upon, rather than things like the experimental setup or conclusions being made. One drawback brought up related to using LLMs to evaluate topics in cases where domain expertise is needed, is valid, but is already addressed somewhat by the limitations section. The authors also commit to expanding on this. I do not see this as a major methodological flaw (and I believe the reviewer agrees, given their scores) but rather something to be discussed more in the paper, if accepted.
Decision: Accept-Main