Topic Modeling for Short Texts with Large Language Models (2024)

Abstract: As conventional topic models rely on word cooccurrence to infer latent topics, topic modeling for short texts has been a long-standing challenge. Large Language Models (LLMs) can potentially overcome this challenge by contextually learning the meanings of words via pretraining. In this paper, we study two approaches to using LLMs for topic modeling: parallel prompting and sequential prompting. Input length limitations prevent LLMs from processing many texts at once. However, an arbitrary number of texts can be handled by LLMs by splitting the texts into smaller subsets and processing them in parallel or sequentially. Our experimental results demonstrate that our methods can identify more coherent topics than existing ones while maintaining the diversity of the induced topics. Furthermore, we found that the inferred topics cover the input texts to some extent, while hallucinated topics are hardly generated.

heading: 1 Introduction
text: Topic modeling is the classical task of discovering latent topics that best describe a set of documents (Blei et al., 2003; Churchill and Singh, 2022). Recently, while neural topic models have worked successfully on various kinds of long documents (Miao et al., 2017; Srivastava and Sutton, 2017; Dieng et al., 2020), they have not been able to handle short texts, such as social media posts and news headlines (Li et al., 2016; Wu et al., 2022).
Large Language Models (LLMs), such as InstructGPT (Ouyang et al., 2022) and GPT-4 (OpenAI, 2023), have shown impressive results on various tasks by providing task instructions in a zeroshot manner (Wang et al., 2023; Kocoń et al., 2023). Since conventional topic models infer the topics of words by relying on word co-occurrence, they perform worse on short texts. In contrast, as LLMs contextually learn the meanings of words by pre-
training on massive text corpora, they could accurately infer the latent topics.
We propose two approaches to using LLMs for topic modeling: parallel prompting and sequential prompting (Figure 1). Due to the input length limitations of LLMs, an input document set must be split into smaller subsets, which are processed individually. Parallel prompting concurrently infers the topics of each subset and merges them to represent the topics of the whole document set. Sequential prompting processes each subset successively, updating the topics in every iteration. We assess our approaches across texts from various domains using multiple evaluation metrics.
The contributions of this study are as follows: 1. We propose parallel and sequential prompting
methods for topic modeling using LLMs. Our methods can handle a large number of texts that cannot be processed in a single run due to the input length limitations of LLMs.
2. We validate the performance of our methods by comparing them with existing models and show that ours can identify more coherent topics than existing models while maintaining
21
the diversity of the induced topics.
3. We assess the document coverage and factuality of the inferred topics, due to concerns that LLMs may focus on only parts of documents or generate hallucinated topics. Evaluation results indicate that those concerns are negligible.

heading: 2 Background
text: Topic modeling is the task of identifying latent topics as a set of topic words representing each topic from a collection of documents (Blei et al., 2003). Topic modeling has conventionally been tackled with probabilistic models such as latent Dirichlet allocation (LDA, Blei et al., 2003). In recent years, however, neural models have come into widespread use due to their high performance (Srivastava and Sutton, 2017; Dieng et al., 2020; Grootendorst, 2022).
It is known that topic modeling for short texts is difficult for current topic models due to data sparsity (Li et al., 2016; Wu et al., 2022). TSCTM (Wu et al., 2022) is a current state-of-the-art neural topic model for short texts. This model addresses data sparsity by learning representations of documents using VQ-VAE (van den Oord et al., 2017), contrastive learning, and incorporation of data augmentation into the learning.
BERTopic (Grootendorst, 2022) uses a pretrained encoder, Sentence-BERT (Reimers and Gurevych, 2019), to obtain clusters of documents and assigns topic words to each cluster by using a class-based TF-IDF procedure. Another related study is Stammbach et al. (2023), in which LLMs are utilized to automatically evaluate topic quality. However, our study is the first to explore how well LLMs perform topic modeling.

heading: 3 Topic Modeling with LLMs
text: We introduce two approaches to performing topic modeling with LLMs: parallel prompting and sequential prompting. For these approaches, we apply common preprocessing, which involves randomly splitting a document set into subsets with the same size, smaller than the context length of the LLMs.
Parallel Prompting In the parallel prompting, LLMs identify topics for each subset in parallel by prompting the subset and the instruction of topic modeling. The topics of each subset are then
merged by LLMs. We use two kinds of prompts as shown in Table 1: (i) a ParTM prompt for parallel topic modeling for each subset, and (ii) a ParMrg prompt for merging the topics from the results
Sequential Prompting In the sequential prompting, LLMs identify topics for each subset sequentially, considering the topics previously identified for the previous subset. We use the ParTM for the first subset, then use a SeqTM prompt in Table 1 for the other subsets. This prompt contains topics identified in the prior subset and instructions for referring to them.

heading: 4 Experiments
text: We investigate how well our methods perform topic modeling for short texts.

heading: 4.1 Dataset
text: We employ three tokenized datasets provided by Zhang et al. (2021): GoogleNewsT (Rakib et al., 2020), Tweet (Yin and Wang, 2016), and StackOverFlow.1 Following Wu et al. (2022), the datasets are preprocessed as follows: (i) characters are converted to lower case; (ii) words with two or fewer letters are removed; (iii) words appearing fewer than five times are filtered out. We then split each preprocessed dataset into subsets for
1https://www.kaggle.com/competitions/ predict-closed-questions-on-stack-overflow/ data?select=train.zip
our methods, setting the size at 10002 and truncating the remaining example. Table 2 shows the final statistics of the datasets we use. Note that baseline models take the union of subsets as input, and each subset contains different examples for each run.

heading: 4.2 Model
text: We evaluate our approaches with GPT-3.5 (gpt3.5-turbo-0125) and GPT-4 (gpt-4-0125-preview) provided by the OpenAI API.3 For baseline models, we employ the three models mentioned in Section 2: LDA4, TSCTM4, and BERTopic.5 Additionally, we report the results of each baseline model with data augmentation. Regarding data augmentation techniques and the hyperparameters of TSCTM, we follow the original settings that were used in
2In preliminary experiments, we checked the performance of our methods with subset sizes of 250, 500, and 1000. See Appendix A.3.
3In preliminary experiments, we also tried Llama 2 (Touvron et al., 2023), but we found that it was not sufficiently controllable for its output to be used in our approach. See Appendix A.2.
4https://github.com/BobXWu/TopMost 5https://maartengr.github.io/BERTopic
prior research (Wu et al., 2022).6

heading: 4.3 Evaluation
text: We evaluate the models under the condition that the number of topics is 5 or 15, and the number of topic words for each topic is 5. For evaluation metrics, we employ two widely used metrics for topic quality and two new metrics to assess possible issues of LLMs, i.e., the possibility of outputting topics reflecting only a very limited documents or hallucinated topics not included in documents. We run each model five times and report the average scores.
Topic Coherence and Diversity Following Wu et al. (2022), we calculate the coherence value7 (Cv, Röder et al., 2015) with Wikipedia for topic coherence, and the topic uniqueness (TU, Nan et al., 2019) to assess the diversity in the inferred topics.
Document Coverage We are concerned that LLMs infer topics that reflect only a very limited
6The details can be found in Appendix B.1. 7https://github.com/dice-group/Palmetto
documents. Thus, we propose the metric document coverage, which measures the extent to which discovered topics cover documents. Document coverage is defined as follows:
DC = #(dref that contains at least one wtopic)
# (dref )
where dref is a document within the reference document collection, and wtopic is the topic word constituting the outputted topics. A higher DC means that discovered topics cover more reference documents. In this experiment, we use the preprocessed datasets without augmentation as references.
Factuality Another potential issue is hallucination, where topics discovered by LLMs may not be included in given documents. Therefore we introduce factuality, which measures the degree to which topic words are composed from the vocabulary in the reference documents. Factuality is defined as follows:
Fa = #(wtopic present in at least one dref )
# (wtopic)
A higher Fa indicates that more topic words are composed from the vocabulary in the reference documents. Note that the factuality could be less than one in existing topic modeling with data augmentation due to word substitution using out-ofvocabulary words of the documents.

heading: 5 Results and Discussion
text: Topic Quality Table 3 shows that the topics discovered by our methods are relatively high-quality both in terms of coherence (Cv) and diversity (TU).8 For coherence in particular, GPT-4 achieved the state-of-the-art performance in all settings, with up to 40 % improvement. For instance, the scores on GoogleNewsT have risen by 41% (from 0.439 to 0.618) and 28% (from 0.437 to 0.561), respectively, for each setting of the number of topics.
8Examples of topics are given in Appendix C.1.
Document Coverage Table 4 reports that LLMs showed relatively lower scores for document coverage (DC) than the best baseline models. This means that the topics discovered by LLMs often cover fewer documents than those discovered by the baseline models. However, note that there is a trade-off between topic coherence (Cv) and document coverage. For example, LDAAug achieved the highest coverage on GoogleNewsT but showed the lowest coherence, with the exception of TSCTM and TSCTMAug.
Factuality As shown in Table 4, LLMs showed lower scores for factuality (Fa) than the baseline models, particularly those without augmentation. This indicates that some topic words output by LLMs are not included in the documents. However, their factuality loss was less than 5% in almost all settings. Furthermore, we analyzed these non-existent words and found that most were not problematic enough to mislead topic interpretation; these include synonyms, derivatives, and related words of the ones in the documents.9 This suggests that LLMs do not generate hallucinated topics that would cause misinterpretation of the content.
Parallel and Sequential Prompting Table 3 and Table 4 show that the parallel prompting approach can identify topics with better coherence and document coverage than the sequential prompting one. To analyze the superior performance of the parallel approach, we calculated Cv and DC of topics before merging. Table 5 shows that Cv and DC scores before merging were worse than those of the parallel approach, demonstrating that the merging process can improve both their coherence and document coverage. On the other hand, we analyzed the transition of topics during the sequential approach and then observe that it tended to update the previously identified topic very little due to strict adher-
9Examples of non-existent words and analysis details are provided in Appendix C.2.
ence to our instructions, leading to lower document coverage compared with the parallel approach.10
Qualitative Analysis We conducted a qualitative analysis of the representative results that achieved the median topic coherence (Cv) across five trials using the GoogleNewsT dataset under five topics and five topic words. Table 6 demonstrates that BERTopic, the best baseline model for Cv, has the potential to identify topics encompassing multiple themes, while our methods using LLMs discover highly consistent and distinct topics. For instance, topic #1 identified by BERTopic could be considered to contain three distinct themes (Kanye West, Thanksgiving, and Xbox), while GPT-4Par and GPT-4Seq effectively separated these into topics #1, #2, and #3, respectively.

heading: 6 Conclusion
text: In this study, we proposed two approaches to using LLMs for topic modeling: parallel prompting and sequential prompting. We implemented our methods on GPT-3.5 and GPT-4 and evaluated their performance on three datasets together with three existing topic models. In the evaluation, in addition to the well-known metrics for topic quality, we introduced two new metrics, document coverage and factuality, to assess the potential issues with LLMs reflecting only some documents or outputting hallucinated topics. The results showed that LLMs could find higher-quality topics than existing methods, and the impact of these issues was not
10Examples and further analysis are provided in Appendix C.3.
remarkable in practice. Future work will include improving our methods to enable topic assignment to each document.

heading: Acknowledgements
text: We thank the three anonymous reviewers for their helpful comments and suggestions, which improved this paper. This work was supported by JSPS KAKENHI Grant Number JP24H00809, Japan.

heading: A Preliminary Experiments
text: In preliminary experiments, we tested different prompts and subset sizes to determine which maximize the performance of our methods.

heading: A.1 Prompts
text: We first considered the ParTM prompt and then proceeded to the ParMrg and the SeqTM prompts.
ParTM We checked three kinds of prompts, which are shown in Table 7. Finally, we tentatively selected a Direct prompt as a ParTM prompt, which achieved the highest performance. We also considered the effects from inserting the following phrases, which were expected to improve scores for topic coherence, diversity, and document coverage, respectively.
Cv “NOTE: Make top words for each topic likely to occur together in the documents”
TU “NOTE: Make the top words unique across topics.”
DC “NOTE: Maximize the number of documents that contain at least one of the top words.”
However, we found that none of these can positively influence LLMs’ performance in our methods. Therefore, we selected a Direct prompt without phrase insertion as the ParTM prompt.
ParMrg Regarding the ParTM prompt, we created a Base ParMrg prompt, which has a similar structure to the ParTM (Table 8). We then considered the insertion of the following phrases:
Goal “We aim to identify topics for the entire document set by merging the topic modeling results for each subset.”
Detail “NOTE: Outputs should reflect the topics before merging as much as possible. Output should contain topics that often appear before merging and not have ones that don’t appear much before merging.”
Experimental results showed our methods performed the best when we inserted both the Goal phrase and the Detail phrase into the Base ParTM.
Consequently, we employed a Base ParTM prompt with both phrases as the ParTM prompt for the parallel approach.
SeqTM Similar to the prompt for parallel, we first created a simple Base SeqTM prompt for the sequential approach in Table 8, after which we validated the effect from inserting the following phrases.
Goal “We aim to identify topics for the entire document set by sequentially updating tentative topics identified from each subset, considering topics identified just before from another subset.”
Detail “NOTE: Outputs should be the same as the previous topics as much as possible. You can change them minimally only when the given documents don’t include them much, and a new topic needs to be added to describe the documents.”
We also found that the insertion of both of the above phrases was most effective at improving the performance of the sequential method. Thus, we utilized a Base SeqTM prompt that incorporates both phrases as the SeqTM prompt for the sequential approach.

heading: A.2 Llama 2
text: In preliminary experiments, we also tried using Llama-2-7b-chat11 and Llama-2-13b-chat11 as LLMs for our methods and found that it is difficult for Llama 2 (Touvron et al., 2023) to perform topic modeling regardless of the prompts and the subset size we use. Table 9 shows the outputs of Llama 2 when given the ParTM prompt with a subset size of 100 on GoogleNewsT. Llama 2 could not make adequate output for the number of topics and topic words in line with our instructions, while GPT-3.5 and GPT-4 could do so consistently under identical settings.

heading: A.3 Subset Size
text: We used 250, 500, and 1000 as options for the subset size. It would be difficult for the subset size to exceed 1000 due to the context length of GPT3.5 (gpt-3.5-turbo-0125), which we planned to use for the main experiments.
We ran the parallel and the sequential methods with GPT-3.5 on GoogleNewsT for each subset size. Table 10 presents the average scores of each method for five runs. There was a tendency for
11https://huggingface.co/ collections/meta-llama/ llama-2-family-661da1f90a9d678b6f55773b
topic coherence to improve as the subset size increased, but we could not discern any tendency for the other metrics. We ultimately selected 1000 as the subset size because the performance of each model was relatively high in all metrics under that setting.
Note, however, that using our proposed methods with the subset size of 250 or 500 could enable discovery of competitive or higher-quality topics compared with the existing models shown in Table 3 and Table 4. This suggests our methods could perform well regardless of the context length of LLMs applied them.

heading: B Experimental Details
text: B.1 Implementation Details
We run TSCTM for 200 epochs. In the case without data augmentation, we run it with temperature as 0.5 and weight contrast as 1.0. In the case with data augmentation, we run it with temperature as 0.07, weight contrast as 3.0, and same quant as 0.001. For data augmentation, we apply WordNet12 and Contextual Augmenter3 (Kobayashi, 2018) with 30% word replacement, and filtered low-frequency words as in the preprocessing. Each Augmenter randomly replaces words in an input text with synonyms defined by WordNet and with words predicted by BERT (Devlin et al., 2019)13, respectively. We utilized the original configurations of gpt-3.5-turbo-0125, gpt-4-0125-preview, and BERTopic without modification.

heading: B.2 Examples of Prompts
text: Table 11 shows examples of prompts used in the experiment.

heading: C Result Details

heading: C.1 Examples of Topics
text: Following Wu et al. (2022), we randomly select some examples of topics identified by LDA, BERTopic, and our proposed methods with GPT-4.

heading: C.2 Examples of Topic Words Not Included in the Documents
text: gree, and watson) or words that do not exist in the real world (e.g., dorffiefskee). Such words are considered harmful because they may induce misinterpretation of topics. However, only a small number of such words were found, and most of them were synonyms, derivatives, or related words in the documents.

heading: C.3 Examples of the Processing
text: Table C.3 shows specific the concrete examples of topics identified for each subset and the final output to demonstrate the processing in our methods. In the parallel approach, we find that LLM reasonably merges topics from each subset. For instance, bold topics in each subset are merged into one topic in the final output, using words from both subsets. On the other hand, in the sequential approach the final output is the same as the topics for the first subset except for the one pair of bold words. This indicates that LLMs with the the sequential approach could too strictly retain topics from the previous subset, and thus they cannot output topics that sufficiently reflect the entire set.

heading: D Limitations
text: We do not thoroughly consider whether pretraining and instruction-tuning datasets of GPT-3.5 and GPT-4 might contain the datasets used in this study. Since topic modeling is an unsupervised task and we change the order of the samples randomly, we do not consider them able to utilize their knowledge about these datasets in our experiment.

heading: Future Work
text: Future work will include improving our methods to enable topic assignment to each document