DeTiME: Diffusion-Enhanced Topic Modeling using Encoder-decoder based LLM
Authors: Weijie Xu
Wenxiang Hu
Fanyou Wu
Srinivasan H. Sengamedu
Venue: EMNLP 2023 Findings (2023)
Keywords: Topic Modeling
Diffusion
Encoder-Decoder LLM
FlanT5
CNN
TL;DR: The paper proposes a framework Diffusion-Enhanced Topic Modeling using Encoder-decoder based LLM to generate highly clustered topics and topics related contents.

Abstract: In the burgeoning field of natural language processing, Neural Topic Models (NTMs) and Large Language Models (LLMs) have emerged as areas of significant research interest. Despite this, NTMs primarily utilize contextual embeddings from LLMs, which are not optimal for clustering or capable for topic generation. Our study addresses this gap by introducing a novel framework named Diffusion-Enhanced Topic Modeling using Encoder-Decoder-based LLMs (DeTiME). DeTiME leverages Encoder-Decoder-based LLMs to produce highly clusterable embeddings that could generate topics that exhibit both superior clusterability and enhanced semantic coherence compared to existing methods. Additionally, by exploiting the power of diffusion, our framework also provides the capability to generate content relevant to the identified topics. This dual functionality allows users to efficiently produce highly clustered topics and related content simultaneously. DeTiME's potential extends to generating clustered embeddings as well. Notably, our proposed framework proves to be efficient to train and exhibits high adaptability, demonstrating its potential for a wide array of applications.

DeTiME: Diffusion-Enhanced Topic Modeling using Encoder-decoder based LLM
Weijie Xu, Wenxiang Hu, Fanyou Wu, Srinivasan H. Sengamedu Amazon
weijiexu@amazzn.com com
Abstract
the Neural Topic Model (Miao et al., 2018: Dieng et al., 2020) has attracted attention due its better ficxibility and scalability. The topic generated through the reconstruction of the bag-of-word rep- resentations of the document (Miao et al., 2018).
In the burgeoning field of natural language processing, Neural Topic Models (NTMs) and Large Language Models (LL.ss) have emerged as areas of significant research interest. De- spite this, NTMs primarily utilize contextual embeddings from LLMs, which are not optimal for clustering or capable for topic generation. Our study addresses this gap by introducing novel framework named Inffusion-Enhaneee Topic Modeling using Encoder Decoder- based LLMs (DeTiME), DeTiME leverages Encoder- Decoder-basse LLMS to produce highly clus- terable embeddings that could generate topics that exhibit both superior clusterability and en- hanced semantic coherence compared to exist- ing methods. Additionally, by exploiting the power of diffusion, our framework also pro- vides the capability to generate content relevant the identified topics. This dual functionality allows users to efficiently produce highly clus- tered topics and related content simultaneously. DeTiME's potential extends to generating clus- tered embeddings as well. Notably, our pro- posed framework proves be efficient to train and exhibits high adaptability, demonstrating its potential for a wide array applications.
The progress of large language model (LLM) (Vaswani et al., 2017; Radford et al., 2019) brings significant advancements in the NLP community. Sentence embedding is the process of converting sentences into numerical vectors in high-dimensional space. LLM-based sentence embedding has been applied to topic modeling by using it to reconstruct bag word representation of documents (Bianchi et al., 2021a), to cluster document directly (Grootendorst, 2022) or both (Han et al., 2023). Sentence embedding- based models have been shown to achieve high performance regarding coherence and diversity (Zhang al., 2022). Embeddings with higher clusterability are likely perform well in classification tasks. However, sentence embeddings are in general not perform well in clustering. The best performed sentence embedding has an average v-measure (Rosenberg and Hirschberg, 2007) below 0.44 even if it uses kmeans and set the cluster equal to the number of different labels (Muennighoff et al., 2022). This means that their clusterability can be even lower when the latent dimension increases. Lastly, language modeling is a powerful generative tool (Brown et al., 2020). While topic modeling has been utilized for generation ( (Wang et al., 2019), its integration with Large Language Models (LLMs) for generation remains less explored.
1 Introduction
Topic modeling methods, such (Blei et al., 2003), are unsupervised approaches for discov- ering latent structures in documents and achiev- ing great performance (Blei et al., 2009). These methods take a list of documents as input, gen- erate a defined number of topics, and can further produce keywords and related documents for each topic. In recent years, topic modeling methods have been widely used in various fields such finance (Aziz al., 2019), healthcare (Bhattacharya et al., 2017), education (Zhao al., 2021a,b), marketing (Reisenbichler, 2019), and social science (Roberts et al., 2013). With the development of Variational Autoencoder (VAE) (Kingma and Welling, 2013),
In this study, we introduce DeTiME, an innova- tive topic modeling framework that exploits the pabilities of the encoder-decoder Large Language Model (LLM). Specifically, we design task train an adapted encoder-decoder I.I., as depicted in Figure 2. We generate an embedding using this architecture, which exhibits high clusterability

---

compared established models illustrated in Figure 1. Furthermore, we design topic modeling approach using the last hidden layer of our modi- fied LLM encoder as input. This technique notably outperforms standard methods across all pertinent metrics. Additionally, we leverage diffusion and our proposed framework to generate relevant docu- ments. Our major contributions are as follows:
Compon idwamm bemac ehen mesteds
1. We modify the encoder-decoder LLM and de- sign a task create an embedding ideal for topic modeling, even using a smaller model.
Corpue VD letaee Dotte meleleeedokk
2. The fabricated embeddings outperform exist- ing methods in terms of clusterability
3. We devise a topic modeling method based on the embedding that achieves superior results in both clusterability and semantic coherence, compared to the relevant topic modeling meth- ods.
Figure I: A summary of few of our findings: (1)
4. We demonstrate the ability to produce relevant content based on this model by harnessing dif- fusion, indicating potential practical applica- tions.
Our embeddings outperform the best clusterable meth- ods (selected from (Muennighoff et al., 2022)). (2) The same framework with slightly different finetuned task(DeTiME Training) does not perform well. (3)
When compressed, embeddings excel in higher di- mensions, making them ideal for topic modeling. De- tailed settings is in Appendix E.
5. Our framework exhibits flexibility it can be seamlessly adapted to various encoder- decoder LLMs and neural topic modeling methods, broadening its applicability in the field.
on extensive datasets. FlanT5 (Chung al., 2022) enhances T5 by finetuning instructions across multiple datasets. Compared to encoder only (Bert) or decoder only model(GPT), encoder- decoder models such as FlanT5 allow the encoder to extract vital input information for output generation (Rothe et al., 2020).
By documenting detailed methodology and empir- ical results, we aim to inspire further research in this domain, and provide strong foundation for future work on topic modeling and LLMs.
2 Related work
Prefix tuning (Li and Liang, 2021) modifies a fixed-length "prefix" of parameters prepended the input during fine-tuning, significantly reducing the number of parameters required. This efficiency doesn't compromise performance; it often matches or surpasses traditional fine-tuning methods across various NLP tasks. The technique enables the model learn task-specific initial hidden states for LLM, steering the generation process appropri- ately without hindering the model's generality due to the fine-tuning task.
2.1 Language Modeling
Recent transformer-based models, such as BERT (Devlin al., 2019), GPT-3 (Brown al., 2020), and GPT-4 (OpenAI, 2023) have achieved unmatched performance numerous language tasks. Utilizing self-attention mechanisms, they capture context from both past and future tokens, generating coherent text. These rapidly evolving Large Language Models (LLMs) carry significant implications for diverse sectors and society. T5 (Raffel et al., 2020) treats every NLP task text-to-text problem, using a standard format with input and output as text sequences. employs an encoder-decoder framework and pretrained
2.2 Sentence Embedding
Contextual embeddings aim to encode sentence semantics a machine-readable format. Word em- beddings like Word2Vec (Mikolov et al., 2013)

---

and GloVe (Pennington al., 2014) capture word- level meaning but struggle with larger text struc- tures. Advanced models like the Universal Sen- tence Encoder (USE) (Cer et al., 2018) and In- ferSent (Conneau et al., 2018) were developed to better capture sentence nuances. USE employs transformer or Deep Averages Networks, while InferSent uses bidirectional LSTM with max pooling. Bentence-BERT (Reimers and Gurevych, 2019) utilizes siamese BERT-Networks. However, these models often struggle to capture context- dependent sentence meanings, resulting in lower clusterability. This might be due to their reliance on contrastive loss on sentence pairs, which might focus on specific similarities rather than the overall semantic relationship.
Denoising Diffusion Probabilistic Models (DDPM) (Ho al., 2020) have garnered attention for generating high-quality samples sans adversar- ial training, sometimes surpassing other generative models. Speedier sampling was achieved in (Song et al., 2022) with denoising diffusion implicit models. The success of image generation models like CLIP (Radford et al., 2021), Stable Diffusion (Rombach al., 2022), and Midjourny (Oppen- laender, 2022) leveraged such diffusion-based methods. Their use extends NLP tasks including natural language generation, sentiment analysis, and machine translation (Zou et al., 2023). has also demonstrated that the diffusion model is able to generate high-quality text from noise samples in the continuous embedding space(Li et al., 2022; Gong al., 2023; Gao et al., 2022; Lin et al., 2023b). Yet, diffusion hasn't 't been used for topic modeling as content generation tool.
2.3 Topic Modeling
The Neural Topic Model (NTM) (Miao al., 2016) employs variational inference but struggles with semantics and interpretability, while the Embed- ding Topic Model (ETM) (Dieng et al., 2019) uses pre-trained word embeddings to capture semantics. However, NTMs rely on bag-of-word representa- tions, limiting their ability to capture document semantics effectively.
Methods
The goal of this paper is to create framework that leverages encoder-decoder LLM to generate topics that is highly clusterable and able generate topic related sentence. To achieve that, we need to create an embedding that could be used to generate text as well as be highly clusterable. Thus, we designed a specific task and dataset for our use case. We add CNN encoder and decoder on top of Flan'T5 to generate that can easily fit into neural topic mod- eling for further dimension reduction. We further design variational autoencoder to take the output of the CNN encoder as input and generate topics and reconstruct embeddings. This is achieved by autoencoders. The first autoencoder is a varia- tional autoencoder which generates topic distribu- tion and reconstructs bag of words representations. To reconstruct the embeddings from enc2. We use another autoencoder to generate embeddings from topic distribution and reconstructed bag of words. The detailed structure and name are in Figure 3. We do not train finetune FlanT5 and CNN dur- ing the topic modeling process which makes our methods cost-effective. We then leverage diffusson to generate high quality text that represents the document.
The Contextual Topic Model (CTM) (Bianchi et al., 2021a) uses sentence embeddings and bag of words as input reconstruct bag of words em- beddings, while BERTopic (Grootendorst, 2022) combines sentence embedding and clustering tech- niques like UMAP and HDBSCAN for topic gen- eration. Other models (Han et al., 2023) use both clustering techniques and reconstruction to create high-quality topics. Nonetheless, contextual em- bedding based topic modeling methods lack re- construction process only reconstruct bag of words representations. These disadvantages limit its ability to generate relevant content. We exam- ined other related works in Appendix
2.4 Diffusion
Drawing inspiration from non-equilibrium ther- modynamics, the diffusion model adds noise to the data distribution in forward process and learns reverse denoising process -ihh--Dickstein et al., 2015). (Song and Ermon, 2020) further ap- plied this for high-quality image generation, com- parable leading likelihood-bar
This section contains four components. First, we present the dataset and define the finetuned task. Second, we elaborate on our modified FlanT5 and the fine-tuning strategy. The third component intro- duces a variational autoencoder designed for topic
models and GANs (Goodfellow al., 2014), but with more stable training and generation due to iterative diffu- sion.

---

VAC GROVOR
VAE dUOOd
cmc
Thalo
Ward
cRRC)
I' Dispi
Daue Nat
doc
does
No
Ni Dutes
DM seceder
Phe-T seceder
Figure 2: DeTiME framework. We have 4 encoders and 4 decoders. enco and ency are compressing the input document lower dimension. encg to construct topic distribution. dee to reconstruct hag of words representations. enc; is extract the hidden dimension from the reconstructed bag of words. deer, decs and dec; is to reconstruuctrephrase the input document. In our method, we name the number of dimensions for embedding Dtoken and maximum sequence length Ni. The dimension of the compressed vector Dembed. The number topies equals T. The dimension of vocabulary is NBow. The dimension of topic embeddings is Dropie.
modeling and generation. Finally, we utilize dif- fusion to generate content relevant to the derived topics.
for comparing meaning representations, to form our dataset (Agirre et al., 2012, 2013, 2014, 2015, 2016). We select pairs with scores above 80 per- cent of the maximum, yielding a total of 22,278 pairs. This dataset addresses the limitations of ex- isting paraphrase datasets, which are either domain- specific (Dolan and Brockett, 2005; Gohsen et al., 2023), generated by potentially unreliable lan- guage models (Shumailov et al., 2023). Our com- posite dataset diverse, including data from news, captions, and forums.
3.1 Tasks and Finetune Dataset
To achieve effective topic modeling methods, aim to generate embeddings that are highly cluster- able and capable of generating document-relevant topics. We utilize paraphrase dataset in which the input and output sentences are equivalent in meaning. Such equivalent sentences should be- long to similar topics, thereby aiding us in gen- erating similar sentences. In contrast to methods that use the same sentence for both input and out- put, our task assists the language model in learning the semantic meaning of sentences rather than sim- ply memorizing the embeddings. As illustrated in Figure. 1, the DeTiMEE-trining model represents the model generated by the task where the input and output are identical contents. As you can see, the clusterability of this method is substantially lower than ours. Thus, rephrase task is effective generate clusterable contents. Moreover, the paraphrase task is not sufficiently easy (Vahtola al., 2022) and is less likely to impair the utility of the language model. We concatenate similar sentence pairs from the STS benchmark, a dataset
3.2 Modified Encoder Decoder LLM
The motivation for this nested autoencoder struc- ture stems from the limitation of existing sentence embeddings, which struggle reconstruct sen- tences as they are primarily trained using con- trastive learning (Reimers and Gurevych, 2019) rather than reconstruction. In other words, simi- lar sentences are distributed close to each other in the learned embedded vector space. We choose an encoder-decoder model due to its ability to preserve essential information through encoding process. Specifically, encoder-decoder approaches, like T5, encapsulate vital information in the encoder's final hidden state. We can compress this final hidden state to create our embeddings. FlanTS (Chung

---

al., 2022) outperforms T5 in standard tasks by leveraging (Wei al., 2023) and instruction fine-tuning (Chung et al., 2022). We believe that the final hidden layer of a fine-tuned FlanT5 can represent the input information.
(VAB) aiming to reconstruct a bag of words pro- duces high-quality topic embeddings. Our VAE has two encoders and two decoders. ency is used to encode the output of the CNN encoder (e) into a topic distribution t. encg has two parts: the first is a multi-layer perceptron (MLP) that maps the input to a lower dimension, and the second consists of two MLPs to generate the mean and the log of the standard deviation vector of size T: p, log(o) ency(e). We sample a latent repre- sentation using the mean and standard deviation:
The purpose of CNN to compress output from FlanT5 encoder to create embeddings for topic modeling as illustrated in Append F. Using the encoder output as an embedding leads to exces- sive length and dimensionality, causing sparsely distributed vectors, poor clusterability, and issues in downstream tasks like topic modeling. To ad- dress this, incorporate variational autoencoder reconstruct FlanT5's final encoder hidden layer. We trained MIP, RNN, and CNN-based autoen- coders, but MLP introduced too many parameters and underperformed. LSTM, bidirectional LSTM, and GRU (Sherstinsky, 2020), with varied attention schemes (Xia et al., 2021), mostly yielded empty results or identical output embeddings, likely due to the FlanT5 encoder's non-sequential informa- tion processing. Applying ID convolution on the sequence dimension allowed for dimensionality reduction, with nearby embeddings showing high correlation, suggesting possible compression using a convolutional network on the sequence side. We can adapt the same framework to other exist- ing encoder decoder LLM such as BART (Lewis et al., 2019).
~ (N, σ), and apply a softmax function to generate the topic distribution
softmar(n). The dec3 is used to decode the topic distribu- tion t into a bag-of-words representation X'. Ex- isting research (Dieng et al., 2020) shows that topic-words similarity matrix offers better qual- ity in reconstructions. The decoder consists of two matrices. We use a vocabulary embedding matrix ey E RPropiccX XNBoww where Dropic rep- resents the dimension of word embeddings and Bow represents the dimension of the vocabulary. The decoder & learns a topic embedding matrix er E TTTxDropix The topic-to-word distribution is denoted
E = softmazeerre))
(1)
=x=
(2)
We utilize Parameter Efficient Fine-tuning (PEFT) because it reduces the number of param- eters to be fine-tuned, making the process more efficient and often yielding comparable or even su- perior performance to traditional fine-tuning (Liu al., 2022). We adopt prefix fine-tuning (Li and Liang, 2021) our work. During fine-tuning, we train both prefix fine-tuning related parameters and the CNN-based autoencoder for the paraphrase tasks. We then use the output from the CNN-based autoencoder's encoder for downstream topic mod- eling tasks. In our experiment, we use relatively small model FlanT5 base (248M parameters) to illustrate the effectiveness of our framework.
Here, X' rrepresents the reconstructed bag of words. The product of the generated topic distribution and this matrix E yields a bag-of-words reconstruction.
The enc; is neural network that encodes the generated bag of words back to a vector t', hav- ing the same dimension as the topic embeddings dimension: t' cnc)(X). We add residual con- nections between two compressed vectors and use a neural network to generate input embeddings:
è' = decs (t +()
(3)
It's necessary to reconstruct input embeddings (c) to be fed into the decoder to reconstruct the rephrased input sentence. We believe that the re- constructed bag of words can enhance the ability of sentence reconstruction. The residual connection helps the model leverage both the reconstructed bag of words and topic distribution to reconstruct input embeddings. This simplifies our input embedding reconstruction and ensures that the topic embed- dings can capture semantic information from the output of the CNN decoder e. Our VAE leverages
3.3
VAE structure for topic modeling
Our VAE serves two purposes. First, it generates highly clusterable topic distribution. Second, it reconstructs the output of the CNN encoder e, en- abling be input into the decoder of the CNN autoencoder. Prior research (Srivastava and Sutton, 2017) demonstrated that a Variational Autoencoder

---

only bag of words representations and contextual embeddings. Our VAE can also take other con- textual embeddings as input. Our loss function has three components: the reconstruction loss for the bag of words, the reconstruction loss for input embeddings using mean square error, and the KL Divergence for the normal distribution. The loss for single input is as follows:
=(x,,) 1) time step is obtained by the diffusor as following:
2' - X, + Sinusoid(t)
2² FCFOMP (2!)
2 = (CCOMP(E?)
2t FC3(=
2 - FORECONSS (2 + +2²)
2 2(x,, FC (2 + 2
(6)
L = XXlogXX) + (c- (c-c')² KI ())
This diffusor consists of 2 fully connected lay- ers FCCOMP to compress the input and 2 fully- connected layers FCRECONST to reconstruct. We also add residual connections between compress and reconstruct layers. Similar to UNet (Ron- neberger et al., 2015), the Sinusoidal positional embeddings Sinusoid(() is used encode time. Then, in generating component, this trained dif- fusor is used to denoise the embedding after the dee, in Figure 3. The intuition behind this denois- ing process is as follows. The forward process of diffusion itself is a process that converts the un- known and complex data distribution into one (nor- mal distribution in our case) that is easy sample from. By adding back the learned noise with small iterative steps, we are able to take a sample from the noise subspace (support a simple distribution) to the data subspace (support the unknown data distribution). Similarly, for an embedding obtained from the topic modeling that deviates from the em- bedding distribution corresponding to the unknown input data distribution, should also be able to take this embedding back to the area supporting the original embedding distribution.
3.4 Diffusion for content generation
Our pretrained model can compress the text and em- bed them in low-dimensional space while keeping the semantic information and high-quality cluster- ing. Is natural to wonder if this pretrained model can be used to generate topic-guided text. One of the challenges that the decompression process in the pretrained model may induce noise, loss some information and thus the quality of the generated text will be impacted. Specifically, the latent di- mension (i.e. the vector space of 2" before the deo2 in Figure 3) is several orders of magnitude lower than the dimension of embedding vector e' in DeTiME. When we reconstruct text from latent vectors, it may hugely deviate from any reasonable input for F5anT5 decoder dec3.
To overcome this, we have leveraged the dif- fusion models to denoise the generated text em- bedding from the topic modeling with structure as shown in Figure 3. It has demonstrated that the dif- fusion model is able generate high-quality text from noise samples the continuous embedding space (Li et al., 2022; Gong et al., 2023; Gao et al., 2022; Lin et al., 2023b). In the training compo- nent, we employ DDPM-schedules Autoencoder with residual connections as the diffusor (Ho et al., 2020) in the text embedding continuous space (i.c. the space after enc2 in Figure 3) using the embed- ded vectors obtained from the pretrained model. Specifically, during the forward process, the Gaus- sian noises is gradually added to Xo according to a variance schedule BI...., By, the noisy sample at time step is expressed as
Experimental Results
4.1 Topic Modeling
Dataset Our experiments are conducted on labeled benchmark datasets for topic modeling: AgNews (Zhang et al., 2016), 20Newsgroups (Lang, 1995)
and bbc-news (Greene and Cunningham, 2006). The average document length varies from 38 to 425. We use the text as it is for the contextual embedding generation. To get bag of words, we use the word tokenizer from nltk to tokenize, move digits and words with lengths less than 3, and remove stop words and words that appear less than 10 time. Additional details on the dataset and places to download processed data are available in Appendix B.
q( (o, = N
VI- anl)
(5)
Oz with a = Br Our diffusor is trained to minimize the squared error between the predicted and true noise. The predicted noise
where a. = 17

---

Methods
Purity
NMI
Km-Purity
Km-NMI
diveriity
C.
EIM
0.4677
0.04
0,2902 + 0.07
0,4063 = 0.07
0,2400
0.08
0.1177 = 0.05
0.5594
+ 0.01
GSM
0.2701
0.02
0.0687
1: 0.03
0.3167 + 0.03
0.1312 0.03
0.2991 + 0.01
0.3495
+ 0.01
VONT
0.3727 0.02
0.1004
0.03
0.4941
0.05
0,2088 0.05
0.6937 0.06
0.5151
+
NVDM
0.4254 0.04
0.2373
1. 0.07
0.3768 1 0.07
0.2138
0.05
0.2633
1 0.05
0.4715
1 0.02
0.3637
0.003
0.1087
0.6796
0.6703
CTM
0.4307 0.03
0.1611 +
0.4191 + 0.04
0.1819 = 0.05
0.7198 + 0.01
0.6966
+ 0.02
DeTIME Dow
+
0.03
0.02
0.7186
1 0.004
DeTME resi
0.3239 0.01
0.1098 + 0.01
0.4230 + 0.0T
0.1741 0.02
0.6802 + 0.01
0.7435
+ 0.002
DeTIME
0.03
0.5929 0.04
0.3463 0.05
0.6913 0.02
0.7203
Table 1: The main results for all clusterability metrics, diversity, and coherence (C,). The number of topics is 20. The best and second-best scores of each dataset are highlighted in boldface and with underline, respectively. The result represents the average value obtained from three datascts, where each dataset was processed 10 times to compute the mean and standard deviation.
Baseline Methods We compare with common NTM methods and contextual embedding based methods. We explain the reasons for choos- ing these methods in Appendix D. These meth- ods include: NVDM (Wang and YANG, 2020). VAE architecture for topic modeling with the en- coder is implemented by multilayer perceptron, the variational distribution is a Gaussian distribution; GSM (Miao et al., 2018), an NTM replaces the Brrichee-MMuliioomia parameterization in LDA with Gaussian Softmax; ETM (Dieng et al., 2020), an NTM model which incorporates word embed- ding to model topics; vONT (Xu al., 2023e), vMF based NTM where they the radius of vMF distribution equal to 10; CTM (Bianchi et al., 202 trains variational autoencoder to recon- struct bag of words representation using both con- textual embeddings as well as bag of words repre- sentation. ZTM (Bianchi et al., 2021b) is similar to CTM but only use contextual embeddings; De- TiME resi is the DeTiME model with resid- ual connections. The reconstruction of embedding hugely dependent on the reconstructed bag of words; De TiME bow the DeTiME model with- out reconstruction of bag of words and is used to represent topics.
Training
Generating
del
Figure 3: The diffusion framework based on the main framework Figure 2. In the training component, a DDPM-schedule Autoencoder with residual connec- tions diffusor is trained using the embedding vectors obtained from the encz. generating part, the trained diffusor is used denoise the embedding vectors trans- formed from the topic vectors hidden space before the generation. It's important note that normal- ized the hidden space before passing it to the decz.
Settings In experiment setting, The hyper- parameter setting used for all baseline models and DeTiME is the same as (Burkhardt and Kramer, 2019). For neural topic modeling and our encoder and decoder, we use a fully-connected neural net- work with two hidden layers of half of the hidden dimension and one quarter of hidden dimension and GELU (Hendrycks and Gimpel, 2023) as the activation function followed by dropout layer. We use Adam (Kingma and Ba, 2017) the optimizer with learning rate 0.001 and use batch size 256. We use (Smith and Topin, 2018) as scheduler and use learning rate 0.001. We use 0.0005 learning rate for the DeTiME bow because the loss may

---

overflow when the learning rate is 0.001. We use word embeddings (Mikolov al., 2013) to rep- resent word embeddings on the dataset for vONT, ETM, and DeTiME and keep it trainable for De- TiME. For vONT, we set the radius of the vMF distribution equal to 10. For CTM and ZTM, we use all-mppet-base-v2 as our embeddings since it performs the best in clusterability in Figure 1. We use the same way to find key words as suggested by CTM. Our code written in PyTorch and all the models are trained on AWS using ml..2.8xlarge (NVIDIA K80). Detailed code implementations for methods and metrics are in Appendix C
performance. By eliminating the bag of words reconstruction components, we found decrease
in diversity and clusterability, indicating the im- portance of this component in boosting purity and NMI. When we removed the residual connection, we observed an improvement in coherence but a de- crease in clusterabilityy This trade-off suggests that the absence of residual connection may prevent the topic distribution from effectively capturing the information from embeddings, thus reducing clus- terability. DeTiME resi performs better than ZTM in clusterability related metrics, which confirms that our embedding is more clusterable than ex- isting sentence embeddings.
Evaluation Metries We measure the topic clus- terability, diversity, and semantic coherence of the model. To measure clusterabilityy we assign ev- ery document the topic with the highest probability the clustering label and compute Top-Purity and Normalized Mutual
4.2 Diffusion for content generation
To evaluate how the diffusor improves the qual- ity of the generated text, we compared the gener- ated text before and after the diffusion. Specifi- cally, we utilized the Flesch Reading Ease (FRF), Flesch-Kincaid Grade Level (FKGL), and Dale- Chall Readability Score (DCRS) to measure the readability of the generated text before and after the diffusion (Goldsack et al., 2022). In general, higher FRE (lower FKGL and DCRS) indicates that the text easier to read. In this experiment, we generated 1000 random topic vectors and passed them to dec2, then the denoising process is fol- lowed to generate text. The main results are shown in Table 2. As observed, after the denoising pro- cess, the FRE increases significantly across all datasets, which indicates that diffusion makes the content easier to understand. Meanwhile, the value of FKGL and DCRS decreases from T = 500 to T 1000. One of the reasons for the low score of FKGL and DCRS T 0s that some of the samples contain only repeated words, making them easy to understand. Overall, after more steps in dif- fusion, the generated text becomes more readable for a lower grade. This experiment demonstrates that our generated content achieves higher read- ability, indicating the potential of our frame- work to generate topic-relevant content.
(Topp-Ni as metrics (Nguyen al., 2018) to evaluate alignment. Both of them range from 1. A higher score reflects better clustering performance. We further apply the KMeans algorithm to topic proportions z and use the clustered documents to report pu- rity(Km-Purity) and NMI Km-NMI (Zhao et al., 2020). We set the number of clusters to be the number of topics for the KMeans algorithm. Topic coherence(C.)) uses the one-set segmentation to count word co-occurrences and the cosine similar- ity as the similarity measure. Compared to other metries, C is able to capture semantic coherence. We only benchmark C. because most of coherence metrics are similar to each other (Lim and Lauw, 2023). For diversity, we measure the uniqueness of the words across all topics divided by total key- words. For each topic, set the number of key- words equal 25, Furthermore, we run all these metries 10 times. We report averaged mean and standard deviation. We also include evaluations on Perplexity in Appendix G
Results The experiment shows that DeTiME out- performs all other methods in NMI, Km-NMI, and Km-Purity, which underscores its ability to gener- ate highly clusterable topic distributions. Fur- thermore, DeTiME has the second highest scores in coherence(The highest score is also a TiME variation), validating the exceptional semantic co- herence of topics generated from our methods. Observations reveal that the CTM and DeTiME's high diversity scores highlight the benefit of incor- porating bag of words inputs, enhancing diversity
Human Evaluation To ensure the generated content is valuable to humans, human evalua- tion conducted with regard to the text gener- ated after diffusion, as seen in Figure 3. In this evaluation, we generated 400 pieces of text. Each piece was evaluated for fluency, grammar, and re- dundancy by three different human annotators, as suggested by (Celikyilmaz et al., 2021). We com-

---

Datasets
J0Newsproups
bbc-news
AgNews
Time posnt
T=
T= 1000
2' 0
500
1000
7 500
1000
FRE
-2.96600
51.1590
54.2467
6.8600
36,5889
60,9407
36.6200
64,1707
63.1074
FRGL.
53,200
10,2017
9,985
30,2000
12,6860
9.1856
8,4000
9A876
8.6781
DCRS
7,3500
8.4788
7,822
4,0100
8.3304
8.2010
66,5500
8,189
8.1089
Table The average readability scores different time steps during the denoising process. A general increase readability is observed.
pared our results with baseline through t-tests and found that the generated text exhibited fluency and grammatical correctness with statistical signifi- cance (p < le 14). This demonstrates that our generated contents are of high quality. More de- tails about the survey setup, results, and examples of generated text can be found in Appendix A.
generating a summarization for each topic, and subsequently concatenating them. This framework can also be extended to hierarchical topic model- ing (Chen et al., 2023; Shahid et al., 2023; Eshima and Mochihashi, 2023), mitigate data sparsity for short text topic modeling (Wu et al., 2022), gen- crate topic-relevant and coherent long texts (Yang et al., 2022), and construct network of topics gether with meaningful relationships between them (Byrne et al., 2022).
5 Conclusion and Future Work
We have developed a framework DeTiME for gen- crating highly clusterable embeddings, loveraging the strengths paraphrase tasks, FlanT5, and CNN. In addition to this, we introduced a variational au- toencoder structure capable of reconstructing em- beddings while simultaneously producing highly coherent, diverse, and clusterable topics. Our de- sign incorporates diffusion process for generating content that provides representative depictions of various topics. The flexibility our embedding generation structure allows for easy adaptation to other encoder-decoder language model architec- tures, climinating the need for retraining the en- tire framework, thereby ensuring cost-efffetivveness. Additionally, our variational autoencoder structure is versatile, and capable of being applied any contextual embeddings. Other methods could fur- ther improve with larger LLM.
Moving forward, we aim to further improve the performance of our embeddings by training on larger models such as Flan-T5-XL. Benchmark- ing other Pre-training with Fine-Tuning (PEFT) methods, such as LORA, may also enhance our system's performance. Given the high clusterabil- ity of our embeddings, we plan to extend our work to semi-supervised document classification (Xu et al., 2023b,c; Balepur et al., 2023; Lin et al., 2023a). This framework could be applied to iden- tify the most representative documents within ex- tensive document collections. This functionality could make our model suitable for generation topic guided generation (Xu et al., 2023a) Finally, we en- visage utilizing this framework to generate superior summarizations for large documents. This could be achieved by training a decoder for summarization,

---

6 Limitations
References
While our study has made significant strides in its domain, we acknowledge certain limitations that present themselves as opportunities for future research and optimization. Firstly, we have not yet benchmarked our model with other encoder- decoder frameworks such as BART, or with alter- native PEFT methods like LORA, leaving room for potential performance enhancement. We be- lieve that the diversity could further improve with diversity aware coherence loss (Li et al., 2023). Secondly, our model has yet to reach the full poten- tial of FlanT5 due to current model size constraints. This implics that scaling up the model could fur- ther improve its performance. Thirdly, we have not fine-tuned the number of dimensions for the CNN encoder output or explored structures beyond basic CNN, LSTM, and MLP, both of which could enhance our current performance. Fourthly, We noted a relatively high variance in DeTiME's per- formance, we interpret this as a consequence of the complicated autoencoder structure. Lastly, we have not benchmarked all coherence metrics. Though many metrics have similarities and some may not consider semantic word meaning, a more extensive benchmarking could provide richer evaluation of our approach. Despite these limitations, each of these points serves as a promising direction for future research, thereby helping to further elevate our model's 's capabilities.
Eneko Agirr,, Carmen Banea, Claire Cardie, Daniel Cer, Mona Diab, Aitor Gonzalez-Agire, Weiwei Guo, Iñigo Lopez-Gazpio, Montse Maritxalar, Rada Mihalcea, German Rigau, Larraitz Uria, and Janyce Wiebe. 2015. SemEval-2015 task 2: Semantic tex- tual similarity, English, Spanish and pilot on inter- pretability. In Proceedings of the 9th International Workshop on Semuntic Evaluation (SemEval 2015), pages 252-263, Deaver, Colorado. Association for Computational Linguistics.
Encko Agirre, Carmen Banea, Claire Cardie, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Weiwei Guo, Rada Mihalcea, German Rigau, and Janyce Wiebe, 2014. SemEval-2001 task 10: Multilingual semantic textual similarity. In Proceedings of the 8th Interna- tional Workshop on Semantic Evaluation (SemEval 2014), pages 81-91, Dublin, Ireland. Association for Computational Linguistics.
Eneko Agirre, Carmen Banea, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Rada Mihalcea, German Rigau, and Janyce Wiebe. 2016. SemEval-2016 task 1 Semantic textual similarity, monolingual and cross-lingual evaluation. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016) pages 497-511, San Diego, Califor- nia. Association for Computational Linguistics.
aut
Eneko Agirre, Daniel Cer, Mona Diab, and Aitor Gonzalez-Agirre. 2012. SemEval-22022 task 6: A pilot on semantic textual similarity. In *SEM 2012: The First Joint Conference on Lexical and Compu- lational Semantics Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop Semantic Evaluation (SemEval 2012), pages 385- 393, Montréal, Canada. Association for Computa- tional Linguistics.
Eneko Agirre, Daniel Cer, Mona Diab, Aitor Gonzalez- Agirre, and Weiwei Guo. 2013. *SEM 2013 shared task: Semantic textual similarity. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume I: Proceedings of the Main Confer- ence and the Shared Task: Semantic Textual Similar- ity. pages 32 32-43, Atlanta, Georgia, USA. Association for Computational Linguistics.
Melissa Ailem, Bowen Zhang, and Fei Sha. 2019. Topic augmented generator for abstractive summarization. ArXiv, https//90.00020
Saqib Aziz, Michael Dowling, Helmi Hammami, and Anke Piepenbrink. 2019. Machine learning in fi- nance: A topic modeling approach. European Finan- ciul Management, n/a(n/a).
Nishant Balepur, Shivam Agarwal, Karthik Venkat Ra- manan, Susik Yoon, Diyi Yang, and Jiawei Han. 2023. DynaMiTE: Discovering explosive topic evolutions with user guidance. In Findings of the Association

---

for Computational Linguistics: ACL 2023, pages 194- 217, Toronto, Canada. Association for Computational Linguistics.
Daniel Cer. Yinfei Yang. Sheng yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St. John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, Yun-Hsuan Sung, Brian Strope, and Ray Kurzweil. 2018. Universal sentence encoder.
1z Beltagy, Matthew E. Peters, and Arman Cohan, 2020. Longformer: The long--ocuune document transformer.
HeGang Chen, Pengbo Mao, Yuyin Lu, and Yanghui Rao. 2023. Nonlinear structural equation model guided Gaussian mixture hierarchical topic modeling. In Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 10377-10390, Toronto, Canada. Association for Computational Linguisties.
Moumita Bhattacharya, Claudine Jurkovitz, and Hagit Shatkay. 2017. Identifying patterns of associated- conditions through topic models of electronic medi- cal records. CoRR, abs/1711.1000 1.10960.
Federico Bianchi, Silvia Terragni, and Dirk Hovy.
2021a. Pre-training is a hot topic: Contextualized document embeddings improve topic coherence. In Proceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 759-766, Online, Association for Computational Linguisties.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Al- bert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdh- cry, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang. Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Ja- cob Devlin, Adam Roberts, Denny Zhou, Quoe V. Le. and Jason Wei. 2022. Scaling cnatructionfinettune language models.
Federico Bianchi, Silvia Terragni, Dirk Hovy, Debora Nozza, and Elisabetta Fersini. 2021b. Cross-lingua contextualized topic models with zero-shot learning. In Proceedings of the 16th Conference of the Euro- pean Chapter of the Association for Computational Linguistics: Main Volume, pages 1676-1683, Online. Association for Computational Linguistics.
Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes. 2018. Supervised learning of universal sentence representations from natural language inference data.
David M. Blei, Thomas L. Griffiths, and Michael I. Jor- dan. 2009. The nested chinese restaurant process and hayesian nonparametric inference of topic hierar- chies.
Jeremy Costello and Marek Reformat. 2023a. Rein- forcement learning for topic models. In Findings of the Association for Computational Linguistics: ACL 2023, pages 4332-4351, Toronto, Canada. Associa- tion for Computational Linguistics.
David M Blei. Andrew Ng. and Michael Jordan. 2003. Latent dirichlet allocation. the Journal of muchine Learning research, 3:923-1022.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger. Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler. Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Matcusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners.
Jeremy Costello and Marek Z Reformat. 2023b. Rein- forcement learning for topic models, arXiv preprint wwXivv2305.0484
Peng Cui and Le Hu. 2021. Topic-guided abstractive multi-document summarization. In Findings of the Association for Computational Linguisties: EMNLP 2021, pages 1463-1472, Punta Cana, Dominican Re- public. Association for Computational Linguisties.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understand- ing.
Sophie Burkhardt and Stefan Kramer. 2019, Decoupling sparsity and smoothness in the dirichlet variational autoencoder topic model. Journal of Machine Lear- ing Research, 20(131):1-27..
Adji B. Dieng, Francisco J. R. Ruiz, and David M. Blei. 2019. Topic modeling in embedding spaces.
Ciarán Byme, Danijela Horak, Karo Moilanen, and Amandla Mahona. 2022. Topic modeling with topo- logical data analysis. In Proceedings of the 2022 Con- ference Empirical Methods in Natural Language Processing, pages 11513, Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics.
Adji B Dieng, Francisco JR Ruiz, and David M Blei. 2020. Topic modeling in embedding spaces. Trans- actions of the Association for Computational Linguis- tics, 8:439 453.
Thanh-Nam Doan and Tuan-Anh Hoang. 2021. Bench- marking neural topic models: An empirical study. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4363-4368.
Asli Celikyilmaz, Elizabeth Clark, and Jianfeng Gao. 2021. Evaluation of text generation: A survey.

---

William B. Dolan and Chris Brockett. 2005, Automati- cally constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing ( (IWP2005).
Diederik Kingma and Max Welling. 2013. Auto-
encoding variational bayes.
Ken Lang. 1995. Newsweeder: Learning to filter net- news. In Machine Learning Proceedings 1995, pages 331-339. Elsevier.
Shusei Eshima and Daichi Mochihashi. 2023. Scale- invariant infinite hierarchical topie model. In Find- ings of the Association for Computational Linguisties: pages 11731-11744 Toronto, Canada. As- sociation for Computational Linguistics.
Mike Lewis. Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zeullmmyer. 2019. Bart: De- noising senence-to-seqeene pre-training for natural language generation, translation, and comprehension.
Zhujin Gao, Junliang Guo, Xu Tan, Yongxin Zhu, Fang Zhang, Jiang Bian, and Linli Xu. 2022. Difformer: Empowering diffusion model on embedding space for text generation. arXiv preprint arXiv:2212.094111
Jingling Li, Mozhi Zhang, Keyulu Xu, John P. Dick- erson, and Jimmy Ba. 2021. How does neural network's 's architecture impact its robustness to noisy labels?
Marcel Gohsen, Matthias Hagen, Martin Potthast, and Benno Stein. 2023. Paraphrase acquisition from im- age captions. In Proceedings of the 17th Conference of the European Chapter of the Association for Com-
Raymond Li, Felipe Gonzalez-Pizara Linzi Xing.
Gabriel Murray, and Giuseppe Carenini. 2023. Diversity- aware coherence loss for improving neu- ral topic models. In Proceedings of the 6 61st Annual Meeting of the Association for Computational Lin- guisties (Volume Short Papers), pages 1710-1722, Toronto, Canada. Association for Computational Lin- guistics.
putational Linguistics, pages 3348-3358, Dubrovnik, Croatia. Association for Computational Linguistics.
Tomas Goldsack, Zhihao 7hang, Chenghua Lin, and Carolina Scarton. 2022, Making science simple: Cor- pora for the lay summarisation of scientific literature.
Shansan Gong, Mukai Li, Jiangtao Feng, Zhiyong Wu, and Lingpeng Kong. 2023. Diffuseq: Sequence to sequence text generation with diffusion models. In The Eleventh International Conference on Learning Representations.
Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation.
Xiang Lisa Li, John Thickstun, Ishaan Gulrajani, Percy Liang, and Tatsunori Hashimoto. 2022. Diffusion- LM improves controllable text generation. In Ad- vances in Neural Information Processing Systems.
Ian J.
Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial networks.
Jia Peng Lim and Hady Lauw. 2023, Large-scale corre- lation analysis of automated metrics for topic models. In Proceedings of the 6l Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pages 13874-13898, Toronto, Canada. Association for Computational Linguistics.
Derek Greene and draig Cunningham. 2006. Practical solutions to the problem of diagonal dominance in kernel document clustering. In Proceedings of the 23rd International Conference on Machine Learn- ing, ICML '06, page 377-384, New York, NY, USA. Association for Computing Machinery.
Yang Lin, Xin Gao, Xu Chu, Yasha Wang. Junfeng Zhao, and Chao Chen. 2023a. Enhancing neural topic model with multi-level supervisions from seed words. In Findings of the Association for Computa- tional Linguistics: ACL 2023, pages 13361-13377. Toronto, Canada. Association for Computational Lin- guistics.
Maarten Grootendorst. 2022. Bertopie: Neural topic modeling with class-based if-idf procedure.
Amulya Gupta and Zhu Zhang. 2021.
Vector- quantization- based topic modeling, ACM Trans. In- tell. Syst. Technol. 12(3).
Zhenghao Lin, Yeyun Gong, Yelong Shen, Tong Wu, Zhihao Fan, Chen Lin, Nan Duan, and Weizhu Chen. 2023b. Text generation with diffusion language mod- els: A pre- traiiiinnng approach with continuous para- graph denoise. In Proceedings of the 40th Interna- tional Conference on Machine Learning, 1CML'23. JMLR.org.
Sungwon Han, Mingi Shin, Sungkyu Park, Changwook Jung, and Mecyoung Cha. 2023. Unified neural topic model via contrastive learning and term weighting. Proceedings of the 17th Conference of the Euro- pean Chapter of the Association for Computational Linguistics, pages 1794-1800 1809..
Dan Hendrycks and Kevin Gimpel. 2023. Gaussian еτror linear units (gelus).
Haokun Liu, Derck Tam, Mohammed Mugeeth, Jay Mo- hta, Tenghao Huang, Mohit Bansal, and Colin Raffel. 2022. Few-shot parameter-efffcient fine-tuning is better and cheaper than in-context learning.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020. De- noising diffusion probabilistic models.
Yishu Miao, Edward Grefenstette, and Phil Blunsom. 2017. Discovering discrete latent topics with neu- ral variational inference. In Proceedings of the 34th
Diederik P. Kingma and Jimmy Ba. 2017. Adam:
method for stochastic optimization.

---

International Conference on Machine Learning, vol- ume 70 of Proceedings of Machine Learning Re- search, pages 2410-2419. PMIR.
Nils Reimers and Iryna Gurevych. 2019, Sentence-bert, Sentence embeddings using siamese bert-networks.
Reutterer Reisenbichler, M. 2019. Topic modeling marketing: recent advances and rescarch opportuni- ties. J Bus Econ, 89.
Yishu Miao, Edward Grefenstette, and Phil Blunsom. 2018. Discovering discrete latent topics with neural variational inference.
M. Roberts, B. Stewart, D. Tingley, and E. Airoldi. 2013. The structural topic model and applied social science. Neural Information Processing Society.
Yishu Miao, Lei Yu, and Phil Blunsom. 2016. Neu- ral variational inference for text processing. In In- lernational conference muchine learning, pages 1727-1736. PMLR.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022. High- resolution image synthesis with latent diffusion mad- cls.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean, 2013. Efficient estimation of word representa- tions in vector space.
Olaf Ronneherger, Philipp Fischer, and Thomas Brox. 2015. U-net: Convolutional networks for biomedical image segmentation.
Niklas Muennighoff. Nouamane Tazi, Loic Magne, and Nils Reimers. 2022. Mteb: Massive text embedding benchmark. In Conference of the European Chapter of the Association for Computational Linguistics.
Andrew Rosenberg and Julia Hirschberg. 2007. V-
measure: A conditional entropy- external clus- ter evaluation measure. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLA) pages 410- 420, Prague, Czech Republic. Association for Com- putational Linguistics.
Dat Quoe Nguyen, Richard Billingsley, Lan Du, and Mark Johnson. 2018. Improving topic models with latent feature word representations.
Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gus- tavo Hernández Abrego, Ji Ma. Vincent Y. Zhao, Yi Luan, Keith B. Hall, Ming-Wes Wei Chang, and Yinfei Yang. 2021a. Large dual encoders are generalizable retrievers.
Sascha Rothe, Shashi Narayan, and Aliaksei Severyn. 2020. Leveraging pre-trained checkpoints for se- quence generation tasks. Transactions of the Associ- ation for Computational Linguistics. 8:264-280.
Jianmo Ni, Gustavo Hernández Ábrego, Noah Constant, Ji Ma, Keith B. Hall, Daniel Cer, and Yinfci Yang. 2021b. Sentence-t5: Scalable sentence encoders from pre-trained text-to-text models.
Simra Shahid, Tanay Anand, Nikitha Srikanth, Sumit Bhatia, Balaji Krishnamurthy, and Nikaash Puri. 2023. HyHTM: Hyperbolic geometry- geometry-based based hierar- chical topic model. In Findings of the Association for Computational Linguistics: ACL 2023, pages 11672- 11688, Toronto, Canada. Association for Computa- tional Linguistics.
OpenAI. 2023. Gpt-4 technical report.
Jonas Oppenlaender. 2022. The creativity of text-to-
image generation, In Proceedings of the 25th Inter- national Academic Mindtrek Conference. ACM.
Alex Sherstinsky. 2020.
Fundamentals of recurrent neural network (RNN) and long short-term memory (LSTM) network. Physica D: Nonlinear Phenomena, 404:132306.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Confer- ence on Empirical Methods in Natural Language Pro- cessing (EMNLP), pages 1332- 1543, Doha, Qatar. Association for Computational Linguisties.
Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Yarin Gal, Nicolas Papernot, and Ross Anderson. 2023. The curse of recursion: Training on generated data makes models forget.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learn- ing transferable visual models from natural language supervision.
Leslie N. Smith and Nicholay Topin. 2018 Super- convergence: Very fast training of neural networks using large learning rates.
Jascha Sohl-Dickstein, Eiiic A. Weiss, Niru Mah- eswaranathan, and Surya Ganguli. 2015. Deep un- supervised learning using nonequilibrium thermody- namies.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAl blog, 18):9
Jiaming Song, Chenlin Meng, and Stefano Ermon. 2022. Denoising diffusion implicit models.
Colin Raffel, Noam Shazeer. Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former.
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie- Yan Liu. 2020. Mpnet: Masked and permuted pre- training for language understanding.

---

Yang Song and Stefano Ermon. 2020. Generative mod- eling by estimating gradients of the data distribution.
Computational Linguistics: ACL 2023, pages 11387- 11402, Toronto, Canada. Association for Computa- tional Linguisties.
Akash Srivastava and Charles Sutton. 2017. Autoen- coding variational inference for topic models. arXiv preprint arXiv: lato:::03000000 00888
Weijie Xu, Jay Desai, "SHS" Srinivasan Sengamedu. Xiaoyu Jiang, and Francis lannacci. 2023b. S2vntm: semi-supervised vmp neural topic modeling. In ICLR 2023 Workshop on Practical Machine Learning for Developing Countries.
Teemu Vahoola, Mathias Creutz, and Jörg Tiedemann. 2022. It is not easy to detect paraphrases: Analysing semantic similarity with antonyms and negation us- ing the new SemAntoNeg benchmark. In Proceed- ings of the Fifth BlackboxNLP Workshop on Analyz- ing and Interpreting Neural Networks for NLP, pages 249-262, Abu Dhabi, United Arab Emirates (Hybrid), Association for Computational Linguistics.
Weijie Xu, Billy Jiang, Jay Desai, Bin Han, Fugin Yan, and Francis lannacci. 2023c. Kdstm: Neural semi- supervised topic model-ing with knowledge distilla- tion. In ICLR 2022 Workshop on Practical Machine Learning for Developing Countries.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017, Attention all you need.
Weijie Xu, Xiaoyu Jiang, Srinivasan H. Sengamedu, Francis lamnacci, and Jinjin Zhao. 2023d. vontss: vmf based semi-super neural topic modeling with optimal transport.
Boyu Wang, Linhai Zhang, Deyu Zhou, Yi Cao, and Jiandong Ding. 2023. Neural topic modeling based on cycle adversarial training and contrastive learning. In Findings of the Association for Computational Linguistics; ACI. 2023, pages 9730-9731, Toronto, Canada. Association for Computational Linguisties.
Weijie Xu, Xiaoyu Jiang, Srinivasan Sengamedu Hanu- mantha Rao, Francis lannacci, and Jinjin Zhao. 2023e. vONTSS: vMF based semi-supervised neural topic modeling with optimal transport. Findings of the Association for Computational Linguistics: ACL 2023, pages 4433-4457, Toronto, Canada. Associa- tion for Computational Linguisties.
Wenlin Wang. Zhe Gan, Hongteng Xu, Ruiyi Zhang. Guoyin Wang, Dinghan Shen, Changyou Chen, and Lawrence Carin. 2019, Topic-guided variational au- toencoders for text generation.
Erguang Yang. Mingtong Liu, Deyi Xiong, Yujie Zhang, Yufeng Chen, and Jinan Xu. 2022. Long text genera- tion with topic-aware discrete latent variable model. In Proceedings of the 2022 Conference on Empiri- cal Methods in Natural Language Processing, pages 8 8100-8107, Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics.
Xinyi Wang and YI YANG. 2020. Neural topic model with attention for supervised learning. In Proceed- ings of the Twenty Third International Conference on Artificial Intelligence and Statistics, volume 108 of Proceedings of Machine Learning Research, pages 1147-1156. PMLR.
Haopeng Zhang, Xiao Liu, and Jiawei Zhang. 2023. Dif- fuSum: Generation enhanced extractive summariza- tion with diffusion. In Findings of the Association for Computational Linguistics: ACL 2023, pages 13089- 13100, Toronto, Canada. Association for Computa- tional Linguisties.
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoe Le, and Denny Zhou. 2023. Chain-of-thought prompting elic- its reasoning in large language models.
Xiaobao Wu, Xinshuai Dong, Thong Nguyen, and Anh Tuan Luu. 2023, Effective neural topic mod- eling with embedding clustering regularization.
Xiang Zhang. Junbo Zhao, and Yann LeCun. 2016. Character-level convolutional networks for text clas- sification.
Xiaobao Wu, Anh Tuan Luu, and Xinshuai Dong. 2022. Mitigating data sparsity for short text topic modeling by topic-semantic contrastive learning. In Proceed- ings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2748-2760 Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Zihan Zhang, Meng Fang. Ling Chen, and Mohammad- Reza Namazi- Rad. 2022. Is neural topic modelling better than clustering? an empirical study on cluster- ing with contextual embeddings for topics. In North American Chapter of the Association for Computa- tional Linguistics.
Jun Xia, Yunwen Feng, Cheng Lu, Chengwei Fei, and Xiaofeng Xue. 2021. Lstm- multi-layer self- attention method for remaining useful life estimation of mechanical systems. Engineering Failure Analy- sis, 125: 105385.
He Zhao, Dinh Phung. Viet Huynh. Trung Le, and Wray Buntine. 2020. Neural topic model via optimal trans- port.
Jinjin Zhao, Kim I Larson, Weijie Xu, Neelesh Gattani, and Candace Thille. 2021a. Targeted feedback gen- eration for constructed-response questions. AAAI 2021 Workshop on Al Education.
Chumpu Xu, Jing Li, Piji Li, and Min Yang. 2023a. Topic-guided self-introduction generation for social media users. In Findings of the Association for

---

Jinjin Zhao, Weijie Xu, and Candace Thille. 2021b. End-to-end question generation to assist formative assessment design for conceptual knowledge learning. In AETS 2021.
Fieeryy
200
Eoo nons
200
54-
Deyu Zhou, Xuemeng Hu, and Rui Wang. 2020. Neural topic modeling by incorporating document relation- ship graph. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Process- ing (EMNLP), pages 3790-3796, Online. Association for Computational Linguistics.
152
Hao Zou, Zac Myung Kim, and Dongyeop Kang. 2023.
-25
Diffusion models in nip: A survey.
Redlndarcy
233ee graus
Appendix
52-
A Qualitative study
-20
-25
-3.5
0.3
2.1
3.0
this appendix, we mainly discuss how we set up the qualitative survey for diffusion-baad based text generation.
Figure 4: The histogram of survey scores from fluency, grammar, and redundancy perspectives.
As mentioned in the main content, we mainly measure the fluency, grammar, and redundancy of the generated text. Based on this reference (Ce- likyilmaz al., 2021), have designed the corre- sponding questions in Table. 3. For each question, five answer options are listed from strong negative to strong positive, and a score assigned to each option. In this survey, we have sampled 400 one- hot topic vectors and generated text following the generating component in fig. for each datasets. We then leverage the Amazon Mechanical Turk to evaluate the quality of each generated sentence. In this process, We have requested three independent reviewers mitigate the individual bias, and the average score calculated for each sample. The histogram the collected scores shown in fig.4. At the end, we have employed a t-test to evaluate if this survey is statistically significant. The null hypothesis has been tested against the one-sided al- ternative that the mean of the population is greater than 0 for fluency and grammar, and the null hy- pothesis against the one-sided alternative that the mean of the population is less than 1 for redun- dancy. The p < le - 14 have been obtained and thus we can reject the null hypothesis for all of them.
ing process and five generated text with denoising from the same topic vector z".
B Datasets
We have created a huggingface account to upload all relevant data used for training our modified FlanTS: htaps:s//huggnngaaeee wwjzdss/pretriinstt
We use the same account to upload all raw data for topic modeling you can see: https: https://hugginggacce. /tgp://cccccc/taatttsst/s/////// ag_news, hts.eessshtttss//uunggggggeeeeeo xwjzds//bbcnnwws, and https://hugginnfacee co co//attsttts//jic////eeeeerrrrgs have uploaded text after preprocessing here:
We
ag_nees__eema__trainn, https///hhh///gggiiiigaaae co oo//attsssts/////occoo/oreeeeeea and http:://uuggingfacccccodaaaaaa htards/2__nessgrruss_eeemmm
We have uploaded words used for bag of words here:
http://hwggnnaaaee.cccc///tttt/s ag_ htt.wwss.kkeoorrrsstttt////iiiig datsssts/wwjpdeebbbeeeeeeeevver
co/ and
We use the ratings and word intrusion tasks as human evaluations of topic quality. We recruit crowdworkers using Amazon Mechanical Turk in- side Amazon Sagemaker. We pay workers 0.024 per task. We select crowdworkers per task for 400 generated contents per task.
ht https::/huggingfccccoo/ cotttaatsaccscttttttttttttssttte /wwjds///////////ss 20 nww.groupssppcc -eawordss...ddss... Overall, we use 3 datasets that combines different domain to evaluate the performance.
(1) AgNews We use the same AG's News dataset from (Zhang et al., 2016). Overall it has 4 classes and, 30000 documents per class. Classes categories
In Table.4 below, we present a comparison be- tween a sample text generated without the denois-

---

Table 3: The setup of survey
metrics Fluency
question
options
Is the language in the sentence fluent?
not flunnt all (-2)
not fluent (-1)
neutral
fluent (1)
very fluent
Grammar
How grammatical the generated is?
not grammatical at all (-2)
not grammatical (-1)
neutral
gramuaticaal (1)
perfectly grammatical (2)
Redundancy
How repetitive or redundant the generated text is?
not redundant all (-2)
not redundant (-1)
neutral (0)
redundant (1)
very redundant (2)
Table 4: The generated text with and without denoising
The text generated without denoising "I'm not sure this good idea or not, Imm sure good
The generated with denoising act of removing a bacteriophage from a plant a source of danger." "a few years ago blond man was driving a honda civic car."
idea."
"the act of putting letter or symbol in docu- ment." "the man, who philanthropist died in a car crash in april, 2000."
"the act of stealing car."
include World, Sports, Business, and Sci/Tech.
I-dimension convolution layers with GELU as the activation function. In channel is 256, 32, and 4. Kernal size is and stride is 1 and padding is 1. We set dropout after the convolution layer with dropout rate equal to 0.2. We have not systematically finetuned these parameters, We trained our modified FlanT5 for 3 times and choose the lowest validation loss model as our model to run topic modeling. It took less than 20 hours for single gpu to finetune task.
(2) bbe-news (Lang, 1995) has 2225 texts from bbe news, which consists of 5 categories in total. The five categorics we want to identify are Sports, Business, Politics, Tech, and Entertainment.
(3) 20Newsgroups (Lang, 1995) is collection of newsgroup posts. We only select 20 categories here. Compare to the previous datasets, 20 cate- gories newsgroup is small so we can check the per- formance of our methods on small datasets. Also, the number of topics is larger than the previous one.
Code for comparable methods Code we used to implement GSM is https://gi thug.thhu.... comb Nawaiian/eaaalaocnmeeee
with topic covariance penalty equals to The code we used to implement ETM is https: //gbb com/ com/adj i Weiinn//TT ntm.py in zip file where we rewrite and includes all relevant topic modeling methods.
C Code
Code for our architecture Enc, and Enca is modified from https://uuggingfee co/transfforers//v.. 0. http://33........oee.e 0.2//modules/
transformers/mooder innelllnnt___a_ Batinng_t...... t5. htm1#T5Model.
forward by modifying the forward process. Training process for modified T5 We use google/flan-t5-bass
The code we used to implement CTM and ZTM is https: //gt thub. www.MiiaalPrroo...oooooo contextualized-topii models For CTM and ZTM, We the number of samples for topic pre- dictions equal to 5 and used their default prepro- cessing methods. The code we used implement VONT is derived from https: t/githb... thub. com/ Yongfeeian//Neut Document-
basic model. We use 20 percent data the validation set. We train 20 epochs when validation loss deteriorates consistently for 3 epochs. We set the number of virtual tokens equal to 20. We set the learning rate to 0.01. For the CNN encoder we have
The

---

code we used sentence embeddings vectors is from huggingface: https://hugginge: co/ sentence- transformers gsm-vae.py is where we implement our version of topic modeling.
2023) have code. We only include methods that leverage language models do an apple-to-apple comparison and exclude methods using graph neu- ral network (Zhou et al., 2020) or reinforcement learning (Costello and Reformat, 2023b).
Code for metric diversity is implemented us- ing scripts: https://gi thub. com/adj aoinenddddengg/ ETM/blob/masterr//ue httpssttteetilll.p.. py line 4. C. is im- plemented using gensim..oollls..ccrreeemmeeeee where coherence 'C,' Top-NMI implemented using mnctrics.normalieddn, in fogce from sklearn. Top-Purity is implemented by definitions. km based is implemented by the sklearn package kmeans
E Settings for clusterability evaluations
Compare existing sentence embedding methods with our proposed embedding on standard clus- terability metrics such as purity and NMI Ag- News dataset (Zhang et al., 2016). We compare our methods with GTR-T5 (Ni al., 2021a), Sentence- T5 (Ni et al., 2021b) and Mpnet (Song et al., 2020). We choose the largest version for all of them. De- TiME training is the embedding finetuned on the same dataset but we use the same input and output instead of rephrasing. We train 2 layer MLP neu- ral network variational autoencoder without soft- max suggested by (Miao et al., 2017). We choose the mean to represent the hidden dimension of the input. We train each embedding 10 times get the confidence band and average. The consistency of DeTiME's clusterability from 20 to 50 epochs suggests its potential suitability for topic modeling for the large number of topics.
D Compared Methods Selection
Sentence Embedding We choose GTR-T5 and Sentence-T5 because they the only two em- beddings that we are aware of T5 base models. They also perform well in clustering tasks (Muen- nighoff et al., 2022). We choose Mpnet because is commonly used in sentence embeddings and is the second best method in clustering, Benchmark- ing all these methods shows that our method is superior in sentence embeddings when the number of topics is large.
Topic Modeling There are many neural topic modeling methods but no standard benchmarks. For neural topic modeling methods, we choose NVDM because it performs well in (Doan and Hoang, 2021). We choose ETM because it is com- monly used and is the first one to leverage word em- beddings to topic modeling. We choose vONT (Xu al., 2023d) because it performs well in cluster- ability topic modeling metrics. We choose GSM because it also applicd softmax after sampling from the gaussian distribution. We think a similar comparison.
F
the purpose of CNN encoder
The theoretical advantage of CNN is to extract local features, reduce dimension reduction and be robust to noise(Li et al., 2021). In our cases, it helps to further extract important and local features from LLM encoder output and reduce dimensions.
The purpose of CNN is to compress output from FlanT5 encoder to create embeddings for topic modeling. The output of FlanT5 encoder is 256 (maximum sequence length used in our pre-train model) * 768 (embedding dimension) 393216. To illustrate points, rerun the same cluster- ability experiment but replacing CNN encoder with MLP. The topic purity drops from 0.614 to 0.396 and NMT drops from 0.31 to 0.05. This shows that CNN encoder helps the framework to achieve high clusterability and suitable for topic modeling. For efficiency, since the input dimension of encoder is 393216 1 and the output is 3072. MLP will require parameters 1207962624 parameters while CNN only reuqires 49624 parameters. This makes CNN is easy load and much effeient to train.
For contextualized topic modeling, we choose CTM and ZTM because they are the best perform- ing ones with code available. We exclude methods such as Veeic2Vec or Berttopic because it hard to define the number of topics or get the embeddings of documents to calculate clusterability. While many methods are derived from CTM, they either do not have code or are hard to use. For example, (Wu et al., 2023) is hard to process data for bbc news in the same format. In the future, we would like to benchmark methods such as (Costello and Reformat, 2023a) which leverage reinforcement learning, (Wang et al., 2023) which leverage adver- sairal training. Other methods such as (Han et al.,
This dimension is too high for any NTM to ex- tract information. Thus, we need to compress the obtained embeddings for topic modeling. Here,

---

using MLP only is hard build representations that incorporate information across the entire in- put text sequence as we just concatenate the em- beddings of each tokens and then fed into MLP. the same time, the authors in (Beltagy et al., 2020) have shown that very lightweight convolu- tion can perform competitively to the best reported self-attention results. This shows that CNN is ef- fective on extracting information from attention layer. Based on this, we thus leverage CNN en- coder to build representations that capture infor- mation across the text sequence. Also, by reduc- ing the number of oulput channels, we are able to obtain embeddings with reduced dimensions (4 768=3072 in our method), which hugely speed up the training.
(Cui and Hu, 2021) has leveraged Bert as part of encoder. However, they used NTM that taken bag of words as input and Bert/Grrah neural network as encoder. We instead use encoder and decoder LLM and we take embeddings from encoder as input to Neural Topic Modeling. Also, their goal is sum- marization but our goal is topic aware generation. (Ailem et al., 2019) propose new decoder where the output summary is generated by conditioning on both the input text and the latent topies of the document. The latent topics, identified by topic model such as LDA, reveals more global semantic information that can used to bias the decoder to generate words. In our work, we have leveraged a more promising topic modeling based encoder- decoder LLM. Also, the diffusion model is used to generate nigh-quality topic aware text, instead of summarization. (Zhang al., 2023) proposed to directly generating the desired summary sen- tence representations with diffusion models and ex- tracting sentences based on sentence representation matching. Even though this work leveraged the dis- tribution of embedded vectors of text for matching, it does not leverage the topic modeling, In compar- ison, our work have leveraged topic modeling, and also is able generate high-quality topic-aware text instead of extractive summarization.
Perplexity Evaluation
We have measured the perplexity for our method and other methods the dataset AgNews, and the results vNTM: 1479.32, ETM 692.17, NVDM: 1734.28, GSM 684.31 DeTiME: 612.71. This strengthen our conclusion of our work that De- TiME is promising in topic modeling. We use the same set up as (Gupta and Zhang, 2021) for this experiment.
Perplexity is not applied in topic-aware content generation and has not been used in topic modeling lately. We had not reported perplexity as the topic- aware content is generated from the sampled latent topic embeddings, where the ground truth (i.c. text sequence ground truth) is not available. The reason we used sampled latent topic embeddings is that we mainly focus how diffusion improve the quality of topic aware text generation.
Related Work
To the best of our knowledge, we are the first to fine- tune and modify encode-decoder LLM (i.e. Flan- T5) in topic modeling, and the first one to use dif- fusion in topic aware content generation with topic modeling, and integrate both in one unified frame- work. We do not find simpler structures to solve topic modeling and topic aware generation using encoder decoder LLM. For topic aware content generation using diffusion, there is compara- ble work and we have to establish all baselines ourselves for this. We have list other comparable works below and how our work distinguish from them:


Reviews:
[{"review_id": "6e9IO3218c", "reviewer": "Reviewer_Y61q", "summary": "", "questions": "1.\tWhat is the purpose of CNN encoder?\n\n2.\tWhy are there 2 rows for NVDM in Table2?\n\n3.\tWhat is the motivation of using Diffusion models for Topic Modeling?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks the reviewer for their thoughtful feedback! We will address them one by one. \n\n1. Question: What is the purpose of CNN encoder?\n\nAnswer: The purpose of CNN is to compress output from FlanT5 encoder to create embeddings for topic modeling. The output of FlanT5 encoder is 256 (maximum sequence length used in our pre-train model) * 768 (embedding dimension) = 393216. To illustrate our points, we rerun the same clusterability experiment but replacing CNN encoder with MLP. The topic purity drops from 0.614 to 0.396 and NMI drops from 0.31 to 0.05. This shows that CNN encoder helps the framework to achieve high clusterability and suitable for topic modeling.\n\nThis dimension is too high for any NTM to extract information. Thus, we need to compress the obtained embeddings for topic modeling. Here, using MLP only is hard to build representations that incorporate information across the entire input text sequence as we just concatenate the embeddings of each tokens and then fed into MLP. In the same time, the authors in [1, 2] have shown that a very lightweight convolution can perform competitively to the best reported self-attention results. This shows that CNN is effective on extracting information from attention layer. Based on this, we thus leverage CNN  encoder to build representations that capture information across the text sequence. Also, by reducing the number of output channels, we are able to obtain embeddings with reduced dimensions (4 * 768=3072 in our method), which hugely speed up the training. We also explained more in line 301-319\n\n\n\n[1] Beltagy, Iz, Matthew E. Peters, and Arman Cohan. \"Longformer: The long-document transformer.\" arXiv preprint arXiv:2004.05150 (2020).\n\n[2] Wu, Felix, et al. \"Pay less attention with lightweight and dynamic convolutions.\" arXiv preprint arXiv:1901.10430 (2019).\n\n\n\n2. Question: This paper is missing Perplexity which is one of the important metrics to quantify the quality of content generation.\n\nAnswer: We have run additional experiments to show the effectiveness of our methods in perplexity.\n\nWe have measured the perplexity for our method and other methods on the dataset AgNews, and the results are vNTM: 1479.32, ETM 692.17, NVDM: 1734.28, GSM 684.31 DeTiME: 612.71. This strengthen our conclusion of our work that DeTiME is promising in topic modeling. We use the same set up as [1, 2] for this experiment.\n\nPerplexity is not applied in topic-aware content generation and has not been used in topic modeling lately. We had not reported perplexity as the topic-aware content is generated from the sampled latent topic embeddings, where the ground truth (i.e. text sequence ground truth) is not available. The reason we used sampled latent topic embeddings is that we are mainly focus on how diffusion can improve the quality of topic aware text generation. \n\nFor topic modeling, we agreed that the perplexity is an important metric for topic modeling in LDA [3], but this metric has not been used recently. By search the papers title using key words topic model, we found 0/7 EMNLP 2022 papers have include perplexity in their work. Specifically, in [4], it mentioned that perplexity actually correlated negatively with human determinations of coherence as estimated using behavioral measures. To compare, 5 of them have used topic coherence and/or topic diversity, which have been included in our work.\n\n\n[1] Amulya Gupta and Zhu Zhang. 2021. Vector-Quantization-Based Topic Modeling. ACM Trans. Intell. Syst. Technol. 12, 3, Article 34 (June 2021), 30 pages. https://doi.org/10.1145/3450946\n\n[2] Amulya Gupta and Zhu Zhang. 2023. Neural Topic Modeling via Discrete Variational Inference.  ACM Transactions on Intelligent Systems and Technology 14.2: 1-33.\n\n\n[3] David M. Blei, Andrew Y. Ng, and Michael I. Jordan. 2003. Latent dirichlet allocation. J. Mach. Learn. Res. 3, null (3/1/2003), 993–1022.\n\n[4] Alexander Miserlis Hoyle, Pranav Goel, Rupak Sarkar, and Philip Resnik. 2022. Are Neural Topic Models Broken?. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 5321–5344, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.\n\n\n\n\n3. Question: What is the motivation of using Diffusion models for Topic Modeling?There is no clear motivation on why diffusion models can be utilized for Topic Modeling except the weak claim in Line 212-213. \n\nAnswer:  The motivation of using diffusion models is to improve the quality of generated contents. The topic modeling part of DeTIME may induce noise during the text reconstruction, and using Diffusion models is helpful to denoise the reconstructed continuous embedding vectors and thus to improve the quality of topic aware generated text. Specifically, the latent dimension is 20 but the input length of FlanT5 decoder is 393216 in DeTIME. When we reconstruct text from latent vectors, it may hugely deviate from any reasonable input for FlanT5 decoder. Diffusion can thus help through denoising to find the most closed and suitable embeddings for FlanT5 decoder . That is, we are mostly leveraging the diffusion to improve the quality of topic-aware text generation. Several papers[1,2,3,4] have demonstrated the advantages using diffusion model to generate high-quality text from noise samples in the continuous embedding space. We gave more detailed explanations in line 426-440.\n\n[1] Li, Xiang, et al. \"Diffusion-lm improves controllable text generation.\" Advances in Neural Information Processing Systems 35 (2022): 4328-4343.\n\n[2] Gong, Shansan, et al. \"Diffuseq: Sequence to sequence text generation with diffusion models.\" arXiv preprint arXiv:2210.08933 (2022).\n\n[3] Gao, Zhujin, et al. \"Difformer: Empowering diffusion model on embedding space for text generation.\" arXiv preprint arXiv:2212.09412 (2022).\n\n[4] Lin, Zhenghao, et al. \"Text Generation with Diffusion Language Models: A Pre-training Approach with Continuous Paragraph Denoise\" arXiv preprint arXiv:2212.11685 (2023).\n\n4. Question: A brief description of Diffusion models is missing. \n\nAnswer: Thanks for your comments. We have introduce DDPM in line 199-213. Also, we have documented the detail of diffuser in DeTIME in line 414-424. We agree that we need more details for diffusion and will add more references about using diffusion in text generation and corresponding brief descriptions in the final version.\n\n5. Question: I am not sure about the purpose of section 3.1 when “Datasets” and “Tasks” are introduced in section 4.1. \n\nAnswer: Section 3.1 explains datasets for pre-training. Section 4.1 are datasets for downstream tasks. We recognize how the current structure of the paper makes this distinction confusing.  To address this,  we will restructure the paper to introduce the datasets in section 3.\n\n6. Question: 1. It is difficult to understand the model description from Sections 3.2-3.3. For instance, dec1 is not mentioned in the text except in the caption of Figure2. My suggestion is that the Figure 2 and description in the text should match. The authors are requested to clearly explain all the different modules thoroughly available in Figure2.\n\nAnswer: we agree and will edit figure 2 and description to match in the final version. \n\n7. Question: Why are there 2 rows for NVDM in Table2? \n\nAnswer: it is a typo, the second one should be GSM. We will edit it in the final version.\n\n8. Question: Neural Topic Modeling via Discrete Variational Inference (https://dl.acm.org/doi/abs/10.1145/3570509)\n\nAnswer: Thanks for bring this reference to us. We believe this is an important work and will cite this paper in the final version. We will also benchmark DVITM_{full} in our final draft when the code is available.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.\tDeTiME performs better than the neural baseline models.", "reasons_to_reject": "1.\tThe paper is loosely written. \n\n      a.\tThere is no clear motivation on why diffusion models can be utilized for Topic Modeling except the weak claim in Line212-213.\n\n      b.\tI am not sure about the purpose of section 3.1 when “Datasets” and “Tasks” are introduced in section 4.1.\n\n      c.\tIt is difficult to understand the model description from Sections 3.2-3.3. For instance, dec1 is not mentioned in the text except in the caption of Figure2. My suggestion is that the Figure 2 and description in the text should match. The authors are requested to clearly explain all the different modules thoroughly available in Figure2.\n\n2.\tA brief description of Diffusion models is missing.\n\n3.\tThis paper is missing Perplexity which is one of the important metrics to quantify the quality of content generation.", "questions_to_authors": "1.\tWhat is the purpose of CNN encoder?\n\n2.\tWhy are there 2 rows for NVDM in Table2?\n\n3.\tWhat is the motivation of using Diffusion models for Topic Modeling?", "paper_topic_and_main_contributions": "For neural topic modeling, this paper proposes DeTiME (Diffusion-Enhanced Topic Modeling using Encoder-Decoder-based LLMs). Specifically, the model is built upon the following modules: Flan-T5 (Encoder-Decoder LLM), CNN (Convolutional Neural Network), VAE (Variational Auto-Encoder), and Diffusion. Experiments on 3 datasets: 20Ng, bbc-news and AgNews, show the effectiveness of the proposed model.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "1.\tNeural Topic Modeling via Discrete Variational Inference (https://dl.acm.org/doi/abs/10.1145/3570509)", "Typos_Grammar_Style_and_Presentation_Improvements": "Please correct the grammar of following sentence. Line 061-062: However, sentence embeddings are in general not perform well in clustering."}, "extra_text": {}}, {"review_id": "fiUp3eyBY9", "reviewer": "Reviewer_q8m2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful review and spending time on understanding our work.  Yes, finding an embedding method for topic modeling is really hard. That is the exact idea that leads to this work! Yes, without any piece of module, the framework won't work.\n\nWe will take the suggestions to heart and add more visualization, provide more examples, reduce bold fonts and apply to more datasets in our final draft. We will polish appendix and fix minor issues.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "\nI think the authors have a good observation that sentence embeddings do not lead to good clustering results. The proposed framework to me makes sense. The authors are able to back up their claims using experimental results. \n\nRegarding the experimental sections, I do have some suggestions:\n\n1. The authors could consider some visualization methods or whatever to provide some more insights on why the new embeddings are better compared to previous methods.\n\n2. The authors might want to provide more examples to demonstrate the meaning of the clusters and the associated sentences.\n\n3. The authors also might want to apply this method to more datasets to test the generalization of it. ", "reasons_to_reject": "\nI do not have major reasons to reject this paper.", "questions_to_authors": "", "paper_topic_and_main_contributions": "\nIn this paper, the authors present a study of generating embeddings that are suitable for clustering (and topic modeling) tasks. To do this, the authors propose an encoder-decoder LLM that is able to generate clusterable topics and related topic sentences. The authors validate their experimental results on some standard labeled benchmark datasets, including AgNews, 20News, and bbc-news. Experimental results show that the proposed method is outperforming several identified baselines.\n\nIn general, I agree with the authors' motivation that the currently available sentence embeddings are not totally suitable for clustering tasks. It is especially true for some domain-specific applications. The idea for conducting such a study is solid. Regarding the methods, although it is not totally new for each module used in the proposed framework. Nonetheless, the framework as a whole can not be treated as simple add-together of different blocks or incremental work. For the experiments, I am fine with the authors' choices of baselines along with the human evaluation. In sum, I think it is a solid work.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "\nSomehow I think the paper is written a little bit rushed. The Appendix is not polished. Also, I understand the authors use the bold fonts to emphasize their points, but I do not think it is necessary to have so much bolded text throughout the whole paper.\n\nOther minor issues include:\n\n- \"fig.3\" -> \"Figure 3\" or \"Fig. 3\". \n\n- line 520 and line 538, a blank space after the bracket"}}, {"review_id": "rNjMfaOZZ0", "reviewer": "Reviewer_WMyA", "summary": "", "questions": "Please see the reject.", "limitations": "", "rating": 2, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks the reviewer for their thoughtful feedback! I will address them one by one.\n\n1. Question: While DeTiME consists of CNN, VAE, and T5 (encoder and decoder) and employs a diffusion process, topic models with simpler structures have been proposed to tackle the same issues. The paper appears to be well-cited in the existing literature, but cites few models that attempt the same problem.\n\nAnswer: Thanks for your comments. To the best of our knowledge, we are the first to finetune and modify encode-decoder LLM (i.e. Flan-T5) in topic modeling, and the first one to use diffusion in topic aware content generation with topic modeling, and integrate both in one unified framework. We do not find simpler structures to solve topic modeling and topic aware generation using encoder decoder LLM.  For topic aware content generation using diffusion, there is no comparable work and we have to establish all baselines ourselves for this. We have list other comparable works below and how our work distinguish from them:\n\n    [1] has leveraged Bert as part of encoder. However, they used NTM that taken bag of words as input and Bert/Graph neural network as encoder. We instead use encoder and decoder LLM and we take embeddings from encoder as input to Neural Topic Modeling. Also, their goal is summarization but our goal is topic aware generation. \n    [2] propose a new decoder where the output summary is generated by conditioning on both the input text and the latent topics of the document. The latent topics, identified by a topic model such as LDA, reveals more global semantic information that can be used to bias the decoder to generate words. In our work, we have leveraged a more promising topic modeling based on encoder-decoder LLM. Also, the diffusion model is used to generate high-quality topic aware text, instead of summarization.\n    [3] proposed to directly generating the desired summary sentence representations with diffusion models and extracting sentences based on sentence representation matching. Even though this work leveraged the distribution of embedded vectors of text for matching, it does not leverage the topic modeling. In comparison, our work have leveraged topic modeling, and also is able to generate high-quality topic-aware text instead of extractive summarization.\n\nWe will add those comparison in revised version. We have done extensive literature on relevant work. If you find any other relevant work that is not covered by this, please share with us. We would love to citing and showing the difference between our work and other work.   \n[1] Cui, P., & Hu, L. (2021). Topic-guided abstractive multi-document summarization. arXiv preprint arXiv:2110.11207.\n\n[2] Ailem, Melissa, Bowen Zhang, and Fei Sha. \"Topic augmented generator for abstractive summarization.\" arXiv preprint arXiv:1908.07026 (2019).\n\n[3] Zhang, Haopeng, Xiao Liu, and Jiawei Zhang. \"Diffusum: Generation enhanced extractive summarization with diffusion.\" arXiv preprint arXiv:2305.01735 (2023).\n\n2. Question: Weak motivation for why these models were chosen and combined in this way.\n\nAnswer: Please see the detailed explanations of each modules and their combination:\n\nFlanT5: We need encoder/decoder model to extract the embeddings and generate relevant contents. At the point when we submitted the paper, FlanT5 did good job on both. Thus, we select FlanT5. Please find details in line 295-300.  \n\nCNN: This is used to compress the encoder output which has 393216 dimensions. To illustrate our points, we rerun the same clusterability experiment but replacing CNN encoder to MLP. The topic purity drops from 0.614 to 0.396 and NMI drops from 0.31 to 0.05. This shows that CNN encoder help the framework to achieve high clusterability and suitable for topic modeling. Please find details in line 301-319 and answer 1 for the reviewer Y61q\n\nVAE: Like all other neural topic modeling, this is used to generate topics. We chosen the exact structure because it performs well in both topic modeling relevant metrics compared to other variations see Table 1.\n\nDiffusion: The topic modeling part of DeTIME may induce noise during the text reconstruction, and using Diffusion models is helpful to denoise the reconstructed continuous embedding vectors and thus to improve the quality of topic aware generated text. Please find more details in 426-440 and answer 3 in reviewer Y61q\n\nCombination: Encoder Decoder model can give us a good embedding but the dimension is too high. Thus, we put CNN inside Encoder Decoder model. We need VAE to create Neural Topic Modeling method. NTM performs well when the dimension is not too high. Thus, we put it inside CNN encoder. We use diffusion because our latent dimension is low. Thus, we need diffusion to denoise it and generate high quality output. \n\n3. Question: For the above reasons, the number of benchmark models in the experiment is small and the number of evaluation items is also small.\n\nAnswer: For topic modeling, the number of benchmark models in our work is 6 and the number of evaluation metrics are 6 and evaluation datasets are 3. While we agree that there are many models and metric in topic modeling, we believe we covered most of main stream models and important variations, which strongly supports our conclusion in the paper. We explain the detailed reason why we pick these models in line 1172 - 1199. \n\nFor content generation, we benchmark our method at different diffusion time stamps across 3 metrics and 3 datasets as well as qualitative study. To the best of our knowledge, we are the first to leverage diffusion in topic aware content generation with topic modeling. Therefore there is no comparable work and we have to establish all baselines ourselves for content generation by diffusion.\n\n\n4. Question: Lack of research on hierarchical topic models and topic models integrated with neural networks.\n\nAnswer: All 6 selected benchmark models are topic models integrated with neural networks as explained in line 456-481. Thanks for bringing hierarchical topic models up, we agree that extending DeTiME to hierarchical topic modeling is our next step.  We exclude hierarchical topic models this time because DeTiME is a general topic modeling method and we only benchmark general topic modeling methods.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* Proposed framework, DeTiME, aims to close the gap that Neural Topic Models (NTMs) have leveraged contextual embeddings from LLMs, neglecting the potential benefits of harnessing the overall structure.\n\n* DeTiME is designed to exploit the capabilities of the encoder-decoder Large Language Model (LLM) and train an adapted encoder-decoder LLM.\n\n* Limitation is reasonable \n\n* Easy to read.", "reasons_to_reject": "* The structure of the framework is complex\n  * While DeTiME consists of CNN, VAE, and T5 (encoder and decoder) and employs a diffusion process,\n    topic models with simpler structures have been proposed to tackle the same issues.\n  * Weak motivation for why these models were chosen and combined in this way.\n\n* Weak literature review and experimentation\n  * The paper appears to be well-cited in the existing literature, but cites few models that attempt the same problem.\n  * For the above reasons, the number of benchmark models in the experiment is small and the number of evaluation items is also small.", "questions_to_authors": "Please see the reject.", "paper_topic_and_main_contributions": "This paper proposes a framework named Diffusion-Enhanced Topic Modeling using Encoder-Decoder-based LLMs (DeTiME).\nDeTiME is designed to generate topics that exhibit both superior clusterability and enhanced semantic coherence compared to existing methods.\nDeTiME leverages the strengths of paraphrase tasks, FlanT5, CNN, and a variational autoencoder structure capable of reconstructing embeddings while simultaneously producing highly coherent, diverse, and clusterable topics.\nBy exploiting the power of diffusion, \nDeTiME also provides the capability to generate content relevant to the identified topics. \nThis dual functionality allows users to efficiently produce highly clustered topics and related content simultaneously. \nDeTiME’s potential extends to generating clustered embeddings as well.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Lack of research on hierarchical topic models and topic models integrated with neural networks."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Almost all the reviewers have agreed that this is an interesting paper and has shown some good observations.

However, concerns have been raised, though not that critical, on the lacking details and experimental designs.

During the rebuttal period, the authors have partly addressed these concerns and only left some long-term concerns.
Decision: Accept-Findings