SAE-NTM: Sentence-Aware Encoder for Neural Topic Modeling
Year: 2023

Abstract:
Incorporating external knowledge, such as pretrained language models (PLMs), into neural topic modeling has achieved great success in recent years. However, employing PLMs for topic modeling generally ignores the maximum sequence length of PLMs and the interaction between external knowledge and bag-of-words (BOW). To this end, we propose a sentenceaware encoder for neural topic modeling, which adopts fine-grained sentence embeddings as external knowledge to entirely utilize the semantic information of input documents. We introduce sentence-aware attention for document representation, where BOW enables the model to attend on topical sentences that convey topic-related cues. Experiments on three benchmark datasets show that our framework outperforms other state-of-the-art neural topic models in topic coherence. Further, we demonstrate that the proposed approach can yield better latent document-topic features through improvement on the document classification.

heading: 1 Introduction
text: Topic models have been widely used to identify human-interpretable topics and learn text representations, which have been applied for various tasks in Natural Language Processing (NLP) such as information retrieval (Lu et al., 2011), summarization (Nguyen et al., 2021), and semantic similarity detection (Peinelt et al., 2020). A typical topic models is based on the latent Dirichlet allocation (LDA) (Blei et al., 2003) and Bayesian inference. However, to avoid the complex and expensive iterative inference of conventional topic models, topic modeling with the deep neural network has been the leading research direction in this field (Miao et al., 2016; Srivastava and Sutton, 2017; Ding et al., 2018).
Neural topic models (NTMs) usually exploit the BOW representation as input, disregarding the
∗Corresponding Author
syntactic and semantic relationships among the words in a document, thus leading to relatively inferior quality of topics. Recently, pre-trained language models (PLMs) (Kenton and Toutanova, 2019; Reimers and Gurevych, 2019) demonstrate their strong ability to capture sentential coherence by achieving state-of-the-art performance on many natural language processing tasks. Therefore, several approaches have been proposed to incorporate external knowledge into topic models to address the limitations of BOW. A typical method to take external knowledge as additional features (Bianchi et al., 2021; Jin et al., 2021) concentrates the outputs of PLMs with BOW data. Another way (Hoyle et al., 2020) is to distill the knowledge of the teacher PLMs to generate a smoothed pseudo-document, which guides the training of a student topic model.
However, there are still limitations to the above approaches. Firstly, the document-level sequences are too long to be modeled, since the token-level sequence in the context is usually considered as input to the PLMs. Extracting the document-level semantic embedding with PLMs as external knowledge ignores the restriction on sequence length, which loses massive semantic information from input text. Secondly, the difference in learning objectives between NTMs and PLMs makes it challenging to incorporate external knowledge. The encoder of NTMs is designed to handle the sparse BOW data, unable to take into account the dense contextual document embedding from PLMs.
To address these limitations, we build upon the framework of variational autoencoders (VAE) (Kingma and Welling, 2013) and propose a sentence-aware encoder for incorporating external semantic knowledge into topic models. The proposed approach integrates the advantages of NTMs and PLMs as encoders. Specifically, the encoder of the topic model is responsible for processing document-level BOW data like most NTMs, while the PLMs is used to encode sentence-
106
𝑫𝒐𝒄𝒖𝒎𝒆𝒏𝒕 𝑹𝒆𝒑𝒓𝒆𝒔𝒆𝒏𝒕𝒂𝒕𝒊𝒐𝒏𝒔
level semantic information as its original training objective. Different from previous approaches, our proposed framework considers cross-attention (Vaswani et al., 2017) between the BOW data and sentence embeddings, which leverages fine-grained semantic information for topic discovery.
To summarize, the main contributions of this paper are as followed: (1) We propose a novel framework SAE-NTM: Sentences-Aware Encoder for Neural Topic Modeling which leverages the cross-attention for incorporating external semantic knowledge in a sentence-aware manner. (2) Quantitative and qualitative experiments demonstrate that our proposed approach significantly outperforms the existing state-of-the-art topic models in topic coherence. (3) We show that the BOW-guided attention yields practical latent document-topic features, achieving better performance on the document classification task.

heading: 2 Methodology

heading: 2.1 SAE: Sentence-Aware Encoder
text: In this section, we introduce the sentence-level semantic information as external knowledge and propose a method to efficiently combine BOW data with external knowledge for document representations, as shown in Figure 1.
Encoder for bag-of-words and sentence sequences. Neural topic models with variational autoencoders usually take high-dimensional, sparse word counts xi as input and transform it into a lowdimensional dense feature hi to fit the variational autoencoders framework as formulated in Eq.1.
hi = Enc (xi) (1)
Where Enc : RV → RL is usually a multi-layer
perceptron (MLP) for the inference of the ith document representation.
Complementary to the orderless BoW, the context of the document carries more affluent and more sophisticated semantic information. And it can be represented as contextual embeddings by pretrained language models (e.g., BERT (Kenton and Toutanova, 2019)) from large corpora, which have a fine-grained ability to capture aspects of linguistic context. In this paper, we employ sentencetransformers (Reimers and Gurevych, 2019) to encode each sentence in the document as follows:
{ hi1, · · · , him } = Trans ({ si1, · · · , sim }) (2)
where sij is a sequence of tokens and h i j is the aggregated contextual embedding from the pre-trained sentence-transformers for the jth sentence.
Sentence-aware Attention. The contextual embeddings { hi1, · · · , him } and BOW representation hi jointly constitute the input of sentence-aware encoder. Then sentence-aware attention is employed to accomplish the interaction of word counts and semantic embeddings formulated in Eq.3.
di =
j=m∑
j=1
αijh i j
αij = exp(score(hi, hij))∑k=m
k=1 exp(score(hi, h i k))
(3)
Where the representation di of the ith document is a weighted sum of contextual embeddings{ hi1, · · · , him } and αij is the normalized attention of the jth sentence. Typically, the scoring function score is scaled dot-product attention. Sentence embeddings as external knowledge provide rich textual information, while the BOW data guides topic model in the assignment of attention on topical sentences, which contributes to capturing the co-occurrence patterns of the words.

heading: 2.2 Variational inference
text: Starting with the document representation, variational inference (Kingma and Welling, 2013) consider Logistic-Normal distribution as the posterior distribution q(z | x), whose mean µi and variance σi vectors are separately derived from the document representation through a linear layer. Then the reparameterization trick in Eq.4 is used to estimate the gradient.
zi = softmax (µi + σi · εi) (4)
where εi ∼ N (0, 1) denotes samples from the normal distribution and zi is the latent document-topic vector. Next, it attempts to reconstruct the original BOW data xi by modeling the words distributions of topics ϕ as follows:
xreci ∼ Multi ( softmax ( ziϕ T ) , N ) (5)
where ϕ ∈ RV ×K is the word-topic matrix and N is a vector of document lengths.
Finally, SAE-NTM are trained by maximizing the Evidence Lower Bound (ELBO) of the marginal likelihood of the BoW data:
L(x) = −Eq [log p(x | z)] + KL [q(z | x)∥p(z)] (6)
where log p(x | z), q(z | x) and p(z) are respectively the reconstructed data likelihood, the posterior distribution and prior Dirichlet distribution.

heading: 3 Experiments
text: In this section, we design empirical experiments to answer the following questions of concern in topic modeling. First, how effectively does SAE-NTM perform quantitatively and qualitatively in terms of topic quality? Second, how does SAE-NTM perform in automated document-topic inference for downstream tasks? Besides, more details about the impact of external knowledge on topic modeling can be found in Appendix A.

heading: 3.1 Experimental Settings
text: Datasets. We evaluate our proposed SAE-NTM on three benchmark datasets, which differ significantly in the domain, vocabulary size, and document length: 20Newsgroups (20NG, Lang, 1995) 1, Wikitext-103 (Wiki, Merity et al., 2016) 2, IMDb movie reviews (IMDb, Maas et al., 2011) 3. For
1qwone.com/~jason/20Newsgroups 2s3.amazonaws.com/research.metamind.io/ wikitext/wikitext-103-v1 3ai.stanford.edu/Ëœamaas/data/sentiment
consistency with prior work, we adopt the same preprocessing steps and train/dev/test split from the original papers for 20NG (i.e., 48/12/40), Wiki (i.e., 70/15/15), IMDb (i.e., 50/25/25).
Baselines. We compare our model with existing state-of-the-art neural topic models: W-LDA (Nan et al., 2019) is a neural model with wassestein autoencoder, which approximates the Dirichlet prior by minimizing Maximum Mean Discrepancy. SCHOLAR (Card et al., 2018) is a VAE-based neural topic model with a logistic normal prior to facilitate approximate Bayesian inference and provide a flexible way to incorporate document metadata. SCHOLAR+BAT (Hoyle et al., 2020) is a knowledge-distilled neural topic model where a BERT-based autoencoder as a teacher provides contextual knowledge for the student model. CTM (Bianchi et al., 2021) is a combined topic model with the incorporation of contextualized document embeddings in neural topic models. CLNTM (Nguyen and Luu, 2021) is a contrastive learning version of the neural topic model through a wordbased sampling strategy.

heading: 3.2 Evaluation in topic coherence
text: Since topic models aim to discover a set of latent topics that are meaningful and useful for humans (Chang et al., 2009), we evaluate topic coherence using the Normalized Mutual Pointwise Information (NPMI) which is significantly correlated with human judgments on topic quality (Aletras and Stevenson, 2013; Lau et al., 2014). Specifically, we first select the top 10 words under each topic generated by topic models, and then estimate NPMI scores with reference co-occurrence counts from the held-out corpus, e.g. the dev or test split.
As shown in Table 1, we report the results of the average NPMI over 5 runs with different random seeds for initialization for robustness. It can be observed that our model yields the most coher-
ent topics across all baselines for three benchmark datasets in NPMI scores. This demonstrates that our method promotes the overall quality of generated topics. More importantly, our model not only significantly outperforms the baseline without external knowledge such as SCHOLAR, but also surpasses other state-of-the-art neural topic models that incorporate external knowledge, such as CTM, SCHOLAR+BAT. It suggests that our approach is more efficient than others for incorporating external knowledge into neural topic models.
In addition to the quantitative evaluation, we also randomly extract sample topics from three datasets to gain an intuitive view on the quality of generated topics, as shown in Table 2. Obviously, the topic words generated by our model capture the concept of topics in the document rather than the baseline model. For example, it can be noticed that in the 20NG dataset our words are closely related to encryption (agency, wiretap, etc.), rather than some common words (industry, manufacturer, etc.) from SCHOLAR. The words generated by our model in Wiki are more focused on cantata and opera, while SCHOLAR drifts gradually away from the music topic to paintings. Similarly in the IMDb dataset, the topic words generated by our model reflect religion-related themes, which is different from SCHOLAR including off-topic words such as views, racist, etc.

heading: 3.3 Document Classification
text: Since the latent vectors inferred by neural topic models can be applied as text features (Nan et al., 2019), we employ the downstream task of document classification to compare the predictive performance of the models in addition to the evaluation of topic coherence. Specifically, we collect latent document-topic features from the trained neural topic models setting number of topics to 50 and use these vectors as inputs to train a Random Forest classifier on the training split separately.
We report classification accuracy on the test split of 20NG and IMDb in Table 3. It is worth noting that we aim to evaluate the predictive capability of topic models by the performance in document classification, rather than training the model to obtain higher accuracy. The document-topic features provided by our proposed model achieve best accuracy for all the datasets with a significant improvement. It demonstrates that the proposed sentence-aware encoder not only discovers topics that are more meaningful to humans, but also learns better latent document features.

heading: 4 Conclusions
text: In this paper, we propose a Sentence-Aware Encoder for Neural Topic Modeling framework: SAENTM to incorporate external knowledge into neural topic models. The proposed method can capture document information by performing attention on sequential sentences in a bag-of-words guided manner. Extensive experiments have shown that our framework can achieve state-of-the-art performance in topic coherence and encode better latent document-topic features. In the future, we would like to explore the possibility of integrating our approach with neural topic models built on other frameworks, such as generative adversarial training (Nan et al., 2019; Wang et al., 2020).
Limitations
The proposed model with sentence-aware encoder aims to efficiently incorporate external knowledge and bag-of-words for topic modeling, which means that in this work we are mainly interested in how documents should be encoded for topic inference. However, the decoder of topic models can also be coupled with word embeddings through factorization, such as embedded topic models (Dieng et al., 2020). It is worth exploring how hierarchical semantic embeddings can be employed for topic modeling with our model.
In this paper, we do not conduct any fine-tuning for the pre-trained language model. Our approach reveals how the frozen pre-trained language model can be effectively used to improve the performance of the topic model with limited computational overhead, given that the parameter size of the pretrained language model is much larger than that of the topic model. Moreover, fine-tuning pre-trained language models for topic modeling as an unsupervised learning task (Mueller and Dredze, 2021) is challenging.

heading: A Analysis on individual topics
text: To evaluate whether our improvements are meaningful on individual topics, we directly compare each of the aligned topics generated by the baseline SCHOLAR without external knowledge and our model. Follow previous works (Hoyle et al., 2020; Nguyen and Luu, 2021), we align the topics by using a variation of competitive linking to greedily approximate the optimal weight of the bipartite graph matching. And the weight of each link is calculated based on the similarity between their word distributions as measured Jenson-Shannon (JS) divergence (Wong and You, 1985; Lin, 1991). We iteratively select the topic pair with the lowest score based on JS divergence, separate the two topics from the topic list, and repeat until the rest JS score exceeds a certain threshold.
Figure 2 shows the JS-divergences for aligned topic pairs for three benchmark corpora. Based on visual inspection, we choose the most aligned 44 topic pairs to conduct the comparison, since there is no conceptual relationship between topic pairs
beyond this point and employ the same threshold across all three datasets for simplicity. Considering these aligned topic pairs conceptually related, we explore the impact of external knowledge on the baseline topic model on a topic-by-topic basis as shown in Figure 3. It can be observed that the number of topics with high NPMI scores from our model is apparently more than that of the baseline model. This means that the overall promotion achieved by our approach can be interpreted as identifying the topic space generated by the baseline models and in most cases, improving the coherence of individual topics.

heading: Future Work
text: In the future, we would like to explore the possibility of integrating our approach with neural topic models built on other frameworks, such as generative adversarial training (Nan et al. , 2019; Wang et al. , 2020). Limitations
The proposed model with sentence-aware encoder aims to efficiently incorporate external knowledge and bag-of-words for topic modeling, which means that in this work we are mainly interested in how documents should be encoded for topic inference. However, the decoder of topic models can also be coupled with word embeddings through factorization, such as embedded topic models (Dieng et al. , 2020). It is worth exploring how hierarchical semantic embeddings can be employed for topic modeling with our model. In this paper, we do not conduct any fine-tuning for the pre-trained language model. Our approach reveals how the frozen pre-trained language model can be effectively used to improve the performance of the topic model with limited computational overhead, given that the parameter size of the pretrained language model is much larger than that of the topic model. Moreover, fine-tuning pre-trained language models for topic modeling as an unsupervised learning task (Mueller and Dredze, 2021) is challenging