Enhancing Neural Topic Model with Multi-Level Supervisions from Seed Words
Year: 2023

Abstract:
Efforts have been made to apply topic seed words to improve the topic interpretability of topic models. However, due to the semantic diversity of natural language, supervisions from seed words could be ambiguous, making it hard to be incorporated into the current neural topic models. In this paper, we propose SeededNTM, a neural topic model enhanced with supervisions from seed words on both word and document levels. We introduce a context-dependency assumption to alleviate the ambiguities with context document information, and an auto-adaptation mechanism to automatically balance between multi-level information. Moreover, an intra-sample consistency regularizer is proposed to deal with noisy supervisions via encouraging perturbation and semantic consistency. Extensive experiments on multiple datasets show that SeededNTM can derive semantically meaningful topics and outperforms the state-of-the-art seeded topic models in terms of topic quality and classification accuracy.

heading: 1 Introduction
text: Unsupervised topic models, despite their efficiency in uncovering the underlying latent topics in text corpora (Blei et al., 2003), may suffer from poor topic interpretability as the semantic interpretability of latent space is poorly explored (Chang et al., 2009; Newman et al., 2011; Eshima et al., 2020)
∗Corresponding Author
and the generated topics may not match users’ desires (Jagarlamudi et al., 2012; Gallagher et al., 2017; Harandizadeh et al., 2022). To address this problem, topic seed words are incorporated as additional prior knowledge to provide richer semantic information and indicate users’ preferences. Compared to sample-wise information like document labels, seed words can be easier to access, more widely applicable, and with a milder level of human bias.
Many works in conventional topic models incorporate seed words as guidance. Some works extend Latent Dirichlet Allocation (LDA) into seeded models (Andrzejewski and Zhu, 2009; Jagarlamudi et al., 2012; Li et al., 2016; Eshima et al., 2020), and some draw inspiration from information theory (Gallagher et al., 2017) or word embeddings (Meng et al., 2020a). While most of the conventional topic models struggle with the growing number of topics and documents, with the recent development of neural topic models (NTM), keyETM (Harandizadeh et al., 2022) is proposed to incorporate seed words into NTM to combine the advantages of NTM of scalability on large datasets.
However, keyETM only focuses on regularizing word-topic relations with seed words and fails to combine document-level topic information, which is essential as the semantics of words may vary under different context documents. As shown in Figure 1(a), under different contexts, the word ’apple’ has different semantic meanings and may belong to
13361
different topics, even if it co-occurs with the seed word ’company’. This inspires us to incorporate supervisions from seed words into NTM on both word and document level and balance information from both levels for better inference of topics, thus achieving better topic interpretability.
There still remain challenges to effectively combining multi-level supervisions from seed words into the current framework of NTM. Firstly, the mean-field assumption made in current NTMs prevents the model from combining topic preferences of words and documents because they are assumed to be conditionally independent. Secondly, as shown in Figure 1(b), document level supervisions from seed words can be noisy due to the semantic ambiguity of natural languages. Previous work (Li et al., 2018) tried to tackle the problem via a neighbor consistency regularization. However, the neighbor-based method can be time-consuming, limiting the scalability on large datasets, and noisy neighbors may cause cumulative errors.
To address these challenges, we propose a novel neural topic model SeededNTM, which incorporates seed words as supervisions and autoadaptively balances information from both word and document level. During variational inference, we drop the mean-field assumption and make a context-dependency assumption to assist the inference of per-word topic assignment with context document information. Based on this assumption, we implement an auto-adaptation mechanism between multi-level information inspired by the idea of product of experts (Hinton, 2002). Moreover, to deal with the noisy document supervisions, we propose a novel regularizer that encourages
intra-sample consistency to avoid time-consuming neighbor finding and cumulative errors. The regularizer encourages consistency between perturbed samples to preserve local structures and consistency between the semantics of outputs from different encoders to improve robustness.
Our contributions are summarized as follows: • We propose SeededNTM, a novel neural topic
model that leverages supervisions from seed words on both word and document level. • We propose a reasonable context-dependency assumption and develop an auto-adaptation mechanism to automatically balance between word level and document level information. • We propose an intra-sample consistency regularizer to deal with noises from document level supervisions by encouraging both perturbation and semantic consistency,. • Extensive experiments on three public datasets show that SeededNTM can derive semantically meaningful topics and outperforms the state-of-the-art seeded topic models in terms of NPMI and classification accuracy.

heading: 2 Related Works

heading: 2.1 Neural Topic Model
text: The recent developments of neural variational inference (Kingma and Welling, 2014; Rezende et al., 2014) enable the application of neural networks on topic models to deal with scalability issues. NVDM (Miao et al., 2016) and ProdLDA (Srivastava and Sutton, 2017) are two representative works. Gaussian and logistic normal distribution are leveraged as approximations of the Dirichlet prior in the original LDA. Subsequently, various works have
been proposed (Nan et al., 2019; Dieng et al., 2020; Nguyen and Luu, 2021), aiming for better inference of topics. Among these works, the most relevant to our work is VRTM (Rezaee and Ferraro, 2020). It explicitly models each word’s the topic assignments zn while other works collapse them for simplicity. However, the mean-field assumption in VRTM prevents the model from combining context document information when inferring words’ topic preferences, limiting its performance.

heading: 2.2 Topic Model with Prior Knowledge
text: Introducing prior knowledge into topic models has been a widely adopted way to improve topic interpretability. Some works (Bianchi et al., 2021a,b) incorporate pre-trained embeddings of words and documents to convey prior knowledge from additional datasets. Though effective, topic models with pre-trained embeddings remain unsupervised, and cannot mine information based on users’ interests. Sample-wise knowledge, like labels (Blei and Mcauliffe, 2008; Wang and Yang, 2020) and covariates (Eisenstein et al., 2011; Card et al., 2018) can reflect the semantic structure information of the corpus but can be difficult to acquire and may introduce strong biases. On the other hand, topic seed words, as a kind of topic-wise knowledge, can be easier to access and more applicable. SeededLDA (Jagarlamudi et al., 2012) paired each topic with a seed topic and biased documents to topics if they have corresponding seed words. And keyATM (Eshima et al., 2020) improved upon SeededLDA by allowing topics with no seed word and better empirical hyperparameters. Anchored CorEx (Gallagher et al., 2017) proposed an information-theoretic framework and incorporates seed words by anchoring them to topics. CatE (Meng et al., 2020a) took category names as seed words and learned a discriminative embedding space for topics and words. And SEETOPIC (Zhang et al., 2022) improved upon CatE by using BERT to handle out-of vocabulary seed words. Recently, to combine the advantages of NTMs on scalability, keyETM (Harandizadeh et al., 2022) is proposed to incorporate seed words into NTM by regularizing word-topic relations with seed words and pre-trained word embeddings.

heading: 2.3 Weakly-Supervised Text Classification
text: Weakly-supervised text classification is a branch of classification task to build a text classifier with a few relevant words or descriptions for each cat-
egory and no sample-wise labels. Because of the similar settings with seeded topic modeling, a few topic model-based methods are proposed (Chen et al., 2015; Li et al., 2016, 2018), and some recent works (Meng et al., 2020b; Wang et al., 2021; Zhang et al., 2021) attempt to bootstrap the seed word list to obtain stronger supervisions.
Despite similar settings, weakly-supervised text classification and seeded topic modeling differ in many aspects. While seeded topic modeling aims at discovering latent semantic structures of current corpus and focuses on the interpretability of learned topics, weakly-supervised text classification aims to build classifiers that generalize well on unseen data and focuses on the validity of the document-category partitions. Unsupervised topics are allowed in seeded topic modeling, and documents are interpreted as mixtures of multiple topics, while in weakly-supervised text classification, every category is assumed to be known in advance, and a document may be assumed to belong to a single category.

heading: 3 Background

heading: 3.1 Problem Formulation
text: Consider a corpus with D documents, where each document d contains Nd words wd = {wd1, wd2, . . . , wdNd}, each belonging to a vocabulary of size V . And suppose that we have K topics, each provided with a set of Lk seed words denoted by Sk = {sk1, sk2, . . . , skLk}. Our goal is to derive topics from the corpus that are semantically coherent with corresponding seed word sets.

heading: 3.2 Generative Story and Variational Inference
text: Our model builds on the generative story in (Srivastava and Sutton, 2017), where the Dirichlet prior is approximated via a logistic normal distribution. The generative story is summarized as follows, where α is the parameter for prior distribution and βk denotes the word distribution for the k-th topic:
For document d, draw topic distribution θ ∼ LN (μ0(α), σ20(α)); For wdn in this document:
Draw topic zdn ∼ Cat(θ); Draw word wdn ∼ Cat(βzdn);
Based on the generative story, variational inference is used to approximate posterior distribution of latent variables θd and zd = {zd1, zd2, . . . , zdNd} to maximize the likelihood
of observed data. And the evidence lower bound (ELBO) can be derived as
L(w) =Eq(θ,z|w) log (p(w|θ, z;β))
− Eq(θ,z|w) log ( q(θ, z|w) p(θ, z) )
=− (Lrec + Lkl),
(1)
where q(θ, z|w) is the joint variational distribution.

heading: 4 Methodology
text: In this section, we introduce our proposed SeededNTM. We start by introducing the model architecture and the designs of multi-level pseudo supervisions. Then we focus on our proposed auto-adaptation mechanism based on contextdependency assumption and our noise-reduction consistency regularizer. Finally, we introduce our training objective and summarize the training procedure with Algorithm 1.

heading: 4.1 Model Architecture

heading: 4.1.1 Document Encoder
text: A multi-layer network is used as document encoder to infer the document-topic distributions θ for document d with a word set w. The words are first encoded into word embedding vectors Ed = {e1, e2, . . . , eNd} and then averaged to obtain the document embedding ed. Then the mean vector μ and the diagonal of the covariance matrix σ2 are further encoded with two sub-networks μ = fμ(ed) and σ2 = fσ(ed), and the document-topic distribution is sampled via the reparameterization trick with ∼ N (0, I) and θ = softmax(μ + σ · ). The above procedure is donoted as θ = Fd(d).

heading: 4.1.2 Word Encoder
text: Word encoder encodes words to local word-topic preferences φ. For a word wn, it is first encoded to the embedding vector en, followed by a feedforward network activated with a softmax function. The above procedure is donoted as φn = Fw(wn).

heading: 4.1.3 Topic Decoder
text: The decoder contains topic-word distribution and reconstructs documents with topic mixtures. Inspired by (Eisenstein et al., 2011), we disassemble topics in log-space into three parts, background m, regular topic ηr, and seed topic ηs. The background term is estimated with the overall log frequencies of words from the corpus, and both regular and
seed topics act as additional deviations on m. The possibility βkv for word wv in topic k is
βkv = exp(mv + η
r kv + η s kv)∑
v exp(mv + η r kv + η s kv)
, (2)
where ηrk is a V -dimensional parameter vector whose elements at positions corresponding to Sk are fixed to zero. And ηsk is defined as
ηskv = { κ, wv ∈ Sk, 0, otherwise, v ∈ {1, · · · , V }, (3)
where κ is a hyperparameter of seeding strength.

heading: 4.2 Multi-Level Supervisions

heading: 4.2.1 Document Level Supervision
text: With seed words, we can regularize the inferred document-topic distribution θ with the pseudo distribution θ̂ which is estimated via the tf-idf scores of seed words appearing in the document. Formally, for a document d, its corresponding θ̂ is
θ̂k = exp 1Lk ∑ s∈Sk tfidf(s, d)∑
k ( exp 1Lk ∑ s∈Sk tfidf(s, d) ) ,
k ∈ {1, . . . ,K}. (4)
And we regularize θ by minimizing the KL divergence between θ and θ̂,
Ld(θ, θ̂) = KL(θ̂‖θ) = ∑
k
θ̂k log( θ̂k θk ). (5)

heading: 4.2.2 Word Level Supervision
text: Local word-topic preferences φ can also be regularized by seed words. We estimate the pseudo word-topic distribution φ̂ with co-occurrence measured by the conditional possibility p(w|s) = df(w, s)/df(s) of word w and seed word s, where df(·) is the number of documents containing s or both s and w. And the pseudo possibility for word wn belonging to topic k is
φ̂nk =
τ Lk ∑ s∈Sk p(wn|s)∑
k ( τ Lk ∑ s∈Sk p(wn|s) ) , (6)
where τ is a temperature factor to sharpen the distribution. And we also use KL divergence to minimize the distance between φ̂n and φn,
Lw(φn, φ̂n) = KL(φ̂n‖φn) = ∑
k
φ̂nk log( φ̂nk φnk ).
(7)

heading: 4.3 Auto-Adaptation of Multi-Level Information
text: In previous work (Rezaee and Ferraro, 2020), the inferred posterior distribution q(θ, z|w) is decomposed with a mean-field assumption as
q(θ, z|w) = q(θ|w) ∏
n
q(zn|wn), (8)
but as we mentioned before, per-word topic preferences can be ambiguous without context document information. Therefore, instead of mean-field assumption, we introduce a context-dependency assumption by taking document topic distribution θ into consideration,
q(θ, z|w) = q(θ|w) ∏
n
q(zn|wn, θ). (9)
As zn is now conditioned on both wn and θ, how to properly balance information from word and document remains unsolved. Inspired by the idea of product of experts (Hinton, 2002), we propose an auto-adaptation mechanism to automatically combine local word-topic preference φn and the global document-topic preference θ and implement the combination as products of two distributions,
ϕnk = q(zn = k|θ, wn) = φnkθk∑ k(φnkθk) . (10)
In this way, we avoid manually weighting the global and local topic preferences and achieve autoadaptation between multi-level information. Potential ambiguities in per-word topic preferences get re-weighted by the global document-topic distributions, and topics with higher probabilities in both distributions are further encouraged.

heading: 4.4 Noise-Reduction Consistency Regularizer
text: Document level supervisions can be biased by seed words’ semantic diversity and ambiguity. To avoid time-consuming nearest neighbor method (Li et al., 2018), inspired by recent works in noisy label learning (Li et al., 2020; Englesson and Azizpour, 2021), we propose a consistency regularizer that encourages intra-sample consistency.
In this regularizer, we encourage outputs from the document encoder to be consistent with perturbed samples, θ′ = Fd(d′) = Fd(A(d)), where A is an data augmentation function. Each perturbed sample can be viewed as a neighbor with the original sample in feature space, and by encouraging perturbation consistency, we can preserve local structures without finding nearest neighbors.
Moreover, we encourage consistency with the outputs from the word encoder. The word encoder takes supervisions from the word-word cooccurrences and contains more fine-grained information than the document level. By encouraging consistency with the predictions of the word encoder on document embeddings, θ′′ = Fw(d), we incorporate semantic information from the word level to help correct the predictions from the document encoder and improve its robustness to noises.
We use the symmetric KL Divergence to measure the distance between two distributions, and our consistency regularizer is summarized as follows.
SKL(a, b) =KL(a‖b) +KL(b‖a), Lc(d) =SKL(θ, θ′) + SKL(θ, θ′′). (11)
Algorithm 1 The SeededNTM training procedure. Input: corpus D, topic number K, seed word sets S = {S1, S1, . . . , SK}, initial KL annealing factor λ0, hyperparameters λ1, λ2, λ3, max iteration number T . for t from 1 to T do
randomly sample a batch of B documents; Lbatch ← 0; λ0 ← min(λ0 + 1T , 1.0); compute βk for each topic k by Eq.3; for each document d in the batch do
compute θ with encoder Fd; compute φn for each wn with encoder Fw; compute ϕd = {ϕ1, ..., ϕn} by Eq.10; Lbatch ← Lbatch + Ltr by Eq.13
end for update model parameters with ∇Lbatch
end for

heading: 4.5 Training Objectives
text: With the new assumption in Eq.9, Lrec and Lkl in Eq.1 can be further derived as
Lrec =− ∑
n,k
ϕnk log βkwn ,
Lkl =KL ( N (μ, σ2)‖N (μ0, σ20) )
+ ∑
n
KL (ϕn‖θ) . (12)
Detailed derivations can be found in Appendix A. Our final training objectives is
Ltr = Lrec + λ0Lkl + λ1Ld + λ2Lw + λ3Lc, (13) where λ0 is KL annealing factor and gradually increases to 1 during training and λ1, λ2, λ3 are hyperparameters. The overall structure of SeededNTM is shown in Figure 2, and the training procedure is described in Algorithm 1.

heading: 5 Experiments

heading: 5.1 Datasets
text: We conduct our experiments on three datasets: 20 Newsgroups, UIUC Yahoo Answers, and DBPedia. 20 Newsgroups (Lang, 1995) is a dataset that contains around 20,000 newsgroup documents and is commonly used in the topic modeling field. And to verify our model’s scalability, we adopt two other larger datasets, the UIUC Yahoo Answers dataset (Chang et al., 2008) and DBPedia (Zhang
et al., 2015), which contain 150,000 and 630,000 samples, respectively. We preprocess each dataset and split them for training and testing. The detailed procedure of preprocessing and the statistical summaries for each dataset can be viewed in Appendix B.

heading: 5.2 Seed Words Extraction
text: To avoid human biases, we follow (Jagarlamudi et al., 2012; Gallagher et al., 2017) and adopt an automatic approach to extract seed words. For each dataset, we set the topic number K the same as its class number, and use Information Gain (IG) to identify the words having the highest mutual information with the class. Specifically, IG of a word w in class c is
IG(w, c) = H(c)−H(c|w), (14)
where H(c) is the entropy of class c and H(c|w) denotes the conditional entropy of c given w. For each class, we choose the top L words with the highest IG scores as seed words.

heading: 5.3 Evaluation of Topic Coherence
text: Evaluation Metrics. We use Normalized Pointwise Mutual Information (NPMI), to evaluate the coherence of learned topics. NPMI between words wi and wj is defined as:
NPMI(wi, wj) = log
p(wi,wj) p(wi)p(wj)
− log p(wi, wj) . (15)
As we are dealing with topic models with seed words, we take the top N non-seed words and predefined L seed words for each topic and measure NPMI among the N + L words. For unsupervised methods, we pick the top N + L words. By considering both seed and non-seed words, the NPMI scores can measure how well the learned topics fit the predefined aspects of interests. Also, the score implicitly reflects topic diversity, as topics with a high coherence score with seed words are more likely to be diverse as long as their seed words are distinct. We report NPMI with N = 10, L = 5 on both train and test sets.
Baselines. We compare SeededNTM with the following baselines. For unsupervised topic models, we compare with LDA (Blei et al., 2003), which is a representative conventional neural topic models, and CombinedTM (Bianchi et al., 2021a), which enhances prodLDA (Srivastava and Sutton,
2017) with contextualized embeddings from BERT. For seed-guided topic models, we compare with SeededLDA (Jagarlamudi et al., 2012), STM (Li et al., 2016), Anchored CorEx (Gallagher et al., 2017), CatE (Meng et al., 2020a), keyATM (Eshima et al., 2020) and keyETM (Harandizadeh et al., 2022), which we have introduced in related works.
Results. The performances of topic coherence are reported in Table 1. As we can see, most seeded topic models achieve better topic coherence than unsupervised ones as the seed words provide additional semantic information. SeededNTM outperforms the baselines in most settings, demonstrating the effectiveness of our approach. Note that the advantages become more significant on the largest datasets, DBPedia, indicating its scalability when facing datasets of huge scale. We can find that keyETM sometimes performs worse performances than conventional methods like STM and keyATM, indicating the necessity to incorporate document level information. Anchor CorEx and CatE are strong baselines on some occasions, as Anchor CorEx has an information-theory-based objective similar to NPMI, and CatE takes the order
of words as additional information when learning embeddings.

heading: 5.4 Evaluation of Classification
text: Evaluation Metrics. Except for evaluating coherence of learned topics, we evaluate how well the document-topic distribution is learned with a classification task. Here we take the maximum probability in the document topic distribution as the predicted label to test topic models’ ability to extract semantic information from documents. We use Macro and Micro F1 scores as the evaluation metrics. As most baselines cannot predict on new data, we report the results on the train set and take the test set for validation.
Baselines. We compare SeededNTM on classification with the aforementioned baselines except for the unsupervised ones. Specifically, we follow CatE’s original paper and use WeSTClass model (Meng et al., 2018) to classify its outputs.
Results. Table 1 summarizes the F1 scores on three datasets. SeededNTM outperforms other baseline models on most occasions, indicating our model can understand the semantics of the documents and learn more reliable and helpful topic distributions for each document. Among the baselines methods, seededNTM, STM, and keyATM achieve better performances on three datasets, as they incorporate information from seed words on both levels.

heading: 5.5 Ablation Studies
text: We analyze the effects of different modules of SeededNTM by comparing among the following variants: 1) SeededNTM-noise: SeededNTM without the consistency regularizer, 2) SeededNTMNN: SeededNTM without the consistency regu-
larizer and with a neighbor-based noise-reduction method as in (Li et al., 2018). 3) SeededNTMw.o.doc: SeededNTM without document encoder, 4) SeededNTM-w.o.word: SeededNTM without word encoder, 5) SeededNTM-mean: SeededNTM with the mean-field assumption as in (Rezaee and Ferraro, 2020).
Results are provided in Table 2, from which we can draw the following conclusions. The effectiveness of the noise-reduction method can be proved by the comparisons between variants with and without noise regularizer. Both SeededNTM-NN and original SeededNTM outperform SeededNTMnoise. And the effectiveness of our intra-sample consistency regularizer can be further demonstrated by the improvements of SeededNTM over SeededNTM-NN. The decreases in SeededNTMw.o.doc and SeededNTM-w.o.word indicate the importance of information on both document and word levels. Moreover, the decay on SeededNTMmean proves the effectiveness of our proposed assumption and the necessity to balance context document information when modeling per-word topic assignments.

heading: 5.6 Human Evaluation
text: Apart from automated evaluation metrics, we hope to further demonstrate our model’s ability to discover semantically meaningful topics through the judgements from human, as automated metrics can be sometimes biased (Hoyle et al., 2021).
Metrics: We adopt two human evaluation metrics: accuracy in the word intrusion task (Chang et al., 2009) and MACC score (Meng et al., 2020a). In the word intrusion task, participants are given the top words of a certain topic and an intruding
word from another topic, and are asked to identify the intruding term. For MACC, participants need to classify whether the top words are consistent with their corresponding seed word set, i.e.
MACC = 1
K
K∑
k=1
1
N
N∑
i=1
(wki ∈ Sk), (16)
where (wki ∈ Sk) indicates whether word wki belongs to the topic with seed word set Sk. We conduct human evaluation on UIUC Yahoo Answer dataset, and take the top 5 words from each topic for evaluation. For each metric, we invite 10 graduate students to independently fulfil the task, and the participants in two groups do not overlap to avoid information leak. More details can be viewed in Appendix C. The results are shown in Table 3. We can find that SeededNTM achieves best results on both metrics, which further demonstrates the quality of our learned topics.
Case Study: Here we present part of topics learned by SeededNTM on Yahoo Answer dataset along with topics learned by baselines methods using the same seed words in the aforementioned experiments in Table 4. We can find that some baselines like Anchor CorEx, keyATM, and KeyETM, tend to put high weights on several commonly used words like ’play’, ’great’, ’good’, while SeededNTM tends to pay attention to words that are more specific such as ’nintendo’, a Japanese multinational video game company who releases the game ’Pokemon’, and ’rowling’, the author of Harry Potter, and ’ttc’, meaning ’trying to conceive’.
Besides presentations of topics, we conduct more other qualitative experiments under different settings to verify the generalization ability of
our model. Please refer to Appendix C for more information.

heading: 6 Conclusions
text: In this paper, we propose SeededNTM to improve topic interpretability together with scalability. We leverage supervisions from seed words on both word and document levels and propose a contextdependency assumption. An auto-adaptation mechanism is designed to balance word and context document information. Moreover, we propose an intra-sample consistency regularizer to deal with noisy document level supervisions. Perturbation consistency and semantic consistency are encouraged to improve the model’s robustness to noises. Through quantitative and qualitative experiments on three datasets, we demonstrate that SeededNTM can derive semantically meaningful topics and outperforms state-of-the-art baselines.
Limitations
Our model improves topic interpretability of NTM with seed words, but we believe there are still limitations to be explored in the future works. For the methodology part, large-scale pre-trained language models could be considered to provide more context information when incorporating seed words. For the experiment part, only single label dataset are used for extracting seed words, and more explorations on multi-label datasets should be conducted.

heading: Acknowledgments
text: This work is supported by the National Natural Science Foundation of China (No.62172011) and the Fundamental Research Funds for the Central Universities.

heading: A Derivation of ELBO-based Loss
text: The Evidence Lower Bound (ELBO) for our model is
ELBO(w) =Eq(θ,z|w) log p(w|θ, z;β)
− Eq(θ,z|w) log ( q(θ, z|w) p(θ, z) ) .
(A.1) To maxmize the ELBO, we minimize its oppo-
site number as training loss, which is
Lelbo =− Eq(θ,z|w) log p(w|θ, z;β)
+ Eq(θ,z|w) log ( q(θ, z|w) p(θ, z) ) . (A.2)
And we denote
Lrec = −Eq(θ,z|w) log p(w|θ, z;β),
Lkl = Eq(θ,z|w) log ( q(θ, z|w) p(θ, z) ) ,
Lelbo = Lrec + Lkl.
(A.3)
For the posterior q(θ, z|w), we have
q(θ, z|w) = q(θ|w) ∏
n
q(zn|θ, wn). (A.4)
For p(w|θ, z;β), we have
p(w|θ, z;β) = ∏
n
p(wn|zn;β). (A.5)
So for Lrec we have
Lrec = −Eq(θ,z|w) log p(w|θ, z;β) = −Eq(θ|w) ∑
n
Eq(zn|θ,wn) log p(wn|zn;β).
(A.6) The expectation Eq(θ|w) can be estimated using a sample-based method by sampling θ ∼ q(θ|w), and given θ, ϕnk = q(zn = k|θ, wn) can be computed with Eq.10. So we have
Lrec ≈ − ∑
n,k
ϕnk log βkwn . (A.7)
For Lkl we have
Lkl = Eq(θ,z|w) log ( q(θ, z|w) p(θ, z) )
= Eq(θ|w) log ( q(θ|w) p(θ) ) + Eq(θ|w) ∑
n
Eq(zn|θ,wn) log ( q(zn|θ, wn) p(zn|θ) )
= KL (q(θ|w)‖p(θ)) + Eq(θ|w) ∑
n
KL (q(zn|θ, wn)‖p(zn|θ)) .
(A.8) The former term can be approximated using Laplace approximation to the Dirichlet prior, and can be calculated in closed form as KL ( N (μ, σ2)‖N (μ0, σ20) ) (Srivastava and Sutton, 2017). And the latter term can be estimated by Monte Carlo sampling with θ ∼ q(θ|w):
Eq(θ|w) ∑
n
KL (q(zn|θ, wn)‖p(zn|θ))
≈ ∑
n
KL(ϕn‖θ). (A.9)

heading: B More Details of Datasets
text: B.1 Dataset Descriptions
Three datasets are used in out experiments: 20 Newsgroups, UIUC Yahoo Answers, and DBPedia. 20 Newsgroups (Lang, 1995) is a collection of newsgroup documents containing 11,000 train samples and 7,000 test samples in 20 classes. It is a common dataset that is widely used in topic modeling field. The UIUC Yahoo Answers dataset (Chang et al., 2008) contains 150,000 question-answer pairs belonging to 15 categories. It has been used in topic models in (Card et al., 2018). DBPedia (Zhang et al., 2015) is extracted from Wikipedia and contains 560,000 train samples and 70,000 test samples belonging to 14 ontology classes. Similar datasets (though much smaller) from Wikipedia have been adopted to test topic models in (Nguyen and Luu, 2021).
B.2 Preprocess Procedures for Datasets
We preprocess documents in each dataset by tokenizing, filtering out stop words, words whose document frequency above 70%, and words appearing in less than around 100 documents (depending on the dataset). The final vocabulary sizes for each dataset after preprocessing vary from 2,000
to 20,000. Then we remove the documents shorter than two words.
Specifically, for the UIUC Yahoo Answer dataset, we follow the approach used in (Card et al., 2018), and drop the Cars and Transportation and Social Science classes and merge Arts and Arts and Humanities into one class, producing 15 categories, each with 10,000 documents.
As for the augmentation functions A, we use the word level augmentation method proposed in (Xie et al., 2020) by randomly replacing words with lower tf-idf scores. Around of 10% words are replaced in our experiments.
B.3 Statistics of Datasets We summarize the statistics for the three datasets after preprocessing in Table.B.1

heading: C More Experimental Details
text: C.1 Implementation Datails As for the training environment, we implement our method based on PyTorch 1.6.0 with Python 3.7.9 and perform our experiments on 4 GeForce RTX 2080Ti. For model structure, the dimension for our word embedding layer is 300, and the dimension for the hidden layer in the document encoder is 256. We use a 0.2 dropout rate in our encoder during training. We present our choices for hyperparameters in Table.C.1. Hyperparameters are determined by grid search on the smallest dataset, 20 Newsgroups, and fine-tuned on other two large datasets. The final hyperparameters are shown in Table C.1.
C.2 Baselines We give detailed descriptions of our baselines here.
• LDA (Blei et al., 2003): LDA is one of the most popular unsupervised conventional topic models that deduce posterior distribution via Gibbs sampling or variational inference.
• CombinedTM: CombinedTM enhances topic model with contextualized embeddings from
pretrained language model to improve model’s semantic expression ability and leads to more coherent topics. CombinedTM is an extension of prodLDA (Srivastava and Sutton, 2017), one of the most representative neural topic models.
• SeededLDA (Jagarlamudi et al., 2012): SeededLDA pairs each regular topic with a topic containing only seed words and biases documents’ topic preferences in Gibbs sampling if they contain seed words.
• STM (Li et al., 2016): STM is a topic modelbased weakly-supervised text classification method that incorporates both document and word level supervisions to improve classification accuracies.
• Anchored CorEx (Gallagher et al., 2017): Anchored CorEx is based on an informationtheoretic framework and tries to derive maximally informative topics based on seed words.
• CatE (Meng et al., 2020a): CatE aims at deriving topics with a single seed word for each topic. It uses a word embedding method and tries to learn a discriminative embedding space for both topics and words.
• keyATM (Eshima et al., 2020): keyATM improves upon SeededLDA by allowing some seed-word-free topics.
• keyETM (Harandizadeh et al., 2022): keyETM incorporates seed words into NTM by regularizing word-topic and topic-word distributions on word level with seed words and pre-trained word embeddings.
C.3 Running Time for Topic Models with Seed Words
In Table C.4 we report the running time of each topic model that is supervised by seed words.
C.4 Comparisons with Weakly-Supervised Classification Methods
We compare SeededNTM with two recent weaklysupervised classification methods, XClass (Wang et al., 2021) and ClassKG (Zhang et al., 2021). Both models take seed words or class names as label information and bootstrap seed word lists in some ranked orders, which could be viewed as
topics. For fair comparison, we replace the BERTbased document classifier in both methods with a multi-layer MLP classifier same as SeededNTM. The results are shown in Table C.3. We can find that SeededNTM outperforms both methods on NPMI and F1 score, which may indicates that the unsupervised losses in NTM might help the model make better use of the unlabeled or noisy-labeled documents.
C.5 Class Names as Seed Words Besides seed words extracted automatically with information gain, we also take the class names (one or two words for each class) as seed words and run the experiments on Yahoo Answer dataset. Results are shown in Table C.5. We can draw similar conclusions from these results as in the main paper.
C.6 Details of Human Evaluation To perform human evaluations on topic quality, we invite 20 graduate students majoring in computer science, who participate the experiments for course credits. The age of the participants ranges from 21 to 24 years old. We divide the participants into two non-overlapping groups for word intrusion and MACC respectively to avoid information leak. All participants are informed that they are performing evaluations for an automatic method and none of their privacy information are collected during the experiments. The surveys used in human evaluation experiments are shown in Figure C.1.
C.7 More Qualitative Results Here we present some more qualitative results under different settings which are not included in the main paper.
C.7.1 Topic with Incomplete Seed Words In the experiments of the main paper, seed words are assumed to be complete and accurately represent latent topics in the corpus. However, in practical situations, users may only be interested in part of the corpus or have little prior knowledge, leading to incomplete seed words. To simulate such situations, we preserve seed words for only three topics and leave other topics unsupervised. We present the results of SeededNTM along with the two latest baselines, keyATM and keyETM in Table C.6.
For three supervised topics, SeededNTM can discover words related to the seed words as it does under complete seed words, while keyATM and keyETM produce semantically incoherent topics, such as irrelevant words "god" and "world" appearing in the topic ’Education&Reference’ from keyETM. SeededNTM can also discover meaningful unsupervised topics similar to the seeded topics in former experiments, such as ’Pets’ and ’Computer&Internet’, while keyATM and keyETM find incoherent or unrelated topics. Moreover, new topics which are not included in the original seed word sets can also be discovered by SeededNTM, such as ’Craigslist’, a famous American classified advertisements website.
C.7.2 Noisy Seed Words The seed word set may contain irrelevant words in real-world practice due to users’ mistakes or unfamiliarity with the corpus. To simulate such situations, we manually intrude irrelevant words from other topics into the seed words. The results are shown in Table C.7, from which SeededNTM can still find meaningful topics when there are noisy intrusions in the seed words, while keyATM and keyETM provide topics that are less explicit and coherent.

heading: D Potential Risks
text: As for potential risks of our model, seeded topic models can be used to trace a specific topic, so
it is possible that it’s used to track someone’s information from texts collected from the internet, violating personal privacy.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Limiatation section
3 A2. Did you discuss any potential risks of your work? Appendix Section D
3 A3. Do the abstract and introduction summarize the paper’s main claims? Abstract and Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Left blank.
3 B1. Did you cite the creators of artifacts you used? Section 5.1
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Not applicable. Left blank.
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Appendix Section B.1
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Appendix Section B.1
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Appendix Section B.2 and B.3
C 3 Did you run computational experiments? Left blank.
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Appendix Section C.1 and C.3
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Appendix Section C.1
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Table 1 in Section 5, Appendix Table C.2
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Appendix Section B.2
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Appendix Section C.6
3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Appendix Section C.6
3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Appendix Section C.6
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
3 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Appendix Section C.6

heading: Future Work
text: Limitations
Our model improves topic interpretability of NTM with seed words, but we believe there are still limitations to be explored in the future works. For the methodology part, large-scale pre-trained language models could be considered to provide more context information when incorporating seed words. For the experiment part, only single label dataset are used for extracting seed words, and more explorations on multi-label datasets should be conducted