Modeling Dynamic Topics in Chain-Free Fashion by Evolution-Tracking Contrastive Learning and Unassociated Word Exclusion (2024)

Abstract: Dynamic topic models track the evolution of topics in sequential documents, which have derived various applications like trend analysis and opinion mining. However, existing models suffer from repetitive topic and unassociated topic issues, failing to reveal the evolution and hindering further applications. To address these issues, we break the tradition of simply chaining topics in existing work and propose a novel neural Chain-Free Dynamic Topic Model. We introduce a new evolution-tracking contrastive learning method that builds the similarity relations among dynamic topics. This not only tracks topic evolution but also maintains topic diversity, mitigating the repetitive topic issue. To avoid unassociated topics, we further present an unassociated word exclusion method that consistently excludes unassociated words from discovered topics. Extensive experiments demonstrate our model significantly outperforms state-of-the-art baselines, tracking topic evolution with high-quality topics, showing better performance on downstream tasks, and remaining robust to the hyperparameter for evolution intensities. Our code is available at https://github.com/bobxwu/CFDTM.

heading: 1 Introduction
text: Dynamic topic models seek to discover the evolution of latent topics in sequential documents divided by time slice. For example, Figure 1 illustrates Topic#1 about Covid-19 and Topic#20 about Ukraine evolve from 2020 to 2022 in the documents divided by year. This evolution reveals how topics emerge, grow, and decline due to trends and events. Such evolution has been employed in various downstream applications, e.g., opinion mining, trend tracking, and sentiment analysis (Wu and Li, 2019; Sha et al., 2020; Li et al., 2020; Hu et al., 2015; Greene and Cross, 2017; Li et al., 2021; Murakami et al., 2021; Churchill and Singh, 2022).
Existing dynamic topic models can be classified into two types: (i) probabilistic dynamic topic models (Blei and Lafferty, 2006; Wang et al., 2008), learning through Variational Inference (Blei et al., 2017) or Gibbs sampling (Griffiths and Steyvers, 2004), and (ii) neural dynamic topic models (Dieng et al., 2019; Zhang and Lauw, 2022; Miyamoto
3088
et al., 2023; Wu et al., 2024a), learning via gradient back-propagation.
However, these existing models mostly chain topics via Markov chains to capture topic evolution, suffering from two vital issues: (i) Repetitive Topics: topics within a time slice are repetitive with similar semantics. Table 1 exemplifies that Topic#21, #30, and #40 in the year 2015 all include repeating words like “president” and “trump”. As illustrated in Figure 2, simply chaining topics via Markov chains pushes topics to gather together and cannot separate them. These topics are less distinguishable and fail to uncover the complete semantics of their time slices. (ii) Unassociated Topics: topics are unassociated with their corresponding time slices. Table 1 shows that Topic#2 in 2017 and 2018 refer to Covid-19 with words “covid” and “coronavirus”, but Covid-19 does not outbreak in 2017 or 2018. This is because simply chaining topics via Markov chains may force topics across time slices to be overly related. These topics become less associated with their time slices, and the real topics in these slices are unrevealed. Consequently, these two issues bring about low-quality dynamic topics and hinder tracking topic evolution, which thus impairs downstream applications.
To solve the above issues, we break the tradition of chaining topics via Markov chains and propose a novel neural Chain-Free Dynamic Topic Model (CFDTM). First, we propose Evolution-Tracking Contrastive learning (ETC) to address the repetitive topic issue. ETC adaptively builds positive and negative relations among dynamic topics. Building positive relations tracks the topic evolution with different intensities. More importantly, building negative relations encourages topics within each slice to be distinct, maintaining topic diversity and thus alleviating the repetitive topic issue.
Second, we propose a new Unassociated Word Exclusion (UWE) method to solve the unassociated topic issue. UWE finds the top related words of topics at each time slice and identifies which of them do not belong to this slice as unassociated words. Then UWE explicitly excludes these unassociated words from topics to refine topic semantics, such as excluding the unassociated word “covid” from Topic#2 in 2017 in Table 1. This mitigates the unassociated topic issue and also enhances the robustness of our model to the hyperparameter for evolution intensities. Figure 2b visualizes the effectiveness of our CFDTM. Our contributions can be concluded as follows:
• To our best knowledge, we are the first to propose and investigate both the repetitive and unassociated topic issues in dynamic topic modeling.
• We propose a novel chain-free dynamic topic model with a new evolution-tracking contrastive learning method that tracks topic evolution and avoids producing repetitive topics.
• We further propose a new unassociated word exclusion method that excludes unassociated words from topics, which effectively alleviates the unassociated topic issue.
• We conduct extensive experiments on benchmark datasets and show our model consistently outperforms baselines, capturing topic evolution with high-quality topics and achieving higher downstream performance, with robustness to the hyperparameter for evolution intensities.

heading: 2 Related Work
text: Probabilistic Dynamic Topic Modeling Topic modeling aims to understand documents in unsupervised fashion with latent topics, deriving various text analysis (Nguyen et al., 2023; Liu et al., 2023; Mao et al., 2023, 2024a,b). Blei and Lafferty (2006) first propose Dynamic Topic Model (DTM) based on LDA (Blei et al., 2003). DTM adopts state space models to chain the natural parameters of latent topics with Gaussian noise and uses Kalman filter and wavelet regression as variational approximations. Afterward, Wang et al. (2008) introduce DTM under continuous time settings; Caron et al. (2012) extend DTM to nonparametric settings. Several other extensions are also proposed (Wang and
McCallum, 2006; Iwata et al., 2010; Bhadury et al., 2016; Jähnichen et al., 2018; Hida et al., 2018). They use Variational Inference or Gibbs sampling to optimize model parameters.
Neural Dynamic Topic Modeling Due to the success of neural topic modeling (Miao et al., 2016; Srivastava and Sutton, 2017; Wu et al., 2020a,b, 2021, 2024c,b), neural dynamic topic modeling has attracted more attention (Balepur et al., 2023). Dieng et al. (2019) first propose DETM in the framework of VAE (Kingma and Welling, 2014; Rezende et al., 2014). Later, Zhang and Lauw (2022) capture evolution from temporal document networks; Cvejoski et al. (2023) focus on modeling topic activities over time; Rahimi et al. (2023) cluster documents to find dynamic topics but cannot infer their topic distributions. Miyamoto et al. (2023) model the dependencies among topics with an attention mechanism. They all chain topics via Markov chains following probabilistic models. Differently instead of Markov chains, we propose the novel evolution-tracking contrastive learning and unassociated word exclusion methods to track topic evolution and address the repetitive topic and unassociated topic issues.
Contrastive Learning The goal of contrastive learning is to learn the similarity relations among samples (Hadsell et al., 2006; Oh Song et al., 2016; Van den Oord et al., 2018; Frosst et al., 2019; He et al., 2020; Nguyen et al., 2022, 2024a). It has become a prevalent self-supervised fashion in vision and textual fields (Chen et al., 2020; Xie et al., 2021; Gao et al., 2021; Zhao et al., 2021). Several studies apply contrastive learning in static topic models (Wu et al., 2022, 2023a; Nguyen and Luu, 2021; Nguyen et al., 2024b; Zhou et al., 2023; Wang et al., 2023; Han et al., 2023). Rather than these static topic models, we focus on dynamic topic models, which motivates our new evolutiontracking contrastive learning.

heading: 3 Methodology
text: We recall the problem setting of dynamic topic modeling and present our novel evolution-tracking contrastive learning and unassociated word exclusion. Finally we introduce the Chain-Free Dynamic Topic Model (CFDTM) on these two methods.

heading: 3.1 Problem Setting and Notations
text: Following DTM (Blei and Lafferty, 2006), we introduce the problem setting of dynamic topic mod-
Genera on of Sequen al Documents
Evolu on-Tracking Contras ve Learning
Embedding Space
k by adjustable intensity hyperparameter λ
(t), and
pushes it away from φ(t)k′ (k
′ ̸=k), for instance k′ = k−1 and k + 1 here.
eling. As shown in Figure 1, consider sequential document collections divided by T time slices, for example by year. Slice t has Nt documents with x(t,d) as the d-th document at slice t. The entire vocabulary set is V , and the vocabulary set of slice t is V(t). We aim to discover K latent topics at each slice, and topics at slice t evolve from topics at slice t−1, e.g., Topic#1 in 2022 evolves from Topic#1 in 2021 in Figure 1. We generally preset the evolution intensity through hyperparameters: we use strong intensity if documents evolve dramatically and weak otherwise, like the Gaussian variance in DTM (Blei and Lafferty, 2006). Following LDA (Blei et al., 2003), each topic is defined as a distribution over all words (topic-word distribution): Topic#k at slice t is defined as β(t)k ∈ R|V|. Then β(t)=(β
(t) 1 , . . . ,β (t) K )∈R|V|×K is the topic-word
distribution matrix of slice t. The same topic at different slices reveals the evolution. We also infer the topic distribution of document x(t,d) (doc-topic distribution), denoted as θ(t,d)∈∆K , where ∆K is a probability simplex.

heading: 3.2 Parameterizing Topics as Embeddings
text: We begin by parameterizing topics. Following Miao et al. (2017); Dieng et al. (2020), we project words in the vocabulary into a D-dimensional embedding space as |V| word embeddings: W = (w1, . . . ,w|V|)∈RD×|V|. We reuse word embeddings for different slices for parameter efficiency. Similarly, we project topics at each time slice into the same space: slice t has K topic embeddings φ(t)=(φ
(t) 1 , . . . ,φ (t) K )∈RD×K , and φ (t) k denotes
the topic embedding of Topic#k at slice t. Each topic (word) embedding represents the semantics of the topic (word). Following Wu et al. (2023b), we formulate β
(t), topic-word distribution matrix at slice t, as
β (t) k,i = exp(−∥φ(t)k −wi∥2/π)∑K k′=1 exp(−∥φ (t) k′ −wi∥2/π) . (1)
Here β(t)k,i models the correlation between word i and Topic#k at slice t with a scale hyperparameter π, calculated as the Euclidean distance between their embeddings with normalization along all topics. As such, a word relates to a topic if their embeddings are close and away from other topic embeddings, which cooperates with our next evolution-tracking contrastive learning.

heading: 3.3 Evolution-Tracking Contrastive Learning
text: To track topic evolution, we abandon traditionally chaining topics via Markov chains and propose the novel Evolution-Tracking Contrastive learning (ETC) to avoid repetitive topics. Figure 3 illustrates our ETC method.
Positive Relations among Dynamic Topics We first build positive relations among dynamic topics from a contrastive learning perspective to track their evolution. Recall that topics at slice t evolve from topics at slice t−1 via different intensities (Blei and Lafferty, 2006). Hence we model their topic embeddings (φ(t)k ,φ (t−1) k ) as positive pairs and build their positive relations with evolution intensity hyperparameter λ(t):
Lpos = T∑
t=2
K∑
k=1
−λ(t)g(φ(t)k ,φ (t−1) k ) (2)
where a positive pair is the embeddings of Topic#k at slice t and Topic#k at slice t−1. Here following InfoNCE (Van den Oord et al., 2018), g(·, ·) measures the similarity between two embeddings,
modeled as a scaled cosine function (Wu et al., 2018): g(a, b)=cos(a, b)/τ with τ as a temperature hyperparameter (Wu et al., 2022). Eq. (2) pulls these topic embeddings (positive pairs) close to each other in the semantic space. This encourages these topics to cover related semantics, so we can track their evolution with the later topic modeling objective in Sec. 3.5.
Following the common practice (Blei and Lafferty, 2006; Dieng et al., 2019; Miyamoto et al., 2023), we use hyperparameter λ(t) to adaptively adjust evolution intensities between time slices. If topics evolve slightly between slice t−1 and t, we use a large λ(t); otherwise a small λ(t) if they evolve dramatically (e.g., new events emerge). We later demonstrate the strong robustness of our model to this hyperparameter in Sec. 4.3.
Negative Relations among Dynamic Topics The above positive relations only track topic evolution, but cannot prevent topics within a slice from being similar, which leads to repetitive topics as shown in Figure 2 (See empirical results in Sec. 4.2 and 4.4). To address this issue, we further build negative relations among dynamic topics. Specifically, we require a topic to be different from others at each time slice to avoid repetitive topics. Similar to Eq. (2), we model the embeddings of different topics at a time slice as negative pairs and build their negative relations as
Lneg=γ T∑
t=1
K∑
k=1
log ∑
k′ ̸=k exp(g(φ
(t) k ,φ (t) k′ )) (3)
where γ is a weight hyperparameter. Eq. (3) pushes these topic embeddings (negative pairs) away from each other in the semantic space, forcing them to cover distinct semantics. Hence topics at each slice become different from each other, enhancing topic diversity and alleviating the repetitive topic issue.
Objective for Evolution-Tracking Contrastive Learning Combining the objectives for positive and negative relations (Eq. (2) and (3)), we formulate the objective for Evolution-Tracking Contrastive learning (ETC) as
LETC = Lpos + Lneg. (4)
As illustrated in Figure 2, this objective builds positive and negative relations among dynamic topics, which captures topic evolution and mitigates the repetitive topic issue as well.

heading: 3.4 Unassociated Word Exclusion
text: To address the unassociated topic issue, we further break the tradition of simply chaining topics and propose the Unassociated Word Exclusion (UWE) as illustrated in Figure 4.
What Causes Unassociated Topics? As aforementioned, previous dynamic topic models produce unassociated topics (See examples in Table 1). We believe the reason is that they only chain topics across time slices through Markov chains to track topic evolution (Blei and Lafferty, 2006; Dieng et al., 2019; Miyamoto et al., 2023). This pushes the chained topics at different slices to include similar words. In consequence, topics may be overly related, where a topic could be polluted by the words that do not belong to its slice but to other slices. Thus these words are unassociated with its slice, which incurs the unassociated topic issue.
To solve this issue, one may wonder if we can simply weaken the actual required evolution intensity between chained topics. Unfortunately, this is challenging and may fail to track topic evolution, missing the goals of dynamic topic modeling.
Excluding Unassociated Words To this end, we propose the Unassociated Word Exclusion (UWE) method, which refines topic semantics by excluding unassociated words. We first sample unassociated words from the top related words of topics. To be specific, we denote V(t)top as the set of top words of all topics at time slice t, which is sampled as
V(t)top = K⋃
k=1
topK(β (t) k , Ntop). (5)
Here we use topK function to sample Ntop top words from discovered topics according to their topic-word distribution β(t)k . Then V (t) top is the union of the selected top words from all topics. Later as illustrated in Figure 4, we formulate the set of unassociated words V(t)UW as
V(t)UW = V(t)top \ V(t), (6)
where we identify the top words that do not exist in V(t), the vocabulary set of slice t. These words are unassociated since they do not belong to slice t, such as the “covid” of Topic#2 in 2017 in Table 1.
Then we aim to exclude the unassociated words from discovered topics. For this purpose, we might directly mask these words in each topic, but unfortunately this simple way cannot fully improve
dynamic topic quality as it leaves topic semantics unrefined (See ablation studies in Sec. 4.4). Thus to refine topic semantics, we propose to model topic embeddings and the embeddings of unassociated words as negative pairs: (φ(t)k ,wid(x)) where x ∈ V(t)UW and id(x) returns the index of word x. Similar to Eq. (3), we formulate the objective for UWE with negative pairs as
LUWE= T∑
t=1
K∑
k=1
log ∑
x∈V(t)UWE
exp(g(φ (t) k ,wid(x))). (7)
It refines topic semantics by pushing topic embeddings away from the embeddings of unassociated words. This excludes unassociated words from topics and hence mitigates the unassociated topic issue. This also keeps topics associated to their time slices even if they are overly related by large evolution intensity hyperparameter λ(t), which increases the robustness of our model to the hyperparameter λ(t) (See experiment supports in Sec. 4.3).

heading: 3.5 Chain-Free Dynamic Topic Model
text: In this section, we combine the above ETC and UWE methods with the generation of sequential documents to formulate our CFDTM.
Generation of Sequential Documents As illustrated in Figure 3, our generation process follows VAE as Dieng et al. (2019). Specifically for document x(t,d), we use a latent variable r(t,d) following a logistic normal prior: p(r(t,d)) = LN (µ0,Σ0) where µ0 and Σ0 are the mean the diagonal covariance matrix. We model its variational distribution as qΘ(r(t,d)|x(t,d)) = N (µ(t,d),Σ(t,d)). To model parameters µ(t,d),Σ(t,d), we adopt a neural network encoder fΘ parameterized by Θ with the Bag-
of-Words of x(t,d) as inputs. Here we reuse this encoder for documents at different time slices for parameter efficiency as Dieng et al. (2019). Through the reparameterization trick of VAE (Kingma and Welling, 2014), we sample r(t,d) as
r(t,d)=µ(t,d)+(Σ(t,d))1/2ϵ, ϵ∼N (0, I). (8)
We model the doc-topic distribution via a softmax function as θ(t,d)=softmax(r(t,d)). Then we generate words in x(t,d) by sampling from a multinomial distribution: x ∼ Mult(softmax(β(t)θ(t,d))) following Srivastava and Sutton (2017). See more details about the generation process in Appendix B.
Topic Modeling Objective Based on the above generation process of sequential documents, we write the topic modeling objective following the ELBO of VAE (Dieng et al., 2019) as
LTM= T∑
t=1
Nt∑
d=1
−(x(t,d))⊤ log(softmax(β(t)θ(t,d)))
+ KL [ qΘ(r (t,d)|x(t,d))∥p(r(t,d)) ]
(9)
where we model β(t) as Eq. (1). The first term is the reconstruction error between the input and reconstructed document with β(t) in Eq. (1). The second term is the KL divergence between the prior and variational distributions.
Overall Objective for CFDTM We formulate the overall objective function for our CFDTM by combining Eq. (4), (7) and (9) as
min Θ,W,{φ(t)}Tt=1
LTM + LETC + λUWELUWE (10)
where λUWE is a weight hyperparameter of LUWE. Note the weight parameters of LETC are included in Eq. (4). In sum, LTM learns doc-topic distributions and latent topics from sequential documents; LETC
tracks topic evolution and avoids repetitive topics by building similarity relations among dynamic topics; LUWE mitigates the unassociated topic issue by excluding unassociated words from topics.

heading: 4 Experiment
text: In this section, we conduct comprehensive experiments and show that our model achieves stateof-the-art performance and effectively solves the repetitive topic and unassociated topic issues.

heading: 4.1 Experiment Setup
text: Datasets We experiment with the following benchmark datasets: (i) NeurIPS 1 contains papers published between 1987 and 2017 at the NeurIPS conference. (ii) ACL (Bird et al., 2008) is an article collection between 1973 and 2006 from ACL Anthology. (iii) UN (Baturo et al., 2017) 2 includes the statement transcriptions at the United Nations from 1970 to 2015. (iv) NYT 3 contains news articles in the New York Times from 2012 to 2022 with 12 categories, like “Arts”, “Business”, and “Health”. (v) WHO (Li et al., 2020) contains articles about non-pharmacological interventions from the World Health Organization, divided by week from January to May 2020. We follow the toolkit TopMost (Wu et al., 2023c) 4 to preprocess these datasets. See also Appendix A for details. Note that NYT and WHO have relatively shorter documents than other datasets.
Baseline Models We consider the following dynamic topic models as baselines: (i) DTM (Blei and Lafferty, 2006), a widely-used probabilistic
1https://www.kaggle.com/datasets/benhamner/ nips-papers
2https://www.kaggle.com/unitednations/ un-general-debates
3https://huggingface.co/datasets/Matthewww/ nyt_news
4https://github.com/bobxwu/topmost
dynamic topic model; (ii) NDTM (Dieng et al., 2019), a neural model extending DTM with neural variational inference; (iii) NDTM-b, a variant of NDTM using our Eq. (1) to model topic-word distributions. We propose this baseline for fair comparisons. (iv) DETM (Dieng et al., 2019), a neural dynamic topic model with pre-trained word embeddings. (v) BERTopic (Grootendorst, 2022), a clustering-based topic discovery model via document embeddings. (vi) DSNTM (Miyamoto et al., 2023), the latest neural model using attention mechanism to model dependencies among dynamic topics. Some studies are inapplicable for comparison since they focus on different problem settings: Zhang and Lauw (2022) deal with temporal document networks instead of sequential documents; Cvejoski et al. (2023) models the activities of static topics instead of the evolution of dynamic topics. (Rahimi et al., 2023) only cluster words to form topics and cannot specify the number of topics or infer doc-topic distributions. We fine-tune the hyperparameters of these baselines like the Gaussian variance of DTM and the min similarity of BERTopic. See implementation details of our model in Appendix B.

heading: 4.2 Dynamic Topic Quality
text: Evaluation Metrics To compare dynamic topic models, we evaluate the quality of topics from two aspects following Dieng et al. (2019): (i) Topic Coherence (TC), referring to the coherence of top words in a topic. We employ the popular CV as the coherence metric (Röder et al., 2015), which outperforms early metrics like NPMI (Lau et al., 2014). To measure the association between a topic and its time slice as well, we compute its TC with the documents of its slice as a reference corpus to estimate the word occurrence probabilities. As such, higher TC means this topic is more coherent within the documents of the slice, indicating a
stronger association with the slice. Therefore, TC can confirm if the unassociated topic issue happens. (ii) Topic Diversity (TD), referring to difference between discovered topics (Dieng et al., 2020). For a topic at a slice, we compute the proportion of its top words that only occur once and also exist at that slice. Higher TD indicates topics are more distinct from others, so TD can verify if the repetitive topic issue exists. We take the average TC and TD over all time slices. We set the number of topics (K) as 50 following Dieng et al. (2019) and report the average results of 5 runs.
Result Analysis Table 2 summarizes the topic coherence (TC) and diversity (TD) results on benchmark datasets. We have the following observations: (i) CFDTM significantly outperforms baselines in terms of TD. We see baseline models commonly generate repetitive topics as indicated by their low TD scores. For instance, the TD score of CFDTM is 0.879 on ACL whereas the highest baseline only is 0.651. This demonstrates that CFDTM produces more diverse topics than baselines. The improvements result from our evolution-tracking contrastive learning that encourages topics to be distinct from each other. Hence it mitigates the repetitive topic issues while tracking topic evolution. (ii) CFDTM achieves the best TC scores with significant improvements. For instance, CFDTM has a TC score of 0.581 on NeurIPS while the runnerup is only 0.455. This implies the topics of CFDTM are more coherent and associated with their time slices than these baselines. The reason lies in that CFDTM adopts our unassociated word exclusion method, which excludes unassociated words from topics. Thus it alleviates the unassociated topic issue and better captures topic evolution. We mention that merely modeling doc-topic distributions as Eq. (1) cannot result in these improvements since NDTM-b is incomparable to CFDTM.
The above results manifest that our CFDTM ef-

heading: 2007 Topic#5 2012 Topic#5 2017 Topic#5 2007 Topic#15 2012 Topic#15 2017 Topic#15 2007 Topic#21 2012 Topic#21 2017 Topic#21
text: fectively tracks topic evolution with more diverse, coherent, and associated topics, which can benefit downstream tasks and applications. See case studies of word and topic evolution in Sec. 4.6 and appendix D.

heading: 4.3 Robustness to Evolution Intensity Hyperparameter
text: We demonstrate the robustness of our model to the evolution intensity hyperparameter λ(t). Figure 7 reports the topic coherence (TC) and diversity (TD) results with varying λ(t). We see that TC and TD scores remain relatively stable with larger λ(t). This can be attributed to our UWE method which excludes unassociated words from topics even if they are overly related by large λ(t). These validate that our method is robust to the evolution intensity hyperparameter. This advantage is vital as it effectively reduces the efforts to find suitable intensity hyperparameters, overcoming a difficult challenge in previous work (Dieng et al., 2019).

heading: 4.4 Ablation Study
text: We conduct ablation studies to show the necessity of our proposed two methods: Evolution-Tracking Contrastive learning (ETC) and Unassociated Word
Exclusion (UWE). From Table 3, we have the following discoveries: (i) ETC effectively mitigates the repetitive topic issue. Compared to CFDTM, TD greatly degrades if without ETC (w/o ETC). For example, the TD decreases from 0.846 to 0.466 on NeurIPS. Besides, TD reduces more if building positive relations without negative relations (w/o negative). This implies only building positive relations worsens the repetitive topic issue. (ii) UWE can alleviate the unassociated topic issue. Table 3 shows the TC of CFDTM are significantly better than the case without UWE (w/o UWE), e.g., 0.581 vs. 0.483 on NeurIPS. More importantly, CFDTM reaches higher TC than directly masking unassociated words (w/ masking). This is because direct masking cannot leverage these words to refine topic semantics. In summary, these results demonstrate the necessity of our ETC and UWE for improving dynamic topic modeling performance.

heading: 4.5 Text Classification and Clustering
text: Apart from dynamic topic quality, we compare the quality of learned doc-topic distributions through
downstream tasks: text classification and clustering. In detail, we leverage the document categories of the NYT dataset for evaluation. For text classification, we train SVM classifiers with learned doc-topic distributions as features and predict the categories of testing documents following Wu et al. (2023b). This performance is evaluated by Accuracy and F1. For text clustering, we use the most significant topics in the doc-topic distributions as clustering assignments and evaluate this performance by widely-used metrics Purity and NMI following Wu et al. (2023b). Note that our purpose is not to achieve state-of-the-art classification or clustering performance but to compare the quality of doc-topic distributions.
Figure 6 shows that our CFDTM consistently surpasses all baselines in terms of both classification and clustering performance. We mention that the improvements of CFDTM over baselines are statistically significant at 0.05 level. These results manifest that our model generates more accurate doc-topic distributions for downstream classification and clustering tasks.

heading: 4.6 Case Study: Evolution of Topics
text: Furthermore, we report case studies to show our model captures the evolution of topics. Figure 5 illustrate some topics discovered by our model from NeurIPS, which evolve from the year 2007 to 2017. We see Topic#5 tracks the trend of word embeddings and attention mechanism in the NLP field. In addition, Topic#15 recognizes the popularity of convolutional neural networks and adversarial training in computer vision, indicated by the appearances of the words “convolutional”, “adversarial”, and “gan”. Topic#21 focuses on the evolution of speech processing, which also captures the application of neural networks as implied by the words “dnn” and “gru”. See full topic lists of different models in Appendix E.

heading: 5 Conclusion
text: In this paper, we propose CFDTM, a new chainfree neural dynamic topic model. To break the tradition of simply chaining topics in previous work, CFDTM employs the novel evolution-tracking contrastive learning and unassociated word exclusion methods. Experiments demonstrate that our CFDTM consistently outperforms baselines and effectively mitigates the repetitive topic and unassociated topic issues. CFDTM tracks topic evo-
lution with higher coherence and diversity, and achieves better performance on downstream tasks. Our model also shows robustness to the hyperparameter for evolution intensities.
Limitations
Our proposed method has achieved promising improvements by mitigating the repetitive topic and unassociated topic issues in dynamic topic modeling, but we consider the following limitations as future work: • Extend our model to discover dynamic topics
from multilingual corpora. Numerous multilingual corpora are prevalent in the current Internet (Zosa and Granroth-Wilding, 2019). It would inspire more applications to discover and compare historical trends from different cultures and languages by extending our method to the multilingual setting.
• Leverage Large Language Models (LLMs) for dynamic topic modeling. Popular LLMs retain rich knowledge from pretraining corpora (Petroni et al., 2019; Pan et al., 2023; Wu et al., 2024d), but they cannot directly discover topics from a large dataset. Alternatively, we may leverage the prior knowledge in LLMs to further improve the performance of dynamic topic modeling.

heading: Acknowledgements
text: This research/project is supported by the National Research Foundation, Singapore under its AI Singapore Programme (AISG Award No: AISG2-TC2022-005).

heading: A Pre-processing Datasets
text: To pre-process datasets, we conduct the following the steps of Wu et al. (2023c) 5: (i) tokenize documents and convert to lowercase; (ii) remove punctuation; (iii) remove tokens that include numbers; (iv) remove tokens less than 3 characters; (v) remove stopwords.
Table 5 summarizes the statistics of datasets after pre-processing.

heading: B Model Implementation
text: Generation of Sequential Documents As mentioned in Sec. 3.5, the generation process of sequential documents in our model is in the framework of VAE (Kingma and Ba, 2014; Rezende et al., 2014). Following Srivastava and Sutton (2017); Wu et al. (2022, 2023b), the prior distribution is modeled with Laplace approximation (Hennig et al., 2012) to approximate a symmetric Dirichlet prior as µ0,k = 0 and Σ0,kk = (K − 1)/K. The encoder network fΘ is a two-layer MLP with softplus as the activation function, concatenated with two single layers each for the mean and covariance matrix. The encoder takes the Bag-of-Words of documents as inputs and outputs the mean and covariance matrix parameters, µ(t,d) and Σ(t,d). We reuse the encoder for all time slices to save model
5https://github.com/bobxwu/topmost
2012 2014 2016 2018 2020 2022
Topic "Russia-Ukraine War" ukraine russian war invasion
2012 2014 2016 2018 2020 2022
Topic "Disease" covid vaccine symptoms cancer
2012 2014 2016 2018 2020 2022
Topic "Europe" brexit boris theresa energy
2012 2014 2016 2018 2020 2022
Topic "Natural Disaster" earthquake nepal hurricane irma
Figure 8: Case study. Evolution of word probability. X-axis denotes years; Y-axis denotes word probabilities.
parameters as Dieng et al. (2019). The decoder generates documents with doc-topic distribution θ(t,d) and the topic-word distribution matrix of the corresponding time slice β(t). We use pre-trained 200-dimensional GloVe (Peng et al., 2014) to initialize the word embeddings W in Eq. (1).
Hyperparameter Selection We set π in Eq. (1) as 1.0, Ntop in Eq. (5) as 15, and τ in Eq. (2), (3) and (7) as 0.1. We use Adam (Kingma and Ba, 2014) to optimize model parameters and run our model for 800 epochs with a learning rate as 0.002.
See our code for more implementation details.

heading: C Robustness to the Number of Topics
text: Apart from the aforementioned results under K = 50 in Table 2, we experiment with different numbers of topics on NeurIPS to verify the robustness of our model. Table 4 reports the performance un-
der K = 60, 80, 100. We observe that our CFDTM outperforms all baseline methods concerning both topic coherence and diversity. These results validate the robustness of our model to the number of topics.

heading: D Case Study: Evolution of Word Probabilities
text: We conduct case studies to show our model captures the evolution of word probabilities of topics. Following Dieng et al. (2019), Figure 8 plots the word probability evolution in the topics (Eq. (1)) discovered by our CFDTM from the NYT dataset. In detail, we see that our model successfully captures the Crimea Crisis in 2014 and the RussiaUkraine War in 2022 as shown by the probability increments of the words “war” and “invasion”. For the topic “Disease”, our approach discerns the explosion of Covid-19 in 2020, indicated by the increase of “covid” and “vaccine”. We notice people focus on cancer before Covid-19. In addition, our model discovers Brexit around 2019 and the energy crisis after 2020 under the topic “Europe”. Under the topic “Natural Disaster”, our model reveals the Nepal earthquake in 2015 and Hurricane Irma in 2017 as implied by the word probabilities.

heading: E Full Topic Lists
text: Here are the discovered topics of different models in 2017 of NeurIPS (the latest time slice). DETM
Topic#1: data set problem learning algorithm number based using results one
Topic#2: networks neural network deep layer weights learning convolutional arxiv training
Topic#3: model data using figure learning approach one based used set
Topic#4: layer convolutional network layers image model cnn input feature networks
Topic#5: image object images segmentation detection flow ground network vision pose
Topic#6: model data figure one set based using used learning two
Topic#7: matrix rank matrices tensor low algorithm norm entries singular spectral
Topic#8: gan adversarial generator discriminator image images generative gans training samples
Topic#9: user items users item model revenue data recommendation ranking information
Topic#10: model learning set number function two one first using linear
Topic#11: learning data set problem two model also one algorithm systems
Topic#12: policy state learning function value reward optimal action algorithm probability
Topic#13: wasserstein transportation function model pipeline data earth nov coming one
Topic#14: neural brain time data information activity neurons model fig spike
Topic#15: model using set one algorithm learning results number based two
Topic#16: algorithm submodular set approximation function functions log theorem privacy problem
Topic#17: clustering algorithm cluster clusters points means algorithms graph let number
Topic#18: recurrent lstm memory rnn neural sequence model attention arxiv models
Topic#19: gaussian posterior inference bayesian model prior distribution log variational likelihood
Topic#20: log estimator theorem error estimation bound analysis probability results rate
Topic#21: model algorithm number data learning set time one given using
Topic#22: one set figure model learning function using given two data
Topic#23: using data method learning model function based one methods set
Topic#24: model data models group structure methods latent variables using analysis
Topic#25: kernel space kernels random function functions features approximation data learning
Topic#26: networks training deep neural layer network dropout layers learning arxiv
Topic#27: learning task tasks data transfer model multi performance features different
Topic#28: causal variables learning set data treatment discrimination fair model two
Topic#29: features feature classification accuracy prediction learning cost set training datasets
Topic#30: inference variational latent log models model distribution variables generative likelihood
Topic#31: loss learning risk function algorithm bounds bound generalization empirical regression
Topic#32: algorithm data time distributed algorithms parallel computation number machine computing
Topic#33: model training learning neural models task language arxiv output sequence
Topic#34: policy agent learning agents reward reinforcement state model policies environment
Topic#35: set problem algorithm one using data number method first function
Topic#36: communication workers worker protocol asynchronous decentralized server message attack attacks
Topic#37: regret algorithm bandit online bound arm problem log bandits optimal
Topic#38: distribution random samples distributions sample estimation probability density model models
Topic#39: algorithm number problem set time one data algorithms given using
Topic#40: embedding embeddings neural word vectors words representations vector text learning
Topic#41: label learning labels data classifier class classification domain training labeled
Topic#42: graph node nodes graphs network networks tree edge edges set
Topic#43: gradient optimization stochastic methods sampling batch variance function method step
Topic#44: set one function two number size given also based used
Topic#45: convex optimization algorithm gradient convergence descent stochastic problem method problems
Topic#46: michel succinctly guo nielsen fulfilled jonas misspecified herein plethora gross
Topic#47: attention visual model question image object prediction video dataset models
Topic#48: time state process dynamics model transition states systems processes system
Topic#49: distance metric data hash points function embedding nearest point hashing
Topic#50: sparse linear solution problem convex lasso min local condition optimization
DSNTM
Topic#1: functions sessions repeated function constant potential behavior questions performances programming
Topic#2: used modified use using compact also algorithm encoding commonly improve
Topic#3: characters divide google ideas actors intersection texts differences divided text
Topic#4: functionals cns hlt torr multivariate marcel recursion lyapunov ell rna
Topic#5: networks programming network data telephone broadcast programs settings sites yahoo
Topic#6: capital sources statistical largest found popular large smaller bulk widely
Topic#7: multiplier optimization sensor occupancy damping compression kernel sigmoid disk segmentation
Topic#8: structures highest level status living class skills million built cognitive
Topic#9: absolute theory model practical dimension scope social political serious party
Topic#10: recovery georgia ari gallant sustained complicate undergoes welch efforts fluid
Topic#11: chart jia flip degree elevation deng illinois compression unsuccessful stacked
Topic#12: bracket completely category lasts cells status attains mediated tile flattened
Topic#13: features version music compositions versions feature experimental contains programming generic
Topic#14: algorithm algorithms detection adaptive inference recursive automatically mathematically simplest stimuli
Topic#15: performance performances inadequate objects evidence effectiveness immune abilities insufficient adequate
Topic#16: learning skills abilities hebbian tricks learn opportunities teach positioning methodologies
Topic#17: experience benefits form say law hard david says bill deal
Topic#18: period represented since december february proceedings october appearance january basis
Topic#19: carnegie summary programming grove news larry computing annual java ibm
Topic#20: data element contains strings sheet fiber graph component funding dramatic
Topic#21: high layers air volume science layer royal highest scientists filled
Topic#22: enhancement spontaneous associated segmentation pat approximations strategies substitutions term smooth
Topic#23: two several three kitchen including like one number little small
Topic#24: use appropriate equipment fairly manner fire clean buy measures necessary
Topic#25: information document details text provide matrix data clarity management false
Topic#26: algorithm gaussian algorithms mapped theme reality sequence structure format linear
Topic#27: isometry modulo vanilla multiplicative polynomial descending hamiltonian simoncelli leftmost diag
Topic#28: accuracy gauge align values coordinate footprint style size dimensions measurements
Topic#29: method convergence divergence quantum notation clustering stochastic sampling machines macroscopic
Topic#30: functions complementary optical units algorithms sharing software points systems discrete
Topic#31: optimization optimal variables analyze constraint algorithms behaviors dynamics heuristic equilibrium
Topic#32: physically images finite independent magnetic image web endpoints arrays site
Topic#33: algorithm algorithms estimator parameterized bayesian decoding viterbi encoding optimization asymptotically
Topic#34: satisfying richness refines nicely nervous formalizes carbonell diverse musical automating
Topic#35: unused pool property layer permanent reuse thumb unwanted potential plastic
Topic#36: subjective probability judgment methods race decisions spatial formula projects develop
Topic#37: hope believe impossible solution solve find cause still say important
Topic#38: elimination kyoto processing division bases seed reductions eliminating eliminated regulatory
Topic#39: residuals theoretical modeling solving mathematical empirical length numerical ratio approximate
Topic#40: stock character updates draw bid general force role jobs meeting
Topic#41: models problems term problem production forced companies company many names
Topic#42: model changed change changes models design almost different prediction new
Topic#43: treat want make able good throw think get achieve accurate
Topic#44: millions estimated families men knew compensation alarm discrimination execution magnitude
Topic#45: videos dirichlet eigenvectors oscillatory mathieu unbounded nicol distributions magnitudes sigmoid
Topic#46: affine maze heaviside computational singularities tensor logarithms gradients algebraic pdf
Topic#47: svd kernels decomposition bursting learner maximization auditory sift ocr predicate
Topic#48: function appropriate input reserve determine criteria parameters ensure information healthy
Topic#49: process application psychology community applications management overview language systems implementation
Topic#50: basic code tuning range base assembly comprises community consists ranges
CFDTM
Topic#1: policy reward agent reinforcement state policies value action actions planning
Topic#2: loss bounds bound complexity risk hypothesis online generalization upper empirical
Topic#3: sup lim inf strategy strategies lugosi ergodic feasible experts objectives
Topic#4: parameterizations munos convolved gaussianity toy nonasymptotic berkeley costly vedaldi partitioned
Topic#5: word words attention language items embeddings question sentence embedding answering
Topic#6: constraints springerverlag constraint entropy resulting mrf assignment configurations maximum segment
Topic#7: group interactions topic structure gibbs topics tree additive groups trees
Topic#8: hlog majorization citeseer lowrank afosr krizhevsky convnet salakhutdinov slices subnetwork
Topic#9: clustering clusters cluster algorithm queries algorithms query number means cost
Topic#10: overcomplete rmse osindero balcan lowdimensional rbms dbns kkt yannlecuncom exdb
Topic#11: variational posterior inference likelihood distribution latent bayesian sampling generative samples
Topic#12: fukumizu guestrin rkhs multiarmed symp steyvers dkl supx noising dbn
Topic#13: identifiability convolving bigram theart adaptivity rescale multiresolution scholkopf interpolate rbms
Topic#14: layer networks layers deep neural network convolutional activations weights batch
Topic#15: image images object pose vision convolutional objects adversarial gan flow
Topic#16: rule rules realization group music abstraction compositional raw lasso path
Topic#17: isomap roweis highdimensional parameterize whye lies saul regarded name put
Topic#18: libsvm approximator bfgs wwwcsientuedutw cjlin minf reparameterization olkopf overfits mdps
Topic#19: decorrelated denoise denoised indexhtml gehler tresp twodimensional centroid tsochantaridis loosely
Topic#20: lasso selection regularized screening regularization logistic argmin biometrika sparsity regression
Topic#21: speaker voice speech mel gru phoneme mos synthesis audio speakers
Topic#22: convergence games equilibrium nash equilibria game gan player descent divergence
Topic#23: kernel kernels space points invariant metric fourier quadrature embedding gaussian
Topic#24: components correlations mit component mixtures solid ica knn eigenvectors obtained
Topic#25: regret arm bandit arms bound reward bandits optimal rewards expected
Topic#26: estimator estimation density sample estimators statistics distributions statistical iid high
Topic#27: stimulus sensory information memory stimuli recall strength choice retrieval items
Topic#28: optimization convex gradient convergence descent stochastic iteration problems methods accelerated
Topic#29: warmuth vempala stochasticity littlestone servedio adversarially mehryar olkopf schapire yishay
Topic#30: andez nonstationary symmetrized comput zabih universit parametrize tardos auai submodularity
Topic#31: revenue discrimination resolving proxy price causal prices market race unresolved
Topic#32: transductive abbeel pomdp discretize maxj mdps discriminatively boykov tractably timesteps
Topic#33: tsitsiklis ihler componentwise hassibi parallelizable labelings isit willsky limt little
Topic#34: classification task label classifier learning tasks labels class dataset domain
Topic#35: pomdps maxk jmlr liblinear herbrich graepel tsochantaridis joachims vayatis puterman
Topic#36: tishby chervonenkis sompolinsky ijcai markram initialisation datapoints vapnik thresholded factorizes
Topic#37: rmax svms cristianini vectorized semisupervised tresp adaboost cholesky linearize regularizes
Topic#38: itti conll treebank compensate leading pontil alter great enhances enforced
Topic#39: matrix rank matrices low entries tensor norm singular completion sparse
Topic#40: unnormalized felzenszwalb sudderth buhmann ijcv perona groundtruth bmvc sublinear resampled
Topic#41: neurosci neurophysiol movshon spatio glms paninski schraudolph preserving exponentiated scholkopf
Topic#42: brain activity responses fig neurons neuroscience connectivity spike spikes neuron
Topic#43: invariances olshausen bergstra pretraining resample vedaldi loglikelihood convolved subgradients rescale
Topic#44: learnability hyperparameter heckerman timit universit borgwardt multiagent rithm inpainting downward
Topic#45: shadlen heskes knn xti axt devroye yedidia orfi kkt enyi
Topic#46: srebro elementwise regularizers ritov xxt nonnegativity juditsky negahban omnipress tsybakov
Topic#47: initialised probabilit vldb montr kernelized independencies auai delta things kinds
Topic#48: velocity described motion direction retina appropriate orientation system position cells
Topic#49: graph nodes node graphs edges edge vertices influence vertex set
Topic#50: sparse linear solution problem convex lasso min local condition optimization

heading: Future Work
text: Limitations
Our proposed method has achieved promising improvements by mitigating the repetitive topic and unassociated topic issues in dynamic topic modeling, but we consider the following limitations as future work: • Extend our model to discover dynamic topics
from multilingual corpora. Numerous multilingual corpora are prevalent in the current Internet (Zosa and Granroth-Wilding, 2019). It would inspire more applications to discover and compare historical trends from different cultures and languages by extending our method to the multilingual setting. • Leverage Large Language Models (LLMs) for dynamic topic modeling. Popular LLMs retain rich knowledge from pretraining corpora (Petroni et al. , 2019; Pan et al. , 2023; Wu et al. , 2024d), but they cannot directly discover topics from a large dataset. Alternatively, we may leverage the prior knowledge in LLMs to further improve the performance of dynamic topic modeling