Dynamic Structured Neural Topic Model with Self-Attention Mechanism
Year: 2023

Abstract:
This study presents a dynamic structured neural topic model, which can handle the timeseries development of topics while capturing their dependencies. Our model captures the topic branching and merging processes by modeling topic dependencies based on a selfattention mechanism. Additionally, we introduce citation regularization, which induces attention weights to represent citation relations by modeling text and citations jointly. Our model outperforms a prior dynamic embedded topic model (Dieng et al., 2019) regarding perplexity and coherence, while maintaining sufficient diversity across topics. Furthermore, we confirm that our model can potentially predict emerging topics from academic literature.

heading: 1 Introduction
text: Topic models are dominant tools for discovering the underlying semantic structure in a collection of documents. As a part of such topic models that can capture the chronological transition of topics have been intensively studied in recent years.
The dynamic topic model (DTM; Blei and Lafferty, 2006) is a pioneering work that captures the time-series evolution of topics. It successfully visualizes the changes in the topic proportion and the word distributions of each topic over time. Recently, neural networks have empowered topic models to handle a significant collection of documents. The dynamic embedded topic model (D-ETM; Dieng et al., 2019) introduces word embeddings and amortized variational inference into DTM, which significantly improves topic quality while reducing computational time. D-ETM is widely applied to large-scale time-series documents, such as scientific papers and social media (Churchill and Singh, 2022; Murakami et al., 2021).
However, DTM and D-ETM assume that topics evolve independently without interaction. This assumption is inappropriate, particularly for mod-
eling scientific papers, where documents are dependent on each other through citation relations. For example, the recent text-to-image techniques are evolved from multiple topics such as image processing, natural language processing, and deep learning. Conventional dynamic topic models cannot capture how past topics contributed to the emergence of new topics. Furthermore, these models cannot predict emerging topics because the posterior word distribution of topics is parameterized for each time step, as explained later in detail.
To overcome these challenges, we propose a dynamic structured neural topic model (DSNTM), which captures the dependencies among topics over time (Fig. 1). Specifically, DSNTM models topic dependencies based on a self-attention mechanism (Vaswani et al., 2017; Lin et al., 2017), which reveals how past topics branches or merges into new topics. We can quantitatively evaluate which past topics contributes to the emergence of new topic by observing attention weights. In addition, the selfattention mechanism shares the parameters used for inferring topics at each time step, enabling the model to predict emerging topics.
Additionally, we introduce citation regularization, which induces attention weights to reflect the citation relations among documents. Citation
5916
regularization enables DSNTM to model text and citation jointly, improving the inferred topics’ quality and accurately capturing their transitions. Due to the high expressive power of the self-attention mechanism and additional citation information, our model can capture the complex branching and merging processes of topics over time.
In the experiment, we used datasets consisting of over 20,000 scientific papers on computer science and natural language processing retrieved from the Semantic Scholar Open Research Corpus (S2ORC; Lo et al., 2020). Experimental results show that DSNTM outperforms recent neural topic models (Dieng et al., 2019, 2020) regarding perplexity and coherence, while maintaining sufficient diversity across topics. We also confirmed that DSNTM accurately captures the topic branching and merging processes and can potentially predict emerging topics in the academic literature.

heading: 2 Related Work
text: Extending the dynamic topic model (Blei and Lafferty, 2006), several variants have been proposed, such as the dependent Dirichlet processes mixture model (Lin et al., 2010), infinite dynamic topic model (Ahmed and Xing, 2010), and D-ETM (Dieng et al., 2019). However, these studies treat timeseries changes in topics independently and cannot capture dependencies among topics.
Relating to structured topic models, several models have been proposed such as tree-structured topic model (Griffiths et al., 2003; Isonuma et al., 2020; Chen et al., 2021) and pachinko allocation models (Li and McCallum, 2006; Mimno et al., 2007). In addition, several studies have modeled the structure among documents by jointly modeling citation network and text (Nallapati et al., 2008; Tu et al., 2010; Chang and Blei, 2010; Lim and Buntine, 2015). However, these studies were not intended to track the time series transitions of topics.
The dynamic and static topic model (DSTM; Hida et al., 2018) extends the pachinko allocation model to capture the dynamic structure over time and the static structure among topics at each time step. DSTM has several drawbacks against our model. It cannot capture topic dependencies across multiple time steps, and thus, incorporating citation information is challenging. In addition, it cannot predict emerging topics as topic transitions are parameterized for each time step. Moreover, collapsed Gibbs sampling is used to infer posteriors,
which is not scalable for large datasets. The dynamic topic model on networked documents (NetDTM; Zhang and Lauw, 2022) models time-series documents and citation networks simultaneously. However, NetDTM does not capture the relations between topics nor predict emerging topics, which significantly differ from ours.
Contrary to the aforementioned studies, our work introduces amortized variational inference using the self-attention mechanism. This inference technique enables us to capture the topic branching and merging process across multiple time steps with significant scalability. Furthermore, our model can predict emerging topics from past topics.

heading: 3 Background
text: We first review the embedded topic model (ETM; Dieng et al., 2020), and then explain D-ETM, which combines ETM with DTM before introducing our DSNTM.

heading: 3.1 Embedded Topic Model (ETM)
text: ETM (Dieng et al., 2020) is a topic model that introduces word embeddings into LDA. The generative process of documents is the following:
1. For each document index d ∈ {1, . . . , D}: Draw topic proportion: θd ∼ LN (0, I) (1) 2. For each word index n∈{1, . . . , Nd} in d: Draw topic assignment: zd,n ∼ Cat(θd) (2) Draw word: wd,n ∼ Cat(βzd,n) (3)
Here, Cat(·) and LN (·, ·) denote the categorical distribution and logistic-normal distribution (Atchison and Shen, 1980), respectively. βk ∈ RV represents the word distribution of the kth topic computed as follows:
βk = softmax(ρ ⊤αk). (4)
where ρ ∈ RL×V denotes the L-dimensional word embeddings of the entire vocabulary. The ρv ∈ RL corresponds to the vth word embedding. αk ∈ RL denotes the embedding representation of the kth topic in the semantic space of words, which is called topic embedding.

heading: 3.2 Dynamic Embedded Topic Model (D-ETM)
text: D-ETM (Dieng et al., 2019) analyzes time-series documents by changing the topics over time. Contrary to ETM, D-ETM assumes a discrete-time
Markov chain for the topic embedding in Eq. (5) and the topic proportion mean in Eq. (7). The generative process of documents is described as follows:
1. For each time step t ∈ {1, . . . , T}: Draw word distribution for each topic k:
α (t) k ∼ N (α (t−1) k , σ 2I) (5) β (t) k = softmax(ρ ⊤α(t)k ) (6) Draw topic proportion mean:
ηt ∼ N (ηt−1, δ2I) (7) 2. For each document index d∈{1, . . . , D}:
Draw topic proportion: θd∼LN (ηtd , γ2I) (8) 3. For each word index n∈{1, . . . , Nd} in d:
Draw topic assignment: zd,n ∼ Cat(θd) (9) Draw word: wd,n ∼ Cat(β(td)zd,n) (10)
where α(t)k and β (t) k are the topic embedding and word distribution assigned to the kth topic in the tth time step, respectively. σ, δ , and γ are model hyperparameters, which control the variance of normal distributions.
Dieng et al. (2019) approximate the posterior distribution of α, η and θ with amortized variational inference (Kingma and Welling, 2014; Rezende et al., 2014). Particularly, for the topic embeddings α, the mean-field family is used for the approximation as follows:
q(α (t) k ) = N (µ (t) k ,σ (t) k ) (11)
q(α) = ∏
k
∏
t
q(α (t) k ) (12)
where µ(t)k ∈RL and σ (t) k ∈RL are learnable vectors representing the mean and variance of α(t)k , respectively.
However, this mean-field approximation has two limitations.
Dependencies among topics cannot be modeled Eq. (12) assumes that topics are independent of each other. This assumption is typically inappropriate for time-series documents. For instance, academic topics sometimes emerge from interactions among several past topics. Topic dependencies must be modeled to consider such interactions.
Emerging topics cannot be predicted D-ETM infers a topic by parameterizing µ(t)k and σ (t) k for each time step t. As documents in the tth time step
are used to infer these parameters, topics cannot be inferred for the time steps that are not contained in the dataset. The parameters must be shared across all time steps to predict emerging topics.
To overcome these limitations, our DSNTM introduces the self-attention mechanism to infer those parameters. The self-attention mechanism enables DSNTM to capture the topic dependencies, while sharing the parameters across all time steps.

heading: 4 Dynamic Structured Neural Topic Model (DSNTM)
text: This section describes the proposed DSNTM. The generative process of documents is the same as that of D-ETM.

heading: 4.1 Inference of Topic Embeddings
text: Contrary to D-ETM, we use structured variational inference to infer the topic embeddings. We compute a topic embedding from all previous topic embeddings using the self-attention mechanism.
α̃ (t) k = self − attention(α̃ (1:t−1) 1:K ) (13)
q(α (t) k |α̃ (1:t−1) 1:K ) = N (fµ(α̃ (t) k ), fσ(α̃ (t) k )) (14)
where α̃(t)k ∈ RL denotes the transformed topic embedding. fµ and fσ are multi-layer perceptrons (MLP) that convert α̃(t)k to a variational normal distribution.
Computation of Self-attention We present an outline of the self-attention mechanism in Fig. 2. To calculate Eq. (13), we obtain the key K
(1:t−1) 1:K ∈ RK(t−1)×L and value V (1:t−1) 1:K ∈ RK(t−1)×L from all previous transformed topic embeddings α̃(1:t−1)1:K . On the other hand, the query q (t−1) k ∈RL is obtained from the kth transformed topic embedding α̃(t−1)k at time step t−1.
K (1:t−1) 1:K = fk(α̃ (1:t−1) 1:K ) (15) V (1:t−1) 1:K = fv(α̃ (1:t−1) 1:K ) (16)
q (t−1) k = fq(α̃ (t−1) k ) (17)
where fk, fv and fq denote MLPs. Subsequently, we compute the attention weight a(t)k ∈ RK(t−1) between each past topic and the kth topic at time step t, similar to Vaswani et al. (2017).
a (t) k = softmax(
q (t−1) k K (1:t−1)⊤ 1:K√ L ) (18)
Then, we obtain α̃(t)k by calculating the sum of V
(1:t−1) 1:K weighted by the attention weights. We use a residual connection to obtain α̃(t)k for preventing gradient explosion and disappearance (Simonyan and Zisserman, 2014).
∆α̃ (t) k = a (t) k V (1:t−1) 1:K (19)
α̃ (t) k = LayerNorm(∆α̃ (t) k + α̃ (t−1) k ) (20)
Here, we use the layer normalization (Ba et al., 2016) to compute the transformed topic embeddings. At the time step t = 1, we initialize the transformed topic embeddings α̃(1)k from a normal distribution.
α̃ (1) k ∼ N (0, σ2I) (21)
Note that the prior distribution assumes that α(t)k is drawn from a normal distribution as follows: α
(t) k ∼N (α (t−1) k , σ
2I). This assumption regularizes α(t)k to be close to its previous topic α (t−1) k .
Motivations behind Self-attention We use the self-attention mechanism for the following reasons:
(1) Instead of the self-attention mechanism, we can also capture the dependencies among topics by simply parameterizing the topic embeddings as follows:
α̃ (t) 1:K=Wα̃ (t−1) 1:K (22)
Here, W ∈ RK×K is a learnable weight matrix, where wi,j represents the dependency between topic i and j. However, this parameterization cannot capture the dependencies across multiple time
steps, and a method that allows an arbitrary number of inputs is required. The self-attention mechanism, which handles an arbitrary-length sequence, satisfies this requirement and captures the topic dependencies across multiple time steps.
(2) The self-attention mechanism parameterizes MLP fq, fk, and fv to compute the embeddings of subsequent topics from previous topics. As the parameters of MLPs are shared over time, the selfattention mechanism allows DSNTM to predict the emerging topic embeddings.

heading: 4.2 Overall Inference and ELBO
text: Under our proposed probabilistic model, the likelihood of documents is given by
p(w1:D|σ, δ, γ)
=
∫ {∏
d
∏
n
∑
zd,n
p(wd,n|β(td)zd,n)p(zd,n|θd)p(θd|ηtd) }
{∏
t
∏
k
p(ηt|ηt−1)p(αtk|αt−1k ) } dθdηdα
=
∫ {∏
d
∏
n
(β(td) · θd)wd,np(θd|ηtd) }
{∏
t
∏
k
p(ηt|ηt−1)p(αtk|αt−1k ) } dθdηdα (23)
Subsequently, let q(θ,η,α) be the variational distribution of the posterior distribution p(θ,η,α|w1:D). Following D-ETM, q(θ,η,α) is computed as follows:
q(θ,η,α)
= ∏
d
q(θd|ηtd ,wd)× ∏
t
q(ηt|η1:t−1, w̃t)
× ∏
t
∏
k
q(α (t) k |α̃ (1:t−1) 1:K ) (24)
q(θd|ηtd ,wd) = LN (fµ(θ̃td), fσ(θ̃td)) θ̃td = fθ([ηtd ;wd]) (25) q(ηt|η1:t−1, w̃t) = N (fµ(η̃t), fσ(η̃t)) η̃t = [ht;ηt−1] (26)
where wd is the bag-of-words (BoW) representation of document d. ht is the hidden state of a longshort term memory network (LSTM; Hochreiter and Schmidhuber, 1997) that uses the normalized BoW representation w̃t of all documents at time t as input. [·; ·] denotes the concatenation of vectors.
The evidence lower bound (ELBO) for the document log-likelihood is derived as follows:
Ldoc= ∑
d
E q(θd)q(ηt)q(α (t) k )
[ w⊤d log(β (td) · θd) ]
− ∑
t
∑
k
DKL
[ q(α
(t) k |α (1:t−1) 1:K )||p(α (t) k |α (t−1) k )
]
− ∑
d
DKL [ q(θd|ηtd ,wd)||p(θd|ηtd) ]
− ∑
t
DKL [ q(ηt|η1:t−1, w̃t)||p(ηt|ηt−1) ] (27)

heading: 5 Citation Regularization
text: DSNTM has difficulty interpreting the attention weights among topics. Ideally, the attention should model the dependency among topics, representing citation relations between documents. Therefore, we let the attention weights to be interpretable by regularizing them to correspond with citation relations. Citation regularization also improves the quality of the inferred topics by jointly modeling the text and citations.
To regularize the attention weights, we model the citations between documents based on the topic proportion θ, the attention weights a, and the paper proportion ϕ as shown in Fig. 3. Formally, for each document pair (i, j) ∈ {1, . . . , D} × {1, . . . , D}, the citation is modeled as follows:
1. Draw citing topic assignment: zi ∼ Cat(θi) (28)
2. Draw cited topic assignment: zj∼Cat(a(ti)zi ) (29)
3. Draw cited document: dj ∼ Cat(ϕzj ) (30)
where a(ti)k ∈ RK(ti−1) is the attention weight, which denotes the probability distribution across all previous topics zj ∈ {1, . . . ,K}×{1, . . . , ti− 1}. The paper proportion ϕk ∈ RD denotes the probability distribution of cited documents where a topic k is assigned, as explained in next section.

heading: 5.1 Obtaining paper proportion
text: From Bayes’ theorem, we calculate the probability where a paper dj ∈ {1, . . . , D} is cited according to a topic zj as follows:
p(dj |zj) = p(zj |dj)p(dj)
p(zj) ∝ p(zj |dj) (31)
Here, we assume that the prior p(dj) is uniformly distributed, and p(zj) can be ignored because it is constant regardless of dj . As p(dj |zj)=ϕ(dj)zj and p(zj |dj)=θ(zj)dj , ϕ (dj) zj can be simply computed by normalizing θ(zj)dj :
ϕ (dj) zj = θ (zj) dj∑
dj θ (zj) dj
(32)

heading: 5.2 Overall Inference and ELBO
text: Hence, under our modeling assumption, the likelihood of a citation ci,j ∈ {0, 1} is given by
p(ci,j=1|θ,α)
= K∑
k
K∑
k′ p(dj |ϕk′)p(zj=k′|a(ti)k )p(zi=k|θi)
(33)
where ci,j=1 indicates that the document di cites the document dj . Finally, the likelihood of both documents and citations can be described as fol-
lows:
p(w1:D, c1,1, . . . , cD,D|σ, δ, γ)
=
∫ {∏
i
∏
n
(β(ti) · θi)wi,np(θi|ηti) }
{∏
t
∏
k
p(ηt|ηt−1)p(αtk|αt−1k ) }
{∏
i
∏
j
p(ci,j |θ,α) } dθdηdα (34)
The ELBO for both document and citation loglikelihood is derived as follows: L= ∑
i
Eq(θi) [ w⊤i log(β (ti) · θi) ]
− ∑
t
∑
k
DKL
[ q(α
(t) k |α (1:t−1) 1:K )||p(α (t) k |α (t−1) k )
]
− ∑
i
DKL [ q(θi|ηti ,wi)||p(θi|ηti) ]
− ∑
t
DKL [ q(ηt|ηt−1, w̃t)||p(ηt|ηt−1) ]
+ ∑
i
∑
j
E q(θi)q(ηt)q(α (t) k )
[ log p(ci,j |θ,α) ]
= Ldoc + Lcit (35)
Here, Ldoc is defined in Eq. (27), and Lcit is defined as the following equation:
Lcit = D∑
i
D∑
j
BCE [p(ci,j |θ,α), ci,j ] (36)
where BCE denotes the binary cross entropy.
6 Experiment
6.1 Experimental Setup1
Dataset In our experiments, we used ACL and CS dataset, which were based on the Semantic Scholar Open Research Corpus (S2ORC; Lo et al., 2020)2. S2ORC contains over 136 million academic papers, each of which contains publication year, abstract text, cited paper’s data, ACL ID, and field of study. We used the abstracts of papers that are published at *ACL conferences (ACL ID is not “None”) from 2006 to 2019 for ACL dataset. For CS dataset, we used the abstracts of papers where the field of study includes “Computer Science” and are published from 2006 to 2019. We used the top 40,000 papers w.r.t. the number of citations for CS dataset.
1https://github.com/miyamotononno/DSNTM 2https://github.com/allenai/s2orc
The papers were randomly splitted into 3:1:1 ratio for training, validation, and evaluation. We also filtered out stop words, i.e., words with a document frequency of 70% or above, words appearing in less than ten documents, numbers, punctuation marks, and stop words used in Dieng et al. (2019). The papers published in two consecutive years were grouped into a single time step so that each time step contained a sufficient number of papers. For example, papers published between 2006 and 2007 were grouped into t=1. The statistics of the datasets are summarized in Table 1.
Baseline Methods As baseline methods, we measured the performance of ETM (Dieng et al., 2020) and D-ETM (Dieng et al., 2019) by using the published code of ETM3 and D-ETM4. To evaluate the effectiveness of the self-attention mechanism, we also compared DSNTM that adopts a linear layer instead of the self-attention as shown in Eq. (22). We denote it by “DSNTM w/o self-attention.”
Implementation Details The number of topics was set to 20 for all models and kept constant over time for fair comparison with the baseline models. Hyperparameters of each model were tuned based on the validation perplexity in ACL dataset. Further details are provided in Appendix A.

heading: 6.2 Experimental Results
text: We quantitatively evaluated the performance of topic models using the following three criteria. We run each model eight times and show the average performance and its 95% confidence interval in Table 2 and 3. Lower is better for perplexity, while higher is better for coherence and diversity.
Perplexity We used perplexity (Rosen-Zvi et al., 2004) to evaluate the generalization ability of topic models as a generative model. It measures the
3https://github.com/adjidieng/ETM 4https://github.com/adjidieng/DETM
ability to predict words in unseen documents. Perplexity is computed as follows:
Perplexity = exp ( − ∑D
d=1 log p(wd)∑D d=1Nd
) (37)
where Nd is the number of words in the test document d. As computing p(wd) is intractable, we calculated perplexity using ELBO following Miao et al. (2017); Srivastava and Sutton (2017).
Across the two datasets, our DSNTM achieved a lower perplexity than the baseline models. DSNTM outperformed D-ETM by a large margin specifically for CS dataset. In addition, DSNTM with citation regularization outperformed DSNTM, indicating that citation information contributed to the generalization ability of the topic model.
Coherence We measured topic coherence by calculating the average pointwise mutual information (Mimno et al., 2011) to assess the interpretability of topics. Specifically, we used the normalized pointwise mutual information (NPMI; Lau et al., 2014) of the two words included in the top 10 most likely words of the topic k.
Coherence
= 1
K
K∑
k
1
45
10∑
i=1
10∑
j=i+1
NPMI(w (k) i ,w (k) j ) (38)
NPMI is calculated using the following formula:
NPMI(wi, wj) = log
P (wi,wj) P (wi)P (wj)
logP (wi, wj) (39)
where P (wi, wj) is the probability where words wi and wj co-occurs in a document, and P (wi) is the marginal probability of word wi.
Our DSNTM significantly outperformed ETM and DSNTM without self-attention, while achieving a slightly lower score than D-ETM. However, the citation regularization let DSNTM outperform D-ETM. This result demonstrates that the topic interpretability is sufficiently ensured by the citation regularization.
Diversity We calculated the percentage of unique words in the top 25 frequent words of all topics to measure the diversity of topics following Dieng et al. (2020, 2019).
Diversity= Nu 25K
(40)
where Nu denotes the number of unique words that appear in all topics. Both models achieved competitive scores with the baseline models. This result indicates that our models improves the topic quality, while ensuring sufficient topic diversity.

heading: 7 Discussion

heading: 7.1 Visualization of Topic Transition
text: We discuss that the attention weights capture academic topics merging and branching processes.
Fig. 4 presents an example of topic merging on CS dataset. We show a topic about motion tracking in 2018-2019 (i.e., citing topic) and the two most influential topics on its emergence with respect to the attention weights in 2016-2017 (i.e.,
cited topic). Each cited topic represents motion tracking and reinforcement learning. To investigate the validity of this topic merging, we checked the citation relations between the top 50 papers w.r.t. the citing topic’s paper proportion and the top 50 papers w.r.t. the cited topic’s paper proportion using the test dataset. While many papers on the citing topic refer to papers on motion tracking in previous years, some papers refer to papers on reinforcement learning. As reinforcement learning is used as the learning method of a tracker to achieve a light computation and satisfactory tracking accuracy for object tracking, the topic of reinforcement learning greatly influences the topic of object tracking. The attention weights reveal the merging process of academic topics.
Subsequently, we present an example of topic branching using ACL dataset (Fig. 5). We show a topic about machine translation in 2014-2015 (i.e., cited topic) and three subsequent topics that are most highly influenced by the cited topic (i.e., citing topic). Each citing topic describes machine translation (2016-2017) and neural network (2016- 2017 and 2018-2019). We assessed the validity of this branching in the same manner as topic merging. As of 2014-2015, statistical machine translation (SMT) was predominant, whereas neural machine translation (NMT) was a nascent area in machine translation research. After 2016, NMT was intensively studied by incorporating SMT knowledge of SMT, while NMT models were imported into other text generation tasks (e.g., summarization). This trend induced the topics on neural network in 2016- 2019. DSNTM successfully captures such topic branching processes in the academic literature.
Finally, we present an overview of the topic transition process using ACL dataset (Fig. 6). The topics in the first, second, and third rows represent graph, neural network, and social media, respectively. We can follow the prevalence of the neural

heading: 2016-2017 2018-20192014-2015
text: network techniques to other research areas, such as graphs and social media, by observing the frequent words in the topic and attention weights. DSNTM enables us to grasp the current trends in the research area without following the citations of articles.

heading: 7.2 Prediction of Emerging Topics
text: In this section, we discuss the predictive performance of emerging topics using DSNTM on CS dataset. We trained DSNTM on the papers in 2006- 2017 and predicted topics in 2018-2019 by computing the posterior word distribution for each topic using the self-attention mechanism (Eq. (6), (13), and (14)). We then prepared another model trained on the papers from 2006-2019. The topics inferred by this model are regarded as a proxy for the ground truth of predicted topics. We discuss the quality of the predicted topics by comparing the inferred topics in 2018-2019 across two models.
To measure the prediction performance, we calculated the minimum KL divergence between the word distribution of the predicted topics and ground truth topics: ∑K i=1minj DKL[β pred i ,β truth j ]/K. This value measures the difference between the predicted topics and their nearest ground truth topics.
We computed this value for topics in 2018-2019 and compared it with the average value computed for topics in 2006-2017, which provides the baseline of the predictive performance.
We show that the average KL divergence for 2018-2019 (i.e., predictive performance) is 5.33, whereas that for 2006-2017 (i.e., baseline) is 5.24. This result indicates that the predicted topics are sufficiently close to the ground truth topics. Although further studies are needed, this result suggests that DSNTM can potentially predict emerging topics in the academic literature.

heading: 8 Conclusion
text: In this study, we proposed a novel dynamicstructured neural topic model, DSNTM, which captures dependencies among topics using the selfattention mechanism. We also introduced a citation regularizer, which induces the attention weights to correspond to citation relations.
Experimental results demonstrated that DNSTM outperforms previous dynamic topic models regarding perplexity and coherence while maintaining sufficient diversity across topics. In addition, DSNTM can identify the process of topic merging and branching while showing the potential to predict emerging topics. We expect that DSNTM will make it easier for non-specialists to keep track of the evolution of topics in a given research area without retracing the citations of copious articles and assist their search for a novel topic.
Limitations
As a limitation of the modeling assumption, DSNTM assumes that the number of topics is constant over time; however, this assumption is inappropriate for some time-series documents, such as scientific papers. As the number of scientific papers is increasing annually, increasing the number of topics over time would be appropriate for modeling the time-series evolution of academic literature.
We used the abstracts of the papers as text, and the attention was computed using textual information. However, citations mainly appear in the body text when a paper cites other papers. Therefore, there might be a discrepancy between the attention among topics and the citation relation among papers because the attention cannot not consider information in the body text. In future work, it would be desirable to evaluate our model using a corpus containing the body text of the papers.
Generally, topic models sometimes infer the incorrect information about topics, such as the frequent words appearing in topics, the topic proportion in each document, and the dependencies among topics. It would be the potential risk to induce the misunderstanding of users.
Ethics Statement
Our study complies with the ACL Ethics Policy. We used S2ORC (Lo et al., 2020, CC BY-NC 4.0), PyTorch (Paszke et al., 2019, BSD-style license) as scientific artifacts. Our study is conducted under the licenses and terms of the scientific artifacts. S2ORC is a collection of academic papers and generally does not contain any information that uniquely identifies individual people or offensive content. We did not use the author’s information in our experiments.

heading: Acknowledgements
text: We would like to thank the anonymous reviewers for their valuable feedback. This work was supported by NEDO JPNP20006, JST ACT-X JPMJAX1904, JST CREST JPMJCR21D1, Japan.

heading: B Computing the Influence Among Topics
text: In Fig. 5, we do not directly use the attention weights to represent how much a past topic influences the emergence of new topics. This section describes its reason and how to calculate the influence of a topic on emerging topics.
We assume that the influence of a topic zj on the emergence of topic zi can be represented by the probability where zi emerges given zj . From Bayes’ theorem, we can calculate its probability as follows:
p(zi|zj) = p(zj |zi)p(zi)
p(zj) ∝ p(zj |zi)p(zi) (41)
where p(zj) can be ignored because it is constant regardless of zi. p(zj |zi) is represented by the attention weight from zi to zj , denoted as azi→zj . p(zi) indicates the marginal probability where topic zi appears across all documents, which is calculated by the sum of its topic proportions across all documents.
p(zi) = ∑
d
p(zi|d)p(d)
= ∑
d
θ (zi) d
(42)
where we assume that p(d) is uniformly distributed. Thus, we can obtain probability p(zi|zj) as follows:
vi,j = azi→zj ∑
d
θ (zi) d (43)
p(zi|zj) = vi,j∑ j vi,j
(44)
Therefore, we calculate the influence of a topic zj on the emergence of topic zi by considering the marginal probability of topic zi.
5https://github.com/adjidieng/ETM 6https://github.com/jhlau/topic_
interpretability
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Section "Limitations"
7 A2. Did you discuss any potential risks of your work? Section "Limitations"

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Abstract, Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 6,7, Appendix A

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section 6.1, Appendix A
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? No response.

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Section "Ethics Statement"

heading: 3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any
text: information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Section "Ethical Statement"

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Section 6.1
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 6.1
C 3 Did you run computational experiments? Section 6, 7

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Appendix A
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Section 6.1, Appendix A

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Appendix A

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Appendix A
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: In future work, it would be desirable to evaluate our model using a corpus containing the body text of the papers. Generally, topic models sometimes infer the incorrect information about topics, such as the frequent words appearing in topics, the topic proportion in each document, and the dependencies among topics. It would be the potential risk to induce the misunderstanding of users