# A Method for Constructing Supervised Time Topic Model Based on Variational Autoencoder

Authors: Zhinan Gou; Yan Li; Zheng Huo
Journal: Scientific Programming
 (2021)

---

## Abstract

Topic modeling is a probabilistic generation model to find the representative topic of a document and has been successfully applied to various document-related tasks in recent years. Especially in the supervised topic model and time topic model, many methods have achieved some success. The supervised topic model can learn topics from documents annotated with multiple labels and the time topic model can learn topics that evolve over time in a sequentially organized corpus. However, there are some documents with multiple labels and time-stamped in reality, which need to construct a supervised time topic model to achieve document-related tasks. There are few research papers on the supervised time topic model. To solve this problem, we propose a method for constructing a supervised time topic model. By analysing the generative process of the supervised topic model and time topic model, respectively, we introduce the construction process of the supervised time topic model based on variational autoencoder in detail and conduct preliminary experiments. Experimental results demonstrate that the supervised time topic model outperforms several state-of-the-art topic models.


---

## Body

## 1. Introduction

Nowadays many kinds of text information like news, blogs, books, and social network accompany the daily lives of people. Some traditional methods are difficult to get effective information from an increasingly large amount of data. The probabilistic topic model is a new technology that can help people to organize, index, search, and browse these kinds of large data automatically.Latent Dirichlet allocation (LDA) [1] is a classical topic model to find the representative topic of a document. During the past decade, LDA has been successfully applied to various document-related tasks, such as text classification [2], clustering [3], and summarization [4]. However, as a static topic model, LDA has two limitations.Firstly, the number of topics is hard to determine in the static topic model. To select the best number of topics, most of the methods use the lowest perplexity or the largest likelihood estimate value of the topic model by comparing different numbers of topics [1, 5, 6]. In fact, these methods lead to the fact that the latent topic content is difficult to interpret. Secondly, the text information can be exchanged in the static topic model [1]. Moreover, the static topic model assumes that the text information is unordered with each other [7]. This simplified assumption is inappropriate and unrealistic [8].In order to overcome these limitations, the supervised topic model and time topic model are proposed, respectively. In recent years, methods of the supervised topic model and time topic model have a variety of applications.The objective of the supervised topic model is to learn topics from documents annotated with multiple labels. To the best of our knowledge, labeled LDA [9] is the classical supervised topic model that matches the multiple topics to the labels in the document. The number of topics is determined by the metadata of the document (such as labels), and topic terms have a better way to interpret topics [10]. In addition, the supervised topic model supports a variety of applications, such as social event analysis [11], abnormal event detection [12], document classification [13], and tag recommendation [14].The objective of the time topic model is to construct a topic model that evolves over time in a sequentially organized corpus. To the best of our knowledge, the dynamic topic model is the first time topic model that captures the evolution of topics in a sequentially organized corpus of documents [7]. Based on the time topic model, a series of applications are researched, for example, dynamic features extraction [15], automated behaviour analysis [16], travel recommendation [17], and tracking urban geotopics [18].However, there are some documents with multiple labels and time-stamped in reality, for example, a scientific paper has keywords and is time-stamped. Park et al. proposed a similar supervised time topic model, the main idea of which is on generating the numerical time-series variables as supervised metadata that is a single topic of each time slice [19]. This model is not considered as a supervised time topic model which can learn a topic from documents with multiple labels and which are time-stamped and apply the topic to various document-related tasks. As a result, it is necessary to propose a method to construct a supervised time topic model. To the best of our knowledge, this is the first work of constructing a supervised time topic model. The contributions of this paper are as follows.We reveal and discuss the limitations of the main current relevant works on the topic model, and thus to construct a supervised time topic model is very necessary. We propose a method for constructing a supervised time topic model based on variational autoencoders, denoted by ST-TM, which is designed to deal with documents that have multiple labels and are time-stamped. The reasoning and constructing process of ST-TM is presented in detail. For a preliminary evaluation, we compare ST-TM with the state-of-the-art methods in the experiments. The results show that our proposed method is more effective.The rest of the paper is organized as follows: we firstly review related research works. Secondly, the method for constructing a supervised time topic model based on variational autoencoders is described in detail. Then, the experiments of our proposed method are conducted. Finally, this paper is concluded.


## 2. Related Works

In this section, we review some representative works on the supervised topic models and time topic models in detail, respectively. Based on analysing the limitations of related works, we present a supervised time topic model to address these limitations.


### 2.1. Supervised Topic Model

To the best of our knowledge, supervised latent Dirichlet allocation is the first supervised topic model that adds to a topic associated with each document and uses variational methods to handle intractable posterior expectations [20]. Since then, several supervised topic models have been proposed, for example, discriminative LDA [21] and maximum entropy discrimination LDA [22]. The above methods train only a single topic for each document.It is well known that a document with only a single topic is inappropriate. For example, a document on social education includes both social and educational topics. Based on this fact, some supervised topic models with multiple topics are proposed. Labeled LDA matches the multiple topics to the labels in the document [9]. The number of topics is determined by the metadata of the document (such as labels), and topic terms have a better way to interpret topics [10]. Partially labeled LDA learns latent topic structure within the scope of observed, human interpretable labels [23]. Nonparametric labeled LDA uses the Dirichlet process with mixed random measures as a base distribution of the hierarchical Dirichlet process framework [24]. Dependency-LDA further considers the label frequency and label dependency observed in the training data for constructing the supervised topic model [2].We review Labeled LDA (L-LDA) that is representative work on the supervised topic model. In order to contain the supervision, L-LDA applies a 1 : 1 correspondence between topics and labels. In addition to labels, keywords of scientific papers and categories of news are also considered as topics [10]. L-LDA is a probabilistic graphical model that describes a process for generating a labeled document collection. The graphical model of L-LDA is shown in Figure 1.Figure 1
The graphical model of L-LDA.In Figure1, two observable parts of the grey fill c and w are explicit variables that indicate labels and terms, respectively. Nodes with no fill are implicit variables and unobservable. Unlike LDA, both the label set c and the topic prior α influence the topic mixture θ. L-LDA assumes that each document d is restricted to a multinomial distribution θd over labels which are part of the corpus. Each label is represented as a topic c that is a multinomial distribution Φc over the terms. The generative process of L-LDA is shown in Table 1. Different from LDA, the whole generative process of L-LDA contains topic constraint. The more detailed descriptions of L-LDA are presented in literature [9].Table 1
The generative process of L-LDA.
StepsDescription1For each topicc∈C:2Generate the multinomial topic distributions over vocabularyΦc∼Dirichletβc3End for4For each documentd∈D:5Generate the multinomial topic distributions over documentθd∼Dirichletαd according to the label set of each document d6For each termWd,n∈d:7Generate topicZd,n∼Multinomialθd8Generate termWd,n∼MultinomialΦZd,n9End for10End for


### 2.2. Time Topic Model

The time topic model is developed to analyse the time evolution of topics in large document collections. To the best of our knowledge, the dynamic topic model is the first time topic model that captures the evolution of topics in a sequentially organized corpus of documents [7]. In order to simplify the reasoning procedure of the time topic model, the method uses a variational autoencoder to construct the time topic model [8]. To build cross-lingual tools, a multilingual dynamic topic model is proposed that can capture cross-lingual topics that evolve across time [25]. We review the dynamic topic model (DTM) that is representative work on time topic model. DTM supposes that the data is divided by time slice, for example, by decades. The graphical model of DTM is shown in Figure 2.Figure 2
The graphical model of DTM.In Figure2, β is the vector of natural parameters for topic evolving over time and α is the vector of mean parameters of the logistic normal distribution for the topic proportions. However, the Dirichlet is not amenable to the sequential model. Literature [7] chains the vectors of parameters α and β in a state-space model that evolves with Gaussian noise, respectively. By chaining together topics and topic proportion distributions, DTM has sequentially tied a collection of topic models. The generative process for slice t of a sequential corpus is shown in Table 2. The whole process of DTM is restrained by time slice t; the more detailed descriptions of DTM are presented in the literature [7].Table 2
The generative process of DTM.
StepsDescription1Draw topicsβt|βt−1∼Nβt−1,σ2I2Drawαt|αt−1∼Nαt−1,δ2I3For each documentd∈Dt:4Generate the multinomial topic distributions over documentθt,d∼Nαt,a2I5For each termWt,d,n∈d:6Generate topicZd,n∼Multinomialθt,d7Generate termWt,d,n∼Multinomialβt,Zd,n8End for9End forUsing the above methods, the supervised topic model and time topic model are implemented for mining topics. However, some documents have multiple labels and are time-stamped in reality; for example, the scientific paper has keywords and is time-stamped. And there are few research papers on the supervised time topic model to learn topics for these documents. As a result, it is necessary to propose a method to construct a supervised time topic model for documents with multiple labels and time-stamped.


## 2.1. Supervised Topic Model

To the best of our knowledge, supervised latent Dirichlet allocation is the first supervised topic model that adds to a topic associated with each document and uses variational methods to handle intractable posterior expectations [20]. Since then, several supervised topic models have been proposed, for example, discriminative LDA [21] and maximum entropy discrimination LDA [22]. The above methods train only a single topic for each document.It is well known that a document with only a single topic is inappropriate. For example, a document on social education includes both social and educational topics. Based on this fact, some supervised topic models with multiple topics are proposed. Labeled LDA matches the multiple topics to the labels in the document [9]. The number of topics is determined by the metadata of the document (such as labels), and topic terms have a better way to interpret topics [10]. Partially labeled LDA learns latent topic structure within the scope of observed, human interpretable labels [23]. Nonparametric labeled LDA uses the Dirichlet process with mixed random measures as a base distribution of the hierarchical Dirichlet process framework [24]. Dependency-LDA further considers the label frequency and label dependency observed in the training data for constructing the supervised topic model [2].We review Labeled LDA (L-LDA) that is representative work on the supervised topic model. In order to contain the supervision, L-LDA applies a 1 : 1 correspondence between topics and labels. In addition to labels, keywords of scientific papers and categories of news are also considered as topics [10]. L-LDA is a probabilistic graphical model that describes a process for generating a labeled document collection. The graphical model of L-LDA is shown in Figure 1.Figure 1
The graphical model of L-LDA.In Figure1, two observable parts of the grey fill c and w are explicit variables that indicate labels and terms, respectively. Nodes with no fill are implicit variables and unobservable. Unlike LDA, both the label set c and the topic prior α influence the topic mixture θ. L-LDA assumes that each document d is restricted to a multinomial distribution θd over labels which are part of the corpus. Each label is represented as a topic c that is a multinomial distribution Φc over the terms. The generative process of L-LDA is shown in Table 1. Different from LDA, the whole generative process of L-LDA contains topic constraint. The more detailed descriptions of L-LDA are presented in literature [9].Table 1
The generative process of L-LDA.
StepsDescription1For each topicc∈C:2Generate the multinomial topic distributions over vocabularyΦc∼Dirichletβc3End for4For each documentd∈D:5Generate the multinomial topic distributions over documentθd∼Dirichletαd according to the label set of each document d6For each termWd,n∈d:7Generate topicZd,n∼Multinomialθd8Generate termWd,n∼MultinomialΦZd,n9End for10End for


## 2.2. Time Topic Model

The time topic model is developed to analyse the time evolution of topics in large document collections. To the best of our knowledge, the dynamic topic model is the first time topic model that captures the evolution of topics in a sequentially organized corpus of documents [7]. In order to simplify the reasoning procedure of the time topic model, the method uses a variational autoencoder to construct the time topic model [8]. To build cross-lingual tools, a multilingual dynamic topic model is proposed that can capture cross-lingual topics that evolve across time [25]. We review the dynamic topic model (DTM) that is representative work on time topic model. DTM supposes that the data is divided by time slice, for example, by decades. The graphical model of DTM is shown in Figure 2.Figure 2
The graphical model of DTM.In Figure2, β is the vector of natural parameters for topic evolving over time and α is the vector of mean parameters of the logistic normal distribution for the topic proportions. However, the Dirichlet is not amenable to the sequential model. Literature [7] chains the vectors of parameters α and β in a state-space model that evolves with Gaussian noise, respectively. By chaining together topics and topic proportion distributions, DTM has sequentially tied a collection of topic models. The generative process for slice t of a sequential corpus is shown in Table 2. The whole process of DTM is restrained by time slice t; the more detailed descriptions of DTM are presented in the literature [7].Table 2
The generative process of DTM.
StepsDescription1Draw topicsβt|βt−1∼Nβt−1,σ2I2Drawαt|αt−1∼Nαt−1,δ2I3For each documentd∈Dt:4Generate the multinomial topic distributions over documentθt,d∼Nαt,a2I5For each termWt,d,n∈d:6Generate topicZd,n∼Multinomialθt,d7Generate termWt,d,n∼Multinomialβt,Zd,n8End for9End forUsing the above methods, the supervised topic model and time topic model are implemented for mining topics. However, some documents have multiple labels and are time-stamped in reality; for example, the scientific paper has keywords and is time-stamped. And there are few research papers on the supervised time topic model to learn topics for these documents. As a result, it is necessary to propose a method to construct a supervised time topic model for documents with multiple labels and time-stamped.


## 3. Methods

In order to make the proposed method for constructing a supervised time topic model easier to describe and understand, we summarize some major notations needed in our formulation in Table3.Table 3
Notations description.
NotationDescriptiontTime slicetcThe supervised topicCThe set of topics in a text corpusdDocumentWTermnThe number of terms in documentdβtThe word-topic distribution parameters in time slicetθtThe document-specific topic parameters in time slicetzThe vector of words-topicwtInput documents in time slicetμtThe mean parameters for inputting documents in time slicetσtThe covariance parameters for inputting documents in time slicetwt′Generate documents in time slicetμt′The mean parameters for generating documents in time slicetσt′The covariance parameters for generating documents in time slicetThe method of constructing a supervised time topic model is based on the generative process of the supervised topic model and time topic model. The difficulty is the reasoning of the loss function. In this section, we describe the method for constructing a supervised time topic model based on variational autoencoders in detail and denoted by ST-TM; the graphical model of ST-TM is shown in Figure3.Figure 3
The graphical model of ST-TM.The generative process of ST-TM is different from L-LDA and DTM by dividing time slices and having the supervision for each time slice, respectively. Moreover, unlike DTM, ST-TM removes transitive dependencies forα and β so that the reasoning process of the model can further simplify. The generative process of ST-TM is shown in Table 4.Table 4
The generative process of ST-TM.
StepsDescription1For each topicc∈C in time slice t:2Generate the multinomial topic distributions over vocabularyβt,c according to βt∼Nμt,σ2I3End for4Drawαt∼Nμt,σ2I5For each documentd∈Dt:6Generate the multinomial topic distributions over documentθt,d∼Nθt′|αt,a2I according to θt∼Nαt,a2I and the constraint matrix of document-topic θt′7For each termWt,d,n∈d:8Generate topicZd,n∼Multinomialθt,d9Generate termWt,d,n∼Multinomialβt,Zd,n10End for11End forThe implementation of ST-TM depends on a variational autoencoder that is a deep learning technique for learning latent representations. The variational autoencoder is composed of an encoder and decoder. In time slicet, the encoder is to generate a variational approximate posterior distribution qϕθt|wt of the document-topic. Decoder estimates the optimal generation probability pwt′|θt,βt with qϕθt|wt known. The network structure of the variational autoencoder of the topic model is shown in Figure 4.Figure 4
The network structure of variational autoencoder of the topic model.In Figure4, the model assumes qθ|w obeys diagonalization of the covariance matrices of the Gaussians distribution. The encoder trains the potential mean and variance for the vector of term-topic z. In order to generate documents w′ that are as close to the input documents w, the decoder estimates the better probabilistic model pw′|θ,z according to the known qθ|w. The generation process of θ needs to use backpropagation to calculate the gradient of the error function. Generally, we want to minimize an expected cost by using gradient descent that requires computing the gradients. The generation process of θ adopts the reparametrization trick that is used in computing the gradients. The graphical model of the reparametrization trick is shown in Figure 5.Figure 5
The graphical model of the reparametrization trick.In Figure5, the random variable z∼qz|w (at left) can be reparameterized as a Gaussian function and a random variable ε. More specifically, the reparametrization trick in this paper involves a function of a standard Gaussian variable ε∼N0,I that is used to rewrite z∼Nμ,σ, such that z=μ+σ⊙ε. Therefore, μ and σ only involve linear operation that can be optimized by stochastic gradient descent algorithm easily.In this paper, the encoder has a 2-layer neural network to generate the variational approximate posterior distributionqϕθt|wt of document-topic. The decoder estimates the better probabilistic model pwt′|θt,βt according to the known qϕθt|wt. According to the generation process of ST-TM, the marginal likelihood function of generating documents w′ as a variational target is shown in equations (1) and (2):(1)pwt′|αt,βt,θt′=∫∏n=1N∑zt,nKpwt,n′|zt,n,βtpzt,n|θt⋅pθt|αt;θt′dθt,(2)pwt′|αt,βt,θt′=∫∏n=1Npwt,n′|θt,βtpθt|αt;θt′dθt.Unlike Table3, n is the number of documents in time slice t. The variational functions qϕθt|wt,θt′ are introduced for calculating logpwt′|αt,βt,θt′. According to Jensen’s inequality, the marginal logarithmic likelihood function lwt′,αt,βt,θt′ of generated document wt′ is as follows:(3)lwt′,αt,βt,θt′=log∫θtqϕθt|wt,θt′pwt′|θt,βt∗pθt|αt;θt′qϕθt|wt,θt′dθt,lwt′,αt,βt,θt′≥∫θtqϕθt|wt,θt′∗logpwt′|θt,βtdθt+∫θtqϕθt|wt,θt′∗logpθt|αt;θt′qϕθt|wt,θt′dθt,lwt′,αt,βt,θt′≥Eqϕθt|wt,θt′logpwt′|θt,βt−KLqϕθt|wt,θt′pθt|αt;θt′,Eqϕθt|wt,θt′logpwt′|θt,βt−KLqϕθt|wt,θt′pθt|αt;θt′≜ℒq,βt,αt;wt,θt′,where ℒq,βt,αt;wt,θt′ is the variational lower bound for the marginal logarithmic likelihood function lwt′,αt,βt,θt′, lwt′,αt,βt,θt′=ℒq,βt,αt;wt,θt′ if and only if qϕθt|wt,θt′=pθt|αt;θt′. In order to find the best qϕθt|wt,θt′ which should be able to approximate the true posterior distribution pθt|αt;θt′, the most common approximate calculation of lwt′,αt,βt,θt′ uses the Kullback–Leibler divergence (KL-divergence) between qϕθt|wt,θt′ and pθt|αt;θt′. Our aim is to get argminKLqϕθt|wt,θt′pθt|αt;θt′ which is different from the precise KL-divergence lwt′,αt,βt,θt′−ℒq,βt,αt;wt,θt′. Details are shown as follows:(4)argminKLqϕθt|wt,θt′pθt|αt;θt′=argminlwt′,αt,βt,θt′−ℒq,βt,αt;wt,θt′,(5)argminKLqϕθt|wt,θt′pθt|αt;θt′=argmaxℒq,βt,αt;wt,θt′,(6)argminKLqϕθt|wt,θt′pθt|αt;θt′=argmaxEqϕθt|wt,θt′logpwt′|θt,βt−KLqϕθt|wt,θt′pθt|αt;θt′,where the first part is the expectation of reconstruction error and the second part is the KL-divergence of the approximate posterior distribution from the true posterior distribution. In t time slice, wt′ has N independent samples (wt1′, wt2′, …, wtN′). As the first part of equation (6), Eqϕθt|wt,θt′logpwt′|θt,βt is generated by each sample with one point and then Eqϕθt|wti,θt′logpwti′|θt,βt≈logpwti′|θt,βt and obeys multivariate Bernoulli distribution. The expectation of reconstruction error can be measured by the cross entropy cost function, which is often used to measure the difference between the predicted value and actual value. In addition, the decrease of the learning rate of the loss function can be avoided in the process of gradient descent [26]. The logarithmic likelihood of pwt′|θt,βt uses the negative cross entropy denoted by L1, where L1=−wt∗logpwt′|θt,βt.The state variablesβt remove transitive dependency in order to simplify the complex reasoning process. Gaussian sampling is used to initialize the state variables βt denoted by L2, then L2=βt∼Nμt,σ2I. In ST-TM, the encoder is to generate a variational approximate posterior distribution qϕθt|wt,θt′ of the document-topic. We assume qϕθt|wt,θt′ obeys diagonalization of the covariance matrices of the Gaussians distribution. Then, logqϕθt|wt,θt′=logNwt;μt′,σt′⋅θt′, where the mean μt′ and the covariance σt′ are generated by the neural network. Following the above reparametrization trick, we rewrite qϕ with an auxiliary parameter ε that makes θt=σμt′+σt′1/2∗ε⋅θt′. We apply this trick to obtain an estimator of the variational lower bound [27].For the true posterior distribution,θt obeys a logarithmic Gaussian distribution with the mean μt and the variance σt, then logpθt|wt,θt′;αt=logNwt;μt,σt⋅θt′, where the covariance σt is the hyperparameters, and the mean μt=αt that is initialized by the Gaussian random sampling αt∼N1,σt2I to take the place of αt=1. According to the probability density function of multivariate Gaussian distribution, the second part of (6) denoted by Lkl is inferred as follows:(7)KLqϕθt|wt,θt′pθt|wt,θt′;αt=Eqϕlogqϕθt|wt,θt′−logpθt|wt,θt′;αt=12Eqϕ−logσt′−x−μt′Tσt′−1x−μt′+logσt+x−μtTσt−1x−μt=12logσtσt′+12Eqϕ−x−μt′Tσt′−1x−μt′+x−μtTσt−1x−μt=12logσtσt′+12Eqϕ−trσt′−1x−μt′x−μt′T+trσt−1x−μtx−μtT=12logσtσt′+12Eqϕ−trσt′−1x−μt′x−μt′T+12Eqϕtrσt−1x−μtx−μtT=12logσtσt′−12trEqϕσt′−1x−μt′x−μt′T+12trEqϕσt−1x−μtx−μtT=12logσtσt′−12trσt′−1Eqϕx−μt′x−μt′T+12trEqϕσt−1xxT−μtxT−xμtT+μtμtT=12logσtσt′−12trσt′−1σt′+12trσt−1EqϕxxT−μtxT−xμtT+μtμtT=12logσtσt′−12θt′+12trσt−1σt′+μt′μt′T−μtμt′T−μt′μtT+μtμtT=12logσtσt′−θt′+trσt−1σt′+trσt−1μt′μt′T−μtμt′T−μt′μtT+μtμtT=12logσtσt′−θt′+trσt−1σt′+trσt−1μt′μt′T−σt−1μtμt′T−σt−1μt′μtT+σt−1μtμtT=12logσtσt′−θt′+trσt−1σt′+trμt′Tσt−1μt′−2μt′Tσt−1μt+μtTσt−1μt=12logσtσt′−θt′+trσt−1σt′+μt−μt′Tσt−1μt−μt′,where θt′ is the column number of document-topic constraint matrix that is the number of topics in time slice t. In conclusion, the final variational objective can be represented by loss function ℒΘ as follows:(8)ℒΘ=argmaxL1+L2−Lkl,ℒΘ=argmin−L1−L2+Lkl,ℒΘ=argminwt∗logpwt′|θt,βt+βt+Lkl,ℒΘ=argmin∑iDwt,i∗logπβtπμt′+σt′∗ε⋅θt′+βt+12trσt−1σt′+μt−μt′Tσt−1μt−μt′−θt′+logσtσt′,where πβt=expβt,w/∑wexpβt,w, πμt′+σt′∗ε=expμt,d′+σt,d′∗ε/∑dexpμt,d′+σt,d′∗ε.


## 4. Experimental Results and Analysis

In this section, we conduct experiments to verify the efficiency of the proposed method on a real dataset. Our approach is compared with the state-of-the-art topic models.


### 4.1. Experiment Environment and Dataset

The experiments were executed on a personal computer with an AMD FX-Series CPU FX-8350 @4.0 GHz, the processor having eight physical computing cores, 24 GB DDR3 RAM memory @1600 MHz. The machine was running on Windows 7 (64 bits) operating system, TensorFlow 1.4.0 with CPU support only, and Python 3.6.To demonstrate the efficiency of our proposed approach, we use the paper corpus of SIGIR (International ACM SIGIR Conference on Research and Development in Information Retrieval) as the dataset. The main purpose of SIGIR is to show new technologies and achievements related to information retrieval. The dataset is constructed by abstracts of papers that are from the 2018, 2013, and 2009 annual conferences. Experimental data consisting of 564 papers with 76 topics and about 90000 words can be downloaded from the official SIGIR website. We divided the dataset into 3 time slices by year.Our neural networks structure is composed of an encoder and decoder. The encoder has one input layer and two hidden layers, and the decoder has two hidden layers and one output layer. Input layer and output layer have the same number of neurons as the number of papers, which changes in different time slices. Every hidden layer has 100 hidden neurons according to the existing experimental conclusion [8]. Moreover, the learning rate is 0.002 and the number of iterations is 300 that can achieve convergence for our dataset.


### 4.2. Evaluation Metrics

It is difficult to choose the appropriate evaluation method of the topic model. Although the traditional evaluating method uses perplexity to evaluate the qualitative of topics, it is generally not well suited to express the semantic property of topic terms [28]. Lau et al. [29] use Normalized Pointwise Mutual Information (NPMI) to evaluate the qualitative of topics, and NPMI is close to the ordinary understanding and judgment of people.The experiment is evaluated byNPMIk that can express the coherence of topic k, and NPMIk is given as(9)NPMIk=∑i=1M−1∑j=i+1Mlogpwi,wj/pwipwj−logpwi,wj,where M is the number of topic terms in topic k. wi and wj are the ith topic term and the jth topic term, respectively. pwi, pwj, and pwi,wj are the probability of the ith topic term, the probability of the jth topic term, and the cooccurrence probability of the ith and the jth topic term in topic k. Experiment results depend on the average topic coherence of K topics at last. The higher value the average of NPMI is, the better the topic model is.


### 4.3. Baseline Methods

To validate the effectiveness of our method, three state-of-the-art topic model methods have been compared with ST-TM. The first baseline is collapsed Gibbs sampling on the LDA model denoted by SDTM Gibbs [30]. The second one is the static topic model based on autoencoding variational inference denoted by LDA VAE [31]. The last one uses a variational autoencoder to construct a time topic model denoted by DTM VAE [8]. The details of all methods are provided in Table 5. All baseline methods are conducted by the original paper of the corresponding methods.Table 5
Details of all methods.
MethodVariational autoencoderTime slicesSupervisionSDTM GibbsNoYesYesLDA VAEYesNoNoDTM VAEYesYesNoST-TMYesYesYes


### 4.4. Comparison with Baseline Methods

We conduct a preliminary experiment. The convergence process of each method is shown in Figure6. The convergence of Gibbs sampling depends on the perplexity value which is a measurement of stability for the topic model. The loss function is used to determine the convergence of other methods that are based on a variational autoencoder.Figure 6
The convergence process of all methods. (a) Gibbs sampling. (b) Other methods.
(a)(b)All methods on NPMI value are shown in Figure7. Our method has the highest NPMI value at 0.316, while the other methods are at 0.083, 0.169, and 0.299, respectively. Comparing to the traditional Gibbs sampling method of constructing the topic model, the variational autoencoder method is more efficient. The supervised time topic model outperforms the static topic model and unsupervised time topic model on the semantic property of topic terms. The result of the preliminary experiment shows that ST-TM has excellent performance at interpretation of the topic.Figure 7
Comparison of all methods on NPMI.


### 4.5. Discussion

The proposed method for constructing the supervised time topic model is verified through the public dataset, and the experimental results show that the average of NPMI is improved with time slices and supervision. The proposed method is easier to operate compared with the traditional Gibbs sampling method of constructing the topic model [30] and can achieve a good result. In addition, the proposed method is based on the basic structure of the variational autoencoder method [31] and introduces time slices and supervision for understanding the supervised time topic model. The largest advantage of this work is in solving the problem of constructing a supervised time topic model for some documents with multiple labels and which are time-stamped to achieve document-related tasks.


## 4.1. Experiment Environment and Dataset

The experiments were executed on a personal computer with an AMD FX-Series CPU FX-8350 @4.0 GHz, the processor having eight physical computing cores, 24 GB DDR3 RAM memory @1600 MHz. The machine was running on Windows 7 (64 bits) operating system, TensorFlow 1.4.0 with CPU support only, and Python 3.6.To demonstrate the efficiency of our proposed approach, we use the paper corpus of SIGIR (International ACM SIGIR Conference on Research and Development in Information Retrieval) as the dataset. The main purpose of SIGIR is to show new technologies and achievements related to information retrieval. The dataset is constructed by abstracts of papers that are from the 2018, 2013, and 2009 annual conferences. Experimental data consisting of 564 papers with 76 topics and about 90000 words can be downloaded from the official SIGIR website. We divided the dataset into 3 time slices by year.Our neural networks structure is composed of an encoder and decoder. The encoder has one input layer and two hidden layers, and the decoder has two hidden layers and one output layer. Input layer and output layer have the same number of neurons as the number of papers, which changes in different time slices. Every hidden layer has 100 hidden neurons according to the existing experimental conclusion [8]. Moreover, the learning rate is 0.002 and the number of iterations is 300 that can achieve convergence for our dataset.


## 4.2. Evaluation Metrics

It is difficult to choose the appropriate evaluation method of the topic model. Although the traditional evaluating method uses perplexity to evaluate the qualitative of topics, it is generally not well suited to express the semantic property of topic terms [28]. Lau et al. [29] use Normalized Pointwise Mutual Information (NPMI) to evaluate the qualitative of topics, and NPMI is close to the ordinary understanding and judgment of people.The experiment is evaluated byNPMIk that can express the coherence of topic k, and NPMIk is given as(9)NPMIk=∑i=1M−1∑j=i+1Mlogpwi,wj/pwipwj−logpwi,wj,where M is the number of topic terms in topic k. wi and wj are the ith topic term and the jth topic term, respectively. pwi, pwj, and pwi,wj are the probability of the ith topic term, the probability of the jth topic term, and the cooccurrence probability of the ith and the jth topic term in topic k. Experiment results depend on the average topic coherence of K topics at last. The higher value the average of NPMI is, the better the topic model is.


## 4.3. Baseline Methods

To validate the effectiveness of our method, three state-of-the-art topic model methods have been compared with ST-TM. The first baseline is collapsed Gibbs sampling on the LDA model denoted by SDTM Gibbs [30]. The second one is the static topic model based on autoencoding variational inference denoted by LDA VAE [31]. The last one uses a variational autoencoder to construct a time topic model denoted by DTM VAE [8]. The details of all methods are provided in Table 5. All baseline methods are conducted by the original paper of the corresponding methods.Table 5
Details of all methods.
MethodVariational autoencoderTime slicesSupervisionSDTM GibbsNoYesYesLDA VAEYesNoNoDTM VAEYesYesNoST-TMYesYesYes


## 4.4. Comparison with Baseline Methods

We conduct a preliminary experiment. The convergence process of each method is shown in Figure6. The convergence of Gibbs sampling depends on the perplexity value which is a measurement of stability for the topic model. The loss function is used to determine the convergence of other methods that are based on a variational autoencoder.Figure 6
The convergence process of all methods. (a) Gibbs sampling. (b) Other methods.
(a)(b)All methods on NPMI value are shown in Figure7. Our method has the highest NPMI value at 0.316, while the other methods are at 0.083, 0.169, and 0.299, respectively. Comparing to the traditional Gibbs sampling method of constructing the topic model, the variational autoencoder method is more efficient. The supervised time topic model outperforms the static topic model and unsupervised time topic model on the semantic property of topic terms. The result of the preliminary experiment shows that ST-TM has excellent performance at interpretation of the topic.Figure 7
Comparison of all methods on NPMI.


## 4.5. Discussion

The proposed method for constructing the supervised time topic model is verified through the public dataset, and the experimental results show that the average of NPMI is improved with time slices and supervision. The proposed method is easier to operate compared with the traditional Gibbs sampling method of constructing the topic model [30] and can achieve a good result. In addition, the proposed method is based on the basic structure of the variational autoencoder method [31] and introduces time slices and supervision for understanding the supervised time topic model. The largest advantage of this work is in solving the problem of constructing a supervised time topic model for some documents with multiple labels and which are time-stamped to achieve document-related tasks.


## 5. Conclusions

In this paper, we propose a method for constructing a supervised time topic model based on variational autoencoders. We reveal and discuss the limitations of the main current relevant works on the supervised topic models and time topic models, respectively. We further present the reasoning and constructing process of the supervised time topic model. Specifically, the graphical model of constructing process of supervised time topic model is proposed. The implementation of the method depends on a variational autoencoder that is composed of an encoder and decoder. An encoder is to generate a variational approximate posterior distribution of the document-topic, and a decoder estimates the optimal generation probability. The process of reasoning of the marginal likelihood function of generating documents as a variational target is presented. In addition, we compare our method with baselines in the experiment for a preliminary evaluation. The results show that our method is more effective for the topic model than baseline methods. The contribution of this work is in solving the problem of constructing a supervised time topic model for some documents with multiple labels and time-stamped to achieve document-related tasks. The relevant theories of the topic model can be conducive for researchers to understand, reason, and use the supervised time topic model.In future work, we will improve the method of constructing a supervised time topic model for further completion, refinement, and adaptation. Furthermore, to achieve better performance, we will also apply our constructed supervised time topic model in information retrieval, recommender systems, text classification, and other fields.



---

*Source: 6623689-2021-02-08.xml*