Venue: NAACL 2018

# Deep Temporal-Recurrent-Replicated-Softmax for

Topical Trends over Time

Pankaj Gupta1,2, Subburam Rajaram1, Hinrich Schutze2, Bernt Andrassy1

###### Abstract

Dynamic topic modeling facilitates the identification of topical trends over time in temporal collections of unstructured documents. We introduce a novel unsupervised neural dynamic topic model named as Recurrent Neural Network-Replicated Softmax Model (RNN-RSM), where the discovered topics at each time influence the topic discovery in the subsequent time steps. We account for the temporal ordering of documents by explicitly modeling a joint distribution of latent topical dependencies over time, using distributional estimators with temporal recurrent connections. Applying RNN-RSM to 19 years of articles on NLP research, we demonstrate that compared to state-of-the art topic models, RNN-RSM shows better generalization, topic interpretation, evolution and trends. We also introduce a metric (named as SPAN) to quantify the capability of dynamic topic model to capture word evolution in topics over time.

## 1 Introduction

Topic Detection and Tracking Allan et al. (1998) is an important area of natural language processing to find topically related ideas that evolve over time in a sequence of text collections and exhibit temporal relationships. The temporal aspects of these collections can present valuable insight into the topical structure of the collections and can be quantified by modeling the dynamics of the underlying topics discovered over time.

**Problem Statement**: We aim to generate temporal topical trends or automatic overview timelines of topics for a time sequence collection of documents. This involves the following three tasks in dynamic topic analysis: **(1)**_Topic Structure Detection_ (TSD): Identifying main topics in the document collection. **(2)**_Topic Evolution Detection_ (TED): Detecting the emergence of a new topic and recognizing how it grows or decays over time Allan (2002). **(3)**_Temporal Topic Characterization_ (TTC): Identifying the characteristics for each of the main topics in order to track the words' usage (_keyword trends_) for a topic over time i.e. _topical trend analysis for word evolution_ (Fig 1, Left).

Probabilistic static topic models, such as Latent Dirichlet Allocation (LDA) Blei et al. (2003) and its variants Wang and McCallum (2006); Hall et al. (2008); Gollapalli and Li (2015) have been investigated to examine the emergence of topics from historical documents. Another variant known as Replicated Softmax (RSM) Hinton and Salakhutdinov (2009) has demonstrated better generalization in log-probability and retrieval, compared to LDA. Prior works Iwata et al. (2010); Pruteanu-Malinici et al. (2010); Saha and Sindhwani (2012); Schein et al. (2016) have investigated Bayesian modeling of topics in time-stamped documents. Particularly, Blei and Lafferty (2006) developed a LDA based dynamic topic model (DTM) to capture the evolution of topics in a time sequence collection of documents; however they do not capture explicitly the topic popularity and usage of specific terms over time. We propose a family of probabilistic time series models with distributional estimators to explicitly model the dynamics of the underlying topics, introducing temporal latent topic dependencies (Fig 1, Right).

To model temporal dependencies in high dimen

Figure 1: (Left): Word Usage over time for Topic (_Word Representation_) in scholarly articles. (Right): RSM-based dynamic topic model with explicit temporal topic dependence

sional sequences, such as polyphonic music, the temporal stack of RBMs (Smolensky, 1986; Hinton, 2002) has been investigated to model complex distributions. The Temporal RBM Taylor et al. (2007); Sutskever and Hinton (2007), Recurrent Temporal RBM (RTRBM) (Sutskever et al., 2009) and RNN-RBM (Boulanger-Lewandowski et al., 2012) show success in modeling the temporal dependencies in such symbolic sequences. In addition, RNNs Gupta et al. (2015a); Vu et al. (2016a, b); Gupta et al. (2016) have been recognized for sentence modeling in natural language tasks. We aspire to build neural dynamic topic model called RNN-RSM to model document collections over time and learn temporal topic correlations.

We consider RSM for TSD and introduce the explicit latent topical dependencies for TED and TTC tasks. Fig 1 illustrates our _motivation_, where temporal ordering in document collection \({\bf\hat{V}}^{(t)}\) at each time step \(t\), is modeled by conditioning the latent topic \({\mathbf{h}}^{(t)}\) on the sequence history of latent topics \({\mathbf{h}}^{(0)}\), ..., \({\mathbf{h}}^{(t-1)}\), accumulated with temporal lag. Each RSM discovers latent topics, where the introduction of a bias term in each RSM via the time-feedback latent topic dependencies enables to explicitly model topic evolution and specific topic term usage over time. The temporal connections and RSM biases allow to convey topical information and model relation among the words, in order to deeply analyze the dynamics of the underlying topics. We demonstrate the applicability of proposed **RNN-RSM** by analyzing 19 years of scientific articles from NLP research.

The _contributions_ in this work are:

**(1)** Introduce an unsupervised neural dynamic topic model based on recurrent neural network and RSMs, named as RNN-RSM to explicitly model discovered latent topics (evolution) and word relations (topic characterization) over time.

**(2)** Demonstrate better generalization (log-probability and time stamp prediction), topic interpretation (coherence), evolution and characterization, compared to the state-of-the-art.

**(3)** It is the first work in dynamic topic modeling using undirected stochastic graphical models and deterministic recurrent neural network to model collections of different-sized documents over time, within the generative and neural network framework. The code and data are available at [https://github.com/pgcool/RNN-RSM](https://github.com/pgcool/RNN-RSM).

## 2 The RNN-RSM model

RSM (Fig 2, Left) models are a family of differentiated Restricted Boltzmann Machines (RBMs) (Gehler et al., 2006; Xing et al., 2005; Gupta et al., 2015b, c) that models _word counts_ by sharing the same parameters with multinomial distribution over the observable i.e. it can be interpreted as a single multinomial unit (Fig 2, Middle) sampled as many times as the document size. This facilitates in dealing with the documents of different lengths.

The proposed RNN-RSM model (Fig 2, Right) is a sequence of conditional RSMs1 such that at any time step \(t\), the RSM's bias parameters \({\mathbf{b}}_{{\mathbf{v}}}^{(t)}\)

Figure 2: (Left): RSM for a document \({\mathbf{V}}_{n}\) of \(D_{n}\)=3 words (\(w\)). The bottom layer represents the softmax visible units, that share the same set of weights connected to binary hidden units \({\mathbf{h}}\). (Middle): Interpretation of RSM in which \(D_{n}\) softmax units with identical weights are replaced by a single multinomial unit, sampled \(D_{n}\) times. (Right): Graphical structure of 2-layered **RNN-RSM**, unfolded in time. Single and double headed arrows represent deterministic and stochastic-symmetric connections, respectively. \({\bf\hat{V}}^{(t)}\) and \({\mathbf{h}}^{(t)}\) are binary visible and hidden layers of RSM for a document collection at time, \(t\). \({\mathbf{u}}\) is RNN hidden layer. \(k\): dictionary index for a word \(w\)



and \({\bf b_{h}}^{(t)}\) depend on the output of a deterministic RNN with hidden layer \({\bf u}^{(t-1)}\) in the previous time step, \(t-1\). Similar to RNN-RBM [Boulanger-Lewandowski et al., 2012], we constrain RNN hidden units (\({\bf u}^{(t)}\)) to convey temporal information, while RSM hidden units (\({\bf h}^{(t)}\)) to model conditional distributions. Therefore, parameters (\({\bf b_{v}}^{(t)}\), \({\bf b_{h}}^{(t)}\)) are time-dependent on the sequence history at time \(t\) (via a series of conditional RSMs) denoted by \(\Theta^{(t)}\equiv\{\widehat{\mathbf{V}}^{(r)},{\bf u}^{(r)}|\tau<t\}\), that captures temporal dependencies. The RNN-RSM is defined by its joint probability distribution:

\[P(\widehat{\mathfrak{W}},{\bf H})=P(\{\widehat{\mathbf{V}}^{(t)},{\bf h}^{(t) }\}_{t=1}^{T})=\prod_{t=1}^{T}P(\widehat{\mathbf{V}}^{(t)},{\bf h}^{(t)}| \Theta^{(t)})\]

where \(\widehat{\mathfrak{W}}\) = [\(\widehat{\mathbf{V}}^{(1)},...\widehat{\mathbf{V}}^{(T)}\)] and \({\bf H}\) = [\({\bf h}^{(1)},...{\bf h}^{(T)}\)]. Each \({\bf h}^{(t)}\in\{0,1\}^{F}\) be a binary stochastic hidden topic vector with size \(F\) and \(\widehat{\mathbf{V}}^{(t)}=\{\mathbf{V}_{n}^{(t)}\}_{n=1}^{N^{(t)}}\) be a collection of \(N\) documents at time step \(t\). Let \(\mathbf{V}_{n}^{(t)}\) be a \(K\times D_{n}^{(t)}\) observed binary matrix of the \(n^{th}\) document in the collection where, \(D_{n}^{(t)}\) is the document size and \(K\) is the dictionary size over all the time steps. The conditional distribution (for each unit in hidden or visible) in each RSM at time step, is given by softmax and logistic functions:

\[P(v_{n,i}^{k,(t)}=1|{\bf h}_{n}^{(t)})=\frac{\exp(b_{v,i},^{k,(t)}+\sum_{j=1}^ {F}h_{nj}^{(t)}W_{ij}^{k})}{\sum_{q=1}^{K}\exp(b_{v,i},^{q,(t)}+\sum_{j=1}^{p_{ i}}h_{nj}^{(t)}W_{ij}^{q})}\]

\[P(h_{nj}^{(t)}=1|\mathbf{V}_{n}^{(t)})=\sigma(b_{h,j}^{(t)}+\sum_{i=1}^{D_{n}^ {(t)}}\sum_{k=1}^{K}v_{k,i}^{k,(t)}W_{ij}^{k})\]

where \(P(v_{n,i}^{k,(t)}=1|{\bf h}_{n}^{(t)})\) and \(P(h_{nj}^{(t)}=1|\mathbf{V}_{n}^{(t)})\) are conditional distributions for \(i^{th}\) visible \(v_{n,i}\) and \(j^{th}\) hidden unit \(h_{nj}\) for the \(n^{th}\) document at \(t\). \(W_{ij}^{k}\) is a symmetric interaction term between \(i\) that takes on value \(k\) and \(j\). \(v_{n}^{k,(t)}\) is sampled \(D_{n}^{(t)}\) times with identical weights connected to binary hidden units, resulting in multinomial visibles, therefore the name _Replicated Softmax_. The conditionals across layers are factorized as: \(P(\mathbf{V}_{n}^{(t)}|{\bf h}_{n}^{(t)})=\prod_{u=1}^{D_{n}^{(t)}}P(\mathbf{ v}_{u,i}^{(t)}|{\bf h}_{n}^{(t)})\); \(P({\bf h}_{n}^{(t)}|\mathbf{V}_{n}^{(t)})=\prod_{J}P(h_{nj}^{(t)}|\mathbf{V}_{ n}^{(t)})\).

Since biases of RSM depend on the output of RNN at previous time steps, that allows to propagate the estimated gradient at each RSM backward through time (BPTT). The _RSM biases_ and RNN hidden state \({\bf u}^{(t)}\) at each time step \(t\) are given by:

\[{\bf b_{v}}^{(t)} ={\bf b_{v}}+{\bf W_{uv}}{\bf u}^{(t-1)} \tag{1}\] \[{\bf b_{h}}^{(t)} ={\bf b_{h}}+{\bf W_{uh}}{\bf u}^{(t-1)}\] \[{\bf u}^{(t)} =\tanh({\bf b_{u}}+{\bf W_{uu}}{\bf u}^{(t-1)}+{\bf W_{vu}}\sum_{ n=1}^{N^{(t)}}\hat{\mathbf{v}}_{n}^{(t)}) \tag{2}\]

```
Input: Observed visibles, \(\widehat{\mathfrak{V}}\) = \(\{\langle\hat{\mathbf{V}}^{(0)},\hat{\mathbf{V}}^{(1)},...,\hat{\mathbf{V}}^{( t)},...,\hat{\mathbf{V}}^{(T)}\rangle\)RNN-RSM Parameters: \(\theta\) = {\(\mathbf{W_{wh}}\), \(\mathbf{W_{vb}}\), \(\mathbf{W_{wv}}\), \(\mathbf{W_{wu}}\), \(\mathbf{W_{vv}}\), \(\mathbf{W_{uu}}\), \(\mathbf{b_{v}}\), \(\mathbf{b_{u}}\), \(\mathbf{b_{l}}\), \(\mathbf{b_{v}}^{(t)}\), \(\mathbf{b_{h}}^{(t)},{\bf u}^{(0)}\)
1: Propagate \({\bf u}^{(t)}\) in RNN portion of the graph using eq 2. Compute \({\bf b_{v}}^{(t)}\) and \({\bf b_{h}}^{(t)}\) using eq 1. Generate negatives V\({}^{(t)*}\) using k-step Gibbs sampling. Estimate the gradient of the cost \(C\) w.r.t. parameters of RSM \(\mathbf{W_{vw}}\), \({\bf b_{v}}^{(t)}\) and \({\bf b_{h}}^{(t)}\) using eq 5. Compute gradients (eq 6) w.r.t. RNN connections (\(\mathbf{W_{uh}}\), \(\mathbf{W_{uw}}\), \(\mathbf{W_{uu}}\), \(\mathbf{W_{vu}}\), \(\mathbf{W_{vu}}\), \(\mathbf{W_{vu}}\), \(\mathbf{W_{vu}}\), \(\mathbf{u}^{0}\)) and biases (\(\mathbf{b_{v}}\), \(\mathbf{b_{h}}\), \(\mathbf{b_{u}}\)).
2:Goto step 1 until stopping_criteria (early stopping or maximum iterations reached)
```

**Algorithm 1**Training RNN-RSM with BPTT

where \(\mathbf{W_{uv}}\), \(\mathbf{W_{uh}}\) and \(\mathbf{W_{vu}}\) are weights connecting RNN and RSM portions (Figure 2). \(\mathbf{b_{u}}\) is the bias of \({\bf u}\) and \(\mathbf{W_{uu}}\) is the weight between RNN hidden units. \(\hat{\mathbf{v}}_{n}^{(t)}\) is a vector of \(\hat{v}_{n}^{k}\) (denotes the count for the \(k^{th}\) word in \(n^{th}\) document). notes the count for the \(k^{th}\) word in \(n^{th}\) document). \(\sum_{n=1}^{N^{(t)}}\hat{\mathbf{v}}_{n}^{(t)}\) refers to the sum of observed vectors across documents at time step \(t\) where each document is represented as-

\[\hat{\mathbf{v}}_{n}^{(t)}=[\{\hat{v}_{n}^{k,(t)}\}_{k=1}^{K}]~{}~{}\text{and}~{}~{} \hat{v}_{n}^{k,(t)}=\sum_{i=1}^{D_{n}^{(t)}}v_{n,i}^{k,(t)} \tag{3}\]

where \(v_{n,i}^{k,(t)}\)=1 if visible unit \(i\) takes on \(k^{th}\) value.

In each RSM, a separate RBM is created for each document in the collection at time step \(t\) with \(D_{n}^{(t)}\) softmax units, where \(D_{n}^{(t)}\) is the count of words in the \(n^{th}\) document. Consider a document of \(D_{n}^{(t)}\) words, the _energy_ of the state \(\{\mathbf{V}_{n}^{(t)},{\bf h}_{n}^{(t)}\}\) at time step, \(t\) is given by:

\[E(\mathbf{V}_{n}^{(t)},{\bf h}_{n}^{(t)})= -\sum_{j=1}^{F}\sum_{k=1}^{K}h_{nj}^{(t)}W_{jj}^{k}\hat{v}_{n}^{k,(t)}\] \[-\sum_{k=1}^{K}\hat{v}_{n}^{k,(t)}b_{v}^{k}-D_{n}^{(t)}\sum_{ j=1}^{F}b_{h,j}h_{nj}^{(t)}\]

Observe that the bias terms on hidden units are scaled up by document length to allow hidden units to stabilize when dealing with different-sized documents. The corresponding energy-probability relation in the energy-based model is-

\[P(\mathbf{V}_{n}^{(t)})=\frac{1}{Z_{n}^{(t)}}\sum_{{\bf h}_{n}^{(t)}}\exp(-E( \mathbf{V}_{n}^{(t)},{\bf h}_{n}^{(t)}) 

where \(H(\cdot)\) is the entropy and \(Q\) is the approximating posterior. Similar to Deep Belief Networks Hinton et al. (2006), adding an extra layer improves lower bound on the log probability of data, we introduce the extra layer via RSM biases that propagates the prior via RNN connections. The dependence analogy follows-

\[E({\bf V}_{n}^{(t)},{\bf h}_{n}^{(t)})\propto\frac{1}{{\bf b}_{{\bf v }}^{(t)}}\text{ and }E({\bf V}_{n}^{(t)},{\bf h}_{n}^{(t)})\propto\frac{1}{{\bf b }_{{\bf h}}^{(t)}}\\ \ln P({\bf V}_{n}^{(t)})\propto\frac{1}{E({\bf V}_{n}^{(t)},{\bf h }_{n}^{(t)})};\ln P(\widehat{{\bf V}}_{n}^{(t)})\propto\ln P(\left\{\hat{{\bf V }}_{n}^{\tau}\right\}_{\gamma<t})\]

Observe that the prior is seen as the deterministic hidden representation of latent topics and injected into each hidden state of RSMs, that enables the likelihood of the data to model complex temporal densities i.e. heteroscedasticity in document collections (\(\mathfrak{F}\)) and temporal topics (**H**).

**Gradient Approximations:** The _cost_ in RNN-RSM is: \(C=\sum_{l=1}^{T}C_{t}\equiv\sum_{t=1}^{T}-\ln P(\mathbf{\hat{V}}^{(t)})\)

Due to intractable \(Z\), the gradient of cost at time step \(t\) w.r.t. (with respect to) RSM parameters are approximated by k-step Contrastive Divergence (CD) Hinton (2002). The gradient of the negative log-likelihood of a document collection \(\{{\bf V}_{n}^{(t)}\}_{n=1}^{N^{(t)}}\) w.r.t. RSM parameter \({\bf W_{vh}}\),

\[\begin{split}&\frac{1}{N^{(t)}}\sum_{n=1}^{N^{(t)}}\frac{ \partial(-\ln P({\bf V}_{n}^{(t)}))}{\partial{\bf W_{vh}}}\\ &=\frac{1}{N^{(t)}}\sum_{n=1}^{N^{(t)}}\frac{\partial\mathfrak{F} ({\bf V}_{n}^{(t)})}{\partial{\bf W_{vh}}}-\frac{\partial(-\ln Z_{n}^{(t)})}{ \partial{\bf W_{vh}}}\\ &=\underbrace{\mathsf{E}_{P_{data}}[\frac{\partial\mathfrak{F}( {\bf V}_{n}^{(t)})}{\partial{\bf W_{vh}}}]}_{\text{data-dependent expectation}}-\underbrace{\mathsf{E}_{P_{model}}[\frac{\partial\mathfrak{F}({\bf V }_{n}^{(t)})}{\partial{\bf W_{vh}}}]}_{\text{model's expectation}}\\ &\simeq\frac{1}{N^{(t)}}\sum_{n=1}^{N^{(t)}}\frac{\partial \mathfrak{F}({\bf V}_{n}^{(t)})}{\partial{\bf W_{vh}}}-\frac{\partial \mathfrak{F}({\bf V}_{n}^{(t)*})}{\partial{\bf W_{vh}}}\end{split}\]

The second term is estimated by negative samples \({\bf V}_{n}^{(t)*}\) obtained from k-step Gibbs chain starting at \({\bf V}_{n}^{(t)}\) samples. \(P_{data}(\mathbf{\hat{V}}^{(t)},{\bf h}^{(t)})=P({\bf h}^{(t)}|\mathbf{\hat{V }}^{(t)})P_{data}(\mathbf{\hat{V}}^{(t)})\) and \(P_{data}(\mathbf{\hat{V}}^{(t)})=\frac{1}{N^{(t)}}\sum_{n}^{N^{(t)}}\delta( \mathbf{\hat{V}}^{(t)}-{\bf V}_{n}^{(t)})\) is the empirical distribution on the observable. \(P_{model}({\bf V}_{n}^{(t)*},{\bf h}_{n}^{(t)})\) is defined in eq. 4. The free energy \(\mathfrak{F}({\bf V}_{n}^{(t)})\) is related to normalized probability of \({\bf V}_{n}^{(t)}\) as \(P({\bf V}_{n}^{(t)})\)\(\equiv\exp^{-\mathfrak{F}({\bf V}_{n}^{(t)})}/Z_{n}^{(t)}\) and as follows-

\[\begin{gathered}\mathfrak{F}({\bf V}_{n}^{(t)})=-\sum_{k=1}^{K} \hat{c}_{n}^{k,(t)}b_{v}^{k}-\sum_{j=1}^{F}\log(1+\\ \exp(D_{n}^{(t)}b_{h,j}+\sum_{k=1}^{K}\hat{v}_{n}^{k,(t)}W_{j}^{k }))\end{gathered}\]

Gradient approximations w.r.t. RSM parameters,

\[\begin{gathered}\frac{\partial C_{t}}{\partial{\bf b}_{{\bf v }}^{(t)}}\simeq\sum_{n=1}^{N^{(t)}}\mathbf{\hat{v}}_{n}^{(t)*}-\mathbf{\hat{v }}_{n}^{(t)}\\ \frac{\partial C_{t}}{\partial{\bf b}_{{\bf h}}^{(t)}}\simeq\sum_{ n=1}^{N^{(t)}}\sigma({\bf W_{vh}\hat{v}_{n}^{(t)*}-D_{n}^{(t)}{\bf b_{h}}^{( t)}})\\ -\sigma({\bf W_{vh}\hat{v}_{n}^{(t)}-D_{n}^{(t)}{\bf b_{h}}^{(t)}}) \\ \frac{\partial C_{t}}{\partial{\bf W_{vh}}}\simeq\sum_{t=1}^{T} \sum_{n=1}^{N^{(t)}}\sigma({\bf W_{vh}\hat{v}_{n}^{(t)*}-D_{n}^{(t)}{\bf b_{h} }^{(t)}})\\ \mathbf{\hat{v}}_{n}^{(t)*T}-\sigma({\bf W_{vh}\hat{v}_{n}^{(t)}- D_{n}^{(t)}{\bf b_{h}}^{(t)}})\mathbf{\hat{v}}_{n}^{(t)T}\end{gathered} \tag{5}\]

The estimated gradients w.r.t. RSM biases are back-propagated via hidden-to-bias parameters (eq 1) to compute gradients w.r.t. RNN connections (\({\bf W_{uh}}\), \({\bf W_{uv}}\), \({\bf W_{vu}}\) and \({\bf W_{uu}}\)) and biases (\({\bf b_{h}}\), \({\bf b_{v}}\) and \({\bf b_{u}}\)).

\[\begin{gathered}\frac{\partial C}{\partial{\bf W_{uh}}}=\sum_{t=1}^{ T}\frac{\partial C_{t}}{\partial{\bf b_{h}}^{(t)}}{\bf u}^{(t-1)T}\\ \frac{\partial C}{\partial{\bf W_{vv}}}=\sum_{t=1}^{T}\frac{\partial C _{t}}{\partial{\bf b_{v}}^{(t)}}{\bf u}^{(t-1)T}\\ \frac{\partial C}{\partial{\bf W_{vu}}}=\sum_{t=1}^{T}\frac{\partial C _{t}}{\partial{\bf u}^{(t)}}{\bf u}^{(t)}(1-{\bf u}^{(t)})\sum_{n=1}^{N^{(t)}} \dot{\textbf{v}}_{n}^{(t)T}\\ \frac{\partial C}{\partial{\bf b_{h}}}=\sum_{t=1}^{T}\frac{\partial C _{t}}{\partial{\bf b_{h}}^{(t)}}\text{ and }\frac{\partial C}{\partial{\bf b_{v}}}=\sum_{t=1}^{T}\frac{\partial C _{t}}{\partial{\bf b_{v}}^{(t)}}\\ \frac{\partial C}{\partial{\bf b_{u}}}=\sum_{t=1}^{T}\frac{\partial C _{t}}{\partial{\bf u}^{(t)}}{\bf u}^{(t)}(1-{\bf u}^{(t)})\\ \frac{\partial C}{\partial{\bf W_{uu}}}=\sum_{t=1}^{T}\frac{\partial C _{t}}{\partial{\bf u}^{(t)}}{\bf u}^{(t)}(1-{\bf u}^{(t)}){\bf u}^{(t-1)T} \end{gathered}\] (6) 

For the single-layer RNN-RSM, the BPTT recurrence relation for \(0\leq t<T\) is given by

\[\frac{\partial C_{t}}{\partial\mathbf{u}^{(t)}}=\mathbf{W_{uu}}\frac{ \partial C_{t+1}}{\partial\mathbf{u}^{(t+1)}}\mathbf{u}^{(t+1)}(1-\mathbf{u}^{ (t+1)})\\ +\mathbf{W_{uh}}\frac{\partial C_{t+1}}{\partial\mathbf{b_{h}}^{ (t+1)}}+\mathbf{W_{uv}}\frac{\partial C_{t+1}}{\partial\mathbf{b_{v}}^{(t+1)}}\]

where \(\mathbf{u}^{(0)}\) being a parameter and \(\frac{\partial C_{T}}{\partial\mathbf{u}^{(T)}}=0\).

See _Training RNN-RSM with BPTT_ in Algo 1.

## 3 Evaluation

### Dataset and Experimental Setup

We use the processed dataset Gollapalli and Li (2015), consisting of EMNLP and ACL conference papers from the year 1996 through 2014 (Table 1). We combine papers for each year from the two venues to prepare the document collections over time. We use ExpandRank Wan and Xiao (2008) to extract top 100 keyphrases for each paper, including unigrams and bigrams. We split the bigrams to unigrams to create a dictionary of all unigrams and bigrams. The dictionary size (\(K\)) and word count are 3390 and 5.19 M, respectively.

We evaluate RNN-RSM against static (RSM, LDA) and dynamic (DTM) topics models for topic and keyword evolution in NLP research over time. Individual 19 different RSM and LDA models are trained for each year, while DTM2 and RNN-RSM are trained over the years with 19 time steps, where paper collections for a year is input at each time step. RNN-RSM is initialized with RSM (\(\mathbf{W_{vh}}\), \(\mathbf{b_{v}}\), \(\mathbf{b_{h}}\)) trained for the year 2014.

Footnote 2: [https://radimrehurek.com/gensim/models/dthmmodel.html](https://radimrehurek.com/gensim/models/dthmmodel.html)

We use perplexity to choose the number of topics (=30). See Table 2 for hyperparameters.

### Generalization in Dynamic Topic Models Perplexity:

We compute the perplexity on unobserved documents (\(\widehat{\mathbf{V}}^{(t)}\)) at each time step as

\[\text{PPL}(\widehat{\mathbf{V}}^{(t)},t)=\exp\Big{(}-\frac{1}{N^{(t)}}\frac{ \sum_{n=1}^{N^{(t)}}\log P(\mathbf{V}^{(t)}_{n})}{\sum_{n=1}^{N^{(t)}}D^{(t)}_ {n}}\Big{)}\]

where \(t\) is the time step. \(N^{(t)}\) is the number of documents in a collection (\(\widehat{\mathbf{V}}^{(t)}\)) at time \(t\). Better models have lower perplexity values, suggesting less uncertainties about the documents. For heldout documents, we take 10 documents from each time step i.e. total 190 documents and compute perplexity for 30 topics. Fig (d)d shows the comparison of perplexity values for unobserved documents from DTM and RNN-RSM at each time step. The _SumPPL_ (Table 3) is the sum of PPL values for the held-out sets of each time step.

**Document Time Stamp Prediction:** To further assess the dynamic topics models, we split the document collections at each time step into 80-20% train-test, resulting in \(1067\) held-out documents. We predict the time stamp (dating) of a document by finding the most likely (with the lowest perplexity) location over the time line. See the _mean absolute error_ (_Err_) in year for the held-out in Table 3. Note, we do not use the time stamp as observables during training.

### Tsd, Ted: Topic Evolution over Time Topic Detection:

To extract topics from each RSM, we compute posterior \(P(\widehat{\mathbf{V}}^{(t)}|h_{j}=1)\) by activating a hidden unit and deactivating the rest in a hidden layer. We extract the top 20 terms for every 30 topic set from 1996-2014, resulting in \(|Q|_{max}=19\times 30\times 20\) possible topic terms.

**Topic Popularity:** To determine topic _popularity_, we selected three popular topics (_Sentiment Analysis_, _Word Vector_ and _Dependency Parsing_) in NLP research and create a set3 of key-terms (including unigrams and bigrams) for each topic. We compute cosine similarity of the key-terms defined for each selected topic and topics discovered by the topic models over the years. We consider the discovered topic that is the most similar to the key-terms in the target topic and plot the similarity values in Figure (a)a, (b)b and (b)b. Observe that RNN-RSM shows better topic evolution for the three emerging topics. LDA and RSM show

\begin{table}
\begin{tabular}{c|c|c} \hline
**Parameter** & **Value(s)** & **Optimal** \\ \hline _epochs_ & 1000 & 1000 \\ _CD iterations_ & 15 & 15 \\ _learning rate_ & 0.1, 0.03, 0.001 & 0.001 \\ _hidden size_ & 20, 30, 50 & 30 \\ \hline \end{tabular}
\end{table}
Table 2: Hyperparameters for RNN-RSM model

\begin{table}
\begin{tabular}{c|c|c||c|c|c} \hline \multirow{2}{*}{**model**} & \multicolumn{4}{c}{**metric**} \\ \cline{2-5}  & _SumPPL_ & _Err_ & _mean-COH_ & _median-COH_ & _TTD_ \\ \hline _DTM_ & 10.9 & 8.10 & 0.1514 & 0.1379 & 0.084 \\ _RNN-RSM_ & **3.8** & **7.58** & **0.1620** & **0.1552** & 0.268 \\ \hline \end{tabular}
\end{table}
Table 3: State-of-the-art Comparison: Generalization (_PPL_ and _Err_), Topic Interpretation (_COH_) and Evolution (_TTD_) in DTM and RNN-RSM models 

topical locality in Figure 2(c) attributed to no correlation in topic dynamics over time, while in Figure 2(b), DTM does not capture evolution of topic _Word Vector_.

**Topic Drift (Focus Change):** To compute the topic _focus_ change over the years, we first split the time period 1996-2014 into five parts:{1996, 2000, 2005, 2010, 2014}. The cosine similarity scores are computed between the topic sets discovered in a particular year and the years preceding it in the above set, for example the similarity scores between the topic-terms in (1996, 2000), (1996, 2010), (1996, 2010) and (1996, 2014), respectively. Figure 2(i), 2(j), 2(k) and 2(l) demonstrate that RNN-RSM shows higher _convergence_ in topic focus over the years, compared to LDA and RSM. In RNN-RSM, the topic similarity is gradually increased over time, however not in DTM. The higher similarities in the topic sets indicate that new/existing topics and words do not appear/disappear over time.

We compute topic-term drift (\(TTD\)) to show the changing topics from initial to final year, as

\[TTD=1.0-cosineSimilarity(\mathbf{Q}^{(t)},\mathbf{Q}^{(t^{\prime})})\]

where **Q** is the set of all topic-terms for time step \(t\). Table 3 shows that \(TTD\) (where \(t\)=1996 and \(t^{\prime}\)=\(2014\)) are \(0.268\) and \(0.084\) for RNN-RSM and DTM, respectively. It suggests that the higher number of new topic-terms evolved in RNN-RSM, compared to DTM. Qualitatively, the Table 4 shows the topics observed with the highest and lowest cosine drifts in DTM and RNN-RSM.

In Figure 2(g) and 2(h), we also illustrate the temporal _evolution_ (drift) in the selected topics by computing cosine similarity on their adjacent topic vectors over time. The topic vectors are selected similarly as in computing topic popularity. We observe better TED in RNN-RSM than DTM for the three emerging topics in NLP research. For instance, for the selected topic _Word Vector_, the red line in DTM (Fig 2(h)) shows no drift (for x-axis 00-05, 05-10 and 10-14), suggesting the topic-terms in the adjacent years are similar and does not evolve.

Figure 3: (a, b, c): Topic popularity by LDA, RSM, DTM and RNN-RSM over time (d): Perplexity on the unobserved document collections over time (e, f): Mean and Median Topic Coherence (g, h): Topic Evolution (i,j,k,l): Topic focus change over time. Adj- Adjacent; Sim- Similarity





### Topic Interpretability

Beyond perplexities, we also compute topic coherence [20, 14, 36] to determine the meaningful topics captured. We use the coherence measure proposed by Aletras:2013 that retrieves co-occurrence counts for the set of topic words using Wikipedia as a reference corpus to identify context features (window=5) for each topic word. Relatedness between topic words and context features is measured using normalized pointwise mutual information (NPMI), resulting in a single vector for every topic word. The coherence (_COH_) score is computed as the arithmetic mean of the cosine similarities between all word pairs. Higher scores imply more coherent topics. We use Palmetto4 library to estimate coherence.

Footnote 4: github.com/earthquakesan/palmetto-py

**Quantitative:** We compute mean and median coherence scores for each time step using the corresponding topics, as shown in Fig 3e and 3f. Table 3 shows _mean-COH_ and _median-COH_ scores, computed by mean and median of scores from Fig 3e and 3f, respectively. Observe that RNN-RSM captures topics with higher coherence.

**Qualitative:** Table 5 shows topics (top-10 words) with the highest and lowest coherence scores.

### TTC: Trending Keywords over time

We demonstrate the capability of RNN-RSM to capture word evolution (usage) in topics over time. We define: _keyword-trend_ and SPAN. The _keyword-trend_ is the appearance/disappearance of the keyword in topic-terms detected over time, while SPAN is the length of the longest sequence of the keyword appearance in its keyword trend.

Let \(\widehat{\mathbf{Q}}_{model}=\{\mathbf{Q}^{(t)}_{model}\}_{l=1}^{T}\) be a set of sets5 of topic-terms discovered by the \(model\) (LDA, RSM, DTM and RNN-RSM) over different time steps. Let \(\mathbf{Q}^{(t)}\in\widehat{\mathbf{Q}}_{model}\) be the topic-terms at time step \(t\). The keyword-trend for a keyword \(k\) is a time-ordered sequence of 0s and 1s, as

Footnote 5: a set by **bold** and set of sets by \(\widetilde{\text{bold}}\)

\[\begin{split}&\text{trend}_{k}(\widehat{\mathbf{Q}})=[\text{ find}(k,\mathbf{Q}^{(t)})]_{t=1}^{T}\\ &\text{where;}~{}~{}\text{find}(k,\mathbf{Q}^{(t)})=\begin{cases} 1&\text{if $k\in\mathbf{Q}^{(t)}$}\\ 0&\text{otherwise}\end{cases}\\ \end{split} \tag{7}\]

And the SPAN (\(S_{k}\)) for the \(k\)th keyword is-

\[S_{k}(\widehat{\mathbf{Q}})=\text{length}\big{(}\text{longestOnesSeq}(\text{ trend}_{k}(\widehat{\mathbf{Q}})\big{)}\]

We compute keyword-trend and SPAN for each term from the set of some popular terms. We define average-SPAN for all the topic-terms appearing in the topics discovered over the years,

\[\begin{split}\texttt{avg-SPAN}(\widehat{\mathbf{Q}})& =\frac{1}{||\widehat{\mathbf{Q}}||}\sum_{\{k|\mathbf{Q}^{(t)}\in\widehat{ \mathbf{Q}}\wedge k\in\mathbf{Q}^{(t)}\}}\frac{S_{k}(\widehat{\mathbf{Q}})}{ \hat{v}^{k}}\\ &=\frac{1}{||\widehat{\mathbf{Q}}||}\sum_{\{k|\mathbf{Q}^{(t)}\in \widehat{\mathbf{Q}}\wedge k\in\mathbf{Q}^{(t)}\}}S_{k}^{dict}(\widehat{\mathbf{ Q}})\end{split}\]

where \(||{\widehat{\mathbf{Q}}}||=|\{k|\mathbf{Q}^{(t)}\in{\widehat{\mathbf{Q}}}\wedge k \in\mathbf{Q}^{(t)}\}|\) is the count of unique topic-terms and \(v^{k}\) = \(\sum_{t=1}^{T}\sum_{j=1}^{D_{t}}v_{j,t}^{k}\) denotes the count of \(k^{th}\) keyword.

In Figure 4, the keyword-trends indicate emergence (appearance/disappearance) of the selected popular terms in topics discovered in ACL and EMNLP papers over time. Observe that RNN-RSM captures longer SPANs for popular keywords and better word usage in NLP research. For example: _Word Embedding_ is one of the top keywords, appeared locally (Figure 5) in the recent years. RNN-RSM detects it in the topics from 2010 to 2014, however DTM does not. Similarly, for _Neural Language_. However, _Machine Translation_ and _Language Model_ are globally appeared in the input document collections over time and captured in the topics by RNN-RSM and DTM. We also show keywords (_Rule-set_ and _Seed Words_) that disappeared in topics over time.

Higher SPAN suggests that the model is capable in capturing trending keywords. Table 6 shows corresponding comparison of SPANs for the 13 selected keywords. The SPAN \(S_{k}\) for each keyword is computed from Figure 4. Observe that \(||{\widehat{\mathbf{Q}}}||_{DTM}<||{\widehat{\mathbf{Q}}}||_{RNN-RSM}\) suggests new topics and words emerged over time in RNN-RSM, while higher SPAN values in RNN-RSM suggest better trends. Figure 6 shows how the word usage, captured by DTM and RNN-RSM for the topic _Word Vector_, changes over 19 years in NLP research. RNN-RSM captures popular terms _Word Embedding_ and _Word Representation_ emerged in it.

## 4 Discussion: RNN-RSM vs DTM

**Architecture:** RNN-RSM treats document's stream as high dimensional sequences over time and models the complex conditional probability distribution i.e. _heteroscedasticity_ in document collections and topics over time by a temporal stack of RSMs (undirected graphical model), conditioned on time-feedback connections using RNN Rumelhart et al. (1985). It has two hidden layers: **h** (stochastic binary) to capture topical information, while **u** (deterministic) to convey temporal information via BPTT that models the topic dependence at a time step \(t\) on _all_ the previous steps \(\tau<t\). In contrast, DTM is built upon 

LDA (directed model), where Dirichlet distribution on words is not amenable to sequential modeling, therefore its natural parameters (topic and topic proportion distributions) for each topic are chained, instead of latent topics that results in intractable inference in topic detection and chaining.

**Topic Dynamics:** The introduction of explicit connection in latent topics in RNN-RSM allow new topics and words for the underlying topics to appear or disappear over time by the dynamics of topic correlations. As discussed, the distinction of \({\bf h}\) and \({\bf u}\) permits the latent topic \({\bf h}^{(t)}\) to capture new topics, that may not be captured by \({\bf h}^{(t-1)}\).

DTM assumes a fixed number of global topics and models their distribution over time. However, there is no such assumption in RNN-RSM. We fixed the topic count in RNN-RSM at each time step, since \({\bf W_{vh}}\) is fixed over time and RSM biases turn off/on terms in each topic. However, this is fundamentally different for DTM. E.g. a unique label be assigned to each of the \(30\) topics at any time steps \(t\) and \(t^{\prime}\). DTM follows the sets of topic labels: \(\{TopicLabels^{(t)}\}_{k=1}^{30}=\{ TopicLabels^{(t^{\prime})}\}_{k=1}^{30}\), due to eq (1) in Blei and Lafferty (2006) (discussed in section 5) that limits DTM to capture new (or local) topics or words appeared over time. It corresponds to the keyword-trends (section 3.5).

**Optimization:** The RNN-RSM is based on Gibbs sampling and BPTT for inference while DTM employs complex variational methods, since applying Gibbs sampling is difficult due to the nonconjugacy of the Gaussian and multinomial distributions. Thus, easier learning in RNN-RSM.

For all models, approximations are solely used to compute the likelihood, either using variational approaches or contrastive divergence; perplexity was then computed based on the approximated likelihood. More specifically, we use variational approximations to compute the likelihood for DTM (Blei and Lafferty, 2006). For RSM and RNN-RSM, the respective likelihoods are approximated using the standard Contrastive Divergence (CD). While there are substantial differences between variational approaches and CD, and thus in the manner the likelihood for different models is estimated - both approximations work well for the respective family of models in terms of approximating the true likelihood. Consequently, perplexities computed based on these approximated likelihoods are indeed comparable.

## 5 Conclusion and Future Work

We have proposed a neural temporal topic model which we name as RNN-RSM, based on probabilistic undirected graphical topic model RSM with time-feedback connections via deterministic RNN, to capture temporal relationships in historical documents. The model is the first of its kind that learns topic dynamics in collections of different-sized documents over time, within the generative and neural network framework. The experimental results have demonstrated that RNN-RSM shows better generalization (perplexity and time stamp prediction), topic interpretation (coherence) and evolution (popularity and drift) in scientific articles over time. We also introduced SPAN to illustrate topic characterization.

In future work, we forsee to investigate learning dynamics in variable number of topics over time. It would also be an interesting direction to investigate the effect of the skewness in the distribution of papers over all years. Further, we see a potential application of the proposed model in learning the time-aware i.e. dynamic word embeddings (Aitchison, 2001; Basile et al., 2014; Bamler and Mandt, 2017; Rudolph and Blei, 2018; Yao et al., 2018) in order to capture language evolution over time, instead of document topics.

#### Acknowledgments

We thank Sujatha Das Gollapalli for providing us with the data sets used in the experiments. We express appreciation for our colleagues Florian Buettner, Mark Buckley, Stefan Langer, Ulli Waltinger and Usama Yaseen, and anonymous reviewers for their in-depth review comments. This research was supported by Bundeswirtschaftsministerium (bmwi.de), grant 01MD15010A (Smart Data Web) at Siemens AG- CT Machine Intelligence, Munich Germany.

Figure 6: Word usage for emerging topic _Word Vector_ over time, captured by DTM and RNN-RSM





## References

* Aitchison (2001) Jean Aitchison. 2001. _Language change: progress or decay?_. Cambridge University Press.
* Aletras and Stevenson (2013) Nikolaos Aletras and Mark Stevenson. 2013. Evaluating topic coherence using distributional semantics. In _Proceedings of the 10th International Conference on Computational Semantics (IWCS)_. Potsdam, Germany, pages 13-22.
* Allan (2002) James Allan. 2002. Introduction to topic detection and tracking. In _Topic detection and tracking_, Springer, pages 1-16.
* Allan et al. (1998) James Allan, Jaime G Carbonell, George Doddington, Jonathan Yamron, and Yiming Yang. 1998. Topic detection and tracking pilot study final report. In _Proceedings of the DARPA Broadcast News Transcription and Understanding Workshop_. Virginia, US, pages 194-218.
* Bamler and Mandt (2017) Robert Bamler and Stephan Mandt. 2017. Dynamic word embeddings. In _Proceedings of the 34th International Conference on Machine Learning_. Sydney, Australia, pages 380-389.
* Basile et al. (2014) Pierpaolo Basile, Annalina Caputo, and Giovanni Semeraro. 2014. Analysing word meaning over time by exploiting temporal random indexing. In _Proceedings of the 1st Italian Conference on Computational Linguistics (CLiC-it)_. Pisa University Press, Pisa, Italy.
* Blei and Lafferty (2006) David M. Blei and John D. Lafferty. 2006. Dynamic topic models. In _Proceedings of the 23rd International Conference on Machine Learning_. Association for Computing Machinery, Pittsburgh, Pennsylvania USA, pages 113-120.
* Blei et al. (2003) David M. Blei, Andrew Y. Ng, and Michael I. Jordan. 2003. Latent dirichlet allocation. _Proceedings of Machine Learning Research_ 3(Jan):993-1022.
* Boulanger-Lewandowski et al. (2012) Nicolas Boulanger-Lewandowski, Yoshua Bengio, and Pascal Vincent. 2012. Modeling temporal dependencies in high-dimensional sequences: Application to polyphonic music generation and transcription. In _Proceedings of the 29th International Conference on Machine Learning_. Edinburgh, Scotland UK.
* Chang et al. (2009) Jonathan Chang, Sean Gerrish, Chong Wang, Jordan L. Boyd-Graber, and David M. Blei. 2009. Reading tea leaves: How humans interpret topic models. In _Advances in Neural Information Processing Systems_. Curran Associates, Inc., Vancouver, Canada, pages 288-296.
* Das et al. (2015) Rajarshi Das, Manzil Zaheer, and Chris Dyer. 2015. Gaussian lda for topic models with word embeddings. In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing_. Association for Computational Linguistics, Beijing, China, volume 1, pages 795-804.
* Gehler et al. (2006) Peter V. Gehler, Alex D. Holub, and Max Welling. 2006. The rate adapting poisson model for information retrieval and object recognition. In _Proceedings of the 23rd International Conference on Machine Learning_. Association for Computing Machinery, Pittsburgh, Pennsylvania USA, pages 337-344.
* Dollapalli and Li (2015) Sujatha Das Gollapalli and Xiaoli Li. 2015. Emnlp versus acl: Analyzing nlp research over time. In _Proceedings of the Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, Lisbon, Portugal, pages 2002-2006.
* Gupta et al. (2015a) Pankaj Gupta, Thomas Runkler, Heike Adel, Bernt Andrassy, Hans-Georg Zimmermann, and Hinrich Schutze. 2015a. Deep learning methods for the extraction of relations in natural language text. Technical report, Technical University of Munich, Germany.
* Gupta et al. (2015b) Pankaj Gupta, Thomas Runkler, and Bernt Andrassy. 2015b. Keyword learning for classifying requirements in tender documents. Technical report, Technical University of Munich, Germany.
* Gupta et al. (2016) Pankaj Gupta, Hinrich Schutze, and Bernt Andrassy. 2016. Table filling multi-task recurrent neural network for joint entity and relation extraction. In _Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers_. Osaka, Japan, pages 2537-2547.
* Gupta et al. (2015c) Pankaj Gupta, Udhayaraj Sivalangam, Sebastian Polisert, and Nassir Navab. 2015c. Identifying patients with diabetes using discriminative restricted boltzmann machines. Technical report, Technical University of Munich, Germany.
* Hall et al. (2008) David Hall, Daniel Jurafsky, and Christopher D. Manning. 2008. Studying the history of ideas using topic models. In _Proceedings of the conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, Honolulu, Hawaii, pages 363-371.
* Hinton and Salakhutdinov (2009) Geoffrey Hinton and Ruslan Salakhutdinov. 2009. Replicated softmax: an undirected topic model. In _Advances in Neural Information Processing Systems 22_. Curran Associates, Inc., Vancouver, Canada, pages 1607-1614.
* Hinton (2002) Geoffrey E. Hinton. 2002. Training products of experts by minimizing contrastive divergence. _Neural Computation_ 14(8):1771-1800.
* Hinton et al. (2006) Geoffrey E. Hinton, Simon Osindero, and Yee-Whye Teh. 2006. A fast learning algorithm for deep belief nets. _Neural Computation_ 18(7):1527-1554.
* Iwata et al. (2010) Tomoharu Iwata, Takeshi Yamada, Yasushi Sakurai, and Naonori Ueda. 2010. Online multiscale dynamic topic models. In _Proceedings of the 16th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_. Association for Computing Machinery, Washington DC, USA, pages 663-672

David Newman, Sarvnaz Karimi, and Lawrence Caveddon. 2009. External evaluation of topic models. In Proceedings of the 14th Australasian Document Symposium, Sydney, Australia. Cited by: SS5.
* I. Pruteanu-Malinici, L. Ren, J. Paisley, E. Wang, and L. Carin (2010)Hierarchical bayesian modeling of topics in time-stamped documents. IEEE transactions on pattern analysis and machine intelligence32 (6), pp. 996-1011. Cited by: SS5.
* M. Rudolph and D. Blei (2018)Dynamic pernoulli embeddings for language evolution. In Proceedings of the 27th International Conference on World Wide Web Companion, Lyon, France. Cited by: SS5.
* D. E. Rumelhart, G. E. Hinton, and R. J. Williams (1985)Learning internal representations by error propagation. Technical report California Univ San Diego La Jolla Inst for Cognitive Science. Cited by: SS5.
* A. Saha and V. Sindhwani (2012)Learning evolving and emerging topics in social media: a dynamic nmf approach with temporal regularization. In Proceedings of the 5th ACM International Conference on Web Search and Data Mining, Washington for Computing Machinery, Seattle, Washington USA, pp. 693-702. Cited by: SS5.
* A. Schein, H. Wallach, and M. Zhou (2016)Poisson-gamma dynamical systems. In Advances in Neural Information Processing Systems 29, P. Associates, Inc., Barcelona, Spain, pp. 5005-5013. Cited by: SS5.
* P. Smolensky (1986)Information processing in dynamical systems: foundations of harmony theory. Technical report Colorado University at Boulder Department of Computer Science. Cited by: SS5.
* I. Sutskever and G. Hinton (2007)Learning multilevel distributed representations for high-dimensional sequences. In Proceedings of the 11th International Conference on Artificial Intelligence and Statistics, W. Rico, pp. 548-555. Cited by: SS5.
* I. Sutskever, G. E. Hinton, and G. W. Taylor (2009)The recurrent temporal restricted boltzmann machine. In Advances in Neural Information Processing Systems 22, P. Associates, Inc., Vancouver, Canada, pp. 1601-1608. Cited by: SS5.
* G. W. Taylor, G. E. Hinton, and S. T. Roweis (2007)Modeling human motion using binary latent variables. In Advances in Neural Information Processing Systems 20, P. Associates, Inc., Vancouver, Canada, pp. 1345-1352. Cited by: SS5.
* N. Thang Vu, Heike Adel, P. Gupta, and H. Schutze (2016)Combining recurrent and convolutional neural networks for relation classification. In Proceedings of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 534-539. Cited by: SS5.
* N. Thang Vu, P. Gupta, H. Adel, and H. Schutze (2016)Bi-directional recurrent neural network with ranking loss for spoken language understanding.. In Proceedings of the Acoustics, Speech and Signal Processing (ICASSP), Shanghai, China, pp. 6060-6064. Cited by: SS5.
* X. Wan and J. Xiao (2008)Single document keyphrase extraction using neighborhood knowledge. In Proceedings of the 23rd National Conference on Artificial Intelligence, Chicago, Illinois USA, Vol. 8, pp. 855-860. Cited by: SS5.
* X. Wang and A. McCallum (2006)Topics over time: a non-markov continuous-time model of topical trends. In Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, F. Y. Ng and K. Lee (Eds.), pp. 424-433. Cited by: SS5.
* E. P. Xing, R. Yan, and A. G. Hauptmann (2005)Mining associated text and images with dual-wing harmoniums. In Proceedings of the 21st Conference on Uncertainty in Artificial Intelligence, A. Press, Edinburgh, Scotland UK. Cited by: SS5.
* Z. Yao, Y. Sun, W. Ding, N. Rao, and H. Xiong (2018)Dynamic word embeddings for evolving semantic discovery. In Proceedings of the 11th ACM International Conference on Web Search and Data Mining (WSDM), New York, NY, USA, pp. 673-681. Cited by: SS5.