Title: MODELING EVOLUTION OF LANGUAGE THROUGH TIME WITH NEURAL NETWORKS Anonymous authors Paper under double-blind review

Abstract: Language evolves over time with trends and shifts in technological, political, or cultural contexts. Capturing these variations is important to develop better language models. While recent works tackle temporal drifts by learning diachronic embeddings, we instead propose to integrate a temporal component into a recurrent language model. It takes the form of global latent variables, which are structured in time by a learned non-linear transition function. We perform experiments on three time annotated corpora. Experimental results on language modeling and classification tasks show that our model performs consistently better than temporal word embedding methods in two temporal evaluation settings: prediction and modeling. Moreover, we empirically show that the system is able to predict informative latent states in the future.

1. INTRODUCTION:

Language modeling with deep neural networks is an active research field (Howard & Ruder, 2018; Melis et al., 2018; Merity et al., 2018a; b) . It is a central task in Natural Language Processing (NLP) as it plays a major role in various text related tasks such as: speech recognition (Chiu et al., 2017) , image captioning (Vinyals et al., 2017) , or text generation (Fedus et al., 2018) . The task is to predict the probability distribution of the next word in a text sequence. The standard deep architecture for language models (LMs) has been the Recurrent Neural Network (RNN) for several years (Mikolov et al., 2010) . Moreover, their sophisticated version, LSTM networks (Hochreiter & Schmidhuber, 1997) , are still state of the art in language modeling (Melis et al., 2018; Merity et al., 2018b) , although research on new architectures is very active (Vaswani et al., 2017; Bai et al., 2018) .

Currently, recurrent language models are static and do not consider the various shifts that affect language; meaning of words can shift, new words appear as other vanish, and yesterday's topics are different from tomorrow's. To handle temporal variations in texts, recent research mainly focus on learning distinct word embeddings per timestep (Kim et al., 2014; Kulkarni et al., 2015; Hamilton et al., 2016) , and smoothing them in time (Bamler & Mandt, 2017; Yao et al., 2018) . Indeed, word embeddings are powerful tools to capture and analyze semantic relations between word pairs (Mikolov et al., 2013) . However, learning different embeddings for each timestep leads to learning algorithms with high time and memory complexity. This leads several approximations. For instance, Yao et al. (2018) use alternate optimization that breaks the flow of gradient through time. And Bamler & Mandt (2017) in their smoothing skip-gram algorithm propose complex gradient estimations that involves solving tridiagonal linear systems which cannot be parallelized in time.

In this paper, we propose a dynamic recurrent language model that uses global temporal variables instead of learning distinct embeddings at each timestep. Our contribution is threefold:

• We propose a dynamic recurrent language model in the form of an LSTM conditioned on global latent variables structured in time. By also learning a transition function in the latent space, we are able to predict states at future unseen timesteps. The learning procedure is straightforward thanks to the pathwise derivative (Kingma & Welling, 2014; Rezende et al., 2014) , thus scales to large corpora easily.

• We adapt temporal word embeddings algorithms to recurrent language modeling. We empirically show that these methods are not fit for learning with RNNs, and that the temporal embeddings almost systematically fail to beat non-temporal baselines on downstream classification tasks.

• We perform experiments on three time annotated text corpora with varying sizes and temporal scales. We evaluate the models on language modeling and on downstream classifications tasks.

2. RELATED WORK:

Studying of language evolution has been of interest for a long time in machine learning and information retrieval communities. Topics detection and tracking were among the firsts approaches to study language evolution. In 2002, Kabán & Girolami (2002) proposed a Hidden Markov Model (HMM) to visualize temporal evolution of topics in a textual stream. In 2006, Wang & McCallum (2006) and Blei & Lafferty (2006) proposed non-Markovian models based on Latent Dirichlet Allocations (LDA). While Wang & McCallum (2006) learn distributions of topics through time, Blei & Lafferty (2006) learn word distributions conditioned on latent topics that evolve through time with a State Space Model. However, these methods require to manually tune the number of latent topics, and language models are limited to simple word occurrence distributions. Moreover, these models are usually limited to specific conjugate distributions on the latent variables to allow tractable Variational Inference. Note that Blei & Lafferty (2006) led to several extensions, for instance with multi-scale temporal variable (Iwata et al. (2012) ), or continuous time dependencies (Wang et al. (2012) ).

After the introduction of the Word2Vec model (Mikolov et al., 2013) , numerous papers proposed derivations of the famous skip-gram algorithm for time annotated corpora (Frermann & Lapata, 2016) . All these approaches attempt to acquire a better understanding language evolution by studying shifts in words semantic through time. Among them, Eger & Mehler (2016) learn linear temporal dependencies between word representations. Yao et al. (2018) learns diachronic word representations by matrix factorization with temporal alignment constraints. Bamler & Mandt (2017) proposed a temporal probabilistic skip-gram model with a diffusion prior. Rudolph & Blei (2017) also propose a probabilistic framework that uses exponential family embeddings. Compared to HMM and LDA based approached, the skip-gram algorithm uses standard gradient descent and can be parallelized easily to scale to massive corpora. But all the above approaches learn distinct word embeddings at each timestep, which leads a huge number of trainable parameters. An exception is Rosenfeld & Erk (2018) that combines a static word representation to a scalar timestep in a deep neural network that produces a temporal embedding.

An alternative to these various models is to leverage RNNs for language modeling. A recurrent language model takes a sequence of words of arbitrary size as input and outputs a probability distribution of the next word. Such models are often parameterized by LSTM networks (Hochreiter & Schmidhuber, 1997) . Compared to the skip-gram algorithm that uses a limited context window, recurrent language models operate on sequences of arbitrary length, and can capture long-term dependencies. They are nowadays used at the core of an increasing number of tasks. For instance, as a feature extractor for text classification (Peters et al., 2018) , as a core building block of unsupervised Neural Machine Translation models (Lample et al., 2018) , or as a discriminator for Generative Adversarial Models on text (Yang et al., 2018) .

In this paper, we propose a dynamic language model based on RNNs. The aim is to capture the language evolution through time via an end-to-end framework, where a standard RNN is conditioned by a latent representation of temporal drifts in language. To the best of our knowledge, no RNN LMs methods have been proposed for the extraction of temporal dynamics in text data.

3. MODEL:

We propose a dynamic recurrent language network. It is a State Space Model (SSM) with one global latent state per timestep used to condition an LSTM Language Model. Unlike most current methods that learn complete word embedding matrices for each time step, we only learn one embedding per word, which is augmented with the states of the SSM. The LSTM can capture general language dynamics shared between all timesteps, and uses the temporal states to adapt language dynamics depending on language bias specific to each timestep. We also learn a transition function between states that allows estimation of future states.

x 1 x 2 x T z 1 z 0 z 2 z T N1 N2 NT

3.1. NOTATIONS AND TASK:

We consider text sequences, annotated by discrete timesteps t ∈ {1, 2, . . . , T }, defined over a vocabulary of size V . Let x t ∈ X t be a text sequence of length |x t | published at time t. We denote by x t k the k th term of sequence x t . A corpus X t is composed of N t such sequences. The complete corpus is denoted X = {X 1 , X 2 , . . . , X T }.

In the standard, non-temporal, recurrent language modeling task, the objective is to find parameters θ maximizing the likelihood of the next word given the previous ones for all sentences in a corpus:

EQUATION

where x 1:k is the sequence of the first k words in the sequence x and p θ is parametrized by an RNN with parameters θ that outputs next word probabilities. Specifically, we have

p θ (x k+1 |x 1:k ) = softmax(W h k + b) where W ∈ R V ×d h and b ∈ R V are parameters to learn, h k = f (x k , h k-1 ; v)

is a hidden vector of size d h , and f is the RNN's recurrent function with parameters v. We thus have θ = {U , W , b, v} where U is word embeddings matrix.

3.2. DYNAMIC RECURRENT LANGUAGE MODEL:

Our goal is to extend classic recurrent language models with a dynamic component in order to adapt it to language shifts. We propose to augment the word embeddings of an LSTM LM with global temporal variables that are connected in time with a transition function learned jointly with the LSTM. We use a probabilistic SSM where global latent variables z t ∈ R dz capture information specific at a timestep t and onward, and are decoded by the LSTM to adapt its language model. Figure 1 represents the directed graphical model.

Learning the transition function between latent states enables estimation of future states of the system, where data is not available during training. The transition function is a Gaussian model centered on a non-linear transition function of the previous state, with a diagonal covariance matrix σ 2 :

z t+1 |z t ∼ N (g(z t ; w), σ 2 )

where w are the transition function parameters. Learning a transition function gives the system the freedom to learn interesting latent trajectories. We hypothesize this will yield better performances, compared to a more restrictive diffusion model for instance. For the prior on the first timestep, we learn a vector z 0 that acts as the initial conditions of the system.

Overall, the joint distribution factorizes as follows:

EQUATION

where ψ = (w, σ 2 , z 0 ) are the temporal prior parameters, and Z ∈ R T ×dz is the matrix containing latent variables z t . p θ (x|z t ) is parameterized by an LSTM where the latent state z t is concatenated to every word embedding vectors for a given sequence x.

3.3. INFERENCE:

Learning the generative model in equation 2 requires to infer the latent variables z t . In Bayesian inference, it is done by estimating their posterior p θ,ψ (Z|X) = p θ,ψ (X,Z) p θ,ψ (X,Z)dZ . Unfortunately, the marginalization on Z requires to compute an intractable normalizing integral. We therefore use Variational Inference (VI), and consider a variational distribution q φ (Z) that factorizes across all timesteps:

q φ (Z) = T t=1 q t φ (z t )

where q t φ are independent Gaussian distributions N (µ t , σ 2 t ) with diagonal covariance matrices σ 2 t , and φ is the total set of variational parameters.

Following the derivation in Krishnan et al. ( 2017), we get the following evidence lower bound (ELBO):

EQUATION

with q 0 a Dirac distribution centered on z 0 , and D KL the Kullback-Leibler (KL) divergence.

Since the observation model p θ is an RNN, the model is non-conjugate, and the ELBO in equation 3 cannot be computed in closed form. We thus use the re-parametrization trick (Kingma & Welling, 2014; Rezende et al., 2014) to learn the model.

Global temporal states coupled with variational distributions independent in time offer several learning and computation advantages. This allows the system to deal with strong disruptions in language shifts, for which regularities observed on other steps could not hold. Rather than considerably upsetting the transition function, and thus highly impacting consecutive states, the learning algorithm can choose to ignore such difficult transitions, at a cost depending on the variance σ 2 . This variance σ 2 , learned jointly with the model, allows the learning algorithm to adapt the stochastic transition according to the regularity level of the data.

Moreover, since temporal dependency is broken, the computation of the KL in equation 3 can be computed in parallel, while still maintaining information flow through time. Indeed, since the latent states are global vectors, we can easily sample all of them at every optimization steps, even when T is large, and compute the prior in parallel. We can hence learn the model with mini-batches containing text samples from every timesteps, allowing gradient flow in q t φ from the likelihood and the KL at both past and future timesteps simultaneously through the pathwise derivative.

4. EXPERIMENTAL PROTOCOL:

We evaluate the proposed model together with baselines adapted from the temporal word embedding literature, detailed in section 4.1. We propose two evaluation configurations: prediction and modeling, presented in section 4.2. We perform experiments on three temporal corpora, with different sizes, temporal scales, and language level, described in section 4.3. In section 5.1 we conduct experiments on language modeling, and in section 5.2 we evaluate learned representations on classification tasks. In addition, we present text samples generated by our model in appendix D.

4.1. MODELS AND BASELINES:

In our experiments, we compare the following models:

• LSTM: a standard regularized LSTM. This baseline has no temporal component, but is currently the state-of-the-art in language modeling. • DT: the DiffDtime model presented in Rosenfeld & Erk (2018) . It is a deep model that takes as input learned word embeddings and a timestep which outputs temporal word embeddings. Like our approach, this model learns only one embedding vector per word, but their temporal prior is obtained only by scaling a learned vector with a scalar timestep.

• DWE: the Dynamic Word Embedding model (Bamler & Mandt, 2017) learns Gaussian word embeddings with a probabilistic version of the skip-gram algorithm. This method learns a different set of word embeddings per timestep, that are smoothed in time with a diffusion prior.

• DRLM-Id: the Dynamic Recurrent Language Model proposed in this paper, where the transition function is replaced by the identity matrix so that z t+1 ∼ N (z t , σ 2 ).

• DRLM: the Dynamic Recurrent Language Model proposed in this paper with learned transition function.

Since, to the best of our knowledge, no dynamic recurrent language models have been proposed, we compare our approach to the distributed word embedding models DT and DWE. We adapt these models for recurrent language models by replacing the skip-gram component with an LSTM and discarding the context embeddings. More details can be found in appendix B. We also evaluate the proposed model without transition function (DRLM-Id) to assess its impact on performances.

4.2. TEMPORAL EVALUATION:

To evaluate the models in a temporal context, we propose the two following settings:

Prediction We take the first T p timesteps to train models, and evaluate them on the next timesteps T p + 1 to T , with T the total number of timesteps. For DRLM, we use the transition model g to predict future states z t in time. For DT and DWE we use the embeddings at the last training timestep U Tp . Timestep T p + 1 is used for hyperparameters tuning.

Modeling In this configuration, models are trained and evaluated on all timesteps T , and corpora are randomly split into a training (60%) validation (10%) and test (30%) sets. In this setup, the training set for a given corpus is different from prediction, resulting in slightly different vocabulary sizes and hyperparameters.

We evaluate the models on language modeling and on downstream classification tasks. For language modeling the evaluation metric is perplexity on the respective test sets. We report micro perplexity, computed on the total test set, and macro perplexity, which is the averaged perplexity computed separately at each test timestep. For classification, we report precision, recall, and F1 measures for multi-label classification, and top1, top3, and top5 scores for multi-class classification.

4.3. TEMPORAL CORPORA:

We use three different corpora for our experiment, which differ in terms of time ranges, topics, number of sequence per timestep, and sequence lengths.

-The Semantic Scholar1 corpus (S2) is composed of titles from scientific papers published in machine learning conferences and journals from 1985 to 2017, split by years (33 timesteps). The corpus is composed of 50K titles, representing a total of 500K words.

-The New York Times (Yao et al., 2018) corpus (NYT) is composed of headlines from the New York Times newspaper spanning from 1990 to 2015, also split by years (26 timesteps).

The corpus contains 50K headlines and 500K words.

-The Reddit corpus contains a sample of 3% of the social network's posts presented in Tan & Lee (2015) . It is composed of 100K posts sampled from January 2006 to December 2013 split by quarters (32 timesteps).

4.4. PREPROCESSING AND HYPERPARAMETERS:

For each corpus, the vocabulary is constructed with words appearing at least 5 times in training sets (3 times for S2). The resulting vocabulary sizes are 5K tokens for S2, 8K for NYT and 13K for Reddit. We train a 2 layers AWD-LSTM (Merity et al., 2018b ) for all models with hidden units and word embeddings of size 400. We use weight dropout, variational dropout, embedding dropout, and embeddings weight-tying (except for DWE that learns distinct word embeddings per timestep). Its hyperparameters are fixed across all models for a given corpus, except for input dropout, weight decay and learning rates that are tuned individually for each model.

For language modeling experiments, we decay the learning rate by a factor of 10 when no improvement is seen on the validation set for 10 consecutive epochs. For classification tasks, we linearly anneal the learning rate.

5. RESULTS:

In section 5.1 we present results on the temporal language tasks. In section 5.2, we extract the embeddings trained in section 5.1 and evaluate them on downstream classification tasks.

5.1. LANGUAGE MODELING:

We present here language modeling results for our three corpora in prediction and modeling configurations.

Prediction Figure 2 shows perplexity evolution for the prediction setup (numerical results are provided in appendix C). On the three corpora, both DRLM-Id and DRLM beat all baselines. The standard LSTM always performs better than the DWE and DT baselines that systematically overfit. This shows that LSTM, even without temporal components are powerful, and conditioning them is not trivial. Results on Reddit (figure 2c ) tend to confirm this observation: performances LSTM, DRLM-Id, and DRLM are quasi-equivalent, with a gain of 2 points of micro perplexity for DRLM compared to LSTM. It is a corpus twice larger than the others, with longer sequences. Our analyses is that with sufficient data, and due to the auto-regressive nature of text, LSTM, even without explicit temporal prior, manage to capture temporal biases implicitly.

In the S2 corpus, we can see in figure 2a that, while the perplexity of DRLM-Id tends to converge to LSTM's perplexity, DRLM presents consistent improvement through time. On the NYT corpus, while DRLM-Id and DRLM have significant performance gain compared to LSTM (more than 5 points), the difference between the two models is small, and vanishes with time. The NYT corpus is composed of newspaper headlines that are greatly influenced by exterior factors, while S2 are scientific publications, which are influenced by one another through time. Hence DRLM, thanks to its transition function, is able to predict informative latent states on S2 but not on NYT.

Recursive Inference To validate empirically this hypothesis, we recursively infer the latent states z t of DRLM by maximizing equation 3 with data from X t , and fixing all other parameters. We then evaluate the resulting model at t + 1, next we infer z t+1 on X t+1 , evaluate at t + 2 and so on. We perform the same recursive inference algorithm for variational parameters of DWE. We call the two methods DRLM-F and DWE-F respectively, and present the results in figure 3 .

We first observe that DRLM-F significantly improves long-term performances on NYT, meaning that the trained LSTM is able to interpret latent states z t never seen during training. This is not trivial, given the difficulties and various tricks present in the literature to condition LSTM language models (Bowman et al., 2016; Semeniuta et al., 2017; Yang et al., 2017) . We also see that recursive inference does not improve DRLM results on S2, while DWE results are greatly improved on NYT and even more on S2. This shows that the there is a temporal drift in S2, which is less clear on Reddit since recursive inference does not improve performances neither on DMLR nor on DWE. It then follows that DLRM predicts accurate latent states on the S2 corpus, since there is a temporal drift, but feeding future data does not enhance performances.

The DWE baseline benefits a lot more from recursive inference than DMLR. This is expected since it can adapt each word embedding at each timestep, whereas DRLM-F only infer the distribution of a single vector per timestep. This thus makes DWE-F a good baseline for assessing temporal drift.

Modeling Table figure 1 presents results for the modeling setup. As for prediction, temporal word embeddings baselines also fail to beat the LSTM baseline. All perplexities are lower since the task is easier, but DRLM and DRLM-Id keep their perplexity gain over LSTM. However, although DRLM is always at least better than DRLM-Id, the difference between the two is thinner than in the prediction setup.

5.2. TEXT CLASSIFICATION WITH TEMPORAL WORD EMBEDDINGS:

To further evaluate the representations learned by DRLM, we extract its word embeddings augmented with temporal states, and use them for text classification. For the DT and DWE baselines, we learned temporal embeddings exactly as described in their respective papers.

Classification Model For every classification tasks, we learn a linear classifier that takes as input an average of the embeddings for a given sequence, as done in Joulin et al. (2016) and Shen et al. (2018) . We learn the classifier by maximizing the likelihood

1 N T t=1 N t i=1 y t i log f (Au t i )

where A is a weight matrix, u t i is the averaged temporal embeddings of sequence i at time t. And f is a normalizing function that depends on the task: a softmax for multi-class classification and a sigmoid for multi-label classification.

Tasks For S2, the task is to classify articles' keywords (multi-label with 400 classes). For NYT, the classes are the news section in which articles are published (mono-label with 28 labels). And for Reddit, the task is to classify the subreddit in which posts are submitted (mono-label with 60 labels). Classification results confirm language modeling results. On the S2 corpus, we can see in figure 4a that DRLM has higher F1 score across all timesteps, and performs consistently better in the modeling configuration. However, DRLM-Id is better in prediction for NYT (figure 4b ) except in 2010, which is the validation timestep. And for the same configuration, DWE also performs better than DWE-F, confirming chaotic temporal drifts in the corpus. Reddit is the only corpus where a temporal word embeddings baseline, DWE, manages to beat the LSTM. Overall, our DLMR and DLMR-Id models obtained significantly better classification scores compared to every baselines.

6. CONCLUSION:

We proposed a Dynamic Recurrent Language Model (DRLM) for handling temporal drifts in language. We conditioned and LSTM language model by augmenting its word embeddings with temporal latent variables. The latent variables are global states that capture temporal variations. They are structured in time with a learned transition model, which enables the estimation of future states. The transition model acts as a temporal prior on latent states, allowing their distribution to adapt to disruptive shifts in the data.

Experiments on three corpora with various sizes, time scales, and language levels, showed that our approach beats temporal word embeddings baselines in two temporal evaluation configurations (prediction and modeling) on language modeling and classification tasks. Empirical results also showed that, for certain corpora, our model is able to accurately predict future states at timesteps where no data were available during training.

In this work, we did not address the fact that new words appear at future timesteps. Handling out-ofvocabulary words are of major concern in NLP, and predicting new words is even more challenging.

A promising approach would be to use byte pair encoding (Sennrich et al., 2016) and learn to predict future sub-words combinations.

Also, while our method works well in practice, it is not clear how the LSTM networks internals interpret the latent states. Efficiently conditioning language models is an active research field, as stated in section 5.1. And since convolutional decoders' popularity is rapidly growing (Semeniuta et al., 2017; Yang et al., 2017) , using efficient methods from the computer vision community, like Adaptive Instance Normalization (Huang & Belongie, 2017) , seems a promising research direction.

A COMPLETE INFERENCE DERIVATION:

The ELBO in equation 3 can be obtained as follows:

log P θ,φ (X) = log Z p ψ (Z) T t=1 p θ (X t |z t )dZ = log Z q φ (Z)p ψ (Z) T t=1 p θ (X t |z t ) q φ (Z) dZ ≥ Z q φ (Z) log     p ψ (Z) T t=1 p θ (X t |z t ) q φ (Z)     dZ = T t=1 z t q t φ (z t ) log p θ (X t |z t )dz t + T t=1 z t-1 q t-1 φ (z t-1 ) z t q t φ (z t ) log p ψ (z t |z t-1 ) q t φ (z t ) dz t-1 dz t = T t=1 E q t φ (z t ) log p θ (X t |z t ) - T t=1 E q t-1 φ (z t-1 ) D KL (q t φ (z t ) p ψ (z t |z t-1 ))

= L(θ, ψ, φ) where the inequality is obtained thanks to the Jensen theorem on concave functions.

The KL between two Gaussians owns an analytically closed form. This allows us to rewrite our log-likelihood lower-bound, noted L(θ, ψ, φ), as follows:

L(θ, ψ, φ) = T t=1 E q t φ (z t ) p θ (X t |z t ) + T d 2 - 1 2 T d-1 i=0 log σ 2 i - T t=1 d-1 i=0 log η 2 t,i + T t=1 d-1 i=0 η 2 t,i σ 2 i + T t=1 E q t-1 φ (z t-1 ) (g(z t-1 ; w) -µ t ) (σ 2 ) -1 (g(z t-1 ; w) -µ t )

where we note A the matrix transpose of a matrix A and where σ 2 i and η 2 t,i stand for the i-th component of diagonals σ 2 and η 2 t respectively. This re-writing allows one to improve learning stability w.r.t. a version in which sampling would be done on the KL components too.

B DERIVING TEMPORAL WORD EMBEDDING METHODS FOR RECURRENT LANGUAGE MODELING:

We detail here how we adapt temporal word embeddings baselines to recurrent language modeling. The baselines are Dynamic Word Embeddings (DWE) (Bamler & Mandt, 2017) , and DiffTime (Rosenfeld & Erk, 2018) . For both methods, we get rid of the context embeddings and only keep word embeddings U.

B.1 DYNAMIC WORD EMBEDDINGS:

In DWE (Bamler & Mandt, 2017) , Gaussian word embeddings are learned at each timestep with a temporal diffusion prior:

U t+1 |U t ∼ N U t 1 + σ 2 t /σ 2 0 , 1 σ -2 t + σ -2 0 I

where σ 2 0 and σ 2 t are hyperparameters of the model. We derive their skip-gram algorithm for our setting by maximizing the following approximate ELBO:

L DWE (θ, φ) = T t=1 E q φ (Ut) log p θ (X t |U t ) + E q φ (Ut) log E q φ (Ut-1) [p(U t |U t-1 )] -E q φ (Ut) [log q φ (U t )] (4)

where p θ is paramatrized by an LSTM. q φ is a variational Gaussian distribution that factorizes as:

q φ (U) = T t=1 q φ (U t )

and φ are its parameters.

To learn this model, we sample a mini-batches M that contains text coming from different training timesteps. We must hence rescale the ELBO in equation 4. We do so by estimating the probability that a given word appears in a particular mini-batch:

L minibatch (θ, φ) = |X| |M| E q φ (U M ) x∈M log p θ (x|U M ) + u∈U M 1 (1 -(1 -ν u ) |M| ) T t=1 E q φ (u) log E q φ (ut-1) [p(u t |u t-1 )] -E q φ (ut) [log q φ (u t )]

where U M are the embeddings of words in M, ν u is the apparition frequency of term whose embedding is u in X, and |X| (respectively |M|) is the number of words in X (M). In this formulation, gradient computation does not require any approximation, while allowing it to flow through all timesteps.

B.2 DIFFTIME:

The adaptation of the DiffTime baseline (Rosenfeld & Erk, 2018) is straightforward. It learns a non-linear function d that outputs temporal word embeddings:

u t = d(u, t; φ)

where u is a learned word embedding, t is a scalar timestep, and φ are the function's parameters. We refer the reader to the complete paper for more details on the implementation of d.

For recurrent language modeling adaptation, we simply learn jointly the word embeddings U, the parameters φ of d and the parameters θ of an LSTM by maximizing the following likelihood:

L DT (θ, φ, U) = T t=1 x∈X t |x|-1 k=1 p θ (x k+1 |u t 1:k ) C QUANTITATIVE RESULTS FOR PREDICTION

We report here the quantitative results for the prediction configuration. Language modeling results are shown in table 3 , and text classification results in table 4.

D TEXT GENERATION THROUGH TIME:

We present here texts samples generated by our model. The three word triplets that most often appear in the S2 modeling test set are used as seed for LSTM. Samples are generated by beam search with the DLMR model trained in modeling configuration. Table 5 presents generated samples where the latent state that condition the LSTM evolves from z 0 to z T . 

:

http://labs.semanticscholar.org/corpus/

Figures and Tables:

Figure 1: Directed graphical model.

Figure 2: Perplexity through time for the prediction configuration. Results shown are obtained with texts published at future time periods, not seen during training.

Figure 3: Perplexity through time with recursive inference. DRLM-F and DWE-F are trained on T p timesteps, and then their variational parameters are recursively inferred on data at timestep T p + τ and evaluated at T p + τ + 1. The LSTM baseline is displayed for comparison purposes.

Figure4: Classification results with temporal word embeddings in the prediction configuration. For LSTM, DRLM, and DRLM-Id, word embeddings were pretrained on the language modeling tasks in section 5.1, while for the baselines DT and DWE, they were trained as proposed by their authors.

Modeling perplexity, where training and testing timesteps are the same.

Table content: <table><tr><td/><td/><td>S2</td><td>NYT</td><td>Reddit</td></tr><tr><td>Models</td><td colspan="4">micro macro micro macro micro macro</td></tr><tr><td>LSTM</td><td>62.8</td><td>66.2</td><td colspan="2">109.9 110.4 116.7 123.0</td></tr><tr><td>DT</td><td>70.7</td><td>73.9</td><td colspan="2">125.6 120.4 136.8 147.7</td></tr><tr><td>DWE</td><td>65.9</td><td>69.8</td><td colspan="2">119.9 120.4 129.4 139.6</td></tr><tr><td colspan="2">DRLM-Id 60.6</td><td>61.3</td><td colspan="2">104.0 104.4 115.5 121.5</td></tr><tr><td>DRLM</td><td>60.2</td><td>61.2</td><td colspan="2">103.5 103.9 114.7 120.4</td></tr></table>

Classification results with temporal word embeddings for the modeling configuration.Results Prediction results are displayed in figure4, and the detailed results can be found in appendix C. For the DWE, we also show results when the word embeddings are recursively inferred with the skip-gram filtering algorithm of Bamler & Mandt (2017) (DWE-F). Modeling results are given in table 2.

Table content: <table><tr><td/><td/><td>S2</td><td/><td>NYT</td><td>Reddit</td></tr><tr><td>Models</td><td colspan="3">Precision Recall F1</td><td>top1 top3 top5 top1 top3 top5</td></tr><tr><td>LSTM</td><td>0.427</td><td>0.171</td><td colspan="2">0.221 41.4 63.2 74.3 44.0 66.7 74.8</td></tr><tr><td>DT</td><td>0.272</td><td>0.077</td><td colspan="2">0.110 17.3 40.9 57.5 40.9 56.4 63.3</td></tr><tr><td>DWE</td><td>0.371</td><td>0.129</td><td colspan="2">0.174 24.8 51.0 66.9 44.5 64.0 71.5</td></tr><tr><td colspan="2">DRLM-Id 0.429</td><td>0.176</td><td colspan="2">0.226 44.3 68.2 79.5 45.6 68.6 76.7</td></tr><tr><td>DRLM</td><td>0.435</td><td>0.177</td><td colspan="2">0.227 44.8 68.8 80.0 45.2 68.7 76.8</td></tr></table>



Table content: <table><tr><td/><td/><td/><td colspan="2">: Prediction perplexity</td></tr><tr><td/><td/><td colspan="2">S2</td><td>NYT</td><td>Reddit</td></tr><tr><td/><td>Models</td><td colspan="3">micro macro micro macro micro macro</td></tr><tr><td/><td>LSTM</td><td>84.7</td><td>82.7</td><td>128.5 128.4 125.8 126.1</td></tr><tr><td/><td>DT</td><td>92.0</td><td>89.6</td><td>137.1 137.0 151.1 151.6</td></tr><tr><td/><td>DWE</td><td>87.0</td><td>84.8</td><td>140.1 140.0 136.5 139.9</td></tr><tr><td/><td colspan="2">DRLM-Id 81.2</td><td>79.2</td><td>123.7 123.6 124.7 125.0</td></tr><tr><td/><td>DRLM</td><td>79.7</td><td>77.8</td><td>123.3 123.1 123.9 124.3</td></tr><tr><td/><td/><td colspan="3">Table 4: Prediction classification</td></tr><tr><td/><td/><td>S2</td><td/><td>NYT</td><td>Reddit</td></tr><tr><td>Models</td><td colspan="3">Precision Recall F1</td><td>top1 top3 top5 top1 top3 top5</td></tr><tr><td>LSTM</td><td>0.378</td><td>0.144</td><td colspan="2">0.190 35.1 54.7 65.9 32.0 52.8 63.4</td></tr><tr><td>DT</td><td>0.357</td><td>0.111</td><td colspan="2">0.154 19.1 45.1 62.1 12.5 26.0 33.2</td></tr><tr><td>DWE</td><td>0.358</td><td>0.134</td><td colspan="2">0.177 33.4 55.0 66.8 34.3 51.0 60.8</td></tr><tr><td>DWE-F</td><td>0.317</td><td>0.166</td><td colspan="2">0.186 31.4 53.4 65.0 34.8 50.7 59.7</td></tr><tr><td colspan="2">DRLM-Id 0.385</td><td>0.144</td><td colspan="2">0.193 42.3 66.9 77.6 41.6 58.3 67.0</td></tr><tr><td>DRLM</td><td>0.370</td><td>0.167</td><td colspan="2">0.208 41.2 60.0 68.9 38.0 56.2 66.7</td></tr></table>

Text sequences generating with DMLR conditioned on different timesteps on the S2 corpus. The first three words are uses as seeds, and the samples are generted by beam search.

Table content: <table/>

Peer review (iclr2019):
 This paper aims to improve sequential prediction using LSTM by incorporating another latent variables {z_t} that follow distinct dynamics rather than LSTM itself. Experimental results on comparing with naive LSTM and dynamic word embedding approaches confirmed that the proposed method works better than these baselines. Basically the proposed method is sensible, but this paper has two clear drawbacks: (1) it is not compared with similar previous attempts; and (2) the proposed architecture is not guaranteed to represent "global" contextual behavior rather than local contextual one. (1) There have been numerous attempts in sequential modeling, especially language models, to incorporate a global and contextual factor into prediction. Early attempts used latent dirichlet allocation combined with n-gram models; afterwords, LSTM became prevalent, and in fact there exists a famous baseline of Latent LSTM allocation [1]. Therefore, it is minimally required for this paper to compare the proposed model with latent LSTM. (2) Latent LSTM above has a clear advantage that there is an explicit mechanism of latent topic distribution that governs a specific sequence (global variable). However, the proposed method in this paper, contextual behavior is determined only locally, and might be quite prone to overfit to exceptional observations, in this case, words. Of course, it might handle a global context by chance, but the architecture does not guarantee it. Therefore, such a potential disadvantage should be experimentally investigated by a comparison with a suitable baselines noted above. I agree with the authors that the contexual behavior should be continuous and flexible, such as pursued in this paper. Thus a challenge is how to suitably incorporate a global context into such a representation. Finally, I would like to know what kind of function g is learned through this experiment. Investigating into the strengths and weaknesses of the learned function g is also beneficial for a more flexible language models in the future. [1] Zaheer, M., Ahmed, A., & Smola, A. J. (2017). Latent LSTM Allocation: Joint Clustering and Non-Linear Dynamic Modeling of Sequence Data. In International Conference on Machine Learning (pp. 3967-3976).