Variational Hyper RNN for Sequence Modeling

In this work , we propose a novel probabilistic sequence model that excels at capturing high variability in time series data , both across sequences and within an individual sequence . Our method uses temporal latent variables to capture information about the underlying data pattern and dynamically decodes the latent information into modifications of weights of the base decoder and recurrent model . The efficacy of the proposed method is demonstrated on a range of synthetic and real-world sequential data that exhibit large scale variations , regime shifts , and complex dynamics . 1 INTRODUCTION . Recurrent neural networks ( RNNs ) are the natural architecture for sequential data as they can handle variable-length input and output sequences . Initially invented for natural language processing , long short-term memory ( LSTM ; Hochreiter & Schmidhuber 1997 ) , gated recurrent unit ( GRU ; Cho et al . 2014 ) as well as the later attention-augmented versions ( Vaswani et al. , 2017 ) have found wide-spread successes from language modeling ( Mikolov et al. , 2010 ; Kiros et al. , 2015 ; Jozefowicz et al. , 2016 ) and machine translation ( Bahdanau et al. , 2014 ) to speech recognition ( Graves et al. , 2013 ) and recommendation systems ( Wu et al. , 2017 ) . However , RNNs use deterministic hidden states to process input sequences and model the system dynamics using a set of time-invariant weights , and they do not necessarily have the right inductive bias for time series data outside the originally intended domains . Many natural systems have complex feedback mechanisms and numerous exogenous sources of variabilities . Observations from such systems would contain large variations both across sequences in a dataset as well as within any single sequence ; the dynamics could be switching regimes drastically , and the noise process could also be heteroskedastic . To capture all these intricate patterns in RNN with deterministic hidden states and a fixed set of weights requires learning about the patterns , the subtle deviations from the patterns , the conditions under which regime transitions occur which is not always predictable . Outside of the deep learning literature , many time series models have been proposed to capture specific types of high variabilities . For instance , switching linear dynamical models ( Ackerson & Fu , 1970 ; Ghahramani & Hinton , 1996 ; Murphy , 1998 ; Fox et al. , 2009 ) aim to model complex dynamical systems with a set of simpler linear patterns . Conditional volatility models ( Engle , 1982 ; Bollerslev , 1986 ) are introduced to model time series with heteroscedastic noise process whose noise level itself is a part of the dynamics . However , these models usually encode specific inductive biases in a hard way , and can not learn different behaviors and interpolate among the learned behaviors as deep neural nets . In this work , we propose a new class of neural recurrent latent variable model , called the variational hyper RNN ( VHRNN ) , which can perform dynamic regime identification and re-identification dynamically at inference time . Our model captures complex time series without encoding a large number of patterns in static weights , but instead only encodes base dynamics that can be selected and adapted based on run time observations . Thus it can easily learn to express a rich set of behaviors including but not limited to the ones mentioned above . Our model can dynamically identify the underlying pattern , express uncertainty due to observation noise , lack of information , or model misspecification . As such , VHRNN can model complex patterns with fewer parameters ; and when given lots of parameters , it generalizes better than previous methods . The VHRNN is built upon the previous variational RNN ( VRNN ) models ( Chung et al. , 2015 ) and hypernetworks ( Ha et al. , 2016 ) . The VRNN models introduce stochastic latent variables at every time step , which are inferred using a variational recognition model . The overall model is trained by maximizing the evidence lower bound ( ELBO ) . In VRNN , the latent variables capture the information in the stochastic hidden states and are then fed as input to the RNN and decoding model to produce reconstructed observations . While in our work , the latent variables are decoded to produce the RNN transition weights and observation projection weights in the style of hypernetworks ( Ha et al. , 2016 ) , i.e. , dynamically generating the scaling and bias vectors to adjust the base weights of the RNN . We demonstrate that the proposed VHRNN model is better at capturing different types of variability on several synthetic as well as real-world time series datasets . 2 BACKGROUND AND RELATED WORK . Variational Autoencoder Variational autoencoder ( VAE ) is one of the most popular unsupervised approaches to learning a compact representation from data ( Kingma & Welling , 2013 ) . It uses a variational distribution q ( z|x ) to approximate the intractable posterior distribution of the latent variable z . With the use of variational approximation , it maximizes the evidence lower bound ( ELBO ) of the marginal log-likelihood of data L ( x ) = Eq ( z|x ) [ log p ( x|z ) ] −DKL ( q ( z|x ) ‖ p ( z ) ) ≤ log p ( x ) , where p ( z ) is a prior distribution of z and DKL denotes the Kullback–Leibler ( KL ) divergence . The approximate posterior q ( z|x ) is usually formulated as a Gaussian with a diagonal covariance matrix . Variational RNN for Sequential Data Variational autoencoders have demonstrated impressive performance on non-sequential data like images . Many following works ( Bowman et al. , 2015 ; Chung et al. , 2015 ; Fraccaro et al. , 2016 ; Luo et al. , 2018 ) extend the domain of VAE models to sequential data . Among them , variational RNN ( VRNN ; Chung et al . 2015 ) further incorporate a latent variable at each time step into their models . A prior distribution conditioned on the contextual information and a variational posterior is proposed at each time step to optimize a step-wise variational lower bound . Sampled latent variables from the variational posterior are decoded into the observation at the current time step . The VHRNN model makes use of the same factorization of sequential data and joint distribution of latent variables as in VRNN . However , in VHRNN model , the latent variables also parameterize the weights for decoding and transition in RNN cell across time steps , giving the model more flexibility to deal with variations within and across sequences . Importance Weighted Autoencoder and Filtering Variational Objective A parallel stream of work to improve latent variable models with variational inference study tighter bounds of the data ’ s log-probability than ELBO . Importance Weighted Autoencoder ( IWAE ; Burda et al . 2016 ) estimates a different variational bound of the log-likelihood , which is provably tighter than ELBO . Filtering Variational Objective ( FIVO ; Maddison et al . 2017 ) exploits the temporal structure of sequential data and uses particle filtering to estimate the data log-likelihood . FIVO still computes a step-wise IWAE bound based on the sampled particles at each time step , but it shows better sampling efficiency and tightness than IWAE . We use FIVO as the objective to train and evaluate our models . HyperNetworks Our model is motivated by HyperNetworks ( Ha et al. , 2016 ) which use one network to generate the parameters of another . The dynamic version of HyperNetworks can be applied to sequence data , but due to lack of latent variables , can only capture uncertainty in the output variables . For discrete sequence data such as text , categorical output variables can model multi-model outputs very well ; but on continuous time series with the typical Gaussian output variables , the model is much less capable at dealing with stochasticity . Furthermore , it does not allow straightforward interpretation of the model behaviour using the time-series of KL divergence as we do in Sec . 4 . With the augmentation of latent variables , VHRNN is much more capable of modelling uncertainty . It is worth noting that Bayesian HyperNetworks ( Krueger et al. , 2017 ) also have a latent variable in the context of Hypernetworks . However , the goal of Bayesian HypernNtwork is an improved version of Bayesian neural net to capture model uncertainty . The work of Krueger et al . ( 2017 ) has no recurrent structure and can not be applied to sequential data . Furthermore , the use of normalizing flow dramatically limits the flexibility of the decoder architecture design , unlike in VHRNN . Dezfouli et al . ( 2019 ) recently proposed to learn a disentangled low-dimensional latent space such that samples from the latent space are used to generate parameters of an RNN model . Different latent variables could account for the behaviours of different subjects . In spite of similarity in combining RNN with HyperNetworks , the motivations and architectures are fundamentally different . The work of Dezfouli et al . ( 2019 ) intends to learn a low-dimensional interpretable latent space that represents the difference between subjects in decision-making process while our models tries to better handle the variances both within and across sequences in general time-series data modeling . The work of Dezfouli et al . ( 2019 ) generate a set of parameters that is shared across time steps for each latent variable . In contrast , our model samples a latent variable and dynamically generates non-shared weights at each time step , which we believe is essential for handling variance of dynamics within sequences . 3 MODEL FORMULATION . Variational Hyper RNN A recurrent neural network ( RNN ) can be characterized by ht = gθ ( xt , ht−1 ) , where xt and ht are the observation and hidden state of the RNN at time step t , and θ is the fixed weights of the RNN model . The hidden state ht is often used to generate the output for other learning tasks , e.g. , predicting the observation at the next time step . We augment the RNN with a latent random variable zt , which is also used to output the non-shared parameters of the RNN at time step t. ht = gθ ( zt , ht−1 ) ( xt , zt , ht−1 ) , ( 1 ) where θ ( zt , ht−1 ) is a hypernetwork that generates the parameters of the RNN at time step t. The latent variable zt can also be used to determine the parameters of the generative model p ( xt|z≤t , x < t ) : xt|z≤t , x < t ∼ N ( µdect , Σ dec t ) , where ( µ dec t , Σ dec t ) = φ dec ω ( zt , ht−1 ) ( zt , ht−1 ) . ( 2 ) We hypothesize that the previous observations and latent variables , characterized by ht−1 , define a prior distribution p ( zt|x < t , z < t ) over the latent variable zt , zt|x < t , z < t ∼ N ( µpriort , Σ prior t ) , where ( µ prior t , Σ prior t ) = φ prior ( ht−1 ) . ( 3 ) Eq . 2 and 3 result in the following generation process of sequential data : p ( x≤T , z≤T ) = T∏ t=1 p ( zt|x < t , z < t ) p ( xt|x < t , z≤t ) . ( 4 ) The true posterior distributions of zt conditioned on observations x≤t and latent variables z < t are intractable , posing a challenge in both sampling and learning . Therefore , we introduce an approximate posterior q ( zt|x≤t , z < t ) such that zt|x≤t , z < t ∼ N ( µenct , Σ enc t ) , where ( µ enc t , Σ enc t ) = φ enc ( xt , ht−1 ) . ( 5 ) This approximate posterior distribution enables the model to be trained by maximizing a variational lower bound , e.g. , ELBO ( Kingma & Welling , 2013 ) , IWAE ( Burda et al. , 2016 ) and FIVO ( Maddison et al. , 2017 ) . We refer to the main components of our model , including g , φdec , φenc , φprior as primary networks and refer to the components responsible for generating parameters , θ and ω , as hyper networks in the following sections . Implementation Following the practice of VAE , we parametrize the covariance matrices Σpriort , Σdect and Σ enc t as diagonal matrices . Note that Σ prior t in our model is no longer an identity matrix as in a vanilla VAE ; it is the output of φprior and depends on the hidden state ht−1 at the previous time step . The recurrence model g in Eq . 1 is implemented as an RNN cell , which takes as input xt and zt at each time step t and updates the hidden states ht−1 . The parameters of g are generated by the hyper network θ ( zt , ht−1 ) , as illustrated in Figure 1b . θ is also implemented using an RNN to capture the history of data dynamics , with zt and ht−1 as input at each time step t. However , it is computationally costly to generate all the parameters of g using θ ( zt , ht−1 ) . Following the practice of previous works ( Ha et al. , 2016 ; Krueger et al. , 2017 ) , the hyper network θ maps zt and ht−1 to bias and scaling vectors . The scaling vectors modify the parameters of g by scaling each row of the weight matrices , routing information in the input and hidden state vectors through different channels . To better illustrate this mechanism , we exemplify the recurrence model g using an RNN cell with LSTM-style update rules and gates . Let ∗ ∈ { i , f , g , o } denote the one of the four LSTM-style gates in g. W∗ and U∗ denote the input and recurrent weights of each gate in LSTM cell respectively . The hyper network θ ( zt , ht−1 ) outputs di∗ and dh∗ that are the scaling vectors for the input weights W∗ and recurrent weights U∗ of the recurrent model g in Eq . 1 . The overall implementation of g in Eq . 1 can be described as follows : it = σ ( dii ( zt , ht−1 ) ◦ ( Wiyt ) + dhi ( zt , ht−1 ) ◦ ( Uiht−1 ) ) , ft = σ ( dif ( zt , ht−1 ) ◦ ( Wfyt ) + dhf ( zt , ht−1 ) ◦ ( Ufht−1 ) ) , gt = tanh ( dig ( zt , ht−1 ) ◦ ( Wgyt ) + dhg ( zt , ht−1 ) ◦ ( Ught−1 ) ) , ot = σ ( dio ( zt , ht−1 ) ◦ ( Woyt ) + dho ( zt , ht−1 ) ◦ ( Uoht−1 ) ) , ct = ft ◦ ct−1 + it ◦ gt , ht = ot ◦ tanh ( ct ) , where ◦ denotes the Hadamard product and yt is a fusion ( e.g. , concatenation ) of observation xt and latent variable zt . For simplicity of notation , bias terms are ignored from the above equations . This implementation of the recurrence model in VHRNN is further illustrated in the diagram in Fig . 6 in appendix . Another hyper network ω ( zt , ht−1 ) generates the parameters of the generative model in Eq . 2 . It is implemented as a multilayer perceptron ( MLP ) . Similar to θ ( zt , ht−1 ) , the outputs are the bias and scaling vectors that modify the parameters of the decoder φdecω ( zt , ht−1 ) .

Summary: This paper proposes a variational hyper recurrent neural network which is a combination of the variational RNN and the hypernetwork. The hypernetwork is an RNN whose output modifies the parameters of the variational RNN dynamically at runtime. Overall, this seems like an extension of the idea of using a hypernetwork with the VRNN (rather than the RNN as done in Ha. et. al). The model is trained via the FIVO objective. The model and learning algorithm are compared to the variational RNN and tested on a variety of synthetic settings where the VHRNN outperforms the VRNN in held-out likelihood. The performance gains are investigated on synthetic datasets where the paper notes that the VHRNN is often quicker to adapt variations that happen within seqences (for example, the paper considers a dataset where multiple patterns are stitched together into a sequence and study the changes in the KL divergence and reconstruction at switch points). On four real-world sequential datasets, the paper finds that the model outperforms the VRNN across many configurations and with a fewer number of parameters.