Variational recurrent models for representation learning

1 INTRODUCTION . Modeling sequence data is a central problem in domains such as speech and natural language processing . In this work , we study the problem of learning representations ( features ) of sequence data , with varying degrees of supervision , that can be helpful for downstream sequence labeling tasks . We take a generative approach , inspired by variational autoencoders ( VAEs ) Kingma & Welling ( 2014 ) ; Rezende et al . ( 2014 ) ; Doersch ( 2016 ) for non-sequence data . We assume that the sequence data is generated by a series of latent variables , parameterize the observation likelihood with recurrent neural networks , and maximize ( a lower bound on ) the observation likelihood . If labels are available during training , a discriminative loss can also be imposed on the latent variables . The learned posterior distribution of the latent variables then provides features for the downstream tasks . This intuitively simple approach , however , requires non-trivial extensions of the deep generative models that have been successful for non-sequence data . For a ( non-sequence ) observation x , a VAE assumes a prior distribution , p ( z ) , on the latent variable z which is often simple ( e.g. , N ( 0 , I ) ) , and consists of a generation network parameterizing p ( x|z ) with weights θ and an inference network parameterizing the approximate posterior q ( z|x ) with weights φ . Learning the VAE is done by maximizing an evidence lower bound ( ELBO ) on the observation log-likelihood log pθ ( x ) = log ∫ p ( z ) pθ ( x|z ) dz , defined as ELBO : = Eqφ ( z|x ) [ log pθ ( x|z ) ] −DKL ( qφ ( z|x ) ||p ( z ) ) ≤ log pθ ( x ) . ( 1 ) VAEs can be viewed as autoencoders : The encoder q ( z|x ) maps the input to a latent variable z according to a ( also typically Gaussian ) posterior distribution , and the decoder p ( x|z ) reconstructs the input from samples of z drawn from this posterior . The output of the encoder ( most commonly , the mean of q ( z|x ) ) is often used as a learned representation ( features ) for downstream tasks . Maximizing the ELBO is equivalent to minimizing the reconstruction loss of the output x with respect to the input , plus a regularizer based on the prior distribution of the latent variable . In some variants the importance of the KL term is tuned by weighting the regularizer by a hyperparameter β Higgins et al . ( 2016 ) ; Alemi et al . ( 2016 ) . VAEs have been extended to model sequence data in a number of ways . Fabius & van Amersfoort ( 2014 ) learn a single representation for the entire sequence , while Hsu et al . ( 2017 ) learn both a whole-sequence representation and a set of representations for pre-defined segments within the sequence . For many tasks , such as the ones we consider here , it is desirable to represent a length-T input sequence x1 : T with a corresponding length-T latent sequence z1 : T so as to fit directly into typical recurrent network-based prediction models . Several recent approaches fit this criterion Krishnan et al . ( 2015 ) ; Archer et al . ( 2015 ) ; Chung et al . ( 2015b ) ; Fraccaro et al . ( 2016 ) ; Goyal et al . ( 2017a ) ; Chen et al . ( 2018 ) . Learning is again done by maximizing an ELBO , with the main differences among approaches being the specific forms of the prior p ( z1 : T ) ( typically parameterized so as to capture dynamics in the latent space ) , the generation distribution pθ ( x1 : T |z1 : T ) , and the approximate posterior qφ ( z1 : T |x1 : T ) . For example , direct recurrent connections between stochastic variables Fraccaro et al . ( 2016 ) , as shown in Figure 1 ( a ) , or indirect recurrent connections , e.g . ht−1 → zt−1 → ht → zt Chung et al . ( 2015b ) ; Goyal et al . ( 2017a ) , are often introduced in p ( z1 : T ) to model the dependence between neighboring latent variables . While this is more powerful than a simpler prior for the purpose of generation , it poses challenges for designing the approximate posteriors due to the dependencies among zt ’ s . On the other hand , given z1 : T , the generation model is often fully factorized into T independent terms : pθ ( x1 : T |z1 : T ) = ∏T t=1 pθ ( xt|zt ) . Most prior work on variational recurrent models has focused on generation quality and likelihood evaluation . It is not clear , however , that the learned representations are useful for downstream tasks . The goal of our work is to fill this gap by developing variational recurrent models that produce high-quality representations for prediction tasks , and to investigate several modeling choices . We take an approach , similarly to Chen et al . ( 2018 ) , where the latent distribution p ( z1 : T ) factors over time conditioned on a deterministic hidden state sequence , while the generation model pθ ( x1 : T |z1 : T ) is factored as ∏T t=1 pθ ( xt|z1 : T ) , as shown in Figures 1 ( b , d ) . Also similarly to Chen et al . ( 2018 ) , we use a simple Gaussian prior which is updated during training , and find that this can be very helpful . Unlike Chen et al . ( 2018 ) , however , we consider a wide variety of tasks and levels of supervision . Compared to variational recurrent models with recurrent connections among the stochastic latent variables , this class of models is more efficient and easier to implement , since there are no dependencies between stochastic latent variables and therefore no nested sampling procedure . One key novelty in this work is that , while we have no recurrent connections among the z1 : T , we require each zj to generate each xk with some probability , for all 1 ≤ j , k ≤ T , so that the features are predictive of a large context of observations . We propose a “ stochastic generation ” model that approximates this idea , and we find that this extension is important for improving downstream performance . In addition , we study the proposed models on a range of speech and NLP prediction tasks , in a variety of supervision settings , finding improved performance in all cases . We consider fully supervised , semi-supervised , and weakly supervised ( using multi-view training data ) settings . We believe this is the first attempt at a broad study of recurrent variational representation learning that is applicable to a variety of downstream tasks and levels of supervision . 2 A VARIATIONAL RECURRENT REPRESENTATION LEARNING MODEL . Our variational recurrent representation learning approach , shown in Figure 1 ( b-d ) , aims to learn informative representations while keeping inference simple . As in other recurrent variational models , we optimize an ELBO of the same form as in Equation 1 but where the variables are all sequences of length T ; that is , the prior is p ( z1 : T ) , the posterior is qφ ( z1 : T |x1 : T ) , and the generation model is pθ ( x1 : T |z1 : T ) . 2.1 INFERENCE . Given a length T input x1 : T , the model first maps from the input sequence to a sequence of deterministic recurrent hidden state vectors h1 : T , produced by one or more stacked recurrent neural network layers . Conditioned on ht ( 1 ≤ t ≤ T ) , the posterior distribution of the latent random variable zt is a spherical Gaussian zt|ht ∼ qφ ( zt|ht ) ≡ N ( µφt , diag ( σφt ) ) , [ µφt , log σφt ] = Fφ ( ht ) ( 2 ) Here , qφ is the inference network with parameters φ and Fφ is either a linear transformation or a feedforward neural network with a linear final layer . The full posterior and prior are qφ ( z1 : T |h1 : T ) : = T ∏ t=1 qφ ( zt|ht ) , p ( z1 : T ) : = T ∏ t=1 p ( zt ) ( 3 ) Finally , a reconstruction of the input sequence is generated from the latent stochastic sequence z1 : T . The learned representation of xt , which can be used for downstream tasks , is taken to be the mean µφt of qφ ( zt|ht ) . Note that no direct dependence is assumed among different time steps of z1 : T given h1 : T . This greatly simplifies inference over prior work that directly models dependence between the zt Goyal et al . ( 2017a ) ; Fraccaro et al . ( 2016 ) ; Chung et al . ( 2015b ) ; Krishnan et al . ( 2015 ) , since the distribution of each zt ( given ht ) is a single spherical Gaussian rather than a more complex multimodal distribution that would arise with dependence modeling ( e.g. , using recurrent connections among stochastic nodes ) . Consider a model with stochastic recurrent connections as in Figure 1 ( a ) , which models the conditional probability of zt given the past z1 : t−1 . The marginal distribution of each zt is multimodal , and requires nested sampling . In order to obtain good estimates , a large number of samples of z1 : t−1 may need to be used . In addition , in a model with multimodal marginal distributions of zt , it is not clear what to consider to be the learned representation for use in a downstream tasks ; in particular the mean of zt may have very small probability and thus may not be a good choice . 2.2 STOCHASTIC GENERATION . Whereas our approach simplifies inference , the generation of each xk for 1 ≤ k ≤ T in our model involves samples of all z1 : T , with each zt for 1 ≤ t ≤ T having a different effect on the generation of xk , as shown in Figures 1 ( c , d ) . Intuitively , this generation approach reintroduces some of the relationships between the zt that may have been lost by having no direct dependence modeling . Given a particular choice of k , t , the generative model for xk given zt is also a spherical Gaussian : xk|zt ∼ N ( µθt , diag ( σθt ) ) = pθ ( xk|zt ) , [ µθt , log σθt ] = Fθ ( zt ) ( 4 ) where again Fθ , with parameters θ , can be linear or a feedforward neural network with a linear final layer . Note that the full generative distribution ( Figure 1 ( d ) ) is multimodal , although we never explicitly evaluate it , as described below . Figure 1 ( d ) shows our full generation process . However , this generation process is a bit expensive ; instead we employ stochastic generation ( Figure 1 ( c ) ) , a technique that aims to approximate the generation process . In stochastic generation , we first randomly select a frame xk from x1 : T for time step t. During training we evaluate Eqφ ( zt|ht ) [ log pθ ( xk|zt ) ] for time step t. The way we sample the target to be reconstructed for time step t affects the way zt and ht encode x1 : T . The higher the probability that a distant time frame is selected , the more temporal information should be encoded in the latent variables . Next we describe stochastic generation more formally . Definition 2.1. α ( t , k ) δ , T Given x1 : T and Gaussian distribution p ( t ) ( s ) ≡ N ( t , δ ) , where 1 ≤ t ≤ T . Let c0 : T = { −∞ , 1.5 , 2.5 , ... , T − 0.5 , +∞ } . We define α ( t , k ) δ , T for 1 ≤ t , k ≤ T as α ( t , k ) δ , T : = ∫ ck ck−1 p ( t ) ( s ) ds ( 5 ) Definition 2.2 . Stochastic Generation Given δ and sequence x1 : T , stochastic generation reconstructs xk using samples of zt with probability α ( t , k ) δ , T . That is , the conditional log-likelihood for time step t is Eqφ ( zt|ht ) [ log pθ ( xk|zt ) ] with probability α ( t , k ) δ , T . In stochastic generation , each zt contributes to xk differently based on the delay between time steps t and k. If δ is chosen such that only nearby neighbor xks are produced , it becomes a weighted window generation . Remark 1 shows how a particular instance of stochastic generation relates to a full generation model , with graphical model as described in Figure 1 ( d ) . Remark 1 . Given z1 : T , φ and θ , stochastic generation in expectation is evaluating T ∑ t=1 { Eqφ ( zt|ht ) [ T ∑ k=1 α ( t , k ) δ , T log pθ ( xk|zt ) ] } ( 6 ) Equation 6 is equivalent to computing the expectation of log pθ ( x1 : T |z1 : T ) , where pθ ( x1 : T |z1 : T ) = T ∏ t=1 pθ ( xt|z1 : T ) = T ∏ t=1 T ∏ k=1 pθ ( xk|zt ) α ( t , k ) δ , T ( 7 ) It is straightforward to show that the ELBO from the two perspectives is identical ( see the supplementary material ) . Through stochastic generation , although we are always using simple Gaussian distributions , we are implicitly creating a more complex generative model . Note that , for purposes of generation and likelihood evaluation , we directly use the generative model described in Equation 7 . More detailed derivations can be found in the supplementary material .

Summary: This is largely an experimental paper, proposing and evaluating various modifications of variational recurrent models towards obtaining sequence data representations that are effective in downstream tasks. The highlighted contribution is a "stochastic generation" training procedure in which the training objective evaluates the reconstruction of output sequence elements from individual latent variables independently. The main claim is that the resulting model, augmented with prior updating and/or hierarchical latent variables, improves results w.r.t. the baselines.