Improving Sequential Latent Variable Models with Autoregressive Flows

We propose an approach for sequence modeling based on autoregressive normalizing flows . Each autoregressive transform , acting across time , serves as a moving reference frame for modeling higher-level dynamics . This technique provides a simple , general-purpose method for improving sequence modeling , with connections to existing and classical techniques . We demonstrate the proposed approach both with standalone models , as well as a part of larger sequential latent variable models . Results are presented on three benchmark video datasets , where flowbased dynamics improve log-likelihood performance over baseline models . 1 INTRODUCTION . Data often contain sequential structure , providing a rich signal for learning models of the world . Such models are useful for learning self-supervised representations of sequences ( Li & Mandt , 2018 ; Ha & Schmidhuber , 2018 ) and planning sequences of actions ( Chua et al. , 2018 ; Hafner et al. , 2019 ) . While sequential models have a longstanding tradition in probabilistic modeling ( Kalman et al. , 1960 ) , it is only recently that improved computational techniques , primarily deep networks , have facilitated learning such models from high-dimensional data ( Graves , 2013 ) , particularly video and audio . Dynamics in these models typically contain a combination of stochastic and deterministic variables ( Bayer & Osendorfer , 2014 ; Chung et al. , 2015 ; Gan et al. , 2015 ; Fraccaro et al. , 2016 ) , using simple distributions ( e.g . Gaussian ) to directly model the likelihood of data observations . However , attempting to capture all sequential dependencies with relatively unstructured dynamics may make it more difficult to learn such models . Intuitively , the model should use its dynamical components to track changes in the input instead of simultaneously modeling the entire signal . Rather than expanding the computational capacity of the model , we seek a method for altering the representation of the data to provide a more structured form of dynamics . To incorporate more structured dynamics , we propose an approach for sequence modeling based on autoregressive normalizing flows ( Kingma et al. , 2016 ; Papamakarios et al. , 2017 ) , consisting of one or more autoregressive transforms in time . A single transform is equivalent to a Gaussian autoregressive model . However , by stacking additional transforms or latent variables on top , we can arrive at more expressive models . Each autoregressive transform serves as a moving reference frame in which higher-level structure is modeled . This provides a general mechanism for separating different forms of dynamics , with higher-level stochastic dynamics modeled in the simplified space provided by lower-level deterministic transforms . In fact , as we discuss , this approach generalizes the technique of modeling temporal derivatives to simplify dynamics estimation ( Friston , 2008 ) . We empirically demonstrate this approach , both with standalone autoregressive normalizing flows , as well as by incorporating these flows within more flexible sequential latent variable models . While normalizing flows have been applied in a few sequential contexts previously , we emphasize the use of these models in conjunction with sequential latent variable models . We present experimental results on three benchmark video datasets , showing improved quantitative performance in terms of log-likelihood . In formulating this general technique for improving dynamics estimation in the framework of normalizing flows , we also help to contextualize previous work . 2 BACKGROUND . 2.1 AUTOREGRESSIVE MODELS . Consider modeling discrete sequences of observations , x1 : T ∼ pdata ( x1 : T ) , using a probabilistic model , pθ ( x1 : T ) , with parameters θ. Autoregressive models ( Frey et al. , 1996 ; Bengio & Bengio , 2000 ) use the chain rule of probability to express the joint distribution over all time steps as the product of T conditional distributions . Because of the forward nature of the world , as well as for handling variable-length sequences , these models are often formulated in forward temporal order : pθ ( x1 : T ) = T∏ t=1 pθ ( xt|x < t ) . ( 1 ) Each conditional distribution , pθ ( xt|x < t ) , models the temporal dependence between time steps , i.e . a prediction of the future . For continuous variables , we often assume that each distribution takes a relatively simple form , such as a diagonal Gaussian density : pθ ( xt|x < t ) = N ( xt ; µθ ( x < t ) , diag ( σ2θ ( x < t ) ) ) , ( 2 ) where µθ ( · ) and σθ ( · ) are functions denoting the mean and standard deviation , often sharing parameters over time steps . While these functions may take the entire past sequence of observations as input , e.g . through a recurrent neural network , they may also be restricted to a convolutional window ( van den Oord et al. , 2016a ) . Autoregressive models can also be applied to non-sequential data ( van den Oord et al. , 2016b ) , where they excel at capturing local dependencies . However , due to their restrictive distributional forms , such models often struggle to capture higher-level structure . 2.2 AUTOREGRESSIVE LATENT VARIABLE MODELS . Autoregressive models can be improved by incorporating latent variables , often represented as a corresponding sequence , z1 : T . Classical examples include Gaussian state space models and hidden Markov models ( Murphy , 2012 ) . The joint distribution , pθ ( x1 : T , z1 : T ) , has the following form : pθ ( x1 : T , z1 : T ) = T∏ t=1 pθ ( xt|x < t , z≤t ) pθ ( zt|x < t , z < t ) . ( 3 ) Unlike the simple , parametric form in Eq . 2 , evaluating pθ ( xt|x < t ) now requires integrating over the latent variables , pθ ( xt|x < t ) = ∫ pθ ( xt|x < t , z≤t ) pθ ( z≤t|x < t ) dz≤t , ( 4 ) yielding a more flexible distribution . However , performing this integration in practice is typically intractable , requiring approximate inference techniques , like variational inference ( Jordan et al. , 1998 ) . Recent works have parameterized these models with deep neural networks , e.g . ( Chung et al. , 2015 ; Gan et al. , 2015 ; Fraccaro et al. , 2016 ; Karl et al. , 2017 ) , using amortized variational inference ( Kingma & Welling , 2014 ; Rezende et al. , 2014 ) for inference and learning . Typically , the conditional likelihood , pθ ( xt|x < t , z≤t ) , and the prior , pθ ( zt|x < t , z < t ) , are Gaussian densities , with temporal conditioning handled through deterministic recurrent networks and the stochastic latent variables . Such models have demonstrated success in audio ( Chung et al. , 2015 ; Fraccaro et al. , 2016 ) and video modeling ( Xue et al. , 2016 ; Gemici et al. , 2017 ; Denton & Fergus , 2018 ; He et al. , 2018 ; Li & Mandt , 2018 ) . However , design choices for these models remain an active area of research , with each model proposing new combinations of deterministic and stochastic dynamics . 2.3 AUTOREGRESSIVE FLOWS . Our approach is based on affine autoregressive normalizing flows ( Kingma et al. , 2016 ; Papamakarios et al. , 2017 ) . Here , we review this basic concept , continuing with the perspective of temporal sequences , however , it is worth noting that these flows were initially developed and demonstrated in static settings . Kingma et al . ( 2016 ) noted that sampling from an autoregressive Gaussian model is an invertible transform , resulting in a normalizing flow ( Rippel & Adams , 2013 ; Dinh et al. , 2015 ; 2017 ; Rezende & Mohamed , 2015 ) . Flow-based models transform between simple and complex probability distributions while maintaining exact likelihood evaluation . To see their connection to autoregressive models , we can express sampling a Gaussian random variable , xt ∼ pθ ( xt|x < t ) ( Eq . 2 ) , using the reparameterization trick ( Kingma & Welling , 2014 ; Rezende et al. , 2014 ) : xt = µθ ( x < t ) + σθ ( x < t ) yt , ( 5 ) where yt ∼ N ( yt ; 0 , I ) is an auxiliary random variable and denotes element-wise multiplication . Thus , xt is an invertible transform of yt , with the inverse given as yt = xt − µθ ( x < t ) σθ ( x < t ) , ( 6 ) where division is performed element-wise . The inverse transform in Eq . 6 acts to normalize ( hence , normalizing flow ) and therefore decorrelate x1 : T . Given the functional mapping between yt and xt in Eq . 5 , the change of variables formula converts between probabilities in each space : log pθ ( x1 : T ) = log pθ ( y1 : T ) − log ∣∣∣∣det ( ∂x1 : T∂y1 : T ) ∣∣∣∣ . ( 7 ) By the construction of Eqs . 5 and 6 , the Jacobian in Eq . 7 is triangular , enabling efficient evaluation as the product of diagonal terms : log ∣∣∣∣det ( ∂x1 : T∂y1 : T ) ∣∣∣∣ = T∑ t=1 ∑ i log σθ , i ( x < t ) , ( 8 ) where i denotes the observation dimension , e.g . pixel . For a Gaussian autoregressive model , pθ ( y1 : T ) = N ( y1 : T ; 0 , I ) . With these components , the change of variables formula ( Eq . 7 ) provides an equivalent method for sampling and evaluating the model , pθ ( x1 : T ) , from Eqs . 1 and 2 . We can improve upon this simple set-up by chaining together multiple transforms , effectively resulting in a hierarchical autoregressive model . Letting ym1 : T denote the variables after them th transform , the change of variables formula for M transforms is log pθ ( x1 : T ) = log pθ ( y M 1 : T ) − log ∣∣∣∣det ( ∂x1 : T∂y11 : T ) ∣∣∣∣−M−1∑ m=1 log ∣∣∣∣det ( ∂ym1 : T∂ym+11 : T ) ∣∣∣∣ . ( 9 ) Autoregressive flows were initially considered in the contexts of variational inference ( Kingma et al. , 2016 ) and generative modeling ( Papamakarios et al. , 2017 ) . These approaches are , in fact , generalizations of previous approaches with affine transforms ( Dinh et al. , 2015 ; 2017 ) . While autoregressive flows are well-suited for sequential data , as mentioned previously , these approaches , as well as many recent approaches ( Huang et al. , 2018 ; Oliva et al. , 2018 ; Kingma & Dhariwal , 2018 ) , were initially applied in static settings , such as images . More recent works have started applying flow-based models to sequential data . For instance , van den Oord et al . ( 2018 ) and Ping et al . ( 2019 ) distill autoregressive speech models into flow-based models . Prenger et al . ( 2019 ) and Kim et al . ( 2019 ) instead train these models directly . Kumar et al . ( 2019 ) use a flow to model individual video frames , with an autoregressive prior modeling dynamics across time steps . Rhinehart et al . ( 2018 ) and Rhinehart et al . ( 2019 ) use autoregressive flows for modeling vehicle motion , and Henter et al . ( 2019 ) use flows for motion synthesis with motion-capture data . Ziegler & Rush ( 2019 ) learn distributions over sequences of discrete observations ( e.g. , text ) by using flows to model dynamics of continuous latent variables . Like these recent works , we apply flow-based models to sequential data . However , we demonstrate that autoregressive flows can serve as a useful , general-purpose technique for improving sequence modeling as components of sequential latent variable models . To the best of our knowledge , our work is the first to focus on the aspect of using flows to pre-process sequential data to improve downstream dynamics modeling . Finally , we utilize affine flows ( Eq . 5 ) in this work . This family of flows includes methods like NICE ( Dinh et al. , 2015 ) , RealNVP ( Dinh et al. , 2017 ) , IAF ( Kingma et al. , 2016 ) , MAF ( Papamakarios et al. , 2017 ) , and GLOW ( Kingma & Dhariwal , 2018 ) . However , there has been recent work in non-affine flows ( Huang et al. , 2018 ; Jaini et al. , 2019 ; Durkan et al. , 2019 ) , which may offer further flexibility . We chose to investigate affine flows for their relative simplicity and connections to previous techniques , however , the use of non-affine flows could result in additional improvements .

Summary: The paper proposes to combine the video modeling approaches based on autoregressive flows (e.g. Kumar’19) with amortized variational inference (e.g. Denton’18), wherein an autoregressive latent variable model optimized with variational inference is extended with an autoregressive flow that further transforms the output of the latent variable model while allowing to compute exact conditional probability. This is motivated with a physical intuition, where a dynamics model can benefit from decorrelating the inputs, and it is demonstrated that layers of autoregressive flows can represent derivatives of the original signal. In a proof-of-concept experiment, it is shown that using a layer of autoregressive flow improves NLL of a latent variable model.