Improving Sequential Latent Variable Models with Autoregressive Flows

We propose an approach for sequence modeling based on autoregressive normalizing flows . Each autoregressive transform , acting across time , serves as a moving reference frame for modeling higher-level dynamics . This technique provides a simple , general-purpose method for improving sequence modeling , with connections to existing and classical techniques . We demonstrate the proposed approach both with standalone models , as well as a part of larger sequential latent variable models . Results are presented on three benchmark video datasets , where flowbased dynamics improve log-likelihood performance over baseline models . 1 INTRODUCTION . Data often contain sequential structure , providing a rich signal for learning models of the world . Such models are useful for learning self-supervised representations of sequences ( Li & Mandt , 2018 ; Ha & Schmidhuber , 2018 ) and planning sequences of actions ( Chua et al. , 2018 ; Hafner et al. , 2019 ) . While sequential models have a longstanding tradition in probabilistic modeling ( Kalman et al. , 1960 ) , it is only recently that improved computational techniques , primarily deep networks , have facilitated learning such models from high-dimensional data ( Graves , 2013 ) , particularly video and audio . Dynamics in these models typically contain a combination of stochastic and deterministic variables ( Bayer & Osendorfer , 2014 ; Chung et al. , 2015 ; Gan et al. , 2015 ; Fraccaro et al. , 2016 ) , using simple distributions ( e.g . Gaussian ) to directly model the likelihood of data observations . However , attempting to capture all sequential dependencies with relatively unstructured dynamics may make it more difficult to learn such models . Intuitively , the model should use its dynamical components to track changes in the input instead of simultaneously modeling the entire signal . Rather than expanding the computational capacity of the model , we seek a method for altering the representation of the data to provide a more structured form of dynamics . To incorporate more structured dynamics , we propose an approach for sequence modeling based on autoregressive normalizing flows ( Kingma et al. , 2016 ; Papamakarios et al. , 2017 ) , consisting of one or more autoregressive transforms in time . A single transform is equivalent to a Gaussian autoregressive model . However , by stacking additional transforms or latent variables on top , we can arrive at more expressive models . Each autoregressive transform serves as a moving reference frame in which higher-level structure is modeled . This provides a general mechanism for separating different forms of dynamics , with higher-level stochastic dynamics modeled in the simplified space provided by lower-level deterministic transforms . In fact , as we discuss , this approach generalizes the technique of modeling temporal derivatives to simplify dynamics estimation ( Friston , 2008 ) . We empirically demonstrate this approach , both with standalone autoregressive normalizing flows , as well as by incorporating these flows within more flexible sequential latent variable models . While normalizing flows have been applied in a few sequential contexts previously , we emphasize the use of these models in conjunction with sequential latent variable models . We present experimental results on three benchmark video datasets , showing improved quantitative performance in terms of log-likelihood . In formulating this general technique for improving dynamics estimation in the framework of normalizing flows , we also help to contextualize previous work . 2 BACKGROUND . 2.1 AUTOREGRESSIVE MODELS . Consider modeling discrete sequences of observations , x1 : T ∼ pdata ( x1 : T ) , using a probabilistic model , pθ ( x1 : T ) , with parameters θ. Autoregressive models ( Frey et al. , 1996 ; Bengio & Bengio , 2000 ) use the chain rule of probability to express the joint distribution over all time steps as the product of T conditional distributions . Because of the forward nature of the world , as well as for handling variable-length sequences , these models are often formulated in forward temporal order : pθ ( x1 : T ) = T∏ t=1 pθ ( xt|x < t ) . ( 1 ) Each conditional distribution , pθ ( xt|x < t ) , models the temporal dependence between time steps , i.e . a prediction of the future . For continuous variables , we often assume that each distribution takes a relatively simple form , such as a diagonal Gaussian density : pθ ( xt|x < t ) = N ( xt ; µθ ( x < t ) , diag ( σ2θ ( x < t ) ) ) , ( 2 ) where µθ ( · ) and σθ ( · ) are functions denoting the mean and standard deviation , often sharing parameters over time steps . While these functions may take the entire past sequence of observations as input , e.g . through a recurrent neural network , they may also be restricted to a convolutional window ( van den Oord et al. , 2016a ) . Autoregressive models can also be applied to non-sequential data ( van den Oord et al. , 2016b ) , where they excel at capturing local dependencies . However , due to their restrictive distributional forms , such models often struggle to capture higher-level structure . 2.2 AUTOREGRESSIVE LATENT VARIABLE MODELS . Autoregressive models can be improved by incorporating latent variables , often represented as a corresponding sequence , z1 : T . Classical examples include Gaussian state space models and hidden Markov models ( Murphy , 2012 ) . The joint distribution , pθ ( x1 : T , z1 : T ) , has the following form : pθ ( x1 : T , z1 : T ) = T∏ t=1 pθ ( xt|x < t , z≤t ) pθ ( zt|x < t , z < t ) . ( 3 ) Unlike the simple , parametric form in Eq . 2 , evaluating pθ ( xt|x < t ) now requires integrating over the latent variables , pθ ( xt|x < t ) = ∫ pθ ( xt|x < t , z≤t ) pθ ( z≤t|x < t ) dz≤t , ( 4 ) yielding a more flexible distribution . However , performing this integration in practice is typically intractable , requiring approximate inference techniques , like variational inference ( Jordan et al. , 1998 ) . Recent works have parameterized these models with deep neural networks , e.g . ( Chung et al. , 2015 ; Gan et al. , 2015 ; Fraccaro et al. , 2016 ; Karl et al. , 2017 ) , using amortized variational inference ( Kingma & Welling , 2014 ; Rezende et al. , 2014 ) for inference and learning . Typically , the conditional likelihood , pθ ( xt|x < t , z≤t ) , and the prior , pθ ( zt|x < t , z < t ) , are Gaussian densities , with temporal conditioning handled through deterministic recurrent networks and the stochastic latent variables . Such models have demonstrated success in audio ( Chung et al. , 2015 ; Fraccaro et al. , 2016 ) and video modeling ( Xue et al. , 2016 ; Gemici et al. , 2017 ; Denton & Fergus , 2018 ; He et al. , 2018 ; Li & Mandt , 2018 ) . However , design choices for these models remain an active area of research , with each model proposing new combinations of deterministic and stochastic dynamics . 2.3 AUTOREGRESSIVE FLOWS . Our approach is based on affine autoregressive normalizing flows ( Kingma et al. , 2016 ; Papamakarios et al. , 2017 ) . Here , we review this basic concept , continuing with the perspective of temporal sequences , however , it is worth noting that these flows were initially developed and demonstrated in static settings . Kingma et al . ( 2016 ) noted that sampling from an autoregressive Gaussian model is an invertible transform , resulting in a normalizing flow ( Rippel & Adams , 2013 ; Dinh et al. , 2015 ; 2017 ; Rezende & Mohamed , 2015 ) . Flow-based models transform between simple and complex probability distributions while maintaining exact likelihood evaluation . To see their connection to autoregressive models , we can express sampling a Gaussian random variable , xt ∼ pθ ( xt|x < t ) ( Eq . 2 ) , using the reparameterization trick ( Kingma & Welling , 2014 ; Rezende et al. , 2014 ) : xt = µθ ( x < t ) + σθ ( x < t ) yt , ( 5 ) where yt ∼ N ( yt ; 0 , I ) is an auxiliary random variable and denotes element-wise multiplication . Thus , xt is an invertible transform of yt , with the inverse given as yt = xt − µθ ( x < t ) σθ ( x < t ) , ( 6 ) where division is performed element-wise . The inverse transform in Eq . 6 acts to normalize ( hence , normalizing flow ) and therefore decorrelate x1 : T . Given the functional mapping between yt and xt in Eq . 5 , the change of variables formula converts between probabilities in each space : log pθ ( x1 : T ) = log pθ ( y1 : T ) − log ∣∣∣∣det ( ∂x1 : T∂y1 : T ) ∣∣∣∣ . ( 7 ) By the construction of Eqs . 5 and 6 , the Jacobian in Eq . 7 is triangular , enabling efficient evaluation as the product of diagonal terms : log ∣∣∣∣det ( ∂x1 : T∂y1 : T ) ∣∣∣∣ = T∑ t=1 ∑ i log σθ , i ( x < t ) , ( 8 ) where i denotes the observation dimension , e.g . pixel . For a Gaussian autoregressive model , pθ ( y1 : T ) = N ( y1 : T ; 0 , I ) . With these components , the change of variables formula ( Eq . 7 ) provides an equivalent method for sampling and evaluating the model , pθ ( x1 : T ) , from Eqs . 1 and 2 . We can improve upon this simple set-up by chaining together multiple transforms , effectively resulting in a hierarchical autoregressive model . Letting ym1 : T denote the variables after them th transform , the change of variables formula for M transforms is log pθ ( x1 : T ) = log pθ ( y M 1 : T ) − log ∣∣∣∣det ( ∂x1 : T∂y11 : T ) ∣∣∣∣−M−1∑ m=1 log ∣∣∣∣det ( ∂ym1 : T∂ym+11 : T ) ∣∣∣∣ . ( 9 ) Autoregressive flows were initially considered in the contexts of variational inference ( Kingma et al. , 2016 ) and generative modeling ( Papamakarios et al. , 2017 ) . These approaches are , in fact , generalizations of previous approaches with affine transforms ( Dinh et al. , 2015 ; 2017 ) . While autoregressive flows are well-suited for sequential data , as mentioned previously , these approaches , as well as many recent approaches ( Huang et al. , 2018 ; Oliva et al. , 2018 ; Kingma & Dhariwal , 2018 ) , were initially applied in static settings , such as images . More recent works have started applying flow-based models to sequential data . For instance , van den Oord et al . ( 2018 ) and Ping et al . ( 2019 ) distill autoregressive speech models into flow-based models . Prenger et al . ( 2019 ) and Kim et al . ( 2019 ) instead train these models directly . Kumar et al . ( 2019 ) use a flow to model individual video frames , with an autoregressive prior modeling dynamics across time steps . Rhinehart et al . ( 2018 ) and Rhinehart et al . ( 2019 ) use autoregressive flows for modeling vehicle motion , and Henter et al . ( 2019 ) use flows for motion synthesis with motion-capture data . Ziegler & Rush ( 2019 ) learn distributions over sequences of discrete observations ( e.g. , text ) by using flows to model dynamics of continuous latent variables . Like these recent works , we apply flow-based models to sequential data . However , we demonstrate that autoregressive flows can serve as a useful , general-purpose technique for improving sequence modeling as components of sequential latent variable models . To the best of our knowledge , our work is the first to focus on the aspect of using flows to pre-process sequential data to improve downstream dynamics modeling . Finally , we utilize affine flows ( Eq . 5 ) in this work . This family of flows includes methods like NICE ( Dinh et al. , 2015 ) , RealNVP ( Dinh et al. , 2017 ) , IAF ( Kingma et al. , 2016 ) , MAF ( Papamakarios et al. , 2017 ) , and GLOW ( Kingma & Dhariwal , 2018 ) . However , there has been recent work in non-affine flows ( Huang et al. , 2018 ; Jaini et al. , 2019 ; Durkan et al. , 2019 ) , which may offer further flexibility . We chose to investigate affine flows for their relative simplicity and connections to previous techniques , however , the use of non-affine flows could result in additional improvements .

Summary: This paper proposes to model temporal sequences using autoregressive flows across time steps, that allow to model more explicitly temporal changes of the input, i.e. how the input x_t has changed w.r.t x_{<t}. As also stated by the authors, this is a generalization of other work that instead of modelling the input at each time step, models temporal differences between consecutive time steps.