Title: IMPROVING SEQUENTIAL LATENT VARIABLE MODELS WITH AUTOREGRESSIVE FLOWS

Abstract: We propose an approach for sequence modeling based on autoregressive normalizing flows. Each autoregressive transform, acting across time, serves as a moving reference frame for modeling higher-level dynamics. This technique provides a simple, general-purpose method for improving sequence modeling, with connections to existing and classical techniques. We demonstrate the proposed approach both with standalone models, as well as a part of larger sequential latent variable models. Results are presented on three benchmark video datasets, where flowbased dynamics improve log-likelihood performance over baseline models.

1. INTRODUCTION:

Data often contain sequential structure, providing a rich signal for learning models of the world. Such models are useful for learning self-supervised representations of sequences (Li & Mandt, 2018; Ha & Schmidhuber, 2018) and planning sequences of actions (Chua et al., 2018; Hafner et al., 2019) . While sequential models have a longstanding tradition in probabilistic modeling (Kalman et al., 1960) , it is only recently that improved computational techniques, primarily deep networks, have facilitated learning such models from high-dimensional data (Graves, 2013) , particularly video and audio. Dynamics in these models typically contain a combination of stochastic and deterministic variables (Bayer & Osendorfer, 2014; Chung et al., 2015; Gan et al., 2015; Fraccaro et al., 2016) , using simple distributions (e.g. Gaussian) to directly model the likelihood of data observations. However, attempting to capture all sequential dependencies with relatively unstructured dynamics may make it more difficult to learn such models. Intuitively, the model should use its dynamical components to track changes in the input instead of simultaneously modeling the entire signal. Rather than expanding the computational capacity of the model, we seek a method for altering the representation of the data to provide a more structured form of dynamics.

To incorporate more structured dynamics, we propose an approach for sequence modeling based on autoregressive normalizing flows (Kingma et al., 2016; Papamakarios et al., 2017) , consisting of one or more autoregressive transforms in time. A single transform is equivalent to a Gaussian autoregressive model. However, by stacking additional transforms or latent variables on top, we can arrive at more expressive models. Each autoregressive transform serves as a moving reference frame in which higher-level structure is modeled. This provides a general mechanism for separating different forms of dynamics, with higher-level stochastic dynamics modeled in the simplified space provided by lower-level deterministic transforms. In fact, as we discuss, this approach generalizes the technique of modeling temporal derivatives to simplify dynamics estimation (Friston, 2008) .

We empirically demonstrate this approach, both with standalone autoregressive normalizing flows, as well as by incorporating these flows within more flexible sequential latent variable models. While normalizing flows have been applied in a few sequential contexts previously, we emphasize the use of these models in conjunction with sequential latent variable models. We present experimental results on three benchmark video datasets, showing improved quantitative performance in terms of log-likelihood. In formulating this general technique for improving dynamics estimation in the framework of normalizing flows, we also help to contextualize previous work. Figure 1 : Affine Autoregressive Transforms. Computational diagrams for forward and inverse affine autoregressive transforms (Papamakarios et al., 2017) . Each y t is an affine transform of x t , with the affine parameters potentially non-linear functions of x <t . The inverse transform is capable of converting a correlated input, x 1:T , into a less correlated variable, y 1:T .

2. BACKGROUND:



2.1. AUTOREGRESSIVE MODELS:

Consider modeling discrete sequences of observations, x 1:T ∼ p data (x 1:T ), using a probabilistic model, p θ (x 1:T ), with parameters θ. Autoregressive models (Frey et al., 1996; Bengio & Bengio, 2000) use the chain rule of probability to express the joint distribution over all time steps as the product of T conditional distributions. Because of the forward nature of the world, as well as for handling variable-length sequences, these models are often formulated in forward temporal order:

EQUATION

Each conditional distribution, p θ (x t |x <t ), models the temporal dependence between time steps, i.e. a prediction of the future. For continuous variables, we often assume that each distribution takes a relatively simple form, such as a diagonal Gaussian density:

EQUATION

where µ θ (•) and σ θ (•) are functions denoting the mean and standard deviation, often sharing parameters over time steps. While these functions may take the entire past sequence of observations as input, e.g. through a recurrent neural network, they may also be restricted to a convolutional window (van den Oord et al., 2016a) . Autoregressive models can also be applied to non-sequential data (van den Oord et al., 2016b) , where they excel at capturing local dependencies. However, due to their restrictive distributional forms, such models often struggle to capture higher-level structure.

2.2. AUTOREGRESSIVE LATENT VARIABLE MODELS:

Autoregressive models can be improved by incorporating latent variables, often represented as a corresponding sequence, z 1:T . Classical examples include Gaussian state space models and hidden Markov models (Murphy, 2012) . The joint distribution, p θ (x 1:T , z 1:T ), has the following form:

EQUATION

Unlike the simple, parametric form in Eq. 2, evaluating p θ (x t |x <t ) now requires integrating over the latent variables,

EQUATION

yielding a more flexible distribution. However, performing this integration in practice is typically intractable, requiring approximate inference techniques, like variational inference (Jordan et al., 1998) . Recent works have parameterized these models with deep neural networks, e.g. (Chung et al., 2015; Gan et al., 2015; Fraccaro et al., 2016; Karl et al., 2017) , using amortized variational inference (Kingma & Welling, 2014; Rezende et al., 2014) for inference and learning. Typically, the conditional likelihood, p θ (x t |x <t , z ≤t ), and the prior, p θ (z t |x <t , z <t ), are Gaussian densities, with temporal conditioning handled through deterministic recurrent networks and the stochastic latent variables. Such models have demonstrated success in audio (Chung et al., 2015; Fraccaro et al., 2016) and video modeling (Xue et al., 2016; Gemici et al., 2017; Denton & Fergus, 2018; He et al., 2018; Li & Mandt, 2018) . However, design choices for these models remain an active area of research, with each model proposing new combinations of deterministic and stochastic dynamics.

2.3. AUTOREGRESSIVE FLOWS:

Our approach is based on affine autoregressive normalizing flows (Kingma et al., 2016; Papamakarios et al., 2017) . Here, we review this basic concept, continuing with the perspective of temporal sequences, however, it is worth noting that these flows were initially developed and demonstrated in static settings. Kingma et al. (2016) noted that sampling from an autoregressive Gaussian model is an invertible transform, resulting in a normalizing flow (Rippel & Adams, 2013; Dinh et al., 2015; 2017; Rezende & Mohamed, 2015) . Flow-based models transform between simple and complex probability distributions while maintaining exact likelihood evaluation. To see their connection to autoregressive models, we can express sampling a Gaussian random variable, x t ∼ p θ (x t |x <t ) (Eq. 2), using the reparameterization trick (Kingma & Welling, 2014; Rezende et al., 2014) :

EQUATION

where y t ∼ N (y t ; 0, I) is an auxiliary random variable and denotes element-wise multiplication. Thus, x t is an invertible transform of y t , with the inverse given as

EQUATION

where division is performed element-wise. The inverse transform in Eq. 6 acts to normalize (hence, normalizing flow) and therefore decorrelate x 1:T . Given the functional mapping between y t and x t in Eq. 5, the change of variables formula converts between probabilities in each space: log p θ (x 1:T ) = log p θ (y 1:T )log det ∂x 1:T ∂y 1:T .

By the construction of Eqs. 5 and 6, the Jacobian in Eq. 7 is triangular, enabling efficient evaluation as the product of diagonal terms:

EQUATION

where i denotes the observation dimension, e.g. pixel. For a Gaussian autoregressive model, p θ (y 1:T ) = N (y 1:T ; 0, I). With these components, the change of variables formula (Eq. 7) provides an equivalent method for sampling and evaluating the model, p θ (x 1:T ), from Eqs. 1 and 2.

We can improve upon this simple set-up by chaining together multiple transforms, effectively resulting in a hierarchical autoregressive model. Letting y m 1:T denote the variables after the m th transform, the change of variables formula for M transforms is

EQUATION

)

Autoregressive flows were initially considered in the contexts of variational inference (Kingma et al., 2016) and generative modeling (Papamakarios et al., 2017) . These approaches are, in fact, generalizations of previous approaches with affine transforms (Dinh et al., 2015; 2017) . While autoregressive flows are well-suited for sequential data, as mentioned previously, these approaches, as well as many recent approaches (Huang et al., 2018; Oliva et al., 2018; Kingma & Dhariwal, 2018) , were initially applied in static settings, such as images. by using flows to model dynamics of continuous latent variables. Like these recent works, we apply flow-based models to sequential data. However, we demonstrate that autoregressive flows can serve as a useful, general-purpose technique for improving sequence modeling as components of sequential latent variable models. To the best of our knowledge, our work is the first to focus on the aspect of using flows to pre-process sequential data to improve downstream dynamics modeling.

Finally, we utilize affine flows (Eq. 5) in this work. This family of flows includes methods like NICE (Dinh et al., 2015) , RealNVP (Dinh et al., 2017) , IAF (Kingma et al., 2016) , MAF (Papamakarios et al., 2017) , and GLOW (Kingma & Dhariwal, 2018). However, there has been recent work in non-affine flows (Huang et al., 2018; Jaini et al., 2019; Durkan et al., 2019) , which may offer further flexibility. We chose to investigate affine flows for their relative simplicity and connections to previous techniques, however, the use of non-affine flows could result in additional improvements.

3. METHOD:

We now describe our approach for sequence modeling with autoregressive flows. Although the core idea is a relatively straightforward extension of autoregressive flows, we show how this simple technique can be incorporated within autoregressive latent variable models (Section 2.2), providing a general-purpose approach for improving dynamics modeling. We first motivate the benefits of affine autoregressive transforms in the context of sequence modeling with a simple example.

3.1. A MOTIVATING EXAMPLE:

Consider the discrete dynamical system defined by the following set of equations:

EQUATION

EQUATION

where w t ∼ N (w t ; 0, Σ). We can express x t and u t in probabilistic terms as

EQUATION

EQUATION

)

Physically, this describes the noisy dynamics of a particle with momentum and mass 1, subject to Gaussian noise. That is, x represents position, u represents velocity, and w represents stochastic forces. If we consider the dynamics at the level of x, we can use the fact that

EQUATION

Thus, we see that in the space of x, the dynamics are second-order Markov, requiring knowledge of the past two time steps. However, at the level of u (Eq. 13), the dynamics are first-order Markov, requiring only the previous time step. Yet, note that u t is, in fact, an affine autoregressive transform of x t because u t = x t -x t-1 is a special case of the general form xt-µ θ (x<t) σ θ (x<t) . In Eq. 10, we see that the Jacobian of this transform is ∂x t /∂u t = I, so, from the change of variables formula, we have p(x t |x t-1 , x t-2 ) = p(u t |u t-1 ). In other words, an affine autoregressive transform has allowed us to convert a second-order Markov system into a first-order Markov system, thereby simplifying the dynamics. Continuing this process to move to w t = u tu t-1 , we arrive at a representation that is entirely temporally decorrelated, i.e. no dynamics, because p(w t ) = N (w t ; 0, Σ). A sample from this system is shown in Figure 2 , illustrating this process of temporal decorrelation.

The special case of modeling temporal changes, In fact, ∆x t is a finite differences approximation of the generalized velocity (Friston, 2008) of x, a classic modeling technique in dynamical models and control (Kalman et al., 1960) , redefining the state-space to be first-order Markov. Affine autoregressive flows offer a generalization of this technique, allowing for non-linear transform parameters and flows consisting of multiple transforms, with each transform serving to successively decorrelate the input sequence in time. In analogy with generalized velocity, each transform serves as a moving reference frame, allowing us to focus model capacity on less correlated fluctuations rather than the highly temporally correlated raw signal.

u t = x t -x t-1 = ∆x t ,

3.2. AUTOREGRESSIVE FLOWS ON SEQUENCES:

We apply autoregressive flows across time steps within a sequence, x 1:T ∈ R T ×D . That is, the observation at each time step, x t ∈ R D , is modeled as an autoregressive function of past observations, x <t ∈ R t-1×D , and a random variable, y t ∈ R D (Figure 3a ). We consider flows of the form given in Eq. 5, where µ θ (x <t ) and σ θ (x <t ) are parameterized by neural networks. In constructing chains of flows, we denote the shift and scale functions at the m th transform as µ m θ (•) and σ m θ (•) respectively. We then calculate y m using the corresponding inverse transform:

EQUATION

)

After the final (M th ) transform, the base distribution, p θ (y M 1:T ), can range from a simple distribution, e.g. N (y M 1:T ; 0, I), in the case of a flow-based model, up to more complicated distributions in the case of other latent variable models (Section 3.3). While flows of greater depth can improve model capacity, such transforms have limiting drawbacks. In particular, 1) they require that the outputs maintain the same dimensionality as the inputs, R T ×D , 2) they are restricted to affine transforms, and 3) these transforms operate element-wise within a time step. As we discuss in the next section, we can combine autoregressive flows with non-invertible sequential latent variable models (Section 2.2), which do not have these restrictions.

3.3. LATENT VARIABLE MODELS WITH AUTOREGRESSIVE FLOWS:

We can use autoregressive flows as a component in parameterizing the dynamics within autoregressive latent variable models. To simplify notation, we consider this set-up with a single transform, but a chain of multiple transforms (Section 3.2) can be applied within each flow.

3.3.1. MODEL FORMULATION:

Let us consider parameterizing the conditional likelihood, p θ (x t |x <t , z ≤t ), within a latent variable model using an autoregressive flow (Figure 3b ). To do so, we express a base conditional distribution for y t , denoted as p θ (y t |y <t , z ≤t ), which is then transformed into x t via the affine transform in Eq. 5. We have written p θ (y t |y <t , z ≤t ) with conditioning on y <t , however, by removing temporal correlations to arrive at y 1:T , our hope is that these dynamics can be primarily modeled through z 1:T . Using the change of variables formula, we can express the latent variable model's log-joint distribution as where the joint distribution over y 1:T and z 1:T , in general, is given as

EQUATION

EQUATION

Note that the latent prior, p θ (z t |y <t , z <t ), can be equivalently conditioned on x <t or y <t , as there is a one-to-one mapping between these variables. We could also consider parameterizing the prior with autoregressive flows, or even constructing a hierarchy of latent variables. However, we leave these extensions for future work, opting to first introduce the basic concept here.

3.3.2. VARIATIONAL INFERENCE & LEARNING:

Training a latent variable model via maximum likelihood requires marginalizing over the latent variables to evaluate the marginal log-likelihood of observations: log p θ (x 1:T ) = log p θ (x 1:T , z 1:T )dz 1:T . This marginalization is typically intractable, requiring the use of approximate inference methods. Variational inference (Jordan et al., 1998) introduces an approximate posterior distribution, q(z 1:T |x 1:T ), which provides a lower bound on the marginal log-likelihood:

EQUATION

referred to as the evidence lower bound (ELBO). Often, we assume q(z 1:T |x 1:T ) is a structured distribution, attempting to explicitly capture the model's temporal dependencies across z 1:T . We can consider both filtering or smoothing inference, however, we focus on the case of filtering, with

EQUATION

The conditional dependencies in q can be modeled through a direct, amortized function, e.g. using a recurrent network (Chung et al., 2015) , or through optimization (Marino et al., 2018) . Again, note that we can condition q on x ≤t or y ≤t , as there exists a one-to-one mapping between these variables. With the model's joint distribution (Eq. 16) and approximate posterior (Eq. 19), we can then evaluate the ELBO. We derive the ELBO for this set-up in Appendix A, yielding

EQUATION

This expression makes it clear that a flow-based conditional likelihood amounts to learning a latent variable model on top of the intermediate learned space provided by y, with an additional factor in the objective penalizing the scaling between x and y. 

4. EVALUATION:

We demonstrate and evaluate the proposed framework on three benchmark video datasets: Moving MNIST (Srivastava et al., 2015) , KTH Actions (Schuldt et al., 2004) , and BAIR Robot Pushing (Ebert et al., 2017) . Experimental setups are described in Section 4.1, followed by a set of qualitative experiments in Section 4.2. In Section 4.3, we provide quantitative comparisons across different model classes. Further implementation details and visualizations can be found in Appendix B. Anonymized code is available at the following link.

4.1. EXPERIMENTAL SETUP:

We implement three classes of models: 1) standalone autoregressive flow-based models, 2) sequential latent variable models, and 3) sequential latent variable models with flow-based conditional likelihoods. Flows are implemented with convolutional networks, taking in a fixed window of previous frames and outputting shift, µ θ , and scale, σ θ , parameters. The sequential latent variable models consist of convolutional and recurrent networks for both the encoder and decoder networks, following the basic form of architecture that has been previously employed in video modeling (Denton & Fergus, 2018; Ha & Schmidhuber, 2018; Hafner et al., 2019) .

In the case of a regular sequential latent variable model, the conditional likelihood is a Gaussian that models the frame, x t . In the case of a flow-based conditional likelihood, we model the noise variable, y t , with a Gaussian. In our experiments, the flow components have vastly fewer parameters than the sequential latent variable models. In addition, for models with flow-based conditional likelihoods, we restrict the number of parameters to enable a fairer comparison. These models have fewer parameters than the baseline sequential latent variable models (with non-flow-based conditional likelihoods). See Appendix B for parameter comparisons and architecture details. Finally, flow-based conditional likelihoods only add a constant computational cost per time-step, requiring a single forward pass per time step for both evaluation and generation. 8 in the Appendix. In Figure 9 in the Appendix, we also visualize these quantities for a flow-based conditional likelihood with two transforms.

From these visualizations, we can make a few observations. The shift parameters (second row) tend to capture the static background, blurring around regions of uncertainty. The scale parameters (third row), on the other hand, tend to focus on regions of higher uncertainty, as expected. The resulting noise variables (bottom row) display any remaining structure not modeled by the flow. In comparing standalone flow-based models with flow-based conditional likelihoods in sequential latent variable models, we see that the latter qualitatively contains more structure in y, e.g. dots (Figure 4b , fourth row) or sharper edges (Figure 4d , fourth row). This is expected, as the noise distribution is more expressive in this case. With a relatively simple dataset, like Moving MNIST, a single flow can reasonably decorrelate the input, yielding white noise images (Figure 4a , fourth row). However, with natural image datasets like KTH Actions and BAIR Robot Pushing, a large degree of structure is still present in these images, motivating the use of additional model capacity to model this signal. In Appendix C.1, we quantify the degree of temporal decorrelation performed by flow-based models by evaluating the empirical correlation between frames at successive time steps for both the data, x, and the noise variables, y. In Appendix C.2, we provide additional qualitative results.

4.3. QUANTITATIVE EVALUATION:

Log-likelihood results for each model class are shown in Table 1 . We report the average test loglikelihood in nats per pixel per channel for flow-based models and the lower bound on this quantity for sequential latent variable models. Standalone flow-based models perform surprisingly well, even outperforming sequential latent variable models in some cases. Increasing flow depth from 1 to 2 generally results in improved performance. Sequential latent variable models with flow-based conditional likelihoods outperform their baseline counterparts, despite having fewer parameters. One reason for this disparity is overfitting. Comparing with the training performance reported in Table 3 , we see that sequential latent variable models with flow-based conditional likelihoods overfit less. This is particularly apparent on KTH Actions, which contains training and test sets with a high degree of separation (different identities and activities). This suggests that removing static components, like backgrounds, yields a reconstruction space that is better for generalization.

The quantitative results in Table 1 are for a representative sequential latent variable model with a standard convolutional encoder-decoder architecture and fully-connected latent variables. However, many previous works do not evaluate proper lower bounds on log-likelihood, using techniques like down-weighting KL divergences (Denton & Fergus, 2018; Ha & Schmidhuber, 2018; Lee et al., 2018) . Indeed, Marino et al. (2018) train SVG (Denton & Fergus, 2018) with a proper lower bound and report a lower bound of -2.86 nats per pixel on KTH Actions, on-par with our results. Kumar et al. (2019) report log-likelihood results on BAIR Robot Pushing, obtaining -1.3 nats per pixel, substantially higher than our results. However, their model is significantly larger than the models presented here, consisting of 3 levels of latent variables, each containing 24 steps of flows.

5. CONCLUSION:

We have presented a technique for improving sequence modeling based on autoregressive normalizing flows. This technique uses affine transforms to temporally decorrelate sequential data, thereby simplifying the estimation of dynamics. We have drawn connections to classical approaches, which involve modeling temporal derivatives. Finally, we have empirically shown how this technique can improve sequential latent variable models.

A LOWER BOUND DERIVATION:

Consider the model defined in Section 3.3.1, with the conditional likelihood parameterized with autoregressive flows. That is, we parameterize

EQUATION

The joint distribution over all time steps is then given as

EQUATION

To perform variational inference, we consider a filtering approximate posterior of the form

EQUATION

We can then plug these expressions into the evidence lower bound:

EQUATION

= E q(z 1:T |x 1:T ) log 

B EXPERIMENT DETAILS:

We store a fixed number of past frames in the buffer of each transform, to generate the shift and scale for the transform. For each stack of flow, 4 convolutional layers with kernel size (3, 3), stride 1 and padding 1 are applied first on each data observation in the buffer, preserving the data shape. The outputs are concatenated along the channel dimension and go through another four convolutional layers also with kernel size (3, 3), stride 1 and padding 1. Finally, separate convolutional layers with the same kernel size, stride and padding are used to generate shift and scale respectively.

For latent variable models, we use a DC-GAN structure (Radford et al., 2015) , with 4 layers of convolutional layers of kernel size (4, 4), stride 2 and padding 1 before another convolutional layer of kernel size (4, 4), stride 1 and no padding to encode the data. The encoded data is sent to an LSTM (Hochreiter & Schmidhuber, 1997) followed by fully connected layers to generate the mean and log-variance for estimating the approximate posterior distribution of the latent variable, z t . The conditional prior distribution is modeled with another LSTM followed by fully connected layers, taking the previous latent variable as input. The decoder take the inverse structure of the encoder.

In the SLVM, we use 2 LSTM layers for modelling the conditional prior and approximate posterior distributions, while in the combined model we use 1 LSTM layer for each.

We use the Adam optimizer (Kingma & Ba, 2014) with a learning rate of 1 × 10 -4 to train all the models. For Moving MNIST, we use a batch size of 16 and train for 200, 000 iterations for latent variable models and 100, 000 iterations for flow-based and latent variable models with flow-based likelihoods. For BAIR Robot Pushing, we use a batch size of 8 and train for 200, 000 iterations for all models. For KTH dataset we use a batch size of 8 and train for 90, 000 iterations for all models. Batch norm (Ioffe & Szegedy, 2015) is applied to all convolutional layers that do not directly generate distribution or transform parameters. We randomly crop sequence of length 13 from all sequences and evaluate on the last 10 frames. (For 2-flow models we crop sequence of length 16 to fill up all buffers.) Anonymized code is available at the following link. 64, 4, 2, 1)   conv(128, 4, 2, 1)   conv(256, 4, 2, 1)   conv(512, 4, 2, 1 To quantitatively confirm the temporal decorrelation resulting from this process, we evaluate the empirical correlation between successive frames, averaged over spatial locations and channels, for the data observations and noise variables. This is an average normalized version of the auto-covariance of each signal with a time delay of 1 time step. Specifically, we estimate the temporal correlation as

EQUATION

where x (i,j,k) denotes the value of the image at location (i, j) and channel k, µ (i,j,k) denotes the mean of this dimension, and σ (i,j,k) denotes the standard deviation of this dimension. H, W, and C respectively denote the height, width, and number of channels of the observations.

We evaluated this quantity for data examples, x, and noise variables, y, for SLVM w/ 1-AF. The results for training sequences are shown in Table 4 . In Figure 7 , we plot this quantity during training for KTH Actions. We see that flows do indeed result in a decrease in temporal correlation. Note that because correlation is a measure of linear dependence, one cannot conclude from these results alone ) the lower-level predicted shift, µ 1 θ (x <t ), for the frame, 3) the predicted scale, σ 1 θ (x <t ), for the frame. On the right side, from top to bottom, we have 1) the higer-level predicted shift, µ 2 θ (x <t ), for the frame, 3) the predicted scale, σ 2 θ (x <t ), for the frame and 4) the noise, y t , obtained from the inverse transform. 

:



Figures and Tables:

More recent works have started applying flow-based models to sequential data. For instance, van den Oord et al. (2018) andPing et al. (2019) distill autoregressive speech models into flow-based models.

Figure 2: Motivating Example. Plots are shown for a sample of x 1:T (left), u 1:T (center), and w 1:T (right). Here, w 1:T ∼ N (w 1:T ; 0, I), and u and x are initialized at 0. Moving from x → u → w via affine transforms results in successively less temporal correlation and therefore simpler dynamics.

is a common pre-processing technique; for recent examples, see Deisenroth et al. (2013); Chua et al. (2018); Kumar et al. (2019).

Figure 3: Graphical Models. Diagrams for (a) a single-transform affine autoregressive flow-based model, with random variables, y 1:T ∼ N (y 1:T ; 0, I), and (b) a sequential latent variable model with a flow-based conditional likelihood.We have depicted simplified dynamics for y 1:T and z 1:T for clarity, however, in general, these can be non-Markov. The flow removes low-level temporal correlations in x 1:T , whereas the latent variables, z 1:T , capture any remaining structure in y 1:T .

Figure 4: Flow Visualization. Visualization of the flow component for (a), (c) standalone flowbased models and (b), (d) sequential latent variable models with flow-based conditional likelihoods for Moving MNIST and BAIR Robot Pushing. From top to bottom, each figure shows 1) the original frames, x t , 2) the predicted shift, µ θ (x <t ), for the frame, 3) the predicted scale, σ θ (x <t ), for the frame, and 4) the noise, y t , obtained from the inverse transform.

are set up with with kernel size (3,3) stride 1 and padding 1.

Figure 5: Implementation Visualization of the autoregressive flow.

Figure 6: Model Architecture Diagrams. Diagrams are shown for the (a) approximate posterior, (b) conditional prior,and (c) conditional likelihood of the sequential latent variable model. conv denotes a convolutional layer, LSTM denotes a long short-term memory layer, fc denotes a fullyconnected layer, and t conv denotes a transposed convolutional layer. For conv and t conv layers, the numbers in parentheses respectively denote the number of filters, filter size, stride, and padding of the layer. For fc and LSTM layers, the number in parentheses denotes the number of units.

Figure 7: Temporal Correlation During Training. corr y during training for SLVM w/ 1-AF on the KTH Actions. Temporal correlation decreases substantially during training.

Figure 9: SLVM w/ 2-AF Visualization on Moving MNIST. Visualization of the flow component for sequential latent variable models with 2-layer flow-based conditional likelihoods for Moving MNIST. From top to bottom on the left side, each figure shows 1) the original frames, x t , 2) the lower-level predicted shift, µ 1θ (x <t ), for the frame, 3) the predicted scale, σ 1 θ (x <t ), for the frame. On the right side, from top to bottom, we have 1) the higer-level predicted shift, µ 2 θ (x <t ), for the frame, 3) the predicted scale, σ 2 θ (x <t ), for the frame and 4) the noise, y t , obtained from the inverse transform.

Figure 10: Generated Moving MNIST Samples. Samples frame sequences generated from a 2-AF model.

Figure 11: Generated BAIR Robot Pushing Samples. Samples frame sequences generated from SLVM w/ 1-AF.

Quantitative Comparison. Average test log-likelihood (higher is better) in nats per pixel per channel for Moving MNIST, BAIR Robot Pushing, and KTH Actions. For flow-based models (1-AF and 2-AF), we report the average log-likelihood. For sequential latent variable models (SLVM and SLVM w/ 1-AF), we report the average lower bound on the log-likelihood.To better understand the behavior of autoregressive flows on sequences, we visualize each component as an image. In Figure4, we show the data, x t , shift, µ θ , scale, σ θ , and noise variable, y t , for standalone flow-based models (left) and flow-based conditional likelihoods (right) on random sequences from the Moving MNIST and BAIR Robot Pushing datasets. Similar visualizations for KTH Actions are shown in Figure

Table content: <table><tr><td/><td>M-MNIST</td><td>BAIR</td><td>KTH</td></tr><tr><td>1-AF 2-AF SLVM SLVM w/ 1-AF</td><td colspan="3">-2.15 -2.13 ≥ -1.92 ≥ -3.57 ≥ -4.63 -3.05 -3.34 -2.90 -3.35 ≥ -1.86 ≥ -2.35 ≥ -2.39</td></tr><tr><td>4.2 QUALITATIVE EVALUATION</td><td/><td/></tr></table>

(y t |y <t , z ≤t )log q(z t |x ≤t , z <t ) p θ (z t |x <t , z <t ) ≤t |x ≤t ) log p θ (y t |y <t , z ≤t )log q(z t |x ≤t , z <t ) p θ (z t |x <t , z <t )Because there exists a one-to-one mapping between x 1:T and y 1:T , we can equivalently condition the approximate posterior and the prior on y, i.e. ≤t |y ≤t ) log p θ (y t |y <t , z ≤t )log q(z t |y ≤t , z <t ) p θ (z t |y <t , z <t )

Table content: <table><tr><td/><td/><td/><td/><td colspan="2">T t=1</td><td>p θ (y t |y &lt;t , z ≤t ) det</td><td>∂x t ∂y t</td><td>-1</td><td>p θ (z t |x &lt;t , z &lt;t )</td></tr><tr><td/><td/><td/><td/><td/><td>T</td></tr><tr><td/><td/><td/><td colspan="2">-log</td><td>t=1</td><td>q(z t |x ≤t , z &lt;t )</td><td>(27)</td></tr><tr><td colspan="3">= E q(z 1:T |x 1:T )</td><td>T t=1</td><td colspan="2">log p θ -log det</td><td>∂x t ∂y t</td><td>. (28)</td></tr><tr><td colspan="6">Finally, in the filtering setting, we can rewrite the expectation, bringing it inside of the sum (see</td></tr><tr><td colspan="6">Gemici et al. (2017); Marino et al. (2018)):</td></tr><tr><td>L =</td><td>T t=1</td><td colspan="4">E q(z -log det</td><td>∂x t ∂y t</td><td>. (29)</td></tr><tr><td>L =</td><td>T t=1</td><td colspan="4">E q(z -log det</td><td>∂x t ∂y t</td><td>. (30)</td></tr></table>

variable model. conv denotes a convolutional layer, LSTM denotes a long short-term memory layer, fc denotes a fullyconnected layer, and t conv denotes a transposed convolutional layer. For conv and t conv layers, the numbers in parentheses respectively denote the number of filters, filter size, stride, and padding of the layer. For fc and LSTM layers, the number in parentheses denotes the number of units. Number of parameters for each model on each dataset. Flow-based models contain relatively few parameters as compared with the SLVM, as our flows consist primarily of 3 × 3 convolutions with limited channels. In the SLVM, we use 2 LSTM layers for modelling the prior and posterior distribution of latent variable while in the combined model we use 1 LSTM layer for each.The qualitative results in Figures4 and 8demonstrate that flows are capable of removing much of the structure of the observations, resulting in whitened noise images.

Table content: <table><tr><td>Model</td><td colspan="2">1-AF 2-AF SLVM SLVM w/ 1-AF</td></tr><tr><td>Moving Mnist</td><td>343k 686k 11302k</td><td>10592k</td></tr><tr><td colspan="2">BAIR Robot Pushing 363k 726k 11325k</td><td>10643k</td></tr><tr><td>KTH Action</td><td>343k 686k 11302k</td><td>10592k</td></tr><tr><td colspan="2">C ADDITIONAL EXPERIMENTAL RESULTS</td><td/></tr><tr><td colspan="3">C.1 QUANTITATIVE EVALUATION OF TEMPORAL DECORRELATION</td></tr></table>

Training Quantitative Comparison. Average training log-likelihood (higher is better) in nats per pixel per channel for Moving MNIST, BAIR Robot Pushing, and KTH Actions. For flowbased models (1-AF and 2-AF), we report the average log-likelihood. For sequential latent variable models (SLVM and SLVM w/ 1-AF), we report the average lower bound on the log-likelihood.

Table content: <table><tr><td/><td>M-MNIST</td><td>BAIR</td><td>KTH</td></tr><tr><td>1-AF 2-AF SLVM SLVM w/ 1-AF</td><td colspan="3">-2.06 -2.04 ≥ -1.93 ≥ -3.46 ≥ -3.05 -2.98 -2.95 -2.76 -2.95 ≥ -1.85 ≥ -2.31 ≥ -2.21</td></tr></table>

Temporal Correlation. Temporal correlation (Eq. 31) between successive time steps for data observations, x, and noise variables, y, for SLVM w/ 1-AF.

Table content: <table><tr><td/><td/><td colspan="2">M-MNIST BAIR KTH</td><td/></tr><tr><td/><td>corr x</td><td>0.24</td><td>0.87 0.96</td><td/></tr><tr><td/><td>corr y</td><td>0.02</td><td>0.43 0.31</td><td/></tr><tr><td>0</td><td>10000</td><td>20000</td><td>30000</td><td>40000</td></tr><tr><td/><td/><td colspan="2">Training Iterations</td><td/></tr></table>

Peer review (iclr2020):
Summary The paper proposes to combine the video modeling approaches based on autoregressive flows (e.g. Kumar’19) with amortized variational inference (e.g. Denton’18), wherein an autoregressive latent variable model optimized with variational inference is extended with an autoregressive flow that further transforms the output of the latent variable model while allowing to compute exact conditional probability. This is motivated with a physical intuition, where a dynamics model can benefit from decorrelating the inputs, and it is demonstrated that layers of autoregressive flows can represent derivatives of the original signal. In a proof-of-concept experiment, it is shown that using a layer of autoregressive flow improves NLL of a latent variable model. Decision The paper presents an interesting method and tackles an important problem. At the same time, the properties of the proposed method are not well exposed and the experimental evaluation is incomplete. Moreover, the motivation of the paper is confusingly disconnected from the proposed model. I rate this paper as borderline, but am hopeful that some of the issues will be clarified during the discussion period. Pros - The paper is well-motivated and tackles a significant problem. - The proposed method is novel. - The paper is well-written. Cons - The experimental evaluation is incomplete and does not expose the properties of the method fully. Comparisons to prior art are missing. (see below) - The motivation is disconnected from the proposed model. The introduction of the paper motivates a model that hierarchically decorrelates a sequence of frames to arrive at a fully factorized model, which is later motivated with a physical example. However, the method proposed in the paper is instead a single layer of autoregressive flow on top of a powerful latent variable model! This is expressed in the title, but only glossed over in the abstract and introduction. The writing has to be updated to coherently focus on the contribution of the paper. Questions (ordered by decreasing importance) 1. In table 1, quantitative results are reported for the introduced methods. It is shown that introducing autoregressive flows achieves better likelihood and better generalization. However, quantitative comparisons with published methods that were evaluated on these datasets are missing, such as Denton’18 and Kumar’19. A quick calculation shows that Kumar et al. achieves a log-likelihood of -0.43 in Table 1 when converted to this paper’s metric, although it is possible my conversion is incorrect. Is the presented model competitive with previously published results? 2. No qualitative generation results are presented. Since the model achieves a high likelihood it is likely to do well on one-frame prediction, and possibly would even work on autoregressive multi-step prediction. Is the model capable of generation of diverse and plausible video? 3. The paper has a lengthy section 3.1 that convincingly explains that decorrelating latent variables in time is important for sequence modeling. However the proposed approach in fact produces latents that are correlated in time! Since the prior over latent variables is conditioned on past frames, the model can in fact learn a correlated representation and still achieve optimal likelihood. Moreover, the position of both the digit and the robot arm could be seen in what should be the decorrelated image in Fig 4. Is there solid quantitative (or even qualitative) evidence that the model learns a ‘more decorrelated’ representation beyond the fact that it copies the background and that the likelihood improves? The evaluation in this paper does not convince me that the model learns a temporally decorrelated representation. 4. Were modern techniques beyond affine flows considered, such as from Kingma’18, Kumar’19? Two layers of affine flows are likely insufficient to model the complexity of these data, which makes the comparison to the purely flow-based models somewhat unfair. 5. It is stated that the paper is “the first to demonstrate flows across time steps for video data”, however, the related work by Kumar et al. proposes a somewhat similar model in which conditional flows are used to model video data. Do Kumar et al. not “demonstrate flows across time steps”? Minor comments 1. Eq (10) and (12) seem to be inconsistent. Perhaps x_t = x_t-1 + u_t-1 was meant in eq (10)? 2. Line before eq(14): it not true that u_t-1 = x_t-1 - x_t-2. It would be true if the deterministic x_t = x_t-1 + u_t-1 model was assumed instead of the gaussian N(x_t; x_t-1 + u_t-1, Sigma). It is possible that eq(14) is still correct as the variance of Gaussians is additive. 3. The following work uses autoregressive flows for modeling temporal dynamics and should be cited: Rhinehart’18,19 Rhinehart et al, Deep Imitative Models for Flexible Inference, Planning, and Control Rhinehart et al, PRECOG: PREdiction Conditioned On Goals in Visual Multi-Agent Settings --------------------- Update 11.19 ----------------------- The newly provided experiments support some of the claims of the paper. In particular, I appreciate the plot showing that the proposed method successfully learns a more decorrelated representation over time, and the provided qualitative samples from the model. The authors also clarified my questions about motivation. At the same time, the proposed method is not shown to compare well to state-of-the-art approaches. I am leaning towards accepting the paper, but I believe the method would have a much larger impact if its properties were more fully exposed. == comparison with Denton&Fergus*18 (SVG) == When trained with beta=1, as the authors suggest for comparison, this method is known to perform poorly. There are two possible ways of alleviating this: 1) to train with the modified objective as in the paper but evaluate the true lower bound on the likelihood, or 2) interpret the beta as the fixed variance of the decoder distribution. Given the results the authors have provided, I believe the latter option will lead to SVG outperforming the proposed approach. == Correlation plot == Thanks for performing this experiment! While measuring correlation only captures linear dependencies, which is likely mostly the background image, this plot shows that the model indeed learns to (linearly) decorrelate the frames in the sequence. == Samples == Thanks for providing samples from the model! While the performance on BAIR is not quite convincing, the MNIST samples look very good. = Kumar et al. comparison == The author*s response convinces me that the proposed model is significantly different from Kumar et al. in scope, as Kumar et al simply use a per-frame normalizing flow encoder coupled with a sequential prior. == eqs. 10, 12 == The authors* response cleared my confusion, the equations are correct.