Improving Sequential Latent Variable Models with Autoregressive Flows
Authors: Joseph Marino
Lei Chen
Jiawei He
Stephan Mandt
Venue:  (2020)
Keywords: Autoregressive Flows
Sequence Modeling
Latent Variable Models
Video Modeling
Variational Inference

Abstract: We propose an approach for sequence modeling based on autoregressive normalizing flows. Each autoregressive transform, acting across time, serves as a moving reference frame for modeling higher-level dynamics. This technique provides a simple, general-purpose method for improving sequence modeling, with connections to existing and classical techniques. We demonstrate the proposed approach both with standalone models, as well as a part of larger sequential latent variable models. Results are presented on three benchmark video datasets, where flow-based dynamics improve log-likelihood performance over baseline models.

Under review as a conference paper at ICLR 2020
IMPROVING SEQUENTIAL LATENT VARIABLE MODELS WITH AUTOREGRESSIVE FLOWS
Anonymous authors Paper under double-blind review
ABSTRACT
We propose approach for sequence modeling based on autoregressive normal- izing flows. Each autoregressive transform, acting across time, serves as a moving reference frame for modeling higher-level dynamics. This technique provides a simple, general-purpose method for improving sequence modeling, with connec- tions existing and classical techniques. We demonstrate the proposed approach both with standalone models, as well as a part of larger sequential latent variable models. Results are presented on three benchmark video datasets, where flow- based dynamics improve log- likelhood performance over baseline models.
INTRODUCTION
Data often contain sequential structure, providing a rich signal for learning models of the world. Such models useful for learning sule-super representations of sequences (Li & Mandt, 2018; Ha & Schmidhuber, 2018) and planning sequences of actions (Chua et al., 2018; Hafner al,, 2019). While sequential models have a longstanding tradition in probabilistic modeling (Kalman et al., 1960), it is only recently that improved computational techniques, primarily deep networks. have facilitated learning such models from high- sigh-dimensional data (Graves, 2013), particularly video and audio. Dynamics in these models typically contain combination of stochastic and deterministic variables (Bayer & Osendorfer, 2014; Chung al., 2015; Gan et al., 2015; Fraccaro et al., 2016), using simple distributions (e.g. Gaussian) to directly model the likelihood of data observations. However, attempting to capture all sequential dependencies with relatively unstructured dynamics may make it more difficult to learn such models. Intuitively, the model should use its dynamical components to track changes in the input instead of simultaneously modeling the entire signal. Rather than expanding the computational capacity of the model, seek method for altering the representation of the data to provide more structured form of dynamics.
To incorporate more structured dynamics, on autoregressive normalizing flows (Kingma et al., 2016; Papamakarios et al., 2017), consisting of one or more autoregressive transforms in time. single transform is equivalent to a Gaussian autoregressive model. However, by stacking additional transforms or latent variables on top, we can arrive more expressive models. Each autoregressive transform serves moving reference frame in which higher-level structure is modeled. This provides general mechanism for separating different forms of dynamics, with higher-level stochastic dynamics modeled in the simplified space provided by lower-level deterministic transforms. In fact, we discuss, this approach generalizes the technique of modeling temporal derivatives to simplify dynamics estimation (Friston, 2008).
propose approach for sequence modeling based
We empirically demonstrate this approach, both with standalone autoregressive normalizing flows, as well as by incorporating these flows within more flexible sequential latent variable models. While normalizing flows have been applied in few sequential contexts previously, we emphasize the use of these models in conjunction with sequential latent variable models. We present experimental results three benchmark video datasets, showing improved quantitative performance in terms of log-like likeliooooo.. In formulating this general technique for improving dynamics estimation in the framework of normalizing flows, also help contextualize previous work.

---

Under review as a conference paper at ICLR 2020
Y+
...
(a) Forward Transform (Eq. 5)
(b) Inverse Transform (Eq. 6)
Figure 1: Affine Autoregressive Transforms. Computational diagrams for forward and inverse affine autoregressive transforms (Papamakarios et al., 2017). Each Yt is an affine transform of x,, with the affine parameters potentially non-linear functions of xxt- The inverse transform is capable of converting a correlated input, X1.T, into a less correlated variable, Y1.7-
2 BACKGROUND
2.1 AUTOREGRESSIVE MODELS
Consider modeling discrete sequences of observations, X1:T (Xi.T using probabilistic model, Pe (X1:)) with parameters 0. Autoregressive models (Frey et al., 1996; Bengio & Bengio, 2000) use the chain rule of probability to express the joint distribution over time steps as the product of I' conditional distributions. Because of the forward nature of the world, as well as for handling variable-length sequences, these models are often formulated in forward temporal order:
T
po (X1:T) = II (x(
Each conditional distribution, pe (X, (xe).. models the temporal dependence between time steps, i.c. a prediction of the future. For continuous variables, we often assume that each distribution takes relatively simple form, such diagonal Gaussian density:
po (x, x = No diag(o)(x<:)),
where (-) and (.) are functions denoting the mean and standard deviation, often sharing pa- rameters over time steps. While these functions may take the entire past sequence of observations as input, c.g. through a recurrent neural network, they may also be restricted to a convolutional window (van den Oord et al., 2016a). Autoregressive models can also be applied to non-sequential data (van den Oord et al., 2016b), where they excel at capturing local dependencies. However, due to their restrictive distributional forms, such models often struggle to capture higher-level structure.
2.2 AUTOREGRESSIVE LATENT VARIABLE MODELS
Autoregressive models can be improved by incorporating latent variables, often represented as corresponding sequence, Z1-r- Classical examples include Gaussian state space models and hidden Markov models (Murphy, 2012). The joint distribution, Po(x Zi:r), has the following form:
( ZI.T) = II
(x Ze. (xe Zex)
(3)
1=1
Unlike the simple, parametric form in Eq. 2, evaluating (x, (xx now requires integrating over the latent variables,
(x(x((x<e) =
yielding a more flexible distribution. However, performing this integration in practice is typically intractable, requiring approximate inference techniques, like variational inference (Jordan et al.,
2

---

Under review as a conference paper at ICLR 2020
1998). Recent works have parameterized these models with deep neural networks, e.g. (Chung et al., 2015; Gan et al., 2015; Fraccaro et al., 2016; Karl et al., 2017), using amortized variational inference (Kingma & Welling, 2014; Rezende et al., 2014) for inference and learning. Typically, the conditional likelihood, po (gxx//Xxe xx: k.ge), and the prior, po Xx1, ?,te), are Gaussian densities, with temporal conditioning handled through deterministic recurrent networks and the stochastic latent variables. Such models have demonstrated success in audio (Chung et al., 2015; Fraccaro et al,, 2016) and video modeling (Xue et al., 2016; Gemici et al., 2017; Denton & Fergus, 2018; He al,, 2018; Li & Mandt, 2018). However, design choices for these models remain an active area of research, with each model proposing new combinations of deterministic and stochastic dynamics.
2.3 AUTOREGRESSIVE FLOWS
Our approach is based affine autoregressive normalizing flows (Kingma et al., 2016; Papamakar- ios et al., 2017). Here, we review this basic concept, continuing with the perspective of temporal sequences, however, is worth noting that these flows were initially developed and demonstrated in static settings. Kingma et al. (2016) noted that sampling from an autoregressive Gaussian model is an invertible transform, resulting in a normalizing flow (Rippel & Adams, 2013; Dinh et al., 2015; 2017; Rezende & Mohamed, 2015). Flow-based models transform between simple and complex probability distributions while maintaining exact likelihood evaluation. To see their connection to autoregressive models, we can express sampling a Gaussian random variable, X; pa (x((Xet)) (Eq. 2), using the reparameterization trick (Kingma Welling, 2014; Rezende al., 2014):
X = o (xxx)) + (x(xx() @ yt,
(5)
where y, (yr;; O. I) is auxiliary random variable and @ denotes element-wiss multiplication. Thus, x, an invertible transform of ys, with the inverse given as
yz X, (X (Xx1)
where division is performed element-wise. The inverse transform in Eq. acts normalize (hence, normalizing flow) and therefore decorrelate XI.T. Given the functional mapping between y, and X, in Eq. 5, the change of variables formula converts between probabilities in each space:
@x1:T 0y
log (x = log (y( - log det
By the construction of Eqs. 5 and 6, the Jacobian in Eq. 7 is triangular, enabling efficient evaluation as the product of diagonal terms:
@xi:T dy
log det
ΣΣ
log (x<c),,
(8)
where i denotes the observation dimension, c.g. pixel. For Gaussian autoregressive model, pe(yi.)) = NY1::; O, I). With these components, the change of variables formula (Eq. 7) provides an equivalent method for sampling and evaluating the model, po (x1:)) from Eqs. and 2.
We can improve upon this simple set-up by chaining together multiple transforms, effectively result- ing in a hierarchical autoregressive model. Letting Yi. denote the variables after the m" transform, the change of variables formula for M transforms
OxLT dy
Ty 1:T
log (x log pa (y - log det
Σ log det
(9)
Autoregressive flows were initially considered in the contexts of variational inference (Kingma et al., 2016) and generative modeling (Papamakarios al., 2017). These approaches are, in fact, general- izations of previous approaches with affine transforms (Dinh et al., 2015; 2017). While autoregres- sive flows are well-suited for sequential data, mentioned previously, these approaches, as well as many recent approaches (Huang et al., 2018; Oliva et al., 2018; Kingma & Dhariwal, 2018), were initially applied in static settings, such as images.
More recent works have started applying flow-based models to sequential data. For instance, van den Oord et al. (2018) and Ping et al. (2019) distill autoregressive speech models into flow- models.
3

---

Under review as a conference paper at ICLR 2020
X;:T
W.T
60
30-
Ap mayh
10-
-30
Figure 2: Motivating Example. Plots are shown for sample of X1:T (left), ui. (center), and WI:T (right). Here, WI. ~ (V(w.: 0, I), and u and x are initialized at O. Moving from x - u + w via affine transforms results in successively less temporal correlation and therefore simpler dynamics.
Prenger al. (2019) and Kim al. (2019) instead train these models directly. Kumar et al. (2019) use flow to model individual video frames, with an autoregressive prior modeling dynamics across time steps. Rhinehart et al. (2018) and Rhinehart et al. (2019) use autoregressive flows for mod- cling vehicle motion, and Henter al. (2019) use flows for motion synthesis with motion-capture data. Ziegler Rush (2019) learn distributions over sequences of discrete observations (e.g., text) by using flows to model dynamics of continuous latent variables. Like these recent works, apply flow-based models to sequential data. However, we demonstrate that autoregressive flows can serve useful, general-purpos tial latent variable models. To the best of our knowledge, our work is the first to focus on the aspect of using flows pre-process sequential data improve downstream dynamics modeling.
technique for improving sequence modeling components of sequen-
Finally, we utilize affine flows (Eq. 5) in this work. This family of flows includes methods like NICE (Dinh et al., 2015), RealNVP (Dinh et al., 2017), IAF (Kingma et al., 2016), MAF (Papamakarios et al., 2017), and GLOW (Kingma & Dhariwal, 2018). However, there has been recent work in non-affine flows (Huang et al., 2018 Jaini et al., 2019; Durkan et al., 2019), which may offer fur- ther flexibility. We chose to investigate affine flows for their relative simplicity and connections to previous techniques, however, the use of non-affine flows could result in additional improvements.
METHOD
We describe approach for sequence modeling with autoregressive flows. Although the core idea is a relatively straightforward extension of autoregressive flows, we show how this simple technique can be incorporated within autoregressive latent variable models (Section 2.2), providing a purppse approach for improving dynamics modeling. We first motivate the benefits of affine autoregressive transforms the context of sequence modeling with a simple example.
3.1 A MOTIVATING EXAMPLE
Consider the discrete dynamical system defined by the following set of equations:
+ Ur:
(10)
u, u, + w.,
(11)
where w, ~ NWw:;; 0, E). We can express x, and u, in probabilistic terms as
X, N(x; x,-1 u1 E),
(12)
Nu, u. E).
(13)
Physically, this describes the noisy dynamics of particle with momentum and mass 1, subject to Gaussian noise. That is, x represents position, u represents velocity, and w represents stochastic forces. If we consider the dynamics at the level of x,
use the fact that u,-1 X X
to write
p(x,|x4-1, x,-2) =N(xx;Xe-) + X X,-2, E).
(14)
4

---

Under review as a conference paper at ICLR 2020
Thus, see that in the space of x. the dynamics second-order Markov, requiring knowledge of the past two time steps. However, at the level of u (Eq. 13), the dynamics are first-order Markov, requiring only the previous time step. Yet, note that u, is, in fact, affine autoregressive transform x because u, = x - is special case of the general form xx-oo(xee, Eq. 10, we see that the Jacobian of this transform is 0x:/ /Ou; = I, so, from the change of variables formula, we have p(x,/Xx 1 x = p(u, (u,-)). In other words, an affine autoregressive transform has allowed us to convert a second-order Markov system into first-order Markov system, thereby simplifying the dynamics. Continuing this process to move to W, = u: - 1. we arrive a representation that is entirely temporally decorrelated, i.e. no dynamics, because p(w,) = (w,,;; 0, Σ). A sample from this system is shown in Figure 2. illustrating this process temporal decorrelation.
do(xc1)
The special case of modeling temporal changes, 02 = X, - Ax,, is common pre-processing technique; for recent examples, see Deisenroth ct al. (2013); Chua ct al. (2018); Kumar et al. (2019). In fact, Ax; is finite differences approximation of the generalized velocity (Friston, 2008) of x, a classic modeling technique in dynamical models and control (Kalman et al., 1960), redefining the state-space to be first-order Markov. Affine autoregressive flows offer a generalization of this technique, allowing for non-linear transform parameters and flows consisting of multiple transforms, with each transform serving successively decorrelate the input sequence in time. In analogy with generalized velocity, each transform serves as a moving reference frame, allowing us to focus model capacity on less correlated fluctuations rather than the highly temporally correlated raw signal.
3.2 AUTOREGRESSIVE FLOWS ON SEQUENCES
We apply autoregressive flows across time steps within sequence, XL. RTXDD That is, the observation at each time step, x, E RR,, is modeled as an autoregressive function of past observa- tions, x Ix and random variable, y: R (Figure 3a). We consider flows of the form given in Eq. 5, where μs (x(x)) and og(xx1) are parameterized by neural networks. In constructing chains of flows, we denote the shift and scale functions at the mhth transform as ug M (-) and o respectively. We then calculate yd using the corresponding inverse transform:
1 (y <t
(15)
<t
After the final (1 transform, the base distribution, Po ). can range from a simple distribution, e.g. NN M 1:7; 0, I), in the case of a flow-based model, up to more complicated distributions in the case of other latent variable models (Section 3.3). While flows of greater depth can improve model capacity, such transforms have limiting drawbacks. In particular, 1) they require that the outputs maintain the same dimensionality as the inputs, RT'D,, 2) they are restricted to affine transforms, and 3) these transforms operate element-wise within time step. As we discuss in the next section, we can combine autoregressive flows with non invertible sequential latent variable models (Section 2.2), which do not have these restrictions.
3.3 LATENT VARIABLE MODELS WITH AUTOREGRESSIVE FLOWS
We can use autoregressive flows as a component in parameterizing the dynamics within autoregres- sive latent variable models. To simplify notation, we consider this set-up with single transform, but a chain of multiple transforms (Section 3.2) can be applied within each flow.
3.3.1 MODEL FORMULATION
Let us consider parameterizing the conditional likelihood, (x X Zgte). within latent variable model using autoregressive flow (Figure 3b). To do so, express base conditional distribution for Yt, denoted as Po (yrly cr, Zgz), which is then transformed into x, via the affine transform in Eq. 5. We have written po (Y:ly ct, Z.ge) with conditioning on y<r, however, by removing temporal correlations to arrive at Yı.T, our hope is that these dynamics can be primarily modeled through distribution as
Z1.T. Using the change of variables formula, we can express the latent variable model's log-joint
TX1.T Ty..
log (x 21-T) = log Pe (VI.7, ZiT) - log det
(16)
5

---

Under review as a conference paper at ICLR 2020
(b)
Figure 3: Graphical Models. Diagrams for (a) single-transform affine autoregressive flow-based model, with random variables, V1:r ~ (y1r: 0,I), and (b) a sequential latent variable model with flow-based conditional likelihood. We have depicted simplified dynamics for yı:r and 21:r for clarity, however, in general, these can be non-Markov. The flow removes low-level temporal correlations in X1:7, whereas the latent variables, Z1.T, capture any remaining structure in yı.7.
where the joint distribution over Yı.r and 2I.T, in general, given as
Pe (e Z1:T)
II pe te
(17)
Note that the latent prior, po(Ze/y<t>> zce), can be equivalently conditioned on X<t ycu, as there is one-to-one mapping between these variables. We could also consider parameterizing the prior with autoregressive flows, even constructing hierarchy latent variables. However, we leave these extensions for future work, opting to first introduce the basic concept here.
3.3.2 VARIATIONAL INFERENCE & LEARNING
Training a latent variable model via maximum likelihood requires marginalizing over the latent variables evaluate the marginal log-likelihood of observations: log (x) log J' (xI.T Zi.7 d1:T) This marginalization is typically intractable, requiring the use of approx- imate inference methods. Variational inference (Jordan et al., 1998) introduces an approximate posterior distribution, (Z X1.)), which provides lower bound on the marginal likelioo;::ood:
fogpe(x1:)) 2 [(x1T: g. 0) (x [log (X1:7, ZT.T) log q(z (1:T)), (18)
referred to the evidence lower bound (ELBO). Often, assume ((11)(()::) is structured distribution, attempting to explicitly capture the model's temporal dependencies across 2i:r- We can consider both filtering or smoothing inference, however, focus the case of filtering, with
g X1.T)
II (2(x(xx(, [=1
(19)
The conditional dependencies in 9 can be modeled through a direct, amortized function, e.g. using recurrent network (Chung al., 2015), or through optimization (Marino et al., 2018). Again, note that we can condition on X or Yst, as there exists a one-to-one mapping between these variables. With the model's joint distribution (Eq. 16) and approximate posterior (Eq. 19), then evaluate the ELBO. We derive the ELBO for this set-up in Appendix A, yielding
|
C Σ Ea(nesbosi (xs))) log logpo(yelye2 25e) log q(2, sy,,y,,,, Zct)
ixx
log det
(20)
Z Ye1, Z21)
Zzi)
Ty
This expression makes it clear that flowe conditional likelihood amounts learning latent variable model on top of the intermediate learned space provided by y, with an additional factor in the objective penalizing the scaling between x and

---

Under review as a conference paper at ICLR 2020
S
5
5
S
F5
S
5
5
55
15
S
S
16
16
15
6
5
5
5
55
15
55
S
(d)
Figure 4: Flow Visualization. Visualization of the flow component for (a), (c) standalone flow- based models and (b), (d) sequential latent variable models with flow-based conditional likelihoods for Moving MNIST and BAIR Robot Pushing. From top bottom, each figure shows 1) the original frames, Xt, 2) the predicted shift, (x(xx)), for the frame, 3) the predicted scale, ae (xxt), for the frame, and 4) the noise, yt, obtained from the inverse transform.
4 EVALUATION
We demonstrate and evaluate the proposed framework on three benchmark video datasets: Moving MNIST (Srivastava et al., 2015), KTTH Actions (Schuldt et al., 2004), and BAIR Robot Pushing (Ebert et al., 2017). Experimental setups are described Section 4.1, followed by a set of qualitative experiments in Section 4.2. In Section 4.3, provide quantitative comparisons across different model classes. Further implementation details and visualizations can be found in Appendix B. Anonymized code is available at the following link.
4.1 EXPERIMENTAL. SETUP
We implement three classes of models: 1) standalone autoregressive flow-based models, 2) sequen- tial latent variable models, and 3) sequential latent variable models with flow-based conditional likelihoods. Flows are implemented with convolutional networks, taking in fixed window of previ- ous frames and outputting shift, 14g, and scale, da. parameters. The sequential latent variable models consist of convolutional and recurrent networks for both the encoder and decoder networks, follow- ing the basic form of architecture that has been previously employed in video modeling (Denton & Fergus, 2018; Ha & Schmidhuber, 2018; Hafner al., 2019).
In the case of regular sequential latent variable model, the conditional likelihood is a Gaussian that models the frame, Xt. the case of flow-based conditional likelihood, we model the noise variable, yt, with Gaussian. In our experiments, the flow components have vastly fewer parame- ters than the sequential latent variable models. In addition, for models with flow-based conditional likelihoods, we restrict the number of parameters to enable fairer comparison. These models have fewer parameters than the baseline sequential latent variable models (with non-flow based condi- tional likelihoods). See Appendix B for parameter comparisons and architecture details. Finally, flow-based conditional likelihoods only add a constant computational cost per time-step, requiring single forward pass per time step for both evaluation and generation.

---

Under review as a conference paper at ICLR 2020
Table 1: Quantitative Comparison. Average test log-likelihood (higher is better) in nats per pixel per channel for Moving MNIST, BAIR Robot Pushing, and KTT Actions. For flow-based models (I-AF and 2-AF), we report the average log- oogliiklihood. For sequential latent variable models (SLVM and SLVM w/ I-AF), we report the average lower bound on the log-likelihood.
M-MNIST
BAIR
KTH
1-AF
-2.1
-3.05
-3.34
2-AF
-2113
-2.90
-3.35
SLVM
1.92
-3.57
-4.63
SLVM w/ I-AF
2 -188
2 -235
2 -2.39
4.2 QUALITATIVE EVALUATION
To better understand the behavior of autoregressive flows on sequences, we visualize each compo- nent as an image. In Figure 4, we show the data, Xx, shift, 140, scale, 0p. and noise variable, yt, for standalone flow-based models (left) and flow-based conditional likelihoods (right) on random sequences from the Moving MNIST and BAIR Robot Pushing datasets. Similar visualizations for KTH Actions are shown in Figure in the Appendix. In Figure 9 in the Appendix, we also visualize these quantities for fowebased conditional likelihood with two transforms.
From these visualizations, we can make a few observations. The shift parameters (second row) tend to capture the static background, blurrigg around regions of uncertainty. The scale parameters (third row), on the other hand, tend to focus on regions of higher uncertainty, as expected. The resulting noise variables (bottom row) display any remaining structure not modeled by the flow. In comparing standalone flow-based models with flow-based conditional likelihoods in sequential latent variable models, we see that the latter qualitatively contains more structure in y, e.g. dots (Figure 4b, fourth row) or sharper edges (Figure 4d, fourth row). This is expected, as the noise distribution is more expressive in this case. With a relatively simple dataset, like Moving MNIST, single flow can reasonably decorrelate the input, yielding white noise images (Figure 4a, fourth row). However, with natural image datasets like KTH Actions and BAIR Robot Pushing, large degree of structure is still present in these images, motivating the use of additional model capacity to model this signal. In Appendix C.I, we quantifyy the degree of temporal decorrelation performed by flow-based models by evaluating the empirical correlation between frames at successive time steps for both the data, x, and the noise variables, y. In Appendix C.2, we provide additional qualitative results.
4.3 QUANTITATIVE EVALUATION
Log-likelihood results for each model class are shown in Table 1. We report the average test log- likelihood nats per pixel per channel for flow-based models and the lower bound on this quantity for sequential latent variable models. Standalone flow-based models perform surprisingly well, even outperforming sequential latent variable models in some cases. Increasing flow depth from to 2 generally results in improved performance. Sequential latent variable models with flow-based conditional likelihoods outperform their baseline counterparts, despite having fewer parameters. One reason for this disparity is overfitting. Comparing with the training performance reported in Table 3. we see that sequential latent variable models with flow-based conditional likelihoods overfit less. This is particularly apparent on KTH Actions, which contains training and test sets with high degree of separation (different identities and activities). This suggests that removing static components, like backgrounds, yields reconstruction space that is better for generalization.
The quantitative results in Table are for a representative sequential latent variable model with a standard convolutional encoder-decoder architecture and fully-connected latent variables. However, many previous works do not evaluate proper lower bounds on log-likelihood using techniques like down-weighting KL divergences (Denton & Fergus, 2018; Ha & Schmidhuber, 2018; Lee et al., 2018). Indeed, Marino et al. (2018) train SVG (Denton & Fergus, 2018) with proper lower bound and report a lower bound of 2.86 nats per pixel on KTH Actions, on-par with our results. Kumar et al. (2019) report log-likeling likelihood results BAIR Robot Pushing, obtaining -13 nats per pixel, substantially higher than our results. However, their model is significantly larger than the models presented here, consisting of 3 levels of latent variables, each containing 24 steps of flows.
8

---

Under review as a conference paper at ICLR 2020
5 CONCLUSION
We have presented technique for improving sequence modeling based on autoregressive normal- izing flows. This technique uses affine transforms temporally decorrelate sequential data, thereby simplifying the estimation of dynamics. We have drawn connections to classical approaches, which involve modeling temporal derivatives. Finally, we have empirically shown how this technique can improve sequential latent variable models.
REFERENCES
Justin Bayer and Christian Osendorfer. Learning stochastic recurrent networks. In NeurIPS 2014 Workshop on Advances in Variational Inference, 2014.
Yoshua Bengio and Samy Bengio. Modeling high-dimensional discrete data with multi-layer neural networks. In Advances in Neural Information Processing Systems, pp. 400-406, 2000.
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learn- ing handful of trials using probabilistic dynamics models. In Advances Neural Information Processing Systems, pp. 4754-4765, 2018.
Junyoung Chung. Kyle Kastner, Laurent Dinh, Kratarth Goel, Aaron C Courville, and Yoshua Ben- gio. A recurrent latent variable model for sequential data. In Advances in neural information processing systems, pp. 2980-2988, 2015.
Mare Peter Deisenroth, Dieter Fox, and Carl Edward Rasmussen. Gaussian processes for data- efficient learning in roboties and control. IEEE transactions on pattern analysis and machine intelligence, 37(2))408-444 2013.
Emily Denton and Rob Fergus. Stochastic video generation with a learned prior. In International Conference on Machine Learning, pp. 1182-1191, 2018.
Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components esti- mation. In International Conference on Learning Representations, 2015.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. In International Conference Learning Representations, 2017.
Conor Durkan, Artur Bekasov, lain Murray, and George Papamakarios. Neural spline flows. arXiv preprint 190x:01906.04032, 2019.
Frederik Ebert, Chelsea Finn, Alex X Lee, and Sergey Levine. Self-supervised visual planning with temporal skip connections. In Conference on Robot Learning, 2017.
Marco Fraccaro, Søren Kaae Sønderby, Ulrch Paquet, and Ole Winther. Sequential neural models with stochastic layers. In Advances in neural information processing systems, pp. 2199-2207, 2016.
Brendan Frey, Geoffrey E Hinton, and Peter Dayan. Does the wake-sleep algorithm produce good density estimators? In Advances in neural information processing systems, pp. 661-667, 1996.
Karl Friston. Hierarchical models in the brain. PLoS computational biology, 4( 1(1002210 2008
Zhe Gan, Chunyuan Li, Ricardo Henao, David E Carlson, and Lawrence Carin. Deep temporal sigmoid belief networks for sequence modeling. In Advances in Neural Information Processing Systems, 2015.
Mevlana Gemici, Chia-Chun Hung, Adam Santoro, Greg Wayne, Shakir Mohamed, Danilo J Rezende, David Amos, and Timothy Lillicrap. Generative temporal models with memory. arXiv preprint 1rXiv:1702.0466 2017.
Alex Graves. Generating sequences with recurrent neural networks.
arXiv preprint
arXiv: 100..308..080 0850, 2013.

---

Under review as a conference paper at ICLR 2020
David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution. In Advances in Neural Information Processing Systems, pp. 2450-2462, 2018.
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In International Conference on Machine Learning, pp. 2565-2565, 2019.
Jiawei He, Andreas Lehrmann, Joseph Marino, Greg Mori, and Leonid Sigal. Probabilistic video generation using holistic attribute control. In Proceedings of the European Conference on Com- puter Vision (ECCV), pp. 452-467, 2018.
Gustav Eje Henter, Simon Alexanderson, and Jonas Beskow. Moglow: Probabilistic and controllable motion synthesis using normalising flows. arXiv preprint 1905.0699000656 2019.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Chin-Wei Huang, David Krueger, Alexandre Lacoste, and Aaron Courville. Neural autoregressive flows. In International Conference Machine Learning, pp. 2083 2092, 2018.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:/502.03167, 2015.
Priyank Jaini, Kira A Selby, and Yaoliang Yu. Sum-of-squares polynomial flow. In International Conference on Machine Learning, pp. 3009-3018, 2019.
Michael Jordan, Zoubin Ghahramani, Tommi S Jaakkola, and Lawrence K Saul. An introduction to variational methods for graphical models. NATO ASI SERIES D BEHAVIOURAL AND SOCIAL SCIENCES, 89:105-162, 1998.
Rudolph Emil Kalman et al. A new approach to linear filtering and prediction problems. Journal of basic Engineering, 82(1):35-45 45, 1960.
Maximilian Karl, Maximilian Soelch, Justin Bayer, and Patrick van der Smagt. Deep variational bayes filters: Unsupervised learning of state space models from raw data. In International Con- ference on Learning Representations, 2017.
Sungwon Kim, Sang-Gil Lee, Jongyoon Song, Jaehyeon Kim, and Sungroh Yoon. Flowavenet: A generative flow for raw audio. In International Conference on Machine Learning, pp. 3370-3378, 2019.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization, arXiv preprint 1.Xiv:1412.69800 2014.
Diederik P Kingma and Max Welling. Stochastic gradient vb and the variational auto-encoder. In Proceedings of the International Conference on Learning Representations, 2014.
Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible Ix1 convolutions. In Advances in Neural Information Processing Systems, pp. 10215-10224, 2018.
Durk P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, llya Sutskever, and Max Welling. Im- proved variational inference with inverse autoregressive flow. In Advances in neural information processing systems, pp. 4743-4751, 2016.
Manoj Kumar, Mohammad Babaeizadeh, Dumitru Erhan, Chelsea Finn, Sergey Levine, Laurent Dinh, and Durk Kingma. Videoflow: A flow-based generative model for video. arXiv preprint arXiv:/90m /90..90031144 01434. 2019.
Alex X Lee, Richard Zhang, Frederik Ebert, Pieter Abbeel, Chelsea Finn, and Sergey Levine. Stochastic adversarial video prediction. arXiv preprint arXiv:1804.01523, 2018.
Yingzhen and Stephan Mandt. A deep generative model for disentangled representations of sequential data. In International Conference on Machine Learning, 2018.
10

---

Under review as a conference paper at ICLR 2020
Joseph Marino, Milan Cvitkovic, and Yisong Yue. A general method for amortizing variational filtering. In Advances in Neural Information Processing Systems, pp. 7857-7868, 2018.
Kevin P Murphy, Machine learning: probabillisicc perspective. MIT press, 2012.
Junier Oliva, Avinava Dubey, Manzil Zaheer, Barnabas Poczos, Ruslan Salakhutdinov, Eric Xing, and Jeff Schneider. Transformation autoregressive networks. International Conference on Machine Learning, pp. 3895-3904 2018.
George Papamakarios, Theo Pavlakou, and lain Murray. Masked autoregressive flow for density estimation. In Advances in Neural Information Processing Systems, pp. 2338-2347, 2017.
Wei Ping, Kainan Peng, and Jitong Chen. Clarinet: Parallel wave generation in end-to-end text-to- speech. In International Conference on Learning Representations, 2019.
Ryan Prenger, Rafael Valle, and Bryan Catanzaro. Waveglow: A flow-based generative network for speech synthesis. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp- 3617-3621, IEEE, 2019.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06433 2015.
Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows In Interna-
tional Conference on Machine Learning, pp. 1530-1538, 2015.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference deep generative models. Proceedings of the International Conference on Machine Learning, pp. 1278-1288 2014.
Nicholas Rhinchart, Kris M Kitani, and Paul Veraza. R2p2: A reparameterized pushforward policy for diverse, precise generative path forecasting. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 772-788, 2018.
Nicholas Rhinehart, Rowan McAllister, Kris Kitani, and Sergey Levine. Precog: Prediction condi- tioned on goals in visual multi- settings. 2019.
Oren Rippel and Ryan Prescott Adams. High-dimensionn probability estimation with deep density models. arXiv preprint arXiv: 1302.512,, 2013.
Christian Schuldt, Ivan Laptev, and Barbara Caputo. Recognizing human actions: local svm approach. In International Conference Pattern Recognition, 2004.
Nitish Srivastava, Elman Mansimov, and Ruslan Salakhudinov. Unsupervised learning of video representations using Istms. In International conference on machine learning, pp. 843-852, 2015.
Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. arXiv preprint 16Xiv:1609.0 03499 2016a.
Aaron den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. In International Conference on Machine Learning, pp. 1747-1756, 2016b.
Aaron van den Oord, Yazhe Li, Igor Babuschkin, Karen Simonyan, Oriol Vinyals, Koray Kavukcuoglu, George Driessche, Edward Lockhart, Luis Cobo, Florian Stimberg, et al. Parallel wavenet: Fast high-fidelity speech synthesis. In International Conference on Machine Learning, pp. 3915-3923, 2018.
Tianfan Xue, Jiajun Wu, Katherine Bouman, and Bill Freeman. Visual dynamics: Probabilistic future frame synthesis via cross convolutional networks. In Advances in Neural Information Processing Systems, 2016.
Zachary Ziegler and Alexander Rush. Latent normalizing flows for discrete sequences. Interna- tional Conference on Machine Learning, pp. 7673-7682, 2019.
11

---

Under review as a conference paper at ICLR 2020
A LOWER BOUND DERIVATION
Consider the model defined Section 3.3.1, with the conditional likelihood parameterized with autoregressive flows. That we parameterize
X; mo(xx=) (x((x2)) y
(21)
yielding
(x::
(o0x))Xet, po (y det
(22)
oy:
The joint distribution over all time steps is then given
Po (x Z1.T)
po 2ge xxes Zeee)
(23)
oxe
pa Poly Zge)
det
po (e.,
(24)
oy:
To perform variational inference, we consider filtering approximate posterior of the form
g(Zi.T X1:T) II [g(
(25)
We can then plug these expressions into the evidence lower bound:
C E ((
[log pe (XL.T, Z1:T) log q(z1 [x
(26)
det (xx, Ty, 1-1
log
II po polyylyyc (y
po(2s (x1,
T
II q ((,, 2211)
log
(27)
log Po (y:ly (y - log (2, x x 222))
Oxx
E. ( (
log det
(28)
(z X Zee)
Oy:
Finally, in the filtering setting, we can rewrite the expectation, bringing it inside of the sum (see Gemici et al. (2017); Marino al. (2018)):
g(n, (x,, 2cx) log det po (z X Zet))
Fx+
|]
C Σ
(x(ee) (vep log
(29)
Ty,
Because there exists one-to-one mapping between X1:T and VI:T, equivalently condition the approximate posterior and the prior on y, i.e.
Σ By logro(yily age) log ((, 22x) log det
Ox.
L =
(30)
po Yet, Zee) Ty:
12

---

Under review as a conference paper at ICLR 2020
B EXPERIMENT DETAILS
We store fixed number of past frames in the buffer of each transform, generate the shift and scale for the transform. For each stack flow, 4 convolutional layers with kernel size (3, 3), stride and padding are applied first each data observation in the buffer, preserving the data shape. The outputs are concatenated along the channel dimension and go through another four convolutional layers also with kerne size (3.3), stride 1 and padding Finally, separate convolutional layers with the same kernel size, stride and padding are used to generate shift and scale respectively.
For latent variable models, we use DC-GAN structure (Radford et al., 2015), with 4 layers of convolutional layers of kernel size (4, 4), stride 2 and padding before another convolutional layer of kernel size (4, 4), stride and no padding to encode the data. The encoded data is sent to an LSTM (Hochreiter & Schmidhuber, 1997) followed by fully connected layers to generate the mean and log-variance for estimating the approximate posterior distribution of the latent variable, 2,, The conditional prior distribution is modeled with another LSTM followed by fully connected layers, taking the previous latent variable as input. The decoder take the inverse structure of the encoder. In the SLVM, use LSTM layers for modelling the conditional prior and approximate posterior distributions, while in the combined model we use LSTT layer for each.
We the Adam optimizer (Kingma & Ba, 2014) with learning rate of 10 to train all the models. For Moving MNIST, we use a batch size of 16 and train for 200, 000 iterations for latent variable models and 100, 000 iterations for flow-based and latent variable models with flow-based likelihoods. For BAIR Robot Pushing, we use a batch size of 8 and train for 200, 000 iterations for all models. For KTH dataset we models. Batch norm (loffe & Szegedy, 2015) is applied to all convolutional layers that do not directly generate distribution transform parameters. We randomly crop sequence of length 13 from all sequences and evaluate on the last 10 frames. (For 2-flow models we crop sequence of length 16 fill up all buffers.) Anonymized code is available the following link.
batch size of 8 and train for 90, 000 iterations for all
de
140
layer CNN
8 2
1 layer CNN
t-3
t-2 t-1
All convolution layers set with with kernel size (3.3) stride and padding
Figure 5: Implementation Visualization of the autoregressive flow.
13

---

Under review as a conference paper at ICLR 2020
CRNNTOO 4.7.5
(a)
(b)
Figure 6: Model Architecture Diagrams. Diagrams shown for the (a) approximate posterior, (b) conditional prior, and (c) conditional likelihood of the sequential latent variable model. conv denotes a convolutional layer, LSTM denotes long short-term memory layer, fc denotes fully- connected layer, and t_conv denotes transposed convolutional layer. For conv and t_conv layers, the numbers in parentheses respectively denote the number of filters, filter size, stride, and padding of the layer. For fc and LSTM layers, the number in parentheses denotes the number of units.
Table 2: Number of parameters for each model on each dataset. Flow-based models contain relatively few parameters as compared with the SLVM, as our flows consist primarily of convolutions with limited channels. In the SLVM, we use 2 LSTM layers for modelling the prior and posterior distribution latent variable while in the combined model we use L.STM layer for each.
Model
1-AF
2-AF
SLVM
SLVM w/ 1-AF
Moving Mnist
343k
686k
11302k
10592k
BAIR Robot Pushing
363k
726k
11325k
10643k
KTH Action
343k
686k
11302k
10592k
C ADDITIONAL EXPERIMENTAL RESULTS
C.I QUANTITATIVE EVALUATION OF TEMPORAL DECORRELATION
The qualitative results in Figures 4 and 8 demonstrate that flows are capable of removing much of the structure of the observations, resulting in whitened noise images. To quantitatively confirm the temporal decorrelation resulting from this process, we evaluate empirical correlation between successive frames, averaged over spatial locations and channels, for the data observations and noise variables. This is an average normalized version of the cuto-covariance of each signal with a time delay of time step. Specifically, we estimate the temporal correlation
H.W.C. Σ
(a) (6.j.) ((
(R...) -nllss)))
corrx = C+
E Ea (5.5
(31)
(o((..10))2
where (1..)) denotes the value the image at location (i.j) and channel k, ((...)) denotes the mean of this dimension, and (i.j.)) denotes the standard deviation of this dimension. H, W, and C respectively denote the height, width, and number of channels of the observations.
We evaluated this quantity for data examples, x, and noise variables, y. for SLVM w/ 1-AF. The results for training sequences shown in Table 4. In Figure we plot this quantity during training for KTH Actions. We see that flows do indeed result a decrease in temporal correlation. Note that because correlation is measure of linear dependence, cannot conclude from these results alone
14

---

Under review as a conference paper at ICLR 2020
Table Training Quantitative Comparison. Average training log-likelihood (higher is better) in nats per pixel per channel for Moving MNIST, BAIR Robot Pushing, and KTT Actions. For flow- based models (1-AF and 2-AF), we report the average log-likelihood. For sequential latent variable models (SLVM and SLVM w/ I-AF), we report the average lower bound the sogliikli.hod.
M-MNIST
BAIR
KTH
1-AF
2206
-298
-2.95
2-AF
-204
-2.76
-2.95
SLVM
1.93
-3.46
-3.05
SLVM w/ I-AF
2 1.85
2 -2331
2 -2.21
Table 4: Temporal Correlation. Temporal correlation (Eq. 31) between successive time steps for data observations, x, and noise variables, y, for SLVM w/ 1-AF.
M-MNIST
BAIR
KTH
COITx
0.24
0.87
0.96
corry
0.02
0.43
0.31
0.5-
0.5
0.4-
0.3-
10000
20000
10:03
Training Herations
Figure 7: Temporal Correlation During Training. corry during training for SLVM w/ 1-AF on the KTT Actions. Temporal correlation decreases substantially during training.
that the flows have resulted in simplified temporal structure. However, these results agree with the qualitative and quantitative results presented in Section 4, suggesting that autoregressive flows can yield sequences with simpler dynamics.
15

---

Under review as a conference paper at ICLR 2020
C.2
ADDITIONAL QUALITATIVE RESULTS
(a)
(b)
Figure 8: Flow Visualization on KTH Action. Visualization the flow component for (a) stan- dalone fow-based models and (b) sequential latent variable models with flow-based conditional likelihoods for KTH Actions. From top to bottom, each figure shows 1) the original frames, x,. 2) the predicted shift, (x<t), for the frame, the predicted scale, oa (x(x))), for the frame, and 4) the noise, Yt. obtained from the inverse transform.
5
5
5
S
5
j
5
5
S
S
5
6
5
<
6
S
$
Figure 9: SLVM w/ 2-AF Visualization on Moving MNIST. Visualization of the flow component for sequential latent variable models with 2-layer flow-based conditional likelihoods for Moving MNIST, From top bottom on the left side, each figure shows 1) the original frames, X,, 2) the lower-level predicted shift, (x,e). for the frame, 3) the predicted scale, (xxr), for the frame. On the right side, from top to bottom, we have I) the higer-level predicted shift, (x)r), for the frame, 3) the predicted scale, oj (x)),, for the frame and 4) the noise, yı, obtained from the inverse transform.
16

---

Under review as a conference paper at ICLR 2020
B
5
88
Bg
2
5
20
L
3
2
2
82
&
R
2
2
8
4 4
#
S
S
3
3
Figure 10: Generated Moving MNIST Samples. Samples frame sequences generated from a 2-AF model.
Figure 11: Generated BAIR Robot Pushing Samples. Samples frame sequences generated from SLVM w/ I-AF.
17


Reviews:
[{"review_id": "Skld8grnYB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to model temporal sequences using autoregressive flows across time steps, that allow to model more explicitly temporal changes of the input, i.e. how the input x_t has changed w.r.t x_{<t}. As also stated by the authors, this is a generalization of other work that instead of modelling the input at each time step, models temporal differences between consecutive time steps.\nTo the best of my knowledge, this is the first work that models normalizing flows in the sequential setting in this way (to be fair however, the idea is fairly obvious).\n\nOverall I found the paper interesting, and I think it is well written, so I am leaning towards acceptance. My biggest concern in the paper is the experimental section that could be improved in several ways:\n- the paper misses broader perfoemance comparisons against other state of the art models, in particular videoflow which is quite related to the models introduced in this paper.\n- how does the model perform on longer sequences, e.g. for long term generation? I would expect that such a direct dependence of the temporal dynamics on the frames of the video may make it hard for the model to coherently predict future latent states for many time steps.\n- What would happen if we used the same trick of modelling the conditional likelihood in this way in other SOTA models?\n- what are the computational requirements of the models presented in this paper? ", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "HylbHoHSFS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nSummary \nThe paper proposes to combine the video modeling approaches based on autoregressive flows (e.g. Kumar’19) with amortized variational inference (e.g. Denton’18), wherein an autoregressive latent variable model optimized with variational inference is extended with an autoregressive flow that further transforms the output of the latent variable model while allowing to compute exact conditional probability. This is motivated with a physical intuition, where a dynamics model can benefit from decorrelating the inputs, and it is demonstrated that layers of autoregressive flows can represent derivatives of the original signal. In a proof-of-concept experiment, it is shown that using a layer of autoregressive flow improves NLL of a latent variable model.\n\nDecision\nThe paper presents an interesting method and tackles an important problem. At the same time, the properties of the proposed method are not well exposed and the experimental evaluation is incomplete. Moreover, the motivation of the paper is confusingly disconnected from the proposed model. I rate this paper as borderline, but am hopeful that some of the issues will be clarified during the discussion period.\n\nPros\n- The paper is well-motivated and tackles a significant problem.\n- The proposed method is novel.\n- The paper is well-written.\n\nCons\n- The experimental evaluation is incomplete and does not expose the properties of the method fully. Comparisons to prior art are missing. (see below)\n- The motivation is disconnected from the proposed model. The introduction of the paper motivates a model that hierarchically decorrelates a sequence of frames to arrive at a fully factorized model, which is later motivated with a physical example. However, the method proposed in the paper is instead a single layer of autoregressive flow on top of a powerful latent variable model! This is expressed in the title, but only glossed over in the abstract and introduction. The writing has to be updated to coherently focus on the contribution of the paper. \n\nQuestions (ordered by decreasing importance)\n1. In table 1, quantitative results are reported for the introduced methods. It is shown that introducing autoregressive flows achieves better likelihood and better generalization. However, quantitative comparisons with published methods that were evaluated on these datasets are missing, such as Denton’18 and Kumar’19. A quick calculation shows that Kumar et al. achieves a log-likelihood of -0.43 in Table 1 when converted to this paper’s metric, although it is possible my conversion is incorrect. Is the presented model competitive with previously published results? \n2. No qualitative generation results are presented. Since the model achieves a high likelihood it is likely to do well on one-frame prediction, and possibly would even work on autoregressive multi-step prediction. Is the model capable of generation of diverse and plausible video?\n3. The paper has a lengthy section 3.1 that convincingly explains that decorrelating latent variables in time is important for sequence modeling. However the proposed approach in fact produces latents that are correlated in time! Since the prior over latent variables is conditioned on past frames, the model can in fact learn a correlated representation and still achieve optimal likelihood. Moreover, the position of both the digit and the robot arm could be seen in what should be the decorrelated image in Fig 4. Is there solid quantitative (or even qualitative) evidence that the model learns a ‘more decorrelated’ representation beyond the fact that it copies the background and that the likelihood improves? The evaluation in this paper does not convince me that the model learns a temporally decorrelated representation.\n4. Were modern techniques beyond affine flows considered, such as from Kingma’18, Kumar’19? Two layers of affine flows are likely insufficient to model the complexity of these data, which makes the comparison to the purely flow-based models somewhat unfair.\n5. It is stated that the paper is “the first to demonstrate flows across time steps for video data”, however, the related work by Kumar et al. proposes a somewhat similar model in which conditional flows are used to model video data. Do Kumar et al. not “demonstrate flows across time steps”?\n\nMinor comments\n1. Eq (10) and (12) seem to be inconsistent. Perhaps x_t = x_t-1 + u_t-1 was meant in eq (10)?\n2. Line before eq(14): it not true that u_t-1 = x_t-1 - x_t-2. It would be true if the deterministic x_t = x_t-1 + u_t-1 model was assumed instead of the gaussian N(x_t; x_t-1 + u_t-1, Sigma). It is possible that eq(14) is still correct as the variance of Gaussians is additive.\n3. The following work uses autoregressive flows for modeling temporal dynamics and should be cited: Rhinehart’18,19\n\nRhinehart et al, Deep Imitative Models for Flexible Inference, Planning, and Control\nRhinehart et al, PRECOG: PREdiction Conditioned On Goals in Visual Multi-Agent Settings\n\n--------------------- Update 11.19 -----------------------\nThe newly provided experiments support some of the claims of the paper. In particular, I appreciate the plot showing that the proposed method successfully learns a more decorrelated representation over time, and the provided qualitative samples from the model. The authors also clarified my questions about motivation. At the same time, the proposed method is not shown to compare well to state-of-the-art approaches. I am leaning towards accepting the paper, but I believe the method would have a much larger impact if its properties were more fully exposed.\n\n== comparison with Denton&Fergus'18 (SVG) ==\nWhen trained with beta=1, as the authors suggest for comparison, this method is known to perform poorly. There are two possible ways of alleviating this: 1) to train with the modified objective as in the paper but evaluate the true lower bound on the likelihood, or 2) interpret the beta as the fixed variance of the decoder distribution. Given the results the authors have provided, I believe the latter option will lead to SVG outperforming the proposed approach.\n\n== Correlation plot == \nThanks for performing this experiment! While measuring correlation only captures linear dependencies, which is likely mostly the background image, this plot shows that the model indeed learns to (linearly) decorrelate the frames in the sequence. \n\n== Samples == \nThanks for providing samples from the model! While the performance on BAIR is not quite convincing, the MNIST samples look very good.\n\n= Kumar et al. comparison ==\nThe author's response convinces me that the proposed model is significantly different from Kumar et al. in scope, as Kumar et al simply use a per-frame normalizing flow encoder coupled with a sequential prior.\n\n== eqs. 10, 12 ==\nThe authors' response cleared my confusion, the equations are correct.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "HyejGEhEFS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThe paper discusses ways to use autoregressive flows in sequence modelling. Two main variants are considered:\n(a) An affine autoregressive flow directly modelling the data.\n(b) An affine autoregressive flow whose base distribution is a sequential VAE; equivalently, a sequential VAE whose decoder is an affine autoregressive flow.\n\nPros:\n\nThe paper is very well written and crystal clear. I particularly appreciated the motivating example that shows how each layer of an affine autoregressive flow reduces the order of a linear dynamical system by 1, and the connections with modelling temporal changes and moving reference frames.\n\nThe methods are technically correct and well-motivated. The experiments are done well.\n\nOverall, the paper scores high on writing and technical quality.\n\nCons:\n\nIn my opinion, the paper scores low on novelty and original contribution.\n\nIn general, it's not clear to me what the claimed contribution is. More specifically:\n\nIs the claimed contribution new methodology for modelling sequences? In my opinion, using flows as VAE decoders, or adding latent variables to a flow model and training it variationally, are standard applications of existing techniques and I wouldn't consider them particularly novel.\n\nIs the claimed contribution improved modelling performance? The main results are that (a) replacing Gaussian decoders with autoregressive flows improves performance, and (b) adding latent variables to the base distribution of an affine autoregressive flow also improves performance. Both of these results are exactly what one would expect from our experience with these methods. Other than that, the paper doesn't present any results that indicate the particular models used enable us to do things we couldn't do before, or improve against the state of the art in sequence modelling.\n\nIs the claimed contribution useful representations? The motivation for using the flow in this particular way as a VAE decoder is that the flow will model low-level correlations whereas the latent variables will capture high-level dynamics. However, the experiments (e.g. the visualizations) don't support this claim, and the usefulness of the learned representations hasn't been demonstrated in an alternative way,\n\nDecision:\n\nEven though the paper is technically correct and well written, my decision is weak reject because of the lack of novelty and original contribution.\n\nSuggestions for improvement:\n\nMy main suggestion to the authors is to keep up the good work, but also reflect on what the specific contribution of the paper is, and try to make a stronger case for it. Some minor suggestions/corrections follow:\n\nEq. (8): As written, the expression makes little sense as \\sigma is a vector. I understand that there is supposed to be a sum over the elements of log\\sigma, so I'd suggest expressing that more clearly.\n\nEq. (9): It seems to me that the last Jacobian is upside down.\n\nIn general, it would be good to be more thorough on how this paper is similar to related work and how it differs. There is also this related work which may be good to discuss:\n\nLatent Normalizing Flows for Discrete Sequences, https://arxiv.org/abs/1901.10548\n\nIn the particle analogy of the motivating example of section 3.1, it would be good to say explicitly that x is the position, u is the velocity and w is the force, to make the example even more intuitive.\n\nThe paper only considers affine autoregressive flows, but there has been a lot of recent work on non-affine autoregressive flows that are more expressive, for example:\n\nNeural Autoregressive Flows, https://arxiv.org/abs/1804.00779\nSum-Of-Squares Polynomial Flow, https://arxiv.org/abs/1905.02325\nNeural Spline Flows, https://arxiv.org/abs/1906.04032\n\nSuch flows could improve the experimental results of the paper. At the very least, it would be good to discuss them as more flexible alternatives.\n\nIn section 3.2, a third and very significant limitation of the flows discussed here is that they act elementwise on the dimensions (e.g. pixels) of y_t.\n\nIn the experimental section, it would be good to describe on a high level what the architecture of the VAE is, especially the architecture of the prior and the encoder, and the types of distributions used there (e.g. diagonal Gaussians or otherwise).\n\nIt would be good to show samples from the models in the experimental results.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper scores low on novelty. The experiments and model analysis are not very strong.
Decision: Reject