Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper proposes to model temporal sequences using autoregressive flows across time steps, that allow to model more explicitly temporal changes of the input, i.e. how the input x_t has changed w.r.t x_{<t}. As also stated by the authors, this is a generalization of other work that instead of modelling the input at each time step, models temporal differences between consecutive time steps.
To the best of my knowledge, this is the first work that models normalizing flows in the sequential setting in this way (to be fair however, the idea is fairly obvious).

Overall I found the paper interesting, and I think it is well written, so I am leaning towards acceptance. My biggest concern in the paper is the experimental section that could be improved in several ways:
- the paper misses broader perfoemance comparisons against other state of the art models, in particular videoflow which is quite related to the models introduced in this paper.
- how does the model perform on longer sequences, e.g. for long term generation? I would expect that such a direct dependence of the temporal dynamics on the frames of the video may make it hard for the model to coherently predict future latent states for many time steps.
- What would happen if we used the same trick of modelling the conditional likelihood in this way in other SOTA models?
- what are the computational requirements of the models presented in this paper? 
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: 
Summary 
The paper proposes to combine the video modeling approaches based on autoregressive flows (e.g. Kumar’19) with amortized variational inference (e.g. Denton’18), wherein an autoregressive latent variable model optimized with variational inference is extended with an autoregressive flow that further transforms the output of the latent variable model while allowing to compute exact conditional probability. This is motivated with a physical intuition, where a dynamics model can benefit from decorrelating the inputs, and it is demonstrated that layers of autoregressive flows can represent derivatives of the original signal. In a proof-of-concept experiment, it is shown that using a layer of autoregressive flow improves NLL of a latent variable model.

Decision
The paper presents an interesting method and tackles an important problem. At the same time, the properties of the proposed method are not well exposed and the experimental evaluation is incomplete. Moreover, the motivation of the paper is confusingly disconnected from the proposed model. I rate this paper as borderline, but am hopeful that some of the issues will be clarified during the discussion period.

Pros
- The paper is well-motivated and tackles a significant problem.
- The proposed method is novel.
- The paper is well-written.

Cons
- The experimental evaluation is incomplete and does not expose the properties of the method fully. Comparisons to prior art are missing. (see below)
- The motivation is disconnected from the proposed model. The introduction of the paper motivates a model that hierarchically decorrelates a sequence of frames to arrive at a fully factorized model, which is later motivated with a physical example. However, the method proposed in the paper is instead a single layer of autoregressive flow on top of a powerful latent variable model! This is expressed in the title, but only glossed over in the abstract and introduction. The writing has to be updated to coherently focus on the contribution of the paper. 

Questions (ordered by decreasing importance)
1. In table 1, quantitative results are reported for the introduced methods. It is shown that introducing autoregressive flows achieves better likelihood and better generalization. However, quantitative comparisons with published methods that were evaluated on these datasets are missing, such as Denton’18 and Kumar’19. A quick calculation shows that Kumar et al. achieves a log-likelihood of -0.43 in Table 1 when converted to this paper’s metric, although it is possible my conversion is incorrect. Is the presented model competitive with previously published results? 
2. No qualitative generation results are presented. Since the model achieves a high likelihood it is likely to do well on one-frame prediction, and possibly would even work on autoregressive multi-step prediction. Is the model capable of generation of diverse and plausible video?
3. The paper has a lengthy section 3.1 that convincingly explains that decorrelating latent variables in time is important for sequence modeling. However the proposed approach in fact produces latents that are correlated in time! Since the prior over latent variables is conditioned on past frames, the model can in fact learn a correlated representation and still achieve optimal likelihood. Moreover, the position of both the digit and the robot arm could be seen in what should be the decorrelated image in Fig 4. Is there solid quantitative (or even qualitative) evidence that the model learns a ‘more decorrelated’ representation beyond the fact that it copies the background and that the likelihood improves? The evaluation in this paper does not convince me that the model learns a temporally decorrelated representation.
4. Were modern techniques beyond affine flows considered, such as from Kingma’18, Kumar’19? Two layers of affine flows are likely insufficient to model the complexity of these data, which makes the comparison to the purely flow-based models somewhat unfair.
5. It is stated that the paper is “the first to demonstrate flows across time steps for video data”, however, the related work by Kumar et al. proposes a somewhat similar model in which conditional flows are used to model video data. Do Kumar et al. not “demonstrate flows across time steps”?

Minor comments
1. Eq (10) and (12) seem to be inconsistent. Perhaps x_t = x_t-1 + u_t-1 was meant in eq (10)?
2. Line before eq(14): it not true that u_t-1 = x_t-1 - x_t-2. It would be true if the deterministic x_t = x_t-1 + u_t-1 model was assumed instead of the gaussian N(x_t; x_t-1 + u_t-1, Sigma). It is possible that eq(14) is still correct as the variance of Gaussians is additive.
3. The following work uses autoregressive flows for modeling temporal dynamics and should be cited: Rhinehart’18,19

Rhinehart et al, Deep Imitative Models for Flexible Inference, Planning, and Control
Rhinehart et al, PRECOG: PREdiction Conditioned On Goals in Visual Multi-Agent Settings

--------------------- Update 11.19 -----------------------
The newly provided experiments support some of the claims of the paper. In particular, I appreciate the plot showing that the proposed method successfully learns a more decorrelated representation over time, and the provided qualitative samples from the model. The authors also clarified my questions about motivation. At the same time, the proposed method is not shown to compare well to state-of-the-art approaches. I am leaning towards accepting the paper, but I believe the method would have a much larger impact if its properties were more fully exposed.

== comparison with Denton&Fergus'18 (SVG) ==
When trained with beta=1, as the authors suggest for comparison, this method is known to perform poorly. There are two possible ways of alleviating this: 1) to train with the modified objective as in the paper but evaluate the true lower bound on the likelihood, or 2) interpret the beta as the fixed variance of the decoder distribution. Given the results the authors have provided, I believe the latter option will lead to SVG outperforming the proposed approach.

== Correlation plot == 
Thanks for performing this experiment! While measuring correlation only captures linear dependencies, which is likely mostly the background image, this plot shows that the model indeed learns to (linearly) decorrelate the frames in the sequence. 

== Samples == 
Thanks for providing samples from the model! While the performance on BAIR is not quite convincing, the MNIST samples look very good.

= Kumar et al. comparison ==
The author's response convinces me that the proposed model is significantly different from Kumar et al. in scope, as Kumar et al simply use a per-frame normalizing flow encoder coupled with a sequential prior.

== eqs. 10, 12 ==
The authors' response cleared my confusion, the equations are correct.
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: Summary:

The paper discusses ways to use autoregressive flows in sequence modelling. Two main variants are considered:
(a) An affine autoregressive flow directly modelling the data.
(b) An affine autoregressive flow whose base distribution is a sequential VAE; equivalently, a sequential VAE whose decoder is an affine autoregressive flow.

Pros:

The paper is very well written and crystal clear. I particularly appreciated the motivating example that shows how each layer of an affine autoregressive flow reduces the order of a linear dynamical system by 1, and the connections with modelling temporal changes and moving reference frames.

The methods are technically correct and well-motivated. The experiments are done well.

Overall, the paper scores high on writing and technical quality.

Cons:

In my opinion, the paper scores low on novelty and original contribution.

In general, it's not clear to me what the claimed contribution is. More specifically:

Is the claimed contribution new methodology for modelling sequences? In my opinion, using flows as VAE decoders, or adding latent variables to a flow model and training it variationally, are standard applications of existing techniques and I wouldn't consider them particularly novel.

Is the claimed contribution improved modelling performance? The main results are that (a) replacing Gaussian decoders with autoregressive flows improves performance, and (b) adding latent variables to the base distribution of an affine autoregressive flow also improves performance. Both of these results are exactly what one would expect from our experience with these methods. Other than that, the paper doesn't present any results that indicate the particular models used enable us to do things we couldn't do before, or improve against the state of the art in sequence modelling.

Is the claimed contribution useful representations? The motivation for using the flow in this particular way as a VAE decoder is that the flow will model low-level correlations whereas the latent variables will capture high-level dynamics. However, the experiments (e.g. the visualizations) don't support this claim, and the usefulness of the learned representations hasn't been demonstrated in an alternative way,

Decision:

Even though the paper is technically correct and well written, my decision is weak reject because of the lack of novelty and original contribution.

Suggestions for improvement:

My main suggestion to the authors is to keep up the good work, but also reflect on what the specific contribution of the paper is, and try to make a stronger case for it. Some minor suggestions/corrections follow:

Eq. (8): As written, the expression makes little sense as \sigma is a vector. I understand that there is supposed to be a sum over the elements of log\sigma, so I'd suggest expressing that more clearly.

Eq. (9): It seems to me that the last Jacobian is upside down.

In general, it would be good to be more thorough on how this paper is similar to related work and how it differs. There is also this related work which may be good to discuss:

Latent Normalizing Flows for Discrete Sequences, https://arxiv.org/abs/1901.10548

In the particle analogy of the motivating example of section 3.1, it would be good to say explicitly that x is the position, u is the velocity and w is the force, to make the example even more intuitive.

The paper only considers affine autoregressive flows, but there has been a lot of recent work on non-affine autoregressive flows that are more expressive, for example:

Neural Autoregressive Flows, https://arxiv.org/abs/1804.00779
Sum-Of-Squares Polynomial Flow, https://arxiv.org/abs/1905.02325
Neural Spline Flows, https://arxiv.org/abs/1906.04032

Such flows could improve the experimental results of the paper. At the very least, it would be good to discuss them as more flexible alternatives.

In section 3.2, a third and very significant limitation of the flows discussed here is that they act elementwise on the dimensions (e.g. pixels) of y_t.

In the experimental section, it would be good to describe on a high level what the architecture of the VAE is, especially the architecture of the prior and the encoder, and the types of distributions used there (e.g. diagonal Gaussians or otherwise).

It would be good to show samples from the models in the experimental results.
review_title: Official Blind Review #3

Meta-review: The paper scores low on novelty. The experiments and model analysis are not very strong.
Decision: Reject