Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 1: Reject
review_content: The paper demonstrates how normalising flows can be conditioned. The method is then demonstrated on a set of sequential experiments which show improvements over the considered base lines.

I recommend rejection of the paper, but I can see me changing that assessment if certain improvements are made. The central points are:
- the paper has errors,
- the paper does not respect some related work and has been published previously in parts,
- the paper has a claim that is unsupported in my view,
- the paper is overcrowded with annoying marketing language; the word "novel" appears 16 times according to my pdf viewer.

In general I like the idea, and the presentation seems solid to a large degree. However, the above points are a show stopper for me personally.

For one, the statements 

- p(y|x) = p(y|x, z) p(z | x) and
- p(y|x) = p(y|z) p(z|x),

are problematic. I would like the authors to clarify how they arrive at these.

The paper starts with the claim that "prior work [...] imposes a uni-modal standard Gaussian prior on the lagent variables". This is just wrong. The whole literature of stochastic recurrent models does not do this. See  [1, 2] for starting points. Since the authors place their work in the setup of sequential prediction, this is what has to be respected.

Further, the authors do not seem to be aware of a recently published work [3] that adresses *exactly* this problem. To quote from their abstract: "To this end, we modify the latent variable model by defining the likelihood as a function of the latent vari- able only and [sic] introduce an expressive multimodal prior to enable the model for capturing semantically meaningful features of the data."

I have two more questions with respect to the proposed regularisations.

First, I would ask the authors to comment on the relationship of cR and the method proposed in [3]. To me, it appears as if cR is not novel, but has instead been proposed in [3] previously.

Second, pR fixes the variance of q. The authors claim that the normalising flow of the conditional can undo this fixing by adequately scaling the prior. Hence, so the claim, the expressivity of the model is not reduced.

This prohibits the posteriors of two distinct data points to share the same mean but not share the same variance. 

I request the authors to make a more formal analysis of this, as I do am not convinced how the expressivity of the model is maintained and what influence this has on the ELBO.



References
[1] Bayer, Justin, and Christian Osendorfer. "Learning stochastic recurrent networks." arXiv preprint arXiv:1411.7610 (2014).
[2] Chung, Junyoung, et al. "A recurrent latent variable model for sequential data." Advances in neural information processing systems. 2015.
[3] Klushyn, Alexej, et al. "Increasing the Generalisaton Capacity of Conditional VAEs." International Conference on Artificial Neural Networks. Springer, Cham, 2019.
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: The work proposes a method to improve conditional VAE with a learnable prior distribution using normalizing flow. The authors also design two regularization methods for the CF-VAE to improve training stability and avoid posterior collapse. The paper is clearly motivated and easy to follow. Experiment results on MNIST, Stanford Drone and HighD datasets show the proposed that the model achieves better results than previous state-of-the-art models by significant margins.

However, the reviewer has the following comments on improving the paper:

The motivation of the conditional normalizing flow design could be made more clear. The posterior regularization originates from the problem that the log Jacobian term encourages contraction of the base distribution. The log Jacobian term would be zero and would not encourage the contraction of the base distribution if the normalizing flow was volume-preserving, like NICE (http://proceedings.mlr.press/v37/rezende15.pdf, https://arxiv.org/pdf/1410.8516.pdf), which could be to convert into a conditional normalizing flow. On the MNIST results, the CF-VAE model with the proposed conditional normalizing flow even has worse performance than the affine flow model without the regularization. Therefore, clarifying the motivation behind this design choice is important.

The work claims the two regularization methods are used to avoid a low-entropy prior and posterior collapse. But the claims are not fully substantiated in the experimental results. It would be better if the paper explicitly compares the CF-VAE models with and without regularizations in terms of the entropy of prior distribution and KL divergence.
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: The paper proposes a combination of conditional VAE wtih normalising flows priors and posterior regularisation strategies to capture the diversity of multi-modal trajectories of complex motion patterns. The paper argues that more flexible priors over the latent space can provide posteriors that more closely resemble the trajectories observed in the training data. To this end, the paper presents a derivation of the evidence lower bound for VAEs with normalising flows and discusses the effect of fixing the variance of the posterior to reduce instability during training. Additionally, it shows that conditioning the regularisation on whether or not the dataset contains a dominating mode leads to more diversity and captures minor modes more effectively. Experiments are reported on sequence datasets of handwritten digits, and two datasets with trajectories of vehicles in traffic. 

A central point the paper makes is the importance of prior distributions for the latent space in VAEs such that it can capture diverse modes of trajectories. It is well known that more flexible priors such as MoG lead to better generative power as shown in Tomczak and Welling, 2018. The paper focuses on an extension of the work by Ziegler and Rush, 2019 which proposes normalising flows as priors to capture sequences, conditioned on the initial part of the trajectory. This extension is relatively simple, but does address the specifics of the problem well. 

In general, the paper is well written and clear. The main innovation, in my opinion, is the combination of several ideas applied to the problem of sequence prediction. While none of the ideas, in isolation, are significantly new, the combination can be useful to this particular problem. However, I would like feedback from the authors on the following two main points below which are the main weaknesses of the paper:

1) Posterior regularisation: The posterior regularisation strategies, while intuitive, are very ad-hoc and somewhat contrary to the Bayesian framework. It is difficult to see how the variance in pR and the "dominant mode" detector in cR can be estimated automatically. Within a Bayesian framework it would be much more natural to place a prior distribution over the variance and marginalise it out within the variational inference procedure. For the other regularisation (cR), how is the dominant mode detected? 

2) Experiments: A major concern reported throughout the paper is the instability of training and the risk for overfitting. I do not think the experiments demonstrate how stable and robust the method is to different initialisations, seeds, training data shuffles, etc. I strongly suggest the authors to run cross validation experiments and report the mean and standard deviation for all methods being compared. Also, how sensitive are the results to different values of C? How to decide whether to use cR or not when don't have access to the ground truth?
review_title: Official Blind Review #3

Meta-review: The novelty of the proposed work is a very weak factor, the idea has been explored in various forms in previous work.
Decision: Reject