Venue: ICLR 2018 conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper proposes to increase the expressivity of the variational approximation in VAEs using a new convolutional parameterization of normalizing flows. Starting from the planar flow proposed in Rezende & Mohammed 2015 using a vector inner product followed by a nonliniarity+element-wise scaling the authors suggests to replace inner product with a shifted 1-D convolution. This reduces the number of parameters used from 2*d to k + d and importantly still maintains the linear time computation of the determinant. This approach feels so straightforward that i’m surprised that it have not been tried before. The authors present results on a synthetic task as well as MNIST and OMNIGLOT. Please find some more detailed comments/questions below


Q1) I feel that section 3 could be more detailed about how the convolution normalizing flow relate to normalizing flow, inverse autoregressive flow and the masked-convolution used in real NVP? Especifically a) is it correct that convolutional normalizing flow trades global connectivity for more expressivity locally? b) Can convolutional flow be seen as faster but ´more restricted version of the LSTM implemented inverse autoregressive flow (full lower triangular jacobian vs k off diagonal elements per row in convolutional normalizing flow) 

Q2) I miss some more baselines in the experimental section. Did the authors compare the convolutional normalizing flow with e.g. Inverse Autoregressive flow or Auxiliary latent variables? 


Q3) Albeit the MNIST results seems convincing - and to a lesser degree the OMNIGLOT ones - I miss results on larger natural image benchmark datasets like cifar10 and ImageNet or preferably other modalities like text? Would it be possible to include results on any of these datasets?

Overall i think the idea is nice and potentially useful due to the ease of implementation and speed of convolutional operations. However I think the authors needs to 1) better describe how their method differs from prior work and 2) compare their method to more baselines for the experiments to fully convincing

review_title: Review of Convolutional Normalizing Flows

final_score: rating: null
review_content: The conclusion of the paper says "density estimates on MNIST show significant improvements over state-of-the-art methods". This is misleading, as the results table ignores all recent results in this area. E.g. PixelVAE, Lossy VAE, PixelCNN, and IAF-VAE (some of which are cited) all obtain much better results. Is there any reason the proposed method should not be compared against these newer methods?
review_title: Why no comparison to recent results?

final_score: rating: 3: Clear rejection
review_content: The authors propose a new method for improving the flexibility of the encoder in VAEs, called ConvFlow. If I understand correctly (please correct me if not) the proposed method is a simplification of Inverse Autoregressive Flow as proposed by Kingma et al. Both of these methods use causal convolution to construct a normalizing flow with tractable Jacobian determinant. The difference is that Kingma et al. used 2d convolution (as well a fully connected architectures) where the authors of this paper propose to use 1d convolution. The novelty therefore seems limited.

The current version of the paper does not present convincing experimental results. The proposed method performs less well than previously proposed methods. If the authors were to update the experimental results to show equal or better performance to SOTA, with an analysis showing their method is indeed computationally less expensive, I would be willing to increase my rating.
review_title: novelty is limited. currently no impressive experimental results

final_score: rating: 3: Clear rejection
review_content: In this paper, the authors propose a type of Normalizing Flows (Rezende and Mohamed, 2015) for Variational Autoencoders (Kingma and Welling, 2014; Rezende et al., 2014) they call Convolutional Normalizing Flows.
More particularly, it aims at extending on the Planar Flow scheme proposed in Rezende and Mohamed (2015). The authors notice an improvement through their method over Normalizing Flows, IWAE with diagonal gaussian approximation, and standard Variational Autoencoders. 
As noted by AnonReviewer3, several baselines are missing. But the authors partly address that issue in the comment section for the MNIST dataset.
The requirement of h being bijective seems wrong. For example, if h was a rectifier nonlinearity in the zero-derivative regime, the Jacobian determinant of the ConvFlow would be 1. 
More importantly, the main issue is that this paper might need to highlight the fundamental difference between their proposed method and Inverse Autoregressive Flow (Kingma et al., 2016). The proposed connectivity pattern proposed for the convolution in order to make the Jacobian determinant computation is exactly the same as Inverse Autoregressive Flow and the authors seems to be aware of the order dependence of their architecture which is every similar to autoregressive models. This presentation of the paper can be misleading concerning the true innovation in the model trained. Proposing ConvFlow as a type of Inverse Autoregressive Flow would be more accurate and would allow to highlight better the innovation of the work.
Since this work does not offer additional significant insight over Inverse Autoregressive Flow, its value should be on demonstrating the efficiency of the proposed method. MNIST and Omniglot seems insufficient for that purpose given currently published work.
In the current state, I can't recommend the paper for acceptance. 


Danilo Jimenez Rezende, Shakir Mohamed: Variational Inference with Normalizing Flows. ICML 2015
Danilo Jimenez Rezende, Shakir Mohamed, Daan Wierstra: Stochastic Back-propagation and Variational Inference in Deep Latent Gaussian Models. ICML 2014
Diederik P. Kingma, Max Welling: Auto-Encoding Variational Bayes. ICLR 2014
Diederik P. Kingma, Tim Salimans, Rafal Józefowicz, Xi Chen, Ilya Sutskever, Max Welling: Improving Variational Autoencoders with Inverse Autoregressive Flow. NIPS 2016
review_title: A form of Inverse Autoregressive Flow?

Meta-review: Thank you for submitting you paper to ICLR. ICLR. Although there revision has improved the paper, the consensus from the reviewers is that this is not quite ready for publication.
Decision: Reject