Venue: ICLR 2017 conference

Reviews:

final_score: rating: null
review_content: The authors claim that “as opposed to [variational autoencoders], real NVP does not rely on fixed form reconstruction cost like an L2.” I can see where the authors get their intuition, nevertheless this statement seems overly simplified/wrong. For small decoder noise, a larger latent space and assuming a strong enough encoder so that the lower bound is close to the likelihood, the differences between what real NVP and a VAE optimize seem to become very small*. Both use a very similar generative model and (approximately) optimize its likelihood. Am I missing something/can you clarify what you mean?

* Note that I am not trying to say that VAE and real NVP are the same, just that their objective functions _can be_ very similar.
review_title: Likelihood vs L2 reconstruction

final_score: rating: 7: Good paper, accept
review_content: Building on earlier work on a model called NICE, this paper presents an approach to constructing deep feed-forward generative models. The model is evaluated on several datasets. While it does not achieve state-of-the-art performance, it advances an interesting class of models. The paper is mostly well written and clear.

Given that inference and generation are both efficient and exact, and given that this represents a main advantage over other models, it would be great if the authors could provide some motivating example applications where this is needed/would be useful.

The authors claim that “unlike both variational autoencoders and GANs, our technique is able to learn a semantically meaningful latent space which is as high dimensional as the input space.” Where is the evidence for this claim? I didn’t see any analysis of the semantic meaningfulness of the latent space learned by real NVP. Stronger evidence that the learned representations are actually useful for downstream tasks would be nice.

I still think the author’s intuitions around the “fixed reconstruction cost of L2” are very vague. The factorial Gaussian assumption itself does not limit the generative model, it merely smoothes an otherwise arbitrary distribution, and to a degree which can be arbitrarily small, p(x) = \int p(z) N(x | f(z), \sigma^2) dz. How a lose lower bound plays into this is not clear from the paper.
review_title: null

final_score: rating: null
review_content: Why use a cosine interpolation instead of a linear one (Fig. 6)?
Why aren't you using your batch normalization with moving average on other datasets than CIFAR-10 (Appendix E)?
Some train and valid results in Table 1 seem pretty close. Are they statistically different?
review_title: Latent space interpolation, batch norm with moving average and close train/valid results

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This paper proposes a new generative model that uses real-valued non-volume preserving transformations in order to achieve efficient and exact inference and sampling of data points.
The authors use the change-of-variable technique to obtain a model distribution of the data from a simple prior distribution on a latent variable. By carefully designing the bijective function used in the change-of-variable technique, they obtain a Jacobian that is triangular and allows for efficient computation.

Generative models with tractable inference and efficient sampling are an active research area and this paper definitely contributes to this field.

While not achieving state-of-the-art, they are not far behind. This doesn't change the fact that the proposed method is innovative and worth exploring as it tries to bridge the gap between auto-regressive models, variational autoencoders and generative adversarial networks.

The authors clearly mention the difference and similarities with other types of generative models that are being actively researched.
Compared to autoregressive models, the proposed approach offers fast sampling.
Compared to generative adversarial networks, Real NVP offers a tractable log-likelihood evaluation.
Compared to variational autoencoders, the inference is exact.
Compared to deep Boltzmann machines, the learning of the proposed method is tractable.
It is clear that Real NVP goal is to bridge the gap between existing and popular generative models.

The paper presents a lot of interesting experiments showing the capabilities of the proposed technique. Making the code available online will certainly contribute to the field. Is there any intention of releasing the code?

Typo: (Section 3.7) We also "use apply" batch normalization
review_title: Interesting ideas, great empirical work, overall good contribution

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This paper presents a clever way of training a generative model which allows for exact inference, sampling and log likelihood evaluation. The main idea here is to make the Jacobian that comes when using the change of variables formula (from data to latents) triangular - this makes the determinant easy to calculate and hence learning possible. 
The paper nicely presents this core idea and a way to achieve this - by choosing special "routings" between the latents and data such that part of the transformation is identity and part some complex function of the input (a deep net, for example) the resulting Jacobian has a tractable structure. This routing can be cascaded to achieve even more complex transformation. 

On the experimental side, the model is trained on several datasets and the results are quite convincing, both in sample quality and quantitive measures. 
I would be very happy to see if this model is useful with other types of tasks and if the resulting latent representation help with classification or inference such as image restoration.

In summary - the paper is nicely written, results are quite good and the model is interesting - I'm happy to recommend acceptance.


review_title: Interesting and important work

Meta-review: This paper clearly lays out the recipe for a family of invertible density models, explores a few extensions, and demonstrates the overall power of the approach. The work is building closely on previous approaches, so it suffers a bit in terms of originality. However, I expect this overall approach to become a standard tool for model-building.
 
 The main weakness of this paper is that it did not explore the computational tradeoffs of this approach against related methods. However, the paper already had a lot of content.
Decision: Accept (Poster)