Title: Density estimation using Real NVP
Authors: Laurent Dinh
Jascha Sohl-Dickstein
Samy Bengio
Venue: ICLR 2017
Keywords: Deep learning
Unsupervised Learning

Abstract: Unsupervised learning of probabilistic models is a central yet challenging problem in machine learning. Specifically, designing models with tractable learning, sampling, inference and evaluation is crucial in solving this task. We extend the space of such models using real-valued non-volume preserving (real NVP) transformations, a set of powerful invertible and learnable transformations, resulting in an unsupervised learning algorithm with exact log-likelihood computation, exact sampling, exact inference of latent variables, and an interpretable latent space. We demonstrate its ability to model natural images on four datasets through sampling, log-likelihood evaluation and latent variable manipulations.

TL;DR: Efficient invertible neural networks for density estimation and generation

Review discussion:
Why use a cosine interpolation instead of a linear one (Fig. 6)?
Why aren't you using your batch normalization with moving average on other datasets than CIFAR-10 (Appendix E)?
Some train and valid results in Table 1 seem pretty close. Are they statistically different?
Hi,
I used cosine interpolation because, in high dimension, the mass of the gaussian distribution is mainly concentrated on a sphere (the norm-2 of a multivariate standard gaussian variable follows a chi-squared distribution).
Hence, a cosine interpolation would mainly travel in area of high probability according to the prior distribution.
We are using batch normalization with moving average only on CIFAR-10 because this is where it had noticeable effect, the choice of the batch normalization scheme for the other datasets did not seem to make a difference.
The train and valid results remained close during training on large scale datasets, which seems to suggest that we are not overfitting.
The authors claim that “as opposed to [variational autoencoders], real NVP does not rely on fixed form reconstruction cost like an L2.” I can see where the authors get their intuition, nevertheless this statement seems overly simplified/wrong. For small decoder noise, a larger latent space and assuming a strong enough encoder so that the lower bound is close to the likelihood, the differences between what real NVP and a VAE optimize seem to become very small*. Both use a very similar generative model and (approximately) optimize its likelihood. Am I missing something/can you clarify what you mean?
* Note that I am not trying to say that VAE and real NVP are the same, just that their objective functions _can be_ very similar.
Hi,
Their objective functions _can be_ very similar indeed as Real NVP optimizes the exact log-likelihood whereas VAE optimizes the Evidence Lower Bound of the log-likelihood and their closeness depends on how much the approximate posterior q(z|x) matches the true posterior p(z|x)=p(z)p(x|z)/p(x). The Variational Lossy Autoencoder paper provides further discussion on the subject: https://openreview.net/forum?id=BysvGP5ee.
However, we are referring to the standard VAE algorithms from (Kingma and Welling, 2013) and (Rezende et al., 2014) where p(x|z) is factorial. This results in a fixed form reconstruction loss -log(p(x|z)), a L2 loss for a conditional gaussian p(x|z).
Even if you have a small decoder noise, i.e. p(x|z) has small entropy, this reconstruction loss remains fixed nonetheless. I agree that it would not be the case if p(x|z) was a more complex conditional distribution as in "PixelVAE: A Latent Variable Model for Natural Images" and "Variational Lossy Autoencoder", two papers that you will find among ICLR submissions this year.
Relying on fixed form reconstruction cost is not an issue in an infinite capacity setting for either algorithm.
Building on earlier work on a model called NICE, this paper presents an approach to constructing deep feed-forward generative models. The model is evaluated on several datasets.
While it does not achieve state-of-the-art performance, it advances an interesting class of models. The paper is mostly well written and clear.
Given that inference and generation are both efficient and exact, and given that this represents a main advantage over other models, it would be great if the authors could provide some motivating example applications where this is needed/would be useful.
The authors claim that “unlike both variational autoencoders and GANs, our technique is able to learn a semantically meaningful latent space which is as high dimensional as the input space.” Where is the evidence for this claim? I didn’t see any analysis of the semantic meaningfulness of the latent space learned by real NVP. Stronger evidence that the learned representations are actually useful for downstream tasks would be nice.
I still think the author’s intuitions around the “fixed reconstruction cost of L2” are very vague. The factorial Gaussian assumption itself does not limit the generative model, it merely smoothes an otherwise arbitrary distribution, and to a degree which can be arbitrarily small, p(x) = \int p(z) N(x | f(z), \sigma^2) dz. How a lose lower bound plays into this is not clear from the paper.
Thanks for your input!
"Given that inference and generation are both efficient and exact, and given that this represents a main advantage over other models, it would be great if the authors could provide some motivating example applications where this is needed/would be useful."
Efficient inference is necessary for tractable training in most probabilistic models and efficient sampling is useful for application that would require fast sampling, e.g. music and speech synthesis and real-time rendering. Having exact inference allows us to accurately and exactly evaluate our models, and could be highly beneficial for tasks such as semi-supervised learning, which require accurately estimating a latent state. Having exact sampling as opposed to MCMC sampling efficiently prevents us from having highly correlated samples (and poor diversity of samples).
We have added this to the submission as an additional argument.
"The authors claim that “unlike both variational autoencoders and GANs, our technique is able to learn a semantically meaningful latent space which is as high dimensional as the input space.” Where is the evidence for this claim? I didn’t see any analysis of the semantic meaningfulness of the latent space learned by real NVP. Stronger evidence that the learned representations are actually useful for downstream tasks would be nice."
Thank you for pointing out this claim. The aim of this sentence was to focus more on the "as high dimensional as the input space" aspect as opposed to VAEs and GANs, which previously used effectively low dimensional latent codes for their models.
As for showing that the latent space is semantically meaningful, based on your feedback we have added an additional experiment to the paper (Appendix F), showing that in class-conditional image generation latent variables have a consistent semantic meaning across classes.
"I still think the author’s intuitions around the “fixed reconstruction cost of L2” are very vague. The factorial Gaussian assumption itself does not limit the generative model, it merely smoothes an otherwise arbitrary distribution, and to a degree which can be arbitrarily small, p(x) = \int p(z) N(x | f(z), \sigma^2) dz. How a lose lower bound plays into this is not clear from the paper."
What you are saying is true but your statement is about the class of generative models associated to Variational Auto-Encoders. The Variational Auto-Encoder learning also includes an encoder for the approximate posterior. Often this approximation is not enough which often results in under exploiting the capacity that these generative models (the decoders) have. The issue here is trainability.
This lack of trainability is due to the gap between the approximate posterior and the real posterior. As the reconstruction cost becomes more flexible, the *real* posterior p(z|x) = p(z)p(x|z)/p(x) becomes more able to fit the family of approximate posterior considered.
Since the problem of using fixed form reconstruction cost in VAEs is already mentioned in "Autoencoding beyond pixels using a learned similarity metric", we chose not to put a focus on explaining this issue.
This paper proposes a new generative model that uses real-valued non-volume preserving transformations in order to achieve efficient and exact inference and sampling of data points.
The authors use the change-of-variable technique to obtain a model distribution of the data from a simple prior distribution on a latent variable. By carefully designing the bijective function used in the change-of-variable technique, they obtain a Jacobian that is triangular and allows for efficient computation.
Generative models with tractable inference and efficient sampling are an active research area and this paper definitely contributes to this field.
While not achieving state-of-the-art, they are not far behind. This doesn't change the fact that the proposed method is innovative and worth exploring as it tries to bridge the gap between auto-regressive models, variational autoencoders and generative adversarial networks.
The authors clearly mention the difference and similarities with other types of generative models that are being actively researched.
Compared to autoregressive models, the proposed approach offers fast sampling.
Compared to generative adversarial networks, Real NVP offers a tractable log-likelihood evaluation.
Compared to variational autoencoders, the inference is exact.
Compared to deep Boltzmann machines, the learning of the proposed method is tractable.
It is clear that Real NVP goal is to bridge the gap between existing and popular generative models.
The paper presents a lot of interesting experiments showing the capabilities of the proposed technique. Making the code available online will certainly contribute to the field. Is there any intention of releasing the code?
Typo: (Section 3.7) We also "use apply" batch normalization
Hi,
Thanks you for reviewing our paper.
"The paper presents a lot of interesting experiments showing the capabilities of the proposed technique. Making the code available online will certainly contribute to the field. Is there any intention of releasing the code?"
Our code is now included in the Tensorflow models open source repository (https://github.com/tensorflow/models/tree/master/real_nvp), in addition to that, our model has been reimplemented  by Taesung Park on Github: https://github.com/taesung89/real-nvp
We will definitely correct the typo you mentioned.
This paper presents a clever way of training a generative model which allows for exact inference, sampling and log likelihood evaluation. The main idea here is to make the Jacobian that comes when using the change of variables formula (from data to latents) triangular - this makes the determinant easy to calculate and hence learning possible.
The paper nicely presents this core idea and a way to achieve this - by choosing special "routings" between the latents and data such that part of the transformation is identity and part some complex function of the input (a deep net, for example) the resulting Jacobian has a tractable structure. This routing can be cascaded to achieve even more complex transformation.
On the experimental side, the model is trained on several datasets and the results are quite convincing, both in sample quality and quantitive measures.
I would be very happy to see if this model is useful with other types of tasks and if the resulting latent representation help with classification or inference such as image restoration.
In summary - the paper is nicely written, results are quite good and the model is interesting - I'm happy to recommend acceptance.

Meta-review:
role: PC
data: comment: This paper clearly lays out the recipe for a family of invertible density models, explores a few extensions, and demonstrates the overall power of the approach. The work is building closely on previous approaches, so it suffers a bit in terms of originality. However, I expect this overall approach to become a standard tool for model-building.
 
 The main weakness of this paper is that it did not explore the computational tradeoffs of this approach against related methods. However, the paper already had a lot of content.

Decision: Accept (Poster)