Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: The paper proposes a new method for improving generative properties of VAE model. The idea is to train VAE in two stages: at first, train the vanilla VAE, then at the second stage freeze the encoder part and train the decoder part as a GAN generator with an additional regularizer which encourages cycle consistency in the latent space. Also the authors claim that other VAE-GAN hybrids which try to improve VAE model are “misguided” and poor samples and reconstructions of VAE are the consequence of minimum description length problem. 

Concerns:
1) The main concern about this paper is the inaccuracy and very general statements without theoretical or empirical justification. For example, the authors claim that “the whole point of VAEs is to capture only compressible information and discard information specific to any particular image”. What is the definition of “only compressible information” or “information specific to any particular image”? Is there an experiment which can support this statement? Other examples of such general statements: “strength of a VAE is that it builds a model of the dataset that does not over-fit”, “the latent code does not contain enough information to do the reconstruction”, “VAEs are not broken and “fixing” them is actually likely to break them”. 
2) Poor experiment comparisons with other baselines. The authors compare their method only with the vanilla VAE which is clearly insufficient. The authors claim that other VAE-GAN hybrids break the “strength of the VAE”. Could you please provide examples of problems and provide experiments where VAE-GAN baselines will be worse than VAE or the proposed method?
3) The paper structure is very confusing. The main part and experiments part are mixed. Therefore, it is hard to follow the text. The experiment setup is not clearly stated. For example, it is unclear for which dataset luminance-normalized Laplacian was computed. 

Overall, the paper proposes a new method and  gives an alternative view on the VAE model. However, statements from the paper are very pretentious and are not rigorously proven. Also there are not empirical comparisons with other VAE-GAN hybrids. Therefore, I would suggest rejecting the current version.

--------------------------------------------------------
Update after author rebuttal

Thank you for your thoughtful response. However, I still think that the paper does not give new insights about VAE model and has poor experiment justifications of their statements. Considering MDL interpretation of VAE it is not new (see [1]). Therefore, the contribution of this paper is very limited. 

After reviewing the other reviews and the author rebuttal, I do not change my original score.

[1] Xi Chen et al.,  Variational  Lossy  Autoencoder, 2016

review_title: Official Blind Review #1

final_score: rating: 1: Reject
review_content: Summary:
This paper proposes a hybrid VAE-GAN model, called the latent space renderer-GAN (LSR-GAN), with the goal to “imagine” the latent space of a VAE, and to improve the decoding and sampling quality of a VAE. First, a VAE-like model is trained, after which the encoder weights are frozen, and the decoder is trained as the generator of a GAN (together with an auxiliary discriminator). The generator loss also contains a reconstruction-like term in the latent space, described by the negative log density of the encoding distribution of the original latent conditioned on the output of the generator: -log q(z|g(z)). 

Decision: reject
This paper contains incorrect claims. Leaving those mistakes aside, the experiments don’t lead to new insights and no comparison against other VAE-GAN hybrids is made. 

Supporting arguments for decision:
In the introduction the authors state “This is a consequence of the well known evidence lower bound or ELBO objective function consisting of a negative log-probability of generating the original image from the latent representation (this is often implemented as a mean squared error between the image and the reconstruction, although as we argue in Appendix A this term should be proportional to the logarithm of the mean squared error)...”. This statement is surprising and I don’t see how it can be correct. If -log p(x|z) is something like the log of the mean squared error, then the density p(x|z) should be a squared error function, which is not even a valid distribution.

To be more precise, according to the authors, if one takes a Gaussian p(x|z), the reconstruction error should be modeled with  

-log p(x|z) = log 1/N sum (x_i - \hat x_i)^2  + const 					(1)

where \hat x_i is a function of z. Note the logarithm on the right-hand side here. In appendix A, the authors observe that most other implementations instead optimize 

-log p(x|z) = sum_{i=1}^N (x_i -\hat x_i)^2/(2\sigma^2) + N/2 log(2 pi sigma^2) 		(2)

Where often sigma is set to ½ (so that the last term on the lhs of (2) drops out when gradients are taken). The authors claim the latter is incorrect because sigma should be equal to the (biased) empirical variance. They then insert sigma=empirical variance into (2), take derivatives while ignoring the dependence of sigma on \hat x_i, and then arrive at something like eq. 1. They also argue that those implementations that use sigma=½ are actually optimizing a beta-VAE because of this “incorrect” prefactor of the reconstruction error. 
I’m confident that the above claims are not correct. 
As an example, using a Gaussian decoder distribution requires parameterizing the mean and variance of that Gaussian distribution as a function of z. Here, setting sigma = ½ (as is commonly done in other literature) is valid, contrary to the above claim. One is always allowed to just set the variance to a constant and ignore its dependence on z. This just leaves you with a less flexible distribution to model p(x|z).

The authors appear to use (1) as the reconstruction error term in the ELBO during VAE optimization, and then play with different prefactors of the KL term similar to beta-VAEs. The combination of the reconstruction error (1) and the prefactor no longer makes this a VAE, but more like a regularized auto-encoder with an unconventional reconstruction error that should not be interpreted as coming from the negative of a log density. Due to the logarithm in front of the mean squared error, the loss function is less sensitive to large errors in reconstruction. It is surprising that the reconstruction error does not cause overflow as log (0) --> - infinity. 

Comments on experiments and related work: 
- Experiments only show images of reconstructions of a VAE and LSR-GAN, a mean squared error plot of reconstruction errors and an accuracy plot of a classifier evaluated on the reconstructions produced by both models. In terms of MSE the LSR-GAN actually performs worse than their VAE baseline. The authors also train a classifier on cifar-10 and measure its accuracy using the ground truth labels and the reconstructed images of a VAE and LSR-GAN. Here the LSR-GAN performs marginally better than the VAE, but the overall accuracy is very poor.
- No experimental comparison is made against other VAE-GAN hybrids. Related work on hybrid VAE-GANS is discussed in the appendix, not in the main paper.

Other sections:
- Section 3 contains a very long interpretation of the minimum description length (MDL). It is unclear what the goal of this section is, as it does not lead to any results, nor does it help bring the point across as to why the proposed model is good at imagining the latent space of a VAE.


Additional feedback to improve the paper (not part of decision assessment):
The additional loss term of the generator is very similar to what is used in reweighted wake sleep, although this is not mentioned. It could be worth making a connection here. 
In the conclusion the authors state “VAEs are often taken to be a pauper’s GAN’. This is not a very scientific statement and can be perceived as insulting for various reasons. Please rephrase this.

review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: This paper proposes to augment the VAE objective with two additional terms: (1) a GAN-like objective that ensures that the generated samples are not distinguishable from real samples, and (2) an additional regularizer to make sure that the latent variable that generated the image can be reconstructed from the VAE encoder. 

Overall the paper is written very well and was a pleasure to read. In particular I appreciated the differences between the present work and the many VAE/GAN hybrids in section B of the Appendix. However, after reading the paper a couple of times, it was still not clear to me what the "main point" of the paper is. To be more specific:

- From the perspective of obtaining good unconditional samples from a VAE-like model, the authors do not compare their approach against like methods like IntroVAE (https://arxiv.org/pdf/1807.06358.pdf) which also adds an adversarial objective to the VAE. Qualitatively at least, it seems apparent that the generated samples here are worse in quality that the IntroVAE work.

- From the perspective of learning more faithful reconstructions, it seems clear from Figure 5(a) that a regular beta-VAE does better. And to me it is not clear why we would want faithful reconstructions?

- The main novelty of the proposed method is in adding an extra term to the generator loss controlled by lambda (equations 1 and 4). In the appendix the authors experiment with varying the lambda parameter and find that generally having lambda > 0 produces "better" metrics, though the choice of these metrics are somewhat questionable. It would be great to see the generated samples as \lambda is varied (to me this is more interesting than seeing the generations as \beta is varied).

Further, I take several issues with the authors' point of view regarding the current state-of-affairs in VAEs:

1. "However, one of their perceived problems is their reconstruction performance." 

I somewhat disagree with this characterization. Sure, there has been much work on modifying the VAE objective such that the latent variable is not ignored (i.e. posterior collapse), but the point of these works is not to get "better reconstruction performance". 

2. "However, having a model that does not over-fit the dataset can be useful, but in this case the decoder of a standard
VAE should not be regarded as a generative model—that is not its purpose. If we wish to generate
realistic looking images we need to imagine the information discarded by the encoder"

I am not sure I understand this characterization. The decoder is by definition a generative model. Depending on the decoder/encoder capacities (e.g. PixelCNN decoder vs DeConvNet decoder), different types of information will be encoded in the latent space.

3. "The job of the decoder in a variational autoencoder is to reconstruct the image only using information
that can be compressed. Image specific information is ignored. For example, information about the
precise shape of an object is probably not compressible. As a result the decoder tends to hedge its
bets and has a blurry outline."

Again, all of this is dependent on how the encoder/decoder is parameterized. I do not agree that "information about the precise shape of an object is probably not compressible". For example see https://hal.archives-ouvertes.fr/hal-01676326/document

4. "VAEs are often taken to be a pauper’s GAN. That is, a method for generating samples that is easier
to train than a GAN, but gives slightly worse results."

My view on VAEs is that they are a way of training latent variable models with likelihood training. One potential application of this is to generate samples, but that is not the only (nor the primary) application. 


Finally, I hope I am not coming across as nitpicking or overly combative, but I am genuinely confused as to the problem that this paper is addressing. I look forward to discussing further among other reviews and the authors during the rebuttal period.

[Response to author rebuttal]

Thank you very much for your thoughtful response. However, I must say that I just fundamentally disagree with motivations and some of the statements made in the paper. In particular, there seems to be some conflation (I could be misunderstanding) of "good reconstruction" vs "good generation". For example:

- "Our point is that it does not generate a realistic image by design (in contrast to a GAN which is designed to generate a realistic image)." 

I am not sure I understand. Would you say an autoregressive model (e.g. PixelCNN) does not generate realistic images by design? (Clearly, they do!). Would you say that autoregressive language models (e.g. GPT2) do not generative realistic language by design? The argument seems to be that likelihood-based training of generative models does not, by design, encourage realistic-looking images (indeed it is true that good likelihood does not *necessarily* imply good generative models). However, there is massive empirical evidence that likelihood training does result in good generation.

After reviewing the other reviews and the author rebuttals, I am maintaining my original score.



review_title: Official Blind Review #3

Meta-review: The paper proposes a new method for improving generative properties of VAE model.  The reviewers unanimously agree that this paper is not ready to be published, particularly being concerned about the unclear objective and potentially misleading claims of the paper. Multiple reviewers pointed out about incorrect claims and statements without theoretical or empirical justification. The reviewers also mention that the paper does not provide new insights about VAE model as MDL interpretation of VAE it is not new.
Decision: Reject