Venue: ICLR 2018 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes an interesting idea--to learn a flexible prior from data by maximizing data likelihood.

It seems that in the prior improvement stage, what you do is training a GAN with CG+dec as the generator while D_I as the discriminator (since you also update dec at the prior improvement stage). So it can also be regarded as GAN trained with an additional enc and D_c, and additional objective. In my opinion, this may explain why your model can generate sharper images.

The experiments do demonstrate the power of their model compared to AAE. However, only the qualitative analysis may not persuade me and more thorough analysis is needed.

1. About the latent space for z. The motivation in AAE is to impose aggregated posterior regularization $D(q(z),p(z))$ where $p(z)$ is chosen as a simple one, e.g., Gaussian. I'm curious how the geometry of the latent space will be, when the code generator is introduced. Maybe some visualization like t-sne will be helpful.
2. Any quantitative analysis? Doing a likelihood analysis like that in the AAE paper will be very informative. 

review_title: Interesting idea, but more thorough analysis is needed.

final_score: rating: 6: Marginally above acceptance threshold
review_content: Recently some interesting work on a role of prior in deep generative models has been presented. The choice of prior may have an impact on the expressiveness of the model [Hoffman and Johnson, 2016]. A few existing work presents methods for learning priors from data for variational autoencoders [Goyal et al., 2017][Tomczak and Welling, 2017].  The work, "VAE with a VampPrior," [Tomczak and Welling, 2017] is missing in references.

The current work focuses on adversarial autoencoder (AAE) and introduces a code generator network to transform a simple prior into one that together with the generator can better fit the data distribution. Adversarial loss is used to train the code generator network, allowing the output of the network could be any distribution. I think the method is quite simple but interesting approach to improve AAEs without hurting the reconstruction. The paper is well written and is easy to read. The method is well described. However, what is missing in this paper is an analysis of learned priors, which help us to better understand its behavior. 

The model is evaluated qualitatively only. What about quantitative evaluation? 


review_title: A simple idea to improve adversarial autoencoders by learning priors

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper propose a simple extension of the adversarial auto-encoders for (conditional) image generation. The general idea is that instead of using Gaussian prior, the propose algorithm uses a "code generator" network  to warp the gaussian distribution, such that the internal prior of the latent encoding space is more expressive and complicated. 

Pros:
- The proposed idea is simple and easy to implement
- The results show improvement in terms of visual quality

Cons:
- I agree that the proposed prior should better capture the data distribution. However, incorporating a generic prior over the latent space plays a vital role as regularisation, this helps avoid model collapse. Adding a complicated code generation network brings too much flexibility for the prior part. This makes the prior and posterior learnable, which makes it easier to fool the regularisation discriminator (think about the latent code and prior code collapsed to two different points). As a result, this weakens the regularisation over the latent encoder space.  
- The above mentioned could be verified through qualitative results. As shown in Fig. 5. I believe this is a result due to the fact that the adversarial loss in the regularisation phase does not a significant influence there. 
- I have some doubts over why AAE works so poorly when the latent dimension is 2000. How to make sure it's not a problem of implementation or the model wasn't trapped into a bad local optima / saddle points. Could you justify this?
- Contributions; this paper propose an improvement over a existing model. However, neither the idea/insights it brought can be applied onto other generative models, nor the improvement bring a significant improvement over the-state-of-the-arts. I am wondering what the community will learn from this paper, or what the author would like to claim as significant contributions. 
review_title: Improving AAE by warping the Gaussian prior using deep networks

Meta-review: The paper proposes learning the prior for AAEs by training a code-generator that is seeded by the standard Gaussian distribution and whose output is taken as the prior. The code generator is trained by minimizing the GAN loss b/w the distribution coming out of the decoder and the real image distribution. The paper also modifies the AAE by replacing the L2 loss in pixel domain with "learned similarity metric" loss inspired by the earlier work (Larsen et al., 2015).

The contribution of the paper is specific to AAE which makes the scope narrow. Even there, the benefits of learning the prior using the proposed method are not clear. Experiments make two claims: (i) improved image generation over AAE, (ii) improved "disentanglement".

Towards (i), the paper compares images generated by AAE with those generated by their model. However, it is not clear if the improved generation quality is due to the use of decoder loss on the learned similarity metric (Larsen at al, 2015), or due to the use of GAN loss in the image space (ie, just having GAN loss over decoder's output w/o having a code generator), or due to learning the prior which is the main contribution of the paper. This has also been hinted at by AnonReviewer1. Hence, it's not clear if the sharper generated images are really due to the learned prior.

Towards (ii), the paper uses InfoGAN inspired objective to generate class conditional images. It shows the class-conditional generated images for AAE and the proposed method. Here AAE is also trained on "learned similarity metric" and augmented with similar InfoGAN type objective so the only difference is in the prior. Authors say the performance of both models is similar on MNIST and SVHN but on CIFAR their model with "learned prior" generates images that match the conditioned-upon labels better. However this claim is also subjective/qualitative and even if true, it is not clear if this is due to learned prior or due to the extra GAN discriminator loss in the image-space -- in other words, how do the results look for AAE + a discriminator in the image space, just like in the proposed model but without a code generator?

The t-SNE plots for the learned prior are also shown but they are only shown when InfoGAN loss is added. The same plots are not shown for AAE with added InfoGAN loss so it is difficult to know the benefits of learning the code-generator as proposed.

Overall, I feel the scope of the paper is narrow and the benefits of learning the prior using the method proposed in the paper are not clearly established by the reported experiments. I am hesitant to recommend acceptance to the main conference in its current form.


Decision: Reject