Venue: ICLR 2018 conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: Update 1/11/18:

I'm happy with the comments from the authors. I think the explanation of non-saturating vs saturating objective is nice, and I've increased the score.

Note though: I absolutely expect a revision at camera-ready if the paper gets accepted (we did not get one).

Original review:
The paper is overall a good contribution. The motivation / insights are interesting, the theory is correct, and the experiments support their claims.

I’m not sure I agree that this is “unifying” GANs and VAEs, rather it places them within the same graphical model perspective. This is very interesting and a valuable way of looking at things, but I don’t see this as reshaping how we think of or use GANs. Maybe a little less hype, a little more connection to other perspectives would be best. In particular, I’d hope the authors would talk a little more about f-GAN, as the variational lower-bound shown in this work is definitely related, though this work uniquely connects the GAN lower bound with VAE by introducing the intractable “posterior”, q(x | y).

Detailed comments:
P1: I see f-GAN as helping link adversarial learning with traditional likelihood-based methods, notably as a dual-formulation of the same problem. It seems like there should be some mention of this.

P2:
what does this mean: “generated samples from the generative model are not leveraged for model learning”. The wording is maybe a little confusing.

P5:
So here I think the connection to f-GAN is even clearer, but it isn’t stated explicitly in the paper: the discriminator defines a lower-bound for a divergence (in this case, the JSD), so it’s natural that there is an alternate formulation in terms of the posterior (as it is called in this work). As f-GAN is fairly well-known, not making this connection here I think isolates this work in a critical way that makes it seem that similar observations haven’t been made.

P6:
"which blocks out fake samples from contributing to learning”: this is an interesting way of thinking about this. One potential issue with VAEs / other MLE-based methods (such as teacher-forcing) is that it requires the model to stay “close” to the real data, while GANs do not have such a restriction. Would you care to comment on this?

P8:
I think both the Hjelm (BGAN) and Che (MaliGAN) are using these weights to address credit assignment with discrete data, but BGAN doesn’t use a MLE generator, as is claimed in this work. 

General experimental comments:
Generally it looks like IWGAN and AA-VAE do as is claimed: IWGANs have better mode coverage (higher inception scores), while AA-VAEs have better likelihoods given that we’re using the generated samples as well as real data. This last one is a nice result, as it’s a general issue with RNNs (teacher forcing) and is why we need things like scheduled sampling to train on the free-running phase. Do you have any comments on this?

It would have been nice to show that this works on harder datasets (CelebA, LSUN, ImageNet).
review_title: Overall good perspective on GANs that connect them to other variational methods

final_score: rating: 7: Good paper, accept
review_content: The paper provides a symmetric modeling perspective ("generation" and "inference" are just different naming, the underlying techniques can be exchanged) to unify existing deep generative models, particularly VAEs and GANs. Someone had to formally do this, and the paper did a good job in describing the new view (by borrowing the notations from adversarial domain adaptation), and demonstrating its benefits (by exchanging the techniques in different research lines). The connection to weak-sleep algorithm is also interesting. Overall this is a good paper and I have little to add to it.

One of the major conclusions is GANs and VAEs minimize the KL Divergence in opposite directions, thus are exposed to different issues, overspreading or missing modes. This has been noted and alleviated in [1].

Is it possible to revise the title of the paper to specifically reflect the proposed idea? Other papers have attempted to unify GAN and VAE from different perspectives [1,2].

[1] Symmetric variational autoencoder and connections to adversarial learning. arXiv:1709.01846
[2] Adversarial variational Bayes: Unifying variational autoencoders and generative adversarial networks. arXiv:1701.04722, 2017.


Minor: In Fig. 1, consider to make “(d)” bold to be consistent with other terms.  
review_title: Good paper

final_score: rating: 6: Marginally above acceptance threshold
review_content: The authors develops a framework interpreting GAN algorithms as performing a form of variational inference on a generative model reconstructing an indicator variable of whether a sample is from the true of generative data distributions. Starting from the ‘non-saturated’ GAN loss the key result (lemma 1) shows that GANs minimizes the KL divergence between the generator(inference) distribution and a posterior distribution implicitly defined by the discriminator. I found the paper IWGAN and especially the AAVAE experiments quite interesting.  However the paper is also very dense and quite hard to follow at times - In general I think the paper would benefit from moving some content (like the wake-sleep part of the paper) to the appendix and concentrating more on the key results and a few more experiments as detailed in the comments / questions below.

Q1) What would happen if the KL-divergence minimizing loss proposed by Huszar (see e.g http://www.inference.vc/an-alternative-update-rule-for-generative-adversarial-networks/) was used instead of the “non-saturated” GAN loss - would the residial JSD terms in Lemma 1 cancel out then?

Q2) In Lemma 1 the negative JSD term looks a bit nasty to me e.g. in addition to KL divergence the GAN loss also maximises the JSD between the data and generative distributions. This JSD term acts in a somewhat opposite direction of the KL-divergence that we are interested in minimizing. Can the authors provide some more detailed comments / analysis on these two somewhat opposed terms - I find this quite important to include given the opposed direction of the JSD versus the KL term and that the JSD is ignored in e.g. section 4.1? secondly did the authors do any experiments on the the relative sizes of these two terms? I imagine it would be possible to perform some low-dimensional toy experiments where both terms were tractable to compute numerically?

Q3) I think the paper could benefit from some intuition / discussion of the posterior term q^r(x|y) in lemma 1 composed on the prior p_theta0(x) and discriminator q^r(y|x). The terms drops out nicely in math however i had a bit of a hard time wrapping my head around what minimizing the KL-divergence between this term and the inference distribution p(xIy). I know this is a kind of open ended question but i think it would greatly aid the reader in understanding the paper if more ‘guidance’ is provided instead of just writing “..by definition this is the posterior.’

Q4) In a similar vein to the above. It would be nice with some more discussion / definitions of the terms in Lemma 2. e.g what does “Here most of the components have exact correspondences (and the same definitions) in GANs and InfoGAN (see Table 1)” mean? 

Q5) The authors state that there is ‘strong connections’ between VAEs and GANs. I agree that both (after some assumptions) both minimize a KL-divergence (table 1) however to me it is not obvious how strong this relation is. Could the authors provide some discussion / thoughts on this topic?

Overall i like this work but also feel that some aspects could be improved: My main concern is that a lot of the analysis hinges on the JSD term being insignificant, but the authors to my knowledge does but provide any prof / indications that this is actually true. Secondly I think the paper would greatly benefit from concentration on fewer topics (e.g. maybe drop the RW topic as it feels a bit like an appendix) and instead provide a more throughout discussion of the theory (lemma 1 + lemma 2) as well as some more experiments wrt JSD term.

review_title: Review of On Unifying Deep Generative Models

Meta-review: This is a thought-provoking paper that places GANs and VAEs in a single framework and, motivated by this perspective, proposes several novel extensions to them. The reviewers made several good suggestions for improving the paper and the authors are expected to make the revisions they promised. The current title of the paper is too general and should be changed to something more directly descriptive of the contents.
Decision: Accept (Poster)