Venue: ICLR 2017 conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper presents a bridging of energy-based models and GANs, where -- starting from the energy-based formalism -- they derive an additional entropy term that is not present in the original GAN formulation and is also absent in the EBGAN model of Zhao et al (2016, cited work). The relation between GANs and energy-based models was, as far as I know, first described in Kim and Bengio (2016, cited work) who also introduced the entropy regularization. It is also discussed in another ICLR submission (Dai et al. "Calibrating Energy-based Generative Adversarial Networks"). There are two novel contribution of this paper: (1) VGAN: the introduction of a novel entropy approximation; (2) VCD: variational contrastive divergence is introduced as a  novel learning algorithm (however, in fact, a different model). The specific motivation for this second contribution is not particularly clear. 

The two contributions offered by the authors are quite interesting and are well motivated in the sense that the address the important problem of avoiding dealing with the generally intractable entropy term. However, unfortunately the authors present no results directly supporting either contribution. For example, it is not at all clear what role, if any, the entropy approximation plays in the samples generated from the VGAN model. Especially in the light of the impressive samples from the EBGAN model that has no corresponding term. The results provided to support the VCD algorithm, come in the form of a comparison of samples and reconstructions. But the samples provided here strike me as slightly less impressive compared to either the VGAN results or the SOTA in the literature - this is, of course, difficult to evaluate. 

The results the authors do present include qualitative results in the form of reasonably compelling samples from the model trained on CIFAR-10, MNIST and SVHN datasets. They also present quantitative results int he form of semi-supervised learning tasks on the MNIST and SVHN. However these
quantitative results are not particularly compelling as they show limited improvement over baselines. Also, there is no reference to the many
existing semi-supervised results on these datasets. 

Summary: The authors identify an important problem and offer two novel and intriguing solutions. Unfortunately the results to not sufficiently support either
contribution.

review_title: Review

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper presents an adversarial training formulation for energy models. Although the relationship between energy-based models and GANs is abundantly clear in the literature, the contributions of this paper seems to be a multimodal energy estimate, and training a transition operator instead of a sampler. The core technical contribution of the paper is not clear. The only two quantitative results in the paper demonstrate that the proposed method learns better features than a traditional GAN, and that the proposed method does a better job in SSL than the authors' chosen baseline. In either case, no comparison to existing literature is made, even though GANs have been used for SSL previously (https://arxiv.org/abs/1606.03498). The qualitative results (visual comparison of quality of samples) is inconclusive.

The lack of comparison to existing literature makes this paper a clear reject. To improve this paper, the authors need to make the core contribution of the paper much better, and situate the paper very clearly wrt existing literature. The motivation for the development of the model should be clearer. Further, experiments need to compare with existing literature.
review_title: Review

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: Our understanding of GAN to date is still vague. Although there have been some efforts relating GAN to energy models, I personally consider that the perspective of this paper, namelying understanding GAN (a variant of GAN, to be more precise) as variational training of an energy model is the most natural and elegant.  The derivation up to equation (5) and the reduction to (7) are very nice.  I think this is the most important contribution of the paper. 

The techniques introduced in sections 5 and 6 are somewhat ad hoc, and lack clarity. Referring to the version I looked at (not sure though if it is the latest), section 6 contain some errors/typos (stuff around p_z(x|\tilde{x}). The presentation of section 6 needs to improve in clarity.  But I think this does not shadow the main contribution of the paper, namely, that perspectives given in sections 2-4.  

Overall I very much enjoy the presented insight of this paper into GAN.

I do have some comment/question regarding equation (7). This equation formulates a variant of GAN, or a model resembling GAN. I am happy to see that the entropy term pops up there, which should save GAN from degenerating its generative distribution or from missing modes.  The swapping of the min-max order in this formulation however makes me wonder if this variant of GAN indeed reflects the "principle" of GAN, or it is in fact a different principle, which happens to gives rise to a model  that "resembles" GAN.  Of course, my question may be merely philosophical rather than mathematical, and I won't expect a precise anwer. Nonetheless, if the author can provide additional insignts on this, it would be appreciated.



review_title: Nice insights, clarity needs improvement

final_score: rating: null
review_content: 1. You use a transition operator to serve as a generator, and it appears that this choice contributes significantly to the results. To understand the role of the transition operator better, have you tried either training your model without transition operator, or training a regular GAN with transition operator?

2. What is H tilde mentioned in algorithm 2? Is it a parametrized model? How do you optimize it?
review_title: Questions

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper drives a variant of GAN's min max game optimizations from EBM's NLL minimization and propose a new generative model from this derivation.

In introduction, could you elaborate why accurate distance metric is needed for unsupervised learning.

It's a bit hard to read the paper as it jumps from points to points without clear connection and laying down the background.  The introduction, refers to many different concepts without any clear connection. And other sections as well is very incomprehensible even if one is familiar with the concepts.

There has been lots of interests in similar area recently, Authors do cite some of them in introduction and related work but the relationship and comparison are missing.

In the results, if it's comparison of the quality of samples it has to include other recent works and compare those as well. And if it's semi-supervised learning, again, the numbers should be compared to other works.

In summary, unfortunately the paper is very clear and cumbersome to read which makes it hard to judge it fairly. I strongly suggest re-write of the paper in more coherent matter to make it easier to read. And also extend the experiments with more comparisons with other works.
review_title: Final Review

final_score: rating: null
review_content: Do you have learning the curves from multiple runs? That could help comparing the stability of the models.
review_title: Learning curve

final_score: rating: null
review_content: Regarding the entropy approximation in equation 9, do you have any reason to think this is a reasonable approximation? You state that there is no theoretical guarantee "recovers the true entropy", but do you have any empirical evidence that this is a reasonable approximation in principle, or that the gradient is a reasonable approximation of the entropy gradient?

You state that maximizing the entropy approximation in equation 9 "serves the same purpose of encouraging the generated samples to be diverse", but it seems like a degenerate solution where all the W_j weights are zero would also maximize equation 9, but not at all encourage diversity. First, do you agree? Second, do you have evidence that this does not happen?

review_title: Questions

Meta-review: This paper is timely since it addresses the connections between energy-based models, GANS, and the general space of generative models. The two principal concerns about the paper are: lack of clarity and coherence in the paper; inability to effectively judge the effectiveness of the method. The responses of the authors address these concern to some extent, but these concerns remain. The paper will have much higher-impact after further introspection, but at present, the paper is not yet ready for acceptance at the conference.
Decision: Reject