Venue: ICLR 2017 conference

Reviews:

final_score: rating: null
review_content: Use encoder-decoder model as D is an interesting and reasonable idea. It looks like transfer knowledge in encoder decoder to G. What is the result of use pretrained encoder-decoder, then freeze D while generating gradient to G?
review_title: Preview question

final_score: rating: 7: Good paper, accept
review_content: This paper proposes a novel extension of generative adversarial networks that replaces the traditional binary classifier discriminator with one that assigns a scalar energy to each point in the generator's output domain. The discriminator minimizes a hinge loss while the generator attempts to generate samples with low energy under the discriminator. The authors show that a Nash equilibrium under these conditions yields a generator that matches the data distribution (assuming infinite capacity). Experiments are conducted with the discriminator taking the form of an autoencoder, optionally including a regularizer that penalizes generated samples having a high cosine similarity to other samples in the minibatch.

Pros:
* The paper is well-written.
* The topic will be of interest to many because it sets the stage for the exploration of a wider variety of discriminators than currently used for training GANs.
* The theorems regarding optimality of the Nash equilibrium appear to be correct.
* Thorough exploration of hyperparameters in the MNIST experiments.
* Semi-supervised results show that contrastive samples from the generator improve classification performance.

Cons:
* The relationship to other works that broaden the scope of the discriminator (e.g. [1]) or use a generative network to provide contrastive samples to an energy-based model ([2]) is not made clear in the paper.
* From visual inspection alone it is difficult to conclude whether EB-GANs produce better samples than DC-GANs on the LSUN and CelebA datasets.
* It is difficult to assess the effect of the PT regularizer beyond visual inspection as the Inception score results are computed with the vanilla EB-GAN.

Specific Comments
* Sec 2.3: It is unclear to me why a reconstruction loss will necessarily produce very different gradient directions.
* Sec 2.4: It is confusing that "pulling-away" is abbreviated as "PT".
* Sec 4.1: It seems strange that the Inception model (trained on natural images) is being used to compute KL scores for MNIST. Using an MNIST-trained CNN to compute Inception-style scores seems to be more appropriate here.
* Figure 3: There is little variation across the histograms, so this figure is not very enlightening.
* Appendix A: In the proof of theorem 2, it is unclear to me why a Nash equilibrium of the system exists.

Typos / Minor Comments
* Abstract: "probabilistic GANs" should probably be "traditional" or "classical" GANs.
* Theorem 2: "A Nash equilibrium ... exists"
* Sec 3: Should be "Several papers were presented"

Overall, I have some concerns with the related work and experimental evaluation sections, but I feel the model is novel enough and is well-justified by the optimality proofs and the quality of the generated samples.

[1] Springenberg, Jost Tobias. "Unsupervised and Semi-supervised Learning with Categorical Generative Adversarial Networks." arXiv preprint arXiv:1511.06390 (2015).
[2] Kim, Taesup, and Yoshua Bengio. "Deep Directed Generative Models with Energy-Based Probability Estimation." arXiv preprint arXiv:1606.03439 (2016).
review_title: Review

final_score: rating: null
review_content: Can you show empirically what impact the introduction of both the hinge loss and the auto-encoder parametrization of energy function have on the training and performance of the EBGAN? In other words, what happens when you remove these features?
review_title: Questions

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This paper is a parallel work to Improving Generative Adversarial Networks with Denoising Feature Matching. The main solution of both papers is introducing autoencoder into discriminator to improve the stability and quality of GAN. Different to Denoising Feature Matching, EBGAN uses encoder-decoder instead of denoising only, and use hingle loss to replace original loss function.

The theoretical results are good, and empirical result of high resolution image is unique among all recent GAN advantages.

I suggest to introduce Improving Generative Adversarial Networks with Denoising Feature Matching as related work.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: This paper introduces an energy-based Generative Adversarial Network (GAN) and provides theoretical and empirical results modeling a number of image datasets (including large-scale versions of categories of ImageNet). As far as I know energy-based GANs (EBGAN) were introduced in Kim and Bengio (2016), but the proposed version makes a number of different design choices. 

First, it does away with the entropy regularization term that Kim and Bengio (2016) introduced to ensure that the GAN discriminator converged to an energy function proportional to the log density of the data (at optimum). This implies that the discriminator in the proposed scheme will become uniform at convergence as discussed in the theoretical section of the paper, however the introductory text seems to imply otherwise -- that one could recover a meaningful score function from the trained energy-function (discriminator). This should be clarified. 

Second, this version of the EBGAN setting includes two innovations: (1) the introduction of the hinge loss in the value function, and (2) the use of an auto-encoder parametrization for the energy function. These innovations are not empirically justified in any way - this is disappointing, as it would be really good to see empirical results supporting the arguments made in support of their introduction. 

The two significant contributions of this paper are the theoretical analysis of the energy-baesd GAN formalism (showing that the optimum corresponds to a Nash equilibrium) and the impressive empirical results on large images that set a new standard in what straight GAN-style models can achieve.

The theoretical results seem solid to me and make a nice contribution.

Regarding the quantitative results in Table 2, it seems not appropriate to bold the EBGAN line when it seems to be statistically indistinguishable form the Rasmus et al (2015) results. Though it is not mentioned here, the use of bold typically indicates the state-of-the-art. 

I think this paper could be be much stronger if the two novel contributions to the energy-based GAN setting were more thoroughly explored with ablation experiments. That being said, I think this paper has already become a contribution other are building on (including at least two other ICLR submissions) and so I think it should be accepted for publication at ICLR. 

review_title: Review

final_score: rating: null
review_content: 
* All experiments are based on the autoencoder formulation for D. Did you try directly predicting the energy e.g. as the output of a convnet?
* The autoencoder formulation of D permits it to be pre-trained on x alone (i.e. without G). Do you think there would be any benefit to doing so, or is it important that G and D be trained together from the start?
review_title: Pre-review questions

Meta-review: The authors have proposed an energy-based rendition of probabilistic GANs, with the addition of auto-encoder and hinge loss to improve stability. Theoretical results involving the Nash equilibrium are also given. Solid paper, well-written. Novel contribution with good empirical and theoretical justification.
Decision: Accept (Poster)