Title: Calibrating Energy-based Generative Adversarial Networks
Authors: Zihang Dai
Amjad Almahairi
Philip Bachman
Eduard Hovy
Aaron Courville
Venue: ICLR 2017
Keywords: Deep learning

Abstract: In this paper, we propose to equip Generative Adversarial Networks with the ability to produce direct energy estimates for samples.
Specifically, we propose a flexible adversarial training framework, and prove this framework not only ensures the generator converges to the true data distribution, but also enables the discriminator to retain the density information at the global optimal.
We derive the analytic form of the induced solution, and analyze the properties.
In order to make the proposed framework trainable in practice, we introduce two effective approximation techniques.
Empirically, the experiment results closely match our theoretical analysis, verifying the discriminator is able to recover the energy of data distribution.


Review discussion:
The submission explores several alternatives to provide the generator function in generative adversarial training with additional gradient information. The exposition starts by describing a general formulation about how this additional gradient information (termed K(p_gen) could be added to the generative adversarial training objective function (Equation 1).
Next, the authors prove that the shape of the optimal discriminator does indeed depend on the added gradient information (Proposition 3.1), which is unsurprising.
Finally, the authors propose three particular alternatives to construct K(p_gen): the negative entropy of the generator distribution, the L2 norm of the generator distribution, and a constant function (which resembles the EBGAN objective of Zhao et al, 2016).
The exposition moves then to an experimental evaluation of the method, which sets K(p_gen) to be the approximate entropy of the generator distribution. At this point, my intuition is that the objective function under study is the vanilla GAN objective, plus a regularization term that encourages diversity (high entropy) in the generator distribution. The hope of the authors is that this regularization will transform the discriminator into an estimate of the energy landscape of the data distribution.
The experimental evaluation proceeds by 1) showing the contour plots of the obtained generator distribution for a 2D problem, 2) studying the generation diversity in MNIST digits, and 3) showing some samples for CIFAR-10 and CelebA. The 2D problem results are convincing, since one can clearly observe that the discriminator scores translate into unnormalized values of the density function. The MNIST results offer good intuition also: the more prototypical digits are assigned larger scores (unnormalized densities) by the discriminator, and the less prototypical digits are assigned smaller scores. The sample experiments from Section 5.3 are less convincing, since no samples from baseline models are provided for comparison.
To this end, I would recommend the authors to clarify three aspects.
First, we have seen that entropy regularization leads to a discriminator that estimates the energy landscape of the data distribution.
But, how does this regularization reshape the generator function? It would be nice to see the mean MNIST digit according to the generator, and some other statistics if possible.
Second, how do the samples produced by the proposed methods compare (visually speaking) to the state-of-the art?
Third, what are the *shortcomings* of this method versus vanilla GAN?
Too much computational overhead? What are the qualitative and quantitative differences between the two entropy estimators proposed in the manuscript?
Overall, a clearly written paper. I vote for acceptance.
As an open question to the authors: What breakthroughs should we pursue to derive a GAN objective where the discriminator is an estimate of the data density function, after training?
Thank you very much for the questions and suggestions. We address your comments one by one as follows.
[Comment 0]: The sample experiments from Section 5.3 are less convincing, since no samples from baseline models are provided for comparison.
[Response]:
- The primary purpose of Section 5.3 is to show that adding the calibrating term K(p_gen) does not degrade sample quality compared to existing convolutional GAN variants (see DCGAN https://arxiv.org/pdf/1511.06434v2.pdf).
- The reason we do not provide samples from baseline models is mostly a space issue. We can add baseline samples in the appendix. For now, a good visual reference is the improved GAN (https://arxiv.org/pdf/1606.03498v1.pdf) as well as https://openai.com/blog/generative-models/ for CIFAR10 samples. Note that the improved GAN uses many additional techniques to improve sample quality, including label information (which gives a big boost). For celebA, the EBGAN (https://openreview.net/pdf?id=ryh9pmcee) paper provides some samples for comparison (see Figure 6). EBGAN also uses additional techniques to boost sample quality, including margin-based cost and a pulling-away term.
- As a proxy measure, the inception scores in Table 1 provide some supporting evidence to the statement of comparable quality.
[Comment 1]: First, we have seen that entropy regularization leads to a discriminator that estimates the energy landscape of the data distribution.
But, how does this regularization reshape the generator function? It would be nice to see the mean MNIST digit according to the generator, and some other statistics if possible.
[Response]:
In theory, the optimal generator distribution p_gen(x) should match the true data distribution p_data(x) as stated by Proposition 3.1. To verify that empirically, we can definitely provide more samples from the trained generator as well as the mean generator sample later in the appendix. For now, from Figure 9, we can already see the generator samples (samples without the white frame) have great quality, and mix well with the real samples.
[Comment 2]: Second, how do the samples produced by the proposed methods compare (visually speaking) to the state-of-the-art?
[Response]
As mentioned above, we believe the sample quality of the proposed method is comparable with the state-of-the-art results, which are arguably either from GAN variants or from Pixel RNN and Pixel CNN (https://arxiv.org/pdf/1601.06759.pdf).
Samples from GAN variants usually have better global structure, while samples from Pixel RNN have better local details. For state-of-the-art samples from GAN variants, please refer to the papers mentioned above.
[Comment 3]: Third, what are the *shortcomings* of this method versus vanilla GAN?
Too much computational overhead? What are the qualitative and quantitative differences between the two entropy estimators proposed in the manuscript?
[Response]:
We think this is a good question worth some systematic future work. For now, based on our empirical observations and some properties of the training objective, we make the following comments:
- Computational overhead is not a big issue. For the KNN based approach, the heaviest computation is to compute the cross-term in the Euclidean distance, which requires a matrix-matrix multiplication of shape [BxD] x [DxB], where B is the mini-batch size and D is the feature size. With a reasonable mini-batch size (we used 200), the computation overhead is equal to that of a fully-connected layer with a weight matrix of shape [DxB], which is quite common.
As for the VI based approach, the inference network is essentially a replica of the discriminator.
Thus, it requires strictly less than double the original computation.
- A potential issue of the proposed formulation is the quality of the approximate entropy gradient.
On the one hand, when the approximate entropy gradient (1) is not accurate, and (2) has a large magnitude, it can significantly slow down the training. Basically, the inaccurate and large-magnitude entropy gradient will act like some chaotic noise, which dominates the discriminator gradient with its large magnitude, pushes the generator to wander around purposelessly, and thus makes the training very slow.
On the other hand, when the estimated entropy gradient is accurate, it can actually speed up the training by encouraging the generator to “explore” regions where p_gen(x) is currently low.
Thus, how to obtain a good entropy gradient estimate is an important topic under our formulation.
[Open question]: What breakthroughs should we pursue to derive a GAN objective where the discriminator is an estimate of the data density function, after training?
[Response]:
Generally speaking, in order to obtain a normalized density estimation, either the training procedure and the parameterization automatically guarantee a normalized result, or some proper post-normalization will be needed. Actually, our current proposed formulation may potentially fall into the second case (see below).
For the first case, the question effectively becomes whether one can derive a GAN-type formulation where the discriminator is self-normalized at the optimum. Intuitively, this self-normalization requirement sounds like posing an additional regularization on the discriminator.
However, it’s difficult to specify the form of the regularization, and it can be a set constraint on the allowed parameterization of the discriminator or some additional term in the training objective.
Either way, the high-level idea is to absorb the explicit normalization step into either the parameterization or the training process.
As for the second case (post normalization), our current proposed formulation offers some directions of exploration, though there are still many challenges as we will discuss in the following.
1. Because we have provided a case in Equation (6) where the discriminator recovers the negative density function plus some underdetermined terms, we can consider how to eliminate the ambiguity caused by the extra underdetermined terms and thus obtain an estimate of the (negative) data density. For this, note that the true difficulty is in the weak support discriminator term $mu(x)$, without which we can calculate the global bias $lambda$ using the normalization condition sum_x p(x) = 1. Actually, there are two situations where $mu(x)$ is zero and can be dropped. The first one is when the data distribution has infinite support.
However, since high-dimensional data usually resides on a low-dimensional manifold, this condition may not hold. The second situation is that we have a clear idea about the boundary of the data support and thus we can only consider the region inside the data support.
But in general, without prior knowledge, identifying the boundary of data support from empirical data is an ill-posed problem.
2. One may also consider whether we can properly renormalize the obtained energy function into the data density. To do the normalization properly, we will actually encounter the same data support problem as mentioned above.
In addition to that, another challenge is that the approximate entropy gradient may have an imprecise scale.
For example, in Equation (10), we normalize the gradient into unit norm and use the hyper-parameter $alpha$ to control the overall scale. Effectively, the imprecise scale is equivalent to adding a multiplicative weight to the calibrating term in Equation (1), i.e. the objective becomes max_{c} min_{p_gen} E_{p_gen} [c(x)] - E_{p_data} [c(x)] + weight * K(p_gen).
Hence, the discriminator calibrated by negative entropy should converge to c(x) = - weight * log p_data(x) + lambda + mu(x). Basically, the weight plays the same role as the “temperature” hyper-parameter in a Gibbs distribution.
Although the temperature does not change the relative ranking of samples in terms of energy/density, it changes the absolute value of the density.
When the temperature is smaller than 1, it will flatten the distribution.
When the temperature is larger than 1, it will make the distribution spikier.
So, the quality of the entropy gradient is again a core issue here.
Finally, since the question is quite open and has different interpretations, if we've misinterpreted the question, we'd be glad to answer further clarifying questions.
The authors present a method for changing the objective of generative adversarial networks such that the discriminator accurately recovers density information about the underlying data distribution. In the course of deriving the changed objective they prove that stability of the discriminator is not guaranteed in the standard GAN setup but can be recovered via an additional entropy regularization term.
The paper is clearly written, including the theoretical derivation. The derivation of the additional regularization term seems valid and is well explained. The experiments also empirically seem to support the claim that the proposed changed objective results in a "better" discriminator. There are only a few issues with the paper in its current form:
- The presentation albeit fairly clear in the details following the initial exposition in 3.1 and the beginning of 3.2 fails to accurately convey the difference between the energy based view of training GANs and the standard GAN.
As a result it took me several passes through the paper to understand why the results don't hold for a standard GAN. I think it would be clearer if you state the connections up-front in 3.1 (perhaps without the additional f-gan perspective) and perhaps add some additional explanation as to how c() is implemented right there or in the experiments (you may want to just add these details in the Appendix, see also comment below).
- The proposed procedure will by construction only result in an improved generator and unless I misunderstand something does not result in improved stability of GAN training. You also don't make such a claim but an uninformed reader might get this wrong impression, especially since you mention improved performance compared to Salimans et al. in the Inception score experiment. It might be worth-while mentioning this early in the paper.
- The experiments, although well designed, mainly convey qualitative results with the exception of the table in the appendix for the toy datasets. I know that evaluating GANs is in itself not an easy task but I wonder whether additional more quantitative experiments could be performed to evaluate the discriminator performance.
For example: one could evaluate how well the final discriminator does separate real from fake examples, how robust its classification is to injected noise (e.g. how classification accuracy changes for noised training data).
Further one might wonder whether the last layer features learned by a discriminator using the changed objective are better suited for use in auxiliary tasks (e.g. classifying objects into categories).
- Main complaint: It is completely unclear what the generator and discriminators look like for the experiments. You mention that code will be available soon but I feel like a short description at least of the form of the energy used should also appear in the paper somewhere (perhaps in the appendix).
Thank you very much for the questions and suggestions. We address your comments one by one as follows.
[Comment 1]: The presentation albeit fairly clear in the details following the initial exposition in 3.1 and the beginning of 3.2 fails to accurately convey the difference between the energy based view of training GANs and the standard GAN.
As a result it took me several passes through the paper to understand why the results don't hold for a standard GAN. I think it would be clearer if you state the connections up-front in 3.1 (perhaps without the additional f-gan perspective) and perhaps add some additional explanation as to how c() is implemented right there or in the experiments (you may want to just add these details in the Appendix, see also comment below).
[Response]:
Thanks for pointing out this potentially confusing point. In order to make it clearer, we have included some additional text following Equation (1) to describe the energy based view of adversarial training along with the intuitive interpretation of Equation (1).
Also, we have included architecture details in the appendix B.1. We believe, however, stating that so early in the text that adding the calibrating term to GAN will not work will diverge the attention of readers, and add some unnecessary burden.
[Comment 2]: The proposed procedure will by construction only result in an improved generator and unless I misunderstand something does not result in improved stability of GAN training. You also don't make such a claim but an uninformed reader might get this wrong impression, especially since you mention improved performance compared to Salimans et al. in the Inception score experiment. It might be worth-while mentioning this early in the paper.
[Response]:
- Firstly, we guess the first sentence in the comment actually reads “The proposed procedure will by construction only result in an improved ‘discriminator’” instead of ‘generator’. Theoretically, under the non-parametric setting, both the original GAN and our proposed formulation guarantee that the generator distribution matches the data distribution, i.e., p_gen(x) = p_data(x).
Therefore, when training reaches the optimal, the generator obtained from the proposed formulation will be indistinguishable to that of the original GAN.
- However, when it comes to the question whether the proposed formulation will improve the training stability or not, we do not have theoretical analysis nor empirical results for verifying it currently.
But the fact that our model achieves better Inception score may suggest that there exist some additional advantages of the energy based training for GAN. To faithfully verify or falsify this statement, much more work (both theoretical and empirical) will be needed. Since it is not the focus of this work, we hope to leave it for future work.
Meanwhile, we will carefully add this discussion into the text, and make sure readers have a precise understanding of the content of our work. Thanks for pointing this out.
[Comment 3]: The experiments, although well designed, mainly convey qualitative results with the exception of the table in the appendix for the toy datasets. I know that evaluating GANs is in itself not an easy task but I wonder whether additional more quantitative experiments could be performed to evaluate the discriminator performance.
For example: one could evaluate how well the final discriminator does separate real from fake examples, how robust its classification is to injected noise (e.g. how classification accuracy changes for noised training data).
Further one might wonder whether the last layer features learned by a discriminator using the changed objective are better suited for use in auxiliary tasks (e.g. classifying objects into categories).
[Response]:
- For quantitative evaluation of the discriminator, we ran experiments on MNIST with the procedure you suggested, i.e., using the last layer features learned by a discriminator as fixed input to train a linear classifier, in order to evaluate the generalization quality of the learned features. We report the result in appendix B.5. The experiment results support the fact that the discriminator from our proposed formulation maintains more information than both GAN and EGAN-Const.
- Using the final discriminator to separate real from fake examples should not be a good evaluation metric, because
(1) At the optimal “fake” samples are essentially real as p_gen(x) = p_data(x).
So, the discriminator will fail to distinguish the generator output (“fake”) from the real.
(2) On the other hand, being able to distinguish real and fake samples does not necessarily mean the discriminator is in a better shape.
Instead, it suggests the generator is not well trained.
- Generally speaking, the core difficulty of evaluating our discriminator lies in that an energy function is measuring the relative “goodness” among samples on the data manifold, rather than distinguishing whether samples are on the data manifold or not (fake vs. real). To evaluate the relative goodness, it is most natural to think about ranking. Unfortunately, we do not have the ground truth ranking for direct evaluation.
Thanks for your response. I appreciate the fact that you went ahead and performed additional experiments to address my questions. I think they are nice additions and paint an even clearer paper.
As I already wrote in my main review I believe this paper should definitely be published at the conference as is also reflected in my score.
I wanted to quickly respond to your responses to give some further comments and will leave my initial review intact to preserve the conversation:
[Response to comment 2]
Yes, I meant to write that the procedure will by construction only result in an improved "discriminator". I agree with your response here, I also of course understand that there is no silver bullet for stabilizing GANs (yet). As you wrote it is just important that we don't convey such a message in our paper (which I also don't believe your paper is doing).
[Response to comment 3]
-Regarding the additional experiment: cool to see that this works as expected, thanks for carrying out the experiment.
-Regarding the use of the final discriminator to separate real from fake exampls: fair points I think you are right that this would likely not be meaningful.
-"To evaluate the relative goodness, it is most natural to think about ranking. Unfortunately, we do not have the ground truth ranking for direct evaluation."
Yes, I  agree that would make most sense.
Another idea for evaluating the learned energy function that goes in a similar direction and might work at least for the low-dimensional cases - and perhaps with some additional regularization could also be made to work for images - would be to use the learned energy in combination with a general purpose gradient based sampler to sample from the data distribution.
That is: you could fix the discriminators after training and do Hamiltonian Monte Carlo sampling using the energy as a proxy for the likelihood for the true distribution using these samples additional evaluations could be performed. To be clear: this is not meant as an additional request for additional experiments I would like to see in this paper; just an additional thought.
Thank you for the additional comments. The HMC idea will be an interesting experiment to do, though it may not be easy to make it work in high-dimensional cases.
Also, it is related to an alternative view of energy-based GAN, which is to train an energy-based model (discriminator) where the negative phase samples come from a trainable directed sampler (generator). We believe future work should take this direction into consideration.
The paper is missing some details in the experimental setup, you refer to the potentially released code but I feel like a few more details regarding the experimental setup should also be included in the main paper or the appendix.
We have included the architecture information in the appendix. The reason why we didn’t specify it initially was that the architecture used is almost identical to that in the DCGAN paper (https://arxiv.org/pdf/1511.06434v2.pdf), except for that the discriminator c() does not have the last sigmoid layer which squashes a scalar value into a probability in [0, 1].
Similarly, following the DCGAN setting, Adam was chosen as the optimization algorithm.
This paper addresses one of the major shortcomings of generative adversarial networks - their lack of mechanism for evaluating held-out data.
While other work such as BiGANs/ALI address this by learning a separate inference network, here the authors propose to change the GAN objective function such that the optimal discriminator is also an energy function, rather than becoming uninformative at the optimal solution. Training this new objective requires gradients of the entropy of the generated data, which are difficult to approximate, and the authors propose two methods to do so, one based on nearest neighbors and one based on a variational lower bound. The results presented show that on toy data the learned discriminator/energy function closely approximates the log probability of the data, and on more complex data the discriminator give a good measure of quality for held out data.
I would say the largest shortcomings of the paper are practical issues around the scalability of the nearest neighbors approximation and accuracy of the variational approximation, which the authors acknowledge.
Also, since entropy estimation and density estimation are such closely linked problems, I wonder if any practical method for EGANs will end up being equivalent to some form of approximate density estimation, exactly the problem GANs were designed to circumvent.
Nonetheless, the elegant mathematical exposition alone makes the paper a worthwhile contribution to the literature.
Also, some quibbles about the writing - it seems that something is missing in the sentence at the top of pg. 5 "Finally, let's whose discriminative power". I'm not sure what the authors mean to say here. And the title undersells the paper - it makes it sound like they are making a small improvement to training an existing model rather than deriving an alternative training framework.
Thank you very much for the comments.
Firstly, we are really sorry about the writing problem at the top of page 5. Due to some careless editing after submission, a paragraph that was originally in the paper was erratically deleted. We have recovered that part and it should read clearly now.
We agree that a scalable entropy estimation method is the core to the proposed formulation.
Also, it is true the entropy estimation problem is closely related to density estimation, especially for the exponential families.
However, for the proposed formulation, what we really need is the estimation of entropy “gradient”, which can be practically easier. As shown by Equation (9) of the updated paper, it amounts to estimating the score function: d log p_gen(x) / d x. In this work, the nearest neighbors approximation is an example of thinking in this direction.
Another direction of thinking is that since we only need a proper gradient, is it possible to build another network to provide the gradient estimation. Actually, this extra network can provide the gradient estimation by backward propagation, or even by forward propagation [1]. For now, the variational inference is an example of using the backward propagation to provide a gradient estimation. More interestingly, as GANs were initially designed to bypass an explicit density estimation, it is conceptually tempting to think about using an adversarial process to get an implicit entropy estimation.
We believe all these ideas are worth exploring as future work.
In addition to these technical possibilities, another “advantage” here is that in theory, we can have infinite samples from the generator to estimate the entropy (gradient), which is usually impossible for normal density estimation based on empirical data.
[1] Jaderberg, Max, et al. "Decoupled neural interfaces using synthetic gradients." arXiv preprint arXiv:1608.05343 (2016).

Meta-review:
role: PC
data: comment: All reviews clearly support acceptance of the paper.

Decision: Accept (Poster)