Title: Activation Maximization Generative Adversarial Nets
Authors: Zhiming Zhou
Han Cai
Shu Rong
Yuxuan Song
Kan Ren
Weinan Zhang
Jun Wang
Yong Yu
Venue: ICLR 2018
Keywords: Generative Adversarial Nets
GANs
Evaluation Metrics
Generative Model
Deep Learning
Adversarial Learning
Inception Score
AM Score

Abstract: Class labels have been empirically shown useful in improving the sample quality of generative adversarial nets (GANs). In this paper, we mathematically study the properties of the current variants of GANs that make use of class label information. With class aware gradient and cross-entropy decomposition, we reveal how class labels and associated losses influence GAN's training. Based on that, we propose Activation Maximization Generative Adversarial Networks (AM-GAN) as an advanced solution. Comprehensive experiments have been conducted to validate our analysis and evaluate the effectiveness of our solution, where AM-GAN outperforms other strong baselines and achieves state-of-the-art Inception Score (8.91) on CIFAR-10. In addition, we demonstrate that, with the Inception ImageNet classifier, Inception Score mainly tracks the diversity of the generator, and there is, however, no reliable evidence that it can reflect the true sample quality. We thus propose a new metric, called AM Score, to provide more accurate estimation on the sample quality. Our proposed model also outperforms the baseline methods in the new metric.

TL;DR: Understand how class labels help GAN training. Propose a new evaluation metric for generative models. 

Review discussion:
I thank the authors for the thoughtful responses and updated manuscript.
Although the manuscript is improved, I still feel it is unfocused and may be substantially improved, thus my review score remains unchanged.
The authors describe a new version of a generative adversarial network (GAN) for generating images that is heavily related to class-conditional GAN's. The authors highlight several additional results on evaluation metrics and demonstrate some favorable analyses using their new proposed GAN.
Major comments:
1) Unfocused presentation. The paper presents a superfluous and extended background section that needs to be cut down substantially. The authors should aim for a concise presentation of their work in 8 pages.
Additionally, the authors present several results (e.g. Section 5.1 on dynamic labeling, Section 6.1 on Inception score) that do not appear to improve the results of the paper, but merely provide commentary. The authors should either defend why these sections are useful or central to the arguments in the paper;
otherwise, remove them.
2) Quantitative evaluation highlight small gains. The gains in Table 1 seem to be quite small and additionally there are no error bars so it is hard to assess what is statistically meaningful. Table 2 highlights some error bars but again the gains some quite small. Given that the AM-GAN seems like a small change from an AC-GAN model, I am not convinced there is much gained using this model.
3) MS-SSIM. The authors' discussion of MS-SSIM is fairly confusing. MS-SSIM is a measure of image similarity between a pair of images.
However, the authors quote an MS-SSIM for various GAN models in Table 3. What does this number mean?  I suspect the authors are calculating some cumululative statistics across many images, but I was not able to find a description, nor understand what these statistics mean.
4) 'Inception score as a diversity measurement.' This argument is not clear to me. Inception scores can be quite high for an individual image indicating that the image 'looks' like a given class in a discriminative model.  If a generative model always generates a single, good image of a 'dog', then the classification score would be quite high but the generative model would be very poor because the images are not diverse.
Hence, I do not see how the inception score captures this property.
If the authors can address all of these points in a substantive manner, I would consider raising my rating.
We sincerely thank you for your constructive advice on our paper. We have substantially revised our paper according to your comments.
1. Unfocused Presentation:
a) Superfluous Preliminary. We shorten the preliminary section and only keep necessary equations that are referred in later sections.
b) Section 6.1 on Inception Score. We have discarded the inessential part of the discussions in Section 6.1 and only kept the definition of Inception Score.
c) Section 5.1 on Dynamic Labeling. Dynamic labeling brings important improvements to AM-GAN, and is applicable to other models that require target class for each generated sample, such as AC-GAN. It is an alternative to predefined labeling, and affects the models largely according to our experiments.
As such, we consider it necessary to keep the discussions about dynamic labeling. We have made the point clearer in the revised version.
2. Quantitative Evaluation:
a) No Error Bar in Table 1. We have added error bars in Table 1, where AM-GAN still consistently outperforms variants of AC-GAN and LabelGAN by a large margin in terms of both Inception Score and AM Score.
b) Table 2 Shows Small Gains. As shown in Table 2, AM-GAN achieves 8.91±0.11 Inception Score on CIFAR-10, which significantly outperforms all the baseline methods, including Improved GAN (8.09±0.07), AC-GAN (8.25±0.07), WGAN-GP + AC (8.42±0.10) and SGAN (8.59±0.11).
When compared to Splitting GAN (8.87±0.09), an orthogonal work to AM-GAN, which enhances the class label information via class splitting, the improvement seems to be less significant.
However, since it is orthogonal to AM-GAN, we can combine them to further improve the results. As it takes some time to conduct additional experiments, we would add such results in the later version.
3. MS-SSIM:
Sorry for missing the descriptions on MS-SSIM. We actually borrow the usage of MS-SSIM from AC-GAN (Odena et al., 2016) which measures the MS-SSIM scores between a set of randomly sampled pairs of images within a given class and uses the mean of the MS-SSIM scores, where a high mean MS-SSIM indicates intra-class mode collapse or low sample diversity in the class. In our paper, we report the maximum of the mean MS-SSIM over the 10 classes in CIFAR-10, with which we judge whether there exists obvious intra-class mode collapse. We have added corresponding descriptions and citations to MS-SSIM in the new version (the first paragraph in Section 7 and the caption of Table 2).
4. Inception Score as a Diversity Measurement:
A direct answer is that Inception Score, i.e. exp(H(E_x[C(x)])-E_x[H(C(x))]), has two terms and, more importantly, when the generator collapsed to a single point, the first term H(E_x[C(x)]) (the entropy of “the mean classification distribution of all samples”) and the second term E_x[H(C(x))] (the mean classification entropy score for each sample) in Inception Score would be actually equal.
Though the second term can be high, the overall Inception Score, in this case, would always be the minimal value 1.0 (exp^0).
It is worth noting that the first and second terms are highly correlated. As in Section 6.3, we provide an alternative explanation by understanding Inception Score in the KL divergence formulation, i.e. exp(E_x[KL(C(x),E_x[C(x)])]), which involves a single term and can be interpreted as it requires each sample’s distribution to be highly different from the overall distribution of the generator. In this view, it measures the sample diversity. We further demonstrate that Inception Score can capture sample diversity well with synthetic experiments:  assuming the generator perfectly generates a subset of the training data, with the subset growing to cover the entire dataset, Inception Score is monotonically increasing. Please also refer to Sections 6.2 and 6.3 in our paper for more details.
The comments have been very useful for us to improve our paper, and we have updated our paper according to your valuable review comments. Please check it.
We have conducted additional experiments on combining AM-GAN with the class-splitting technique proposed in [1] which is orthogonal to our work and has shown to improve quality of generated samples. Unfortunately, we found that it fails to further improve Inception Score in our setting. This might be due to the fact that the quality of split-classes is not good enough, which largely depends on the features it learns and the clustering algorithm it uses. It requires further investigations to make split-classes more effective and we would leave it as future work.
[1] Guillermo, L. Grinblat, Lucas, C. Uzal, and Pablo, M. Granitto. Class-splitting generative adversarial
networks. arXiv preprint arXiv:1709.07359, 2017.
+ Pros:
- The paper properly compares and discusses the connection between AM-GAN and class conditional GANs in the literature (AC-GAN, LabelGAN)
- The experiments are thorough
- Relation to activation maximization in neural visualization is also properly mentioned
- The authors publish code and honestly share that they could not reproduce AC-GAN's results and thus using to its best variant AC-GAN* that they come up with. I find this an important practice worth encouraging!
- The analysis of Inception score is sound.
+ Cons:
- A few presentation/clarity issues as below
- This paper leaves me wonder why AM-GAN rather than simply characterizing D as a 2K-way classifier (1K real vs 1K fake).
+ Clarity:
The paper is generally well-written.
However, there are a few places that can be improved:
- In 2.2, the authors mentioned "In fact, the above formulation is a modified version of the original AC-GAN..", which puts readers confusion whether they were previously just discussed AC-GAN or AC-GAN* (because the previous paragraph says "AC-GAN are defined as..".
- Fig. 2: it's not clear what the authors trying to say if looking at only figures and caption. I'd suggest describe more in the caption and follow the concept figure in Odena et al. 2016.
- A few typos here and there e.g. "[a]n diversity measurement"
+ Originality: AM-GAN is an incremental work by applying AM to GAN.
However, I have no problems with this.
+ Significance:
- Authors show that in quantitative measures, AM-GAN is better than existing GANs on CIFAR-10 / TinyImageNet.
Although I don't find much a real difference by visually comparing of samples of AM-GAN to AC-GAN*.
Overall, this is a good paper with thorough experiments supporting their findings regarding AM-GAN and Inception score!
We sincerely thank you for your constructive feedback. We have revised the paper and fixed the confusing statements. More descriptions about the tables and figures have been added in their captions.
1. Why not characterize the discriminator as a 2K-way classifier (K real vs K fake)?
This is an interesting idea and we have thought about this originally.
However, we did not feel strongly that considering K fake logits would help in our case:
a) Introducing specific real class logits in the discriminator makes it possible to assign a specific target class for each generated sample, which provides a clearer guidance to the generator.
However, a fake class will not be used as the target for the generated sample. In this sense, how and whether we can benefit from using K specific fake class logits are still unknown.
b) Introducing more fake classes does influence the gradient that the generator receives from the discriminator.
When optimizing a generated sample, only the target class logit is encouraged while all the others are otherwise discouraged.
Thus, replacing a single fake class with K fake classes changes the discouraged recipient from the overall fake class to the K specific fake classes. It requires further investigations to figure out whether this will help or not. We leave it as our future work.
This paper is a thorough investigation of various “class aware” GAN architectures. It purposes a variety of modifications on existing approaches and additionally provides extensive analysis of the commonly used Inception Score evaluation metric.
The paper starts by introducing and analyzing two previous class aware GANs - a variant of the Improved GAN architecture used for semi-supervised results (named Label GAN in this work) and AC-GAN, which augments the standard discriminator with an auxiliary classifier to classify both real and generated samples as specific classes.
The paper then discusses the differences between these two approaches and analyzes the loss functions and their corresponding gradients. Label GAN’s loss encourages the generator to assign all probability mass cumulatively across the k-different label classes while the discriminator tries to assign all probability mass to the k+1th output corresponding to a “generated” class. The paper views the generators loss as a form of implicit class target loss.
This analysis motivates the paper’s proposed extension, called Activation Maximization. It corresponds to a variant of Label GAN where the generator is encouraged to maximize the probability of a specific class for every sample instead of just the cumulative probability assigned to label classes. The proposed approach performs strongly according to inception score on CIFAR-10 and includes additional experiments on Tiny Imagenet to further increase confidence in the results.
A discussion throughout the paper involves dealing with the issue of mode collapse - a problem plaguing standard GAN variants.
In particular the paper discusses how variants of class conditioning effect this problem. The paper presents a useful experimental finding - dynamic labeling, where targets are assigned based on whatever the discriminator thinks is the most likely label, helps prevent mode collapse compared to the predefined assignment approach used in AC-GAN / standard class conditioning.
I am unclear how exactly predefined vs dynamic labeling is applied in the case of the Label GAN results in Table 1. The definition of dynamic labeling is specific to the generator as I interpreted it.
But Label GAN includes no class specific loss for the generator. I assume it refers to the form of generator - whether it is class conditional or not - even though it would have no explicit loss for the class conditional version. It would be nice if the authors could clarify the details of this setup.
The paper additionally performs a thorough investigation of the inception score and proposes a new metric the AM score. Through analysis of the behavior of the inception score has been lacking so this is an important contribution as well.
As a reader, I found this paper to be thorough, honest, and thoughtful. It is a strong contribution to the “class aware” GAN literature.
We sincerely thank you for your comprehensive comments on our paper.
1. What does dynamic/predefined labeling in Table 1 means? How does it apply to LabelGAN?
Models with dynamic labeling and predefined labeling settings require different network structures (G and D’s capacities). The “dynamic” and “predefined” in Tables 1 and 3 represent two experimental settings which differ in network structures. Under the “dynamic” setting, if a model requires specific target class (AC-GAN, AM-GAN), we apply dynamic labeling; under the “predefined” setting, if a model requires specific target class, we apply predefined labeling; for models that do not need target class (GAN, LabelGAN), neither of them are applied and the two (“dynamic” and “predefined”) only differ in network structures. In this way, we compare various models with almost identical network structures. We have revised the caption of Table 1. It should be clear now.

Meta-review:
role: PC
data: comment: The authors investigate various class aware GANs and provide extensive analysis of their ability to address mode collapse and sample quality issues. Based on this analysis they propose an extension called Activation Maximization-GAN which tries to push each generated sample to a specific class indicated by the Discriminator. As experiments show, this leads to better sample quality & helps with mode collapse issue. The authors also analyze inception score to measure sample quality and propose a new metric better suited for this task.

Decision: Accept (Poster)