Venue: ICLR 2017 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper introduces a class-conditional GAN as a generative model for images. It introduces two main diagnostic tools for training GANs: one to assess whether a model is making full use of its output resolution and another to measure the diversity of generated samples. Experiments are conducted on the CIFAR-10 and ImageNet datasets.

Pros:
+ The paper is clear and well-written.
+ Experiments performed in the relatively under-explored 128 x 128 ImageNet setting.
+ The proposed MS-SSIM diversity metric appears to be a useful tool for detecting convergence issues in class-conditional GAN models.

Cons:
- AC-GAN model itself is of limited novelty relative to other GAN approaches that condition on class.
- Diversity metric is of limited use for training non class-conditional GANs.
- No experimental comparison of AC-GAN to other class-conditional models.

To my knowledge training GANs on large, diverse images such as 128 x 128 ImageNet images is under-explored ([1] contains just a few samples in this setting). Though the model is not very novel and a comparison to other class-conditional models is lacking, I feel the community will find the diagnostic tools and the thorough exploration of the ImageNet-trained model to be of interest.

* Section 4.2: MS-SSIM is traditionally defined for grayscale images only. How do you extend MS-SSIM to color images in your work? Were they computed channel-wise across R,G, and B?
* Section 4.4: It is difficult to tell whether a single AC-GAN was trained for all of CIFAR-10 or one for each group. If single, why were the samples split into groups for computing Inception Score? And if multiple, the comparison to Salimans et al. is not a direct one. Also it would be helpful to include the real data Inception score as a point of comparison.
* Appendix D: The caption of Figure 9 states that the same number of training steps was taken for each model. From this it seems possible that the models with more classes simply did not converge yet.

[1] Salimans, Tim, et al. "Improved techniques for training GANs." Advances in Neural Information Processing Systems. 2016.
review_title: Review

final_score: rating: null
review_content: 
1. Did you try a comparison of AC-GAN to other class-conditional models? Specifically, plotting Inception accuracy vs. MS-SSIM for AC-GAN compared to e.g. conditional GANs would be insightful. A qualitative comparison of AC-GAN samples vs. other models would be helpful as well.
2. Do you think that the MS-SSIM diversity score is useful for unlabeled datasets, or do you feel its primary use is for class-labeled datasets?
3. In Figure 7, did you try performing the nearest neighbor search using MS-SSIM instead of L1?
review_title: Pre-review questions

final_score: rating: 3: Clear rejection
review_content: Apologies for the late review.

This submission proposes method for class-conditional generative image modeling using auxiliary classifiers. Compared to normal GANs the generator also receives a randomly sampled class label c from the class distribution. The discriminator has two outputs and two corresponding objectives: determine whether a sample is real or generated, and independently to predict the (real or sampled) class label corresponding to the sample.

Figure 2. nicely illustrates related methods - this particular method bears similarities to InfoGANs and Semi-supervised GANs. Compared to infogans, this method also encourages correspondence between the latent c and the real class labels for the real examples (whereas infogans are presented as fully unsupervised).

The authors attempt at evaluating the method quantitatively by looking at the discriminability and diversity of samples. It is found - not surprisingly - that higher resolution improves discriminability (because more information is present).

Discriminability: Figure 3 doesn’t have legends so it is a bit hard to understand what is going on. Furthermore, my understanding is that when evaluating discriminability the authors downsample and then bicubically upsample the image, which is much more like a blurring, very different from retraining all the models to work on low resolution in the first place.

Diversity: The authors try to quantitatively evaluate diversity of samples by measuring the average MS-SSIM between randomly selected pairs of points within each class. I think this method is significantly flawed and limited, for reasons mentioned in (Theis et al, 2015, A note on the evaluation…). In its behaviour, MS-SSIM is not that dissimilar from Euclidean distance - although it is nonlinear and is bounded between -1 and 1. Evaluating diversity/entropy of samples in high dimensions is very hard, especially if the distributions involved are non-trivial for example concentrated around manifolds. Consider for example a generative model which randomly samples just two images. Assuming that the MSSSIM between these two images is -1, this generative model can easily achieve an average MSSSIM score of 0, implying a conclusion that this model has more diversity than the training data itself. Conversely, SSIM is designed not to be sensitive to contrast and average pixel intensity, so if a model is diverse in this sense, that will be ignored by this measure.

Overall, the paper proposes a new way to incorporate class labels into training GAN-type models. As far as I know the particular algorithm is novel, but I consider it incremental compared to what has been done before. I think the proposed evaluation metrics are flawed, especially when evaluating the diversity of the samples for the aforementioned reasons.
review_title: Review

final_score: rating: 6: Marginally above acceptance threshold
review_content: This is a clear, easy to read, highly relevant paper that improves GAN training for images and explores evaluation criterion on GANs. The main contributions are as follows:
- Adding an auxiliary classifier head to a GAN discriminator and training a classification objective in addition to the real/fake objective improves performance. Generator is conditioned on 1-hot encoding of class and is trained to generate the specified class.
- Training different models on different subsets of imagenet classes improves performance.
- They motivate evaluating GAN images by using a perceptual similarity metric (MS-SSIM) on pairs of samples to quantify diversity in the samples (and detect mode collapse)
- They show this metric correlates with a discriminability metric (classification accuracy of pre-trained imagenet model on generated samples) .

The overall novelty of this approach is somewhat lacking in that previous methods have proposed training a classifier head on the discriminator and the discriminability metric proposed is simply the inception score of [1] except with class information. However, I think there is still a contribution to be made my putting these tricks together and successfully demonstrating image synthesis gains. 

Questions for the authors: 
(1) Why do you think splitting the imagenet training into 100 different models improves performance?  Is the issue with the representation of the class? In other words, if an encoding more meaningful that 1-hot vector was used do you still think 100 models would be needed. Ideally we should hope that a generative model can leverage information from different classes to help with the generation of a particular class and also text-image synthesis models [2] have been quite successful when trained on diverse datasets (and these are conditioned on a semantically meaningful text encoding) which suggests to be that the issue is with the representation. 
(2) In section 3 the AC-GAN classification objective (omitting expectation for brevity) is given as L_S = log P(C=c|X_real) + log P(C=c|X_fake) and you say that both the discriminator and generator are trained to maximize this quantity. Obviously the generator would want to maximize  log P(C=c|X_fake) for its given conditioning class c. But can you explain why you would want the discriminator to also maximize the classification accuracy of generated samples? Why not do something similar to the CatGAN paper [3] and train the discriminator to be as uncertain as possible about the generated examples. Seems counterintuitive to me to have both the generator and discriminator trying to optimize the same classification objective rather than not be adversarial wrt to this loss as well as the real/fake loss. 

Overall, this paper makes a clear contribution to GAN research both in terms of image quality and evaluation metrics and I would recommend it for acceptance. 

[1] Salimans et al. Improved Techniques for Training GANs (https://arxiv.org/abs/1606.03498)
[2] Reed et al. Generative Adversarial Text to Image Synthesis (https://arxiv.org/abs/1605.05396)
[3] Jost Tobias Springenberg. Unsupervised and semi-supervised learning with categorical generative adversarial networks (https://arxiv.org/abs/1511.06390)

review_title: review

Meta-review: Ratings summary:
 3: Clear rejection
 6: Marginally above acceptance threshold
 6: Marginally above acceptance threshold
 
 Clear easy to read paper focusing on generating higher quality higher resolution (128x128) pixel imagery with GANs. There were broad concerns however across reviewers that the work is lacking in clearly identifiable novelty. The authorÕs point to a list of novel elements of the work in their rebuttal. However, the most negative reviewer also has issues with the evaluation metrics used.
 
 Thus, unfortunately, the PCs believe that this work isn't ready to appear at the conference.
Decision: Reject