Venue: ICLR 2017 conference

Reviews:

final_score: rating: null
review_content: Hello,

Nice work! A couple of questions:

1. Have you tried your method with AlexNet discriminator for fair comparison in Table 2 ? Your current reasoning about supervised training is somewhat indirect. Just having numbers with AlexNet would be much more convincing.

2. Have you evaluated the performance of the encoder features vs the discriminator features in your models? Is the encoder significantly worse? You argue that "discrimination of real/fake in-paintings is more closely related to the target task of object classification than extracting a feature representation suitable for in-filling", and I tend to disagree: I actually like the reasoning behind Context Encoders. It would be interesting to get a better understanding of what is going on.

3. "We pass the completed image into D rather than the context and the patch as two separate inputs so as to prevent D from simply learning to identify discontinuities along the edge of the missing patch" - could you clarify this? How does feeding the whole completed image solve this?

Thanks!
review_title: pre-review questions

final_score: rating: 5: Marginally below acceptance threshold
review_content: 
After rebuttal:

Thanks for reporting the AlexNet results. The fact that they are not great is not so bad by itself, and as the authors mention, it would be interesting to understand why this happens. But the fact that these results  were not in the paper (and in fact still are not there) is disturbing. Moreover, some claims in the paper look wrong in the light of these results, for example:
- "This suggests that our gains stem from the CC-GAN method rather than the use of a better architecture."
- "Since discrimination of real/fake in-paintings is more closely related to the target task of object classification than extracting a feature representation suitable for in-filling, it is not surprising that we are able to exceed the performance of Pathak et al. (2016) on PASCAL classification."

These statements, and possibly other parts of the paper, have to be updated. I think the paper cannot be published in its current form. Perhaps after a revision.

--------
Initial review:

The paper demonstrates an application of generative adversarial networks (GAN) to unsupervised feature learning. The authors show that the representation learned by the discriminator of a conditional GAN trained for image inpainting performs well on image classification. As a side-effect, fairly convincing inpaintings are produced.

The proposed method combines two existing ideas: using the discriminator of a GAN as a feature learner [Radford et al. 2015] and performing unsupervised feature learning with image inpainting [Pathak et al. 2016]. Therefore conceptual novelty of the paper is limited. On the plus side, the authors implement their idea well and demonstrate state-of-the-art results on STL-10 and good results on Pascal VOC (although Pascal experiments are incomplete, see below). Overall, I am in the borderline mode, and I will gladly raise the score if the authors address my concerns regarding the experiments.

1) Experimental evaluation on Pascal VOC is not quite satisfactory. Comparison with prior work is unfair because the network architecture used by the authors (VGG) is different from the architecture used by all existing methods (AlexNet). It is great that the authors do not try to hide this fact in the paper, but I do not understand why the authors are not willing to simply run their method with AlexNet architecture, although two commenters asked them to do so. Such an experiment would strongly support authors’ claims. Current reasoning that “we thought it reasonable to use more current models while making the difference clear” is not convincing. It is great that better architectures lead to better results, but it is also very important to properly compare to prior work. On a related topic, Doersch et al. also tried using VGG architecture, would it be possible to compare to that? Yet another question: why are you not comparing to [Noroozi&Favaro, ECCV 2016] ? I would also like the authors to address the comment by Richard Zhang.

2) Qualitative inpainting results are incomplete: comparison with previous methods (for instance, [Pathak et al 2016]) is missing, and it is impossible to compare different versions of the proposed method because different images are used for different variants. I realize there may be too little space in the main paper to show all the results, but many more results should be shown in the supplementary material. Quantitative results are missing. Currently the inpainting results are just interesting pictures to look at, but they do not add as much to the paper as they could.
review_title: Final review

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper presents a semi-supervised algorithm for regularizing deep convolutional neural networks.
They propose an adversarial approach for image inpainting where the discriminator learns to identify whether an inpainted image comes from the data distribution or the generator, while at the same time it learns to recognize objects in an image from the data distribution. In experiments, they show the usefulness of their algorithm in which the features learned by the discriminator result in comparable or better object recognition performance to the reported state-of-the-art in two datasets.
Overall, the proposed idea seems a simple yet an effective way for regularize CNNs to improve the classification performance. 
review_title: review

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes a method to incorporate super-resolution and inpainting in the GAN framework for semi-supervised learning using the GAN discriminative features on larger images.

The core idea of the paper is not very novel. The usefulness of the GAN discriminative features for semi-supervised learning is already established in previous works such as CatGAN, DCGAN and Salimans et al. However this paper does a good job in actually getting the semi-supervised GAN framework working on larger images such as STL-10 and Pascal datasets using the proposed context conditioning approach, and achieves the state-of-the-art on these datasets.

I think that the authors should provide the SSL-GAN baseline for the PASCAL dataset as it is very important to compare the contribution of the context conditioning idea with the standard way of using GAN for semi-supervised learning, i.e., SSL-GAN. I can't see why the SSL-GAN can not be applied to the 64*64 and 96*96 version of the Pascal dataset (Table 3). If they have trouble training the vanilla GAN on Pascal even on the 64*64 image size, this should be mentioned in the paper and be explained. I am concerned about this specially because CC-GAN almost matches the SSL-GAN baseline on STL-10, and CC-GAN2, to me, seems like a hacky way to improve upon the core CC-GAN idea. So it would be great to compare CC-GAN and SSL-GAN on some other dataset, even if it is a downsampled PASCAL dataset.
review_title: Review

final_score: rating: null
review_content: Nice work! I have a few questions from the authors: 

I am trying to understand the main contribution of this paper. The usefulness of the GAN discriminative features for semi-supervised learning is already established in previous works such as CatGAN, DCGAN and Salimans et al. This paper claims that the proposed super-resolution approach improves upon the naive way of using the GAN discriminator for semi-supervised learning (SSL-GAN). There are two sets of experiments on STL-10 and PASCAL. The paper mentions "The SSL-GAN [baseline] performs almost as well as the CC-GAN" on STL-10. Also CC-GAN2 only marginally improves upon the baseline. So I am not really convinced that the STL-10 experiment supports the main claim of the paper.
Regarding the PASCAL dataset, I couldn't find the result of the SSL-GAN baseline on this dataset.

Also could you provide more details on how the training of semi-supervised objective is done? Suppose we have K classes. Is the output of discriminator a softmax over K classes and a sigmoid for the real/fake images? The cost function is the weighted average of semi-supervised and unsupervised costs and it seems like both costs are trained jointly from equation 8, but in the experiment section, it is mentioned "We trained for 100 fine-tuned epochs and then fine-tune the discriminator on the 96x96 labeled images". I tried to look up the code but the github repository was empty.

review_title: few questions

final_score: rating: null
review_content: How is the fine-tuning performed? In STL classification experiments, it is mentioned that the model is trained on 64x64 crops and then fine-tuned on 96x96, this means that the last layer of the discriminator will not output a single value. Are you doing data augmentation with randomly sampled 64x64 crops out of each 96x96 image? Or are layers added to perform some kind of pooling?

CC-GAN^2 formulation for semi-supervised learning not specified? Is the equation 8 in Section2.3 with the additional classification objective (eq 7)?

Is it possible to train the discriminator with the images generated from Pathak et al. and compare the performance?

Control experiments displaying the activated areas of the learned discriminator would be helpful to see what the network is learning to focus on compared to the baselines for the object class decision to be made.

review_title: questions

Meta-review: There has been prior work on semi-supervised GAN, though this paper is the first context conditional variant. The novelty of the approach was questioned by two of the reviewers, as the approach seems more incremental. Furthermore, it would have been helpful if the issues one of the reviewer had with statements in the document were addressed.
Decision: Reject