Title: Semi-Supervised Learning with Context-Conditional Generative Adversarial Networks
Authors: Emily Denton
Sam Gross
Rob Fergus
Venue: ICLR 2017
Keywords: Deep learning
Semi-Supervised Learning
Computer vision

Abstract: We introduce a simple semi-supervised learning approach for images based on in-painting using an adversarial loss. Images with random patches removed are presented to a generator whose task is to fill in the hole, based on the surrounding pixels. The in-painted images are then presented to a discriminator network that judges if they are real (unaltered training images) or not. This task acts as a regularizer for standard supervised training of the discriminator. Using our approach we are able to directly train large VGG-style networks in a semi-supervised fashion. We evaluate on STL-10 and PASCAL datasets, where our approach obtains performance comparable or superior to existing methods.


TL;DR: Training GANs to in-paint images produces feature representations that yield leading results on various benchmarks.

Review discussion:
Hello,
Nice work! A couple of questions:
1. Have you tried your method with AlexNet discriminator for fair comparison in Table 2 ? Your current reasoning about supervised training is somewhat indirect. Just having numbers with AlexNet would be much more convincing.
2. Have you evaluated the performance of the encoder features vs the discriminator features in your models? Is the encoder significantly worse? You argue that "discrimination of real/fake in-paintings is more closely related to the target task of object classification than extracting a feature representation suitable for in-filling", and I tend to disagree: I actually like the reasoning behind Context Encoders. It would be interesting to get a better understanding of what is going on.
3. "We pass the completed image into D rather than the context and the patch as two separate inputs so as to prevent D from simply learning to identify discontinuities along the edge of the missing patch" - could you clarify this? How does feeding the whole completed image solve this?
Thanks!
Very nice work! We just wanted to clarify some of the numbers on Table 2 PASCAL VOC classification transfer tests.
Table 4 of https://arxiv.org/pdf/1611.09842v1.pdf (column with "frozen: none, fine-tuned: all") shows a collection of previous methods. For many methods, two values are shown: with and without the Krähenbühl et al. ICLR 2016 rescaling method.
In short, the method rescales the weights so that each layer "learns" at approximately the same rate, and in some cases, has been shown to improve fine-tuning performance. The method takes negligible time to run (a few minutes). The current literature understandably leads to some confusion, as some works concurrent to ICLR 2016 (e.g., Pathak et al. CVPR 16 and Noroozi&Favaro ECCV 16) did not quote the higher, rescaled value.
However, we believe that from this point forward, for each previous method, either both numbers should be quoted, or just the higher, rescaled, value.
Of these, the methods & PASCAL classification values for Agrawal et al. ICCV 15, Wang&Gupta ICCV 15, Doersch et al. ICCV 15, Krähenbühl et al. ICLR 2016, Pathak et al. CVPR 16, and Zhang et al. ECCV 16 (colorization) all currently exist in the peer-reviewed literature (can be traced by following the reference listed under the "Ref" column). The method for Owens et al. ECCV 16 is in the literature, but the classification score for the method is only in this pre-print. The methods proposed by Donahue et al. and Zhang et al. (split-brain auto) are only in ArXiv pre-print.
I also agree with point #1 from "AnonReviewer1" above. The 2% difference in the "from scratch" regime may be bigger or smaller for your pre-training method, and directly using AlexNet would make for an apples-to-apples comparison. I do appreciate the clarity of Table 2 and the accompanying text regarding the architecture difference.
Hi, thanks for your questions!
1. We haven't done a full comparison of our model with an alexnet architecture. Since this is an outdated architecture, we thought it reasonable to use more current models while making the difference clear. I do agree with that direct alexnet numbers would be more compelling, though we did try as much as possible to tease apart the differences stemming from method vs. architecture.
2. We have not evaluated the performance of the encoder features in our model. Thanks for this suggestion, it would be a useful comparison. I can say that the encoder features would almost certainly be worse than the discriminator features just as a result of the architecture differences. The encoder model we used is based on the dcgan architecture which performs far below vgg-style nets on classification.
Also, it is worth mentioning that Pathak et al (context-encoder work) tried to use an alexnet architecture as the encoder but were unable to successfully train the model with an adversarial loss (when using the adversarial loss -- for inpainting -- they used a different model and when using the alexnet model -- for representation learning -- they didn't use an adversarial lose).
3. If we were to pass in the context and patch as two separate inputs then the network would be able to trivially find the points where they should match (by looking for the zero-ed out edges) making the problem much easier than if the completed image is passed in.
Also, since we want the discriminator model to classify images there is an additional incentive to keep the input type as it would be for classification.
Nice work! I have a few questions from the authors:
I am trying to understand the main contribution of this paper. The usefulness of the GAN discriminative features for semi-supervised learning is already established in previous works such as CatGAN, DCGAN and Salimans et al. This paper claims that the proposed super-resolution approach improves upon the naive way of using the GAN discriminator for semi-supervised learning (SSL-GAN). There are two sets of experiments on STL-10 and PASCAL. The paper mentions "The SSL-GAN [baseline] performs almost as well as the CC-GAN" on STL-10.
Also CC-GAN2 only marginally improves upon the baseline.
So I am not really convinced that the STL-10 experiment supports the main claim of the paper.
Regarding the PASCAL dataset, I couldn't find the result of the SSL-GAN baseline on this dataset.
Also could you provide more details on how the training of semi-supervised objective is done? Suppose we have K classes. Is the output of discriminator a softmax over K classes and a sigmoid for the real/fake images? The cost function is the weighted average of semi-supervised and unsupervised costs and it seems like both costs are trained jointly from equation 8, but in the experiment section, it is mentioned "We trained for 100 fine-tuned epochs and then fine-tune the discriminator on the 96x96 labeled images". I tried to look up the code but the github repository was empty.
Thanks for the questions, here is a response, roughly in order of what you asked:
As you mention, usefulness of GAN discriminative features has been shown on several datasets of low resolution images. It is quite difficult to scale GAN training to larger images, particularly when the data distribution is diverse and labels are not available.
For example, LSUN bedrooms and church datasets have been used to train GANs and the samples are of high quality.
However, these models were trained on millions of images from the same class. More recently some methods have scaled GAN training to imagenet (eg. the work of Odena et al. submitted also to ICLR) and achieved impressive results. These results stem largely form their clever exploitation of label information. All of this is to say that, to the best of my knowledge, we still don't know how to generate large images with GANs in cases of diverse data and few labels.
So, the context conditioning we propose in this paper is a way of changing the problem the discriminator is faced with in a simple way that facilitates training on larger images.
I would actually disagree with your claim that "Also CC-GAN2 only marginally improves upon the baseline."..STL-10 is a pretty over-worked dataset for semi-supervised learning , so I think a 2.39% improvement over the previous sota, and an even bigger gain over the ssl-GAN method (suggesting the context conditioning is important), is actually compelling.
Regarding SSL-GAN on PASCAL, at the time we had trouble training a vanilla GAN on PASCAL since the images were quite large. It's possible we could try again by introducing some of the more recent tricks for training GANs that have been proposed since we did this work. (Many of these tricks could of course also be combined with the CC-GAN method too).
Figures 1 and 2 illustrate the model architectures which might elucidate the semi-supervised training. Apologies if this isn't wasn't clear initially. The discriminator has two 'heads', as you describe above: a sigmoid output indicating the probability an image is real of fake and another that is a K dimensional softmax over classes. You are correct in your understanding of the training: we train both jointly with a weighting between them (we weight them evenly, but this is of course a hyperparameter). We then continue to finetune the classifier head, which is what is being described by "100 fine-tuned epochs and then fine-tune the discriminator on the 96x96 labeled images". I hope this is now clear!
And, finally, sorry for the empty link, it'll have the code soon!
How is the fine-tuning performed? In STL classification experiments, it is mentioned that the model is trained on 64x64 crops and then fine-tuned on 96x96, this means that the last layer of the discriminator will not output a single value. Are you doing data augmentation with randomly sampled 64x64 crops out of each 96x96 image? Or are layers added to perform some kind of pooling?
CC-GAN^2 formulation for semi-supervised learning not specified? Is the equation 8 in Section2.3 with the additional classification objective (eq 7)?
Is it possible to train the discriminator with the images generated from Pathak et al. and compare the performance?
Control experiments displaying the activated areas of the learned discriminator would be helpful to see what the network is learning to focus on compared to the baselines for the object class decision to be made.
Finetuning works as follows: the final classification head during training maps from a 512x2x2 set of feature maps to a 10 dimensional softmax output. During finetuning we throw away this final layer and add a new randomly initialize head that maps from 512x4x4 features maps to a 10 dimensional softmax output.
Thus, the feature extractor part of the networks remains the same, and since it is all convolutional, the spatial extent of all the feature maps are simple doubled during this final stage. During the first part of training (i.e. not finetuning) 64x64 crops of the 96x96 image are fed to the classifier so the resolution during finetuning is not different than from training, simply the size.
Yes, your understanding wrt the CC-GAN semi-supervised objective is correct. We will try to make this clearer in the next draft.
We could try to take the generator (encoder/decoder) architecture form their model and train it with our discriminator. It is possible we would get gains from the channel-wise fully connected layer they use. Since they train with a combination of L2 and adversarial loss, it would also be interesting to see is adding this additional L2 objective to our training would help or hurt our method. It would likely produce better inpainting results (though this is not out goal) but could also perhaps result in a worse discriminator since the adversarial examples would no longer be truly adversarial wrt to discriminator.
Thanks for this suggestions, some visualization experiments would definitely be useful.
After rebuttal:
Thanks for reporting the AlexNet results. The fact that they are not great is not so bad by itself, and as the authors mention, it would be interesting to understand why this happens.
But the fact that these results  were not in the paper (and in fact still are not there) is disturbing.
Moreover, some claims in the paper look wrong in the light of these results, for example:
- "This suggests that our gains stem from the CC-GAN method rather than the use of a better architecture."
- "Since discrimination of real/fake in-paintings is more closely related to the target task of object classification than extracting a feature representation suitable for in-filling, it is not surprising that we are able to exceed the performance of Pathak et al. (2016) on PASCAL classification."
These statements, and possibly other parts of the paper, have to be updated. I think the paper cannot be published in its current form. Perhaps after a revision.
Initial review:
The paper demonstrates an application of generative adversarial networks (GAN) to unsupervised feature learning. The authors show that the representation learned by the discriminator of a conditional GAN trained for image inpainting performs well on image classification. As a side-effect, fairly convincing inpaintings are produced.
The proposed method combines two existing ideas: using the discriminator of a GAN as a feature learner [Radford et al. 2015] and performing unsupervised feature learning with image inpainting [Pathak et al. 2016].
Therefore conceptual novelty of the paper is limited. On the plus side, the authors implement their idea well and demonstrate state-of-the-art results on STL-10 and good results on Pascal VOC (although Pascal experiments are incomplete, see below).
Overall, I am in the borderline mode, and I will gladly raise the score if the authors address my concerns regarding the experiments.
1) Experimental evaluation on Pascal VOC is not quite satisfactory. Comparison with prior work is unfair because the network architecture used by the authors (VGG) is different from the architecture used by all existing methods (AlexNet). It is great that the authors do not try to hide this fact in the paper, but I do not understand why the authors are not willing to simply run their method with AlexNet architecture, although two commenters asked them to do so. Such an experiment would strongly support authors’ claims. Current reasoning that “we thought it reasonable to use more current models while making the difference clear” is not convincing. It is great that better architectures lead to better results, but it is also very important to properly compare to prior work. On a related topic, Doersch et al.
also tried using VGG architecture, would it be possible to compare to that?
Yet another question: why are you not comparing to [Noroozi&Favaro, ECCV 2016] ? I would also like the authors to address the comment by Richard Zhang.
2) Qualitative inpainting results are incomplete: comparison with previous methods (for instance, [Pathak et al 2016]) is missing, and it is impossible to compare different versions of the proposed method because different images are used for different variants. I realize there may be too little space in the main paper to show all the results, but many more results should be shown in the supplementary material. Quantitative results are missing. Currently the inpainting results are just interesting pictures to look at, but they do not add as much to the paper as they could.
Thanks for your comments.
We ran the pascal experiments with the AlexNet architecture: Supervised training of AlexNet at 128x128 achieved 51.3% and this increased to 54.1% with CCGAN training. This gap is much smaller that we saw with VGG architecture, and the absolute numbers are not competitive with other approaches. We still need to investigate why the AlexNet architecture didn't see as much of an improvement as the VGG architecture. The GAN training seemed to be stable, and the in-paintings suggest training didn't collapse, but they aren't visually as good as the VGG net ones (though we shouldn't read to much into this assessment).
Regarding comparison with Noroozi&Favaro, ECCV 2016, this was an oversight on our part and we will include these results. We didn't realize they had applied their approach to pascal. Thanks for pointing this out.
Regarding qualitative in-painting results, we didn't show comparisons with Pathak 2016 or others because our goal is not to do image in-painting. We apologize if this was unclear. Our in-painting results could possibly be improved by added a L1 or L2 loss in pixel space of feature space, but our goal is not in-painting so we didn't consider this.
Rather, we only showed in-painting results as a way of better understanding our model and whether it was learning semantically meaningful in-paintings.
This paper proposes a method to incorporate super-resolution and inpainting in the GAN framework for semi-supervised learning using the GAN discriminative features on larger images.
The core idea of the paper is not very novel. The usefulness of the GAN discriminative features for semi-supervised learning is already established in previous works such as CatGAN, DCGAN and Salimans et al.
However this paper does a good job in actually getting the semi-supervised GAN framework working on larger images such as STL-10 and Pascal datasets using the proposed context conditioning approach, and achieves the state-of-the-art on these datasets.
I think that the authors should provide the SSL-GAN baseline for the PASCAL dataset as it is very important to compare the contribution of the context conditioning idea with the standard way of using GAN for semi-supervised learning, i.e., SSL-GAN. I can't see why the SSL-GAN can not be applied to the 64*64 and 96*96 version of the Pascal dataset (Table 3). If they have trouble training the vanilla GAN on Pascal even on the 64*64 image size, this should be mentioned in the paper and be explained. I am concerned about this specially because CC-GAN almost matches the SSL-GAN baseline on STL-10, and CC-GAN2, to me, seems like a hacky way to improve upon the core CC-GAN idea.
So it would be great to compare CC-GAN and SSL-GAN on some other dataset, even if it is a downsampled PASCAL dataset.
Thanks for your comments.
Although it is true that prior work (CatGan, DCGAN) and concurrent work (Salimans) showed the usefulness of using the discriminator for semi-supervised learning, be think there is an additional contribution in this work in two respects: (1) We should that large vgg-style architectures, typically known for classification performance, can successfully be used for the discriminator architecture in a GAN setting and (2) proposing a simple and effective way of scaling GAN training to larger images form diverse sets of classes.
We will run the SSL-GAN baseline for pascal and report the numbers. Thanks for this suggestion.
This paper presents a semi-supervised algorithm for regularizing deep convolutional neural networks.
They propose an adversarial approach for image inpainting where the discriminator learns to identify whether an inpainted image comes from the data distribution or the generator, while at the same time it learns to recognize objects in an image from the data distribution. In experiments, they show the usefulness of their algorithm in which the features learned by the discriminator result in comparable or better object recognition performance to the reported state-of-the-art in two datasets.
Overall, the proposed idea seems a simple yet an effective way for regularize CNNs to improve the classification performance.
Thanks for your comments.

Meta-review:
role: PC
data: comment: There has been prior work on semi-supervised GAN, though this paper is the first context conditional variant. The novelty of the approach was questioned by two of the reviewers, as the approach seems more incremental. Furthermore, it would have been helpful if the issues one of the reviewer had with statements in the document were addressed.

Decision: Reject