Title: TEACHING GAN TO GENERATE PER-PIXEL ANNOTA-TION

Abstract: We propose a method for joint image and per-pixel annotation synthesis with GAN. We demonstrate that GAN has good high-level representation of target data that can be easily projected to semantic segmentation masks. This method can be used to create a training dataset for teaching separate semantic segmentation network. Our experiments show that such segmentation network successfully generalizes on real data. Additionally, the method outperforms supervised training when the number of training samples is small, and works on variety of different scenes and classes. The source code of the proposed method will be publicly available.

:



Figures and Tables:

Peer review (iclr2020):
Overall: My understanding of the method is this: We take a pre-trained GAN and generate some small number of samples. We annotate those samples w/ per-pixel annotations. We then train a model that takes GAN intermediate representations and yields semantic segmentations, so that we can generate a large dataset of (GAN Sample, Semantic Segmentation) pairs, which can then be used to train a semantic segmentation model. If you want to train a semantic segmentation model for a large data set of images with no per-pixel annotations, this could potentially be useful. However, if you buy this motivating scenario (I think I do, but I don*t know much about semantic segmentation), why couldn*t we have taken the labeling budget we spent to label the GAN samples, used it to label some fraction of the real images, and then use some semi-supervised learning technique on those. Is the answer *semi-supervised learning doesn*t work yet for segmentation, even though it does seem to be working well for object recognition*? Regardless, the writing of this paper leaves a lot to be desired. I think that the authors would have to substantially improve clarity in order for me to reccommend acceptance. I would like to see other reviewers more familiar with semantic segmentation weigh in as well: how realistic is this problem setting? If we found ourselves in this problem setting, would this be a sensible way to solve the problem? The work would benefit from clarifying those points. Detailed Comments: Presumably you did not mean for your title to be *TEACHING GAN TO GENERATE PER-PIXEL ANNOTATION CONFERENCE SUBMISSIONS* The introduction is pretty abrupt... > We propose method for joint image and per-pixel annotation synthesis *a* method > We show that separate semantic segmentation network trained on synthetic dataset generalizes on real images. *a* separate > The show that GANs lear internal *they*, *learn* > Interestingly, training takes a few minutes and Decoder successfully learns on a small number of training examples Can you be a bit more specific than this? How few images can you use? How costly is it to annotate those images. > After Decoder is trained we can create a large dataset of pairs of GAN-generated images and corresponding masks predicted by Decoder. At this point in the paper, I don*t know why you want to do this? Presumably to train a segmentation model later, but the paper should read in such a way that I don*t have to guess that. > sythetic typo Minor: table 1 and 2 should have captions that tell you how to read them. Otherwise readers have to scan the text for the description. > We test two variants of backbone This is the first I*ve seen backbone mentioned. What is that? I don*t know much about semantic segmentation. Can you give intuition for why the jump in Intersection over union is bigger than the jump in accuracy? For fig 3, why are the first and 3rd examples so dark? THe top and bottom results look identical for the 2nd and 4th examples, so the darkness seems important.