Learning to Annotate Part Segmentation with Gradient Matching
Authors: Yu Yang
Xiaotian Cheng
Hakan Bilen
Xiangyang Ji
Venue: ICLR 2022 Poster (2022)
Keywords: semi-supervised learning
part segmentation
semantic segmentation
generative models
gradient matching

Abstract: The success of state-of-the-art deep neural networks heavily relies on the presence of large-scale labelled datasets, which are extremely expensive and time-consuming to annotate. This paper focuses on tackling semi-supervised part segmentation tasks by generating high-quality images with a pre-trained GAN and labelling the generated images with an automatic annotator. In particular, we formulate the annotator learning as a learning-to-learn problem. Given a pre-trained GAN, the annotator learns to label object parts in a set of randomly generated images such that a part segmentation model trained on these synthetic images with their predicted labels obtains low segmentation error on a small validation set of manually labelled images. We further reduce this nested-loop optimization problem to a simple gradient matching problem and efficiently solve it with an iterative algorithm. We show that our method can learn annotators from a broad range of labelled images including real images, generated images, and even analytically rendered images. Our method is evaluated with semi-supervised part segmentation tasks and significantly outperforms other semi-supervised competitors when the amount of labelled examples is extremely limited.

Published as a conference paper at ICLR 2022
LEARNING TO ANNOTATE PART SEGMENTATION WITH GRADIENT MATCHING
Yu Yang
Xiaotian Cheng Department of Automation
Department of Automation Tsinghua University, BNRist yang-yul 60ma is. ts tsinghua. weiighaa.dd..cmm
Tsinghua University, BNRist
cxt200ma i 1s. ts testttnninn siniguuaa...... edu. cn
Hakan Bilen
Xiangyang Ji
School of Informatics University of Edinburgh hbi lenêed. ac. uk
Department of Automation Tsinghua University, BNRist xy tts i nghua . edu. on
ABSTRACT
The success state-of-thh-aa deep neural networks heavily relies on the presence of large-scale labelled datasets, which are extremely expensive and time-consuming to annotate. This paper focuses on tackling semi-supervised part segmentation tasks by generating high-quality images with a pre-trained GAN and labelling the generated images with an automatic annotator. In particular, we formulate the annotator learning as a Pearning-to-leant problem. Given pre-trained GAN, the annotator learns to label object parts in set of randomly generated images such that a part segmentation model trained on these synthetic images with their predicted labels obtains low segmentation error small validation set of manually labelled images. We further reduce this nested-loop optimization problem to a simple gradient matching problem and efficiently solve it with iterative algorithm. We show that our method can learn annotators from a broad range of labelled images including real images, generated images, and even analytically rendered images. Our method is evaluated with semi-supervised part segmentation tasks and significantly outperforms other semi-supervised competitors when the amount of labelled examples is extremely limited.
INTRODUCTION
In recent years, deep neural networks have shown a remarkable ability to learn complex visual concepts from large quantities of labelled data. However, collecting manual annotations for growing body of visual concepts remains a costly practice, especially for pixel-wise prediction tasks such as semantic segmentation. A common and effective way of reducing the dependence on manual labels is first learning representations from unlabeled data (Pathak et al., 2016; Noroozi & Favaro, 2016; Gidaris et al., 2018; He al., 2020) and then transferring learned representations to supervised learning task (Chen et al., 2020; Zhai et al., 2019). The insight of this paradigm is that some of the transferred representations are informative, easily identified, and disentangled, such that a small amount of labelled data suffices to train the target task.
promising direction for learning such representations is using generative models (Kingma Welling, 2014; Goodfellow et al., 2014) since they attain the capability of synthesizing photorealistic images and disentangling factors of variation in the dataset. Recent methods (Goetschalekx et al., 2019; Shen et al., 2020; Karras et al., 2019; Plumerault et al., 2020; Jahanian et al., 2020; Voynov & Babenko, 2020; Seingarn-Eliezer et al., 2021; Härkönen et al., 2020) show that certain factors such as object shape and position in the images synthesized by generative adversarial networks (GANs) (Goodfellow et al., 2014) can be individually controlled by manipulating latent features. Building the success of powerful GAN models, Zhang et al. (2021) and Tritrong et al. (2021) demonstrate that feature maps in StyleGAN (Karras et al., 2019: 2020b;a) can be mapped to semantic segmentation masks or keypoint heatmaps through shallow decoder, denoted annotator in this

---

Published as a conference paper at ICLR 2022
paper. Remarkably, the annotator can be trained after few synthesized images are manually labelled, and the joint generator and trained annotator can be used as an infinite labelled data generator. Nonetheless, labelling synthesized images with "human in the loop" has two shortcomings: (i) it can be difficult to label the generated images of low quality; (ii) it requires continual human effort whenever the generator is changed.
One straightforward way to mitigate these limitations is to inverse the generation process to obtain generator features for given images. In this way, annotator, which only receives generator features as input, can be trained from existing labelled data, not necessarily generated data. However, inversion methods can not ensure exact recovery of the generator features, leading to degraded performance of annotator (see Section C.I). Another way to address this issue is to align the joint distribution of generated images and labels with that of real ones using adversarial learning as in SemanticGAN (Li et al., 2021). However, adversarial learning is notoriously unstable (Brock et al., 201 8) and requires a large amount of data to prevent discriminators from overfitting (Karras al., 2020a).
In this paper, we formulate the annotator learning as a learning-to- to-leamm problem the annotator learns to label a set of randomly generated images such that a segmentation network trained on these auto- matically labelled images obtains low prediction error on a small validation set of manually labelled images. This problem involves solving a nested-loop optimization, where the inner loop optimizes the segmentation network and the outer loop optimizes the annotator based on the solution of inner-loop optimization. Instead of directly using end-to-end gradient-based meta-learning techniques (Li et al., 2019; Pham et al., 2021), reduce this complex and expensive optimization problem into simple gradient matching problem and efficiently solve it with an iterative algorithm. We show that our method obtains performance comparable the supervised learning counterparts (Zhang et al., 2021; Tritrong et al., 2021) yet overcomes their shortcomings such that training annotators can utilize labelled data within a broader range, including real data, synthetic data, and even oft-of-domain data.
Our method requires large quantity of unlabeled data for pre-training GANs and a relatively small amount of labelled data for training annotators, which drops into a semi-supervised learning setting. Our method relies on unconditional GANs, of which the performance limit the application of our approach. As the state-of-the-ar GANs only produce appealing results on single-class images but struggle to model complex scenes, we focus part segmentation of particular classes and avoid multi-object scenes.
Our contribution can be summarized as follows. (i) We formulate the learning of annotations for GAN-generater images as a learning-to-leam problem and propose an algorithm based on gradient matching to solve it. Consequently, a broad range of labelled data, including real data, synthetic data, and even out-of-domain data, is applicable. (ii) We empirically show that our method significantly outperforms other semi-supervised segmentation methods in the few-label regime.'
2 RELATED WORK
Semi-supervised learning Semi-supervised learning (SSL) (Zhu, 2005) augments the training of neural networks from a small amount of labelled data with a large-scale unlabeled dataset. A rich body of work regularizes networks on unlabeled data with consistency regularization. In particular, networks are required to make consistent predictions over previous training iterations (Laine & Aila, 2017: Tarvainen & Valpola, 2017: Izmailov al., 2018), noisy inputs (Miyato et al., 201 40, or augmented inputs (Berthelot et al., 2019b;a; Sohn et al., 2020). This paradigm is also interpreted as framework where the teacher produces pseudo labels for unlabelled data to supervise the training of students. Pseudo labels can be enhanced with heuristics (Lee et al., 2013; Berthelot et al., 2019a) or optimized towards better generalization (Pham et al., 2021; Li et al., 2019). Apart from image classification tasks, the above ideas are also successfully applied to semantic segmentation tasks (Hung et al., 2018; Ouali et al., 2020; Ke et al., 2020; French et al., 2020). Our work can also be interpreted from beacher-and-stuce that adopt homogeneous network structures for teacher and student, our work employs generative models as teacher and discriminative models student.
perspective. In contrast to the above methods
Other SSL approaches explore the utility of unlabelled data from a representation learning perspective. In particular, deep neural networks are first optimized on the unlabeled data with self-supervised
Code available https: http//gtthtu..coo yangyunngyyyyyy 12/1 12/1amm
2

---

Published as a conference paper at ICLR 2022
learning tasks (Pathak et al., 2016; Noroozi & Favaro, 2016; Gidaris et al., 2018; He et al., 2020) to gain powerful and versatile representations, and then finetuned on the labelled data to perform target tasks (Chen et al., 2020; Zhai et al., 2019). In addition, generative models can also learn efficient and transferable representations without labels. To this end, one promising direction is to decode generative features into dense annotations such as segmentation masks and keypoint heatmap to facilitate SSL, as done in SemanticGAN (Li et al., 2021), DataselGAN (Zhang et al., 2021), RepurposeGAN (Tritrong el al., 2021) and our work. However, our work differs from these works in that we formulate the annotator learning as learning-too -o-iinn-tt--ennn problem that is solved with gradient matching. Particularly, unlike SemantieGAN (Li et al., 2021) that learns annotator and data generator jointly, our method presumes fixed pre-trained GAN and learns annotator only, which circumvents the complication of joint learning.
Semantic Part Segmentation Semantic part segmentation, which decomposes rigid or non-rigid objects into several parts, is of great significance in tremendous computer-vision tasks such as human parsing (Dong et al., 2014; Nie et al., 2018; Fang et al., 2018; Gong et al., 2017), pose estimation (Yang Shotton et al., 2011; Xia et al., 2017) and 3D object understanding (Yi et al., 2016; Song et al., 2017). The challenge of part segmentation arises from viewpoint changes, occlusion, and lack of clear boundaries for certain parts. Existing datasets manage alleviate the problem by enriching the data and annotations. CelebAMask-HH (Lee et al., 2020a) is a large-scale face image dataset annotated with masks of face components. Chen et al. (2014) provides additional part segmentation upon PASCAL VOC dataset (Everingham et al., 2010), Those datasets meet the demands of latest data-driven and deep neural network based method (Lee et al., 2020b; Wang et al., 2015; Wang & Yuille, 2015). However, well-curated part segmentation dataset containing fine-grained pixel-wise annotation usually requires time-consuming manual labelling work, which holds back the development of this field. This problem also motivates the latest work (Zhang et al., 2021; Li et al., 2021; Tritrong et al., 2021) to leverage a generative model as an infinite data generator in semi-supervised learning setting, which gains promising results.
& Ramanan, 2011;
Gradient matching Gradient matching is an important technique in meta learning (Li et al., 2018; Sariyildiz& & Cinbis, 2019), which is shown to be effective in domain generalization (Li et al., 2018), zero-shot classification (Sariyildiz & Cinbis, 2019), and dataset condensation (Zhao et al., 2021; Zhao & Bilen, 2021) etc. Although our work's 's general principle and gradient matching loss function motivated to solve different problem. In particular, Gradient Matching Network (GMN) (Sariyildiz & Cinbis, 2019) employs gradient matching to train conditional generative model towards application on zero-shot classification. Dataset Condensation (DC) (Zhao et al., 2021: Zhao & Bilen, 2021) condenses a large-scale dataset into a small synthetic one with gradient matching. Our work shows that gradient matching also an effective way to learn annotations of GAN-generated images from limited labelled examples. Moreover, in contrast to DC, where the synthesized images are typically not realistic, our work is concerned about photo-realistic synthetic images.
resemble those in the above works,
METHOD
3.1 PRELIMINARY
Let G denote generative model trained from dataset Dio without labels. G can be trained via adversarial learning (Goodfellow et al., 2014) such that it can generate photo-realistic images. The generation process is formulated
(1) where z denotes a random variable drawn from a prior distribution P. which is typically normal distribution, x denotes the generated image, and h represents the hidden features in G. An annotator A., is constructed to decode into labels
= G(zz, 2~h P.
y = (h),
where w parameterizes the annotator. Let D, denote a manually labeled dataset. Given fixed G, our problem is to learn annotator A., from D, such that A. can annotate images generated from G.
3

---

Published as a conference paper at ICLR 2022
G Aw
Synthetic Data
Labeled Data
z
Sg
t
S
Cg(0,,) w)
Gradient Matching
()
Figure 1: Illustration of learning to annotate with gradient matching. The procedure is as follows. (i) The gradients of segmentation network So on labeled examples are computed, denoted (0). (ii) A batch of synthetic data is randomly generated by generator G and labeled by annotator Ag. (iii) The gradients of 5g on synthetic data are computed, denoted as VoLg (0,w) w). (iv) Gradient matching between VoLi(0) and VeL, (0,w) is computed to optimize Aw.
Revisiting supervised annotator learning DatasetGAN (Zhang et al., 2021) and Repur- PoseGAN (Tritrong et al., 2021) train annotators in supervised learning manner. First, set of synthetic images {x,]_ generated from G are selected and annotated by humans and their generator features {hr) are reserved. These features and manual labels {y:}
J constitute
labeled dataset D; {(x, y-)) Second, an annotator is trained by minimizing the loss computed between annotator prediction and ground truth labels,
arg arg b(,,,.)) (Aw (b)
(3)
where f denotes per-pixel cross-entropy function for segmentation tasks. This method, however, requires Dj to a synthetic and generator-specifec set, which has the following two defects. (i) Details might be illegible in synthetic images, which increases the difficulty of annotation. (ii) Data needs to be re-labelled whenever the generator is changed. Although projecting labelled images into GAN latent space can alleviate the above drawbacks, it leads degraded performance due inexact inversion (see Section C.1 in the appendix).
3.2 LEARNING TO ANNOTATE WITH GRADIENT MATCHING
Learning to learn et al. (2019); Pham et al. (2021). In particular, we introduce a segmentation network Sg, deep neural network parameterized with It takes as input an image and outputs segmentation mask. This segmentation network learns the target task only from the automatically labeled generated images, which is formulated as
We formulate the problem of learning Aw a learning-to-learn problem like Li
g (w) arg min (l)) f(S (10 Au
C,(0,w)
where f denotes the loss function for the target task. The optimal solution of Equation 4 depends on the w, which is therefore denoted as 8* (w:). ) The performance of learned segmentation network can be evaluated with loss on the labeled dataset Dj
(0 (w)) = (x (Sg (w) (x)
(5)
The aim of learning the annotator is produce high-quality labels such that the learned segmentation network achieve minimal loss on D,, which formulated as
min (w),,,
(6)
where ('(w) = arg min (0) 6
This problem is nested-loop optimization problem, where the inner loop optimizes the segmentation network (Equation 4) and the outer loop optimizes the annotator based on the solution of the inner loop. This optimization problem does not require manual labelling of generated images. To this end, any labelled images, including real labelled images, can serve D,. While this problem can be solved with MAML-like (Finn et al., 2017) approaches (Li et al., 2019; Pham et al., 2021), its solution involves expensive unrolling of the computational graph over multiple states, which is memory and computation intensive. Hence, we reduce into a simple gradient matching problem as follows. See Section C.1 in the appendix for further comparison and discussion.
4

---

Published as a conference paper at ICLR 2022
Gradient matching Inspired by Li et al. (2018), we reduce the nested- optimization problem (Equation 6) into a gradient matching problem. First. as commonly done in gradient prasde meta learning approaches, the optimization of segmentation network (Equation is approximated with one-step gradient descent,
0 (w) x 0g (00 W),
where denotes the learning rate and lg denotes the initialized parameter. By plugging this equation into (w)), unfolding Taylor series and omitting the higher order term, we have
(6 (w)) Li(0- -n----- (V
-((0)
VoL +...
2J L((0)) - Vi (o)) (00.w)
The last approximation is conditioned on V w|ll S e, where is very small amount. By further omitting the constant term (())) in Equation 8, we obtain the approximation of Equation as
min (z(
s.t. VnV will s.
We notice connection between this constrained optimization problem with the following uncon- strained optimization problem
min V VoL (00,
(10)
where to minimize this objective, one needs to simultaneously maximize the dot product between the gradients and minimize the norm of ( w). Note that V. (e,, [V),, (1-
Vn o (90.ww))
"
V. V.L [V V [.,,00, [.Co(00]] and the constant term and constant coefficient can be omitted without changing the optimal solution. We re-write the learning objective of Equation 10 as
V (V Vel,
Cym (Oo, w) D(VoLi(Oo), (ol,( =
(11)
which computes the cosine distance between the gradients of segmentation network automatically labeled generated data and manually labeled data (see Scction B in appendix for dealing with gradient matching multi-layer neural networks). To this end, the nested-loop optimization (Equation 6) is reduced into a simple one that minimizes the gradient matching loss (Equation 11). An illustration of this procedure is presented in Figure 1.
Algorithm Note that the gradient matching problem learns an annotator for a particular network parameterized with dg However, a desirable annotator should produce universal automatic labels for training any segmentation network with previously unseen parameters. Hence, we present an alternate training algorithm of annotator and segmentation network in Algorithm where the segmentation network is iteratively updated using the automatically labelled synthetic data, and the annotator is iteratively updated with gradient matching. This algorithm allows the annotator be optimized over various segmentation network states, leading to aegmentaiion-netoorr-aganno gradient matching problem for each segmentation network parameter solved with K K-step gradient descent, and typically K 1 works well in practice. Implementation details are available in the appendix (see Section B).
annotator: The
EXPERIMENTS
4.1
SETUP
Datasets We evaluate our method six part segmentation datasets: CelebA, Pascal-Horse, Pascal- Aeroplane, Car-20, Cat-16, and Face-34, CelebA (Liu et al., 2015) large-scale human face dataset where 30,000 images are annotated with up to 19 part classes, also known CelebAMask- HQ (Lee et al., 2020a). We consider a subset face classes and use the first 28,000 images as an unlabeled set, the last 500 images Li et al. (2021). Hiscal-Horse and Pascal- Pascal-Aerophe Aeroplane are constructed by taking images of horse and aeroplane from PascalPart (Chen al., 2014) which provides detailed part segmentation annotations
test set, and the rest 1500 images a labelled set, in
5

---

Published as a conference paper at ICLR 2022
Algorithm 1: Learning to annotate with gradient matching.
Inputs
G
trained generator
Di
set labeled examples
w, Aw,
initial annotator parameters, annotator, and learning rate for annotator
0. So. no
initial segmentor parameters, segmentor, and learning rate for segmentor
T, K, and total number of optimization steps, interval of updating segmentor, and batch size
for < to T do
// update annetator Aw
B. + y() DD Dij
1/ sample a batch of labeled examples // compute gradients B 1/ sample a batch of synthetic data
Voli 1 EE (o.()((x))))ss)
Bg + (xb)))
VoLg + 1 E Vof(So(z)); A (h;)))
// compute gradienta on B, // compute gradient matching loss // update annetator parameters
Cym D(VeLg, Voli)
w 7/ VuLgm
// update segmentor S
ift mod K then
Bg + {x A (b;))),
1/ sample batch of synthetic data // compute loss on synthetic data // update segmentor parameters
Lg 2
Aw,)))
0- 8 VoLy
12
end
end
A segmentor Sy
for images from Pascal VOC 2010 (Everingham et al.). The selected images are cropped according to bounding box annotations. Full process details are available in the appendix. We finally obtain 180 training images, 34 validation images, and 225 test images to constitute Pascal-Horse, 180 training images, 78 validation images, and 269 test images constitute Puscal-Aeroplane In Pascal-Hoo and Pascal- Pascal-Aeroplane, additional images from LSUN (Yu et al., 2015) are utilized as unlabeled set. Car-20, Cat- 16, and Face-34, released by Zhang al. (2021), ), are three part-segmentation datasets annotated with 20, 16, and 34 part classes for car, cat, and human face, respectively. We use the same datasets to make a fair comparison to DataselGAN (Zhang et al., 2021). In these datasets, all the training images are images generated from pre- trained StyleGAN (Karras al., 2019) while all the test images are real images. Car-20 contains 16 training images and 10 test images; Cat-16 16 contains 30 training images and 20 test images; Face-34 contains 16 training images and 20 test images.
Pre-trained generators We models of StyleGAN family (Karras al., 2019; 2020b;a) that either trained by ourselves or publicly available. Details are provided in the appendix (see Section B).
Evaluation We use mean intersection over union (mloU) to evaluate the performance of segmenta- tion networks. On CelebA, Pascal-Horse, and Pascal-Aeroplam we use the validation set to select checkpoints and report mloU across all foreground classes, denoted as "FG-mloU", on the test set. On Car-20, Cat- 16 and Face-34, follow the setting of DatasetGAN (Zhang et al., 202 and report the cross-validation mloU across all classes, including background on the test set.
4.2 SEMI-SUPERVISED PART SEGMENTATION
Since our method has mild requirement for the labelled data, in this section, we show the effec- tiveness of our method under three different cireumstances: real images as labelled data, synthetic images as labelled data, and out-of-domain images as labelled data.
Real images as labeled data Following SemanticGAN (Li et al., 2021), our methods are compared against SSL methods that have codes available" including Mean Teacher (MT) (Tarvainen Valpola,
We adopt the implementation in https: / thub. com wow///KkKe/////cccccoccccccccce/ xel 881. For these SSL. meth- ods, the real unlaheled images used as unlabeled set.

---

Published as a conference paper at ICLR 2022
CelebA
Pascal-Horse
-erepllaee
40-
50-
30-
Number of labelled ecamples (lgglsalle)
Numbar of labelled examples
Nomber of labellad examples
MT (Li et al., 2020)
AdvSSL (Hung al., 2018)
Ours
GCT (Ke al., 2020)
SemantiGAAN al., 2021)
Figure 2: Benchmark on CelebA (Leff), Pascal-Horse (Middle), and Pascal-Aeroplane (Right). The y-axes are FG-mloU (%) on test set.
Cat-16
Face-34 34
Car-20
Methods
S
annotations:
annotations: 16
# annotations: 16
# classes:: 16
# classes: 16
classss:: 20
TL
DeepLabv3
21.58 0.61
45.77 + 1.51
33.91 + 0.57
SSL (Mittal et al., 2019)t
DeepLabv3
24.85 0.35
48.17 = 0.66
44.51 = 0.94
DatasetGAN
StyleGAN
DeepLabv34
32.63
0.68
54.55 = 0.25
67.53 = 2.58
(Zhang et al., 2021)
StyleGAN
U-Net*
31.36 0.76
53.84 +: 0.41
66.27 1:: 2.75
StyleGAN
DeepLabv3
33.89 0.43
52.58
0.61
63.55 = 2.25
StyleGAN
U-Net
32.64 0.74
53.69
0.54
60.45 = 2.42
Ours
StyleGAN2
DeepLabv3
33.56 1 0.17
55.10
0.39
61.21 = 2.07
StyleGAN2
U-Net
31.90 + 0.75
53.58 = 0.45
58.30 = 2.64
Our downstream segmentation performance
Cat-16
Face-34
Car-20
Source \ Downstream
DeepLabv3
U-Net
DeepLabv3
U-Net
DeepLaby3
U-Net
DeepLab
33.38 1 0.66 33.39 0.74
55.11
0.63 54.77 1 0.32
63.47 1 2.33 62.72 2.89
U-Net
33.38 + 0.40 32.42 0.62
54.05
0.40 53.80 + 1.06 63.22 2.42 62.25 2.77
Table 1: Comparisons to DatasetGAN on Car-20, Cat-16, Face-34. The performance is evaluated with ml).(%) TL: transfer learning. SSL: semi-supervised learning. f: Results taken from Zhang et al. (2021). $: Up-to-date performance from DatasetGAN github repository. #: Results obtained by ourselves using DatasetGAN source codes.
2017; Li al., 2020), Guided Collaborative Training (GCT) (Ke al., 2020), an adversarial- learning-bast -based supe-super segmentation method (AdvSSL) (Hung et al., 2018), as well as SemanticGAN (Li et al., 2021). We benchmark performances on CelebA, Pascal-Hae Horse, and Pascal- Acroplane with respect to different amount of labelled data and present the results in Figure 2. Qualitative results are available the appendix (see Figure D.7, D.8, D.9). Our method significantly outperforms other SSL. methods when labelled data is extremely limited. When the number of labelled images decreases, the performance of our method drops mildly while the performances of other methods decrease drastically. We attribute it to utilization of the highly interpretable generator features. Notably, SemanticGAN (Li et al., 2021) also exploits the generative features to produce segmentation masks, but its performance degrades faster than methods the number of labelled images decreases. We conjecture it is due to that the adversarial learning employed by SemanticGAN typically requires sufficient data prevent discriminators from overfitting. The edges of our method become marginal under the case of a large number of labels. One possible reason is that the quality of pre-trained GAN needs be further improved, which requires more research in the future. (see Section C.5 for more discussion)
Synthetic images as labeled data We comprehensively compare our method to supervised learning counterpart, DatasetGAN (Zhang et al., 2021) Car-20, Cat-16, and Face-34. We follow
7

---

Published as a conference paper at ICLR 2022
(a) Out-of-domain labeled examples
Generated images with learned annotations
Figure Cross-domain annotator learning demo. (a) Out-of-domain labelled examples: car images and ground truth segmentations rendered from 3D CAD model. (b) Generated images with learned annotations: our method learns a reasonable annotator for GAN generated images.
the same setting as in DatasetGAN, using precisely the same synthetic images as training data. Quantitative results are presented Table and qualitative comparison available in the appendix (see Figure 1.12, 11 D.10). First, our method achieves the same performance level with Dataset- GAN on Face-34 and Cat- 16. Second, as merit of our method, we can upgrade the generator to StyleGAN2 (Karras et al., 2020b;a) without requiring extra human effort. However, upgrading the generator brings modest improvement on Face-34 and Cat- 16 and even bit decrease on Car-20. This result suggests that synthesis quality measured by metrics like FID not precisely depict the disentanglement of generator features. Finally, our method could adapt to different segmentation architectures such DeepLabvv3 (Chen al., 2017) and U-Net (Ronneberger al., 2015).
We further use our trained annotator to generate synthetic segmentation datasets for training down- stream segmentation networks. Table shows the results when training annotators with gradient matching and different downstream networks. First, our learned synthetic datasets can be used to train downstream segmentation networks of different architec- ture that achieve good performances. This result indicates that our learned synthetic dataset is not architecture-sper speciffc. Second, DeepLabv3 as a source network generally leads higher performances than U-Net, suggesting that the segmentation networks used for gradient matching affect annotator learning in our method.
different source networks for
Out-of-domain images labeled data We further consider challenge where out-of-domain images constitute labelled data. In particular, we render a set of car images from annotated 3D CAD models provided by CGPart (Liu al., 2021) using graphic tools (e.g. blender). Since 3D models are annotated with parts, the ground truth part segmentation of these rendered images are available via 3D projection. These rendered images vary in viewpoints and textures and have a significant domain gap compared to realistic car images due to inexact 3D models, unrealistic textures, artificial lighting etc., raising challenges for learning annotators. Despite so, as presented in Figure 3, our method still learns reasonable annotators that produce quality segmentation labels for GAN generated images. is noteworthy that it is nearly impossible for DataselGAN (Zhang et al., 2021) and RepurposeGAN (Tritrong et al., 2021) to utilize such out-of-domain labelled data.
4.3 ABLATION STUDY
The frequency of updating segmentation network The results of an ablation study w.r.t. the interval of updating segmentation network, K, is shown in Figure Experiments are run on Face-34 and Cat- 16 with DeepLabv3 as segmentation network and StyleGAN as generator such that pairs of generator features and ground truth segmentation are available to measure the performance of the annotator quantitatively. It shows that the annotator learns significantly faster when the segmentation network is updated more frequently. The visualization shows that very detailed parts, e.g. pupils in Face-34 34 and eyes in Cat- 16t-16, emerge sooner if the segmentation network is updated more frequently. is more effective to match gradients for various segmentation network parameters than optimize gradient matching for single segmentation network parameters.
Gradient matching on partial segmentation network parameters As the implementation of gra- dient matching requires maintenance of computation graph of hackpropagation through segmentation networks, one may be concerned about the training efficiency for very deep neural networks. Here
hhttps://q1 fu24, w224.github. wtithub..o/cgpaa... io/cgpatt/
8

---

Published as a conference paper at ICLR 2022
# Stop 30")
4
8
12
16
20
24
28
32
36
1
(a) Annotator performance
(b) Generated Image
(c) Evolution Automatic labels
Figure 4: Ablation study w.r.t. K. Evolution of the learned annotator on Face-34 (first row) and Cat-16 (second row). (a) Annotator performance (mloU (%) on training data w.r.t. number of updating annotator steps (K). (b) Examples of generated images. (c) Evolution of automatic labels.
Blocks
All
Res2~Head
Resd~HHead Res3>Heead
Res5~Head
Head
Time step
3.34x
3.34x
3.28x
3.08
1.62x
1x
mloU (%)
33.71 + 0.77 34.38 + 0.49 33.81 = 0.99 33.78 + 0.70 34.12 0.57
33.56 0.17
Table 2: Ablation study w.rt. blocks for matching gradients. Evaluation is performed on Cat- where StyleGAN2 is the generator and DeepLabv3 is the segmentation network. "Time / step" is measured as the ratio with respect to the default setting. * denotes the default setting.
we present a simple strategy to trade off the training efficiency and overall performance: gradient matching can be done only on the part of segmentation network parameters. Table 2 presents an ablation study about matching blocks is the segmentation network. We consider network blocks in DeepLabv3 with ResNet-101 as back- bone from bottom to top: Resl, 1, Res2, Res3, Res4 4 and Res5 that residual blocks in the backbone, and Head that is the segmentation head. Results show that skipping the bottom blocks for gradient matching improves the training speed while the segmentation performance is affected little. We conjecture the reasons can be two folds: (i) the backbone parameters are pre-trained such that little needs to be changed during training; (ii) the gradients accumulate more randomness as they backpropagate to the bottom blocks of deep neural networks. Therefore, in practice, we only match the gradients of Head in Levv by default. Additional results are available in the appendix (see Section C.3).
Cat-16, where StyleGAN2 is the generator and DeepLabv3
5 CONCLUSION
We propose a gradient-matching-baeed method to learn annotator, which can label generated images with part segmentation by decoding the generator features into segmentation masks. Unlike existing methods that require labelling generated images for training an annotator, our method allows broader range of labelled data, including realistic images, synthetic images, and even rendered images from 3D CAD models. On the benchmark of semi-superee cemi-supervised part segmentation, our method significantly outperforms other supervisdd segmentation methods under the circumstances of extremely limited labelled data but becomes less competitive under large-scale settings, which requires future research. The effectiveness of our method is validated on a variety of single-class datasets, including well-aligned images, e.g. CelebA and Face-34, well images in the wild part of which contain cluttered backgrounds, e.g. Pascal-Horse Horse, Pascal-Aeros Aeropaae,,,,,,,, Car-20 and Cat-16. 16. In terms more complex scenes, discussion and investigation modelling, it is promising to use more powerful generative models and explore more computer vision problems under different challenging settings in the future.
further needed. With the rapid progress of generative
9

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGEMENTS
This work was supported by the National Key R&D Program of China under Grant 2018AAA0102801. National Natural Science Foundation of China under Grant 6 61620106005, and Beijing Municipal Science and Technology Commission grant Z201 100005820005. Hakan Bilen supported by the EPSRC programme grant Visual AI EP/T028572/1.
REPRODUCIBILITY STATEMENT
The appendix provides all the details about the datasets, network structure, and training curriculum. Code to reproduce our main results publicly available at https: /ttp::///iie //gitubbtttt/tt commyangyu12// lagm.
REFERENCES
David Berthelot, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel. Remixmatch: Semi-supervised learning with distribution matching and augmentation anchoring. In ICLR, 2019a.
David Berthelot, Nicholas Carlini, lan Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel. Mixmatch: A holistic approach to semi-supervised learning. NeurIPS, 2019b.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. In ICLR, 2018
Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint urXiv:1706.0558/ 2017.
Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In /CML, 2020.
Xianjie Chen, Roozbeh Mottaghi, Xiaobai Liu, Sanja Fidler, Raquel Urtasun, and Alan Yuille. Detect what you can: Detecting and representing objects using holistic models and body parts. In CVPR, 2014.
Jian Dong, Qiang Chen, Xiaohui Shen, Jianchao Yang, and Shuicheng Yan. Towards unified human parsing and pose estimation. In CVPR, 2014.
M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The PASCAL Visual Object Classes Challenge 2010 (VOC2010) Results. http://www.pascal- http:://www.aaaaae//////////////
Mark Everingham, Lue Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. IJCV, 2010.
Hao-Shu Fang, Guansong Lu, Xiaolin Fang, Jianwen Xie, Yu-Wing Tai, and Cewu Lu. Weakly and semi supervised human body part parsing via pose-guided knowledge transfer. arXiv preprint 10X1v:1805.04310, 2018.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta- meta-learning for fast adaptation of deep networks. ICML, 2017.
Geoffrey French, Samuli Laine, Timo Aila, Michal Mackiewicz, and Graham Finlayson. Semi- supervised semantic segmentation needs strong, varied perturbations. In BMVC, 2020.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. In ICLR, 2018.
Lore Goetschalekx, Alex Andonian, Aude Oliva, and Phillip Isola. Ganalyze: Toward visual definitions cognitive image properties. In ICCV, 2019.
Ke Gong, Xiaodan Liang, Dongyu Zhang, Xiaohui Shen, and Liang Lin. Look into person: Self- supervised strncture-sensitive learning and new benchmark for human parsing. In CVPR, 2017.
10

---

Published as a conference paper at ICLR 2022
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. NeurIPS, 2014.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
Wei Chih Hung, Yi Hsuan Tsai, Yan Ting Liou, Yen-Y Yu Lin, and Ming Hsuan Yang. Adversarial learning for semi-supervised semantic segmentation. In BMVC, 2018.
Erik Härkönen, Aaron Hertzmann, Jaakko Lehtinen, and Sylvain Paris. Ganspace: Discovering interpretable gan controls. In NeurIPS, 2020.
Izmailov, AG Wilson, D Podoprikhin, Vetrov, and T Garipov. Averaging weights leads to wider optima and better generalization. In UAI, 2018.
Ali Jahanian, Lucy Chai, and Phillip Isola. On the" steerability" of generative adversarial networks. In ICLR, 2020.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. CVPR, 2019.
Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. In NeurlPS, 2020a.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In CVPR, 2020b.
Zhanghan Ke, Di Qiu, Kaican Li, Qiong Yan, and Rynson WH Lau. Guided collaborative training for pixel-wise semi-supervised learning. ECCV, 2020.
Diederik Kingma and Max Welling. Auto-encoding variational bayes. In ICLR, 2014.
Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. In ICLR, 2017.
Cheng-Han Lee, Ziwei Liu, Lingyun Wu, and Ping Luo. Maskgan: Towards diverse and interactive facial image manipulation. In CVPR, 2020a.
Cheng-Han Lee, Ziwei Liu, Lingyun Wu, and Ping Luo. Maskgan: Towards diverse and interactive facial image manipulation. In CVPR, 2020b.
Dong-Hyun Lee et al. Pscudo-label: The simple and efficient semi-supervised learning method for deep neural networks. ICML workshop, 2013.
Da Li, Yongxin Yang. Yi-Zhe Song, and Timothy M Hospedales. Learning to generalize: Meta- learning for domain generalization. In AAAI, 2018.
Daiqing Li, Junlin Yang, Karsten Kreis, Antonio Torralba, and Sanja Fidler. Semantic segmentation with generative models: Semi-supervised learning and strong out-of-domain generalization. In CVPR, 2021.
Wei-Hong Li, Chuan-Sheng Foo, and Hakan Bilen. Learning to impute: A general framework for semi-supervised learning. arXiv preprint 19X2v 2019.
Xiaomeng Li, Lequan Yu, Hao Chen, Chi-Wing Fu, Lei Xing, and Pheng-Ann Heng. Transformation-
consistent self-ensembling model for semisupervised medical image segmentation. TNNLS, 2020.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. ECCV, 2014.
Tsung- Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In CVPR, 2017.
Qing Liu, Adam Kortylewski, Zhishuai Zhang, Zizhang Li, Menggi Guo, Qihao Liu, Xiaoding Yuan, Jiteng Mu, Weichao Qiu, and Alan Yuille. Cgpart: A part segmentation dataset based on 3d computer graphics models. arXiv preprint arXiv: 103. 14098, 2021.

---

Published as a conference paper at ICLR 2022
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In ICCV, 2015.
Sudhanshu Mittal, Maxim Tatarchenko, and Thomas Brox. Semi-supervised semantic segmentation with high-and low-level consistency. TPAMI, 2019.
Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii Virtual adversarial training: regularization method for supervised and semi-supervised learning. TPAMI, 2018.
Xuecheng Nie, Jiashi Feng, and Shuicheng Yan. Mutual learning to adapt for joint human parsing and pose estimation. ECCV, 2018.
Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. ECCV, 2016.
Yassine Ouali, Céline Hudelot, and Myriam Tami. Semi-supervised semantic segmentation with cross-consistency training. CVPR, 2020.
Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. In CVPR, 2016.
Hieu Pham, Zihang Dai, Qizhe Xie, and Quoc V Le. Meta pseudo labels. In CVPR, 2021.
Antoine Plumerault, Hervé Le Borgne, and Céline Hudelot. Controlling generative models with continuous factors variations. In ICLR, 2020.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In MICCAI, 2015.
Mert Bulent Sariyildiz and Ramazan Gokberk Cinbis. Gradient matching generative networks for zero-shot learning. In CVPR, 2019.
Yujun Shen, Jinjin Gu, Xiaoou Tang, and Bolei Zhou. Interpreting the latent space of gans for semantic face editing. In CVPR, 2020.
Jamie Shotton, Andrew Fitzgibbon, Mat Cook, Toby Sharp, Mark Finocchio, Richard Moore, Alex Kipman, and Andrew Blake. Real-time human pose recognition in parts from single depth images. In CVPR, 2011.
Dylan Sisson. Pixar one twenty eight. milxar-onettwent--ee
htt. com/
2018.
Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Do- gus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. NeurIPS, 2020.
Yafei Song, Xiaowu Chen, Jia Li, and Qinping Zhao. Embedding 3d geometric features for rigid object part segmentation. ICCV, 2017.
Nurit Spingarn- Eliezerr, Ron Banner, and Tomer Michaeli. Gan stecrability without optimization. ICLR, 2021.
Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. In NewrIPS, 2017.
Nontawat Tritrong, Pitchaporn Rewatbowornwong and Supasorn Suwajanakorn. Repurposing gans for one-shot semantic part segmentation. CVPR, 2021.
Stavros Tsogkas, Iasonas Kokkinos, George Papandreou, and Andrea Vedaldi. Deep learning for semantic part segmentation with high-level guidance. arXiv preprint arXiv: 15... 024500 2015.
Andrey Voynov and Artem Babenko. Unsupervised discovery of interpretable directions in the gan latent space. In ICML, 2020.
12

---

Published as a conference paper at ICLR 2022
Jianyu Wang and Alan L Yuille. Semantic part segmentation using compositional model combining shape and appearance. In CVPR, 2015.
Peng Wang, Xiaohui Shen, Zhe Lin, Scott Cohen, Brian Price, and Alan L Yuille, Joint object and part segmentation using deep learned potentials. In ICCV, 2015.
Fangting Xia, Peng Wang. Xianjie Chen, and Alan L Yuille. Joint multi-person pose estimation and semantic part segmentation. In CVPR, 2017.
Yi Yang and Deva Ramanan. Articulated pose estimation with flexible mixtures-of-parts. In CVPR, 2011.
Li Yi, Vladimir G Kim, Duygu Ceylan, I-Chao Shen, Mengyan Yan, Hao Su, Cewu Lu, Qixing Huang, Alla Sheffer, and Leonidas Guibas. scalable active framework for region annotation in 3d shape collections. ToG, 2016.
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv: 03360, 2015.
Xiaohua Zhai, Avital Oliver, Alexander Kolesnikov, and Lucas Beyer. S41: Self-supervised semi- supervised learning. In ICCV, 2019.
Yuxuan Zhang, Huan Ling, Jun Gao, Kangxue Yin, Jean-Francois Lafleche, Adela Barriuso, Antonio Torralba, and Sanja Fidler. Datasetgan: Efficient labeled data factory with minimal human effort. In CVPR, 2021.
Bo Zhao and Hakan Bilen. Dataset condensation with differentiable siamese augmentation. In ICML, 2021.
Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with gradient matching. In ICLR, 2021.
Yifan Zhao, Jia Li, Yu Zhang, Yafei Song, and Yonghong Tian. Ordinal multi-task part segmentation with recurrent prior generation. TPAMI, 2019.
Xiaojin Jerry Zhu. Semi-supervised learning literature survey. 2005.
13

---

Published as a conference paper at ICLR 2022
A
DATASET DETAILS
CelebA Following Li et al. (2021), we only consider part classes: background, ear, eye, eyebrow, skin, hair, mouth, and nose. Table A. presents the protocol for merging the 19 part classes into 8 classes, CelebAMask-HQ contains 30,000 annotated images in total. We split this dataset into the unlabeled set, training set, validation and test in Table A. (Right).
8 classes
19 classes
Split
Image id
background
background, hat, car.r, neck_1, neck, cloth
Unlabeled set
1~~28,000
ear
Lear, r.ear
Training set (2)
28,001~28,002
eye
Leye, r_eye
Training set (10)
28,001~2280100
eyebrow
eyebrow
Training set (30)
28,001-28,030
skin
Training (150)
28,001-28.1150
hair
hair
Training (1500)
28,001~29.500
mouth
mouth, u lip, lip
Validation set
27,501~288,000
nose
nose
Test set
29,501~30,000
Table A.I: Left: Protocol for merging part classes CelebA, Right: Dataset split on CelebA.
Pascal-Horse & Pascal-Aeroplane Pascal Part (Chen et al., 2014) provides part segmentation annotations of 20 object classes for images in Pascal VOC 2010. Following other work on part segmentation (Tsogkas al., 2015; Zhao et al., 2019; Tritrong et al., 2021), we merge the fine-grained part classes into 6 classes for horse and aeroplane. The merging protocol is presented in Table A.2. We crop the image patches that contain the object of interest according to bounding box annotations. Concretely, discard bounding boxes with IoU with other boxes smaller than 0.05 to ensure single object appears in a single image patch. The patches that have any side less a certain number of pixels (32 for horse; 50 for aeroplane) are also abandoned. The crop regions extended from bounding boxes to square boxes, and the regions outside images are padded with zeros. These processed patches are finally resized to 256 x 256 resolution serve the training and evaluation process. For horse and aeroplane, We further split the patches in official VOC 2010 t train aiinnnn split in training and validation set, and take the patches in official VOC 2010 val split test set. This procedure finally provides 180 images as the training set (i.e. labelled set), 33 images as a validation set, 223 images as the test set in Horse, 180 images as labelled set, 78 images as the validation set and 266 images as the test set in Pascal-Aeroplane. As unlabeled images, we centre-crop the 200,000 images in the large-scale LSUN (Yu et al., 2015) archive and resize them to 256 256 resolution for both horse and aeroplane.
Pascal-Horse
Pascal-Acroplie
6 classes background head
fine- tine-graine classes
classes
fine grained classes background
background
background
head, leye, reye, lear, rear, muzzle torso
body
body
torso
ster,, tail
legs
Ifho, rfho, Ibho, rbho, Ifuleg, Iflleg, rfuleg neck
wing engine engine {d} wheel
lwing, rwing
neck
tail
tail
wheel,[)
Table A.2: Protocol for merging part classes on Pascal Part.
CGPart As a part segmentation dataset composed of 3D CAD models from vehicle categories with 3D part manual annotations, CGPart (Liu et al., 2021) manage to render large amount of image dataset with part mask labels by adopting blender, a well-known graphical software. In this paper, we select category car as the training and evaluation dataset and make moderate modifications upon the rendering pipeline. Parts are merged into 15 classes, where excessively fine-graine rreegrained parts that seldom emerge combined to avoid sample imbalance. Part merging protocol is presented in Table A.3. Viewpoints are sampled around the object with azimuth ranging in full 360 Elevation angle is varied from (I to 80', yet higher sampling probability is assigned to interval (0%).. The distance
14

---

Published as a conference paper at ICLR 2022
from the object to the viewpoint is carefully set to be located the centre and fully in frame. Random colour is painted upon the object, and random texture sampled from COCO dataset (Lin et al., 2014), and Pixar-One- Twenty-Eight (Sisson, 2018) is rendered on the ground and the surroundings. 4, 000 images of 512 x 384 are rendered as a full train dataset. When training, only 1, 000 them are randomly sampled as official trainset. The original val split of CGPart is adopted as test dataset, consisting of 40 images. Test masks are merged under the same merging protocol in the inference and evaluation phases.
CGPart-Car
15 classes background back.bumper car. body
fine-grained classes
background
bumper
frame, right frame
door
back_left_door, back_right.doom feont.left.door, front_right_doo front bumper
front bumper
head_light left light, right hood back license plate, front license.plaa left mirror, right mirror roof
hood
aicence_plate
mirror roof
light
left light, sail. trunck
trunck
wheel
wheel, back_right.whede leeeell front.
window
back_left window, back right window, front left, window, front fronttright left- window.. qught.quarter.wing windoww
windshield
hack_windshield, frodsslwinshieldd
Table A.3: Protocol for merging part classes CGPart.
B IMPLEMENTATION DETAILS
Gradient matching loss Following DC (Zhao et al., 2021), the distance between gradients is measured with cosine similarity. The computation of gradient matching is quite similar as in DC. Here, we re-state this procedure for clarity. In particular, considering multi-layer neural network Sa that is parameterized with the gradient matching loss is computed as an average over layerwise losses as DV VoLg) = + E d(Vaco 6 Li, Vatn Cg), where i denotes the layer index, L denotes the number of layers for gradient matching and
N Σ j=1
A B A
d(A, B)
(12)
B
where A, and B, are flattened gradient vectors for each neural node j. The minor difference of our practice compared DC is that average the layerwise gradient distances whereas DC sum them. It is easier to handle numerical issues using our practice in our problem. In terms of gradient matching for normalization layers such as Batch Normalization (BN), we follow the practice of DC ignore the gradient matching of learnable parameters in BN. The BN layer is set as train mode during gradient matching. We find it works well.
Pre-trained GAN We use StyleGAN family (Karras et al., 2019; 2020b;a) as our pre-trained GANs. We either train GANs by ourselves for each dataset or use publicly available models. The training configurations links of the pre-trained GANs are listed in Table B.4. For GANs trained by ourselves, we use the checkpoints with the lowest FID in the historical ones.
Annotator structure An annotator is neural network that takes the generator features and output segmentation masks as input. DataselGAN (Zhang et al., 2021) and RepurposeGAN (Tritrong et al., 2021) first upsample feature maps to full resolution (resolution of output image) and then concatenate
15

---

Published as a conference paper at ICLR 2022
Dataset
Version
Output resolution
Training configurations links
CelebA
StyleGAN2-ADA
256 x 256
cfg=paper256 -mizzor-Irue
Pascal-Horse
StyleGAN2
256 x 256
aiylegnn2hoore-config-gpp
Puscal-Aeroplane
StyleGAN2
256 x 256
cfg=stylegan2 --aug=noaug
Cat-16
StyleGAN
256 x 256
available in DataserGAN repo
Cat-16
StyleGAN2
256 x 256
Jumcaz220k-puper256-aaaaapppp
Face-34
StyleGAN
512 512
available DatasetGAN repo
Face-34
StyleGAN2-ADA
512 512
-=cfg-paper51 --mi fror-lrue
Car-20
StyleGAN
512 512
available in DatasetGAN repo
Car-20
StyleGAN2
512 x 512
ayllean2--car-confgg-..lll
Table B.4: Pretrained GANs. We use the source codes provided by Karras al. (2020a) to train Style- ohc For CelebA and Face-34 the StyleGAN-ADA is trained CelebAHQ-Mask 28k images at 256 256 and 512 x 512 resolution, respectively, For Pascal-Aeroplane, the StyleGAN2 is trained on 200k images from LSUN airplane.
GAN models: https: //ttubb. com/NViabs httm///aabb/styeeann22 attpsssslllaan--daaaaaaa
these feature maps along the channel. This procedure results highly high-dimensional feature vectors for every pixel. These features are fed into either multiple-layer perceptrons (MLPs) as in DatasetGAN (Zhang et al., 2021) or convolutional neural networks (CNNs) as (Tritrong et al., 2021). Nonetheless, this practice consumes a lot of GPU memory and run painfully slow even high-end modern GPUs. That is also why DatasetGAN cannot consume image feature vectors in batch during training.
Therefore, to make training annotators more efficient, as commonly done in segmentation or detection network architectures, fuse the multi-scale feature maps with feature pyramid network (FPN) (Lin et al., 2017) structure and decode the fused features into segmentation masks with consecutive convolutional layers. This FPN structure saves lot of GPU memories and computation, which allows us to consume multiple full images in one mini-batch during training. An illustration of our annotator structure presented in Figure B.I. For generator features that fed into annotator, use outputs of all convolutional layers in StyleGAN as in DataserGAN (Zhang et al., 2021) and use outputs of all synthesis blocks, each of which typically contains two consecutive convolutional layers. in StyleGAN2.
Generator
1xd conv
1x1 conv
1x 0000
1xl comV
1xl canv
3x canv
3x3 canv
3x3 conv
100 +
Annotator
Figure B.I: Annotator architecture.
16

---

Published as a conference paper at ICLR 2022
Segmentation network structures In our experiments, we employ two prevalent segmentation network structures: U-Net (Ronneberger et al., 2015) and DeepLabv3 (Chen et al., 2017) with 101 pretrained on ImageNet as backbone. For U-Net, reference the implementation from https: / / sigithi thub. com mi 1 es a PPytorch--Dae UNet and train it from scratch on target datasets. For DeepLabv33 experiments on CelebA, Pascal-Horse, and Pascal- Pascal-Acroplane, we report the results (Figure 2) using DeepLabv3 segmentation networks.
the the built-in implementation in PyTTrcc library'. For all the
Training details of our method We use pixelwise cross-entropy loss as f((:.) for training segmen- tation network and computing gradients. The annotator and the segmentation network are optimized with SGD optimizer with learning rate 0.001 and momentum 0.9. By default, we jointly train an annotator and a segmentation network with = 1 and batch size for 150, 000 steps.
Training details of competing semi-supervised methods For MT, GCT and AdvSSL, we train the networks for 10, 000 iterations with batch size 16 for labelled data and 8 for unlabeled data. The learning rate is decayed with a power of 0.9 every iteration. During the training process, the input images are randomly resized to a scale in [160, 640) and cropped to 256 x 256. We use the default settings for the rest of the srpee-prrameters in PixelSSL.
C FURTHER ANALYSIS
C.I COMPARISON TO OTHER BASELINES
Inversion method As discussed in Section 1, the "inversion method" is straightforward way to utilize existing labelled images to train annotators in the supervised learning manner. We evaluate the performance of the inversion method and compare it to our method on Cat-16, Face-34, and Car-20 using StyleGAN2 as the generator. In particular, inversion baseline first projects images into GAN latent space using the projection method provided by Karras al. (2020b). Then the latent style codes are forwarded to the generator to acquire generator features. Finally, it trains the annotator using the generator features and ground truth masks.
Table C.5 shows the quantitative results and Figure C.2 presents examples of reconstruction quality. Despite its accurate image reconstruction, the inversion method fails to combat our method and produces lower-quality automatic labels than ours (see Figure C.3). We believe it is non-trivial to inverse the GAN generation process to acquire features for specific images. Even though the reconstruction of images is satisfying, one can still recover inaccurate generator features, leading to degraded annotator learning.
Methods
G
S
Cat-16
Face-34
Car-20
StyleGAN2
Deepl.ab
33.56 0.17
55.10
0.39
61.21 2.07
Ours
StyleGAN2
U-Net
31.90 = 0.75
53.58 = 0.45
58.30 + 2.64
StyleGAN2
DeepLab
15.14 = 0.27
47.87 = 0.86
52.18 + 2.31
Inversion method
StyleGAN2
U-Net
12.78 = 0.42
48.57 = 0.38
52.41 + 1.69
Table C.5: Comparison of our methods to inversion method.
Pseudo-laheling method Another way to train annotators using existing labeled images in su- pervised learning manner is pseudo-labeling method, which uses a trained segmentation model to predict pscudo labels for the generated images. In particular, this method involves three stages. (i) A segmentation model is trained with the labeled dataset. (ii) The trained segmentation model used to produce pseudo labels for every random image generated from generator. (iii) The generator features and pseudo labels are used to train annotators in the supervised learning manner.
Another way to train annotators using existing labelled images in supervised learning manner is the pseudo-labe labellin method, which uses a trained segmentation model to predict pseudo labels for the generated images. In particular, this method involves three stages. (i) A segmentation model is
hhttee://ppporrrh..orgg///////pp wttprrhhhiiiiiiooeeeeeeeeeeee abantaaenetttllllleeeeeeee /
17

---

Published as a conference paper at ICLR 2022
Figure C.2: Examples of reconstruction quality. The first row shows target images and the second row shows the reconstructed images.
Automatic labels by inversion baseline
Automatic labels by
Figure C.3: Comparison of the automatic labels produced by inversion method v.s. ours.
trained with the labelled dataset. (ii) The trained segmentation model produces pseudo labels for every random image generated from the generator. (iii The generator features and pseudo labels used to train annotators in supervised learning.
Methods
G
S
Cat-16
Face-34
Car-20
StyleGAN2
DeepLab
33.56 = 0.17
55.10 + 0.39
61.21 + 2.07
Ours
StyleGAN2
U-Net
31.90 = 0.75
53.58 1 0.45
58.30 +: 2.64
StyleGAN2
DeepLab
20.21 + 0.65
42.15 + 0.68
12.37 +: 0.91
Pseudo-labeling method
StyleGAN2
U-Net
16.72 = 0.79
43.26 + 1.20
13.48 + 0.48
Table C.6: Comparison of our methods to pseudo-labeling method.
Our method with MAML One straightforward solution to Equation 6 is MAML-like algorithm, which unrolls the inner loop with multi-step stochastic gradient descent, reserves the computation graph, computes the meta loss, and finally backpropagates the gradients for updating the annotator. We refer to this method as "Ours-K M-tep---tt--MMMMM. and summarize this algorithm as Algorithm 2. Table C.7 shows the experimental comparison of gradient matching versus MAML. where we use = 0.1 in MAML for all experiments. The following conclusions can be drawn from these results. First, gradient matching generally leads to higher performance and obtains more consistent results across different datasets and different segmentation network architectures than MAML. Second, MAML probably requires multiple inner-loop steps (more than two) to match the performance of gradient matching, given that it fails to do so with two inner-loop steps. It is very inefficient and requires enormous computation resources. Furthermore, in practice, we observe that MAML appears unstable and sensitive hyperparameters (e.g. inner-loop learning rate) across different datasets and different network architectures, whereas gradient matching requires neither hyperparameters.
dataset-specific network-architectuee-se
18

---

Published as a conference paper at ICLR 2022
Algorithm 2: Learning to annotate with K-step MAMI.
Inputs
G
trained generator
Di
set labeled examples
w, Aw,
initial annotator parameters, annotator, and learning rate for annotator
0. So. initial segmentor parameters, segmentor, and learning rate for segmentor
T and
total number of optimization steps and batch size
and K
inner-loop learning rate inner-loop steps
for t <- to To do
// update annctator Aw
0' -
11 copy segmentor paramsters for simulated SGD // sample batch of synthetic data
Bg + {(x,, Au (h,)))
for -- 1 to K' do
V
Ver (x)),
compute gradients B // update segmentor parameters
0' + 0'
11 Vo Lg
end
B: - {x., y:) ~ 1
// sample a batch of labeled examples // compute meta loss // update annotator parameters
Li t Σi= [x,)(x)),)))
T/w VuLi
// update segmenter Sa
11
B, + A.,
// sample a batch of syathetic data // compute loss syathetic data // update segmentor parameters
Ly B E (S
13
0- 0 VoLo
14
end
Output :annotator A and segmentor Sg
Cat- 16
Face-34
Car-20
Methods
G
S
annotations: 30
# annotations: 16
annotations: 16 # classes: 20
classes: 16
# classes: 16
StyleGAN
DeepLab
33.89 + 0.43
52.58 + 0.61
63.55 + 2.25
StyleGAN
U-Net
32.64 1 0.74
53.69 1 0.54
60.45 1 2.42
Ours-GM
StyleGAN2
DeepLab
33.56 + 0.17
55.10 + 0.39
61.21 + 2.07
StyleGAN2
U-Net
31.90 + 0.75
53.58 + 0.45
58.30 2.64
StyleGAN
DeepLab
32.49 1 0.43
35.24 0.20
54.74 2.67
StyleGAN
U-Net
16.70 + 0.43
23.21 0.07
29.26 0.72
Ours-I-step-MAMI.
StyleGAN2
DeepLab
30.30
33.82 0.24
54.42 3.11
StyleGAN2
U-Net
22.78 1: 0.65
32.64 1 0.18
26.59 + 0.60
StyleGAN
Deepl.al Lab
32.98 + 0.50
39.73 + 0.13
58.22 + 2.17
Ours-2-stp--AAMI
StyleGAN2
DeepLab
29.66 + 0.69
39.04 + 0.23
56.47 = 2.32
Table C.7: Comparisons of our methods using gradient matching (GM) versus MAML Car-20, Cat-16, Face-34 with different network architectures. The performance evaluated with mean intersection over union (mloU(%)) across all part classes plus background class.
C.2 THE IMPACT OF SEGMENTATION NETWORK STATES
We study the impact of segmentation network states, i.e. segmentation parameters, in the alternate learning algorithm. In particular, we compare the training the annotators' training convergence and the performance when different series of segmentation network states are employed for gradient matching. Figure C.4 & Figure C.5 shows the evolution of annotator performance during training process, where "baseline" denotes the default setting and the following settings are further compared. (i) "random re-init": the segmentation network parameters are randomly re- i--ntiiallzed rather than learned from generated images. (ii) ipre-trained"; the segmentation network is pre- pre-trained to acquire acceptable performance (~ 40.7% mIoU evaluated on the test set). (iii) "scratch": the segmentation network Labv3 network with randomly initial backbone rather than pre-trained ImageNet.
19

---

Published as a conference paper at ICLR 2022
(iv) "random init & fixed" and "pre-trained & fixed"; the segmentation network parameters, either randomly initialized or pre- mre-traned, are fixed, respectively.
First, the "fixed" segmentation network results in gradient matching problem single segmenta- tion network state, leading to an annotator with inferior performance than the default setting. This result suggests that matching gradients for various segmentation network states is vital learning annotators. Second, even though requiring an annotator to produce labels crucial, random re-initialization strategy does work well (see results of "random re-init" v.s. "baseline"). Moreover, the results of "pre-trained & fixed" are better than "random init & fixed", which suggest that well-performed segmentation network provides more informative gradients such that by gradient matching. the annotator can be more effectively learned. Third, matching gradients of pre-trained segmentation networks accelerates the learning of annotators (see results of "baseline" v.s. "pre-trained" and "baseline" v.s. "scratch"), suggesting a well-performed segmentation network provides more informative gradients.
baseline
trained fixed rinlcen & fixol
pre-tmined rundoen re-inii
haseline
70
50
40
20
10
20
60
100
120
140
20
60
100
120
140
steps (x10
steps (x
Figure C.4: Comparison of the training con- vergence under different settings of sampling segmentation network states. Annotator perfor- mance (mloU (%) is evaluated on training set of Car-20. Segmentation network is U-Net.
Figure C.5: Comparison of the training conver- gence of segmentation network with pretrained backbone ("baseline") versus segmentation network with randomly initialized backbone ("scratch"). Annotator performance (mloU (%) evaluated on the training set of Car-20. The segmentation network is DeepLabv3.
C.3 TRADE OFF TRAINING EFFICIENCY WITH PERFORMANCE
Gradient matching partial network parameters We present additional results in Table C.8. U-Net takes as input an image and first produce a feature map at multiple downsampled scales with consecutive convolutional and pooling layers. These feature maps are then processed by consecutive convolutional and upsampling layers to be mapped back to the original input scale. Notably, skip connection employed connect the multi-scale feature maps along the downsampling and upsampling paths. We consider the following particular group of layers in U-Net: (i) Up v.s. Down, (ii) Input v.s. Output, and (iii) U1 v.s. U2. Up and Down denote the convolutional layers along the upsampling and downsampling path, respectively. Input and Output denote the input convolutional layer and the output convolutional layer, respectively. U1 and U2 denote the convolutional layers that process feature maps at 1/1 and 1/2 scales, respectively. The results show that matching gradients of layers near the output end more effective than doing so at layers near the input end.
20

---

Published as a conference paper at ICLR 2022
Blocks
All
Up
Down
Input
Output
U1
U2
Time / step
1x
0.63
0.73x
0.73x
0.34x
0.71
0,63
31.90
30.12
14.42
12.40
25.41
30.18
31.22
mloU (%)
#0.75
10.43
+0.23
10.40
:0.77
10.79
10.48
Table C.8: Ablation study w.r.t. blocks for matching gradients on U-Net. Evaluation is performed on Cat-16, where StyleGAN2 is the generator. "Time ssep"" is measured the ratio the case of default setting. denotes the default setting.
Input resolution to segmentation network Another way to improve the training efficiency is to reduce the input resolution to the segmentation network. Figure C.6 and Table C.9 show that reducing the resolution improves the training speed but compromises the annotator performance and downstream segmentation network performance. Therefore, in practice, the input resolution to the segmentation network can be tuned to trade off the training efficiency versus effectiveness.
60
50
40
30
20
128x128-train
256x256-train
10
512x51 2-train
128x128-val
256x2566val
512x51 2-val
0
20
40
60
80
100
120
140
stops (x10²)
Figure C.6: The evolution of annotator performance(mloU (%) evaluated on training and validation set during the training process on Face-34.
Resolution
512 x 512 256 256
128 x 128
Time step
1x
0.45x
0.37x
mloU (%)
51.93 + 0.28
50.71 = 0.39
45.53 + 0.31
Table C.9: Comparisons of the training efficiency (time step) and downstream segmentation (U-Net) performance (mloU (%) w.r.t different input resolution to segmentation network on Face-34. "Time step" is measured ratio with respect the default setting. * denotes the default setting.
C.4 THE IMPACT OF GENERATOR
As our segmentation network is trained only on the synthetic images and labels (Algorithm Line 9~111), the segmentation performance is inevitably affected by the quality of pre-trained GANs. Table C010 confirms this point and shows that premature GANs ones with high FID score generally leads to lower segmentation performance.
21

---

Published as a conference paper at ICLR 2022
CelebA
Pascal- Pascal-Aeroplane
FID
10.21
6.80
5.12
4.29
11.74
7.78
6.25
5.04
FG-mloU (%)
1
73.92
76.61
77.48
78.07
21.69
36.01
38.81
41.21
Table C.10: The segmentation performance with respect to generators of different quality (indicated by FID) on CelebA and Pascal-Aeroplane.
C.5 DISCUSSION ON LARGE-SCAME SETTING
As shown in Fig. 2, the performance our method begins to saturate and even be beaten by other semi-supervised segmentation methods when the number of labels grows large. To investigate the performance of our method under a large-scale setting, we further run experiments with more unlabeled and labelled data and present the results in Table C. It shows that the performance of our method even slightly drops when the number of unlabeled data or the number labels increases and is significantly beaten by supervised learning. We hypothesize the reason to be that method seriously limited by the performance of GANs, which still struggle to fit large-scale datasets. In contrast, supervised learning becomes stronger under large-scale setting since more labels available. How to address this issue requires further research in the future.
Method
unlabeled data
labels
FG-mloU (%)
Supervised learning
29,000
84.32
Ours
CelebA (29,000)
1,500
77.90
Ours
CelebA-traie (29,000)
29,000
76.03
Ours
CelebA-train FFH (99,000)
29,000
75.00
Table C.11: Experiments on human face part segmentation under large-scale setting.
C.6 COMPARISON TO DATASETGAN IN OTHER ASPECTS
Real images v.s. synthetic images as labeled data As our method removes the necessity of labelling synthetic images, we investigate if replacing the labelled synthetic images with real ones could improve the segmentation performance. Table C.12 shows an evaluation
CelebA-test at 512x512 resolution across 8 face classes. DatasetGAN (Zhang et al., 2021), which uses 16 annotated synthetic images for training, achieves 70.01 mloU performance. Our method achieves bit higher performance when using the exactly same synthetic images as DatasetGAN. Moreover, the performance of method be further improved with around 6.7% mloU by replacing the labelled synthetic images with real ones in CelebA-trai This result suggests that the capability of using real labelled data exhibits the superiority of our method over DatasetGAN.
Labeled data CelebA-lest@51244
DatasetGAN (Zhang et al., 2021)1
Syathetic
70.01
Synthetic
72.55
Ours
Real
79.25
Table C.12:
Comparison of the performance when using synthetic images versus real images as labeled data. t: Results taken from Zhang al. (2021).
Our method with more labeled data on Car-20 Considering our method does not match the performance of DatasetGAN on Car-20, we further investigate if this gap can be narrowed with the help of more labelled data. Table C.13 presents the performance of our method when more labelled data is used on Car-20. It shows that with an increased number of labelled data, the performance of method gradually approaches that of DatasetGAN..
D QUALITATIVE RESULTS
22

---

Published as a conference paper at ICLR 2022
DatasetGAN
Ours
G
S
# annotations: 16
annotations: 16
annotations: 25
# annotations: 33
StyleGAN
DeepLab U-Net
67.53 + 2.58
63.55 = 2.25
64.68 + 2.53
64.88 + 2.52
StyleGAN
66.27 + 2.75
60.45 + 2.42
63.09 + 3.49
65.37 + 2.78
Table C.13: Performance of method with respect to different number of labeled images Car-20
dataset.
GT
2
10
labaled
30
150
1500
Figure D.7: Qualitative results of our method on CelebA. The segmentation results our models trained from 2, 10, and 30. 150, 1500 labeled examples are presented.
23

---

Published as a conference paper at ICLR 2022
Input
GT
2
10
labelted
30
100
180
Figure D.8: Qualitative results of our method on Pascal-Horse The segmentation results of our models trained from 10, and 30, 100, 180 labeled examples are presented.
Impux
GT
2
10
Number labeled
30
100
180
Figure D.9: Qualitative results of our method on Pascal-Aeroplane. The segmentation results of our models trained from 10, and 30, 100, 180 labeled examples are presented.
24

---

Published as a conference paper at ICLR 2022
DalasetGAN
Ours
Synthetic Data
Input
GT
Dataset GAN
Ours
Segmentation Results
Figure D.10: Qualitative comparison of our method DataselGAN Car-20
DatasetGAN
Ours
Synthetic Data
Inputt
GT
Dataset GAN
Ours
Segmentation Results
Figure D.11: Qualitative comparison of our method to DataselGAN Face-34.
25

---

Published as a conference paper at ICLR 2022
DatasetGAN
Ours
Synthetic Data
Input
GT
Dataset GAN
Ours
Segmentation Results
Figure D.12: Qualitative comparison of method to DatasetGAN on Cat- 16.
26


Reviews:
[{"review_id": "g1hhvDiMHVn", "reviewer": "Reviewer_muji", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Reducing the annotation cost is a very important thing. In this work, the authors provided a potential and promising way based on DatasetGAN. \n\nMajors. (1) Contributions. In the last paragraph in Section Introduction, the authors might not very clearly summarize their contributions. I suggest they could highlight the contributions one by one. (2) Method. During the generation process in eq. (1), the authors just did the random sample or sample the z according to the labeled data? Like the examples in Fig. 1, the generated data has similar attributes with the labeled data, such as the view. (3) Experiment. (a) Datasets. The authors use three datasets in the experimental part. Maybe, it might not enough to evaluate the effectiveness. DatasetGAN, the most related work, used more than 3 datasets. (b) Dataset setting. The authors need to explain the reason why they followed (Li et al. (2021), not DatasetGAN (Zhang et al., 2021). In table 1, they just copied the numbers from DatasetGAN (Zhang et al., 2021). I suggest the authors could add a summary of all test datasets as Table 1 in (Zhang et al., 2021). (c) Table 1, the authors also need to provide more details about how to get the results of DatasetGAN, which are not the same ones in DatasetGAN. \n\nMinors. (1) There exists some typos, such as “our method achieve same performance” -> ‘achieves’. (2) The citations of TL and SSL in Table 1 might be modified to the original work, and the authors might add some footnotes to indicate they just copied the numbers from (Zhang et al., 2021). \n", "summary_of_the_paper": "In this paper, the authors mainly propose a gradient-matching-based method for part segmentation to reduce the annotation cost. Based on the DatasetGAN, the proposed model also used the Style GAN family to generate high-quality images and remove the human annotations on a handful of synthesized images. Compared with semi-supervised learning methods, their proposed method achieves higher performance on three datasets. ", "summary_of_the_review": "The contributions might need to be clearly summarised and I vote 'the marginally below the acceptance threshold', based on the limited experiment and the unclear experimental settings. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "_VXGf3EX4zm", "reviewer": "Reviewer_318j", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The paper makes a valuable improvement over the DatasetGAN-like approaches by removing the need for annotating synthetic samples. \n- The motivations and the proposed solution is clearly explained in the paper. \n- The approach also appears to be an effective alternative for traditional semi-supervised learning. The experimental comparison to semi-supervised methods mostly appear to be convincing enough.\n- The paper also reports promising results in comparison to DatasetGAN, though the results are a bit mixed in terms of the performance.\n\nWeaknesses:\n- SemanticGAN's relevance is not clearly discussed, despite the experimental comparisons made in Section 4. In particular, it should be clarified in intro/related work, whether this work proposes an alternative approach to what SemanticGAN does, or whether they differ in terms of the data/annotation requirements. Please clarify.\n- The paper seems to have missed a closely related work:  Gradient Matching Generative Networks for Zero-Shot Learning (CVPR 2019). This CVPR’19 paper appears to introduce the “gradient matching loss\" (the same core formulation, and possibly its name as well) for training conditional generative models, as an efficient approximation to inner loop optimization for meta-learning. It trains synthetic train data-generating conditional generative models, such that when a classification model trained over the synthetic samples the resulting model’s loss is low on limited labelled samples, which is clearly closely. While the overall differences are significant enough (eg training conditional generator vs annotator only, segmentation vs zero-shot classification), the core idea, the use of Gradient Matching Loss and the overall goal of leveraging generative models for labelled data generation via meta-learning principles appear to be closely related, therefore, need to be discussed in the paper.\n- The experimental comparison to DatasetGAN is not fully clear (to me). DatasetGAN uses the annotated synthetic samples. Which labelled examples do you use for the proposed approach in these comparison experiments? It would have been valuable to do this in two ways: (i) train over the synthetic images used in DatasetGAN experiments, and (ii) train over alternative real training images.\n- The paper does not discuss whether it can be used on complex scenes, given that it is typically much more difficult to obtain unsupervised generative models of complex scenes.\n- It would have been very valuable to see an evaluation that shows the detailed analysis on the annotation cost vs final segmentation accuracy (in mIoU), when real images are labelled & used with the proposed approach, in comparison to labelling synthetic images to be used in DatasetGAN. Currently it is unclear how many more real images need to be annotated to match the DatasetGAN's sample generation performance, given that DatasetGAN performs better in several cases (Table 1).   \n- Fig 2: what happens if you train fine-tuned the model only on real annotated images, using the same training procedure, starting with pre-trained backbone, with optimized hyper-parameters?\n- How would it perform if you were to train the segmentation model solely on the available real samples, apply it to synthetic images for pseudo-labeling the pixels? This baseline also seems to be missing.\n- Please clarify at which stage(s) the synthetically generated images are being used in the semi-supervised experiments (Figure 2). Do you use the synthetic images as the unsupervised image set?\n- How would it perform if you were to actually do the inner loop optimization? While the computational advantages are clear, the difference is not empirically evaluated. In particular, how would the approach work with a 1st-order / kth-order MAML-based loss? \n", "summary_of_the_paper": "The ultimate goal of the paper is to train an unsupervised generative model, in a semi-supervised manner, as a synthetic training example generator for training segmentation models. Prior works (eg. DatasetGAN) show that one can label a few synthetic images from a pre-trained generative model, and train a segmentation model (called annotation model) over the intermediate representation produced by the generative model. Then, every time a sample is taken from the generative model, the corresponding labels can be produced by applying the annotation model to the corresponding intermediate representations. This provides a valuable way to produce synthetic labelled training examples. However, (some of) these existing approaches require annotating (a limited set of) generated samples for training the annotation model. This is problematic as one needs to re-label such images every time the generative model changes.  \n\nThis paper aims to address this problem by learning the annotation model over annotated real images instead of annotated synthetic images. To this end, the paper proposes a meta-learning approach. The main idea is to learn the annotation model such that when a (student) segmentation model is trained over the generative model outputs and their labels given by the annotation model, the resulting student model’s loss shall be low on real annotated images. This meta-learning idea is implemented as an approximation by using the Gradient Matching Loss.\n", "summary_of_the_review": "The paper proposes a reasonable and interesting meta-learning approach towards leveraging (pre-trained) generative models for constructing synthetic labelled training sample sets. The overall approach is interesting, the results are promising yet the paper (in text and in experimental analysis) seems to have various weakness, as discussed above. \n\nWhile I do *not* consider that the existence of all suggested experimental evaluations is a pre-condition for acceptance, the current experimental analysis seem to be underwhelming in several aspects. In addition, all the textual weaknesses & ambiguities definitely need to be addressed/clarified in the comments/revision.\n\n— Post revision/rebuttal update —\nI have gone over all the reviews, responses and the updates in the paper. The revision and responses address my concerns. The text and experimental evaluation have been improved on several aspects, though there is still room for improvement in experimental analysis. Overall, based on all positive improvements, I increase my recommendation score from 5 to 6.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "K1hwLF4A5cS", "reviewer": "Reviewer_y2iP", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "<Pros>\n\n\n- The authors argued that the proposed method has a few benefits over the prior arts. The limitations of the prior arts (DatasetGAN and RepurposeGAN) are discussed: 1) they are only applicable to generator-specific generated data, and 2) they need manual re-labeling when the generator is changed. In contrast, the proposed method does not have such problems.\n\n\n- In addition, this work does not require manual labeling of generated images, but requires manually labeled real data. \n\n\n- The learned annotator is student-agnostic\n\n\n- A naive counterpart could be the GANinversion baseline. It is well-compared in the supplementary.\n\n\n- The authors tackle a good scope range that has not been dealt with similar works, DatasetGAN and RepurposeGAN.\n\n\n- Diverse experimental setups\n\n\n\n<Cons>\n\n\n- The mathematical derivation and formulation are primarily borrowed from [Zhao et al., ICLR 2021]. In this work, the synthetic dataset in [Zhao et al., ICLR 2021] is parameterized by GAN and the learnable annotator. This is on the borderline.\n\n\n- While the authors argued that the proposed method significantly outperforms the semi-supervised competing methods, but the data regime is limited to low-data regimes. The effectiveness of this approach is unknown for the large semi-supervised data regimes (sufficient labeled data + very large unlabeled data).\n\n\n- While the proposed approach has merits, its practicality is questionable in that the approach assumes the high fidelity pre-trained GAN. It would be very interesting to see how much robust the proposed method is with immaturely pre-trained GANs (i.e., low-quality GAN).\n\n\n- The experiment settings are a bit biased to favorable settings to the proposed methods. Including the above concern, the experiment setups of DatasetGAN and RepurposeGAN are less considered. Are there specific reasons? Also, large-scale experiments and diverse balance setups could have been regarded.\n\n\n<Questions>\n\n- In the 1st paragraph of \"Comparison to DatasetGAN\", the authors mentioned that \"FID cannot depict the disentanglement of generator features.\" What is the relationship of the proposed approach with disentanglement?\n\n\n- The authors argued the non-necessity of labeling generated images as a benefit of the proposed method (including the one in \"Comparison to DatasetGAN\", Sec. 4). However, the proposed method requires the labeled real dataset as well. In order to validate the argument, the amount of the manual annotation cost (the number of labeled data) should be compared.\n\n\n\n<Minor comments>\n\n\n- The notation convention: In the paper, $\\mathbf{\\hat y}$ denotes manual annotation, while $\\mathbf{y}$ denotes the predicted output. Conventionally, it it more often the other way around.\n\n\n- $\\mathcal{D}_l$ is abused for both a real labeled dataset and a labeled dataset of generated images, which may lead to confusion. It would be clearer to distinguish the notation.\n\n- In Algorithm 1, the annotator and segmentor (called as student) are alternatingly optimized by each other. In this manner, the term student is misleading, since the role of the student is altered.\n\n- Typos and grammars: There are many grammar errors, especially with articles, singular and plural forms.\n", "summary_of_the_paper": "This work deals with the problem of training a part segmentation network by automatically synthesizing pairs of images and annotations. The authors propose a training method of an annotator given the well-pretrained generator model. The formulation is well-motivated and converged to the gradient matching loss. \n\nThis work extends [Zhao et al., ICLR 2021] by combining the idea of DatasetGAN and RepurposeGAN in an interesting way.\nThe proposed approach can favor the unlimited number of annotated data with only limited supervised real data + the pre-trained GAN.\n\nThe proposed method shows noticeable improvement over the other baselines and the competing methods in scarcely labeled data regimes (especially semi-supervised learning).\n", "summary_of_the_review": "The submission has a few limitations, including the limited demonstration of its effectiveness in low-data regimes and grammatic errors. However, this work extends [Zhao et al., ICLR 2021] by combining the idea of DatasetGAN and RepurposeGAN in an interesting way. The merits of this work seem to outweigh the demerits.\n\n===== After rebuttal ===== \nThis reviewer stays at the same rating because the merits of this work seem to outweigh the demerits even after the discussion phase.\nIt would have been good to show the limitation and break-down point analyses in the submission.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "HI5dVMRG868", "reviewer": "Reviewer_Z2yc", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper has the following Strengths.\n\n- The problem covered by the paper is important, and a good direction is proposed based on clear motivation.\n- Experiments were sufficiently conducted to verify the effectiveness of the proposed method.\n- The paper is written in an easy-to-understand manner through detailed formulas and figures.\n\nHowever, the following concerns exist.\n\n1. Comparison with the end-to-end gradient-based meta-learning approach\n    - In section 3.2, it is stated that the nested-loop optimization problem can be solved through existing end-to-end gradient-based meta-learning approaches. And the authors claim that those methods are complex and expensive, so they suggest an efficient gradient matching approach. However, I cannot find any experiment to show how efficient and effective the proposed method is compared to them, and this is my biggest concern.\n2. Images in batch to get gradient matching loss\n    - According to Algorithm 1, the images used to compute each gradient appear to be identical. (The indices of $x_i$ for $B_l$ and $B_g$ are the same.) However, in Figure 1, the two images for each gradient are different, causing confusion.\n3. Pascal-{Horse, Aeroplane} dataset\n    - For this paper, images of Horse and Aeroplane were selected from the Pascal Part segmentation dataset for the experiment. Is there any particular reason for choosing these two classes (among 20 classes)?\n", "summary_of_the_paper": "This paper proposes a method to solve the problem that generally requires large-scale labeled datasets to train deep learning models. The proposed method is designed to solve the nested-loop optimization problem (with an annotator that generates a label and a student network that predicts a label) based on a pre-trained GAN that generates images. In particular, the authors solve this problem through a gradient matching approach, and they claim it is much more efficient than the existing end-to-end gradient-based meta-learning approach. This method was applied to the semi-supervised part segmentation task, and its effectiveness was verified through various experiments.\n", "summary_of_the_review": "This paper proposes an intuitive method based on clear motivation. In addition, sufficient experiments have been conducted to prove it, and the paper is written in an easy-to-understand manner. However, one big concern (#1 above) remains, so I give my initial rating as 6.\n\n**[Comments After Author Response]** I appreciate the efforts of the authors for the detailed rebuttal. The authors have well addressed my concerns (other reviewers' as well), so I raise my rating to 8. I strongly recommend that the authors will reflect those contents in the final version.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work deals with training generators of aligned pairs of images and segmentation maps. It is based on the recent DatasetGAN approach, which generates images and maps, but requires human annotations on a handful of generated images. This paper is addressing this problem by learning the annotation model over annotated real images instead of generated ones. To this end, the paper proposes a meta-learning that uses the Gradient Matching Loss.

Overall, the rebuttal provides valuable insight and many issues raised by reviewers have been convincingly answered by the authors.
On the whole, the reviewers converged positively, the novelty and the interest of the proposal stand out clearly, and this despite the lack of very convincing experiments, at least before the rebuttal. Authors are strongly encouraged to take all comments into account for their final version.
Decision: Accept (Poster)