Year: 2021

Abstract:
The success of state-of-the-art deep neural networks heavily relies on the presence of large-scale labeled datasets, which are extremely expensive and time-consuming to annotate. This paper focuses on reducing the annotation cost of part segmentation by generating high-quality images with a pre-trained GAN and labeling the generated images with an automatic annotator. In particular, we formulate the annotator learning as the following learning-to-learn problem. Given a pre-trained GAN, the annotator learns to label object parts in a set of randomly generated images such that a part segmentation model trained on these synthetic images with automatic labels obtains superior performance evaluated on a small set of manually labeled images. We show that this nested-loop optimization problem can be reduced to a simple gradient matching problem, which is further efficiently solved with an iterative algorithm. As our method suits the semi-supervised learning setting, we evaluate our method on semi-supervised part segmentation tasks. Experiments demonstrate that our method is effective under a broad range of settings and significantly outperforms other semi-supervised competitors when the amount of labeled examples is extremely limited.

heading: 1 INTRODUCTION
text: In recent years, deep neural networks have shown a remarkable ability to learn complex visual concepts from large quantities of labeled data. However, collecting manual annotations for a growing body of visual concepts remains an extremely expensive practice, especially for pixelwise prediction tasks such as semantic segmentation. A common and effective way of reducing the dependence on manual labels is first learning representations from unlabeled data (Pathak et al., 2016; Noroozi & Favaro, 2016; Gidaris et al., 2018; He et al., 2020) and then deploying learned representations to a supervised learning task (Chen et al., 2020; Zhai et al., 2019). The insight of this paradigm is that some of the transferred representations are informative, easily identified, and disentangled, such that small amount of labeled data suffices to train the target task.
A promising direction for learning such representations is using generative models (Kingma & Welling, 2014; Goodfellow et al., 2014), since they attain the capability of synthesizing photorealistic images and disentangling factors of variation in dataset. Recent methods (Goetschalckx et al., 2019; Shen et al., 2020; Karras et al., 2019; Plumerault et al., 2020; Jahanian et al., 2020; Voynov & Babenko, 2020; Spingarn-Eliezer et al., 2021; Härkönen et al., 2020) show that certain factors such as object shape and position in the images synthesized by generative adversarial networks (GANs) (Goodfellow et al., 2014) can be individually controlled by manipulating latent features. Building on the success of powerful GAN models, Zhang et al. (2021) and Tritrong et al. (2021) demonstrate that feature maps in StyleGAN (Karras et al., 2019; 2020b;a) can be mapped to semantic segmentation masks or keypoint heatmaps through a shallow decoder, denoted as annotator in this paper. Remarkably the annotator can be trained after few synthesized images are labeled, and the joint generator and trained annotator can be used as an infinite labeled data generator. Nonetheless, labeling synthesized images with “human in the loop” has two shortcomings: (i) it can be difficult to label the generated images of low quality; (ii) it requires continual human effort whenever the generator is changed.
To mitigate these limitations, a natural idea is to learn annotator from a collection of image-label pairs. This is non-trivial because annotator only receives as input generator features which are lacked
in labeled image datasets. One straightforward way to address this issue is to inverse the generation process to obtain generator features for given images. However, as suggested by our experimental results, inversion methods can not ensure exact recovery of the generator features, leading to degraded performance of annotator. Another way to address this issue is to align the joint distribution of generated images and labels with that of real ones using adversarial learning as in SemanticGAN (Li et al., 2021). However, adversarial learning is notoriously unstable (Brock et al., 2018) and requires sufficient data to prevent discriminator from overfitting (Karras et al., 2020a).
In this paper, we formulate the annotator learning as a learning-to-learn problem – the annotator learns to label a set of randomly generated images such that a model, denoted as student, trained on these automatically labeled images obtains superior performance evaluated on a small set of manually labeled images. This problem involves solving a nested-loop optimization, where the inner loop optimizes the student and the outer loop optimizes the annotator based on the solution of optimization of student. In contrast to existing learning-to-learn algorithms (Li et al., 2019; Pham et al., 2021) that employs end-to-end gradient-based meta learning techniques, we reduce this complex and expensive optimization problem into a simple gradient matching problem and efficiently solve it with an iterative algorithm. We show that our method obtains performance comparable to the supervised learning counterparts (Zhang et al., 2021; Tritrong et al., 2021) yet overcomes their shortcomings such that labeled data within a broader range can be utilized. Experiments also show that our method achieves better performance than the naive inversion method and SemanticGAN (Li et al., 2021).
Our method requires a large quantity of unlabeled data for pre-training GANs and a relatively small amount of labeled data for training annotator, which suits a semi-supervised learning setting. Furthermore, considering that state-of-the-art unconditional GANs only produce appealing results on single-class images but struggle to model complex scenes, we evaluate our method on a variety of part segmentation datasets of particular classes such as human face, horse, aeroplane, car, and cat.
Our contribution can be summarized as follows. (i) We formulate the learning of annotations for GANgenerated images as a learning-to-learn problem and propose an algorithm based on gradient matching to solve it. As a consequence, a broad range of labeled data including real data, synthetic data, and even out-of-domain data is applicable. (ii) We empirically show that our method significantly outperforms other semi-supervised segmentation methods under the circumstances of extremely limited labeled data. 1

heading: 2 RELATED WORK
text: Semi-supervised learning Semi-supervised learning (SSL) augments the training of neural networks from a small amount of labeled data with a large-scale unlabeled dataset. A rich body of work regularize networks on unlabeled data with consistency regularization. In particular, networks are required to make consistent prediction over previous training iterations (Laine & Aila, 2017; Tarvainen & Valpola, 2017; Izmailov et al., 2018), noisy inputs (Miyato et al., 2018), or augmented inputs (Berthelot et al., 2019b;a; Sohn et al., 2020). This paradigm is also interpreted as a teacherand-student framework where teacher produces pseudo labels for unlabelled data to supervise the training of students. Pseudo labels can be enhanced with heuristics (Lee et al., 2013; Berthelot et al., 2019a) or optimized towards better generalization (Pham et al., 2021; Li et al., 2019). Apart from image classification tasks, the above ideas are also successfully applied to semantic segmentation tasks (Hung et al., 2018; Ouali et al., 2020; Ke et al., 2020; French et al., 2020). Our work can also be interpreted from teacher-and-student perspective. In contrast to the above methods that adopt homogeneous network structures for teacher and student, our work employ generative models as teacher and discriminative models as student.
Other SSL approaches explore the utility of unlabelled data from representation learning perspective. In particular, deep neural networks are first optimized on the unlabeled data with self-supervised learning tasks (Pathak et al., 2016; Noroozi & Favaro, 2016; Gidaris et al., 2018; He et al., 2020) to gain powerful and versatile representations, and then finetuned on the labeled data to perform target tasks (Chen et al., 2020; Zhai et al., 2019). In addition, efficient and transferable representations can also be learned with generative models that are typically trained without labels. To this end, one promising direction is to decode generative features into dense annotations such as segmentation
1Code will be made public.
masks and keypoint heatmap to facilitate SSL, as done in SemanticGAN (Li et al., 2021), DatasetGAN (Zhang et al., 2021), RepurposeGAN (Tritrong et al., 2021) and our work. However, our work differs from these works in the way that we formulate the annotator learning as a learning-to-learn problem which is solved with gradient matching. Particualrly, unlike SemanticGAN (Li et al., 2021) that learns annotator and data generator jointly, our method presumes a fixed pre-trained GAN and learns annotator only, which circumvents the complication of joint learning.
Semantic Part Segmentation Semantic part segmentation provides dense form of representations for both rigid and non-rigid objects, which is of great significance in tremendous computer-vision tasks such as human parsing (Dong et al., 2014; Nie et al., 2018; Fang et al., 2018; Gong et al., 2017), pose estimation (Yang & Ramanan, 2011; Shotton et al., 2011; Xia et al., 2017) and 3D object understanding (Yi et al., 2016; Song et al., 2017). The challenge of part segmentation arises from viewpoint changes, occlusion, and lack of clear boundaries for certain parts. Existing datasets manage to alleviate the problem by enrich the data and annotations. CelebAMask-HQ (Lee et al., 2020a) is a large-scale face image dataset annotated with masks of face components. Chen et al. (2014) provides additional part segmentation upon PASCAL VOC dataset (Everingham et al., 2010). Those datasets meet the demands of latest data-driven and deep neural network based method (Lee et al., 2020b; Wang et al., 2015; Wang & Yuille, 2015). However, a well-curated part segmentation dataset containing fine-grained pixel-wise annotation usually requires time-consuming manual labeling work, which holds back the development of this field. This problem also motivates the latest work (Zhang et al., 2021; Li et al., 2021; Tritrong et al., 2021) to leverage a generative model as an infinite data generator in a semi-supervised learning setting, which gain promising results.
Gradient matching Gradient matching is an important technique in meta learning (Li et al., 2018; Sariyildiz & Cinbis, 2019), which is shown to be effective in domain generalization (Li et al., 2018), zero-shot classification (Sariyildiz & Cinbis, 2019), and dataset condensation (Zhao et al., 2021; Zhao & Bilen, 2021) etc. Although the general principle and gradient matching loss function in our work resemble those in the above works, we are motivated to solve a totally different problem. In particular, Gradient Matching Network (GMN) (Sariyildiz & Cinbis, 2019) employs gradient matching to train a conditional generative model towards application on zero-shot classification. Dataset Condensation (DC) (Zhao et al., 2021; Zhao & Bilen, 2021) condenses a large-scale dataset into a small synthetic one with gradient matching. Our work shows that gradient matching is also an effective way to learn annotations of GAN-generated images from limited labeled examples. Moreover, in contrast to DC where the synthesized images are typically not realistic at all, our work are concerned about photo-realistic synthetic images.

heading: 3 METHOD

heading: 3.1 PRELIMINARY
text: Let G denote a generative model trained from a dataset Dtot without labels. For example, G can be trained via adversarial learning (Goodfellow et al., 2014) such that it can generate photo-realistic images. The generation process is formulated as
(h,x) = G(z), z ∼ Pz (1)
where z denotes a random variable drawn from a prior distribution Pz which can be a uniform or Gaussian distribution, x denotes the generated image, and h denotes the hidden features in G. An annotator Aω is constructed to decode h into labels
ŷ = Aω(h), (2)
where ω parameterizes the annotator. Let Dl denote a manually labeled dataset. Given a fixed G, our problem is to learn annotator Aω from Dl such that Aω can annotate images generated from G.
Revisiting supervised learning manner DatasetGAN (Zhang et al., 2021) and RepurposeGAN (Tritrong et al., 2021) train annotators in a supervised learning manner. First, a set of synthetic images {xi}Ni=1 generated from G are selected and annotated by humans and their generator features {hi}Ni=1 are reserved. These features and manual labels {yi}Ni=1 constitute the
labeled dataset Dl = {(xi,hi,yi)}Ni=1. Second, an annotator is trained by minimizing the loss computed between annotator prediction and ground truth labels,
ω = argmin ω E(·,h,y)∼Dlf(Aω(h),y), (3)
where f denotes the loss function for specific tasks, e.g. per-pixel cross-entropy function for segmentation tasks. This method, however, requires Dl to be a synthetic and generator-specific set, which has the following two defects. (i) Details might be illegible in synthetic images, which increases the difficulty of annotation. (ii) Data needs to be re-labeled whenever the generator is changed. Although projecting labeled images into GAN latent space can alleviate the above drawbacks, it leads to degenerate performance due to inexact inversion (see Section C.1 in the appendix).

heading: 3.2 LEARNING TO ANNOTATE WITH GRADIENT MATCHING
text: Learning to learn We formulate the problem of learning Aω as a learning-to-learn problem like Li et al. (2019); Pham et al. (2021). In particular, we introduce a student model Sθ which is a deep neural network parameterized with θ. It takes as input an image and outputs a segmentation mask. This student model learns the target task only from the automatically labeled generated images, which is formulated as
θ∗(ω) = argmin θ E(h,x)=G(z),z∼Pzf(Sθ(x), Aω(h))︸ ︷︷ ︸ Lg(θ,ω) . (4)
where f denotes the loss function for the target task. The optimal solution of Equation 4 depends on the ω, which is therefore denoted as θ∗(ω). The performance of learned student can be evaluated with loss on the labeled dataset Dl as
Ll(θ∗(ω)) = E(x,y)∈Dlf(Sθ∗(ω)(x),y). (5)
The aim of learning the annotator is to produce high-quality labels such that the learned student network achieve minimal loss on Dl, which is formulated as
min ω Ll(θ∗(ω)), where θ∗(ω) = argmin
θ Lg(θ, ω). (6)
This is a nested-loop optimization problem, where the inner loop optimizes student (Equation 4) and the outer loop optimizes annotator based on the solution of the inner loop. It is noteworthy that this optimization problem does not require manual labeling of generated images. To this end, any labeled images, including real labeled images, can serve as Dl. While this problem can be solved with MAML-like (Finn et al., 2017) approaches (Li et al., 2019; Pham et al., 2021), its solution involves expensive unrolling of the computational graph over multiple θ states, which is memory and computation intensive. Hence, we reduce it into a simple gradient matching problem as follows. See Section C.1 in the appendix for further comparison and discussion.
Gradient matching Inspired by Li et al. (2018), we reduce the nested-loop optimization problem (Equation 6) into a gradient matching problem. First, as commonly done in gradient-based meta learning approaches, the optimization of student (Equation 4) is approximated with one-step gradient descent, θ∗(ω) ≈ θ0 − η∇θLg(θ0, ω), (7) where η denotes the learning rate and θ0 denotes the initialized parameter. By plugging this equation into Ll(θ∗(ω)), unfolding it as Taylor series and omitting the higher order term, we have
Ll(θ∗(ω)) ≈ Ll(θ0 − η∇θLg(θ0, ω)) = Ll(θ0)− η∇>θ Ll(θ0)∇θLg(θ0, ω) + . . . ≈ Ll(θ0)− η∇>θ Ll(θ0)∇θLg(θ0, ω)
(8)
The last approximation is conditioned on ‖η∇θLg(θt, ω)‖ ≤ , where is a very small amount. By further omitting the constant term Ll(θ0) in Equation 8, we obtain the approximation of Equation 6 as
min ω −∇>θ Ll(θ0)∇θLg(θ0, ω) s.t. ‖η∇θLg(θ0, ω)‖ ≤ . (9)
This constrained optimization problem is further relaxed into the following unconstrained optimization problem
min ω −∇ > θ Ll(θ0)∇θLg(θ0, ω) ‖η∇θLg(θ0, ω)‖ , (10)
where to minimize this objective, one needs to simultaneously maximize the dot product between the gradients and minimize the norm of∇θLg(θ0, ω). Note that−∇ > θ Ll(θ0)∇θLg(θ0,ω) ‖η∇θLg(θ0,ω)‖ = ‖∇θLl(θ0)‖ η (1−
∇>θ Ll(θ0)∇θLg(θ0,ω) ‖∇θLl(θ0)‖‖∇θLg(θ0,ω)‖ )− ‖∇θLl(θ0)‖ η , and the constant term and constant coefficient can be omitted without changing the optimal solution. We re-write the learning objective of Equation 10 as
Lgm(θ0, ω) = D(∇θLl(θ0),∇θLg(θ0, ω)) = 1− ∇>θ Ll(θ0)∇θLg(θ0, ω) ‖∇θLl(θ0)‖‖∇θLg(θ0, ω)‖ , (11)
which computes the cosine distance between the gradients of student network on automatically labeled generated data and manually labeled data (see Section B in appendix for dealing with gradient matching in multi-layer neural networks). To this end, the nested-loop optimization (Equation 6) is reduced into a simple one that minimizes the gradient matching loss (Equation 11). Illustration of this procedure is presented in Figure 1.
Algorithm Note that the gradient matching problem learns an annotator for a particular network parameterized with θ0. However, a desirable annotator should produce universal automatic labels for training any student with previously unseen parameters. Hence, we present an alternate training algorithm of annotator and student in Algorithm 1, where the student is iteratively updated using the automatically labeled synthetic data and the annotator is iteratively updated with gradient matching. This algorithm allows the annotator to be optimized over various student states, and therefore leads to a student-agnostic annotator. The gradient matching problem for each student parameter is solved with K-step gradient descent and typically K = 1 works well in practice. Implementation details are available in the appendix (see Section B).

heading: 4 EXPERIMENTS

heading: 4.1 SETUP
text: Datasets We evaluate our method on six part segmentation datasets: CelebA, Pascal-Horse, Pascal-Aeroplane, Car-20, Cat-16, and Face-34. CelebA (Liu et al., 2015) is a large-scale human face dataset where 30,000 images are annotated with up to 19 part classes, also known as CelebAMaskHQ (Lee et al., 2020a). We consider a subset of 8 face classes and use the first 28,000 images as unlabeled set, the last 500 images as test set, and the rest 1500 images as labeled set, as in Li et al. (2021). We construct Pascal-Horse and Pascal-Aeroplane by taking images of horse and aeroplane from PascalPart (Chen et al., 2014) which provides detailed part segmentation annotations for images from Pascal VOC 2010 (Everingham et al.). The selected images are cropped according to bounding
Algorithm 1: Learning to annotate with gradient matching.

heading: Inputs :
text: G trained generator Dl set of labeled examples ω, Aω , ηa initial annotator parameters, annotator, and learning rate for annotator θ, Sθ , ηs initial segmentor parameters, segmentor, and learning rate for segmentor T , K, and B total number of optimization steps, interval of updating student, and batch size
1 for t← 1 to T do // update annotator Aω 2 Bl ← {(xi,yi) ∼ Dl}Bi=1 // sample a batch of labeled examples 3 ∇θLl ← 1B ∑B i=1∇θf(Sθ(xi),yi) // compute gradients on Bl 4 Bg ← {(xj , Aω(hj))}Bj=1 // sample a batch of synthetic data 5 ∇θLg ← 1B ∑B j=1∇θf(Sθ(xj), Aω(hj)) // compute gradients on Bg 6 Lgm ← D(∇θLg,∇θLl) // compute gradient matching loss 7 ω ← ω − ηa∇ωLgm // update annotator parameters
// update segmentor Sθ 8 if t mod K = 0 then 9 Bg ← {(xj , Aω(hj))}Bj=1 // sample a batch of synthetic data
10 Lg ← 1B ∑B j=1 f(Sθ(xj), Aω(hj)) // compute loss on synthetic data 11 θ ← θ − ηs∇θLg // update segmentor parameters 12 end 13 end
Output :annotator Aω and segmentor Sθ
box annotations. Full process details are available in the appendix. We finally obtain 180 training images, 34 validation images, and 225 test images to constitute Pascal-Horse, and 180 training images, 78 validation images, and 269 test images to constitute Pascal-Aeroplane. In Pascal-Horse and Pascal-Aeroplane, additional images from LSUN (Yu et al., 2015) are utilized as unlabeled set. To make a thorough comparison to DatasetGAN (Zhang et al., 2021), we adopt the same datasets released by Zhang et al. (2021): Car-20, Cat-16, and Face-34, which are manually annotated with 20, 16, and 34 part classes for car, cat, and human face, respectively. In these tree datasets, all the training images are synthetic images generated from pre-trained StyleGAN (Karras et al., 2019) while all the test images are real images. Car-20 contains 16 training images and 10 test images; Cat-16 contains 30 training images and 20 test images; Face-34 contains 16 training images and 20 test images.
Pre-trained generators We use models of StyleGAN family (Karras et al., 2019; 2020b;a) that are either trained by ourselves or publicly available. Details are provided in the appendix (see Section B).
Evaluation We use mean intersection over union (mIoU) to evaluate the performance of segmentation networks. On CelebA, Pascal-Horse, and Pascal-Aeroplane, we use validation set to select checkpoints and report mIoU across all foreground classes, denoted as “FG-mIoU”, on test set. On Car-20, Cat-16 and Face-34, we follow the setting of DatasetGAN (Zhang et al., 2021) and report the cross validation mIoU across all classes including background on test set.

heading: 4.2 SEMI-SUPERVISED PART SEGMENTATION
text: Since our method relaxes the restrictions on the source of labeled data, in this section, we show the effectiveness of our method under three different circumstances: real images as labeled data, synthetic images as labeled data, and out-of-domain images as labeled data.
Real images as labeled data Following SemanticGAN (Li et al., 2021), our methods are compared against SSL methods that have codes available 2, including Mean Teacher (MT) (Tarvainen & Valpola, 2017; Li et al., 2020), Guided Collaborative Training (GCT) (Ke et al., 2020), an adversarial-
2We adopt the implementation in https://github.com/ZHKKKe/PixelSSL. For these SSL methods, the real unlabeled images are used as unlabeled set.
learning-based semi-supervised segmentation method (AdvSSL) (Hung et al., 2018), as well as SemanticGAN (Li et al., 2021). We benchmark performances on CelebA, Pascal-Horse, and PascalAeroplane with respect to different amount of labeled data and present the results in Figure 2. Qualitative results are available in the appendix (see Figure D.7, D.8, D.9). Our method significantly outperforms other SSL methods when labeled data is extremely limited. When the number of labeled images decreases, the performance of our method drops mildly while performances of other methods decreases drastically. We attribute it to our utilization of the highly interpretable generator features. Notably, SemanticGAN (Li et al., 2021) also exploits the generative features to produce segmentation masks but its performance degrades faster than our methods as the number of labeled images decreases. We conjecture it is due to that the adversarial learning employed by SemanticGAN typically requires sufficient data to prevent discriminators from overfitting.
Synthetic images as labeled data We comprehensively compare our methods to a supervised learning counterpart, DatasetGAN (Zhang et al., 2021) on Car-20, Cat-16, and Face-34 with different network structures. Note that on these three datasets, we follow the same setting as in DatasetGAN: we use exactly the same synthetic images as training data. Quantitative results are presented in Table 1
and qualitative comparison is available in the appendix (see Figure D.12, D.11, D.10). First, our method achieves the same performance level with DatasetGAN on Face-34 and Cat-16. Second, as a merit of our method, we can upgrade the generator to StyleGAN2 (Karras et al., 2020b;a) without requiring extra human effort. The results show such update of generator brings modest improvement on Face-34 and Cat-16 and even a bit decrease on Car-20. This suggests that synthesis quality measured by metrics like FID can not precisely depict the disentanglement of generator features. Finally, our methods could adapt to different segmentation architectures such as DeepLabv3 (Chen et al., 2017) and U-Net (Ronneberger et al., 2015). Generally, DeepLabv3 would leads to slightly higher performance than U-Net.
We further use our trained annotator to generate synthetic segmentation datasets for training downstream segmentation networks. Table 1 shows the results when different source networks, which are used for training annotator with gradient matching, and different downstream networks are used. First, our learned synthetic datasets can be used to train downstream segmentation networks of different architecture that achieve good performances. This indicates that our learned synthetic dataset is not architecture-specific. Second, DeepLabv3 as source network generally leads to higher performances than U-Net does. This suggests that in our method the segmentation networks used for gradient matching has an impact on annotator learning.
Out-of-domain images as labeled data We further consider a challenge where out-of-domain images constitute labeled data. In particular, we render a bunch of car images from annotated 3D CAD models provided by CGPart3 (Liu et al., 2021) using graphic tools (e.g. blender). Since 3D models are annotated with parts, the ground truth part segmentation of these rendered images are available via 3D projection. These rendered images vary in viewpoints and textures and have significant domain gap compared to realistic car images due to inexact 3D models, unrealistic textures, unnatural lighting etc., raising challenges for learning annotators. Despite so, as presented in Figure 3, our method still learns reasonable annotators that produce quality segmentation labels for GAN generated images. It is noteworthy that it is nearly impossible for DatasetGAN (Zhang et al., 2021) and RepurposeGAN (Tritrong et al., 2021) to utilize such out-of-domain labeled data.

heading: 4.3 ABLATION STUDY
text: The frequency of updating student The results of an ablation study w.r.t. the interval of updating student network, K, is shown in Figure 4. Experiments are run on Face-34 and Cat-16 with DeepLabv3 as segmentation network and StyleGAN as generator such that pairs of generator features and ground truth segmentation are available to quantitatively measure the performance of annotator. It shows that the annotator learns significantly faster when the segmentation network is updated more frequently. From the visualization, it can be seen that very detailed parts, e.g. pupils in Face-34 and eyes in Cat-16, emerge sooner if segmentation network is updated more frequently. It can be concluded that it is more effective to match gradients for various student states than optimize gradient matching for single student state.
Gradient matching on partial student parameters As the implementation of gradient matching requires maintenance of computation graph of backpropagation through student networks, one may
3https://qliu24.github.io/cgpart/
concern the training efficiency for very deep neural networks. Here we present a simple strategy to trade off the training efficiency and overall performance: gradient matching can be done only on part of student parameters. An ablation study on Cat-16 using StyleGAN2 as generator and DeepLabv3 as student network is done and the results are shown in Table 2. We consider network blocks in DeepLabv3 with ResNet-101 as backbone from bottom to top: Res1, Res2, Res3, Res4 and Res5 that are residual blocks in the backbone, and Head that is the segmentation head. Results show that by skipping the bottom blocks for gradient matching, the training speed is improved while the segmentation performance is affected little. We conjecture the reasons can be two folds: (i) the backbone parameters are pre-trained such that little needs to be changed during training; (ii) the gradients accumulate more randomness as they backpropagate to the bottom blocks of deep neural networks. Therefore, in practice, we only match the gradients of Head in DeepLabv3 by default. Additional results are available in the appendix (see Section C.3).

heading: 5 CONCLUSION
text: We propose a gradient-matching-based method to learn annotator which is able to label generated images with part segmentation by decoding the generator features into segmentation masks. Unlike existing methods that require labeling generated images for training an annotator, our method allows usage of a broader range of labeled data including realistic images, synthetic images, and even rendered images from 3D CAD models. On the benchmark of semi-supervised part segmentation, our method significantly outperform other semi-supervised segmentation methods under the circumstances of extremely limited labeled data. The effectiveness of our method is validated on a variety of single-class datasets including well-aligned images, e.g. CelebA and Face-34, as well as images in the wild part of which contain cluttered backgrounds, e.g. Pascal-Horse, Pascal-Aeroplane, Car-20 and Cat-16. In terms of more complex scenes, discussion and investigation are further needed. With the rapid progress of generative modeling, it is promising to use more powerful generative models and explore more varieties of computer vision problems under different challenging settings in the future.

heading: REPRODUCIBILITY STATEMENT
text: We provide all the details about the datasets, network structure, and training curriculum in the appendix. Code to reproduce our main results will be made public.

heading: A DATASET DETAILS
text: CelebA Following Li et al. (2021), we only consider 8 part classes: background, ear, eye, eyebrow, skin, hair, mouth, and nose. The protocol for merging the 19 part classes into 8 classes is presented in Table A.1(Left). CelebAMask-HQ contains 30,000 annotated images in total. We split this dataset into unlabeled set, training set, validation set and test set as in Table A.1(Right).
Pascal-Horse & Pascal-Aeroplane Pascal Part (Chen et al., 2014) provides part segmentation annotations of 20 object classes for images in Pascal VOC 2010. Following other work on part segmentation (Tsogkas et al., 2015; Zhao et al., 2019; Tritrong et al., 2021), we merge the fine-grained part classes into 6 classes for horse and aeroplane. The merging protocol is presented in Table A.2. We crop the image patches that contain the object of interest according to bounding box annotations. Concretely, bounding boxes that have IoU greater than 0.05 with other boxes or empty annotations are discarded to ensure single object appears in single image patch. The patches that have any side less a certain number of pixels (32 for horse; 50 for aeroplane) are also abandoned. The crop regions are extended from bounding boxes to square boxes and the regions outside images are padded with zero. These processed patches are finally resized to 256× 256 resolution to serve the training and evaluation process. For horse and aeroplane, We further split the patches in official VOC 2010 train split in training and validation set, and take the patches in official VOC 2010 val split as test set. This procedure finally provides 180 images as training set (i.e. labeled set), 33 images as validation set, and 223 images as test set in Pascal-Horse, and 180 images as labeled set, 78 images as validation set and 266 images as test set in Pascal-Aeroplane. As unlabeled images, we center-crop the 200,000 images in the large-scale LSUN (Yu et al., 2015) archive and resize them to 256× 256 resolution for both horse and aeroplane.
CGPart As a part segmentation dataset composed of 3D CAD models from 5 vehicle categories with 3D part manual annotations, CGPart (Liu et al., 2021) manage to render a large amount of image dataset with part mask labels by adopting blender, a well-known graphical software. In this paper, we select category car as training and evaluation dataset and make moderate modifications upon the rendering pipeline. Parts are merged into 15 classes, where excessively fine-grained parts that seldom emerge are combined together to avoid sample imbalance. Part merging protocol is presented in Table A.3. Viewpoints are sampled around the object with azimuth ranged in full 360◦. Elevation angle is varied from 0◦ to 80◦, yet higher sampling probability is assigned to interval [0◦, 30◦]. The distance
from object to viewpoint is carefully set so that the object is located at center and full in frame. Random color is painted upon the object and random texture sampled from COCO dataset (Lin et al.) and Pixar-One-Twenty-Eight (Sisson, 2018) is rendered on the ground and the surroundings. 4, 000 images of 512× 384 are rendered as full train dataset. When training, only 1, 000 of them in total are randomly sampled as official trainset. The original val split of CGPart is adopted as test dataset, consisting of 40 images. Test masks are merged under the same merging protocol in inference and evaluation phases.
B IMPLEMENTATION DETAILS
Gradient matching loss Following DC (Zhao et al., 2021), the distance between gradients is measured with cosine similarity. The computation of gradient matching is quite similar as in DC. Here, we re-state this procedure for clarity. In particular, considering a multi-layer neural network Sθ that is parameterized with θ, the gradient matching loss is computed as an average over layerwise losses as D(∇θLl,∇θLg) = 1L ∑L i=1 d(∇θ(i)Ll,∇θ(i)Lg), where i denotes the layer index, L denotes the number of layers for gradient matching and
d(A,B) = N∑ j=1 ( 1− Aj ·Bj ‖Aj‖ · ‖Bj‖ ) (12)
where Aj and Bj are flattened gradient vectors for each neural node j. The minor difference of our practice compared DC is that we average the layerwise gradient distances whereas DC sum them. It is easier to handle numerical issues by using our practice in our problem. In terms of gradient matching for normalization layers such as Batch Normalization (BN), we follow the practice of DC to ignore the gradient matching of learnable parameters in BN. The BN layer is set as train mode during gradient matching. We find it works well.
Pre-trained GAN We use StyleGAN family (Karras et al., 2019; 2020b;a) as our pre-trained GANs. For each dataset, We either train GANs by ourselves or use publicly available models. The training configurations or links of the pre-trained GANs are listed in Table B.4. For GANs trained by ourselves, we use the checkpoints with lowest FID in the historical ones.
Annotator structure Annotator is a neural network that takes as input the generator features and output segmentation masks. DatasetGAN (Zhang et al., 2021) and RepurposeGAN (Tritrong et al., 2021) first upsample feature maps to full resolution (resolution of output image) and then concatenate
these feature maps along channel. This procedure results in extremely high-dimensional feature vectors for every pixel. These features are fed into either a pixelwise multiple layer perceptrons (MLPs) as in DatasetGAN (Zhang et al., 2021) or a convolutional neural networks (CNNs) as in (Tritrong et al., 2021). Nonetheless, this practice consumes a lot of GPU memory and run painfully slow even on high-end modern GPUs. That is also why DatasetGAN cannot consume all image feature vectors in a batch during training.
Therefore, to make training annotator more efficient, as commonly done in segmentation or detection network architectures, we fuse the multi-scale feature maps with a feature pyramid network (FPN) (Lin et al., 2017) structure and decode the fused features into segmentation masks with consecutive convolutional layers. This FPN structure saves a lot of GPU memories and computation, which allows us to consume multiple full images in one mini-batch during training. An illustration of our annotator structure is presented in Figure B.1. For generator features that are fed into annotator, we use outputs of all convolutional layers in StyleGAN as in DatasetGAN (Zhang et al., 2021) and use outputs of all synthesis blocks, each of which typically contains two consecutive convolutional layers, in StyleGAN2.
Segmentation network structures In our experiments, we employ two prevalent segmentation network structures: U-Net Ronneberger et al. (2015) and DeepLabv3 (Chen et al., 2017) with ResNet101 pretrained on ImageNet as backbone. For U-Net, we reference the implementation from and train it from scratch on target datasets. For DeepLabv3, we use the the built-in implementation in PyTorch library 4. For all the experiments on CelebA, Pascal-Horse, and Pascal-Aeroplane, we report the results (Figure 2) using DeepLabv3 as segmentation networks.
Training details of our method We use pixelwise cross entropy loss as f(·, ·) for training student network and computing gradients. Both the annotator and the student are optimized with SGD optimizer with learning rate 0.001 and momentum 0.9. By default, we jointly train an annotator and a student with K = 1 and batch size 2 for 150, 000 steps.
Training details of competing semi-supervised methods For MT, GCT and AdvSSL, we train the networks for 10, 000 iterations with batch size 16 for labeled data and 8 for unlabeled data. Learning rate is decayed with power 0.9 every iteration. During training process, the input images are randomly resized to a scale in [160, 640] and cropped to 256× 256. For the rest of the hyper-parameters, we use the default settings as in PixelSSL.

heading: C FURTHER ANALYSIS

heading: C.1 COMPARISON TO OTHER BASELINES
text: Inversion baseline One way to make the supervised learning of annotator be able to utilize existing labeled images is to (i) project images into GAN latent space to acquire generator features, (ii) use the joint generator features and ground truth labels to train an annotator. We call this method “inversion baseline”. We evaluate the performance of this method on Cat-16, Face-34, and Car-20 using StyleGAN2 as generator and make a comparison to our method in Table C.5. Concretely, we first use the projection method provided by Karras et al. (2020b) to compute the latent style codes that result in a reconstruction of labeled images. Second, the latent style codes are forwarded to the generator to acquire generator features. We present examples of such reconstruction in Figure C.2 which shows that the reconstruction is quite accurate. However, the performance of inversion baseline fails to combat our method and the automatic labels for a randomly generated image are of low quality (see Figure C.3). We argue that it is non-trivial to inverse the GAN generation process to acquire features for specific images. Even though the reconstruction of images is satisfying, one is still unable to exactly recover the generator features, which leads to an degraded learning of annotator.
4https://pytorch.org/hub/pytorch_vision_deeplabv3_resnet101/
Automatic labels by inversion baseline

heading: Automatic labels by ours
text: Pseudo-labeling baseline Another way to employ existing labeled images to train an annotator in supervised learning manner is to use a trained segmentation model to predict pseudo labels for the generated images. We refer to this method as “pseudo-labeling baseline”. In particular, this baseline method involves three stages: (i) a segmentation model is trained on the labeled data; (ii) the trained segmentation model is used to produce pseudo labels for every random image generated from generator, which further supervise the learning of an annotator; (iii) the joint generator and trained annotator are used for generating automatically labeled data which is further used for training a downstream segmentation network.
As shown in Table C.6, pseudo-labeling baseline performs badly, especially on Car-20, which can be accounted for the following reasons. Our method can be understood as distilling knowledge from a generative model to a discriminative model, whereas this pseudo-labeling baseline is more like the other way around – distilling knowledge from a discriminative model to a generative model. The highly interpretable generator features make our method possible to learn annotations from only a few labeled data. In contrast, the discriminative model can hardly learn to acquire valuable knowledge from extremely limited labeled data.
Our method with MAML One direct solution to Equation 6 is to use MAML-like algorithm, which unrolls the inner loop with multi-step stochastic gradient descent, reserves the computation graph, computes the meta loss, and finally backpropagates the gradients for updating the annotator. We refer to this baseline as “Ours-K-step-MAML” and summarize this algorithm as Algorithm 2. Table C.7 shows the experimental comparison of gradient matching versus MAML, where we use ηi = 0.1 in MAML for all experiments. The following conclusions can be drawn from these results. First, gradient matching generally leads to higher performance and obtains more consistent results across different datasets and different segmentation network architectures than MAML. Second, MAML probably requires multiple inner-loop steps (more than two) to match the performance of gradient matching, given that it fails to do so with two inner-loop steps. This is very inefficient and requires huge computation resources. Furthermore, in practice, we observe that MAML appears unstable and sensitive to hyperparameters (e.g. inner-loop learning rate) across different datasets and
different network architectures, whereas gradient matching nearly requires neither dataset-specific nor network-architecture-specific hyperparameters.
Algorithm 2: Learning to annotate with K-step MAML.

heading: Inputs :
text: G trained generator Dl set of labeled examples ω, Aω , ηω initial annotator parameters, annotator, and learning rate for annotator θ, Sθ , ηθ initial segmentor parameters, student, and learning rate for student T and B total number of optimization steps and batch size ηi and K inner-loop learning rate and inner-loop steps
1 for t← 1 to T do // update annotator Aω 2 θ′ ← θ // copy segmentor parameters for simulated SGD 3 Bg ← {(xj , Aω(hj))}Bj=1 // sample a batch of synthetic data 4 for k ← 1 to K do 5 ∇θ′Lg ← 1B ∑B j=1∇θ′f(Sθ′(xj), Aω(hj)) // compute gradients on Bg 6 θ′ ← θ′ − ηi∇θ′Lg // update segmentor parameters 7 end 8 Bl ← {(xi,yi) ∼ Dl}Bi=1 // sample a batch of labeled examples 9 Ll ← 1B ∑B i=1 f(Sθ(xi),yi) // compute meta loss
10 ω ← ω − ηω∇ωLl // update annotator parameters
// update segmentor Sθ 11 Bg ← {(xj , Aω(hj))}Bj=1 // sample a batch of synthetic data 12 Lg ← 1B ∑B j=1 f(Sθ(xj), Aω(hj)) // compute loss on synthetic data 13 θ ← θ − ηθ∇θLg // update segmentor parameters 14 end
Output :annotator Aω and segmentor Sθ

heading: C.2 THE IMPACT OF STUDENT STATES
text: As we present an alternate learning algorithm between annotator and student, we study the impact of updating student states. In particular, we investigate the training convergence and overall performance of annotator when different series of student states are employed for gradient matching. We present in Figure C.4 & Figure C.5 the evolution of annotator performance evaluated on training set during training process, where “baseline” denotes the default setting and the following settings are further compared. (i) The student parameters are randomly re-initialized rather than learned from generated
images, denoted as “random re-init”. (ii) The student is pretrained to acquire a fair performance (∼ 40.7% mIoU evaluated on test set), denoted as “pre-trained”. (iii) The student is a DeepLabv3 network that has backbone randomly initialized rather than pretrained on ImageNet, denoted as “scratch”. (iv) The student parameters, either randomly initialized or pre-trained, are fixed, denoted as “random init & fixed” and “pre-trained & fixed”, respectively.
First, “Fixed” students result in a gradient matching problem on single student state, which learns annotator with very poor performance compared to the default setting. This result suggests that matching gradients for a series of student states is vital to learning annotator. Second, even though it is crucial to require annotator to produce student-agnostic labels, a randomly re-initialization strategy does not work well (see results of “random re-init” v.s. “baseline”). Moreover, the results of “pre-trained & fixed” is better than “random init & fixed”, which suggest that a well-performed student provides more informative gradients such that by gradient matching the annotator can be more effectively learned. Third, the conclusion that a well-performed student provides more informative gradients can be further verified by the fact that matching gradients of pre-trained students accelerates the learning of annotator (see results of “baseline” v.s. “pre-trained” and “baseline” v.s. “scratch”).

heading: C.3 TRADE OFF TRAINING EFFICIENCY WITH PERFORMANCE
text: Gradient matching on partial network parameters We present additional results on in Table C.8. U-Net takes as input an image and first produce feature map at multiple downsampled scales with consecutive convolutional and pooling layers. These feature maps are then processed by consecutive convolutional and upsampling layers to be mapped back to original input scale. Notably, skip connection is employed to connect the multi-scale feature maps along the downsampling path and upsampling path. We consider the following particular group of layers in U-Net: (i) Up v.s. Down, (ii) Input v.s. Output, and (iii) U1 v.s. U2. Up and Down denote the convolutional layers along the upsampling and downsampling path, respectively. Input and Output denote the input convolutional layer and the output convolutional layer, respectively. U1 and U2 denote the convolutional layers that process feature maps at 1/1 and 1/2 scales, respectively. From the results
we can see that matching gradients of layers near the output end is more effective than doing so at layers near the input end.
Resolution of input to student network Another way to improve the training efficiency is to reduce the resolution of input to student network. Note that the resize operation can be implemented with interpolation which is differentiable. We show in Figure C.6 and Table C.9 that by reducing the resolution, the training speed improves significantly, but the annotator performance and downstream segmentation networks compromises. Therefore, in practice, the student resolution can be tuned by trading off the training efficiency and effectiveness.

heading: C.4 THE IMPACT OF GENERATOR
text: As our segmentation network is trained only on the generated images (Algorithm 1 Line 9 11) and a very shallow network is employed to convert generator features into annotations, the segmentation performance is inevitably affected by the quality of pre-trained GANs. which suggests that the quality of generative features significantly affect the quality of annotations. Table C.10 shows that premature GANs – ones with high FID score – generally leads to lower segmentation performance.

heading: C.5 COMPARISON TO DATASETGAN IN OTHER ASPECTS
text: Real images v.s. synthetic images as labeled data Since our method removes the necessity of labeling synthetic images, we study if replacing the labeled synthetic images with real ones could
improve the segmentation performance. Table C.11 shows an evaluation on CelebA-test at 512×512 resolution across 8 face classes. DatasetGAN, which uses 16 annotated synthetic images for training, achieves 70.01 mIoU performance. Our method achieves a bit higher performance when using the exactly same synthetic images as DatasetGAN. Moreover, the performance of our method can be further improved with around 6.7% mIoU by replacing the labeled synthetic images with real ones in CelebA-train. This suggests that the capability of using real labeled data exhibits the superiority of our method over DatasetGAN.
Our method with more labeled data on Car-20 Considering our method does not match the performance of DatasetGAN on Car-20, we further investigate if this gap can be narrowed with the help of more labeled data. Table C.12 presents the performance of our method when more labeled data is used on Car-20. It shows that with increased number of labeled data, the performance of our method gradually approaches that of DatasetGAN.
D QUALITATIVE RESULTS

Review summaries:
In this paper, the authors mainly propose a gradient-matching-based method for part segmentation to reduce the annotation cost. Based on the DatasetGAN, the proposed model also used the Style GAN family to generate high-quality images and remove the human annotations on a handful of synthesized images. Compared with semi-supervised learning methods, their proposed method achieves higher performance on three datasets. 

This paper proposes a method to solve the problem that generally requires large-scale labeled datasets to train deep learning models. The proposed method is designed to solve the nested-loop optimization problem (with an annotator that generates a label and a student network that predicts a label) based on a pre-trained GAN that generates images. In particular, the authors solve this problem through a gradient matching approach, and they claim it is much more efficient than the existing end-to-end gradient-based meta-learning approach. This method was applied to the semi-supervised part segmentation task, and its effectiveness was verified through various experiments. 

The ultimate goal of the paper is to train an unsupervised generative model, in a semi-supervised manner, as a synthetic training example generator for training segmentation models. Prior works (eg. DatasetGAN) show that one can label a few synthetic images from a pre-trained generative model, and train a segmentation model (called annotation model) over the intermediate representation produced by the generative model. Then, every time a sample is taken from the generative model, the corresponding labels can be produced by applying the annotation model to the corresponding intermediate representations. This provides a valuable way to produce synthetic labelled training examples. However, (some of) these existing approaches require annotating (a limited set of) generated samples for training the annotation model. This is problematic as one needs to re-label such images every time the generative model changes.    This paper aims to address this problem by learning the annotation model over annotated real images instead of annotated synthetic images. To this end, the paper proposes a meta-learning approach. The main idea is to learn the annotation model such that when a (student) segmentation model is trained over the generative model outputs and their labels given by the annotation model, the resulting student model’s loss shall be low on real annotated images. This meta-learning idea is implemented as an approximation by using the Gradient Matching Loss. 

This work deals with the problem of training a part segmentation network by automatically synthesizing pairs of images and annotations. The authors propose a training method of an annotator given the well-pretrained generator model. The formulation is well-motivated and converged to the gradient matching loss.   This work extends [Zhao et al., ICLR 2021] by combining the idea of DatasetGAN and RepurposeGAN in an interesting way. The proposed approach can favor the unlimited number of annotated data with only limited supervised real data + the pre-trained GAN.  The proposed method shows noticeable improvement over the other baselines and the competing methods in scarcely labeled data regimes (especially semi-supervised learning).