A Comprehensive Overhaul of Distilling Unconditional GANs
Authors: Guodong Xu
Yuenan Hou
Ziwei Liu
Chen Change Loy
Venue: ICLR 2022 Submitted (2022)

Abstract: Generative adversarial networks (GANs) have achieved impressive results on various content generation tasks. Yet, their high demand on storage and computation impedes their deployment on resource-constrained devices. Though several GAN compression methods have been proposed to address the problem, most of them focus on conditional GANs. In this paper, we provide a comprehensive overhaul of distilling unconditional GAN, especially for the popular StyleGAN2 architecture. Our key insight is that the main challenge of unconditional GAN distillation lies in the output discrepancy issue, where the teacher and student model yield different outputs given the same input latent code. Standard knowledge distillation losses typically fail under this heterogeneous distillation scenario. We conduct thorough analysis about the reasons and effects of this discrepancy issue, and identify that the style module plays a vital role in determining semantic information of generated images. Based on this finding, we propose a novel initialization strategy for the student model, which can ensure the output consistency to the maximum extent. To further enhance the semantic consistency between the teacher and student model, we present another latent-direction-based distillation loss that preserves the semantic relations in latent space. Extensive experiments demonstrate that our framework achieves state-of-the-art results in StyleGAN2 distillation, outperforming the existing GAN distillation methods by a large margin.

Under review as a conference paper at ICLR 2022
A COMPREHENSIVE OVERHAUL OF DISTILLING UNCONDITIONAL GANs
Anonymous authors Paper under double-blind review
ABSTRACT
Generative adversarial networks (GANs) have achieved impressive results on var- ious content generation tasks. Yet, their high demand on storage and computation impedes their deployment on resource-constrainee devices. Though several GAN compression methods have been proposed to address the problem, most of them focus conditional GANs. In this paper, provide a comprchensive overhaul of distilling unconditional GAN, especially for the popular StyleGAN2 architecture. Our key insight is that the main challenge of unconditional GAN distillation lies in the output discrepancy issue, where the teacher and student model yield different outputs given the same input latent code. Standard knowledge distillation losses typically fail under this heterogeneous distillation scenario. We conduct thorough analysis about the reasons and effects of this discrepancy issue, and identify that the style module plays vital role in determining semantic information of gen- erated images. Based on this finding, we propose novel initialization strategy for the student model, which can ensure the output consistency to the maximum extent. To further enhance the semantic consistency between the teacher and stu- dent model, we present latent-direction--asee distillation loss that preserves the semantic relations in latent space, Extensive experiments demonstrate the effec- tiveness of our approach in distilling StyleGAN2, outperforming existing GAN distillation methods by a large margin. Code and models will be released.
INTRODUCTION
GAN compression (Wang et al., 2020; et al., 2020; Liu al., 2021) has been actively studied to enable the practical deployment of powerful GAN models (Karras et al., 2018: 2019; 2020) on mo- bile applications and edge devices. Among these techniques, knowledge distillation (Hinton et al., 2015) is widely adopted training strategy for GAN compression. The objective of GAN distillation is to transfer the rich dark knowledge from the original model (teacher) to the compressed model (student) so to mitigate the performance gap between these two models. There two distillation manners, i.e., pixel-level and distribution-level. The former minimizes the distance between gener- ated images of two models, while the latter minimizes the distance between distributions. In this work, we focus on the first setting considering its prevalence in GAN compression litcrature (Chen al., 2020: Wang et al., 2020; Li al., 2020; Liu et al., 2021)
The majority of contemporary GAN distillation methods focus on conditional GANs (cGANs) while the distillation of unconditional GANs (uncGANs) is relatively under-explored. Since there is large difference between the learning dynamics of these two types of GANs, distillation methods tailored for eGANs cannot be directly applied to the unconditional setting. Hence, in this work, we comprehensively investigate the distilling of uncGANs, especially StyleGAN2 (Karras et al., 2020), considering its popularity in both academia and industry.
We find that the main difficulty of uncGAN distillation lies in the output discrepancy between the teacher and student model. In fact, the implicit prerequisite of KD is that teacher and student should have similar outputs for the same input, otherwise the mimicking supervision is no longer meaning- ful. This prerequisite is easier to be satisfied in cGAN, because the output space of cGANs can be narrowed down by the given conditional input, especially when the condition is strong (Zhu et al., 2017: Isola et al., 2017) Consider horse->zee -zebba task as an example. An input horse image deter- mines which region should be added with zebra stripes and which region is background that should

---

Under review as a conference paper at ICLR 2022
not be changed. Two generated images in cGAN may differ in some low-level details such as the shape of zebra stripes, but would largely resemble in their structure. Unlike cGAN, as shown in experiments, it is impossible for uncGAN student with random initialization to learn similar mapping function to the teacher, even though we leverage distillation loss to enforce the agreement between the outputs of the two models.
To address the aforementioned output discrepancy problem, we carefully examine each component of the StyleGAN2 student model through comparative experiments. We identify that the style mod- ule plays crucial role in deciding the semantic information of the generated images. Based on this finding, we reach to a simple yet effective initialization strategy for the student model, i.e., inheriting the weights from the teacher style module and keeping the remaining convolutional layers randomly initialized. Such initialization strategy can work well even in heterogeneous distillation where the student architecture is obtained by neural architecture search (NAS) or manual design, and is totally different from the teacher model.
After resolving the output discrepancy problem, we further design a more effective mimicking ob- jective tailored for uncGAN distillation. As opposed to most of existing GAN distillation approaches that merely transfer the knowledge within single image, we propose novel latent-direction-ban relational loss to fully exploit the rich relational knowledge between different images. Specifically, exploit the good linear separability property of StyleGAN2 in latent space and then augment each latent code w by moving it along certain direction such that the resulting image only differs in single semantic factor. Then, we compute the similarity matrix between original images and augmented images and take it as the dark knowledge to be mimicked by the student network. The latent-direction-basee augmentation disentangles various semantic factors and makes the learning of each factor easier, thus yielding better distillation performance.
Our contributions are summarized as follows: To the best of our knowledge, this the first work that uncovers the output discrepancy issue in uncGAN distillation. Through carefully designed comparative experiments, we identify that the style module is the determining factor to ensure out- put consistency. 2) We propose a concise yet effective initialization strategy for the student model to resolve the output discrepancy problem, demonstrating significant gains upon conventional unc- GAN distillation. 3) We further propose a latent-direction--ased distillation loss to employ the rich relational knowledge between different images, and achieve state-of-the-a results in StyleGAN2 distillation, outperforming the existing ootte--o-th-aan- art CAGAN (Liu et al., 2021) by a large margin.
2 RELATED WORK
GAN Compression. We highlight few recent methods among the many GAN compression meth- ods (Shu al., 2019; Chang & Lu, 2020; Chen al., 2020; Wang et al., 2020; Li al., 2020). GAN Slimming (Wang et al., 2020) integrates model distillation, channel pruning and quantization into unified framework. GAN Compression (Li et al., 2020) first searches a compact student architecture via NAS, and then requires the student to mimic the intermediate outputs and synthesized results of the teacher simultancously. A common characteristic shared by these works is that they all focus on the compression of cGANs such Pix2pixGAN (Isola et al., 2017) and CycleGAN (Zhu al., 2017).
The more recent Content-Aware GAN compression (CAGAN) (Liu al., 2021) shifts the attention to compressing unconditional GANs. It first estimates the contribution of each channel in each layer to the generated human faces and eliminates channels with little contribution. Subsequently, the pruned model inherits the well-trained parameters from the original network for both style module and convolutional layers, and are finetuned with adversarial loss and distillation loss afterwards. Though CAGAN involves the compression of uncGAN, it bypasses the issues of model hetero- geneity between the teacher and student model by allowing the student to inherit the well-trained parameters. Such an requirement assumes the student to inherit the main structure of the teachers too despite pruning. As will be shown in the experiments, the performance of CAGAN will greatly degrade in heterogeneous distillation. The proposed mimicking loss cannot guarantee the student to learn a similar mapping as the teacher. Moreover, we find that the content- aware pruning strategy in CAGAN is not an optimal solution for student initialization. With our proposed initialization strat- egy, the student model does not need to inherit any weights from convolutional layers of the teacher but achieve better results.
2

---

Under review as a conference paper at ICLR 2022
Knowledge Distillation. Knowledge distillation (KD) (Hinton al., 2015) is originally proposed to achieve model compression (Buciluuddefined et al., 2006) for image classification, whose target is to transfer the dark knowledge from one multiple cumbersome networks (teacher) to small compact network (student). Vanilla KD (Hinton et al., 2015) proposes to match the outputs of two classifiers by minimizing the KL-ddivegeence of the softened output logits. Besides the output logits, other intermediate outputs such as feature maps (Romero et al., 2014), attention maps (Zagoruyko & Komodakis, 2017), Gram matrices (Yim et al., 2017), pre-activations (Heo al., 2019), rela- tion (Peng et al., 2019; Tung & Mori, 2019) and self- supervvinvv signal (Tian et al., 2020; Xu et al., 2020) can also serve as the dark knowledge. However, should be careful when adapting KD from classification tasks to generation tasks. The output consistency prerequisite is naturally satisfied in image classification since the supervision of labels guarantees different models to converge to similar mappings. As discussed in Sec. 1, the consistency prerequisite does not naturally hold for uncGANs. Therefore, a special distillation technique tailored for uncGANs is required to cope with the output discrepancy problem.
3 METHODOLOGY
3.1 PRELIMINARIES
StyleGANZ. There are two modules in StyleGAN2 (Karras et al., 2020), i.e., a style module S()) that maps Gaussian noise the style vector and a convolution backbone C(.) that takes as input and generate images. The style vector w is fed into the backbone C(-) through the modulated convolution (ModConv) layer (Huang & Belongie, 2017; Karras al., 2020). And StyleGAN2 allows the use of different w vectors in different ModConv layers. The image generation process can be formulated as:
G( 22, =L)
C(w1, W2, wL) =
((()))
where L is the number of ModConv layers in the backbone, and the i-th ModConv layer uses w; that comes from 2,. We define the output consistency condition as:
Gz 22. =L) = Gr(z1, (2, ,,,)),
where the and t represent student and teacher, respectively. Equation 2 suggests that the generated images of two models should be the same if they use the same z at corresponding layers.
StyleGAN2 Compression. A typical StyleGAN2 compression approach (Liu et al., 2021) contains two steps, i.e., pruning and finetuning. In the pruning stage, unimportant/ / unnecessary channels will be removed according to some heuristics (Hu et al., 2016; Li et al., 2017; He et al., 2018; Liu et al., 2021). Note that pruning is only applied to the convolution backbone C(-) and the style module S(.) is kept unchanged. The pruned model will inherit the well-trained weights from the original model for both the style module and the convolution backbone (Liu al., 2021). In the finetuning stage, besides the normal adversarial loss, the pruned model is also required to mimic the original model's output to compensate the performance degradation brought by channel reduction. typical mimicking loss includes RGB loss and LPIPS loss (Zhang al., 2018):
Ligb = [[),(>) Gz())l Lipips = [[F(G.()) - (x),
where F is a well-trained frozen network that computes the perceptual distance between two images. Lrgh and Lipips require that the generated image of student should be close to that of teacher in RGB space and perceptual space, respectively. The final loss function in the finetuning stage is:
-= ACAN + degh Alpips Dipips:
(4)
where X, is the loss weight of each item.
3.2 FRAMEWORK OVERVIEW OF UNCONDITIONAL GAN DISTILLATION
Knowledge distillation is a common strategy that can bring improvements in classification tasks. However, in generation tasks, its prerequisite, namely the student and teacher having consistent outputs for the same input, is rarely mentioned. In the absence of this prerequisite, the influence of mimicking losses to the training of student remains largely unknown. Here, we hypothesize
3

---

Under review as a conference paper at ICLR 2022
Source Imsaee
Intermoliate results
Target Image
dlcart
100
utdeett PIPS lews
-0.00
stedeat
(b) Grad cosine between GAN and RGB loss.
RGB less
LPUPS loss
Demence
0.04-
0.c0
stadeat 2 RGB less
10.00
(a) Qualitative effects of RGB/LPIPS losses.
(c) Grad cosine between GAN and LPPPS loss
Figure 1: (a) Student-1 1 has similar outputs with teacher and Student-2 has different outputs. The left column is the generated images of the original student. The image in the top right corner the teacher output. We demonstrate the intermediate results to show how RGB/LPIPS loss influ- ences the image generation of student. (b)(c) Cosine distance between the gradient of GAN loss and RGB/LPIPS loss. The z axixis denotes training steps. For similar student, RGB/LPIPS loss is cooperating with GAN loss. For dissimilar student, RGB/LPIPS loss is competing with GAN loss.
that RGB or LPIPS loss is not compatible with GAN loss when the output discrepaney occurs and distillation will also bring no benefit to the student. We examine this hypothesis both qualitatively and quantitatively.
Note that the three losses in Eq. 4 serve different roles. LGAN requires the student to generate realistic images while Ligh and Cipips encourage similarity between the generated images by student and those of the teacher. Intuitively, if the generated image of the student are totally different from that of teacher for the same input, figh and Lipips will result in images that are slightly closer to teacher but with much less realism. To examine this hypothesis, we remove the GAN loss in Eq. and keep only RGB or LPIPS loss. We also cut off the gradient backward path between the student generator and generated images. In this condition, the gradient of RGB/LPIPS loss directly works on the images. The change of synthesized images reflects how RGB/LPIPS loss influences the generation process. We select two student models, i.e.. student-1 that has similar output with teacher for the same input and 2 that has totally different outputs from teacher. The effects of RGB//LPIPS loss are shown in Fig. 1a. We can find that the intermediate results are a mixup of source and target images to some extent. If the source image is in the neighbor of the target image (Ist and 2nd rows). the intermediate results are still perceptually realistic. However, if the source image has large distance to the target image (3rd and 4th rows), the intermediate results are no longer realistic. Though RGB and LPIPS losses are reducing the distance between source and target images, they cannot guarantee a smooth and face-like interpolation in dissimilar setting. And this unrealistic intermediate results naturally contradict with GAN loss.
From quantitative perspective, we wish to prove that RGB/LPIPS loss is not compatible with GAN loss in the heterogeneous setting by gradient analysis. In the training process, for each batch, we perform backward propagation for GAN loss, RGB loss and LPIPS loss, respectively, and obtain three gradients of these losses. We then compute the cosine distance between GAN gradient and RGB/LPIPS gradients. As shown in Fig. 1b and Fig. Ic, the cosine distance between GAN gradient and RGB/LPIPS gradients of dissimilar student is always negative, suggesting that RGB/LPIPS gradients are competing with GAN gradients. On the contrary, the cosine distance of similar student is positive, indicating that the distillation loss is driving the model in the same direction as the adversarial loss. Our analysis above suggests that distillation is not beneficial in heterogeneous setting. Having similar outputs for the same input z is the prerequisite for uncGAN distillation.
3.3 EFFECT OF THE STYLE MODULE
As we will show in the experiments, if the student is randomly initialized, it cannot learn consistent outputs teacher even though we leverage RGB/LPIPS loss to force the agreement between the
4

---

Under review as a conference paper at ICLR 2022
outputs of two models. We hypothesize that the style module S(z) plays key role in determining whether two models can have consistent outputs. If the gap between style modules of student and teacher is too large, it is hard for the student to learn outputs consistent with the teacher.
We prove this hypothesis using a proof by contradiction. Suppose the student has a different style module from the teacher and the consistency condition (Eq. 2) is also satisfied. For the convenience of the following discussion, we define:
G( 22; k) = C(wi, w2; k) C(wi,-.. W1, w2, w1--. ,
(5)
where all the ModConv layers use wh except that the k-th layer uses w2. The consistency condition of Eq. 2 requires that:
Go (21, k) Ge 22 k), k L.
(6)
As shown in StyleGAN (Karras et al., 2019), for well-trained model, the w latent space consists of linear subspaces. It should be possible to find direction vectors that consistently correspond to individual factors of variation. An example is shown in Fig. 3. Some individual semantic factors such as pose, glasses and hair color can be controlled by moving the style vector w of certain layer along a certain direction. Suppose the direction p at k-th layer controls the hair color of the generated face. The only difference between Cr(wo, w1 + p; k) and (wo,, w1; k) is that they are the same faces with different hair colors. The movement of w from w1 to w1 + + corresponds to a consecutive change of hair color of the generated face. If we map the back to the noise space:
(7) obviously, the line segment in w space corresponds to a curve in space with two end points 21 and 22 due to the nonlinearity of Sz(z). We denote this curve as 21 Then (Ge(zg, 2; k;]]22 21 represents cluster of faces with different hair colors. According to the consistency constraint,
21 = -.=S-10 (w1), (wn), 22 = S (wi + p),
{G, (Zo. 2; A)]2 22 should be the same cluster as 2; E 2122). We feed 20, 2 21 into the student style module Sz(-):
wn S.(o), whwn S.(Ei2)).
(8) Since S,(-) different from and independent of Se(-). the result w/ wn is still a curve. Thus, the semantic factor of hair color in student model is controlled by a complex curve in space, which contradicts the property of StyleGAN2 that various semantic factors are decoupled well in w space. Hence, having different style modules and consistency condition cannot hold at the same time.
We further conduct experiments to examine our hypothesis. Specifically, we select four students according to whether the style module is from teacher not and whether the convolution is from teacher or not, We use GAN loss, RGB loss and LPIPS loss to train these models. The style module and convolution weights from the teacher's s style module can learn a mapping that aligns well with the teacher's s output, matter how the convolution C(-) is initialized. However, for the student whose style module is randomly initialized, there are no meaningful connections between student's and teacher's outputs. The analysis above clearly shows that the output consistency between student and teacher is determined by the style module.
updated together. The results
shown in Fig. 2. The student that inherits
3.4 STYLE MODULE CONSISTENCY IN GAN DISTILLATION
We have shown that the consistency between student and teacher outputs is the prerequisite of the distillation, and the style module determines whether two generators can have consistent outputs. Hence, make distillation meaningful, it necessary to impose extra constraints to guarantee the consistency between two style modules.
The simplest way is to keep the architecture of the style module unchanged and inherit teacher's parameters directly. In fact, the parameters of style module account for only 7.5% of the parameters of the convolution backbone. The FLOPs of style module account for about 0.005% of the FLOPs of the convolution backbone. Preserving style module is thus feasible in practice.
If there a strong demand on the compression of style module, one can perform a two-stage training to ensure small gap between student and teacher style modules. In the first stage, the student style module is forced to mimic outputs of the teacher style module:
L E N(,,1 ((S.)))
5

---

Under review as a conference paper at ICLR 2022
Teacher
Inherit
Inhorit C()
Inherit 5
Random C()
Random S())
Inherit C()
Random S( Random C()
Figure 2: The style module S(.) determines whether student can learn from the teacher's output. If student inherits teacher's S()), it can learn the teacher's output well no matter how the convolution C() is initialized. If student uses random S(2), it cannot learn the teacher's output matter how the convolution C(.) is initialized.
where D(s-) is a distance metric. Considering S.(-) and S.(-) are both shallow MLPs, the training cost of this stage is negligible (0.59% of the normal GAN training the second stage).
In the second stage, the style module and generator backbone are finetuned together using the loss in Eq. 4. We will explore the effects of compressing the style module Sec, 4.1.
3.5 LATENT-DIRECTION-BR BASED RELATION DISTILLATION
Under the premise that consistency condition is satisfied, we further propose to incorporate relation mimicking into GAN distillation. Conventional relation-based distillation (Tung & Mori, 2019) in classification task computes feature similarity matrices using the samples in a minibatch. Here, we tailor it to better cater StyleGAN2.
Specifically, for given well-trained teacher model, we compute all its meaningful latent directions (LD) that control single semantic factor and store them a dictionary {d1, d2.... dm ). Note that the latent direction is related to specific layer. For example, if d; is computed in k-layer, then only Ce(w, w + d;; k) has single semantic factor difference with (ww) C((w, w + J4j4) does has this property. In the training stage, feed a batch of noise { iiN:N into the style module and obtain { (uNN.. For each w; we randomly sample a latent direction d, from the dictionary. Thus, Ce (w:) and Ce Cuw,, w; + ad,; k) (k is the layer related to d,) are two images with single semantic factor difference with a controls the moving distance. We denote the intermediate features of C (w;) and C, (w,, w; + ad,; ; k) fi and f/. respectively. Then the similarity matrix M between original view and augmentation view can be computed as A. = ft Jí. We then convert the similarity into probability via the softmax operation and minimize the distance using Kl--divergence loss:
Mcj. exp(Ai.))
LLD =
Σ i
Mi, log M
(10)
E exp(A,,))
The final learning objective the combination of Eq. and LLD.
4 EXPERIMENTS
We conduct experiments mainly on StyleGAN2 since it is the most powerful unconditional GAN so far. We use the FFHQ (Karras et al., 2019) dataset. Following CAGAN (Liu et al., 2021), we adopt Fréchet Inception Distance (FID), Perceptual Path Length (PPL) (Karras et al., 2019) and PSNR/LPIPS (Liu et al., 2021) between real and projected images as evaluation metrics. More qualitative results shown in the appendix.

---

Under review as a conference paper at ICLR 2022
Table 1: Effect of initialization. Inheriting both style module and convolution naturally obtains better results than random initialization. Surprisingly, we find that inheriting only style module is the best solution. With this initialization, RGB+LPIPS and RGB+LPIPS+LD obtains 0.38 and 0.51 improvements, respectively, compared to their corresponding results when inherit both modules.
Style Module Initialization
Convolution Initialization
Mimicking Loss
Student FID
No Mimic RGB
10.92
random
random
10.78
RGB + LPIPS
11.27
RGB RGB+LPIPS
10.81
random
inherit
10.88
No Mimic RGB
10.54
9.41
inherit
inherit
RGB + LPIPS
8.61
RGB + LPIPS + LD
8.45
RGB RGB + LPIPS
9,42
inherit
random
8.23
RGB LPIPS + LD
7.94
For the ablation study in Sec. 4.1, we train the models on resolution 256x256 and use a smaller batch size of 8 to save the computation cost. For the comparison with state-of-the-art methods in Sec. 4.2, we train the models on both resolutions of 256 256 and 1024x 1024. We also use a hatch size of 16 that the same as CAGAN (Liu al., 2021) to ensure fair comparison.
4.1 ABLATION STUDY
The Initialization of the Student Model. Previous works usually treat StyleGAN2 as an integral module and initialize style module and convolution backbone in the same way (from scratch or in- herits teacher parameters). Based on our analysis Sec. 3.3 that the style module plays key role in determining the semantics of generated images, here we separate style module S(2) from convo- lution backbone (ww) and test three initialization strategies: both S(z) and C(w) are randomly initialized, 2) both S(z) and C(w) are initialized with teacher weights, 3) only S(z) inherits teacher weights and (ww) is randomly initialized.
The results are shown in Table. 1. For the setting where S(z) and C(w) are both randomly ini- tialized, RGB loss can only bring marginal improvement. RGB+LPIPS even performs worse than No-Mimic, indicating that distillation cannot work well when output discrepancy occurs. If S(z) and ((w) both inherit teacher weights, the mimicking loss can achieve 1-2 FID improvement. To explore the effect of style module, also try inheriting only S(z) and surprisingly find that this initialization obtains the best result. And loading C(w) hampers the performance of RGB+LPIPS and RGB+LPIPS+L.D. This result contradicts with the conclusion in CAGAN (Liu et al., 2021). It shows that the general pruning strategy, i.c., determining which channels should be removed, is not important. Randomly initialization of convolution layers is the optimal solution as long as the style module is kept.
The Effects of Style Module Compression. We conduct experiments investigate how deal with the style module in StyleGAN2 compression. Specifically, we consider three settings: 1) stu- dent has the same style module architecture as teacher but with random initialization, style module has different architecture and uses the two-stage training strategy, 3) student has the same architecture and inherits the weights from the teacher style module. For all the settings, the convolution backbones randomly initialized. For the two-stage setting, also explore how the architecture of the style module and mimicking loss in Eq. affect the final performance. To emphasize the importance of style module, also list the average LI distance between S.(2) and S,(2) before entering the normal GAN training stage.
student
The results shown in Table. 2. 'Random Initialization' obtains the worst FID because the output diserepancy makes the distillation ineffective. The 'Two-Stage' strategy improves the results by narrowing the gap between S,(z) and S,(2). From several two-stage settings, we can find that L.

---

Under review as a conference paper at ICLR 2022
Table 2: How deal with style module in StyleGAN2 distillation. The style module is comprised of MLPs. The numbers inside and outside the "[]" is the number of channels in each layer and the number layers, respectively. The style module of the teacher [ The FLOPs saving only considers FLOPs reduction in the style module.
Style Module Architecture
FLOPs Saving
Setting
D(.-)
[15((z) Sc(=)
Student FID
Random Initialization
[512]*8
0%
N/A
1.027
11.78
[
0%
L1
0.156
9.69
[
0%
L2
0.260
10.80
Two-Stage
[512]*5
37.5%
L11
0.197
10.38
[390]*7+[512]
37.44%
L.1
0.210
10.55
[256]*7[[52]]
68.75%
L1
0.245
10.86
Inheriting
[512]*8
0%
N/A
0
8.30
better mimicking loss than L. and reducing the number of layers is better than reducing the number of channels in each layer. It is also worth noting that there is a strong positive correlation between (=) (=)) and FID, indicating that the gap between (z) and (+)z) determines the output consistency and further determines the influence of distillation.
Though the two-stage strategy brings perfor- mance gains, there is still a large gap between and the 'Inheriting' variant. Thus. con- clude that the modification to the style module will greatly harm the final performance and the two-stage strategy can only mitigate the degra- dation certain degree. Considering that the scale of the original S,(=) is negligible com- pared to the convolution backbone, the best practice in StyleGAN2 compression is to pre- serve the style module architecture and inherit the weights from the teacher style module.
Table 3: Ablation study about relation mimicking. Single View brings marginal improvement. Ran- dom Offset even has negative effect. Our LD loss consistently improves the performances of both RGB and RGB+LPIPS.
Mimicking Loss
LLD
FID
RGB RGB + Random Offset
N/A
9.41
KI.
9.80
RGB Single View
KL
9.47
RGB + LD
L2
9.16
The Effects of Latent-Dirrectin-Baseee Dis- tillation Loss. The proposed latent-direction- based loss is essentially a relation loss. We interested in whether the benefit brought by LLD comes from relation mimicking or from the latent-direction-basee augmentation. Specifically, we consider three variants: 1) Single View, namely the similarity is computed inside the normal samples rather than between normal samples and augmented samples, 2) Random Offset, namely move along random direction to get Sú instead of along the latent direction, 3) Our nutentdirrectoonbasee method (abbreviated as LD).
RGB LD
KL
9.05
RGB + LPIPS RGB LPIPS LD
N/A
8.61
L2
8.64
RGB + LPIPS + LD
KL
8.26
The results are shown in Table. 3. Ringl View obtains comparable results with RGB loss, in- dicating that naive relation mimicking cannot bring improvements. RGB+Random Offset performs worse than RGB loss, demonstrating that the way of augmentation is critical for relation mim- icking. Improper augmentation may be counterproductive. LD improves RGB by around 0.3 FID. And LD still bring 0.16 FID improvement to the RGB+LPIPS variant. Among two kinds of LLD. KL-divergence loss is slightly better than L2 loss.
4.2 COMPARISON WITH STATE-OF-TIE-AARY METHODS
We compare our method with the GAN Slimming (Wang et al., 2020) and CAGAN (Liu al., 2021) methods. Since our method does not focus on the pruning, we directly adopt the student architecture used in CAGAN, i.e., a network that is the same as teacher but with fewer channels. We also compare with CAGAN in heterogeneous setting where the student is not subnet of the teacher. Specifically, we modify the kernel size of the second convolution layer in each resolution block from to 1, thus inheriting teacher convolution parameters is infeasible. Since CAGAN did not notice the output
8

---

Under review as a conference paper at ICLR 2022
Table 4: Comparison with SOTA. "i" (*f") denotes the lower (higher) the better. For the FID and PPL of Full-Size, GAN slimming and CAGAN, we directly use the results reported in CAGAN (Liu al., 2021). Since CAGAN did not release their computation details of PSNR and LPIPS, we compute these two metrics using our own implementation. We also list the PSNR and LPIPS results reported in CAGAN in "O". The PSNR and LPIPS GAN slimming are blank due to the lack of its well-trained checkpoint. Bold font denotes the results that outperform CAGAN. "heter" denotes heterogeneous setting where the student is not a subnet of the teacher.
Model
Image Size
FLOPs
FID (4)
PPL (1)
PSNR (1)
LPIPS (L)
Original Full-Size
256
45.1B
4.5
0.162
34.26 (32.02)
0.057 (0.113)
Baseline
256
4.1B
9,79
0.156
33.14
0,082
GAN slimming
256
5.0B
12.4
0.313
(31.2)
(0.177)
CAGAN
256
4.1B
7.9
0.143
33.34 (31.41)
0.076 (44)
Ours
256
4.1B
7.25
0.135
33.49
0.071
CAGAN-heter
256
2.7B
13.75
0.158
33.19
0,083
Ours-heter
256
2.7B
9.96
0.141
33.54
0.073
Original Full-Size
1024
74.3B
2.7
0.162
33.52 (31.38)
0.075 (0.149)
GAN slimming
1024
23.9B
10.1
0.211
(30.74)
(0.189)
CAGAN
1024
7.0B
7.6
0.157
32.63 (30.96)
0.099 (0. 170)
Ours
1024
7.0B
7.19
0.128
32.70
0.094
discrepancy issue and always initialize the style module and convolution backbone in the same way, we assume it does not inherit weights from teacher in heterogeneous setting. The results are shown in Table. Our method outperforms CAGAN on FID by 0.65 and 0.41 on resolution 256 and x 1024, respectively, showing that our method can generate more realistic images. Note that these improvements are not marginal considering the images generated by CAGAN already have high quality. For the PPL metric that measures the smooth degree of latent space, outperform CÁGAN by 0.008 on resolution 256. The gap is even larger (0.029) on resolution 1024. For PSNR and LPIPS that are related to image projection ability, our method also surpasses CAGAN, demon- strating that our method can model the face distribution in real world better. Our superiority is much more significant in heterogeneous setting, showing that our method can be applied in more general situation where the student is not necessary to be a subnet of the teacher.
5 CONCLUSION
In this paper, we uncover the output discrepancy issue in uncGAN distillation. Through comparative experiments, we find that the style module is the key the output discrepancy and propose a novel initialization strategy of student, which can help resolve the output discrepancy issue. The proposed seceen-ddrection-baae distillation loss further improves the distillation efficacy and we achieve state- of-the-ar art results in StyleGAN2 distillation, outperforming the rival method by a large margin on image realism, latent space smoothness and image projection fidelity.
LIMITATIONS
The analysis and method in this work are tailored for StyleGAN2-like generator, which may affect its universality. It is unknown whether our analysis is still valid for those uncGANs that do not have style module such as BigGAN (Brock et al., 2019) and PGGAN (Karras et al., 2018). Whether conventional mimicking loss such RGB/LPIPS loss solve the output discrepancy and how to initialize them are not clear. Besides, we mainly conduct experiments on FFHQ which is a well- aligned face dataset. Experiments and rooms should be included to examine the effectiveness of our method. Finally, we only consider the output discrepancy issue in unconditional GANs. In fact, this problem also exists in conditional setting when the condition is not strong enough (e.g., the conditional input is the class label). How to analyze the output discrepancy issues of uncGANs and eGANs in a more general form is also direction worth exploring.
more and harder datasets such as ImageNet, outdoor scenes

---

Under review as a conference paper at ICLR 2022
REFERENCES
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. In International Conference on Learning Representations (ICLR), 2019.
Cristian Buciluundefined, Rich Caruana, and Alexandru Niculescu-Mizii. Model compression. In Proceedings of the 12th ACM SIGKDD International Conference Knowledge Discovery and Data Mining, 2006.
Ting-You Yun Chang and Chi-Jen Lu. Tinygan: Distilling biggan for conditional image generation. In The Asian Conference on Computer Vision (ACCV), 2020.
Hanting Chen, Yunhe Wang, Han Shu, Changyuan Wen, Chunjing Xu, Boxin Shi, Chao Xu, and Chang Xu. Distilling portable generative adversarial networks for image translation. arXiv:2003.03519 2020.
Yang He, Guoliang Kang, Xuanyi Dong, Yanwei Fu, and Yi Yang. Soft filter pruning for accelerating deep convolutional neural networks. arXiv: 06866, 2018.
Byeongho Heo, Jeesoo Kim, Sangdoo Yun, Hyojin Park, Nojun Kwak, and Jin Young Choi. A com- prehensive overhaul of feature distillation. In The IEEE International Conference on Computer Vision (ICCV), 2019.
Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in a neural network. In NIPS Deep Learning and Representation Learning Workshop, 2015.
Hengyuan Hu, Rui Peng, Yu-Wing Tai, and Chi-Keung Tang. Network trimming: A data-driven neuron pruning approach towards efficient deep architectures. arXiv: arXiv: 2016.
Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normal- ization. In The IEEE International Conference on Computer Vision (ICCV), 2017.
Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for im- proved quality, stability, and variation. 1rXhv:1710.10190 2018.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyz- ing and improving the image quality of StyleGAN, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. 1rXiv:1608.087/1 2017.
Muyang Li, Ji Lin, Yaoyao Ding, Zhijian Liu, Jun- Yan Zhu, and Song Han. Gan compression: Efficient architectures for interactive conditional gans. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Yuchen Liu, Zhixin Shu, Yijun Li, Zhe Lin, Federico Perazzi, and Yun-Yuan Kung. Content-aware gan compression. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
Baoyun Peng, Xiao Jin, Jiaheng Liu, Dongsheng Li, Yichao Wu, Yu Liu, Shunfeng Zhou, and Zhaoning Zhang. Correlation congruence for knowledge distillation. In The IEFE International Conference on Computer Vision (ICCV), 2019.
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. arXiv: /42.:5/412.65500 2014.
10

---

Under review as a conference paper at ICLR 2022
Han Shu, Yunhe Wang, Xu Jia, Kai Han, Hanting Chen, Chunjing Xu, Qi Tian, and Chang Xu. Co- evolutionary compression for unpaired image translation. In The IEEE International Conference Computer Vision (ICCV), 2019.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. In Inter- national Conference on Learning Representations (ICLR), 2020.
Frederick Tung and Greg Mori. Similarity tional Conference on Computer Vision (ICCV), 2019.
knowledge distillation. In The IEEE Interna-
Haotao Wang, Shupeng Gui, Haichuan Yang. Ji Liu, and Zhangyang Wang. Gan slimming: All- in-one gan compression by unified optimization framework. In The European Conference Computer Vision (ECCV), 2020.
Guodong Xu, Ziwei Liu, Xiaoxiao Li, and Chen Change Loy. Knowledge distillation meets self- supervision. In The European Conference on Computer Vision (ECCV), 2020.
Junho Yim, Donggyu Joo, Jihoon Bae, and Junmo Kim. A gift from knowledge distillation: Fast optimization, network minimization and transfer learning. In IEEE Conference Computer Vision and Puttern Recognition (CVPR), 2017.
Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Improving the per- formance of convolutional neural networks via attention transfer. In International Conference Learning Representations (ICLR), 2017.
Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networkss. The IEEE International Conference on Computer Vision (ICCV), 2017.
11

---

Under review as a conference paper at ICLR 2022
Pose
Lighting
Glasses
Hair Color
(a) Oirgin
(b) Single factor change
Figure 3: StyleGAN2 shows good factorization in the w space. It is possible to control a single semantic factor such as pose, lighting condition, glasses and hair color by moving the style vector of a certain layer along specific direction.
APPENDIX
A IMPLEMENTATION DETAILS
Training Hyperparametees. For style module mimicking (Eq. 9) of the first stage, we use Adam optimizer with initial learning rate of 0.05. We train for 50k steps with batch size 4096. For the normal GAN training the second stage, we use Adam optimizer with initial learning rate of 0.002 and 450k iterations. For the that controls the offset along latent direction, from a Gaussian distribution 5). We set AGAN, Argh, Alpips and ALD to be 1, 3, 3 and 30, respectively.
sample
Evaluation Metries. Fréchet Inception Distance (FID) is commonly used metric to evaluate the realism of generated images. The generated images and real images are fed into inception network and then Fréchet distance is computed between their corresponding feature maps. We use the implementation of FID in CAGAN (Liu et al., 2021). Specifically, we use 50K real images and 50K generated images to compute statistics, respectively. Perceptual Path Length (PPL.) is proposed in StyleGAN (Karras al., 2019) measure the smoothness of latent space. We adopt the PPI. implementation in CAGAN (Liu et al., 2021) for fair comparison. PSNR and LPIPS are proposed by CAGAN to evaluate the image projection ability. A given real image is first mapped back to the latent space through optimizer such as L-BFGS. The projected image is obtained by feeding this resulting latent code to the generator. Then the PSNR and LPIPS distance are computed between projected image and the original image again. A smaller value indicate that the generator model the distribution in real world better. We compute these two metrics using our own implementation.
B STYLEGAN2 LINEAR SEPARABILITY
well-trained StyleGAN2 model is linear separably in latent space. An example is shown in Fig-
ure. 3.
C IMAGE EDITING
We demonstrate the superiority of our method image editing, including style mixing and inter- polation. Given two real face images LA. Ip. we first project them back to the latent space and get WA, wp. Both WA and w/ are of shape L x D, where
the number of convolution layers and
D is the dimension of latent code. For style mixing, we replace the i-th vector in wA with that from WB. We set E [1,3], [5,8] and lincarly combine the latent code with 3 controls the weight: w = 3 WA + (1 3) wp, and then feed w into generator to get the interpolation results. We edit the images resolution 256x256.
[10, 13) for coarse, middle and fine style mixing,
respectively. For interpolation,
The results are shown in Fig. 4. For style mixing, CAGAN always has artifacts in face shape (coarse style) and skin color (middle shape). While our results are more realistic and correspond better with two source images. coarse style case, our result corresponds well on face shape and sense organs with source B. In fine style case, result corresponds well on lighting and skin color with source
12

---

Under review as a conference paper at ICLR 2022
B. For interpolation, we also observe smoother change than CAGAN, showing that method learns a better structure in latent space.
D IMAGE PROJECTION
We show image projection results of our method in Fig. 5. All the real images come from Helen Sct55 (Liu al., 2021) and are not seen in the training stage. Our model reconstructs them with high quality.
E GENERATION RESULTS
We show more generation results of resolution 256x256 and resolution 1024x 1024 in Fig. 6 and Fig. respectively.
13

---

Under review as a conference paper at ICLR 2022
Style Mixing
Interpolation
Coarse from B
Style from B
Style from B
P = 0.25
p= =0.50
# = 0.75
Source A
Teaeher
CAGAN FLOP
Source B
FLOPs
(a) In coarse style mixing, CAGAN generates glasses, which does not appear in both source images. CAGAN also produces blurry images in middle style case. In contrast, style mixing results more realistic and similar to teacher. Our method also shows smoother transition of mouth and haircut in interpolation.
Style Mixing
Interpolation
Coarse Style Middle Style
Fine Style
f 0.25
0.50
0.75
Source A
Teacher 45.1B FLOPs
CAGAN
Source
Ours 4.1B FLOPs
(b) coarse style mixing, our result corresponds better with source on mouth and face shape. In fine style mixing, our result corresponds better with source on skin color. CAGAN also generates artifacts on hair middle and fine style cases. In interpolation, our methods are more consistent with teacher than CAGAN.
Style Mixing
Interpolation
Style from B
Middle Style from
Style from
= 0.25
f - 0.50
f= 0.75
Source
Teacher
CAGAN 4.18 FLOPs
Source B
4.1B FLOPs
(c) CAGAN generates lighting artifacts in coarse case and skin color artifacts in fine case, while our results more realistic. In interpolation of CAGAN, the earrings disappear in 0.25 while appear again in 0.50. contrast, our results are much smoother.
Figure 4: Image editing results.
14

---

Under review as a conference paper at ICLR 2022
Real
Proj.
Real
Proj.
Real
Proj.
Real
Proj.
Figure 5: Image projection results. In each pair, the left image is from real world (not from training set) and the right image is the projected result by our model. Our method can model the real face distribution well.
15

---

Under review as a conference paper at ICLR 2022
Full-Size 45.IB FLOPs
Random (z)
Two-Stage FLOPS
CAGAN 4 FLOPs
Ours 4.18 FLOPS
Ours-heter 2.7B
FLOPS
Figure 6: Synthesized results on resolution 256x256. Random S,(z) denotes the style module S,(2) is randomly initialized. For Two-Stage, we compress the original 8-layer style module into layers. The images of each row are generated using the same input noise z. Note that all the students are trained with mimicking loss. Random S. (2) cannot generate images consistent with teacher due to the different style module. Two-Stage mitigate output discrepancy issue by directly mimicking style module, but there are still semantics differences from teacher. Compared to CAGAN, our generated images have fewer artifacts and are more similar to teacher.
16

---

Under review as a conference paper at ICLR 2022
Full-Size 74.3B FLOPs
CAGAN FLOPs
Ours 7,0B FLOPs
Figure 7: Generation results on resolution 1024x 1024. The synthesized images of Our method are of better quality than CAGAN. In several semantie factors such beard, haireut and glasses, results are more similar to the full-size model even though we do not inherit convolution weights.
17


Reviews:
[{"review_id": "sjBb2F7j_QO", "reviewer": "Reviewer_fZmP", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n  1. Solving the output discrepancy issue for distilling unconditional GAN is an interesting topic. \n  2. Show many ablation studies for different initialization strategies and loss functions for style module and CNN parts.\n\nConcerns:\n  1. Since the paper abstract starts with the motivation of reducing the storage and computation of deploying GAN models, I would like to see a thorough comparison between a distilled model using the proposed method and a same architecture network which is simply trained from the scratch using the original training recipe described in StyleGAN paper. Otherwise, if a same architecture can achieve the same quality by training from scratch, there is no need to do distilling any more. When I read the paper, I guessed the row 1 in Table 1 (random + random + No Mimic) might represent such experiment, but there is no discussion about how this experiment is done.  It also needs to be added to Table 4.\n  2. In Section 3.2 and Figure 1(a), it is unclear to me how to find the similar student models and dissimilar student models. What type of criteria is used to define similar or not similar?\n  3. The paper only shows the results on face images (the aligned FFHQ face images). How does the method work on other types of images? Especially, other types of images, like building/rooms, have more complicate data distribution than aligned face images. The semantic information of these types of images are not that well disentangled as face images. In this case, does inheriting style module still help? Does the latent-direction-based distilling loss still help? \n  4. In addition to RBG+LPIPS+LD losses, would adding adversarial loss also help? ", "summary_of_the_paper": "Typical knowledge distillation loss/pipeline failed for unconditional GAN distillation, due to the output discrepancy between teacher and student model even if same inputs are fed. This paper proposed a framework for distilling unconditional GANs. The framework mainly contains two parts: 1). Inherit the weights from teacher style module for initializing student’s style module based on the observation that style module plays a crucial role in determining the outputs’ semantic information; 2). Propose a latent-direction-based distillation loss to enforce the consistency between teacher’s outputs and student’s outputs. Experiments are mainly shown by comparing with CAGAN and some variants of the proposed methods with different initialization strategies and loss functions. ", "summary_of_the_review": "Overall, the paper tries to address an interesting problem of tackling the output discrepancy issues when distilling unconditional GAN models. The experiment part provides several ablation studies. However, I have several main concerns regarding to: 1). more thorough comparison with the same architecture network trained from scratched and more discussions; 2). try the proposed method on other types of images to validate whether inheriting style modules and latent-direction-based distilling loss still help or not. \n\n\n==============After Rebuttal===========================\n\nThanks the authors for the clarification. I agree with other authors about the limitations of the proposed method, especially about how to extend to other unconditional GANs and other datasets. Thus, I will still give a borderline rating. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "rPt2eSUs6md", "reviewer": "Reviewer_bbKQ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\nA- The paper tackles an interesting problem\nB- The authors are referring and comparing to key previous work (CAGAN). \n\n\nCons:\nC- My biggest concern with the current draft is the experiment section and claims. While the authors have done a good job in comparing to previous work (CAGAN), the comparison is not convincing nor rigourous. Here is why:\n1) Standard deviation is missing in all tables. The authors need to run several train for each experiment  and report their average and standard deviation. All numbers between CAGAN and the proposed method are so close that a standard deviation is required to compare these methods. It is a common practice in GAN literature. I am quite surprise that the authors have omitted it. \n2) The authors claim (in the conclusion) that they outperform previous work by a \n“large margin”. It is not supported by any evidence. I am not going to list all the   quantitative numbers but when the gains are “0.008 or 0.029 etc…” on a metric where previous work had one order of magnitude more gain, to my humble opinion, these are not large. \n3) While the quantitative results can not be interpreted without the standard deviation,  qualitatively comparison are also not convincing.\n\nD- Finally, the technical depth of the paper and contributions are also limited. I am in favour of simple yet effective ideas. Yet, they need to be “effective”. Here, following my comments in the experiment section, it seems that the ideas are simple but not necessarily effective. \n\nHere are some typos to be fixed: \n- “sduent” -> student in Sec 2.\n- “0.31” -> 0.41 in Section 4.2", "summary_of_the_paper": "The authors present a method to distill StyleGAN2, an unconditional Generative adversarial network. They claim that the main challenge lies in the output discrepancy between the teacher and student. They have identified that the style module plays an important role in determining semantic information of generated images. Hence, they have proposed:\ni) to initialise the student model with the inherent weights of the teacher module and keeping the remaining convolutional layers randomly initialised, and \nii) a new distillation loss that preserves the semantic relations in latent space. \nThe authors run all their experiment on StyleGAN2. ", "summary_of_the_review": "Given C and D comments, I can not accept the paper “as is”. Critical points are missing and conclusions can not be drawn without them. I hope that the authors will be able to address my comments. I thank them for their work and highly encourage them to continue this line of work and re-submit. Best.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "m9a9rys2O5", "reviewer": "Reviewer_RUSD", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Pros:**\n\n1. The output discrepancy issue pointed out by the paper is a real problem for distillation, which does not only exist in the StyleGAN2 compression (references [1] and [2]). The valuable conclusion is that this paper provides the solutions for the StyleGAN2 compression. Based on their results, we know that the style module can not be compressed, while CNN backbone can be compressed. Because the style module of the student should inherit from the teacher.\n\n2. The illustrated proof and result are convincing. Section 3.2 shows the following two phenomena: 1) The more similar output between studentNet and teacherNet makes studentNet generate more realistic images. 2) The dissimilar outputs cause GAN loss and distillation loss to have a competitive relationship, thereby affecting the distillation. \n\n3. This paper provides comprehensive experiments, including both qualitative analysis and quantitative results, to show the effectiveness of the proposed framework. The paper is well written, and the authors analyze the reasons step by step and propose reasonable solutions.\n\n[1] J. Han, et al. Fixing the Teacher-Student Knowledge Discrepancy in Distillation.\n\n[2] B. Liu, et al. MetaDistiller: Network Self-Boosting via Meta-Learned Top-Down Distillation.\n\n\n**Cons:**\n\n1. The paper didn’t introduce the difference between the image generation task and recognition task in the knowledge discrepancy problem, and the inspiration of the proposed method on the use of knowledge distillation in other fields. Otherwise, the proposed method is too specific.\n\n2. Based on the results of the references [1] and [2], the output discrepancy issue also exists in the classification task, although there is enough supervised loss for the training of studentNet. Thus, it seems that the above issue also exists in the uncGANs distillation, which is contrary to the authors’ claim in the 3rd paragraph of the introduction. In order to prove the correctness of the hypothesis in the paper, there should be detailed theoretical analysis or experimental results.\nIn section 3.3, the counter-evidence provided by the authors is simple and not convincing. Of course, a well-trained Net and random initialized Net generate different latent space W, and the W of the latter Net is certainly not linearly separable. And, for two different sizes of style modules, the conclusion is the same as above.\n\n3. Insufficient comparative experiments. 1) For Figure 3, there are no results for different initialized CNN backbone. Without them, how can one demonstrate that the style module plays a more important role in the distillation?  And in Table 1, what about the results on the random-inherit setting. 2) The experimental results of tuning hyperparameters should be added in the paper (\\lambda_GAN, \\lambda_rgb, \\lambda_lpips, and \\lambda_LD).  The authors should provide more details of them under different settings, whether they are with the same values in the results of Table 2.\n\n4. Previous work [3] proposed the initialization of studentNet for compressing GAN. This paper claims that the randomly initialized studentGAN can not learn well from teacherGAN. Thus, they provide a two-step training approach. There are some concerns: 1) Since there is a competitive relationship between GAN loss and distillation loss in the early training stage, what happens if you gradually reduce the weight of GAN loss for training (studentGAN with random initialization)? In this case, the style module could also be compressed instead of directly inheriting from teacherNet.\n\n[3] Z. Zhang, et al. P-KDGAN: Progressive knowledge distillation with GANs for one-class novelty detection.\n\n\n**Minor comments:**\n\n1. Table 4 seems not very clear to me. It has many horizontal lines to divide different experimental results, which makes it difficult to read.\n\n\n**Questions during rebuttal period:**\n\n1. Please address and clarify the cons above.\n\n2. In table 1, how do you explain the results, 9.41 vs. 9.42. It seems not compatible with the paper’s claim. In addition, are these numbers the average of several experiments?\nIs the competitive relationship between GAN loss and distillation loss not alleviated in the late training phase? Please provide results similar to Figure 1 (c).\n\n\n**Some typos:**\n\n1. Section 4.2: 0.65 and 0.31 -> 0.65 and 0.41\n", "summary_of_the_paper": "The paper identifies a key factor that affects the knowledge distillation on unconditional GANs, in particular the StyleGAN2 model: the output discrepancy between teacherNet and studentNet. It proposes that the initialization of the StyleGAN2’ style module causes the above problem to be more serious. To verify their claims, the authors show detailed analysis and comparative experiments. They also propose a novel latent-direction-based loss to further improve the distillation. ", "summary_of_the_review": "Please see the content in the main review. I tend to recommend ICLR 2022 to reject this paper. If my concerns can be well solved, I will increase my score.\n\n\n--------------------------- After Rebuttal --------------------------- \n\nThanks for your detailed feedback.\n\nI totally agree with the authors that the discrepancy problem is different between classification and unconditional generation. The discussion in the introduction and results in Figure 2 and Table 1 can well demonstrate their claims.\n\nThe results in Q4 should be added in the paper.\n\nHowever, as other reviewers said, the contribution is limited. The proposed compression method is only verified in one model, StyleGANv2. Thus, I encourage the authors to explore more modules to improve the influence.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "F6_Y-pxUiQE", "reviewer": "Reviewer_F96C", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "#### Strengths:\n* important topic (generative model compression)\n* the paper is relatively easy to follow\n* state of the art model considered together with a standard dataset in the experimental section\n\n#### Weaknesses:\n* the novelty appears incremental and is very tailored to a single architecture (StyleGAN2)\n* the validation is not convincing (a single state of the art architecture is considered and results are reported on a single dataset for this architecture), improvements seem marginal\n* limitations of the proposed approach are not discussed\n\n#### Details on the above-mentioned concerns:\nThe technical novelty of this paper appears somewhat overstated. The proposed initialization seems borrowed from the pruning-based literature in GAN compression, as stated in the paper \"Note that pruning is only applied to the convolution backbone C and the style module S is kept unchanged\". If the style module remains identical to the one used in the pruning-based literature, then the randomly initialization of the convolutional layers of the student network would remain the potential benefit of the proposed approach, as one would have presumably more options to consider in terms of architecture design. Moreover, although the paper argues rather strongly against pixel-wise losses and perceptual losses in the unconditional GAN setting, those losses are still preserved in the model leading to the best performance (perhaps based on the observation on the importance of the style module). The paper also introduces a loss term inspired by the relational distillation introduced in previous work. While the proposed loss appears sound, the improvements it brings seem rather small. Unfortunately, there is no standard deviations reported in the empirical results, making it hard to assess whether those apparently small improvements are significant.\n\n* Conditional GANs is a very broad term, and whereas output discrepancy may be a limited problem in some settings (e.g. the discussed image to image translation task), other settings will likely experience diverse outputs given some condition (e.g. class conditional GANs can generate diverse outputs given a class label). The paper should be more clear wrt the use of this terminology. \n* \"... if the generated image of the student are totally different from that of teacher for the same input, Lrgb and Lpips will result in images that are slightly closer to teacher but with much less realism.\" Perhaps this is a more philosophical question, but from the perspective of model compression, should we expect student/teacher networks to output the same images given a noise vector (student network emulating the mapping learnt by the teacher network) or should we expect the student network to be able to mimic the distribution of images of the teacher network? If the former, the experiments that assume that teacher and student will result in very different generations should be better motivated, why should we care about this scenario? If the latter, it is unclear why the paper does not consider distilling knowledge via a discriminator loss between student and teacher generations.\n* The solution is very tailored to StyleGAN2. It would be worth exploring other state of the art generative models. For example, it has been shown that StyleGAN2-like architectures do not generalize well to large scale and diverse datasets such as ImageNet. What about leveraging BigGAN in an unconditional scenario? (see e.g. https://arxiv.org/abs/1903.02271, https://arxiv.org/abs/2012.02162, https://arxiv.org/abs/2109.05070 for unconditional BigGAN options)\n* The experimental validation could be strengthened by considering additional datasets (e.g. ImageNet or COCO-Stuff).\n* Also, what happens if the unconditional problem is transformed into a conditional one where the condition comes from e.g. clustering the data points (see e.g. https://arxiv.org/abs/1709.07359, https://arxiv.org/abs/1712.04407, https://arxiv.org/abs/2006.10728)?\n* \"Specifically, we select two students with one inheriting teacher’s style module and the other being randomly initialized for the style module. The convolution backbones are both randomly initialized. We keep the style modules of two students frozen and only train their remaining convolution layers.\" Is this the case for the results reported in the rest of the paper? It would seem natural to allow the style module which is randomly initialized to be updated during training. How would that compare to the 2 stage training in terms of results?\n* Limitations of the work should be acknowledged and discussed.\n\n#### Additional comments:\n* Table 1 does not show the full picture. It would be beneficial to include and discuss results for the missing configurations (e.g. LPIPS is never considered on its own, LPIPS + LD is not considered, RGB+LD is only sometimes considered, etc).\n* Although the paper argues that improvements are not marginal, it would be best to assess their significance. Please include std to all results, given the small improvements reported, it is very hard to assess the significance of the results.\n* It would be beneficial to report the teacher FID in all tables to ground the discussion further.\n* In Table 2, it is unclear why report l1 between s(z) for teacher/student since this same quantity is what appears to be optimized in the two stage training. Therefore, it would not appear to be surprising that the gap is narrowed for the 2 stage training when using L1 as distance metric.\n* It is good practice to explain how FID was computed: What was the reference data used? How many samples (both from the reference and generated images) were used to compute the metric?\n* In Table 4, LPIPS the higher the better (arrow is upside down).\n* Please double check bolding in Table 4, sometimes \"ours\" results are bolded but CAGAN achieves slightly higher performance (e.g. LPIPS).\n* Given the topic of the paper, it would seem appropriate to have smaller tables and gaining space to include student/teacher generations side to side for a grid of randomly selected images.\n\n", "summary_of_the_paper": "This paper tackles the problem of image unconditional generative model compression by leveraging the knowledge distillation framework. The paper highlights some challenges of applying KD out of the box on unconditional generative models and proposes to copy a subset of pretrained parameters (style modules) directly from the teacher to the student network. The paper further proposes an additional loss term, which preserves semantic relations in the latent spaces. Experiments are conducted using a StyleGAN2 backbone on the FFHQ dataset.", "summary_of_the_review": "Overall, although the topic covered in this paper is relevant to the ICLR community, the reviewer does not find the paper ready for acceptance. In particular, the contributions seem rather narrow and the empirical validation is not compelling. For details, see the main review.\n\n**Post-rebuttal update:** The reviewer thanks the authors for the feedback which addressed some of their concerns. However, some the reviewer's concerns wrt the stated contributions remain. More precisely, claiming the initialization strategy (already adopted in pruning methods) as contribution appears a bit stretched, and the compression approach being tailored specifically to StyleGANv2 makes the contribution rather narrow. There might be value in the analysis and lessons learnt wrt compressing StyleGANv2, but considering the limited results and their unknown generalizability to other models and more challenging datasets, the reviewer considers that the paper is not ready for acceptance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "-r6xfDFKn6", "reviewer": "Reviewer_MFc1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- This could be a good study paper for the unconditional GAN. The authors conducted several investigations and show the effect of the style module. Both the proposed student initialization and the latent-direction-based distillation are new and interesting for the unconditional GAN distillation.\n- The proposed method achieves better performance than the recent SOTA, including GAN-slimming and CAGAN, on FFHQ dataset.\n- The proposed initialization is well-motivated and very straightforward. Also, the paper is well-written and easy to follow.\n\nWeakness:\n- To keep the final performance, the style module architecture cannot be changed. Although style module is pretty light weighted, it is a potential limitation.\n- Most experiments are conducted on human face domain (FFHQ dataset). In the current form, it is unclear whether the proposed method can work well for other domains, like animals, or scenery images.", "summary_of_the_paper": "The authors firstly conduct investigations and show that it is important to make sure the consistency between the style modules of both teacher and student networks. Based on their observation, a simple yet effective initialization is proposed -- initialize the student style module with that of the teacher one. The authors further compute the latent directions and use that for augmentation. Overall the proposed method is well-motivated and interesting. The proposed method outperforms GAN-slimming and CAGAN. ", "summary_of_the_review": "I think it is a good candidate that could bring contributions to our community. Although the topic is a bit narrow to unconditional GAN, the experiments are thorough. Most of their claimed are supported by their experiments.\n\n--------------------------- After Rebuttal ---------------------------\n\n\nThank you for the feedback. It is good to see more experiments such as the new results on LSUN church dataset. The authors have addressed my questions. However, I agree with other reviewers that the contribution is a bit narrow (or limited). I think this paper is really borderline, and I have no objection for acceptance/rejection.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a method for compressing unconditional generative models by leveraging a knowledge distillation framework. Two reviewers consider the paper slightly above the acceptance threshold for the interesting topic studied in the paper and the simplicity of the method. However, the other three reviewers consider the paper below the acceptance threshold with two reviewers rating the paper slighting below the acceptance threshold and one reviewer rating the paper as not good enough. Several issues were raised, including that the paper only contains results from one unconditional model (StyleGAN2) and that the presented results are not convincing enough. Consolidating the reviews and the rebuttal, the meta-reviewer found the concern raised by the reviewers justified. It would be more ideal if the paper can present results on different unconditional models and more datasets. The authors are encouraged to incorporate the reviewers' feedback to make the paper stronger for a future venue.
Decision: Reject