Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Typical knowledge distillation loss/pipeline failed for unconditional GAN distillation, due to the output discrepancy between teacher and student model even if same inputs are fed. This paper proposed a framework for distilling unconditional GANs. The framework mainly contains two parts: 1). Inherit the weights from teacher style module for initializing student’s style module based on the observation that style module plays a crucial role in determining the outputs’ semantic information; 2). Propose a latent-direction-based distillation loss to enforce the consistency between teacher’s outputs and student’s outputs. Experiments are mainly shown by comparing with CAGAN and some variants of the proposed methods with different initialization strategies and loss functions. 
main_review: Strength:
  1. Solving the output discrepancy issue for distilling unconditional GAN is an interesting topic. 
  2. Show many ablation studies for different initialization strategies and loss functions for style module and CNN parts.

Concerns:
  1. Since the paper abstract starts with the motivation of reducing the storage and computation of deploying GAN models, I would like to see a thorough comparison between a distilled model using the proposed method and a same architecture network which is simply trained from the scratch using the original training recipe described in StyleGAN paper. Otherwise, if a same architecture can achieve the same quality by training from scratch, there is no need to do distilling any more. When I read the paper, I guessed the row 1 in Table 1 (random + random + No Mimic) might represent such experiment, but there is no discussion about how this experiment is done.  It also needs to be added to Table 4.
  2. In Section 3.2 and Figure 1(a), it is unclear to me how to find the similar student models and dissimilar student models. What type of criteria is used to define similar or not similar?
  3. The paper only shows the results on face images (the aligned FFHQ face images). How does the method work on other types of images? Especially, other types of images, like building/rooms, have more complicate data distribution than aligned face images. The semantic information of these types of images are not that well disentangled as face images. In this case, does inheriting style module still help? Does the latent-direction-based distilling loss still help? 
  4. In addition to RBG+LPIPS+LD losses, would adding adversarial loss also help? 
summary_of_the_review: Overall, the paper tries to address an interesting problem of tackling the output discrepancy issues when distilling unconditional GAN models. The experiment part provides several ablation studies. However, I have several main concerns regarding to: 1). more thorough comparison with the same architecture network trained from scratched and more discussions; 2). try the proposed method on other types of images to validate whether inheriting style modules and latent-direction-based distilling loss still help or not. 


==============After Rebuttal===========================

Thanks the authors for the clarification. I agree with other authors about the limitations of the proposed method, especially about how to extend to other unconditional GANs and other datasets. Thus, I will still give a borderline rating.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors present a method to distill StyleGAN2, an unconditional Generative adversarial network. They claim that the main challenge lies in the output discrepancy between the teacher and student. They have identified that the style module plays an important role in determining semantic information of generated images. Hence, they have proposed:
i) to initialise the student model with the inherent weights of the teacher module and keeping the remaining convolutional layers randomly initialised, and 
ii) a new distillation loss that preserves the semantic relations in latent space. 
The authors run all their experiment on StyleGAN2. 
main_review: Pros:
A- The paper tackles an interesting problem
B- The authors are referring and comparing to key previous work (CAGAN). 


Cons:
C- My biggest concern with the current draft is the experiment section and claims. While the authors have done a good job in comparing to previous work (CAGAN), the comparison is not convincing nor rigourous. Here is why:
1) Standard deviation is missing in all tables. The authors need to run several train for each experiment  and report their average and standard deviation. All numbers between CAGAN and the proposed method are so close that a standard deviation is required to compare these methods. It is a common practice in GAN literature. I am quite surprise that the authors have omitted it. 
2) The authors claim (in the conclusion) that they outperform previous work by a 
“large margin”. It is not supported by any evidence. I am not going to list all the   quantitative numbers but when the gains are “0.008 or 0.029 etc…” on a metric where previous work had one order of magnitude more gain, to my humble opinion, these are not large. 
3) While the quantitative results can not be interpreted without the standard deviation,  qualitatively comparison are also not convincing.

D- Finally, the technical depth of the paper and contributions are also limited. I am in favour of simple yet effective ideas. Yet, they need to be “effective”. Here, following my comments in the experiment section, it seems that the ideas are simple but not necessarily effective. 

Here are some typos to be fixed: 
- “sduent” -> student in Sec 2.
- “0.31” -> 0.41 in Section 4.2
summary_of_the_review: Given C and D comments, I can not accept the paper “as is”. Critical points are missing and conclusions can not be drawn without them. I hope that the authors will be able to address my comments. I thank them for their work and highly encourage them to continue this line of work and re-submit. Best.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper tackles the problem of image unconditional generative model compression by leveraging the knowledge distillation framework. The paper highlights some challenges of applying KD out of the box on unconditional generative models and proposes to copy a subset of pretrained parameters (style modules) directly from the teacher to the student network. The paper further proposes an additional loss term, which preserves semantic relations in the latent spaces. Experiments are conducted using a StyleGAN2 backbone on the FFHQ dataset.
main_review: #### Strengths:
* important topic (generative model compression)
* the paper is relatively easy to follow
* state of the art model considered together with a standard dataset in the experimental section

#### Weaknesses:
* the novelty appears incremental and is very tailored to a single architecture (StyleGAN2)
* the validation is not convincing (a single state of the art architecture is considered and results are reported on a single dataset for this architecture), improvements seem marginal
* limitations of the proposed approach are not discussed

#### Details on the above-mentioned concerns:
The technical novelty of this paper appears somewhat overstated. The proposed initialization seems borrowed from the pruning-based literature in GAN compression, as stated in the paper "Note that pruning is only applied to the convolution backbone C and the style module S is kept unchanged". If the style module remains identical to the one used in the pruning-based literature, then the randomly initialization of the convolutional layers of the student network would remain the potential benefit of the proposed approach, as one would have presumably more options to consider in terms of architecture design. Moreover, although the paper argues rather strongly against pixel-wise losses and perceptual losses in the unconditional GAN setting, those losses are still preserved in the model leading to the best performance (perhaps based on the observation on the importance of the style module). The paper also introduces a loss term inspired by the relational distillation introduced in previous work. While the proposed loss appears sound, the improvements it brings seem rather small. Unfortunately, there is no standard deviations reported in the empirical results, making it hard to assess whether those apparently small improvements are significant.

* Conditional GANs is a very broad term, and whereas output discrepancy may be a limited problem in some settings (e.g. the discussed image to image translation task), other settings will likely experience diverse outputs given some condition (e.g. class conditional GANs can generate diverse outputs given a class label). The paper should be more clear wrt the use of this terminology. 
* "... if the generated image of the student are totally different from that of teacher for the same input, Lrgb and Lpips will result in images that are slightly closer to teacher but with much less realism." Perhaps this is a more philosophical question, but from the perspective of model compression, should we expect student/teacher networks to output the same images given a noise vector (student network emulating the mapping learnt by the teacher network) or should we expect the student network to be able to mimic the distribution of images of the teacher network? If the former, the experiments that assume that teacher and student will result in very different generations should be better motivated, why should we care about this scenario? If the latter, it is unclear why the paper does not consider distilling knowledge via a discriminator loss between student and teacher generations.
* The solution is very tailored to StyleGAN2. It would be worth exploring other state of the art generative models. For example, it has been shown that StyleGAN2-like architectures do not generalize well to large scale and diverse datasets such as ImageNet. What about leveraging BigGAN in an unconditional scenario? (see e.g. https://arxiv.org/abs/1903.02271, https://arxiv.org/abs/2012.02162, https://arxiv.org/abs/2109.05070 for unconditional BigGAN options)
* The experimental validation could be strengthened by considering additional datasets (e.g. ImageNet or COCO-Stuff).
* Also, what happens if the unconditional problem is transformed into a conditional one where the condition comes from e.g. clustering the data points (see e.g. https://arxiv.org/abs/1709.07359, https://arxiv.org/abs/1712.04407, https://arxiv.org/abs/2006.10728)?
* "Specifically, we select two students with one inheriting teacher’s style module and the other being randomly initialized for the style module. The convolution backbones are both randomly initialized. We keep the style modules of two students frozen and only train their remaining convolution layers." Is this the case for the results reported in the rest of the paper? It would seem natural to allow the style module which is randomly initialized to be updated during training. How would that compare to the 2 stage training in terms of results?
* Limitations of the work should be acknowledged and discussed.

#### Additional comments:
* Table 1 does not show the full picture. It would be beneficial to include and discuss results for the missing configurations (e.g. LPIPS is never considered on its own, LPIPS + LD is not considered, RGB+LD is only sometimes considered, etc).
* Although the paper argues that improvements are not marginal, it would be best to assess their significance. Please include std to all results, given the small improvements reported, it is very hard to assess the significance of the results.
* It would be beneficial to report the teacher FID in all tables to ground the discussion further.
* In Table 2, it is unclear why report l1 between s(z) for teacher/student since this same quantity is what appears to be optimized in the two stage training. Therefore, it would not appear to be surprising that the gap is narrowed for the 2 stage training when using L1 as distance metric.
* It is good practice to explain how FID was computed: What was the reference data used? How many samples (both from the reference and generated images) were used to compute the metric?
* In Table 4, LPIPS the higher the better (arrow is upside down).
* Please double check bolding in Table 4, sometimes "ours" results are bolded but CAGAN achieves slightly higher performance (e.g. LPIPS).
* Given the topic of the paper, it would seem appropriate to have smaller tables and gaining space to include student/teacher generations side to side for a grid of randomly selected images.


summary_of_the_review: Overall, although the topic covered in this paper is relevant to the ICLR community, the reviewer does not find the paper ready for acceptance. In particular, the contributions seem rather narrow and the empirical validation is not compelling. For details, see the main review.

**Post-rebuttal update:** The reviewer thanks the authors for the feedback which addressed some of their concerns. However, some the reviewer's concerns wrt the stated contributions remain. More precisely, claiming the initialization strategy (already adopted in pruning methods) as contribution appears a bit stretched, and the compression approach being tailored specifically to StyleGANv2 makes the contribution rather narrow. There might be value in the analysis and lessons learnt wrt compressing StyleGANv2, but considering the limited results and their unknown generalizability to other models and more challenging datasets, the reviewer considers that the paper is not ready for acceptance.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper identifies a key factor that affects the knowledge distillation on unconditional GANs, in particular the StyleGAN2 model: the output discrepancy between teacherNet and studentNet. It proposes that the initialization of the StyleGAN2’ style module causes the above problem to be more serious. To verify their claims, the authors show detailed analysis and comparative experiments. They also propose a novel latent-direction-based loss to further improve the distillation. 
main_review: **Pros:**

1. The output discrepancy issue pointed out by the paper is a real problem for distillation, which does not only exist in the StyleGAN2 compression (references [1] and [2]). The valuable conclusion is that this paper provides the solutions for the StyleGAN2 compression. Based on their results, we know that the style module can not be compressed, while CNN backbone can be compressed. Because the style module of the student should inherit from the teacher.

2. The illustrated proof and result are convincing. Section 3.2 shows the following two phenomena: 1) The more similar output between studentNet and teacherNet makes studentNet generate more realistic images. 2) The dissimilar outputs cause GAN loss and distillation loss to have a competitive relationship, thereby affecting the distillation. 

3. This paper provides comprehensive experiments, including both qualitative analysis and quantitative results, to show the effectiveness of the proposed framework. The paper is well written, and the authors analyze the reasons step by step and propose reasonable solutions.

[1] J. Han, et al. Fixing the Teacher-Student Knowledge Discrepancy in Distillation.

[2] B. Liu, et al. MetaDistiller: Network Self-Boosting via Meta-Learned Top-Down Distillation.


**Cons:**

1. The paper didn’t introduce the difference between the image generation task and recognition task in the knowledge discrepancy problem, and the inspiration of the proposed method on the use of knowledge distillation in other fields. Otherwise, the proposed method is too specific.

2. Based on the results of the references [1] and [2], the output discrepancy issue also exists in the classification task, although there is enough supervised loss for the training of studentNet. Thus, it seems that the above issue also exists in the uncGANs distillation, which is contrary to the authors’ claim in the 3rd paragraph of the introduction. In order to prove the correctness of the hypothesis in the paper, there should be detailed theoretical analysis or experimental results.
In section 3.3, the counter-evidence provided by the authors is simple and not convincing. Of course, a well-trained Net and random initialized Net generate different latent space W, and the W of the latter Net is certainly not linearly separable. And, for two different sizes of style modules, the conclusion is the same as above.

3. Insufficient comparative experiments. 1) For Figure 3, there are no results for different initialized CNN backbone. Without them, how can one demonstrate that the style module plays a more important role in the distillation?  And in Table 1, what about the results on the random-inherit setting. 2) The experimental results of tuning hyperparameters should be added in the paper (\lambda_GAN, \lambda_rgb, \lambda_lpips, and \lambda_LD).  The authors should provide more details of them under different settings, whether they are with the same values in the results of Table 2.

4. Previous work [3] proposed the initialization of studentNet for compressing GAN. This paper claims that the randomly initialized studentGAN can not learn well from teacherGAN. Thus, they provide a two-step training approach. There are some concerns: 1) Since there is a competitive relationship between GAN loss and distillation loss in the early training stage, what happens if you gradually reduce the weight of GAN loss for training (studentGAN with random initialization)? In this case, the style module could also be compressed instead of directly inheriting from teacherNet.

[3] Z. Zhang, et al. P-KDGAN: Progressive knowledge distillation with GANs for one-class novelty detection.


**Minor comments:**

1. Table 4 seems not very clear to me. It has many horizontal lines to divide different experimental results, which makes it difficult to read.


**Questions during rebuttal period:**

1. Please address and clarify the cons above.

2. In table 1, how do you explain the results, 9.41 vs. 9.42. It seems not compatible with the paper’s claim. In addition, are these numbers the average of several experiments?
Is the competitive relationship between GAN loss and distillation loss not alleviated in the late training phase? Please provide results similar to Figure 1 (c).


**Some typos:**

1. Section 4.2: 0.65 and 0.31 -> 0.65 and 0.41

summary_of_the_review: Please see the content in the main review. I tend to recommend ICLR 2022 to reject this paper. If my concerns can be well solved, I will increase my score.


--------------------------- After Rebuttal --------------------------- 

Thanks for your detailed feedback.

I totally agree with the authors that the discrepancy problem is different between classification and unconditional generation. The discussion in the introduction and results in Figure 2 and Table 1 can well demonstrate their claims.

The results in Q4 should be added in the paper.

However, as other reviewers said, the contribution is limited. The proposed compression method is only verified in one model, StyleGANv2. Thus, I encourage the authors to explore more modules to improve the influence.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors firstly conduct investigations and show that it is important to make sure the consistency between the style modules of both teacher and student networks. Based on their observation, a simple yet effective initialization is proposed -- initialize the student style module with that of the teacher one. The authors further compute the latent directions and use that for augmentation. Overall the proposed method is well-motivated and interesting. The proposed method outperforms GAN-slimming and CAGAN. 
main_review: Strengths:
- This could be a good study paper for the unconditional GAN. The authors conducted several investigations and show the effect of the style module. Both the proposed student initialization and the latent-direction-based distillation are new and interesting for the unconditional GAN distillation.
- The proposed method achieves better performance than the recent SOTA, including GAN-slimming and CAGAN, on FFHQ dataset.
- The proposed initialization is well-motivated and very straightforward. Also, the paper is well-written and easy to follow.

Weakness:
- To keep the final performance, the style module architecture cannot be changed. Although style module is pretty light weighted, it is a potential limitation.
- Most experiments are conducted on human face domain (FFHQ dataset). In the current form, it is unclear whether the proposed method can work well for other domains, like animals, or scenery images.
summary_of_the_review: I think it is a good candidate that could bring contributions to our community. Although the topic is a bit narrow to unconditional GAN, the experiments are thorough. Most of their claimed are supported by their experiments.

--------------------------- After Rebuttal ---------------------------


Thank you for the feedback. It is good to see more experiments such as the new results on LSUN church dataset. The authors have addressed my questions. However, I agree with other reviewers that the contribution is a bit narrow (or limited). I think this paper is really borderline, and I have no objection for acceptance/rejection.
review_title: null

Meta-review: The paper proposes a method for compressing unconditional generative models by leveraging a knowledge distillation framework. Two reviewers consider the paper slightly above the acceptance threshold for the interesting topic studied in the paper and the simplicity of the method. However, the other three reviewers consider the paper below the acceptance threshold with two reviewers rating the paper slighting below the acceptance threshold and one reviewer rating the paper as not good enough. Several issues were raised, including that the paper only contains results from one unconditional model (StyleGAN2) and that the presented results are not convincing enough. Consolidating the reviews and the rebuttal, the meta-reviewer found the concern raised by the reviewers justified. It would be more ideal if the paper can present results on different unconditional models and more datasets. The authors are encouraged to incorporate the reviewers' feedback to make the paper stronger for a future venue.
Decision: Reject