Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This work revisits the role of discriminator in GAN compression and finds that the capacity mismatch when compressing the generator with a fixed discriminator will lead to model collapse. To this end, this work proposes a generator-discriminator co-operative compression framework to automatically adapt the size of discriminator to maintain the Nash equilibrium with the lightweight generator. The extensive evaluation results validate the superiority of the proposed framework over SOTA GAN compression methods.
main_review: ## Originality and significance
This work identifies the model collapse problem in GAN compression caused by a fixed discriminator which is commonly adopted in previous GAN compression works. The proposed framework can effectively tackle this challenge and achieve new SOTA performance in GAN compression based on extensive experiments. Such simple but effective insights may help the community for future GAN compression techniques.


## Quality and clarity
This paper is logically clear and easy to follow with solid motivating analysis. The proposed framework is technically sound with impressive results. I still have the following questions for this paper:

(1) More discussions about the choice of L_{local} in Eq. (7) are expected. Is there any other potential forms to better maintain the Nash equilibrium, e.g., a weighted sum?

(2) [55] and [11] directly discard the discriminator during GAN compression and only reply on the distillation loss, which can also avoid the model collapse. A potential ablation study is to adopt the distillation loss in Eq. (10) while the student discriminator is discarded.


## After rebuttal

The author responses address my concerns. I will keep the original score, i.e., a clear accept.
limitations_and_societal_impact: The limitations are discussed in the main review section.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This work introduces a GAN compression method that exploits the role of the discriminator. The main contributions are two-fold: (i) maintain the capacity balance between the compressed generator and discriminator by learning to selectively activate the convolutional channels of the discriminator, (ii) promote the performance of the compressed generator by distilling knowledge from the teacher generator and discriminator to the student models in the feature space. The method is validated on synthesis, translation, and super-resolution tasks. 
main_review: Originality

+ To my knowledge, the idea is novel, simple, and intuitive. It may potentially open up a new perspective for GAN compression. 

+ Nevertheless, it's not the first paper exploiting/distilling the knowledge of the discriminator after training. Several recent works have explored this direction [1-4] for improved sampling. It would be good to see a discussion about that in the related work section and place the work in a broader context of GANs. 

Quality

* The proposed method looks technically sound. Both ideas of capacity balance and feature distillation are highly intuitive and well supported by the empirical results. 

* However, I'm not fully convinced by the concrete solution to capacity balance. It's known that balancing the learning of G & D is not trivial and it's common in practice that $L_G + L_D$ is not close to 0. Yet, in Eq.6 & Eq.7, the authors propose to measure and balance the capacity by using the value of $L_G$ & $L_D$. I wonder how accurate this is as an approximation of the model capacity. Also, how is it applied to other forms of adversarial loss? It would be good to see a discussion about the potential limitations. 

Clarity

* The paper is overall well-motivated and easy to follow. 

* Yet, I'm sure not if I fully got the details of online collaborative distillation. E.g., in L182, 'The online scheme optimizes the teacher model and the lightweight student model iteratively and progressively from scratch. Therefore, we do not need a pre-trained teacher model, and directly complete the distillation process in one stage.' Does it refer to learn light-weighted GANs from scratch as opposed to compression? 



[1] Discriminator Rejection Sampling, ICLR'19

[2] Metropolis-Hastings Generative Adversarial Networks, ICML'19

[3] Collaborative Sampling in Generative Adversarial Networks, AAAI'20

[4] Your GAN is Secretly an Energy-based Model and You Should use Discriminator Driven Latent Sampling, NIPS'20
limitations_and_societal_impact: The limitations discussed in the supplementary material are not convincing. They look more like the general limitations of the topic rather than the proposed method.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: In this paper, the authors revisit the role of discriminator in GAN compression and propose a selective activation discriminator to automatically activate the convolutional channels. An online collaborative distillation is adopted to guide the optimization process of the lightweight generator. Experiment results demonstrate the effectiveness of the proposed method.
main_review: Strengths
+ The idea of generator-discriminator cooperative compression scheme for GAN compression is straightforward and reasonable since the balance between generator and discriminator is crucial for GAN training.

+ The importance of discriminator’s channel-number is verified via corresponding experiments, which gives some insights for the image generation community.


Weaknesses
-	My main concern is the lack of novelty. In my view, the proposed mechanism of automatically activating convolutional channels and distillation method are incremental. In addition, the claim that “first attempt …” in line 59 is open to discussion. Some NAS-GAN related works [1][2] also consider the mutual balance between generator and discriminator and search the architectures of both networks simultaneously. The compression and NAS task are essentially the same in my opinion.

-	The ablation studies on table 3 show that the effect of each component is not obvious. Compared with other compression methods, the improvement is also not significant.

-	More analysis about the role of discriminator in GAN Compression should be given since the title of this paper is “Revisiting Discriminator in GAN Compression”. Simply discuss the effect of channel-number is not deep enough. Authors are encouraged to provide more insights.



[1] Gong et al. AutoGAN: Neural Architecture Search for Generative Adversarial Networks. In ICCV. 2019.

[2] Gao et al. AdversarialNAS: Adversarial Neural Architecture Search for GANs. In CVPR. 2020.

---- Update ----

The authors address some of my concerns. However, the novelty of the proposed methods (e.g., selective activation discriminator and global constraint) is still limited in my view, and this work only discusses the effect of channel-number. After reading other comments and author's response, I prefer to keep the original rating 5.
limitations_and_societal_impact: Yes. See Main Review.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: In this paper, the authors revisit the role of discriminator in GAN compression and design a new generator-discriminator cooperative scheme GCC for GAN compression. Specifically, they use a selective activation discriminator to automatically select the convolutional channels, aiming to help maintain the Nash equilibrium and avoid mode collapse. Besides, they also propose an online collaborative distillation method to further boost the compression performance. They achieve impressive performance on three widely-used GAN tasks--image generation, image-to-image translation, and super-resolution.
main_review: * Originality: The novelty is somewhat limited. As the authors mention in the related work, Slimmable GAN also tries to maintain the Nash equilibrium by making the discriminator width proportional to the generator width. I think the authors are just applying some NAS methods to automatically prune the discriminator, which is quite similar to Anycost GAN. Although the authors do a good job analyzing the correspondence of different generator and discriminator sizes, the basic idea behind it is not interesting.

* Quality: 

  * I am curious about why the authors need to binarize $\alpha_{i,j}$ to $I_{i,j}$. I think it is okay to make $O_{i,j}'= \alpha_{i,j}*O_{i,j}$, since they do not need to care about the efficiency of the discriminator, unlike the traditional NAS methods. In this way, they could make the discriminator more flexible and do not need to employ STE for the gradient estimation. Also, they do not need to set the threshold $\tau$.

  * I think the authors need to provide some ablation study on their global coordination constraint $L_{global}$. It is hard to tell whether only their selective activation discriminator works, only their $L_{global}$ works, or both work together. It is also possible that $L_{global}$ works alone with the original discriminator without the selective activation.

* Clarity: 

  * The paper is overall well-written, but I am confusing about the following two points:

    * The author may need to state how to select a good threshold $\tau$.

    * I think the statement of $L_{local}$ and $L_{global}$ is unclear. The authors may need to explain what is "local" and what is "global". In my understanding, they are just trying to match the loss difference of the teacher and student pair. In this way, there should not be absolute value signs in Equation (6) and (7), which means $L_{global}$ should be
      $$
      L_{global} = |(L_G^S-L_{D_{fake}}^S) - (L_G^T-L_{D_{fake}}^T)|.
      $$
      Moreover, the authors also need to verify that when the student pair have the same loss difference as the teacher pair, the student generator and discriminator maintain the optimal equilibrium.

* Significance: The paper results are good, and I think it could indeed speed up the generator significantly in real-world applications.
limitations_and_societal_impact: I think the authors have adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

Meta-review: This paper proposes a generator-discriminator co-training compression method to balance the model capacity of the lightweight generator and discriminator during GANs compression. The reviewers appreciate the idea of balancing generator and discriminator as well as comprehensive results and clear writing, although there are still concerns regarding the technical novelty and the ablation studies. The rebuttal addressed most of the reviewers' concerns and highlighted the difference from prior works (e.g., knowledge distillation for discriminator, NAS for discriminator architecture search). I agree with most reviewers and believe that the paper's contribution is significant enough, and the results are convincing. Therefore, I recommend accepting the paper. 
Decision: Accept (Poster)