Revisiting Discriminator in GAN Compression: A Generator-discriminator Cooperative Compression Scheme
Authors: ShaoJie Li
Jie Wu
Xuefeng Xiao
Fei Chao
Xudong Mao
Rongrong Ji
Venue: NeurIPS 2021 Poster (2021)
Keywords: Generative Adversarial Networks
Cooperative Compression
Online Distillation

Abstract: Recently, a series of algorithms have been explored for GAN compression, which aims to reduce tremendous computational overhead and memory usages when deploying GANs on resource-constrained edge devices. However, most of the existing GAN compression work only focuses on how to compress the generator, while fails to take the discriminator into account. In this work, we revisit the role of discriminator in GAN compression and design a novel generator-discriminator cooperative compression scheme for GAN compression, termed GCC. Within GCC, a selective activation discriminator automatically selects and activates convolutional channels according to a local capacity constraint and a global coordination constraint, which help maintain the Nash equilibrium with the lightweight generator during the adversarial training and avoid mode collapse. The original generator and discriminator are also optimized from scratch, to play as a teacher model to progressively refine the pruned generator and the selective activation discriminator. A novel online collaborative distillation scheme is designed to take full advantage of the intermediate feature of the teacher generator and discriminator to further boost the performance of the lightweight generator. Extensive experiments on various GAN-based generation tasks demonstrate the effectiveness and generalization of GCC. Among them, GCC contributes to reducing 80% computational costs while maintains comparable performance in image translation tasks.

Revisiting Discriminator in GAN Compression: A Generator-discriinaator Cooperative Compression Scheme
Shaojie Li"; Jie Wu', Xuefeng Xiao², Fei Chao', Xudong Mao', Rongrong J
Media Analysis and Computing Lab, Department of Artificial Intelligence, School of Informaties, Xiamen University 2 ByteDance Inc 'Institute of Artificial Intelligence, Xiamen University
Abstract
Recently, series of algorithms have been explored for GAN compression, which aims reduce tremendous computational overhead and memory usages when deploying GANs on resource-constrainee edge devices. However, most of the existing GAN compression work only focuses how compress the generator, while fails to take the discriminator into account. In this work, we revisit the role of discriminator in GAN compression and design a novel minoritorddiscrmminato co- operative compression scheme for GAN compression, termed GCC. Within GCC, a selective activation discriminator automatically selects and activates convolutional channels according to local capacity constraint and global coordination con- straint, which help maintain the Nash equilibrium with the lightweight generator during the adversarial training and avoid mode collapse. The original generator and discriminator are also optimized from scratch, to play as teacher model progressively refine the pruned generator and the selective activation discriminator. A novel online collaborative distillation scheme is designed take full advantage of the intermediate feature of the teacher generator and discriminator to further boost the performance the lightweight generator. Extensive experiments on vari- ous GAN-based generation tasks demonstrate the effectiveness and generalization GCC. Among them, GCC contributes to reducing 80% computational costs while maintains comparable performance in image translation tasks. Our code and models are available at: hhtps: : //t/://iiitt github..... cwwm SSLeo/GCCCCC
Introduction
Generative Adversarial Networks (GANs) have been widely popularized in diverse image synthesis
tasks, such as image generation [66 50], image translation [64] (30] and super resolution [35] 59].
However, the ultra-high computational and memory overhead of GANs hinder its deployment on resource-conssraiined edge devices. To alleviate this issue, a series of traditional model compression
algorithms, i.e., network pruning [8, 36), weight quantization [39 26, low-rank decomposition [I0]
19], knowledge distillation 22 52 and efficient architecture design 29 55
have been applied to
reduce calculational consumptions of GANs.
attempted directly employ network pruning methods compress the generator but obtain unimpressive results, as shown in Figure ((a) The similar phenomenon also occurred in SAGAN shown in Appendix A potential reason that these methods failed take into account the generator and the discriminator must follow the Nash equilibrium state to avoid the mode
Previous work [56] 12
"This work was done when Shaojie Li was a intern at Bytedance Inc.
t "oorrsppndiigg Author: m mniixmu.edd.cc edu.en
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

real fake
D_real
fake
G_GAN
Input
Original
1/4 Generator
4k
2k
iters
(a) Visualization of mode collapse (b) Original generator + original (c) 1/4 channels width generator
discriminator
original discriminator
Figure I: Illustration of model collapse phenomenon. The experiment is conducted on Pix2Pix (30 based the Cityscapes [8] dataset. (a) shows the influence of mode collapse in generating images. Each image generated by the 1/4 channels width generator appears the forgery trace, which is marked
by the red box. (b) and (c) show loss curves of the original generator and the 1/4 channels width generator with the original discriminator, respectively.
1/2 channels
1/2 channels(GCC)
1/2 channels
1/2 channels(GCC)
1/4 channels
1M channels(GCC)
1/4 channels
1/4 channels(GCC))
42
70
60
38
50
36
40
30
32
32
24
16
128
96
80
64
48
32
24
16
channel number discriminntor
channel number In discriminator
(a) SAGAN
(b) Pix2Pix
Figure The performance generators with different compression ratios on various channel number discriminators. (a) is the result of SAGAN 63 on CelebA 441 dataset. (b) is the result of Pix2Pix on Cityscapes dataset. "1/2 channels" means that we reduce network width of generator network to 1/2 of the original size. GCC uses selective activation discriminator without distillation.
collapse in the adversarial learning. In other words, these methods simply compress the generator while maintaining the original capability of the discriminator, which resulted in breaking the balance in adversarial Icarning. As shown in Figure [(b) and [[c) when compressing the generator, the loss of the discriminator gradually tends to zero, such a situation indicates that the capacity of the discriminator significantly surpasses that of the lightweight generator. Furthermore, the capacity imbalance between the generator and discriminator leads to the mode collapse problem [47].
In order to retain the Nash equilibrium between the lightweight generator and discriminator, must revisit the role of the discriminator the procedure of GAN compression. straightforward method is to reduce the capacity of the discriminator to re-maintain the Nash equilibrium between it and the lightweight generator. Therefore, we train a lightweight generator against the discriminators with different channel numbers. The results are shown in Figure 2 we obtain the following observations: i) The channel numbers in the discriminator significantly influence the performance of the lightweight generator; ii) The discriminator with the same channel compression ratio as the generator may not get the best result. iii) The optimal value of channel numbers for discriminators is task-specific. In short, is arduous to choose appropriate channel number for the discriminator in different tasks.
To solve the above issues, design generator-discrimiinato cooperative compression scheme, term GCC in this paper. GCC consists of four sub-networks, i.e., the original uncompressed generator and discriminator, pruned generator, and selective activation discriminator. GCC selectively activates

---

partial convolutional channels of the selective activation discriminator, which guarantees the whole optimization process to maintains in the Nash equilibrium stage. In addition, the compression of the generator will damage its generating ability, and even affect the diversity of generated images, resulting in the occurrence of mode collapse. GCC employs an online collaborative distillation that combines the intermediate features of both the original uncompressed generator and discriminator and then distills them to the pruned generator. The introduction of online collaborative distillation boosts the lightweight generator performance for effective compression. The contributions of GCC can be summarized as follows:
To the best of our knowledge, offer the first attempt to revisit the role of discriminator in GAN compression. We propose a novel selective activation discriminator to automatically select and activate the convolutional channels according local capacity constraint and global coordination constraint. The dynamic equilibrium relationship between the original models provides a guide choose the activated channels in the discriminator.
A novel online collaborative distillation is designed to simultaneously employ intermediate features of teacher generator and discriminator to guide the optimization process of the lightweight generator step by step. These auxiliary intermediate features provide more complementary information into the lightweight generator for generating high-fidelity images.
Extensive experiments on various GAN-based generation tasks (i.e., image generation, image-to-image translation, and super-resolution) demonstrate the effectiveness and gener- alization of GCC. GCC contributes to reducing 80% computational costs while maintains comparable performance in image translation.
2
Related Work
Generative Adversarial Networks, Generative Adversarial Network, i.e., GANs 16 has attracted intense attention and has been extensively studied for a long time. For example, DCGAN 50 greatly enhanced the capability of GANs by introducing the convolutional layer. The recent works [7 employed a series of advanced network architecture to further improve the fidelity of the generated images. In addition, several novel loss functions [7 2 45 were designed to stabilize the adversarial training stage of GANs. popular research direction of GANs is image-to-image translation. The image-to-image translation is designed to transfer images from the source domain to the target domain while retaining the content representation of the image. Image-to-image translation is widely used in computer vision and image processing, such as semantic image synthesis [30, 48). style transfer [64 I, super resolution
33 63 4
35
image painting (49 655 and so on.
Mode Collapse. Mode collapse is the main catastrophic problem during the adversarial training stage of GANs, which leads to the diversity of generated images are monotonous, only fitting part of the real data distribution. To address this issue, recent efforts were focus on new objective functions [47 [7 34), architecture modifications (5) [5] and mini-batch discrimination [54] (3). Actually, the capacity gap between the generator and the discriminator is a more essential reason for mode collapse. In this paper, we provide a different perspective to eliminate the capacity gap, we select the convolutional channel to be activated in the discriminator according to local capacity constraint and a global coordination constraint.
GAN Compression. To reduce huge resource consumption during the inference process of GANs, great efforts have emerged in the field of GAN compression in the recent two years. The search- based methods [56] 37 12 23 440
58) exploited neural architecture search and genetic algorithm to obtain a lightweight generator. However, they fell into the trap of manual pre-defined search space and huge search costs. The prune-based methods [38 31 57 61] directly pruned lightweight generator architecture from the original generator architecture. However, these works failed to take discriminator pruning into account, which would seriously destroy the Nash equilibrium between the generator and the discriminator. The discriminator-fiee methods [5i [2] directly use the large GAN model as a teacher to distill the lightweight generator without discriminator, which also achieved good performance. Slimmable GAN [23) correspondingly shrank the network width of the discriminator with the generator. However, as shown in Fig. 2 there is no linear between the channel numbers of the optimal discriminator and generator. Anycost GAN 40 proposed a generator-condirione discriminator, which generates the discriminator architecture via passing the generator architecture
3

---

Inference
Intermediate Feature
Output Concept
Original
Original
One-shor Pruning
Collaborntive Distillation Loss
GAN Loss
Architecture Loss
Lightweight
Selective Activation
Figure 3: The proposed generator-discriminaaor cooperative compression scheme (GCC).
into the fully connected layer. However, the fully connected layer is only trained through the original GAN loss, it difficult accurately predict the optimal discriminator architecture for the lightweight generator. In this work, we design (GCC) scheme, which employ selective activation discriminator and novel online collaborative distillation to boost the performance of the lightweight generator.
cooperative compression
3 Methodology
This section, illustrate the proposed GCC framework in Figure We first briefly introduce the composition and loss function GAN in Section 3.1, Then, we present how to obtain the lightweight architecture from the original generator in Section 3.2 The selective activation discriminator that automatically chooses activated convolutional channels is illustrated in Section (33) In Section 3.44 we propose novel online collaborative distillation scheme to further improve the performance of the lightweight generator.
3.1 Preliminaries
Generative Adversarial Networks (GANs) contain two fundamental components, generator G and discriminator D. Among them, maps the input z into a fake image for cheating the discriminator, while the discriminator distinguishes the generator outputs from the real images a. The generator and discriminator are alternately optimized via the adversarial term achieve their respective optimization objectives. The optimization objectives defined as follows:
LG = Exxpp()) [c(-DD((2))]
LD Banauu (D((D((x)) + Ex)p(x) UD(D(G(=)))
(2)
DDreal
CDjake
where fG and In the loss functions of the generator and discriminator, respectively. Theoretically, the generator and discriminator maintain the Nash equilibrium when the generator and discriminator have balanced capacity, and the discriminator is cheated. In this paper, we observe from experiments that the Nash equilibrium maintained in the GAN compression stage to avoid model collapse and obtain impressive performance.
3.2 Generator Compression
In order to avoid additional computational overhead brought by the neural architecture search, we directly employ the conventional network pruning methods compress the generator. Previous

---

work 43 revealed that the core of network pruning searches for a suitable sub-architectun from the original network architecture instead of parameter inheritance. Therefore, we leverage the one-shot pruning method [36 42 to obtain the appropriate lightweight generator architecture from the original generator architecture. our experiments, we adopt the slimming [42 to prune the generator when the network contains the Batch Normalization (BN) layers. We use L1-norm pruning [36] on the contrary (without BN layers). The detailed pruning process is described in Appendix B. Finally, we only keep the architecture of the pruned generator and train it from scratch with a selective activation discriminator proposed in Section 3.3
3.3 Selective Activation Discriminator
As shown in Figure 2 the convolutional channel number in the discriminator have an essential impact on the performance of the lightweight generator. Therefore, we design selective activation discriminator to automatically choose and activate convolutional channels via learnable retention factors. We assign a learnable retention factor to each convolution kernel of the original discriminator, each retention factor represents the probability that the corresponding convolution kernel participates in the inference process. When the retention probability is larger than the given threshold, the corresponding convolution kernel is retained, otherwise it is suppressed. We denote retention factors set A {a; T= where ag {aa1, 0:2; 0inn; } IR is the retention factors of the i-th convolution layer and n; is the number of convolution kernel in the i-th layer. Given the threshold т, the retention factor Oij determines whether the j-th convolution kernel of the i-th layer activate or not:
1
if any E |T, if onj E (0,7).
Inj =
(3)
0
Then activated convolution kernel's output O'y is calculated by:
O₂ Z4 * Oij,
(4)
where Oij is the original output feature map. Since the binary function is not differentiable for auj. we employ STE (27 to estimate the gradient of arj as:
dLarch dLorch Daij
(5)
OI,
where Lurch is a loss function, and the retention factor Ozj is manually clipped to the range of [o, 1].
The learning of retention factor is considered the architecture search of the discriminator.
The selective activation discriminator is optimized according to a local capacity constraint and a global coordination constraint. On the one hand, the incremental loss gap between the generator and the discriminator will break the Nash equilibrium the adversarial training scheme, which leads to the discriminator effortlessly distinguishing the authenticity of the input images. The pruned generator and the selective activation discriminator should pursuit capacity balance to maintain the Nash equilibrium. Therefore, local capacity constraint Liocal is designed follows:
Liocal LC Dyachb
where the superscript S denotes the student (compressed model). On the other hand, the loss gap between the original model's 's combination internally reflects the relative capacity relationship between design global coordination constraint to adjust the relative ability of the student
them. Therefore, generator and discriminator to be consistent with that of the teacher combination. This constraint helps pull the loss gap between teacher and student models into close proximity. In this way, the uncompressed model is regarded as teacher model to guide the learning of retention factors to automatically adjust the discriminator capacity to match the lightweight generator. The global coordination constraint Lygobal is calculated by:
Lgtebal Ltocat 1C D.
The superscript 7' represents the teacher (uncompressed) model. Due to the instability of adversarial training, we use Exponential Moving Average (EMA) to stabilize the loss corresponds to the teacher models. In short, the optimization objective of the retention factor is defined as follows:
Lorch C. + Lglobal.
(8)
5

---

where L guarantees the basic identification function of the discriminator. Furthermore, we use the following bilevel optimization T to optimize a: i) Update discriminator weights to minimize LD for / times. ii) Update retention factor to minimize Larch. iii) Repeat step i) and step ii) until the end of training. In the optimization process of the discriminator weights, we freeze the retention factors and vice versa.
3.4 Online Collaborative Distillation
we use the original model as the teacher models guide the learning of the student diseriminator. Most of the existing work merely considered the distillation from the generator, in contrast, we also employ richer intermediate information from the discriminator perform collab- orative distillation, which provides complementary and auxiliary concepts to boost the generation performance. Therefore, we propose a novel online collaborative distillation method to take full advantage of the teacher generator and discriminator combinations to promote the lightweight gener- ator. The online scheme optimizes the teacher model and the lightweight student model iteratively and progressively from scratch. Therefore, we do not need pre- pre-trained teacher model, and directly complete the distillation process in one stage. Experiments show that online knowledge distillation is more efficient than traditional two-stage offline knowledge distillation.
In Section 3.3
Perceptual loss 32 is widely used to calculate the similarity of semantic and perceptual quality between images. It measures the semantic differences via pre-trained image classification network (such as VGG), where the extracted features are less relevant to our generation task. Mcanwhile, the teacher discriminator has a task-oriented ability to distinguish the authenticity of images. Following the work (6, we employ the teacher discriminator replace the pre-trained classification model to extract the features of the images for calculating the perceptual loss. Furthermore, the intermediate feature map of the generator has rich information for generating real-like fake images, and also add them to the calculation of perceptual loss. In this way, the similarity metric is summarized as:
d(:,)) Ym MSE(..)) y)Teature((:))
where Y'e and "/t are pre-defined hyperparameter used for balancing MSE and Texture [3 14 loss functions, respectively. Detailed formulation of Texture loss function is described in Appendix C. Thus, the collahorative distillation loss
Le
LD
Ldistill Σ E. Bamp(z) [d(fiG (z)), G. +
Σ
E Ez-p(=) [OLD] (G (G)( (x)). (z))))
(10)
where G and G. are the intermediate feature maps of the i-th chosen layer in the student and teacher generator, DJ' are the intermediate feature maps of the j-th chosen layer in the teacher discriminator. La / Ln denote the number of the generator / discrininator chosen layers. Je is a 1x learnable convolution transform layer to match the channel dimension of the teacher's feature maps.
4 Experiments
4.1 Setups
To demonstrate the effectiveness of the proposed GCC, conduct extensive experiments series of GAN based tasks:
Image Generation. We employ SAGAN 633 to input random noise to generate face images on the
CelebA [44] dataset. CelebA contains 202,599 celebrity images with a resolution of 178x22 x18. We first center crop them to 160x 160 and then resize them to 64x66 before feeding them into the model. Frechet Inception Distance (FID) 21 is adopted to evaluate the quality of the generated images. The lower score indicates the better the image generation results.
Image-to-Image Translation. Pix2Pix 30 is conditional GAN that uses Unet-based [33 generator to translate paired images. We evaluate pix2pix on the Cityscapes [ dataset, which contains 3475 German street scenes. The images are resized to 256x 256 for training. We follow the work [37] run the DRN-D-105 (6) to calculate mean Intersection over Union (mIOU) for evaluating the quality of the generated images. The higher score indicates the better the image generation performance. In addition, also conduct experiments the unpaired dataset. CycleGAN [64] leverages

---

Table 1: Quantitative comparison with the state-of-the tart GAN compression methods.
Metric
Model
Dataset
Method
MACs
Compression Ratio
mloU(t)
Original
23.45M
24.87
SAGAN
CelebA
Prune
15.45M
34.12% 34.12%
36.60
GCC (Ours)
15.45M
25.21
Original
56.800
61.53
Co-Evolution
56
13,40G
76.41%
96.15
GAN-Slimning 57
11.25G
80.19%
86.09
AutoGAN 12
6.39G
88.75%
83.60
GAN Compression
2.67G
95.30%
64.95
CycleGAN
Horse2zebra
CF-GAN 58
2.65G
95.33%
62.31
CAT 31
2.55G
95.51%
60.18
DMAD 38
2.41G
95.76%
62.96
Prune
2.40G
95.77%
145.1
GCC (Ours)
2,40G
95.77%
59.31
Original
18.6G
42.71
GAN Compression
5.66G
69.57%
40.77
CF-GAN (88
5.62G
69.78%
42.24
Pix2Pix
Cityscapes
CAT 31
5.57G
70.05%
42.53
DMAD 38
3.96G
78.71%
40.53
Prune
3.09G
82.39%
38.12
GCC (Ours)
3.09G
82.39%
42.88
Table 2: Quantitative results on the super resolution task.
Set5
Set14
BSD100
Urban100
Model
MACs
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
SRGAN
145.88G
29.88
0.86
27.07
0.76
26.35
0.72
24.77
0.76
GCC(Ours)
22.79G
30.35
0.87
27.46
0.77
26.58
0.72
24.88
0.76
ResNet 20 generator to translates horse image into zebra image on the horse2zebra dataset extracted from ImageNet [9. This dataset consists of 1,187 horse images 1,474 zebra images, and all images are resized 256x 256 for training. We use FID evaluate the quality of the generated images.
Super Resolution. We apply the proposed GCC compress SRGAN 35 which uses DenseNet- like 24] generator to upscale low-resolution images. The training set and validation set of COCO [I]] dataset combined for the training process. We randomly crop 96 x96 high-resolution images from the original images and then downsample it 4x into low-resolution images as the input of the generator. For testing, we employ Set5 (3) Set14 35). BSD100 46), and Urban 100 (25] as the standard benchmark datasets. Peak Signal-to -o-ltttt-Niiee Ratio (PSNR) (28 and Structural Similarity (SSIM) (60) are used evaluate the quality of the generated images. The higher score indicates the better the image generation performance. More implementation details are illustrated in Appendix D.
4.2 Comparison to the State of the Art
Quantitative Result. Tables I and show the comparison results of different algorithms image generation, image-to-image translation, and super-resolution. From these two tables, we can obtain the following observations: 1) In the task of image translation, our method can greatly reduce the computational costs in each task and achieve comparable performance as the original uncompressed model. For example, GCC obtains lower FID than the original model under the extremely high compression ratio CycleGAN. Specifically. GCC reduces the MACs of CycleGAN from 56.8G to 2.40G, with compression by 23.6x while FID still reduces 2.22. 2) GCC reduces the computational costs by 82.39% and achieves higher mloU than the original model, which establishes new state-of-the-ar performance for the Pix2Pix model. 3) In image generation task, GCC also shows impressive results on low MACs demands of GAN, i.e., SAGAN. Although the original SAGAN only requires 23.45M MACs to generate a 64x64 pixel image, GCC successes to reduce the MACs by 34.12%. 4) For SRGAN in the super-resolution task, GCC helps to significantly reduce 123.09G

---

Real Images
Original
GCC (Orrs
Inpur
Original
GCC(Ours)
Imput
Ground
Original
GCCXOurs)
Input
Ground Truth
Original
GCC(Our))
Figure 4: Qualitative results of SAGAN based on CelebA dataset (top). CycleGAN based on Horse2zebra dataset (middle left), Pix2Pix based Cityscapes dataset (middle right), and SRGAN based on BSD100 Urban100 dataset (bottom). Original represents the images generated by the original uncompressed generator. (Best viewed with zooming in)
MACs, with a compression ratio of 6.4x. It is worth noting that although GCC reduces such high amount of computational costs, the lightweight model achieve higher PSNR and SSIM than the original model on each test dataset. In addition, we also show the results of employing network pruning methods [36] 42 compress the generator and directly perform adversarial training with the original discriminator, abbreviated as "Prune", the experimental results demonstrate the importance and effectiveness of our proposed selective activation discriminator. summary, Our proposed GCC contributes to removes a large number of redundant calculations while maintains the comparable performances of the original model. The results provide a feasible solution for demands on super-computing or mini-computing GAN model.
Qualitative Results. Figure (iguure4deeee depiets the model generation performance each task. Qualitative results reveal that GCC contributes to generating high-fidelity images with low computational demands. More qualitative results are provided in Appendix G.
Acceleration. We also compare acceleration performance on actual hardware between the original generator and the lightweight generator Pix2Pix. The latency is measured two types of computing processors (i.e. Intel Xeon Platinum 8260 CPUs and NVIDIA Tesla T4 GPUs). The original generator takes 75.96ms the lightweight generator only uses 18. 18.68ms and 1.74ms to generate one image on CPUs and GPUs, respectively. GCC helps to achieve 4.07 x and 1.18x acceleration CPU and GPU, respectively.
generate one image CPUs and 2.06ms GPUs. However,
4.3 Ablation Study
Online Collaborative Distillation. Table shows the effectiveness of each components in the online collaborative distillation. The variant Ours w/o Online" represents the traditional two-stage distillation scheme. "Ours w/o D-distillation and "Ours w/o G-distillation' denotes that only adopt intermediate feature information of the teacher generatorrdiisriimnaaoo respectively. We can intuitively observe that the distillation method can indeed further improve the performance the model. Both MSE loss and Texture loss are helpful in the distillation process, and the combination of these two losses achieves better mIOU scores. GCC deploys the distillation process
the distillation stage,

---

Table 3: Ablation study results in online collaborative distillation.
Name
Online
Collaborative
MSE loss
Texture loss
mIOU(1)
Ours w/o distillation
39.17
Ours w/o Texture loss
X
41.80
Ours w/o MSE loss
X
V
40.07
Ours w/o Online
V
V
41.16
Ours w/o D-distillation
V
V
42.31
Ours w/o G-distillation GCC(Ours)
X
V
V
41.43
V
V
V
42.88
D real
D Jake
GAN
2k
4kc
6k
Input
1/4 Generator GCC(Ours)
(a) 1/4 channels width generator with (b) Comparison of visualization re- GCC
sults
Figure 5: Loss curve and visualization results via GCC improvement. The experiment was conducted on Pix2Pix based on the Cityscapes dataset. (Best viewed with zooming in)
into the online stage, where promotes an uncompressed teacher model progressively to gradually guide the learning of the lightweight generator and avoid the complicated two-stage distillation scheme. In Table the online distillation scheme can achieve better performance than the offline distillation. The online collaborative distillation distills the intermediate feature information from both the generator and discriminator of the teacher model. G-only distillation D-only distillation only distills one of them to the lightweight generator, which leads to declining results.
Solution of Mode Collapse. Figure T shows the phenomenon of mode collapse when compressing the generator without GCC. With the introduction of GCC, the 1/4 channels width generator can get rid of mode collapse. We show the improved loss curve and visualization results in Figure [ In Figure 5(a) the selective activation discriminator effectively balances the adversarial training with the lightweight generator. The loss gap between the 1/4 channels width generator and selective activation discriminator is evenly matched to that of the original model shown in Figure ((b)
In Figure 5(b). the 1/4 channels width generator can generate higher quality images with GCC.
Conclusion
During GAN compression, we observe that merely compressing the generator while retaining the original discriminator destroys the Nash equilibrium between them, which further results in mode collapse. In this work, we revisit the role of discriminator in GAN compression and design novel cenerator cooperative compression (GCC) scheme. To ensure normal adversarial training between the lightweight generator and discriminator, we propose a selective activation discriminator to automatically select the convolutional channel to be activated. In addition, we propose a novel online collaborative distillation that simultaneously utilizes the intermediate feature of the teacher generator and discriminator to boost the performance of the lightweight generator. We have conducted experiments on various generation tasks, and the experimental results verify the effectiveness of our proposed GCC.

---

6 Acknowledge
This work is supported by the National Science Fund for Distinguished Young Scholars NN.66025603), the National Natural Science Foundation of China (No. U1705262, No.62072386, No.62272387, No. 22062272389, No.62002305, No.61772443, No.61802324 and N0.61702136) and Guangdong Basic and Applied Basic Research Foundation (No.2019B1515120099)
References
[1] Youssef Alami Mejjati, Christian Richardt, James Tompkin, Darren Cosker, and Kwang Kim. Unsuper- vised attention-guided image-to-image translation. In Proceedings of the Advances in Neural Information Processing Systems (NeurlPS), 2018.
[2] Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In Proceedings of the International Conference on Machine Learning (ICML), pages 214-223, PMLR, 2017.
[3] Marco Bevilacqua, Aline Roumy, Christine Guillemot, and Maric Line Alberi-Morel. Low-complexity single-image super- super-ecsoutton based on nonnegative neighbor embedding. 2012.
[4] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv: 1809. 1006, 2018.
[5] Tong Che, Yanran Li, Athul Paul Jacob, Yoshua Bengio, and Wenjie Li. Mode regularized generative adversarial networks. arXiv preprint wrXivv1612.0221 2016.
[6] Hanting Chen, Yunhe Wang, Han Shu, Changyuan Wen, Chunjing Xu, Boxin Shi, Chao Xu, and Chang Xu. Distilling portable generative adversarial networks for image translation. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), pages 3585-3592, 2020.
[7] Benoît Colson, Patrice Marcotte, and Gilles Savard. An overview bilevel optimization. Annals of Operations Research, pages 235-256, 2007.
[8] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bert Schiele. The cityscapes dataset for semantic urban understanding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pages 3213-
3223, 2016.
[9] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEF Conference Computer Vision and Pattern Recognition (CVPR), pages 248-255, 2009.
[10] Emily L Denton, Wojciech Zaremba, Joan Bruna, Yann LeCun. and Rob Fergus. Exploiting linear structure within convolutional networks for efficient evaluation. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), 2014.
[11] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-leaming for fast adaptation of deep networks. In Proceedings of the International Conference Machine Learning (ICMI.), pages 1 1126-1135, 2017.
[12] Yonggan Fu, Wuyang Chen, Haotao Wang, Haoran Li, Yingyan Lin, and Zhangyang Wang. Autogan- distiller: Searching to compress generative adversarial networks. arXiv preprint arXiv:2006.081 2020.
[13] Leon Gatys, Alexander S Ecker, and Matthias Bethge. Texture synthesis using convolutional neural networks. Proceedings of the Advances Neural Information Processing Systems (NeurIPS), 2015.
[14] Leon Gatys, Alexander Eeker, and Matthias Bethge. Image style transfer using convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Patter Recognition (CVPR), pages 2414-2423, 2016.
[15] Amab Ghosh, Viveka Kulharia, Vinay P Namboodiri, Philip HS Torr, and Puneet K Dokania. Multi-agent diverse generative adversarial networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 8513-8521, 2018
[16] Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. arXiy preprint
2014.
10

---

[17] Ishaan Guirajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Improved training of wasserstein gans. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2017.
[18] Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural networks. arXiv preprint (edi:::500.026300 2015.
[19] Kohei Hayashi, Taiki Yamaguchi. Yohei Sugawara, and Shin-ichi Maeda. Exploring unexplored tensor network decompositions for convolutional neural networks. In Proceedings of the Advances in Neural Information Processing Systems (NeurPPS), 2019.
[20] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016.
[21] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS). 2017.
[22] Geoffrey Hinton. Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint wrXiv:/503.0253/1 2015.
[23] Liang Hou, Zehuan Yuan, Lei Huang, Huawei Shen, Xueqi Cheng, and Changhu Wang. Slimmable generative adversarial networks. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2021.
[24] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convo- lutional networks. In Proceedings of the IEFF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4700-4708, 2017.
[25] Jia-Bin Huang, Abhishek Singh, and Narendra Ahuja. Single image super-resolution from transformed self-cxemplars. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5197-5206, 2015.
[26] Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniy, and Yoshua Bengio. Binarized neural networks. In Proceedings of the Advances in Neural Information Processing Systems (NeurlPS), pages 4114 4122. 2016.
[27] Itay Hubara, Mathhiu Courbariaux, Daniel Soudry, Ran El- Yaniv, and Yoshua Bengio. Binarized neural networks. In Proceedings of the Advances in Neural Information Processing Systems (NeurlPS), pages 4107-4115, 2016.
[28] Quan Huynh-Thu Thu and Mohammed Ghanbari. Scope of validity of psnr in image/video quality assessment. Electronics letters, pages 800 801, 2008
[29] Forrest N landola, Song Han, Matthew W Moskewicz, Khalid Ashraf, William Dally, and Kurt Keutzer. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and< 0.5 mb model size. arXiv preprint arXiv: 1602.07360, 2016.
[30] Phillip Isola, Jun Yan Zhu, Tinghui Zhou, and Alexci A Efros. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1135-133, 2017.
[31] Qing Jin, Jian Ren, Oliver J Woodford, Jiazhuo Wang, Geng Yuan, Yanzhi Wang, and Sergey Tulyakow. Teachers do more than teach: Compressing image-to-image models. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[32] Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super- resolution. In Proceedings of the European Conference Computer Vision (ECCV), pages 694-711, 2016.
[33] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. arXiv preprint arXiv 101.1999..0166 2017.
[34] Naveen Kodali, Jacob Abernethy, James Hays, and Zsolt Kira. How to train your dragan. arXiv preprint 17220 2017.

---

[35] Christian Ledig, Lucas Theis, Ferene Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et al. Photo-realistic single image super- resolution using a generative adversarial network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4681-4690 2017.
[36] Hao Li, Asim Kadav, Igor Durdanovie, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets, arXiv preprint arXiv: (608.0878810000 10, 2016.
[37] Muyang Li, Ji Lin, Yaoyao Ding, Zhijian Liu, Yan Zhu, and Song Han. Gan compression: Efficient architectures for interactive conditional gans. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5284-5294, 2020.
[38] Shaojie Li, Mingbao Lin, Yan Wang. Mingliang Xu, Feiyue Huang, Yongjian Wu, Ling Shao, and Rongrong Ji. Learning efficient gans via differentiable masks and co-attention distillation. arXiv preprint 0mXiv:2011.08382 2020.
[39] Darryl Lin, Sachin Talathi, and Sreekanth Annapureddy, Fixed point quantization of deep convolutional networks. In Proceedings of the International Conference on Machine Learning (ICML), pages 2016.
[40] Ji Lin, Richard Zhang, Frieder Ganz, Song Han, and Yan 7hu. Anycost gans for interactive image synthesis and editing. In Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[41] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Proceedings of the European Conference on Computer Vision (ECCV), pages 740-755 755, 2014.
[42] Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learning efficient convolutional networks through network slimming. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 2736 06662244,222 2017.
[43] Zhuang Liu, Mingje Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. Rethinking the value of network pruming. arXiv preprint Xivi 1 1810. 100005070, 2018.
[44] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 3730-3738, 2015.
[45] Xudong Mao, Qing Li, Haoran Xie, Raymond YK Lau, Zhen Wang, and Stephen Paul Smolley. Least squares generative adversarial networks. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 2794 220242282222222 2017.
[46] David Martin, Charless Fowlkes, Doron Tal, and Jitendra Malik. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 4 416-223, 2001.
[47] Luke Metz, Ben Poole, David Pfau, and Jascha Sohl-Dickstein. Unrolled generative adversarial networks. arXiv preprint arXiv: 1611 163, 2016.
[48] Taesung Park, Ming- Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu. Semantic image syathesis with spatially- adaptive normalization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recogni- tion (CVPR), pages 2337- 2241-23662222 2019.
[49] Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2536-2544, 2016.
[50] Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint No.iv:/5/11004:55 2015.
[51] Yuxi Ren, Jie Wu, Xuefeng Xiao, and Jianchao Yang. Online multi-granularity distillation for gan compression. arXiv preprint arXiv:2108.00 06908, 2021.
[52] Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. arXiv preprint arXiv:1412.6553 2014.
[53] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and assister-assisted intervention, pages 234 241, 2015.
12

---

[54] Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung. Alec Radford. Xi Chen, and Xi Chen. Improved techniques for training gans, In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), 2016.
[55] Mark Sandler, Andrew Howard. Menglong Zhu, Andrey Zhmoginov. and Liang Chich Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pages 4510-4520,, 2018.
[56] Han Shu, Yunhe Wang. Xu Jia, Kai Han, Hanting Chen, Chunjing Xu, Qi Tian, Chang Xu. Co- evolutionary compression for unpaired image translation. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 3235-3244, 2019.
[57] Haotao Wang. Shupeng Gui, Haichuan Yang, Ji Liu, and Zhangyang Wang. Gan slimming: All-in-one gan compression by a unified optimization framework. In Proceedings of the European Conference on Computer Vision (ECCV), pages 54 73, 2020.
[58] Jiahao Wang, Han Shu, Weihao Xia. Yujiu Yang, and Yunhe Wang. Coarse- to to-fine searching for efficient generative adversarial networks. arXiv preprint arXiv:2/04.092.. 2021.
[59] Xintao Wang. Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong, Yu Qiao, and Chen Change Loy. Esrgan: Enhanced super-resolution generative adversarial networks, In Proceedings of the European Conference on Computer Vision (ECCV) Workshops, 2018.
[60] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Ecro P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, pages 600-612, 2004.
[61] Chong Yu and Jeff Pool. Self-super generative adversarial compression. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), pages 8235 8246, 2020.
[62] Fisher Yu, Vladlen Koltun, and Thomas Funkhouser. Dilated residual networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 472-480, 2017.
[63] Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. Self-attention generative adversarial networks. In Proceedings of the International Conference on Machine Learning (ICML), pages 7354-7363, 2019.
[64] Jun Yan Zhu, Taesung Park, Phillip Isola, and Alexei Efros. Unpaired image-to-image to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 2223-2232, 2017.
[65] Jun-Yan Zhu, Richard Zhang, Deepak Pathak, Trevor Darrell, Alexci Efros, Oliver Wang, and Eli Shechtman. Toward multimodal amage-to-image translation. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), 2017.
13


Reviews:
[{"review_id": "xZVqpO-aNde", "reviewer": "Reviewer_aeqa", "summary": "In this paper, the authors revisit the role of discriminator in GAN compression and design a new generator-discriminator cooperative scheme GCC for GAN compression. Specifically, they use a selective activation discriminator to automatically select the convolutional channels, aiming to help maintain the Nash equilibrium and avoid mode collapse. Besides, they also propose an online collaborative distillation method to further boost the compression performance. They achieve impressive performance on three widely-used GAN tasks--image generation, image-to-image translation, and super-resolution.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "* Originality: The novelty is somewhat limited. As the authors mention in the related work, Slimmable GAN also tries to maintain the Nash equilibrium by making the discriminator width proportional to the generator width. I think the authors are just applying some NAS methods to automatically prune the discriminator, which is quite similar to Anycost GAN. Although the authors do a good job analyzing the correspondence of different generator and discriminator sizes, the basic idea behind it is not interesting.\n\n* Quality: \n\n  * I am curious about why the authors need to binarize $\\alpha_{i,j}$ to $I_{i,j}$. I think it is okay to make $O_{i,j}'= \\alpha_{i,j}*O_{i,j}$, since they do not need to care about the efficiency of the discriminator, unlike the traditional NAS methods. In this way, they could make the discriminator more flexible and do not need to employ STE for the gradient estimation. Also, they do not need to set the threshold $\\tau$.\n\n  * I think the authors need to provide some ablation study on their global coordination constraint $L_{global}$. It is hard to tell whether only their selective activation discriminator works, only their $L_{global}$ works, or both work together. It is also possible that $L_{global}$ works alone with the original discriminator without the selective activation.\n\n* Clarity: \n\n  * The paper is overall well-written, but I am confusing about the following two points:\n\n    * The author may need to state how to select a good threshold $\\tau$.\n\n    * I think the statement of $L_{local}$ and $L_{global}$ is unclear. The authors may need to explain what is \"local\" and what is \"global\". In my understanding, they are just trying to match the loss difference of the teacher and student pair. In this way, there should not be absolute value signs in Equation (6) and (7), which means $L_{global}$ should be\n      $$\n      L_{global} = |(L_G^S-L_{D_{fake}}^S) - (L_G^T-L_{D_{fake}}^T)|.\n      $$\n      Moreover, the authors also need to verify that when the student pair have the same loss difference as the teacher pair, the student generator and discriminator maintain the optimal equilibrium.\n\n* Significance: The paper results are good, and I think it could indeed speed up the generator significantly in real-world applications.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I think the authors have adequately addressed the limitations and potential negative societal impact of their work.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "ItOwezZClKp", "reviewer": "Reviewer_Ajkh", "summary": "This work revisits the role of discriminator in GAN compression and finds that the capacity mismatch when compressing the generator with a fixed discriminator will lead to model collapse. To this end, this work proposes a generator-discriminator co-operative compression framework to automatically adapt the size of discriminator to maintain the Nash equilibrium with the lightweight generator. The extensive evaluation results validate the superiority of the proposed framework over SOTA GAN compression methods.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Originality and significance\nThis work identifies the model collapse problem in GAN compression caused by a fixed discriminator which is commonly adopted in previous GAN compression works. The proposed framework can effectively tackle this challenge and achieve new SOTA performance in GAN compression based on extensive experiments. Such simple but effective insights may help the community for future GAN compression techniques.\n\n\n## Quality and clarity\nThis paper is logically clear and easy to follow with solid motivating analysis. The proposed framework is technically sound with impressive results. I still have the following questions for this paper:\n\n(1) More discussions about the choice of L_{local} in Eq. (7) are expected. Is there any other potential forms to better maintain the Nash equilibrium, e.g., a weighted sum?\n\n(2) [55] and [11] directly discard the discriminator during GAN compression and only reply on the distillation loss, which can also avoid the model collapse. A potential ablation study is to adopt the distillation loss in Eq. (10) while the student discriminator is discarded.\n\n\n## After rebuttal\n\nThe author responses address my concerns. I will keep the original score, i.e., a clear accept.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitations are discussed in the main review section.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.5"}, "extra_text": {}}, {"review_id": "6wntc40aPMQ", "reviewer": "Reviewer_sJWF", "summary": "In this paper, the authors revisit the role of discriminator in GAN compression and propose a selective activation discriminator to automatically activate the convolutional channels. An online collaborative distillation is adopted to guide the optimization process of the lightweight generator. Experiment results demonstrate the effectiveness of the proposed method.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n+ The idea of generator-discriminator cooperative compression scheme for GAN compression is straightforward and reasonable since the balance between generator and discriminator is crucial for GAN training.\n\n+ The importance of discriminator’s channel-number is verified via corresponding experiments, which gives some insights for the image generation community.\n\n\nWeaknesses\n-\tMy main concern is the lack of novelty. In my view, the proposed mechanism of automatically activating convolutional channels and distillation method are incremental. In addition, the claim that “first attempt …” in line 59 is open to discussion. Some NAS-GAN related works [1][2] also consider the mutual balance between generator and discriminator and search the architectures of both networks simultaneously. The compression and NAS task are essentially the same in my opinion.\n\n-\tThe ablation studies on table 3 show that the effect of each component is not obvious. Compared with other compression methods, the improvement is also not significant.\n\n-\tMore analysis about the role of discriminator in GAN Compression should be given since the title of this paper is “Revisiting Discriminator in GAN Compression”. Simply discuss the effect of channel-number is not deep enough. Authors are encouraged to provide more insights.\n\n\n\n[1] Gong et al. AutoGAN: Neural Architecture Search for Generative Adversarial Networks. In ICCV. 2019.\n\n[2] Gao et al. AdversarialNAS: Adversarial Neural Architecture Search for GANs. In CVPR. 2020.\n\n---- Update ----\n\nThe authors address some of my concerns. However, the novelty of the proposed methods (e.g., selective activation discriminator and global constraint) is still limited in my view, and this work only discusses the effect of channel-number. After reading other comments and author's response, I prefer to keep the original rating 5.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes. See Main Review.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "3gNiOxMZbZv", "reviewer": "Reviewer_fP8T", "summary": "This work introduces a GAN compression method that exploits the role of the discriminator. The main contributions are two-fold: (i) maintain the capacity balance between the compressed generator and discriminator by learning to selectively activate the convolutional channels of the discriminator, (ii) promote the performance of the compressed generator by distilling knowledge from the teacher generator and discriminator to the student models in the feature space. The method is validated on synthesis, translation, and super-resolution tasks. ", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality\n\n+ To my knowledge, the idea is novel, simple, and intuitive. It may potentially open up a new perspective for GAN compression. \n\n+ Nevertheless, it's not the first paper exploiting/distilling the knowledge of the discriminator after training. Several recent works have explored this direction [1-4] for improved sampling. It would be good to see a discussion about that in the related work section and place the work in a broader context of GANs. \n\nQuality\n\n* The proposed method looks technically sound. Both ideas of capacity balance and feature distillation are highly intuitive and well supported by the empirical results. \n\n* However, I'm not fully convinced by the concrete solution to capacity balance. It's known that balancing the learning of G & D is not trivial and it's common in practice that $L_G + L_D$ is not close to 0. Yet, in Eq.6 & Eq.7, the authors propose to measure and balance the capacity by using the value of $L_G$ & $L_D$. I wonder how accurate this is as an approximation of the model capacity. Also, how is it applied to other forms of adversarial loss? It would be good to see a discussion about the potential limitations. \n\nClarity\n\n* The paper is overall well-motivated and easy to follow. \n\n* Yet, I'm sure not if I fully got the details of online collaborative distillation. E.g., in L182, 'The online scheme optimizes the teacher model and the lightweight student model iteratively and progressively from scratch. Therefore, we do not need a pre-trained teacher model, and directly complete the distillation process in one stage.' Does it refer to learn light-weighted GANs from scratch as opposed to compression? \n\n\n\n[1] Discriminator Rejection Sampling, ICLR'19\n\n[2] Metropolis-Hastings Generative Adversarial Networks, ICML'19\n\n[3] Collaborative Sampling in Generative Adversarial Networks, AAAI'20\n\n[4] Your GAN is Secretly an Energy-based Model and You Should use Discriminator Driven Latent Sampling, NIPS'20", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitations discussed in the supplementary material are not convincing. They look more like the general limitations of the topic rather than the proposed method.  ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a generator-discriminator co-training compression method to balance the model capacity of the lightweight generator and discriminator during GANs compression. The reviewers appreciate the idea of balancing generator and discriminator as well as comprehensive results and clear writing, although there are still concerns regarding the technical novelty and the ablation studies. The rebuttal addressed most of the reviewers' concerns and highlighted the difference from prior works (e.g., knowledge distillation for discriminator, NAS for discriminator architecture search). I agree with most reviewers and believe that the paper's contribution is significant enough, and the results are convincing. Therefore, I recommend accepting the paper. 
Decision: Accept (Poster)