Revisiting Discriminator in GAN Compression: A Generator-discriminator Cooperative Compression Scheme
Year: 2022

Abstract:
Recently, a series of algorithms have been explored for GAN compression, which aims to reduce tremendous computational overhead and memory usages when deploying GANs on resource-constrained edge devices. However, most of the existing GAN compression work only focuses on how to compress the generator, while fails to take the discriminator into account. In this work, we revisit the role of discriminator in GAN compression and design a novel generator-discriminator cooperative compression scheme for GAN compression, termed GCC. Within GCC, a selective activation discriminator automatically selects and activates convolutional channels according to a local capacity constraint and a global coordination constraint, which help maintain the Nash equilibrium with the lightweight generator during the adversarial training and avoid mode collapse. The original generator and discriminator are also optimized from scratch, to play as a teacher model to progressively refine the pruned generator and the selective activation discriminator. A novel online collaborative distillation scheme is designed to take full advantage of the intermediate feature of the teacher generator and discriminator to further boost the performance of the lightweight generator. Extensive experiments on various GAN-based generation tasks demonstrate the effectiveness and generalization of GCC. Among them, GCC contributes to reducing 80% computational costs while maintains comparable performance in image translation tasks. Our code and models are available at: https://github.com/SJLeo/GCC

heading: 1 Introduction
text: Generative Adversarial Networks (GANs) have been widely popularized in diverse image synthesis tasks, such as image generation [16, 50], image translation [64, 30] and super resolution [35, 59]. However, the ultra-high computational and memory overhead of GANs hinder its deployment on resource-constrained edge devices. To alleviate this issue, a series of traditional model compression algorithms, i.e., network pruning [18, 36], weight quantization [39, 26], low-rank decomposition [10, 19], knowledge distillation [22, 52], and efficient architecture design [29, 55] have been applied to reduce calculational consumptions of GANs.
Previous work [56, 12, 61] attempted to directly employ network pruning methods to compress the generator but obtain unimpressive results, as shown in Figure 1(a). The similar phenomenon also occurred in SAGAN shown in Appendix A. A potential reason is that these methods failed to take into account the generator and the discriminator must follow the Nash equilibrium state to avoid the mode ∗This work was done when Shaojie Li was a intern at Bytedance Inc. †Corresponding Author: rrji@xmu.edu.cn
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
collapse in the adversarial learning. In other words, these methods simply compress the generator while maintaining the original capability of the discriminator, which resulted in breaking the balance in adversarial learning. As shown in Figure 1(b) and 1(c), when compressing the generator, the loss of the discriminator gradually tends to zero, such a situation indicates that the capacity of the discriminator significantly surpasses that of the lightweight generator. Furthermore, the capacity imbalance between the generator and discriminator leads to the mode collapse problem [47].
In order to retain the Nash equilibrium between the lightweight generator and discriminator, we must revisit the role of the discriminator in the procedure of GAN compression. A straightforward method is to reduce the capacity of the discriminator to re-maintain the Nash equilibrium between it and the lightweight generator. Therefore, we train a lightweight generator against the discriminators with different channel numbers. The results are shown in Figure 2, we can obtain the following observations: i) The channel numbers in the discriminator significantly influence the performance of the lightweight generator; ii) The discriminator with the same channel compression ratio as the generator may not get the best result. iii) The optimal value of channel numbers for discriminators is task-specific. In short, it is arduous to choose an appropriate channel number for the discriminator in different tasks.
To solve the above issues, we design a generator-discriminator cooperative compression scheme, term GCC in this paper. GCC consists of four sub-networks, i.e., the original uncompressed generator and discriminator, a pruned generator, and a selective activation discriminator. GCC selectively activates
partial convolutional channels of the selective activation discriminator, which guarantees the whole optimization process to maintains in the Nash equilibrium stage. In addition, the compression of the generator will damage its generating ability, and even affect the diversity of generated images, resulting in the occurrence of mode collapse. GCC employs an online collaborative distillation that combines the intermediate features of both the original uncompressed generator and discriminator and then distills them to the pruned generator. The introduction of online collaborative distillation boosts the lightweight generator performance for effective compression. The contributions of GCC can be summarized as follows:
• To the best of our knowledge, we offer the first attempt to revisit the role of discriminator in GAN compression. We propose a novel selective activation discriminator to automatically select and activate the convolutional channels according to a local capacity constraint and a global coordination constraint. The dynamic equilibrium relationship between the original models provides a guide to choose the activated channels in the discriminator.
• A novel online collaborative distillation is designed to simultaneously employ intermediate features of teacher generator and discriminator to guide the optimization process of the lightweight generator step by step. These auxiliary intermediate features provide more complementary information into the lightweight generator for generating high-fidelity images.
• Extensive experiments on various GAN-based generation tasks (i.e., image generation, image-to-image translation, and super-resolution) demonstrate the effectiveness and generalization of GCC. GCC contributes to reducing 80% computational costs while maintains comparable performance in image translation.

heading: 2 Related Work
text: Generative Adversarial Networks. Generative Adversarial Network, i.e., GANs [16] has attracted intense attention and has been extensively studied for a long time. For example, DCGAN [50] greatly enhanced the capability of GANs by introducing the convolutional layer. The recent works [17, 33, 63, 4] employed a series of advanced network architecture to further improve the fidelity of the generated images. In addition, several novel loss functions [54, 17, 2, 45] were designed to stabilize the adversarial training stage of GANs. A popular research direction of GANs is image-to-image translation. The image-to-image translation is designed to transfer images from the source domain to the target domain while retaining the content representation of the image. Image-to-image translation is widely used in computer vision and image processing, such as semantic image synthesis [30, 48], style transfer [64, 1], super resolution [35, 59], image painting [49, 65] and so on.
Mode Collapse. Mode collapse is the main catastrophic problem during the adversarial training stage of GANs, which leads to the diversity of generated images are monotonous, only fitting part of the real data distribution. To address this issue, recent efforts were focus on new objective functions [47, 17, 34], architecture modifications [5, 15] and mini-batch discrimination [54, 33]. Actually, the capacity gap between the generator and the discriminator is a more essential reason for mode collapse. In this paper, we provide a different perspective to eliminate the capacity gap, we select the convolutional channel to be activated in the discriminator according to a local capacity constraint and a global coordination constraint.
GAN Compression. To reduce huge resource consumption during the inference process of GANs, great efforts have emerged in the field of GAN compression in the recent two years. The searchbased methods [56, 37, 12, 23, 40, 58] exploited neural architecture search and genetic algorithm to obtain a lightweight generator. However, they fell into the trap of manual pre-defined search space and huge search costs. The prune-based methods [38, 31, 57, 61] directly pruned a lightweight generator architecture from the original generator architecture. However, these works failed to take discriminator pruning into account, which would seriously destroy the Nash equilibrium between the generator and the discriminator. The discriminator-free methods [51, 12] directly use the large GAN model as a teacher to distill the lightweight generator without discriminator, which also achieved good performance. Slimmable GAN [23] correspondingly shrank the network width of the discriminator with the generator. However, as shown in Fig. 2, there is no linear between the channel numbers of the optimal discriminator and generator. Anycost GAN [40] proposed a generator-conditioned discriminator, which generates the discriminator architecture via passing the generator architecture
into the fully connected layer. However, the fully connected layer is only trained through the original GAN loss, so it is difficult to accurately predict the optimal discriminator architecture for the lightweight generator. In this work, we design a generator-discriminator cooperative compression (GCC) scheme, which employ a selective activation discriminator and novel online collaborative distillation to boost the performance of the lightweight generator.

heading: 3 Methodology
text: This section, we illustrate the proposed GCC framework in Figure 3. We first briefly introduce the composition and loss function of GAN in Section 3.1. Then, we present how to obtain the lightweight architecture from the original generator in Section 3.2. The selective activation discriminator that automatically chooses activated convolutional channels is illustrated in Section 3.3. In Section 3.4, we propose a novel online collaborative distillation scheme to further improve the performance of the lightweight generator.

heading: 3.1 Preliminaries
text: Generative Adversarial Networks (GANs) contain two fundamental components, generator G and discriminator D. Among them, G maps the input z into a fake image for cheating the discriminator, while the discriminator distinguishes the generator outputs from the real images x. The generator and discriminator are alternately optimized via the adversarial term to achieve their respective optimization objectives. The optimization objectives are defined as follows:
LG = Ez∼p(z) [fG(−D(G(z)))] (1)
LD = Ex∼pdata [fD(−D(x))]︸ ︷︷ ︸ LDreal +Ez∼p(z) [fD(D(G(z)))]︸ ︷︷ ︸ LDfake
(2)
where fG and fD are the loss functions of the generator and discriminator, respectively. Theoretically, the generator and discriminator maintain the Nash equilibrium when the generator and discriminator have a balanced capacity, and the discriminator is cheated. In this paper, we observe from experiments that the Nash equilibrium maintained in the GAN compression stage to avoid model collapse and obtain an impressive performance.

heading: 3.2 Generator Compression
text: In order to avoid additional computational overhead brought by the neural architecture search, we directly employ the conventional network pruning methods to compress the generator. Previous
work [43] revealed that the core of network pruning searches for a suitable sub-architecture from the original network architecture instead of parameter inheritance. Therefore, we leverage the one-shot pruning method [36, 42] to obtain the appropriate lightweight generator architecture from the original generator architecture. In our experiments, we adopt the slimming [42] to prune the generator when the network contains the Batch Normalization (BN) layers. We use L1-norm pruning [36] on the contrary (without BN layers). The detailed pruning process is described in Appendix B. Finally, we only keep the architecture of the pruned generator and train it from scratch with a selective activation discriminator proposed in Section 3.3.

heading: 3.3 Selective Activation Discriminator
text: As shown in Figure 2, the convolutional channel number in the discriminator have an essential impact on the performance of the lightweight generator. Therefore, we design a selective activation discriminator to automatically choose and activate convolutional channels via learnable retention factors. We assign a learnable retention factor to each convolution kernel of the original discriminator, each retention factor represents the probability that the corresponding convolution kernel participates in the inference process. When the retention probability is larger than the given threshold, the corresponding convolution kernel is retained, otherwise it is suppressed. We denote retention factors set as A = {αi}LDi=1 where αi = {αi1, αi2, ..., αini} ∈ Rni is the retention factors of the i-th convolution layer and ni is the number of convolution kernel in the i-th layer. Given the threshold τ , the retention factor αij determines whether the j-th convolution kernel of the i-th layer activate or not:
Iij = { 1 if αij ∈ [τ, 1] 0 if αij ∈ [0, τ). (3)
Then activated convolution kernel’s output O′ij is calculated by:
O′ij = Iij ∗Oij , (4)
where Oij is the original output feature map. Since the binary function is not differentiable for αij , we employ STE [27] to estimate the gradient of αij as:
∂Larch ∂αij = ∂Larch ∂Iij , (5)
where Larch is a loss function, and the retention factor αij is manually clipped to the range of [0, 1]. The learning of retention factor is considered as the architecture search of the discriminator.
The selective activation discriminator is optimized according to a local capacity constraint and a global coordination constraint. On the one hand, the incremental loss gap between the generator and the discriminator will break the Nash equilibrium in the adversarial training scheme, which leads to the discriminator effortlessly distinguishing the authenticity of the input images. The pruned generator and the selective activation discriminator should pursuit capacity balance to maintain the Nash equilibrium. Therefore, a local capacity constraint Llocal is designed as follows:
Llocal = |LSG − LSDfake |, (6)
where the superscript S denotes the student (compressed model). On the other hand, the loss gap between the original model’s combination internally reflects the relative capacity relationship between them. Therefore, we design a global coordination constraint to adjust the relative ability of the student generator and discriminator to be consistent with that of the teacher combination. This constraint helps to pull the loss gap between teacher and student models into close proximity. In this way, the uncompressed model is regarded as teacher model to guide the learning of retention factors to automatically adjust the discriminator capacity to match the lightweight generator. The global coordination constraint Lglobal is calculated by:
Lglobal = |Llocal − |LTG − LTDfake || (7)
The superscript T represents the teacher (uncompressed) model. Due to the instability of adversarial training, we use Exponential Moving Average (EMA) to stabilize the loss corresponds to the teacher models. In short, the optimization objective of the retention factor is defined as follows:
Larch = LSD + Lglobal. (8)
where LSD guarantees the basic identification function of the discriminator. Furthermore, we use the following bilevel optimization [7, 11] to optimize α: i) Update discriminator weights to minimize LD for t times. ii) Update retention factor α to minimize Larch. iii) Repeat step i) and step ii) until the end of training. In the optimization process of the discriminator weights, we freeze the retention factors and vice versa.

heading: 3.4 Online Collaborative Distillation
text: In Section 3.3, we use the original model as the teacher models to guide the learning of the student discriminator. Most of the existing work merely considered the distillation from the generator, in contrast, we also employ richer intermediate information from the discriminator to perform collaborative distillation, which provides complementary and auxiliary concepts to boost the generation performance. Therefore, we propose a novel online collaborative distillation method to take full advantage of the teacher generator and discriminator combinations to promote the lightweight generator. The online scheme optimizes the teacher model and the lightweight student model iteratively and progressively from scratch. Therefore, we do not need a pre-trained teacher model, and directly complete the distillation process in one stage. Experiments show that online knowledge distillation is more efficient than traditional two-stage offline knowledge distillation.
Perceptual loss [32] is widely used to calculate the similarity of semantic and perceptual quality between images. It measures the semantic differences via a pre-trained image classification network (such as VGG), where the extracted features are less relevant to our generation task. Meanwhile, the teacher discriminator has a task-oriented ability to distinguish the authenticity of images. Following the work [6], we employ the teacher discriminator to replace the pre-trained classification model to extract the features of the images for calculating the perceptual loss. Furthermore, the intermediate feature map of the generator has rich information for generating real-like fake images, and we also add them to the calculation of perceptual loss. In this way, the similarity metric is summarized as:
d(·, ·) = γmMSE(·, ·) + γtTexture(·, ·), (9) where γm and γt are pre-defined hyperparameter used for balancing MSE and Texture [13, 14] loss functions, respectively. Detailed formulation of Texture loss function is described in Appendix C. Thus, the collaborative distillation loss is:
Ldistill = LG∑ i=1 Ez∼p(z) [ d(fi(G S i (z)), G T i (z)) ] + LD∑ j=1 Ez∼p(z) [ d(DTj (G S(z)), DTj (G T (z))) ] ,
(10) whereGSi andG T i are the intermediate feature maps of the i-th chosen layer in the student and teacher generator, DTj are the intermediate feature maps of the j-th chosen layer in the teacher discriminator. LG / LD denote the number of the generator / discriminator chosen layers. fi is a 1×1 learnable convolution transform layer to match the channel dimension of the teacher’s feature maps.

heading: 4 Experiments

heading: 4.1 Setups
text: To demonstrate the effectiveness of the proposed GCC, we conduct extensive experiments on a series of GAN based tasks:
Image Generation. We employ SAGAN [63] to input random noise to generate face images on the CelebA [44] dataset. CelebA contains 202,599 celebrity images with a resolution of 178×218. We first center crop them to 160×160 and then resize them to 64×64 before feeding them into the model. Frechet Inception Distance (FID) [21] is adopted to evaluate the quality of the generated images. The lower score indicates the better the image generation results.
Image-to-Image Translation. Pix2Pix [30] is a conditional GAN that uses Unet-based [53] generator to translate paired images. We evaluate pix2pix on the Cityscapes [8] dataset, which contains 3475 German street scenes. The images are resized to 256×256 for training. We follow the work [37] run the DRN-D-105 [62] to calculate mean Intersection over Union (mIOU) for evaluating the quality of the generated images. The higher score indicates the better the image generation performance. In addition, we also conduct experiments on the unpaired dataset. CycleGAN [64] leverages a
ResNet [20] generator to translates horse image into zebra image on the horse2zebra dataset extracted from ImageNet [9]. This dataset consists of 1,187 horse images and 1,474 zebra images, and all images are resized 256×256 for training. We use FID to evaluate the quality of the generated images. Super Resolution. We apply the proposed GCC to compress SRGAN [35], which uses a DenseNetlike [24] generator to upscale low-resolution images. The training set and validation set of COCO [41] dataset are combined for the training process. We randomly crop 96×96 high-resolution images from the original images and then downsample it 4× into low-resolution images as the input of the generator. For testing, we employ Set5 [3], Set14 [35], BSD100 [46], and Urban100 [25] as the standard benchmark datasets. Peak Signal-to-Noise Ratio (PSNR) [28] and Structural Similarity (SSIM) [60] are used to evaluate the quality of the generated images. The higher score indicates the better the image generation performance. More implementation details are illustrated in Appendix D.

heading: 4.2 Comparison to the State of the Art
text: Quantitative Result. Tables 1 and 2 show the comparison results of different algorithms on image generation, image-to-image translation, and super-resolution. From these two tables, we can obtain the following observations: 1) In the task of image translation, our method can greatly reduce the computational costs in each task and achieve comparable performance as the original uncompressed model. For example, GCC obtains a lower FID than the original model under the extremely high compression ratio on CycleGAN. Specifically, GCC reduces the MACs of CycleGAN from 56.8G to 2.40G, with compression by 23.6× while FID still reduces 2.22. 2) GCC reduces the computational costs by 82.39% and achieves a higher mIoU than the original model, which establishes a new state-of-the-art performance for the Pix2Pix model. 3) In image generation task, GCC also shows impressive results on low MACs demands of GAN, i.e., SAGAN. Although the original SAGAN only requires 23.45M MACs to generate a 64×64 pixel image, GCC successes to reduce the MACs by 34.12%. 4) For SRGAN in the super-resolution task, GCC helps to significantly reduce 123.09G
MACs, with a compression ratio of 6.4×. It is worth noting that although GCC reduces such a high amount of computational costs, the lightweight model can achieve higher PSNR and SSIM than the original model on each test dataset. In addition, we also show the results of employing network pruning methods [36, 42] to compress the generator and directly perform adversarial training with the original discriminator, abbreviated as "Prune", the experimental results demonstrate the importance and effectiveness of our proposed selective activation discriminator. In summary, Our proposed GCC contributes to removes a large number of redundant calculations while maintains the comparable performances of the original model. The results provide a feasible solution for demands on a super-computing or mini-computing GAN model.
Qualitative Results. Figure 4 depicts the model generation performance on each task. Qualitative results reveal that GCC contributes to generating high-fidelity images with low computational demands. More qualitative results are provided in Appendix G.
Acceleration. We also compare acceleration performance on actual hardware between the original generator and the lightweight generator on Pix2Pix. The latency is measured on two types of computing processors (i.e. Intel Xeon Platinum 8260 CPUs and NVIDIA Tesla T4 GPUs). The original generator takes 75.96ms to generate one image on CPUs and 2.06ms on GPUs. However, the lightweight generator only uses 18.68ms and 1.74ms to generate one image on CPUs and GPUs, respectively. GCC helps to achieve 4.07× and 1.18× acceleration on CPU and GPU, respectively.

heading: 4.3 Ablation Study
text: Online Collaborative Distillation. Table 3 shows the effectiveness of each components in the online collaborative distillation. The variant "Ours w/o Online" represents the traditional two-stage distillation scheme. "Ours w/o D-distillation" and "Ours w/o G-distillation" denotes that only adopt the intermediate feature information of the teacher generator/discriminator in the distillation stage, respectively. We can intuitively observe that the distillation method can indeed further improve the performance of the model. Both MSE loss and Texture loss are helpful in the distillation process, and the combination of these two losses achieves better mIOU scores. GCC deploys the distillation process
into the online stage, where promotes an uncompressed teacher model progressively to gradually guide the learning of the lightweight generator and avoid the complicated two-stage distillation scheme. In Table 3, the online distillation scheme can achieve better performance than the offline distillation. The online collaborative distillation distills the intermediate feature information from both the generator and discriminator of the teacher model. G-only distillation / D-only distillation only distills one of them to the lightweight generator, which leads to declining results.
Solution of Mode Collapse. Figure 1 shows the phenomenon of mode collapse when compressing the generator without GCC. With the introduction of GCC, the 1/4 channels width generator can get rid of mode collapse. We show the improved loss curve and visualization results in Figure 5. In Figure 5(a), the selective activation discriminator effectively balances the adversarial training with the lightweight generator. The loss gap between the 1/4 channels width generator and selective activation discriminator is more evenly matched to that of the original model as shown in Figure 1(b). In Figure 5(b), the 1/4 channels width generator can generate higher quality images with GCC.

heading: 5 Conclusion
text: During GAN compression, we observe that merely compressing the generator while retaining the original discriminator destroys the Nash equilibrium between them, which further results in mode collapse. In this work, we revisit the role of discriminator in GAN compression and design a novel generator-discriminator cooperative compression (GCC) scheme. To ensure normal adversarial training between the lightweight generator and discriminator, we propose a selective activation discriminator to automatically select the convolutional channel to be activated. In addition, we propose a novel online collaborative distillation that simultaneously utilizes the intermediate feature of the teacher generator and discriminator to boost the performance of the lightweight generator. We have conducted experiments on various generation tasks, and the experimental results verify the effectiveness of our proposed GCC.

heading: 6 Acknowledge
text: This work is supported by the National Science Fund for Distinguished Young Scholars (No.62025603), the National Natural Science Foundation of China (No.U1705262, No.62072386, No.62072387, No.62072389, No.62002305, No.61772443, No.61802324 and No.61702136) and Guangdong Basic and Applied Basic Research Foundation (No.2019B1515120049).

Review summaries:
This work introduces a GAN compression method that exploits the role of the discriminator. The main contributions are two-fold: (i) maintain the capacity balance between the compressed generator and discriminator by learning to selectively activate the convolutional channels of the discriminator, (ii) promote the performance of the compressed generator by distilling knowledge from the teacher generator and discriminator to the student models in the feature space. The method is validated on synthesis, translation, and super-resolution tasks. 

In this paper, the authors revisit the role of discriminator in GAN compression and propose a selective activation discriminator to automatically activate the convolutional channels. An online collaborative distillation is adopted to guide the optimization process of the lightweight generator. Experiment results demonstrate the effectiveness of the proposed method.

In this paper, the authors revisit the role of discriminator in GAN compression and design a new generator-discriminator cooperative scheme GCC for GAN compression. Specifically, they use a selective activation discriminator to automatically select the convolutional channels, aiming to help maintain the Nash equilibrium and avoid mode collapse. Besides, they also propose an online collaborative distillation method to further boost the compression performance. They achieve impressive performance on three widely-used GAN tasks--image generation, image-to-image translation, and super-resolution.

This work revisits the role of discriminator in GAN compression and finds that the capacity mismatch when compressing the generator with a fixed discriminator will lead to model collapse. To this end, this work proposes a generator-discriminator co-operative compression framework to automatically adapt the size of discriminator to maintain the Nash equilibrium with the lightweight generator. The extensive evaluation results validate the superiority of the proposed framework over SOTA GAN compression methods.