Title: SELF-SUPERVISED GAN COMPRESSION

Abstract: Deep learning's success has led to larger and larger models to handle more and more complex tasks; trained models can contain millions of parameters. These large models are compute-and memory-intensive, which makes it a challenge to deploy them with minimized latency, throughput, and storage requirements. Some model compression methods have been successfully applied on image classification and detection or language models, but there has been very little work compressing generative adversarial networks (GANs) performing complex tasks. In this paper, we show that a standard model compression technique, weight pruning, cannot be applied to GANs using existing methods. We then develop a selfsupervised compression technique which uses the trained discriminator to supervise the training of a compressed generator. We show that this framework has a compelling performance to high degrees of sparsity, can be easily applied to new tasks and models, and enables meaningful comparisons between different pruning granularities.

1. INTRODUCTION:

Deep Neural Networks (DNNs) have proved successful in various tasks like computer vision, natural language processing, recommendation systems, and autonomous driving. Modern networks are comprised of millions of parameters, requiring significant storage and computational effort. Though accelerators such as GPUs make realtime performance more accessible, compressing networks for faster inference and simpler deployment is an active area of research. Compression techniques have been applied to many networks, reducing memory requirements and improving their performance. Though these approaches do not always harm accuracy, aggressive compression can adversely affect the behavior of the network. Distillation (Schmidhuber, 1991; Hinton et al., 2015) can improve the accuracy of a compressed network by using information from the original, uncompressed network.

Generative Adversarial Networks (GANs) (Schmidhuber, 1990; Goodfellow et al., 2014) are a class of DNN that consist of two sub-networks: a generative model and a discriminative model. Their training process aims to achieve a Nash Equilibrium between these two sub-models. GANs have been used in semi-supervised and unsupervised learning areas, such as fake dataset synthesis (Radford et al., 2016; Brock et al., 2019) , style transfer (Zhu et al., 2017b; Azadi et al., 2018) , and image-to-image translation (Zhu et al., 2017a; Choi et al., 2018) . As with networks used in other tasks, GANs have millions of parameters and nontrivial computational requirements.

In this work, we explore compressing the generative model of GANs for more efficient deployment. We show that applying standard pruning techniques, with and without distillation, can cause the generator's behavior to no longer achieve the network's goal. Similarly, past work targeted at compressing GANs for simple image synthesis fall short when they are applied to large tasks. In some cases, this result is masked by loss curves that look identical to the original training. By modifying the loss function with a novel combination of the pre-trained discriminator and the original and compressed generators, we can overcome this behavioral degradation and achieve compelling compression rates with little change in the quality of the compressed generator's ouput. We apply our technique to several networks and tasks to show generality. Finally, we study the behavior of compressed generators when pruned with different amounts and types of sparsity, finding that filter pruning, a technique commonly used for accelerating image classification networks, is not trivially applicable to GANs.

Our main contributions are:

• We illustrate that and explain why pruning the generator of a GAN with existing methods is unsatisfactory for complex tasks. (Section 3) • We propose self-supervised compression for the generator in a GAN (Section 4) • We show that our technique can apply to several networks and tasks (Section 5) • We show and analyze qualitative differences in pruning ratio and granularities. (Section 6)

2. RELATED RESEARCH:

A common method of DNN compression is network pruning (Han et al., 2015) : setting the small weights of a trained network to zero and fine-tuning the remaining weights to recover accuracy. Zhu & Gupta (2018) proposed a gradual pruning technique (AGP) to compress the model during the initial training process. Wen et al. (2016) proposed a structured sparsity learning method that uses group regularization to force weights towards zero, leading to pruning groups of weights together. Li et al. (2017) pruned entire filters and their connecting feature maps from models, allowing the network to run with standard dense software libraries. Though it was initially applied to image classification networks, network pruning has been extended to natural language processing tasks (See et al., 2016; Narang et al., 2017) and to recurrent neural networks (RNNs) of all types -vanilla RNNs, GRUs (Cho et al., 2014) , and LSTMs (Hochreiter & Schmidhuber, 1997) . As with classification networks, structured sparsity within recurrent units has been exploited (Wen et al., 2018) .

A complementary method of network compression is quantization. Sharing weight values among a collection of similar weights by hashing (Chen et al., 2015) or clustering (Han et al., 2016) can save storage and bandwidth at runtime. Changing fundamental data types adds the ability to accelerate the arithmetic operations, both in training (Micikevicius et al., 2018) and inference regimes (Jain et al., 2019) .

Several techniques have been devised to combat lost accuracy due to compression, since there is always the chance that the behavior of the network may change in undesirable ways when the network is compressed. Using GANs to generate unique training data (Liu et al., 2018b) and extracting knowledge from an uncompressed network, known as distillation (Hinton et al., 2015) , can help keep accuracy high. Since the pruning process involves many hyperparameters, Lin et al. (2019) use a GAN to guide pruning, and Wang et al. (2019a) structure compression as a reinforcement learning problem; both remove some of the burden from the user.

3. EXISTING TECHNIQUES FAIL TO PRUNE A COMPLEX TASK:

Though there are two networks in a single GAN, the main workload at deployment is usually from the generative model, or generator. For example, in image synthesis and style transfer tasks, the final output images are created solely by the generator. The discriminative model (discriminator) is vital in training, but it is abandoned afterward for many tasks. So, when we try to apply state-of-the-art compression methods to GANs, we focus on the generator for efficient deployment. As we will see, the generative performance of the compressed generators is quite poor for the selected image-toimage translation task. We look at two broad categories of baseline approaches: standard pruning techniques that have been applied to other network architectures, and techniques that were devised to compress the generator of a GAN performing image synthesis. We compare to the dense baseline [a], our technique [b], as well as a small, dense network with the same number of parameters [c] . (Labels correspond to entries in Table 1 , the overview of all techniques, and Figure 1 , results of each technique).

Standard Pruning Techniques. To motivate GAN-specific compression methods, we try variations of two state-of-the-art pruning methods: manually pruning and fine tuning (Han et al., 2015) a trained dense model [d] , and AGP (Zhu & Gupta, 2018) from scratch [e] and during fine-tuning [f]. We also include distillation (Hinton et al., 2015) to improve the performance of the pruned network with manual pruning [g] and AGP fine-tuning [h] . Distillation is typically optional for other network types, since it is possible to get decent accuracy with moderate pruning in isolation. For very aggressive compression or challenging tasks, distillation aims to extract knowledge for the compressed (student) network from original (teacher) network's behavior. We also fix the discriminator of [g] to see if the discriminator was being weakened by the compressed generator [i] .

Targeted GAN Compression. There has been some work in compressing GANs with methods other than pruning, and only one technique applied to an image-to-image translation task. We first examine two approaches similar to ours. Adversarial training (Wang et al., 2018) [j] posits that during distillation of a classification network, the student network can be thought of as a generative model attempting to produce features similar to that of the teacher model. So, a discriminator was trained alongside the student network, trying to distinguish between the student and the teacher. One could apply this technique to compress the generator of a GAN, but we find that its key shortcoming is that it trains a discriminator from scratch. Similarly, distillation has been used to compress GANs in Aguinaldo et al. (2019) [k], but again, the "teacher" discriminator was not used when teaching the "student" generator.

Learned Intermediate Representation Training (LIT) (Koratana et al., 2019) [l] compresses StarGAN by a factor of 1.8× by training a shallower network. Crucially, LIT does not use the pre-trained discriminator in any loss function. Quantized GANs (QGAN) (Wang et al., 2019b) [m] use a training process based on Expectation-Maximization to achieve impressive compression results on small generative tasks with output images of 32x32 or 64x64 pixels. Liu et al. (2018a) find that maintaining a balance between discriminator and generator is key: their approach is to selectively binarize parts of both networks in the training process on the Celeb-A generative task, up to 64x64 pixels. So, we try pruning both networks during the training process [n] .

Experiments. For these experiments, we use StarGAN (Choi et al., 2018) trained with the Distiller (Zmora et al., 2018) library for the pruning. StarGAN1 extends the image-to-image translation capability from two domains to multiple domains within a single unified model. It uses the Celeb-Faces Attributes (CelebA) (Liu et al., 2015) as the dataset. CelebA contains 202,599 images of celebrities' faces, each annotated with 40 binary attributes. As in the original work, we crop the initial images from size 178 × 218 to 178 × 178, then resize them to 128 × 128 and randomly select 2,000 images as the test dataset and use remaining images for training. The aim of StarGAN is facial attribute translation: given some image of a face, it generates new images with five domain attributes changed: 3 different hair colors (black, blond, brown), different gender (male/female), and different age (young/old). Our target sparsity is 50% for each approach.

We stress that we attempted to find good hyperparameters when using the existing techniques, but standard approaches like reducing the learning rate for fine-tuning (Han et al., 2015) , etc., were not helpful. Further, the target sparsity, 50%, is not overly aggressive, and we do not impose any structure; other tasks readily achieve 80%-90% fine-grained sparsity with minimal accuracy impact.

The results of these trials are shown in Figure 1 . Subjectively, it is easy to see that the existing approaches (1c through 1n) produce inferior results to the original, dense generator. Translated facial images from pruning & naïve fine-tuning (1d and 1e) do give unique results for each latent variable, but the images are hardly recognizable as faces. These fine-tuning procedures, along with AGP from scratch (1f) and distillation from intermediate representations (1l), simply did not converge. One-shot pruning and traditional distillation (1g), adversarial learning (1j), knowledge distillation (1k), training a "smaller, dense" half-sized network from scratch (1c) and pruning both generator and discriminator (1n) keep facial features intact, but the image-to-image translation effects are lost to mode collapse (see below). There are obvious mosaic textures and color distortion on the translated images from fine-tuning & distillation (1h), without fine-tuning the original loss (1i), and from the pruned model based on the Expectation-Maximization (E-M) algorithm (1m). However, the translated facial images from a generator compressed with our proposed self-supervised GAN compression method (1b) are more natural, nearly indistinguishable from the dense baseline (1a), matching the quantitative Frechet Inception Distance (FID) scores (Heusel et al., 2017) in Table 1 . While past approaches have worked to prune some networks on other tasks (DCGAN generating MNIST digits, see the supplementary material), we show that they do not succeed on larger imageto-image translation tasks, while our approach works on both. Similarly, though LIT (Koratana et al., 2019) [l] was able to achieve a compression rate of 1.8× on this task by training a shallower network, it does not see the same success at network pruning.

Discussion. It is tempting to think that the loss curves of the experiment for each technique can tell us if the result is good or not. We found that for many of these experiments, the loss curves correctly predicted that the final result would be poor. However, the curves for [h] and [m] look very good -the compressed generator and discriminator losses converge at 0, just as they did for baseline training. It is clear from the results of querying the generative models (Figures 1h and 1m ), though, that this promising convergence is a false positive. In contrast, the curves for our technique predict good performance, and, as we prune more aggressively in Section 6, higher loss values correlate well with worsening FID scores. (Loss curves are provided in the Appendix.)

As pruning and distillation are very effective when compressing models for image classification tasks, why do they fail to compress this generative model? We share three potential reasons:

1. Standard pruning techniques need explicit evaluation metrics; softmax easily reflects the probability distribution and classification accuracy. GANs are typically evaluated subjectively, though some imperfect quantitative metrics have been devised. 2. GAN training is relatively unstable (Arjovsky et al., 2017; Liu et al., 2018a) and sensitive to hyperparameters. The generator and discriminator must be well-matched, and pruning can disrupt this fine balance. 3. The energy of the input and output of a GAN is roughly constant, but other tasks, such as classification, produce an output (1-hot label vector) with much less entropy than the input (three-channel color image of thousands of pixels).

Elaborating on this last point, there is more tolerance in the reduced-information space for the compressed classification model to give the proper output. That is, even if the probability distribution inferred by the original and compressed classification models are not exactly the same, the classified labels can be the same. On the other hand, tasks like style-transfer and dataset synthesis have no obvious energy reduction. We need to keep entropy as high as possible (Kumar et al., 2019) during the compression process to avoid mode collapse -generating the same output for different inputs or tasks. Attempting to train a new discriminator to make the compressed generator behave more like the original generator (Wang et al., 2018) suffers from this issue -the new discriminator quickly falls into a low-entropy solution and cannot escape. Not only does this preclude its use on generative tasks, but it means that the compressed network for any task must also be trained from scratch during the distillation process, or the discriminator will never be able to learn.

4. SELF-SUPERVISED GENERATOR COMPRESSION:

We seek to solve each of the problems highlighted above. Let us restate the general formulation of GAN training: the purpose of the generative model is to generate new samples which are very similar to the real samples, but the purpose of the discriminative model is to distinguish between real samples and those synthesized by the generator. A fully-trained discriminator is good at spotting differences, but a well-trained generator will cause it to believe that the a generated sample is both real and generated with a probability of 0.5. Our main insight follows:

By using this powerful discriminator that is already well-trained on the target data set, we can allow it to stand in as a quantitative subjective judge (point 1, above) -if the discriminator can't tell the difference between real data samples and those produced by the compressed generator, then the compressed generator is of the same quality as the uncompressed generator. A human no longer needs to inspect the results to judge the quality of the compressed generator. This also addresses our second point: by starting with a trained discriminator, we know it is well-matched to the generator and will not be overpowered. Since it is so capable (there is no need to prune it to), it also helps to avoid mode collapse. As distillation progresses, it can adapt to and induce fine changes in the compressed generator, which is initialized from the uncompressed generator.

Since the original discriminator is used as a proxy for a human's subjective evaluation, we refer to this as "self-supervised" compression. We illustrate the workflow in Figure 2 , using a GAN charged with generating a map image from a satellite image in a domain translation task.

In the right part of Figure 2 , the real satellite image (x) goes through the original generative model (G O ) to produce a fake map image (ŷ o ). The corresponding generative loss value is l-G O . Accordingly, in the left part of Figure 2 , the real satellite image (x) goes through the compressed generative model (G C ) to produce a fake map image (ŷ c ). The corresponding generative loss value is l-G C . This is the inference process of the original and compressed generators, expressed as follows:

EQUATION

The overall generative difference is measured between the two corresponding generative losses2 . We use a generative consistent loss function (L GC ) in the bottom of Figure 2 to represent this process. 

EQUATION

So the discriminative difference is measured between two corresponding discriminative losses. We use the discriminative consistent loss function L DC in the top of Figure 2 to represent this process.

EQUATION

The generative and discriminative consistent loss functions (L GC and L DC ) use the weighted normalized Euclidean distance. Taking the StarGAN task as the example (other tasks may use different losses):

L GC (l-G O , l-G C ) = |l-Gen O -l-Gen C |/|l-Gen O | + α|l-Cla O -l-Cla C |/|l-Cla O | +β|l-Rec O -l-Rec C |/|l-Rec O | (7)

where l-Gen is the generation loss term, l-Cla is the classification loss term, and l-Rec is the reconstruction loss term. α and β are the weight ratios among three loss types. (We use the same values of α and β used in the original StarGAN baseline.)

L DC (l-D O , l-D C ) = |l-Dis O -l-Dis C |/|l-Dis O | + δ|l-GP O -l-GP C |/|l-GP O | (8)

where l-Dis is the discriminative loss item, l-GP is the gradient penalty loss item, and δ is a weighting factor (again, we use the same value as the baseline).

The overall loss function of GAN compression consists of generative and discriminative differences:

EQUATION

where λ is the parameter to adjust the percentages between generative and discriminative losses.

We showed promising results with this method above in the context of prior methods. In the following experiments, we investigate how well the method applies to other networks and tasks (Section 5) and how well the method works on different sparsity ratios and pruning granularities (Section 6). 

5. APPLICATION TO NEW TASKS AND NETWORKS:

For the experiments in this section, we choose to prune individual weights in the generator. The final sparsity rate is 50% for all convolution and deconvolution layers in the generator (more aggressive sparsities are discussed in Section 6). Following AGP (Zhu & Gupta, 2018) , we gradually increase the sparsity from 5% at the beginning to our target of 50% halfway through the self-supervised training process, and we set the loss adjustment parameter λ to 0.5 in all experiments. We use PyTorch (Paszke et al., 2017) , implement the pruning and training schedules with Distiller (Zmora et al., 2018) , and train and generate results with a V100 GPU (NVIDIA, 2017) using FP32 to match public baselines. In all experiments, the data sets, data preparation, and baseline training all follow from the public repositories -details are summarized in Table 2 . We start by assuming an extra 10% of the original number of epochs will be required; in some cases, we reduced the overhead to only 1% while maintaining subjective quality. We include representative results for each task, but a more comprehensive collection of outputs for each experiment is included in the Appendix.

Image Synthesis. We apply the proposed compression method to DCGAN (Radford et al., 2016) 3 , a network that learns to synthesize novel images belonging to a given distribution. We task DCGAN with generating images that could belong to the MNIST data set, with results shown in Figure 3 .

Domain Translation.:

We apply the proposed compression method to pix2pix (Isola et al., 2017) 4 , an approach to learn the mapping between paired training examples by applying conditional adversarial networks. In our experiment, the task is synthesizing fake satellite images from label maps and vice-versa. Representative results of this bidirectional task are shown in Figure 4 .

Style Transfer. We apply the proposed compression method to CycleGAN (Zhu et al., 2017a) Image-to-image Translation. In addition to the StarGAN results above (Section 3, Figure 1 ), we apply the proposed compression method to CycleGAN (Zhu et al., 2017a) performing bidirectional translation between zebra and horse images. Results are shown in Figure 6 . (Wang et al., 2004) . We also show FID scores (Heusel et al., 2017) for our results in the results summarized in Table 3 , and a representative output is shown in Figure 7 . These results also include filter-pruned generators (see Section 6).

6. EFFECT OF PRUNING RATIO AND GRANULARITY:

After showing that self-supervised compression applies to many tasks and networks with a moderate, fine-grained sparsity of 50%, we expand the scope of the investigation to include different pruning granularities and rates. From coarse to fine, we can compress and remove the entire filters (3D-level), kernels (2D-level), vectors (1D-level) or individual elements (0D-level). In general, finer-grained pruning results in higher accuracy for a given sparsity rate, but coarser granularities are easier to exploit for performance gains due to their regular structure. Similarly, different sparsity rates, leaving many nonzero weights or few, can result in varying levels of quality in the final network.

We pruned all tasks by removing both single elements (0D) and entire filters (3D). Further, for each granularity, we pruned to final sparsities of 25%, 50%, 75%, and 90%. Representative results for CycleGAN (Monet → Photo) are shown in Figure 8 , but in general, 0D pruning is less invasive, even at higher sparsities. Up to 90% fine-grained sparsity, some fine details faded away in pix2pix, but filter pruning results in drastic color shifts and loss of details at even 25% sparsity.

7. CONCLUSION AND FUTURE WORK:

In this paper, we propose using a pre-trained discriminator to self-supervise the compression of a generative adversarial network. We show that it is effective and applies to many tasks commonly solved with GANs, unlike traditional compression approaches. Comparing the compressed generators with the baseline models on different tasks, we can conclude that the compression method performs well both in subjective and quantitative evaluations. Advantages of the proposed method include:

• The results from the compressed generators are greatly improved over past work.

• The self-supervised compression is much shorter than the original GAN training process.

It only takes 1%-10% training effort to get an optimal compressed generative model. • It is an end-to-end compression schedule that does not require objective evaluation metrics.

• We introduce a single optional hyperparameter (fixed to 0.5 for all our experiments).

We use self-supervised GAN compression to show that pruning whole filters, which can work well for image classification models (Li et al., 2017) , may perform poorly for GAN applications. Even pruned at a moderate sparsity (e.g. 25% in Figure 8 ), the generated image has an obvious color shift and does not transfer the photorealistic style. In contrast, the fine-grained compression stategy works well for all tasks we explored. SRGAN seems to be an exception to filter-pruning's poor results; we have to look closely to see differences, and it's not clear which is subjectively better.

We have not tried to achieve extremely aggressive compression rates with complicated pruning strategies. Different models may be able to tolerate different amounts of pruning when applied to a task, which we leave to future work. Similarly, we have used network pruning to show the importance and utility of the proposed method, but self-supervised compression is general to other techniques, such as quantization, weight sharing, etc. There are other tasks for which GANs can provide compelling results, and newer networks for tasks we have already explored; future work will extend our self-supervised compression method to these new areas. Finally, self-supervised compression may apply to other network types and tasks if a discriminator is trained alongside the teacher and student networks.

Since GANs are typically judged with subjective, qualitative observations, we present several results for each of the experiments in the main paper so readers can see the motivation for the conclusions drawn therein. We organize this document in the following way: 

:

StarGAN baseline implementation: https://github.com/yunjey/StarGAN.

In different GANs, the generative loss may consist of several sub-items. For example, StarGAN combines adversarial loss, domain classification loss and reconstruction loss into overall generative loss.

DCGAN baseline: https://github.com/pytorch/examples/tree/master/dcgan.

Pix2pix, CycleGAN: https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix.

SRGAN baseline implementation: https://github.com/xinntao/BasicSR.

Figures and Tables:

Figure 1: Various approaches to compress StarGAN with network pruning. Each group shows one input face translated with different methods of compressing the network: a. Uncompressed, b. Self-Supervised (ours), c. Small and dense, d. One-shot pruning and fine-tuning, e. AGP as finetuning, f. AGP from scratch, g. One-shot pruning and distilling, h. AGP during distillation, i. AGP during distillation with fixed discriminator, j. Adversarial learning, k. Knowledge distillation, l. Distillation on output of intermediate layers, m. E-M pruning, and n. Prune both G and D models.

Figure 2: Workflow chart of GAN compression process.

Figure 3: Image synthesis on MNIST dataset with DCGAN. Columns 1-3: Handwritten numbers generated by the original generator, pruned generator of 50%, 75% fine-grained sparsity.

Figure 6: Representative image-to-image translation results: CycleGAN.

Figure 7: Representative super resolution results: SRGAN (with enlargements of boxed areas).

Figure 8: Representative results for pruning rate and granularity study of style transfer.

Figure 9: Loss curves of image-to-image translation pruning. (a). Loss curve of StarGAN baseline. Loss curve of training the compressed generator from (b). Self-Supervised fine-tuning (ours), (c). Smaller dense network, (d). One-shot pruning and naive fine-tuning, (e). Gradual pruning and naive fine-tuning, (f). Gradual pruning during the initial training, (g). One-shot pruning and distillation as fine-tuning, (h). Gradual pruning and distillation as fine-tuning, (i). AGP as fine-tuning and distillation without fine-tuning the original loss, (j). Adversarial learning (fine-tuning), (k). Knowledge distillation, (l). Distillation on output of intermediate layers, (m). E-M Quantization, and (n). Prune both G and D models. (a), (c) and (f) start from a randomly-initialized network at epoch 0, others pick up at the end of (a).

Figures 10-12 show outputs of StarGAN compressed with various existing techniques (c-n), and the proposed self-supervised method (b). The baseline output is at the top (a) of each figure for comparison.Each row shows one input face translated to have black hair, blond hair, brown hair, the opposite gender, and a different age, and each row is a different method of compressing the network (the key is identical to that of Figure9).

Figure 10: Example 1 of various approaches to compress StarGAN. 16

Figure 17: Image synthesis: from satellite images to fake label maps. Row 1: Original satellite images, Row 2: Label maps generated by the original generator, Row 3: Label maps generated by the pruned generator, Row 4: Residual difference between generated images in Row 2 and 3, Row 5: Residuals amplified by 10x.

Figure 18: Image synthesis: Two different random seeds, unpruned. Row 1: Original satellite images, Rows 2-3: Generated fake label maps by original generator trained with random seeds 15 and 63, Row 4: Residual difference between generated images in Row 2 and 3. Row 5: Residuals amplified by 10x for higher contrast.

Figure 20: Style transfer: from real photo to Monet artwork style. Row 1: Original photos, Row 2: Monet's style applied by the original generator, Row 3: Monet's style applied by the compressed generator, Row 4: Residual difference between style transferred images in Row 2 and 3 amplified by 10x.

Figure 22: Image-to-image translation experiment: from real horse images to fake zebra images. Row 1: Original real horse images, Row 2: Corresponding translated zebra images by original generator, Row 3: Translated zebra images by compressed generator, Row 4: Residual difference between translated images in Row 2 and 3 amplified by 10x.

Figure 24: Image-to-image translation example 2: facial attribute translation. Columns: 1. Original facial images, 2-4. Translated images to (black, blond, brown) hair colors, 5. Translated images to other gender, 6. Translated images to other age. Rows: Images translated by 1. original generator and 2. compressed generator, 3. Residual difference between Rows 1 and 2, 4. Residuals amplified by 10x.

Figure 25: Image-to-image translation example 3: facial attribute translation. Columns: 1. Original facial images, 2-4. Translated images to (black, blond, brown) hair colors, 5. Translated images to other gender, 6. Translated images to other age. Rows: Images translated by 1. original generator and 2. compressed generator, 3. Residual difference between Rows 1 and 2, 4. Residuals amplified by 10x.

Figure 26: Image-to-image translation example 4: facial attribute translation. Columns: 1. Original facial images, 2-4. Translated images to (black, blond, brown) hair colors, 5. Translated images to other gender, 6. Translated images to other age. Rows: Images translated by 1. original generator and 2. compressed generator, 3. Residual difference between Rows 1 and 2, 4. Residuals amplified by 10x.

Figure 40: Image-to-image translation example 2: filter pruning to different sparsities. Row 1: Baseline generator output. Rows 2-5: Facial attribute translated images by generators pruned to sparsities of 25%, 50%, 75%, 90%.

Figure 41: Image-to-image translation example 2: fine-grained pruning to different sparsities. Row 1: Baseline generator output. Rows 2-5: Facial attribute translated images by generators pruned to sparsities of 25%, 50%, 75%, 90%.

GAN compression comparison (network pruning)

Table content: <table><tr><td/><td colspan="2">Generator(s)</td><td colspan="2">Discriminator</td><td/><td colspan="2">Loss Terms</td><td/><td colspan="2">Results</td></tr><tr><td>Technique</td><td colspan="2">Compressed Init Scheme</td><td>Init Scheme</td><td colspan="5">Fixed L-Gc L-Dc L-Go L-Do</td><td>Qualitative</td><td>FID Score</td></tr><tr><td>(a) No Compression</td><td>Dense</td><td>Random</td><td>Dense,Random</td><td>No</td><td>-</td><td>-</td><td>Yes</td><td>Yes</td><td>Good</td><td>6.113</td></tr><tr><td>(b) Self-Supervised (ours)</td><td colspan="3">Dense,Sparse From Dense Dense,Pretrained</td><td>No</td><td>Yes</td><td>Yes</td><td>Yes</td><td>Yes</td><td>Good</td><td>6.929</td></tr><tr><td>(c) Small &amp; Dense Network</td><td>Dense</td><td>Random</td><td>Dense,Random</td><td>No</td><td>-</td><td>-</td><td>Yes</td><td>Yes</td><td>Mode collapse</td><td>72.821</td></tr><tr><td>(d) One-shot Pruning &amp; Fine-Tuning</td><td>Sparse</td><td colspan="2">From Dense Dense,Pretrained</td><td>No</td><td>Yes</td><td>Yes</td><td>-</td><td>-</td><td>Facial artifacts</td><td>24.404</td></tr><tr><td>(e) Gradual Pruning &amp; Fine-Tuning</td><td>Sparse</td><td>From Dense</td><td>Dense,Random</td><td>No</td><td>Yes</td><td>Yes</td><td>-</td><td>-</td><td>Facial artifacts</td><td>35.677</td></tr><tr><td>(f) Gradual Pruning during Training</td><td>Sparse</td><td>Random</td><td>Dense,Random</td><td>No</td><td>Yes</td><td>Yes</td><td>-</td><td>-</td><td>No faces</td><td>84.941</td></tr><tr><td>(g) One-shot Pruning &amp; Distillation</td><td colspan="2">Dense,Sparse From Dense</td><td>-</td><td>-</td><td>Yes</td><td>-</td><td>Yes</td><td>-</td><td>Mode collapse</td><td>45.461</td></tr><tr><td>(h) (d) &amp; Distillation</td><td colspan="3">Dense,Sparse From Dense Dense,Pretrained</td><td>No</td><td>Yes</td><td>Yes</td><td>Yes</td><td>-</td><td>Color artifacts</td><td>38.985</td></tr><tr><td>(i) (g) &amp; Fix Original Loss</td><td colspan="3">Dense,Sparse From Dense Dense,Pretrained</td><td>Yes</td><td>Yes</td><td>Yes</td><td>-</td><td>-</td><td>Facial artifacts</td><td>15.182</td></tr><tr><td>(j) Adversarial Learning</td><td>Dense,Sparse</td><td>Random</td><td>Dense,Random</td><td>No</td><td>Yes</td><td>Yes</td><td>Yes</td><td>Yes</td><td>Mode collapse</td><td>92.721</td></tr><tr><td>(k) Knowledge Distillation</td><td colspan="2">Dense,Sparse From Dense</td><td>Dense,Random</td><td>No</td><td>Yes</td><td>-</td><td>Yes</td><td>Yes</td><td>Mode collapse</td><td>103.094</td></tr><tr><td>(l) Distill Intermediate (LIT)</td><td colspan="3">Dense,Sparse From Dense Dense,Pretrained</td><td>Yes</td><td>-</td><td>-</td><td>-</td><td>-</td><td>No faces</td><td>194.026</td></tr><tr><td>(m) E-M Pruning</td><td colspan="3">Dense,Sparse From Dense Sparse,Pretrained</td><td>No</td><td>Yes</td><td>Yes</td><td>Yes</td><td>-</td><td>Color artifacts</td><td>159.767</td></tr><tr><td>(n) G &amp; D Both Pruning</td><td colspan="3">Dense,Sparse From Dense Sparse,Pretrained</td><td>No</td><td>Yes</td><td>Yes</td><td>Yes</td><td>-</td><td>Mode collapse</td><td>46.453</td></tr></table>

Tasks and networks overview

Table content: <table><tr><td>Task</td><td>Network</td><td>Dataset</td><td>Resolution</td><td/><td colspan="3">FID Scores when Pruned to</td></tr><tr><td/><td/><td/><td/><td>0% (dense)</td><td>25%</td><td>50%</td><td>75%</td><td>90%</td></tr><tr><td>Image Synthesis</td><td>DCGAN</td><td>MNIST</td><td>64x64</td><td>50.391</td><td colspan="4">50.128 50.634 50.805 51.356</td></tr><tr><td>Domain Translation</td><td>Pix2Pix</td><td>Sat → Map</td><td>256x256</td><td>17.636</td><td colspan="4">17.897 17.990 20.235 24.892</td></tr><tr><td>Domain Translation</td><td>Pix2Pix</td><td>Sat ← Map</td><td>256x256</td><td>30.826</td><td colspan="4">30.628 30.720 34.051 38.936</td></tr><tr><td>Style Transfer</td><td colspan="3">CycleGAN Monet → Photo 256x256</td><td>63.152</td><td colspan="4">63.410 63.662 66.394 70.933</td></tr><tr><td>Style Transfer</td><td colspan="3">CycleGAN Monet ← Photo 256x256</td><td>31.987</td><td colspan="4">32.102 32.346 33.913 41.409</td></tr><tr><td colspan="4">Image-Image Translation CycleGAN Zebra → Horse 256x256</td><td>60.930</td><td colspan="4">61.005 61.102 65.898 68.450</td></tr><tr><td colspan="4">Image-Image Translation CycleGAN Zebra ← Horse 256x256</td><td>52.862</td><td colspan="4">52.631 52.688 58.356 63.274</td></tr><tr><td colspan="2">Image-Image Translation StarGAN</td><td>CelebA</td><td>128x128</td><td>6.113</td><td>6.307</td><td>6.929</td><td>6.714</td><td>7.144</td></tr><tr><td>Super Resolution</td><td>SRGAN</td><td>DIV2K</td><td>≥ 512x512</td><td>14.653</td><td colspan="4">15.236 16.609 17.548 18.376</td></tr></table>

PSNR (dB), SSIM and FID indicators for Validation Datasets .852733 30.761999 30.234316 0.859817 35.514204 30.484014 0.862475 36.824148 Set14 26.643850 0.716294 55.457409 27.314664 0.744525 82.118059 27.417112 0.744101 70.125821 DIV2K Validation 28.205665 0.778364 14.653151 28.875953 0.800625 18.499896 28.974868 0.800767 16.608606

Table content: <table><tr><td>Dataset</td><td colspan="2">Original Generator</td><td/><td colspan="2">Filter-Compressed G</td><td/><td colspan="3">Element-Compressed G</td></tr><tr><td/><td>PSNR</td><td>SSIM</td><td>FID</td><td>PSNR</td><td>SSIM</td><td>FID</td><td>PSNR</td><td>SSIM</td><td>FID</td></tr><tr><td>Set5</td><td>30.063393 0Ground Truth</td><td colspan="2">Dense Generator</td><td/><td colspan="2">Filter-pruned Generator</td><td colspan="2">Fine-grained Generator</td></tr></table>

Peer review (iclr2020):
In this paper, the authors tackle the task of compressing a network. While there are many effective solutions so far regular computer vision tasks, as they demonstrate, they fail catastrophically when applied to generative adversarial networks(GANs). They propose a modification to the classic distillation method, where a *student* network tries to imitate the uncompressed one under the supervision of a fully converged discriminator network. They perform evaluation on multiple tasks from image synthesis to super-resolution. They also study the influence of the compression factor on the quality of the generated images. The task is well motivated and situated in the related literature. The first section is very thorough and extremely efficient at describing the failure modes of existing methods. On one side, the results demonstrated in the evaluation are compelling, on the other side, the compression factor is only 50%, which is much lower than seen in related work. However, as it is shown in section 3 the task may be much harder for GANS than regular models so I still consider it a sizeable contribution. There are a couple of points that require clarification. I personally found the description of the method (Section 4) rather confusing. It is clear what *discriminative loss* is as it is the one used in every GAN. Unfortunately, I could not understand what *generative loss* means in the general case. An example is given for StarGAN in equation (7) and I have a rough idea of what to choose for Style Transfer, Domain Translation, Super Resolution and Image translation. Though, it is unclear to me what to use in the case of image synthesis. The experiments clearly show that it is possible so I think it is necessary to show how this framework is concretely applied to each task at hand. During training, the discriminator only ever saw pictures from the true distribution and the distribution generated by the generator (at each of its training steps). If I understood the framework properly, here, the compressed generator is trained from a random initialization. The distribution it outputs is therefore completely unknown and potentially non overlapping with either of the true or the generator ones. In that case it is hard to predict what the discriminator would do on completely out of distribution samples. I seems reasonable to conjecture that it might consider them *true* because it was never trained on them. Could you provide an explanation of why it is not a problem in practice? Do you have to try multiple initializations? Is the generative loss enough to force the compressed discriminator to match the support of the distribution of the dense generator? I think this paper is novel, tackles a hard task and presents compelling results (albeit using very mild compression ratios). It should be accepted if some clarifications are made in section 3. Minor Remarks: - Figures 2, 9, 42 and 43 are unreadable when printed with a regular color office printer. - It is unclear what it would take an extra 10% of the original number of epochs to train the compressed network. Why couldn*t it be faster, or much longer?