Self-Supervised GAN Compression
Authors: Chong Yu
Jeff Pool
Venue:  (2020)
Keywords: compression
pruning
generative adversarial networks
GAN

Abstract: Deep learning's success has led to larger and larger models to handle more and more complex tasks; trained models can contain millions of parameters. These large models are compute- and memory-intensive, which makes it a challenge to deploy them with minimized latency, throughput, and storage requirements. Some model compression methods have been successfully applied on image classification and detection or language models, but there has been very little work compressing generative adversarial networks (GANs) performing complex tasks. In this paper, we show that a standard model compression technique, weight pruning, cannot be applied to GANs using existing methods. We then develop a self-supervised compression technique which uses the trained discriminator to supervise the training of a compressed generator. We show that this framework has a compelling performance to high degrees of sparsity, generalizes well to new tasks and models, and enables meaningful comparisons between different pruning granularities.

Under review as a conference paper at ICLR 2020
SELF-SUPERVISED GAN COMPRESSION
Anonymous authors Paper under double-blind review
ABSTRACT
Deep learning's 's success has led to larger and larger models to handle more and complex tasks; trained models can contain millions of parameters. These large models are compute- and memory-intensive, which makes it challenge to deploy them with minimized latency, throughput, and storage requirements. Some model compression methods have been successfully applied on image classifica- tion and detection or language models, but there has been very little work com- pressing generative adversarial networks (GANs) performing complex tasks. In this paper, we show that a standard model compression technique, weight prun- ing, cannot be applied to GANs using existing methods. We then develop self- supervised compression technique which uses the trained discriminator to super- vise the training of compressed generator. We show that this framework has a compelling performance to high degrees sparsity, can be easily applied to new tasks and models, and enables meaningful comparisons between different pruning granularities.
INTRODUCTION
Deep Neural Networks (DNNs) have proved successful in various tasks like computer vision, natural language processing, recommendation systems, and autonomous driving. Modern networks are comprised of millions of parameters, requiring significant storage and computational effort. Though accelerators such as GPUs make realtime performance more accessible, compressing networks for faster inference and simpler deployment is an active area of research. Compression techniques have been applied to many networks, reducing memory requirements and improving their performance. Though these approaches do not always harm accuracy, aggressive compression can adversely affect the behavior of the network. Distillation (Schmidhuber, [991 Hinton al. 2015) can improve the accuracy of a compressed network by using information from the original, uncompressed network.
Generative Adversarial Networks (GANs) (Schmidhuber) 1990, Goodfellow et al. 2014) are class of DNN that consist of two sub-networks: a generative model and a discriminative model. Their training process aims to achieve Nash Equilibrium between these two sub-models. GANs have been used in semi-supervised and unsupervised learning areas, such as fake dataset synthesis (Rad- ford et al. 2016 Brock al., 2019), style transfer (Zhu al. 2017b Azadi al., (018), and Image-to-imas translation (Zhu et al. 2017a] Choi et al. 2018). 8). As with networks used in other tasks, GANs have millions of parameters and nontrivial computational requirements.
In this work, we explore compressing the generative model of GANs for more efficient deploy- ment. We show that applying standard pruning techniques, with and without distillation, can cause the generator's 's behavior to no longer achieve the network's goal. Similarly, past work targeted at compressing GANs for simple image synthesis fall short when they are applied to large tasks. In some cases, this result is masked by loss curves that look identical to the original training. By mod- ifying the loss function with a novel combination of the pre-trained discriminator and the original and compressed generators, compression rates with little change in the quality of the compressed generator's ouput. We apply technique several networks and tasks to show generality. Finally, we study the behavior of compressed generators when pruned with different amounts and types of sparsity, finding that filter pruning, technique commonly used for accelerating image classification networks, is not trivially applicable to GANs.
overcome this behavioral degradation and achieve compelling
Our main contributions are:

---

Under review as a conference paper at ICLR 2020
We illustrate that and explain why pruning the generator of GAN with existing methods is unsatisfactory for complex tasks. (Section
We propose self-supervised compression for the generator in a GAN (Section
We show that technique can apply to several networks and tasks (Section
We show and analyze qualitative differences in pruning ratio and granularities. (Section 6
2
RELATED RESEARCH
A common method of DNN compression is network pruning (Han et al. 2015 setting the small weights of trained network to zero and fine-tuning the remaining weights to recover accuracy. Zhu & Gupta (2018) proposed gradual pruning technique (AGP) to compress the model during the initial training process. Wen et al. (2016) proposed a structured sparsity learning method that uses group regularization to force weights towards zero, leading to pruning groups of weights together. Li et al. (2017) pruned entire filters and their connecting feature maps from models, allowing the network to run with standard dense software libraries. Though it was initially applied to image clas- sification networks, network pruning has been extended to natural language processing tasks (See) et al., 2016, Narang al. 2017) and recurrent neural networks (RNNs) of all types vanilla RNNs, GRUs (Cho et al. 2014), and LSTMs (Hochreiter & Schmidhuber 1997). As with classifi- cation networks, structured sparsity within recurrent units has been exploited (Wen et al. 2018).
A complementary method of network compression is quantization. Sharing weight values among a collection of similar weights by hashing (Chen al. 2015) clustering (Han al. (2066) can save storage and bandwidth at runtime. Changing fundamental data types adds the ability to accelerate the arithmetic operations, both in training (Micikevicius et al. 2018) and inference regimes (Jain) et al. 2019).
Several techniques have been devised to combat lost accuracy due compression, since there is always the chance that the behavior of the network may change in undesirable ways when the net- work is compressed. Using GANs to generate unique training data (Liu et al. (0088) and extracting
knowledge from uncompressed network, known distillation (Hinton et al. 2015 can help keep accuracy high. Since the pruning process involves many hyperparameters, [in et al. (2019) use
GAN guide pruning, and Wang al. (2019a) structure compression reinforcement learning problem; both remove some of the burden from the user.
EXISTING TECHNIQUES FAIL TO PRUNE A COMPLEX TASK
Though there are two networks in a single GAN, the main workload at deployment is usually from the generative model, or generator. For example, in image synthesis and style transfer tasks, the final output images are created solely by the generator. The discriminative model (discriminator) is vital in training, but it is abandoned afterward for many tasks. So, when try to apply state-of- thethhr-rt compression methods to GANs, we focus on the generator for efficient deployment. As we will see, the generative performance of the compressed generators is quite poor for the selected image-to- image translation task. We look at two broad categories of baseline approaches: standard pruning techniques that have been applied to compress the generator of a GAN performing image synthesis. We compare to the dense baseline [a], our technique [b], well as a small, dense network with the same number of parameters [c]. (Labels correspond to entries in Table I the overview of all techniques, and Figure results of each technique).
other network architectures, and techniques that were devised
Standard Pruning Techniques. To motivate GAN-specific compression methods, we try variation- s of two state-of-thee-rt pruning methods: manually pruning and fine tuning (Han et al., (015) a trained dense model [d], and AGP (Zhu & Gupta 2018) from scratch [e] and during fine-tuning [f]. We also include distillation (Hinton et al. 2015) to improve the performance of the pruned network with manual pruning [g] and AGP fine-tuning [h]. Distillation is typically optional for other network types, since it is possible to get decent accuracy with moderate pruning in isolation. For very ag- gressive compression or challenging tasks, distillation aims to extract knowledge for the compressed (student) network from original (teacher) network's 's behavior. We also fix the discriminator of [g] to see if the discriminator being weakened by the compressed generator [i].
2

---

Under review as a conference paper at ICLR 2020
Targeted GAN Compression. There has been some work in compressing GANs with methods other than pruning, and only one technique applied to an image-to-image translation task. We first examine two approaches similar to ours. Adversarial training (Wang al. 2018) [j] posits that during distillation of a classification network, the student network can be thought of as a generative model attempting produce features similar to that of the teacher model. So, discriminator was trained alongside the student network, trying to distinguish between the student and the teacher. One could apply this technique to compress the generator of a GAN, but we find that its key shortcoming is that it trains a discriminator from scratch. Similarly, distillation has been used to compress GANs in Aguinaldo et al. (2019) [k], but again, the "teacher" discriminator was not used when teaching the "student" generator.
Learned Intermediate Representation Training (LITT) (Koratana al. 2019) [1] compresses StarGAN
by a factor of 1.8x by training shallower network. Crucially, LIT does not use the pre-trained dis- criminator in any loss function. Quantized GANs (QGAN) (Wang et al., 2019b) [m] use a training process based generative tasks with output images of 32x32 or 64x64 pixels. Liu et al. (018a) find that maintain- ing balance between discriminator and generator is key: their approach is to selectively binarize parts of both networks in the training process on the Celeb-A generative task, up to 64x64 pixels. So, we try pruning both networks during the training process [n].
Iopectation-Maxxmizztinnn to achieve impressive compression results small
Experiments. For these experiments, we use StarGAN CChi et al. 2018) trained with the Dis- tiller (Zmora et al. 2011)) library for the pruning. StarGAn I extends the image-to-image translation capability from two domains to multiple domains within a single unified model. It uses the Celeb- Faces Attributes (CelebA) (Liu et al. 2015) as the dataset. CelebA contains 202,599 images of celebrities' faces, each annotated with 40 binary attributes. As in the original work, we crop the initial images from size 178 x 218 178 178, then resize them to 128 x 128 and randomly select 2,000 images as the test dataset and use remaining images for training. The aim of StarGAN is facial attribute translation: given some image of a face, it generates new images with five domain attributes changed: 3 different hair colors (black, blond, brown), different gender (male/female), and different age (young/old). Our target sparsity is 50% for each approach.
We stress that we attempted to find good hyperparameters when using the existing techniques, but standard approaches like reducing the learning rate for fine-tuning (Han et al. 2015), etc., were not helpful. Further, the target sparsity, 50%, is not overly aggressive, and we do not impose any structure; other tasks readily achieve 80%-90% tie-igained sparsity with minimal accuracy impact.
The results of these trials are shown in Figure I Subjectively, it is easy to see that the existing approaches [Igghrrough] Tnp produce inferior results to the original, dense generator. Translated facial images from pruning & naive fine-tuning and Te) do give unique results for each latent variable, but the images are hardly recognizable as faces. These fine-tuning procedures, along with AGP from scratch Li) and distillation from intermediate representations (T). simply did not converge.
One-shot pruning and traditional distillation (Lg) adversarial learning Li),, knowledge distillation (IK), training "smaller, dense" half-sized network from scratch (Ic) and pruning both generator and discriminator (In) keep facial features intact, but the songe-to-image translation effects are lost to mode collapse (see below). There are obvious mosaic textures and color distortion on the translated images from fine-tuning & distillation (Th), without fine-tuning the original loss [T). and from the pruned model based on the Expectation- Expectation-Maximizzttte Maximization (E-M) algorithm (Im). However, the translated facial images from a generator compressed with our proposed self-supervised GAN compression method (Tb) are more natural, nearly indistinguishable from the dense baseline (Ta) matching the quantitative Frechet Inception Distance (FID) scores (Heusel et al. 2017) in Table I While past approaches have worked to prune some networks on other tasks (DCGAN generating MNIST digits, see the supplementary material), we show that they do not succeed on larger image- to-image translation tasks, while our approach works on both. Similarly, though LIT (Koratanal al. 2019) [I] was able to achieve compression rate of 1.8x on this task by training shallower network, it does not see the same success at network pruning.
Discussion. It is tempting to think that the loss curves of the experiment for each technique can tell us if the result is good or not. We found that for many of these experiments, the loss curves correctly predicted that the final result would be poor. However, the curves for [h] and [m] look very
StarrGNN baseline implementation:
3

---

Under review as a conference paper at ICLR 2020
Table 1: GAN compression comparison (network pruning)
Generatoe(6)
Disertminator
Loss Terms
Results
Technique
Compressed
Init Scheme
Init Scheme
Fixed
L-Ge
L-De
L-Go
L-Do
Quilivative
FID Score
(a) No Compression
Donse
Random
Donse Random
No
Yes
Yes
Good
6.113
Self Supervisdd (ours)
Dpase.parse
Desse Pretraned
No
Yes
Goed
6,929
Small
Network
Random
Randonn
No
Made collagee
72.821
Pruning
Timing
Sparse
Demse.Prttrined
No
Yes
untifacts
(6) Gradual Pruning& & Fine-Tuming Tuming
Sparse
Frum Demse
Dense Randon
No
Yes-
Yes
Fasial antifacts
35.677
(1) Gradual Pruning during Training
Spurse
Random
DauseXandon
No
Yes
Yes
No faces
84,941
(22 One-chet Pruning & Disiilation
Spurse
From Dense
Yos.
Yos
Mede collagee
45 35:46]
0b (d) & Distillation
Dense Spurse
From Dense
Dense, Pretrained
No
Yes
Yes
Yes
Color artifacts
38. 98.
(0(gg) (g) & Fix Original Loss
Danse.Sparc
From Desse
Pietranid
Yes
Yes
Yes
Facial frts
15. 15.182
() Advensriaa Learning
Dense Spure
Random
Deme
No
Yes.
You
Yes
Yes
Mode collapee
92.731
Knowledge Distillation
Deuse Spuree
Ranson
No
Yes
Yes
Mode collapse
103.094
Disill Intemmediati (LJTT)
Spare-Spase
Pretraind
Yes
faces 194,026
Pruning
Spuns
SparsePreeriinn
No
Yes
artifacts
09 & Both Prening
Dyuse-Surre
From Dense
Sparse, Prttinnetttined
No
Yos
Yos
Mode collague
46.453
b
d
k
Figure I: Various approaches to compress StarGAN with network pruning. Each group shows one input face translated with different methods of compressing the network: a. Uncompressed, b. Supervised (ours), c. Small and dense, d. One-shot pruning and tun-tuning. e. AGP as fine- tuning, f. AGP from scratch, g. One-shot pruning and distilling, h. AGP during distillation, i. AGP during distillation with fixed discriminator, j. Adversarial learning, k. Knowledge distillation, Distillation on output of intermediate layers, m. E-M pruning, and n. Prune both G and D models.
good - the compressed generator and discriminator losses converge at 0. just as they did for bascline training. It is clear from the results of querying the generative models (Figures Ih and [m]. though, that this promising convergence is false positive. In contrast, the curves for our technique predict good performance, and, as we prune more aggressively in Section 6 higher loss values correlate well with worsening FID scores. (Loss curves provided in the Appendix.)
As pruning and distillation are very effective when compressing models for image classification tasks, why do they fail to compress this generative model? We share three potential reasons:
4

---

Under review as a conference paper at ICLR 2020
1. Standard pruning techniques need explicit evaluation metrics; softmax easily reflects the probability distribution and classification accuracy. GANs are typically evaluated subjec- tively, though some imperfect quantitative metries have been devised.
2. GAN training is relatively unstable (Arjovsky et al., 2017 Liu et al. 2018a and sensitive to hyperparameters. The generator and discriminator must be well-matched, and pruning can disrupt this fine balance.
3. The energy of the input and output of GAN is roughly constant, but other tasks, such as classification, produce an output (1-hot label vector) with much less entropy than the input (three-channel color image of thousands of pixels).
Elaborating on this last point, there is more tolerance in the reduced-informatio space for the com- pressed classification model to give the proper output. That is, even if the probability distribution inferred by the original and compressed classification models labels can be the same. On the other hand, tasks like style-transtee and dataset synthesis have no obvious energy reduction. We need to keep entropy as high as possible (Kumar et al. 2019) during the compression process to avoid mode collapse generating the same output for different inputs or tasks. Attempting train new discriminator make the compressed generator behave more like the original generator (Wang et al. 2018) suffers from this issue the new discriminator quickly falls into low-entropy solution and cannot escape. Not only does this preclude its use on genera- tive tasks, but it means that the compressed network for any task must also be trained from scratch during the distillation process, or the discriminator will never be able learn.
not exactly the same, the classified
SELF-SUPERVISED GENERATOR COMPRESSION
We seek to solve each of the problems highlighted above. Let restate the general formulation of GAN training: the purpose of the generative model is to generate new samples which are very similar to the real samples, but the purpose of the discriminative model is to distinguish between real samples and those synthesized by the generator. A fully-trained discriminator is good at spotting differences, but a well-trained generator will cause it to believe that the generated sample is both real and generated with a probability of 0.5. Our main insight follows:
By using this powerful discriminator that is already well-trained on the target data set, can allow it to stand in as a quantitative subjective judge (point 1, above) - if the discriminator can't 't tell the difference between real data samples and those produced by the compressed generator, then the compressed generator is of the same quality as the uncompressed generator. A human no longer needs to inspect the results to judge the quality of the compressed generator. This also addresses our second point: by starting with a trained discriminator, we know it is well-matched to the generator and will not be overpowered. Since to avoid mode collapse. As distillation progresses, it can adapt to and induce fine changes in the compressed generator, which is initialized from the uncompressed generator.
so capable (there is no need to prune it to), also helps
Since the original discriminator is used as proxy for a human's 's subjective evaluation, we refer to this as "uulf-supeevviee"" compression. We illustrate the workflow in Figure 2 using a GAN charged with generating& a map image from satellite image in domain translation task.
In the right part of Figure 2 the real satellite image (x) goes through the original generative model (Go) to produce fake map image (f.). The corresponding generative loss value is I-Go. Accord- ingly, in the left part of Figure the real satellite image (x) goes through the compressed generative model (Ge) produce fake map image (j.). The corresponding generative loss value is I-Go- This is the inference process of the original and compressed generators, expressed as follows:
je = Go(z)
(1)
je = Gx
(2)
The overall generative difference is measured between the two corresponding generative losse use generative consistent loss function (Loc) in the bottom of Figure to represent this process.
el We
Lec L-G + 0
In different GANs, the generative loss may consist of several sub-items. For example, StarGAN combines adversarial loss, domain classification loss and reconstruction loss into overall generative loss.
5

---

Under review as a conference paper at ICLR 2020
Real Map Image:
Original Discriminator: Dg
Discriminative Loss: I-D,
Discriminative Consistent Lass Function: Lor
Discriminative Loss: I.Do
Original Discriminator: Do
Compressed Generator: G.)
Original Generator: Go
Fake Map Image from Ge &.
Real Satellite Image:
Fake Map Image Go: Fo
Generative Loss: I-G
Generative Consistent
Generative Loss: I.Go
Loss Function: Agc
Figure 2: Workflow chart of GAN compression process.
Since the GAN training process aims to reduce the differences between real and generated samples, we stick to this principle in the compression process. In the upper right of Figure 2 real map image (y) and fake map image (fo) go through the original discriminative model Do. Do tries to ensure that the distribution of G. is indistinguishable from y using an adversarial loss. The corresponding discriminative loss value is I-Do- In the upper left of Figure 2 real map image (y) and fake map image (je) also go through the original discriminative model Do. In this way, we use the original discriminative model "self-supervisor" The corresponding discriminative loss value I-Dc.
I-Do Do(y.o))
I-Dc Do Uc)
(5)
So the discriminative difference is measured between two corresponding discriminative losses. We use the discriminative consistent loss function LDc in the top of Figure 2 to represent this process.
Loc(l-D,,-DD)) ->
The generative and discriminative consistent loss functions (Lag and Lnc) use the weighted nor- malized Euclidean distance. Taking the StarGAN task as the example (other tasks may use different losses):
L-G(I-G,,-GC)) |1-Geno - I-Gencl//l-Genn + all-Clao - I-Clacl/
+B|1-Reco I-Recol/ ||Reco
where I-Gen is the generation loss term, I-Cla is the classification loss term, and I-Rec is the recon- struction loss term. a and 3 are the weight ratios among three loss types. (We use the same values of a and B used in the original StarGAN baseline.)
LDc (I- I-Dc) |[- I-Dis clDiscl/|I-Diso + 811-GPo I-GPel//I-GPPPP
(8)
where I-Dis is the discriminative loss item, I-GP is the gradient penalty loss item, and is a weighting factor (again, we use the same value as the baseline).
The overall loss function of GAN compression consists of generative and discriminative differences:
Loverell = Lcc(I-Go, + XLDc(l-o,,--D)))
where À is the parameter to adjust the percentages between generative and discriminative losses.
We showed promising results with this method above in the context of prior methods. In the follow- ing experiments, we investigate how well the method applies to other networks and tasks (Section and how well the method works on different sparsity ratios and pruning granularities (Section

---

Under review as a conference paper at ICLR 2020
Table 2: Tasks and networks overview
FID Scores when Pruned
Task
Network
Dutaset
Resolution
(dense)
25%;
50%
Image Synthesis
DOGAN
MNIST
64x64
50.391
50.128
50.634
50.805
51,356
Domain Translation
Pix2Pxx
Sat + Mup
256x256
17.636
17.897
17.990
20.235
24.892
Domain Translation
Pix2Pix
Sat + Map
256x256
30.826
30.628
30.720
34.051
38,936
Style Transfer
CycleGAN
Monet + Photo
256x256
63.152
63.410
63.662
66.394
70,933
Style Transfer
CycleGAN
Monet Photo
256x256
31.987
32.102
32.346
33.913
41.409
Image-Image Translation
CycleGAN
Zebra Horse
256x256
60.930
61.005
61.102
65,898
68.450
Image= Image Translation
CycleGAN
Zehra 4 Horse
256x256
52.862
52.631
52.688
58.356
63.274
image-Image Translation
SturGAN
128x128
6.113
6,307
6,929
6.714
7.144
Super Resolution
SRGAN
DIV2K
2512x512
14,653
15.236
16.609
17.548
18.376
5 APPLICATION TO NEW TASKS AND NETWORKS
For the experiments in this section, we choose to prune individual weights the generator. The final sparsity rate is 50% for all convolution and deconvolution layers in the generator (more aggressive sparsities are discussed in Section 6). Following AGP (Zhu & Guptal 2018). we gradually increase the sparsity from 5% at the beginning our target of 50% halfway through the self-supervised training process, and we set the loss adjustment parameter À to 0.5 in all experiments. We use PyTorch (Paszke et al. 2017), implement the pruming and training schedules with Distiller (Zmora et al. 2018), and train and generate results with a V100 GPU (NVIDIA) 2017) using FP32 to match public baselines. all experiments, the data sets, data preparation, and baseline training all follow from the public repositories details are summarized in Table We start by assuming an extra 10% of the original number of epochs will be required; in some cases, we reduced the overhead to only 1% while maintaining subjective quality, We include representative results for each task, but more comprehensive collection outputs for each experiment included the Appendix.
Image Synthesis. We apply the proposed compression method to DCGAN (Radford et al. 2016 a network that learns to synthesize novel images belonging to given distribution. We task DCGAN with generating images that could belong to the MNIST data set, with results shown in Figure 3
Domain Translation. We apply the proposed compression method to pix2pix (Isola et al. 2017 approach to learn the mapping between paired training examples by applying conditional adver- sarial networks. In our experiment, the task is synthesizing fake satellite images from label maps and vice-versa. Representative results of this bidirectional task are shown in Figure
Style Transfer. We apply the proposed compression method to CycleGAN (Zhu et al. 2017a). used exchange the style of images from source domain to a target domain in the absence of paired training examples. In our experiment, the task is to transfer the style of real photos with that
2
8
4
3
3
4
4
3
2
8
3
9
4
4
3
28
1
5
4
L 4 1.221 I.
4
7
3
0
4
8
2 7
1
7
3
6
1
9
1
250
/
0
/
8
9
5
3
/ 6 8 5 2 6
7
/ 9
1
5
1 7 1 4 0
7 6 8 1 8 5
7//
117/ 1 + 7.67
3
7
3
9
7
8
3
1
4
2 27
5
6
4
9 y
/ 4 6
5
6
6 6 3
6
# 1
5 5664
/
Q
3
6 1
8
1
0
8 2
9 2
4
0 6
4
2
2
3
0
3
2 2
1
2
7
8
1
9
8
/
7 4
7
4
5
7 4
7
4
5 5
1
FID: 32.7786
33.3191
82.1903
Figure 3: Image synthesis on MNIST dataset with DCGAN. Columns 1-3: Handwritten numbers
generated by the original generator, pruned generator of 50%, 75% fine-grained sparsity.
'DCGAN baseline: ht httpa//twwwt..tt...o....//c////a https::/tt///t http://wwwa////////////////////a

---

Under review as a conference paper at ICLR 2020
Input
Dense
50% Sparse
Difference (10x)
Map to Satellite
FID:
35.627
33.366
Satellite
Map
FID:
17.097
17.945
Figure 4: Representative results for domain translation: pix2pix.
Input
Dense
50% Sparse
Difference (10x)
Monet to Photo
FID:
59.381
60.546
Photo to Monet
FID:
35.78/
34.263
Figure 5: Representative results for style transfer: CycleGAN.
of the Monet's paintings. Representative results of this bidirectional task shown in Figure photographs are given the style of Monet's paintings and vice-versa.
Image-to-image Translation. addition the StarGAN results above (Section Figure apply the proposed compression method to CycleGAN (Zhu et al. 2017a) performing bidirectional translation between zebra and horse images. Results shown Figure
Input
Dense
50% Sparse
Difference (10x)
Zebra to Horse
FID:
47.929
48.112
Horse Zebra
FID:
52.627
53.165
Figure 6: Representative mmage-to-image translation results: CycleGAN
8

---

Under review as a conference paper at ICLR 2020
Table PSVR (dB). SSIM and FID indicators for Validation Datasets
Original Generator
Filter Compressedd
Element Compressed G
Dataset
PSNR
SSIM
FID
PSNR
SSIM
FID
PSNR
SSIM
FID
Set5 Set14
30,063393
0.852733
30.761999
30.234316
0.859817
35.514204
30.484014
0.862475
36.824148
26.643850
0,716294
55.457409
27.314664
0,744525
82.118059
27.417112
0,744101
70.125821
Validaaino
28.205665
0.778364
14.653151
28.875953
0.800625
18,499896
28.974868
0.800767
16.608606
Dense
Filter-pruned Generator
Fine-grained Generator
Ground Truth
Generator
Figure 7: Representative super resolution results: SRGAN (with enlargements of boxed areas).
Super Resolution. We apply self-supervised compression to SRGAN (Ledig et al. 2017 which uses a discriminator network trained to differentiate between upscaled and the original high- resolution images. We trained SRGAN the DIV2K data set Agustsson & Timofte
and use
the DIV2K validation images, as well as Set5 Bevilacqua et al. (2012) and Set Zeyde et al. (2010) to report deployment quality. In this task, quality is often evaluated by two metrics: Peak Signal-to- Noise Ratio (PSNR) (Huynh-Thu & Ghanbari 2008) and Structural Similarity (SSIM) (Wang et al.) 2004 We also show FID scores (Heusel al., 2017) for our results in the results summarized in Table 3, and representative output is shown in Figure 7 These results also include filter- filter-pruned generators (see Section 6).
EFFECT OF PRUNING RATIO AND GRANULARITY
After showing that sulf-supe compression applies to many tasks and networks with a mod- erate, fine-grained sparsity of 50%, we expand the scope of the investigation to include different pruming granularities and rates. From coarse to fine, we can compress and remove the entire filter- (3D-level), kernels (2D-level), vectors (ID-level) finer-grained pruning results in higher accuracy for a given sparsity rate, but coarser granularities easier to exploit for performance gains due to their regular structure. Similarly, different spar- sity rates, leaving many nonzero weights or few, can result in varying levels of quality in the final network.
individual elements (OD-level). In general,
We pruned all tasks by removing both single elements (OD) and entire filters (3D). Further, for each granularity, we pruned to final sparsities of 25%, 50%, 75% and 90%. Representative results for CycleGAN (Monet + Photo) are shown in Figure g but in general. OD pruning is less invasive, even higher sparsities. Up 90% fine-grained sparsity, some fine details faded away in pix2pix, but filter pruning results in drastic color shifts and loss of details at even 25% sparsity.
7 CONCLUSION AND FUTURE WORK
In this paper, we propose using pre-trained discriminator to self-supervise the compression of generative adversarial network. We show that effective and applies to many tasks commonly solved with GANs, unlike traditional compression approaches. Comparing the compressed gener- ators with the baseline models on different tasks, we can conclude that the compression method
"SRGAN haseline implementation: https://githuu.t. http.////ommimmto/aaaaaaaaaaaaaa

---

Under review as a conference paper at ICLR 2020
Sparsity 0%
25%
50%
75%
90%
Fine-grained
FID:
32.006
32.462
33.387
34,543
41,251
Filter-pruned
FID:
32.006
82.349
105.884
182.277
204.795
Figure 8: Representative results for pruning rate and granularity study of style transfer.
performs well both subjective and quantitative evaluations. Advantages of the proposed method include:
The results from the compressed generators are greatly improved over past work.
The self-supervised compression is much shorter than the original GAN training process.
It only takes 1%-10% training effort get an optimal compressed generative model.
It is an end-to-end compression schedule that does not require objective evaluation metrics.
We introduce single optional hyperparameter (fixed to 0.5 for all our experiments).
We use supf-sspe GAN compression to show that pruning whole filters, which can work well for image classification models (Li et al. 2017), may perform poorly for GAN applications. Even pruned at a moderate sparsity (e.g. 25% in Figure [). the generated image has an obvious color shift and does transfer the photorealistic style. In contrast, the fine-graines compression stategy works well for all tasks we explored. SRGAN seems to be an exception to filter-pruning's 's poor results; have to look closely to see differences, and it's not clear which subjectively better.
We have not tried to achieve extremely aggressive compression rates with complicated pruning strategies. Different models may be able to tolerate different amounts of pruning when applied to a task, which we leave to future work. Similarly, we have used network pruning to show the importance and utility of the proposed method, but self-supervised compression is general other techniques, such quantization, weight sharing, etc. There are other tasks for which GANs can provide compelling results, and newer networks for tasks we have already explored; future work will extend self-supervised compression method to these new areas. Finally, self-supervised compression may apply to other network types and tasks if discriminator trained alongside the teacher and student networks.
10

---

Under review as a conference paper at ICLR 2020
REFERENCES
Angeline Aguinaldo, Ping- Yeh Chiang, Alexander Gain, Ameya Patil, Kolten Pearson, and Soheil Feizi. Compressing GANs using knowledge distillation. CoRR, abs/1902.00159, 2019. URL
http://arxiv. abs 1902 00159
Eirikur Agustsson and Radu Timofte. Ntire 2017 challenge on single image super-resolution: Dataset and study. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition Workshops, pp. 126-135, 2017.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International Conference on Machine Learning, pp. 214-223, 2017.
Samanch Azadi, Matthew Fisher, Vladimir G Kim, Zhaowen Wang, Eli Shechtman, and Trevor Darrell. Multi-content gan for few-shot font style transfer. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 7564-7573, 2018.
Marco Bevilacqua, Aline Roumy, Christine Guillemot, and Marie Line Alberi-Morel. Low- complexity single-image super-resolution based nonnegative neighbor embedding. 2012.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. In International Conference Learning Representations, 2019.
Wenlin Chen, James Wilson, Stephen Tyree, Kilian Q Weinberger, and Yixin Chen, Compressing neural networks with the hashing trick. In Proceedings the 32nd International Conference on Machine Learning Volume 37, ICML'15, pp. 2285 2294. JMLR.org, 2015. URL http: //dl acll.acm.com wwt.://rrrrttttttooocoomm hwt.:fff2d3304511111 30453366111
Kyunghyun Cho, Bart van Merriënboer, Ça?lar Gülçehre, Dzmitry Bahdanau, Fethi Bougares, Hol- ger Schwenk, and Yoshua Bengio. Learning phrase representations using encoder-decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Meth- ods in Natural Language Processing (EMNLP), pp. 1724-1734. Doha, Qatar, October 2014. As- sociation for Computational Linguistics. URL http:/ www http://wwwwww......attttooooo//
014-1179
Yunjey Choi, Minje Choi, Munyoung Kim, Jung-Woo Ha. Sunghun Kim, and Jacgul Choo. Star- gan: Unified generative adversarial networks for multi-domain image-to-image translation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8789-
8797, 2018
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances neural infor- mation processing systems, pp. 2672-2680, 2014.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In Advances in neural information processing systems, pp. 1135-1143, 2015.
Song Han, Huizi Mao, and William J. Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. In International Conference on Learning Representations, 2016.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge local nash equilibrium. In Advances in Neural Information Processing Systems, pp. 6626-6637, 2017.
Geoffrey Hinton. Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in neural network. In NIPS Deep Learning and Representation Learning Workshop, 2015. URL http::/aaaa
org/ arg/abs/ 503. 02531
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9()): 1735-1780, 1997.
11

---

Under review as a conference paper at ICLR 2020
Q. Huynh-Thu and M. Ghanbari. Scope of validity of psnr in image/video quality assessment. Electronics Letters, 44(13):800-800 801, June 2008. ISSN 0013-5194. doi: 10.10499cl:20080522.
Phillip Isola, Jun Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and paltern recognition, pp. 1 1125-1134, 2017.
Sambhav R. Jain, Albert Gural, Michael Wu, and Chris Dick. Trained uniform quantization for accurate and efficient neural network inference on fixed-point hardware. CoRR, abs/ 1903.08066, 2019.
1190. 08066
Animesh Koratana, Daniel Kang, Peter Bailis, and Matei Zaharia. LIT: Learned intermediate rep- resentation training for model compression. In Proceedings of the International Conference Machine Learning, ICML' 19, 2019.
Rithesh Kumar, Anirudh Goyal, Aaron Courville, and Yoshua Bengio. Maximum entropy generators for energy-based models. arXiv preprint (xXiv:9901.080 /90.19911..00... 2019.
Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro A- costa, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, al. Photo-realistic single image super-resolution using a generative adversarial network. Proceedings of the IEEE con- ference on computer vision and pattern recognition, pp. 4681 1-81-469,, 2017.
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. In International Conference Learning Representations, 2017.
Shaohui Lin, Rongrong Ji, Chenqian Yan, Baochang Zhang, Liujuan Cao, Qixiang Ye, Feiyue Huang, and David Doermann. Towards optimal structured cnn pruning via generative adversarial learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Jinglan Liu, Jiaxin Zhang, Yukun Ding, Xiaowei Xu, Meng Jiang, and Yiyu Shi. PBGen: partial binarization of deconvolution based generators. CoRR, abs/ Abs//802.09153, 2018a. URL http:
//axxis.crrg/ass/11022 09153
Ruishan Liu, Nicolo Fusi, and Lester Mackey. Model compression with generative adversarial networks. In International Conference on Learning Representations, 2018b.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face autributes in the wild. In Proceedings of the IEEE international conference on computer vision, pp. 3730-3738, 2015.
Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory F. Diamos, Erich Elsen, David Gareía, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. Mixed precision training. In International Conference Learning Representations, 2018.
Sharan Narang, Erich Elsen, Gregory Diamos, and Shubho Sengupta. Exploring sparsity in recurrent
neural networks. In International Conference on Learning Representations, 2017.
NVIDIA. NVIDIA Tesla V100 GPU architecture, 2017.
URL
https:
7/1 //mmages.. nvidia a httpd/aa....oooooooooo//oeaaaaaa peppaperr
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In WIPS-WW 2017.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. In International Conference on Learning Repre- sentations, 2016.
J Schmidhuber. Making the world differentiable: On using fully recurrent self-supervised neural net- works for dynamic reinforcement learning and planning in non-stationary environments. Institur für Informatik, Technische Universität München. Technical Report FKI-126, 90, 1990.
12

---

Under review as a conference paper at ICLR 2020
Jürgen Schmidhuber. Neural sequence chunkers. 1991.
Abigail See, Minh-Thang Luong, and Christopher D. Manning. Compression of neural machine translation models via pruning. In CoNLL, 2016.
Kuan Wang, Zhijian Liu, Yujun Lin, Ji Lin, and Song Han. Haq: Hardware-aware automated quan- tization. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2019a.
Peigi Wang, Dongsheng Wang. Yu Ji, Xinfeng Xie, Haoxuan Song, XuXin Liu, Yonggiang Lyu, and Yuan Xie. QGAN: quantized generative adversarial networks. CoRR, boss/1901.0826. 2019b.
URL http://arxiv.cm org/ abs /tts//9911 08263
Yunhe Wang. Chang Xu, Chao Xu, and Dacheng Tao. Adversarial learning of portable student networks. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility structural similarity. IEEE transactions on imuge processing, 13(4):600- 612, 2004.
Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured sparsity in deep neural networks. In Advances in neural information processing systems, pp. 2074-2082, 2016.
Wei Wen, Yuxiong He, Samyam Rajbhandari, Minjia Zhang, Wenhan Wang, Fang Liu, Bin Hu, Yiran Chen, and Hai Li. Learning intrinsic sparse structures within long short-term memory. In International Conference on Learning Representations, 2018.
Roman Zeyde, Michael Elad, and Matan Protter. On single image scale-up using sparse- representations. In International conference curves and surfaces, pp. 711-730. Springer, 2010.
Jun-Ya Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE International Conference Computer Vision, pp. 2223-2232. 2017a
Jun-Yan Zhu, Richard Zhang, Deepak Pathak, Trevor Darrell, Alexei Efros, Oliver Wang, and Eli Shechtman. Toward multimodal image-to-image translation. In Advances in Neural Information Processing Systems, pp. 465 4765,476,200 2017b.
Michael Zhu and Suyog Gupta. To prune, or not to prune: exploring the efficacy of pruning for model compression. In International Conference on Learning Representations, 2018.
Neta Zmora, Guy Jacob, and Gal Novik. Neural network distiller, June 2018. URL https:
//doi (tdoi..or//1.53000 wttp////.../////o/o/l/oo.oooo 1297430
13

---

Under review as a conference paper at ICLR 2020
A APPENDIX
Since GANs are typically judged with subjective, qualitative observations, we present several results for each of the experiments in the main paper readers can see the motivation for the conclusions drawn therein. We organize this document in the following way:
Section
A.
Naïve Compression with StarGAN
Section
A.2
Image Synthesis with DCGAN
Section A.3 3
Domain Translation with Pix2Pix
Section 4
Style Transfer with CycleGAN
Section A.5
Image-Image Translation with CycleGAN
Section 6
Image-Image Translation with StarGAN
Section
A.7
Super Resolution with SRGAN
Section
A.8
Effect of Sparsity Granularity and Ratio
14

---

Under review as a conference paper at ICLR 2020
A.1 NAïVE COMPRESSION: STARGAN
The loss curves for the comparative experiment in Figure are shown in Figure
a
m
Loss curves of image-to-image translation pruning. (a). Loss curve of StarGAN baseline. Loss curve of training the compressed generator from (b). Self-Supervised fine-tuning (ours), (c). Smaller dense network, (d). One-shot pruning and naive fine-tuning, (e). Gradual pruning and naive fine-tuning, (f). Gradual pruning during the initial training, (g). One-shot pruning and distillation as fine-tuning, (h). Gradual pruning and distillation as fine-tuning, (i). AGP as fine-tuning and distil- lation without fine-tuning the original loss, (j). Adversarial learning (fine-tuning), (k). Knowledge distillation,
Figure 9:
(I). Distillation on output of intermediate layers, (m). E-M Quantization, and (n). Prune both G and D models. (a), (c) and (f) start from randomly-initialized network at epoch 0, others pick up at the end (a).
Figures To 12 show outputs of StarGAN compressed with various existing techniques (c-n), and the proposed sulf-supervised method (b). The baseline output at the top (a) of each figure for comparison. Each row shows one input face translated to have black hair, blond hair, brown hair, the opposite gender, and different age, and each row a different method of compressing the network (the key is identical to that of Figure 9).
15

---

Under review as a conference paper at ICLR 2020
01
101
O1
101
101
101
RO
101
01
01
P
m
OI
HOT
Figure 10: Example of various approaches to compress StarGAN,
16

---

Under review as a conference paper at ICLR 2020
m
Figure 11: Example 2 of various approaches to compress StarGAN,
17

---

Under review as a conference paper at ICLR 2020
m
Figure 12: Example 3 of various approaches to compress StarGAN,
18

---

Under review as a conference paper at ICLR 2020
A.2
IMAGE SYNTHESIS: DCGAN (50% AND 75%
FINE- GRAINED SPARSITY)
7
D
9
6
2
3
4
8
2
6
6
8
8
4
3
2
6
6
8
0 0 3 9
3
6
6
0
3
4 6
5
5
6
4
0
4 6 / 9 7 8 /
0
5
5
6
6
0
0
4
4
6
6
0
8
33
4
6
6
4
6
1
6
6
$
2
9
6 2. 1 9 9 4 2 7 6 8 5 2 6
6
4 6
9
/
4 6 2
9 9
/ 9 7 9 // 9
9
7
5
9
7
9
8
7
3
, 7 33777
8
7
6
5
4
8
6
? J
6
5
6
?
J 6 5
6
1
S
9
5
7
9 6
+
3
4
1
9
+ 7 8
4
1
7
7
/
0
7 2 8 3
4
/ 9
0
045
7
4 1
/ 9
045
2
8
4
J
3
4 8 3
3
2
3
44
2
3
4
7
3
6
4
1
7 7 / 9
0 / 2
9
1)
7 3 / 2 6 6 1
/
9
1)/
0
/
9
5
/
1 1
5 4 4 1
1777 7 6 1 8 4 0
7
/ 9
1
5 17/
3 7
3
9
8 9 y 0 8
7 / 4 6
8
8
1
4
7 8 1
67
5 4
d
4
5
6 3 2
6 6 1
6 9
5 O
6 8 2
6 9
4
Q
3
6
8
1
O
44
1
4
2
2
3
1
à
2
8 5
3
2
2
0
3
7 6
1
9
/
7 4
7
4
5
7
7 4
4
5 5 1
FID: 37. 1296
38,6406
42.6728
Figure 13: Image synthesis on MNIST dataset with DCGAN pruned to 50% with fine-grained spar- sity. Column 1: Handwritten numbers generated by the original generator, 2: Handwritten numbers generated by the generator pruned with our method, 3: Handwritten numbers generated by the pruned generator with traditional knowledge distillation adapted for GANs (Aguinaldo al. 2019
4
9
D
6
6
2
3
4
8
2
6
6
0
F
8
2
6
6 &
D 4 a 6 6 3
3
6
6
8
3 3
4
6
05
6
4
9
4
0
p
A
0
9
4
6
4
6
0
3
4 /
6 660.7711
6
4
/
5
5
9
4
9
/
6 3 / 9 9
6
6
9
7
5
9
1
9
9
1757111 10 8
/
173711198
8
7
6
5
4
8
6
6.2.56641
7
4
5
7
9
1 1771
L 1.64197.1
7
7
9
0
6
7 1
7
7 7
2
8
4
J
3
9 8 3 8 8 y
9 2 7 6 8 5 2 6
3
2 18119444
2 88137444
7
O
4
7550411
1.
7
0
9
5
/
1/11111
1/11117:
3
7
J
$
9
7 / 4 6
2 7
1
5
6
4
5 6
6 1
&
S 56 6668
Q
3
6 1
8
1
0
8
6
/
4144 4
0 8 6 9
A
2
3
1
0
2 2
1
2
283
2 2 1
7
1
9
8
/
7 4
7
4
6 5
7 4 4
FID: 37. 1296
118.3160
172.9123
Figure 14: Image synthesis MNIST dataset with DCGAN of 75% fine-grained sparsity. Column 1: Handwritten numbers generated by the original generator, 2: Handwritten numbers generated by the generator pruned with our method, Column 3: Handwritten numbers generated by the pruned generator with traditional knowledge distillation adapted for GANs.
19

---

Under review as a conference paper at ICLR 2020
A.3 DOMAIN TRANSLATION: PIX2PIX (50% TIN-IRRINND SPARSITY)
Figure 15: Image synthesis: from label maps to fake satellite images. Row 1: Original label maps, Row 2: Satellite images generated by the original generator, Row 3: Satellite images generated by the pruned generator, Row 4: Residual difference between generated images in Row and 3, Row 5: Residuals amplified by 10x.
Figure 16: Image synthesis: Two different random seeds, unpruned. Row 1: Original label maps,
Rows 2-3: Generated fake satellite images by original generator trained with random seeds 15 and 63, Row 4: Residual difference between generated images in Row and 3. Row 5: Residuals amplified by 10x for higher contrast.
20

---

Under review as a conference paper at ICLR 2020
Figure 17: Image synthesis: from satellite images to fake label maps. Row 1: Original satellite
images, Row 2: Label maps generated by the original generator, Row 3: Label maps generated by the pruned generator, Row 4: Residual difference between generated images in Row 2 and 3, Row 5: Residuals amplified by 10x.
Figure 18: Image synthesis: Two different random seeds, unpruned. Row Original satellite images, Rows 2-3: Generated fake label maps by original generator trained with random seeds 15 and 63, Row 4: Residual difference between generated images in Row 2 and Row Residuals amplified by 10x for higher contrast.
21

---

Under review as a conference paper at ICLR 2020
A.4 STYLE TRANSFER: CYCLEGAN (50% SPARSITY)
Figure 19: Style transfer: from Monet to real photo style. Row 1: Original artwork images from Monet, Row 2: photographic style applied by the original generator, Row 3: photographic style applied by the compressed generator, Row Residual difference between style transferred images in Row 2 and 3, amplified by 10x.
Figure 20: Style transfer: from real photo to Monet artwork style. Row 1: Original photos, Row 2: Monet's style applied by the original generator, Row 3: Monet's style applied by the compressed generator, Row 4: Residual difference between style transferred images in Row 2 and 3 amplified by 10x.
22

---

Under review as a conference paper at ICLR 2020
A.5
IMAGE-IMAGE TRANSLATION: CYCLEGAN (50% FINE- HINEGGRINND SPARSITY)
Figure 21: Image-to-image translation experiment: from real zebra images fake horse images. Row 1: Original real zebra images, Row 2: Corresponding translated horse images by original generator, Row Translated horse images by compressed generator, Row Residual difference between translated images in Row 2 and 3 amplified by 10x.
Figure 22: Image-to-image translation experiment: from real horse images to fake zebra images. Row 1: Original real horse images, Row 2: Corresponding translated zebra images by original generator, Row 3: Translated zebra images by compressed generator, Row 4: Residual difference between translated images in Row 2 and 3 amplified by 10x.
23

---

Under review as a conference paper at ICLR 2020
A.6 IMAGE-IMAGE TRANSLATION: STARGAN (50% FINI-GRAINED SPARSITY)
P
P UI
P
U1 P or
01
UI
P
P
P
P
P
o
UI
OI
UI
U!
Figure 23: Image-to-imet translation example facial attribute translation. Columns: Original facial images, 2-4. Translated images (black, blond, brown) hair colors, 5. Translated images to other gender, Translated images to other age. Rows: Images translated by 1. original generator and 2. compressed generator, 3, Residual difference between Rows I and 2, Residuals amplified by 10x.
Figure 24: togectto-imaee translation example facial attribute translation. Columns: Original facial images, 2-4. Translated images to (black, blond, brown) hair colors, 5. Translated images to other gender, Translated images to other age. Rows: Images translated by 1. original generator
and 2. compressed generator, 3. Residual difference between Rows I and 2, 4. Residuals amplified by 10x.
24

---

Under review as a conference paper at ICLR 2020
Figure 25: Image-to-image translation example facial attribute translation. Columns: Original facial images, 2-4. Translated images (black, blond, brown) hair colors, 5. Translated images to other gender, 6. Translated images to other age. Rows: Images translated by 1. original generator and 2. compressed generator, 3. Residual difference between Rows I and 2, Residuals amplified by 10x.
Figure 26: Image-to-image -mmge translation example 4: facial attribute translation. Columns: 1. Original facial images, 2-4. Translated images to (black, blond, brown) hair colors, 5. Translated images to other gender, Translated images to other age. Rows: Images translated by 1. original generator and 2. compressed generator, 3. Residual difference between Rows and 2, Residuals amplified by 10x.
25

---

Under review as a conference paper at ICLR 2020
A.7
SUPER RESOLUTION: SRGAN (50%
AND FILTER-PRUNED SPARSITY)
Ground Truth
Baseline
Filter-Pruned
Figure 27: Super resolution experiment. Column 1: Original high resolution images, Columns 2-4: Corresponding generated real high resolution images by original, filter-compressed, element- compressed generators. Each second row provides a detailed view of boxed regions.
26

---

Under review as a conference paper at ICLR 2020
A.8 EFFECT OF SPARSITY GRANULARITY AND RATIO
Pix2Pix: map to satellite domain translation.
Figure 28: Domain translation: filter pruning to different sparsity levels. Row 1: Output of the baseline generator. Rows 2-5: Synthesized satellite images by generators pruned to sparsities of 25%, 50%, 75%, 90%.
Figure 29: Domain translation: fine-grained pruning to different sparsity levels. Row 1: Output of the baseline generator. Rows 2-5: Synthesized satellite images by generators pruned to sparsities of 25%, 50%, 75%, 90%.
27

---

Under review as a conference paper at ICLR 2020
CycleGAN: photographic style applied to Monet's paintings.
Figure 30: Style transfer: filter pruning to different sparsity levels. Row I: Output of the baseline generator. Rows 2-5: Generated real photo style images by generators pruned to sparsities of 25%, 50%, 75%, 90%.
Figure 31: Style transfer: fine- -aningr pruning to different sparsities. Row 1: Output of the baseline generator. Rows 2-5: Generated real photo style images by generators pruned to sparsities of 25%, 50%, 75%, 90%.
28

---

Under review as a conference paper at ICLR 2020
CycleGAN: style of Monet's paintings applied to photographs.
Figure 32: Style transfer: filter pruning to different sparsity levels. Row I: Output of the baseline generator. Rows 2-5: Generated real photo style images by generators pruned to sparsities of 25%, 50%, 75%, 90%.
Figure 33: Style transfer: fine-graine grained pruning to different sparsities. Row 1: Output of the baseline generator. Rows 2-5: Generated real photo style images by generators pruned to sparsities of 25%. 50%, 75%, 90%.
29

---

Under review as a conference paper at ICLR 2020
CycleGAN: zebra to horse image-to- to-inaee translation.
Figure 34: Image-to-image translation: filter pruning different sparsities. Row 1: Bascline gener- ator output. Rows 2-5: Generated real photo style images by generators pruned to sparsities of 25%, 50%, 75%, 90%.
Figure 35: Image-to-ing tingge translation: fine- tne-grained pruning to different sparsities. Row 1: Baseline generator output. Rows 2-5: Generated real photo style images by generators pruned to sparsities of 25%, 50%. 75%, 90%.
30

---

Under review as a conference paper at ICLR 2020
CycleGAN: horse to zebra image-to- to-inage translation.
Figure 36: Image-to-image translation: filter pruning different sparsities. Row 1: Bascline gener- ator output. Rows 2-5: Generated real photo style images by generators pruned to sparsities of 25%, 50%, 75%, 90%.
Figure 37: image-to-image translation: fine rree-graned pruning to different sparsities. Row 1: Baseline generator output. Rows 2-5: Generated real photo style images by generators pruned to sparsities of 25%, 50%. 75%, 90%.
31

---

Under review as a conference paper at ICLR 2020
StarGAN: facial attribute image-to to--mage translation.
P
d1
P
d
P
OI
H11
P
o
P
o1
P PI
Ot
P
OI
P
fo
P
OI
P
on
P
d
P
O1
RO
P
01
P
P
o
P
P
P
P
P
P
Ot
OI
O1
O
P
P
P
P
P
OI
Figure 38: Image-to-image translation example 1: filter pruning to different sparsities. Row 1:
Baseline generator output. Rows 2-5: Facial attribute translated images by generators pruned to
sparsities of 25%, 50%, 75%, 90%.
P
H
P
P
P
H
P
OI
P
o1
P
fo
P
H
P
dor
P
Ho
P Ho
P
o
P el fo
P
P
dor
P
Hor
P on
OL
P
H
P
do
P
O
P
dor
P
Hor
P
or
P P
P
P
P
P
P OI
OL
OI
OL
OI
OI
Figure 39: tangetto-image translation example 1: fine-grained pruning to different sparsities. Row
Baseline generator output. Rows 2-5: Facial attribute translated images by generators pruned to sparsities of 25%, 50%, 75%, 90%.
32

---

Under review as a conference paper at ICLR 2020
Figure 40: Image-to-image translation example 2: filter pruning to different sparsities. Row 1: Baseline generator output. Rows 2-5: Facial attribute translated images by generators pruned to sparsities of 25%, 50%, 75%, 90%.
Figure 41: image-to-image translation example 2: fine- fine-grained pruning different sparsities. Row
1: Baseline generator output. Rows 2-5: Facial attribute translated images by generators pruned to sparsities of 25%, 50%, 75%, 90%.
33

---

Under review as a conference paper at ICLR 2020
25%
50%
75%
90%
Hine-Grained-Pee Fine- Grained- Pruned
Fine- Grained-
Sine-Grained-ao Grained-Pruned
Figure 42: Super resolution: fine- fine-grained pruning to different sparsities. Columns 1-4: Correspond- ing generated real high resolution images by generators pruned to sparsities of 25% 50%, 75%, 90%.
34

---

Under review as a conference paper at ICLR 2020
The loss curves for the comparative experiment in Figure 38 and 40
shown in Figure 4B
Srow of best aerterrn brithee
Generto verter Lesen Caring Statted Tocide
Concune Endflee teon Triern
8-
Figure 43: Loss curves of saaag-tto-image translation experiments of filter pruning to different spar- sities. (a)-(d): Corresponding loss curve of the generator pruned sparsities of 25%, 50%, 75%, 90%.
The loss curves for the comparative experiment in Figure 39 and 41
shown in Figure 47
and Dlel
Genentrant CirBle
Figure 44: Loss curves of image-to toose-tt-inng to-image translation experiments of fineggraine' pruning to differ- ent sparsities. (d): Corresponding loss curve of the generator pruned to sparsities of 25%, 50%, 75%, 90%.
35


Reviews:
[{"review_id": "Sklg5QfVFS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors tackle the task of compressing a network. While there\nare many effective solutions so far regular computer vision tasks, as they demonstrate,\nthey fail catastrophically  when applied to generative adversarial networks(GANs).\nThey propose a modification to the classic distillation method, where a\n\"student\" network tries to imitate the uncompressed one under the supervision of\na fully converged discriminator network. They perform evaluation on multiple\ntasks from image synthesis to super-resolution. They also study the influence\nof the compression factor on the quality of the generated images.\n\nThe task is well motivated and situated in the related literature. The first\nsection is very thorough and extremely efficient at describing the failure modes\nof existing methods. On one side, the results demonstrated in the evaluation are\ncompelling, on the other side, the compression factor is only 50%, which is much\nlower than seen in related work. However, as it is shown in section 3 the task\nmay be much harder for GANS than regular models so I still consider it a \nsizeable contribution.\n\nThere are a couple of points that require clarification. I personally found the\ndescription of the method (Section 4) rather confusing. It is clear\nwhat \"discriminative loss\" is as it is the one used in every GAN.\nUnfortunately, I could not understand what \"generative loss\" means in the general\ncase. An example is given for StarGAN in equation (7) and I have a rough idea of\nwhat to choose for Style Transfer, Domain Translation, Super Resolution and\nImage translation. Though, it is unclear to me what to use in the case of\nimage synthesis. The experiments clearly show that it is possible so I think\nit is necessary to show how this framework is concretely applied to each task at\nhand.\n\nDuring training, the discriminator only ever saw pictures from the true distribution\nand the distribution generated by the generator (at each of its training steps).\nIf I understood the framework properly, here, the compressed generator is trained\nfrom a random initialization. The distribution it outputs is therefore\ncompletely unknown and potentially non overlapping with either of the true or\nthe generator ones. In that case it is hard to predict what the discriminator\nwould do on completely out of distribution samples. I seems reasonable to\nconjecture that it might consider them \"true\" because it was never trained on\nthem. Could you provide an explanation of why it is not a problem in practice?\nDo you have to try multiple initializations? Is the generative\nloss enough to force the compressed discriminator to match the support\nof the distribution of the dense generator?\n\nI think this paper is novel, tackles a hard task and presents compelling results\n(albeit using very mild compression ratios). It should be accepted if some\nclarifications are made in section 3.\n\nMinor Remarks:\n\n- Figures 2, 9, 42 and 43 are unreadable when printed with a regular color office\n  printer.\n- It is unclear what it would take an extra 10% of the original number of epochs to train the compressed network. Why couldn't it be faster, or much longer?\n\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "SJx-uwtJjH", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The problem tackled in the paper is the compression of generators in adversarially trained models. Considering the success of large generative models (like BigGAN) one may wonder if these models can be compressed after being trained to improve practical applicability. \n\nThis paper is focused on the compression of image to image translation models and uses distillation on discriminator's outputs to achieve better results.\n\nMy decision is weak reject.\n\nThe message of the article is misleading. The whole goal of pruning a neural network is to remove filters from it, therefore, reducing the computation or, at least, storage space for the parameters. The attempt to remove filters was presented in the last figure, and it does not work as good as all other results presented in the paper.\n\nThe comparison in Figure 1 is arguably misleading as well. For example, one of the methods that were mentioned (LIT) does achieve a factor of 1.8 model compression, yet the comparison was not carried out directly with that method, but a modification proposed by the authors of this paper.\n\nI would like to see more comparisons in terms of FLOPs or inference time between the baselines, SotA methods, and your proposed method. Weights pruning is simply one of the approaches for model compression, so you cannot ignore the alternatives.\n\nAlso, section 4 probably has to be rewritten, since some unorthodox notation is used. The authors should consider using some reference paper for the notations, like CycleGAN, that was mentioned in the paper. That will improve the clarity and readability of the used objectives.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HkxGS3W9FH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a method to compress GANs. The motivation is that the current compression methods work for other kinds of neural networks (classification, detection), but perform poorly in the GAN scenario. The authors present intuitive reasons for why this is the case. \n\nHowever, the motivation why we would like to compress GANs is unclear to us. The intro mentions: reducing memory requirements and improving their performance. Sure, compressing networks for object detection and classification on mobile devices is really useful. But GANs are mainly used for unsupervised density estimation, why put a GAN generator on a mobile device? But maybe we are missing something here. \n\nTheir “self-supervised” method works by using the pre-trained discriminator network, while compressing only the generator. They show both qualitative and quantitative gains.\n\nThe paper is clear and well-written. It presents a way of pruning GAN generator network and although of limited novelty, it might be an interesting read as it provides extensive and convincing experiments in a clear manner. It does have several parts though which require additional clarification.\n\nThe idea of using the pre-trained discriminator network seems reasonable, but I am missing what the compression method for the generator network actually is (Section 4). From Table 2 I would assume it is pruning, in which case the paper’s contribution is very limited.\n\nThe authors claim that the “self-supervised” method generalizes well to new tasks and models. \"Generalizes\" seems a strong word here, since the procedure compresses only the generator network. A more appropriate way of putting it might be ‘can be applied to other tasks and models.'\n\nIn Section 4 the authors write: “Our main insight is found,” but then they describe the GAN method. What is the actual insight there?\n\nThe qualitative results in Figure 1 suggest that their “self-supervised” method is better than the other baselines. \n\nScores from Table 2 also support the claims, but the table itself is not referenced anywhere in the text.\n\nThe analysis in Section 6 seems out of context with the rest of the paper. It is not clear how it relates to the “self-supervised” method.\n\nMissing related work: 1st paragraph: compressing or distilling one network into another is much older than 2015, dating back to 1991 - see references in section 2 of the overview http://people.idsia.ch/~juergen/deep-learning-miraculous-year-1990-1991.html \nThe GAN principle itself is also much older (1990) - see references in section 5 of the link above.  \n\nGeneral remarks:\n\nIn the first read of Section 3 it is not clear what [a], [b], [c] are.\n\nIt would be good to first refer to Table 1.\n\nTable 1: why is there a “?” only on the “Fixed” column?\n\nIt would be good to have a larger font size in Figure 2, at least the size of the main text font.\n\nIn its current form, the pdf file has 100MBs (8MBs the main paper and the rest is the appendix). One could instead move the images from the appendix to a website and provide a link.\n\nWe might improve our rating provided the comments above were addressed in a satisfactory way in the rebuttal.\n\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper develops a new method for pruning generators of GANs. It has received a mixed set of reviews. Basically, the reviewers agree that the problem is interesting and appreciate that the authors have tried some baseline approaches and verified/demonstrated that they do not work. 

Where the reviewers diverge is on whether the authors have been successful with the new method. In the opinion of the first reviewer, there is little value in achieving low levels (e.g. 50%) of fine-grained sparsity, while the authors have not managed to achieve good performance with filter-level sparsity (as evidenced by Figure 7, Table 3 as well as figures in the appendices). The authors admit that the sparsity levels achieved with their approach cannot be turned into speed improvement without future work.

Furthermore, as pointed out by the first reviewer, the comparison with prior art, in particular with LIT method, which has been reported to successfully compress the same GAN, is missing and the results of LIT have been misrepresented. While the authors argue that their pruning is an "orthogonal technique", and can be applied on top of LIT, this is not verified in any way. In practice, combination of different compression techniques is known to be non-trivial, since they aim to explain the same types of redundancies.

Overall, while this paper comes close, the problems highlighted by the first reviewer have not been resolved convincingly enough for acceptance.
Decision: Reject