Information-Theoretic GAN Compression with Variational Energy-based Model
Authors: Minsoo Kang
Hyewon Yoo
Eunhee Kang
Sehwan Ki
Hyong-Euk Lee
Bohyung Han
Venue: NeurIPS 2022 Accept (2022)
Keywords: GAN Compression
Knowledge Distillation
Mutual Information

Abstract: We propose an information-theoretic knowledge distillation approach for the compression of generative adversarial networks, which aims to maximize the mutual information between teacher and student networks via a variational optimization based on an energy-based model. Because the direct computation of the mutual information in continuous domains is intractable, our approach alternatively optimizes the student network by maximizing the variational lower bound of the mutual information. To achieve a tight lower bound, we introduce an energy-based model relying on a deep neural network to represent a flexible variational distribution that deals with high-dimensional images and consider spatial dependencies between pixels, effectively. Since the proposed method is a generic optimization algorithm, it can be conveniently incorporated into arbitrary generative adversarial networks and even dense prediction networks, e.g., image enhancement models. We demonstrate that the proposed algorithm achieves outstanding performance in model compression of generative adversarial networks consistently when combined with several existing models. 

Information-Theorete GAN Compression with Variational Energy-based Model
Minsoo Kang'
Hyewon Yoo?
Eunhee Kang²
Sehwan Ki
Hyong-Euk Lee²
Bohyung Han
'ECE & 'IPAL, Scoul National University
Samsung Advanced Institute of Technology (SAIT)
{kminsoo, , yoohyewony, bhhan) ac kr
(eunhee. kang, sh1004. ki, hyongeuk. leee @saasung..ccommm com
Abstract
We propose information -ieemration-hooretiic knowledge distillation approach for the com- pression of generative adversarial networks, which aims to maximize the mutual information between teacher and student networks via variational optimization based on an energy-based model. Because the direct computation of the mutual information in continuous domains is intractable, timizes the student network by maximizing the variational lower bound of the mutual information. To achieve tight lower bound, we introduce an energy-based model relying on deep neural network to represent flexible variational distribu- tion that deals with high-dimensional images and consider spatial dependencies between pixels, effectively. Since the proposed method is generic optimization algorithm, it can be conveniently incorporated into arbitrary generative adversarial networks and even dense prediction networks, e.g., image enhancement models. We demonstrate that the proposed algorithm achieves outstanding performance model compression of generative adversarial networks consistently when combined with several existing models.
approach alternatively op-
Introduction
Generative adversarial networks (GANs) ① have accomplished impressive achievements various computer vision tasks including image synthesis (2/5). image-to-image translation [6]]], video pre- diction [10 and many others. However, despite the outstanding performance, their applicability to resource-humee hungryy systems, e.g., edge or mobile devices, is limited due to high computational costs. To deal with this issue, GAN compression, task reducing the size of generator networks, has recently drawn the attention of many machine learning researchers.
The straightforward way to compress generators directly employ traditional network compression techniques such as low-rank approximation [33155 network quantization [1619] pruning [20]- 25]. and knowledge distillation [26 However, the compression of generative models typically involves a higher-dimensionall output space and requires matching the output distributions between the original and compressed models instead the output instances given by the two models. Hence, as reported in [31 the synthesized images obtained from the compressed generators via the images with artifacts and suffering from image quality degradation, which implies that a naïve application of the methods designed for effective for GAN compressions.
channel pruning 34 23 24 are often cetoptimal-produceee
discriminative models with low dimensional output spaces
often utilize knowledge distillation and manage achieve competitive accuracy of student models. Specifically, these methods transfer the knowledge of a teacher network to a student model via intermediate representations,
Existing approaches for compressing GANs B1 35 32
36
38
33
39
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

final outputs, and embedded features 40 41] To this end, the existing algorithms employ & or la norms to compute the difference between the representations of the two networks, but they are not necessarily effective for minimizing the distance between the two distributions.
We propose compress GAN models by maximizing the mutual information for knowledge distilla- tion between the outputs of a teacher and a student. For the compression of generative models, the use of the mutual information between two random variables makes more sense than the direct pixel-wise comparison between two outputs, which is the standard way to transfer knowledge from a teacher to student in discriminative models. However, because the computation of the mutual information is generally intractable in the continuous domain, we employ a variational approach [42], where the lower bound of the mutual information is alternatively optimized using a variational distribution, an approximation for the conditional distribution of the teacher output given the student output. We further adopt an energy-based based model for the flexible representation the variational distribution and tighten the lower bound for effective knowledge transfer. Note that the proposed energy-based model a convenient way to deal with high-dimensional data e-g., images, and an effective tool to handle spatial dependency between pixels within an image. Combined with the existing approaches, our framework achieves substantial performance improvement on GAN compression tasks. The main contributions of our work are summarized below:
We propose a novel energy-based framework of GAN compression for knowledge distilla- tion, which maximizes the mutual information between teacher and student.
The proposed energy-based approach is effective for representing a flexible variational distribution and can be incorporated into many GAN compression methods thanks to its generality.
Experimental results verify that the proposed method improves performance consistently when combined with the existing methods.
The rest of the paper is organized as follows. Section 2 discusses the related work to GAN compres- sion. The details of our approach are described in Section and 4 and the experimental results are presented in Section Finally, we conclude this paper in Section 6
2 Related Work
2.1 Generative Adversarial Networks (GANs)
Generative Adversarial Networks (GANs) N consist of two components, a generator and a discrim- inator, and train the models using an adversarial loss; the generator aims to fool the discriminator by synthesizing realistic images from random noise samples while the discriminator attempts to distinguish the real data from the fake ones. Unlike the vanilla GANs, conditional GANs [3]
control the image generation conditioned on additional imputs such as class label 43 or an image to be translated into another image 6 7 (e.g., Horse + Zebra). The conditional GANs are not confined to the conditional image generation tasks, but applied to various areas such as image super- resolution 44. eext-to-image synthesis (45). and image inpainting [46]. Since the computational costs of the conditional GANs are typically high in terms of FLOPs, the proposed GAN compression algorithm is mainly tested on those networks but we also verify the effectiveness of our approach on the unconditional case.
2.2 Knowledge Distillation
Knowledge distillation techniques aim to transfer knowledge from a powerful network, i.e., teacher,
to compact network, i.e., student, by matching logits [47]. output distributions (26). intermediate ac- tivations [48] or attention maps [27]. Specifically, Hinton et al. (6) learn the student representations by minimizing the relative entropy from its normalized softmax outputs to the teacher's 's while [47 48
minimize the mean squared error between the representations of the teacher and student networks. Both Variational Information Distillation (VID)
49 and Contrastive Representation Distillation (CRD) 500) employ mutual information between teacher and student to estimate the similarity of the representations from the two networks. The former relies on a variational approach [42 for the tractable computation of mutual information between the two networks. On the other hand, the latter derives an InfoNCE-lika like loss function from the lower bound of the mutual information to learn the

---

representations of student. However, VID maximizes the variational lower bound using strong assumption about the variational distribution, a fully factorized Gaussian distribution, and CRD difficult to be employed in generative models due to its contrastive learning objective, which is prone to be trivial in image generation tasks without semantic attribute annotations, e-g- class labels. Our algorithm is related to VII) and CRD since they all attempt to maximize the mutual information between the teacher and student networks, but has clear advantage over them in terms of the flexibility of the variational distribution and the applicability to generative models.
2.3 GAN Compression
GAN compression algorithms typically aim reduce the size of the generator without considering the discriminator since it is not used for inference. Co-Evolution [31] utilizes an evolutionary algo-
rithm [51] reduce the channel size in the generator of CycleGAN
based on a fitmess formulated
based on FLOPs and the mean squared reconstruction error between the original uncompressed and pruned generators. GAN-Compression (35) employs once-for-all network training scheme [52]
combined with the reconstruction loss as well as the intermediate feature distillation loss 488] and
selects the best performing subnetwork given a target budget. Compression and Teaching (CAT)
designs an efficient building block for generator, and prunes the network using magnitude- based channel pruning method [24] while maximizing the similarity between the two networks based on
centered kernel alignment
133 Content- GAN Compression (CAGC) R3 uses the feature
distillation and reconstruction loss based only on the object parts of a generated image while ad- ditionally considering LPIPS 41 as perceptual distance metric for synthesized images. On the contrary, Online Granilrtaallrtty Distillation (OMGD) [39] iteratively trains teacher networks with original conditional GAN objectives while optimizing student network with the total variation loss (54) for spatial smoothness and the distillation loss for making the student mimic the final outputs and the intermediate feature maps of the teachers. Generator-discriminn. Compression (GCC) [55] also adopts the online distillation scheme while selecting channels in the original discriminator to match the reduced capacity of the student generator at each iteration. To validate the effectiveness of our framework, we present how the proposed algorithm can improve performance via the combinations with existing methods such GCC, OMGD, CAGC, CAT, and GAN-Compression.
Cooperative
3 Variational GAN Compression with Energy-based Model
Our goal is to compress generators in GANs through knowledge distillation. Let S and T be random variables of the outputs from a student, G (:::))) ) : X' - y and teacher, G' G'(:0") : X + V,
respectively, where 6 and of are their model parameters. For knowledge transfer from the teacher to the student, we maximize the mutual information between T and S, (T;; S), which is given by
IT; DRI. (p(t, (lp(() p(5)) = H(T) + Ee,g
where (-I-) denotes the Kullback-Leilerr (KL) divergence and H (-) means the entropy function. Note that all the expectations are taken over the joint data distribution, p(t, 8), where t and indicate realizations of the teacher and student output distributions, respectively.
A practical challenge in this objective is that the computation of the mutual information for multivari- ate continuous random variables is typically intractable. To tackle this issue, we adopt a variational method [22, where we derive the lower bound of the mutual information, İ(T: using a variational distribution, q(t)s), which is given by
I(T: 5) = HTT) + E,, (log (t[ss) + E. (Pooooo((())))))l(ttss))
2 (T) + F4,, [l,g (())s)]
: S):
(2)
To maximize the mutual information, we increase the lower bound by minimizing the KL-divergence between the true and variational distributions, which is defined
AT(T; S) = S(T;5)) - i(T:S) = (p(t)s))l((
Variational Information Distillation (VID) 49 optimizes the variational lower bound by employing the fully factorized Gaussian distribution the variational distribution (()ss). However, as noted
3

---

in [56], the statistics of the natural images usually exhibit non-Gaussian properties. Furthermore, optimizing the lower bound using the fully factorized distribution encourages the student generate To the contrary, we employ an energy-based model as flexible distribution for the variational distribution, which is given by
blurry outputs [57 46 58
1
(o
exp
Eo 8t.
(4)
(z(8)
where Eg 3) Rs the energy function defined by deep neural network parametrized with 0 and (s) is the partition function. The variational distribution given by the energy-based model is effective for the student model to maximize the variational lower bound. In addition, differentiated from VID [49]. our approach employs the variational distribution that considers the
spatial dependency among pixels in and s.
Overall, we optimize the energy- based model, the student network, and the teacher network, which is achieved by the following iterative procedure:
1. Fix {Ø² x²) and optimize 0.
Under the fixed parameters ( minimize the objective in (3) with respect to B.
2. Fix {0, Φ²} and optimize o²
Under the fixed parameters {0, 6'), maximize the objective in while minimizing the standard knowledge distillation losses with respect to 6²
3. Optimize d'
Minimize the standard conditional GAN loss 6 with respect d' using training dataset. This step is only required for online knowledge distillation. Otherwise, the teacher model should be learned offline in advance.
These three steps are repeated until convergence.
Training Procedures
This section describes the optimization procedure of our approach for knowledge distillation, which iteratively maximizes the mutual information between teacher and a student using the energy-based model.
4.1
Optimization of Energy-based Model
The first step of the iterative procedure is the optimization of the energy-based model for minimizing E, [Ki (p(t) ()] (t[s))] with respect to 0 given the fixed model parameters {Ø", ot)
4.1.1 Objective
Since the conditional data distribution p(t(s) is constant with respect to the model parameter 0, the energy-based model minimizes the following objective:
min AI(T; -
min B..l log qo (t|s)).
(5)
To optimize the energy-based model using the standard stochastic gradient descent method, we derive the gradient of the objective in (5), which given by
V F | log go [t|s)] Ex, Eo s)] - E, (e
Eo, 8)):
(6)
where the second expectation of the right- hand side requires sampling from go (t(s)) via Markov Chain Monte Carlo (MCMC) method, for example, Gibbs sampling.
4.1.2 Improved MCMC sampling
The main drawback of the MCMC methods is the high computational cost, so we rely on the Langevin dynamics, which leads to the following recursive sample generation:
X
--
VeE
2

---

Teacher Encoder
Teacher Decoder
Teacher Output
Minimize
Maximize Concatenate
Knowledge Distillation
Input
Student Encoder
Student Decoder
Student Output
Energy-based Model
Run MCMC
t
Run MCMC
MCMC
Figure 1: Overview of our knowledge distillation framework for GAN compression. maximizes the variational lower bound of mutual information via an energy-based model.
where w is a sample drawn from (0,, A7) and À is fixed step size. Theoretically, d becomes identical to sample drawn from qe (t(s) as + oo and + [59]. Following Short-Run MCMC [60), also run K- Stetep MCMC starting from an initial distribution, go. to approximate samples from qo (t(s), where K is the number of MCMC steps.
4.1.3 Initial distribution
We initialize e as the output of the student network for the acceleration of the MCMC optimization via the Langevin sampling process. This initialization is good approximation to the model distribution; intuitively, since the student model is supposed to mimic the samples drawn from the data distribution, the optimization of qo (t|s) towards the data distribution p(t[s) from the current student output would be more efficient. As result, the proposed initialization strategy often results in improved performance of student models compared to the existing methods [6] 60 62 65) even with a small number of MCMC steps. On the contrary, we can naturally adopt the data distributions of teacher networks, which are static in Contrastive Divergence (CD) (62) and dynamic in Persistent Contrastive Divergence (PCD) [63] (64, for setting ạo while Short-Rur MCMC (60] can initialize the variational distribution using the uniform noise distribution. Since the data distribution is prone to induce bias in the CD estimate [65] and the random uniform distribution is far from the data distribution, the existing methods require many steps of MCMC and consequently increase training time [60]. In addition, PCD updates all samples in the memory buffer, which leads to significantly increased memory overhead, especially in generation tasks.
4.1.4 Inference
we do not require any MCMC sampling for inference but generate images via a single forward process the student generator. Therefore, the proposed algorithm incurs extra computational cost at inference time.
In contrast to the previous approaches for energy -ased models [60 65 64
4.2
Optimization of Student Generator
4.2.1 Loss for mutual information
The student generator learns to maximize the variational lower bound Ī(T: S) in with respect to 6e. where the objective function is formally given by
max Í(T; S) > max HTT) + Er,, log (t(ss))
max Er, [log qe (t|s)).
(8)
Note that the entropy, HT)), is independent of the parameters in the student network and the objective equivalent to maximize the expected log likelihood as shown the above equation. Since the joint distribution p(t, s) depends Φs, employ reparameterization trick using function G Then, the gradient of the objective function in (8) with respect to &,, derived as follows:
V oe Er,, ! lloo 4o (t|s)]
Ex
(',, V Vo-Eo Eo (G* G) (2; (x)6")) Ex Ex Eo (e]G² (ano)))) Eott, (9)
5

---

Table 1: Performance of VEM in comparison with the sate-of-the-ane compression methods for the Pix2Pix model using the U-Net baseline on the Edges - Shoes and Cityscapes datasets. Methods with asterisk (*) denote reproductions given by running the official codes.
Dataset
Method
MACs
of parameters
FID (J)
mloU (t)
Original
6
18.60G (1.0x)
54.40M (1.0x)
34.31
DMAD
37
2.99G (6.2x)
2.13M (25.5x)
46.95
Edges Shoes
OMGD
39
1.22G (15.3x)
3.40M (16.0x)
27.39
OMGD*
39
+ VID
1.22G (15.3x)
3.40M (16.0x)
28.02
OMGD*
39
VEM (Ours)
1,22G (15.3x)
3.40M (16.0x)
24.59
Original
18.60G (1.0x)
54,40M (1.x)
42.71
DMAD
3,96G (4.7x)
1.73M (31.4))
40.53
GCC [55
3,09G (6.0x)
42,88
Cityscapes
OMGD* 39
1,22G (15.3x)
3.40M (16.0x)
47.52
OMGD*
33
VID 49
1.22G (15.3x)
3.40M (16.0x)
47.79
OMGD*
39
VEM (Ours)
1.22G (15.3x)
3.40M (16.0x)
49.39
Note that we do not need to perform MCMC additionally to obtain the samples from go (t (x)) because they are already available from the training procedure for the energy-based model. Figure illustrates the overview our framework for maximizing the mutual information.
4.2.2 Algorithm-specific loss
The proposed variational lower bound maximization framework using the energy-based model is incorporated into model compression algorithms based knowledge distillation such OMGD (39), GCC [55], GAN-Compression [35], CAT [38] and CAGC 33 The brief descriptions about these
methods are presented Section 2.3 and discuss the details of the segoortth--spciifee losses in the supplementary document.
4.2.3 Combined loss
The full objective function for the student network is given by
L Laloo ATT
(10)
where Calgo is the cegorithh-sppccifee loss and Ī(T; S) is the variational lower bound defined in ( while AMr is byperparameter balancing the two terms.
4.3 Optimization of Teacher Generator
The teacher generator is trained by the standard method of the corresponding GAN algorithm, e.g., Pix2Pix [6] and CycleGAN [7]. Optionally, the teacher model learns to minimize the ( distance between the generated image and the ground-truth when the ground-truth is available. In the of the unsupervised image-to-iman
translation tasks, the cycle consistency loss 7 based on the & distance may enforce the instance mapping between two domains. To sum up, the teacher network is optimized by the adversarial loss, denoted by LGAN, and the reconstruction loss, Liecs as follows:
min max LGAN + Areehres G D
(11)
where Ares is hyperparameter to balance the two loss terms. Note that, for unconditional GANs, there is nothing reconstruct and we train the network based only the GAN loss.
5 Experiments
To validate the effectiveness of the proposed compression algorithm based on the variational energy- based model, referred to as VEM, we conduct extensive experiments on the standard datasets using various architectures. Since VEM is model-agnostic algorithm component, we incorporate the module into existing GAN compression approaches and the combined models are expressed in the form of "[Algorihhm Name] + VEM" throughout this section.

---

Table 2: Performance comparisons between VEM and the state-of-the-an compression methods in CycleGAN model with the ResNet backbones.
Dataset
Method
MACs
# of parameters
FID (4)
Original
F
56.80G (1.0x)
11.30M (1.0x)
61.53
Co- Co-Evolution
31
13,40G (4.2x)
96.15
1133G (23.6x)
86.09
CAoo-GAN-Distillee
32
6.39G (8.9x)
83.60
GAN-Compression
35
2.67G (21.3x)
0.34M (33.2x)
64,95
Horse Zebra
DMAD [37
2,41G (23.6x)
0.20M (40.0x)
62.96
CAT [38
2.55G (22.3x)
60.18
GCC [55
2,40G (23.7x)
59.31
OMGD
[39
1.41G (40.3x)
0.14M (82.5x)
57.14
OMGD*
VEM (Ours)
1.4GG (40.3x)
0.14M (82.5x)
50,83
Original
7
56.80G (L.x)
11.30M (L.x)
79.12
Co-Evolution
31
11,10G (5.1x)
78.58
Auto-GAN- Auto-GAN-Distilleer
32
4.34G (13.1x)
78.33
Summer Winter
DMAD
37
3.18G (17.9x)
0.30M (37.7x)
78.24
OMGD*
39
1,41G (40.3x)
0.14M (82.5x)
75.20
OMGD*
[39
VEM (Ours)
1,41G (40.3x)
0.14M (82.5x)
74.04
Table 3: Performance comparisons between VEM and the existing compression algorithms for unconditional GANs using SAGAN and StyleGAN2 on CelebA and FFHQ, respectively.
Model
Dataset
Method
MACs
Compression rate
FID (J)
Original
23.45M
24.87
Slimming
24
15.45M
34.12% 34.12%
36.60
SAGAN
CelebA
GCC*
55
15.45M
27.91
GCC
VEM (Ours)
15.45M
34.12%
25.27
Original
45.1B
4.50
GAN-Slimming
36
5.0B
88.91%
12.40
StyleGAN2 66
FFHQ
CAGC
4.1B
90.91%
7.90
CAGC 33
VEM (Ours)
4.1B
90.91%
7.48
5.1 Datasets and Models
Image-to-image translation We adopt Pix2Pix (6] on the Edges - Shoes 667 and Cityscapes [68 datasets while taking CycleGAN 77 on Horse > Zebra [7] and Summer - Winter [7]. Note that Pix2Pix is the image-to-image translation network for paired data while CycleGAN is mainly proposed for unpaired image-to-image translation. Following the original models [6] 7). we employ
generators based U-Net [69] and ResNet [70] for Pix2Pix and CycleGAN, respectively. For all the datasets, images are resized to 256 x 256 before feeding them into the models.
Image generation To evaluate the performance of our approach for unconditional GANs, we experiments with StyleGAN2 [66] Flickr-Faces HQ (FFHQ) (71) and Self-Attention GAN (SAGAN) [4 on CelebA (72). As preprocessing, we resize input images to 64 x 64 for CelebA and 256 256 for FFHQ
5.2 Evaluation Metrics
Following the protocol of the previous approaches, we report the Fréchet Inception Distance (FID) 733 scores on Edges + Shoes, Horse - Zebra, Summer->Wineer, CelebA, and FFHQ while the mean Intersection over Union (mloU) is selected for Cityscapes. The FID score is widely used for assessing the quality of the generated images by measuring the distance between the dis- tributions of embedding features extracted from generated and real images, which are given by Inception V3 model 774 pretrained on ImageNet. In the case of the Cityscapes dataset, we adopt DRN-D-10S 75 as model perform semantic segmentation on the translated images and derive mloU scores for the evaluation of image-to-image translation networks.

---

5.3 Implementation Details
The proposed algorithm is implemented in PyTorch 766 based on the publicly available code" Additionally, we incorporate the proposed mutual information maximization framework into existing approaches by using the official codes of GoN-Compressior 355. CAT 38). CAGC
33 and
GCC (55) validate its benefit and generality. For fair comparisons, we do modify neither experimental settings nor network structures of the compared algorithms. Specifically, we set the batch size to 4 for Pix2Pix, for CycleGAN, 64 for SAGAN, and for StyleGAN2 while we adopt
the Adam optimizer with an initial learning rate of 0.0002, which decays to zero linearly.
When we train our energy-based model, we employ the Adam optimizer with a learning rate of 0.0001. Note that we only run 10 steps of Langevin dynamics to draw samples from the energy-basee model to save training time. We set the standard deviation of random noise to 0,005 and use step size of 100 for each gradient step of Langevin dynamics for all datasets except for Cityscapes and FFHQ. For these two datasets, we use a step size of 50 instead. Details about the hyperparameter settings are provided in the supplementary document. In the case of paired datasets such as Edges -> Shoes and Cityscapes, we maximize the mutual information between the outputs given by the student model and the real outputs instead of the teacher outputs.
5.4 Results
Pix2Pix Table presents the results from the Pix2Pix models Edges Shoes and Cityscapes, where the proposed algorithm, denoted by OMGD + VEM, is compared with existing approaches including the vanila OMGD (39]. As shown in the table, OMGD + VEM achieves the best perfor- mance in terms of FID and MAC among all compared algorithms on Edges - Shoes. On Cityscapes, OMGD VEM presents the best mloU by large margins even with the highest compression rate. Note that we report the results from OMGD based on our reproductions using the official code, which are denoted by asterisks (*) in all tables.
We test our compression algorithm for CycleGAN on Horse Zebra and Summer Winter. As shown in Table OMGD + VEM outperforms all the compared methods in both datasets, especially by large margins Horse + Zebra.
CycleGAN
Unconditional GANS We compress SAGAN and StyleGAN2 using GCC 555 and CAGC 33 respectively, Table 3 illustrates that the proposed method is also effective to improve performance of the unconditional GAN compression techniques. In particular, our approach also works well with large-scale generator such as StyleGAN2.
5.5 Ablation Study about Mutual Information Maximization Techniques
To maximize the mutual information between the teacher and student networks, one can use fully-
factorized Gaussian distribution for the variational distribution as VID [49] or employ a contrastive loss similar with infoNCE [77 following the strategy in CRD [50] To validate the effectiveness of
our energy-based models, VEM, in comparison with VID and CRD, we perform an ablation study, where the three add-ons are tested through their integrations into the baselines such as OMGD (3)]
GAN-Compression (35), CAT 38). GCC [55, and CAGC [3)
Table demonstrates that VEM consistently outperforms VID and CRD in all the tested GAN compression algorithms. This is because, compared VIID, we derive a tighter lower bound of the mutual information theoretically and consider spatial dependencies among pixels thanks to the more flexible variational distribution based on the energy function. On the other hand, the application of CRD to generative models may not be beneficial since instance-level contrastive losses are less effective due to large gaps between positive and negative pairs.
http:://ggihabbb.oo/yyyrdaa////
http:///gthhblcccaaatttlllllaaa
http:://giitab..com//s//////
8

---

Table 4: Ablation studies about the approaches to maximizing mutual information. We compare the methods relying on the proposed energy-based model (VEM) with the approaches based on fully factorized Gaussian distribution (VID) and InfoNCE-like contrastive loss (CRD). This experiment is conducted on the Horse Zebra, Summer + Winter, CelebA, and FFHQ datasets.
Model
Dataset
Method
MACs
FID
OMGD*
39
1,41G
57.14
OMGD
39
VID
49
1,41G
65.73
OMGD*
39
CRD
1.4IG
70.59
OMGD*
VEM (Ours)
1.41G
50.83
CAT
38
2.56G
64.79
Horse Zebra
CAT
38
VID
40
2.56G
68.71
CAT
CRD
50
2,56G
67.69
CycleGAN 7
CAT
VEM (Ours)
2.56G
52.44
GAN- DAN-Compression* "
65
2.55G
59.13
GAN Compression * 35
49
2,55G
62.36
GAN-C -oNprrrsressiin * 35
CRD
2,55G
60.27
GAN- MAN-Compresssion
35
VEM (Ours)
2,55G
50.01
OMGD*
39
1.41G
75.20
OMGD
39
49
1.41G
74.39
Summer + Winter
OMGD*
39
+ CRD
1.41G
74.54
OMGD 39
VEM (Ours)
1,4IG
74.04
GCC* [55
15,45M
27.91
GCC (551 + VID
15,45M
26.57
SAGAN 4
CelebA
GCC* [5
CRD 50
15.45M
30.93
GCC [55]
VEM (Ours)
15,45M
25.27
CAGC 33
4.1B
7.90
CAGC 33
+ VID
49
4.1B
7.51
StyleGAN2 66
FFHQ
CAGC 33
CRD
50
4.1B
7.65
CAGC
33
VEM (Ours)
4.1B
7.48
Table 5: Sensitivity analysis for Ami the Cityscapes dataset using Pix2Pix when VEM is combined with OMGD.
Dataset
Method
AMI
MACs
# of parameters
mloU (1)
OMGD* 39
1,22G
(15.3x)
3.40M (16.0x)
47.52
OMGD
39
+ VEM (Ours)
0.05
1.22G (15.3x)
3.40M (16.0x)
47.56
Cityscapes
OMGD*
39
VEM (Ours)
0.10
1.22G (15.3x)
3.40M (16.0x)
48.90
OMGD
39
VEM (Ours)
0.20
1.22G (15.3x)
3.40M (16.0x)
49.39
5.6 Effect of AM for Mutual Information
We analyze the effect of the balancing parameter λMı using the Pix2Pix algorithm, which is tested on the Cityscapes dataset, when VEM combined with OMGD [39]. As shown Table VEM not sensitive to the hyperparameter and improves the performance of OMGD.
5.7 Qualitative Results
Figure visualizes generated images given by the original uncompressed backbones of Pix2Pix and CycleGAN, where we also present the images synthesized by the compressed counterparts using OMGD and OMGD VEM. Pix2Pix is tested on the Edges Shoes and Cityscapes datasets, and CycleGAN is evaluated on the Horse - Zebra and Summer - Winter datasets. The results imply that our approach effective to enhance generation quality. For example, OMGD often produces unusual artifacts in the background with the Cityscapes dataset, which are significantly reduced in OMGD + VEM. 3 generation results given by unconditional GANs on CelebA and FFHQ, Compared with the uncompressed versions, the proposed method preserves often improves image quality even with high compression rates. Also, our approach synthesizes images with better visual quality when combined with the previous GAN compression methods such as GCC and CAGC.

---

Horse Zebra
Edges Shocs
ri
Citsscapes
Summer Winter
Input
Original
OMGD
+VEM
Inpur
Ground-truth
Original
OMGD
+VEM
CycleGAN
Pix2Pix
Figure 2: Qualitative results of CycleGAN on Horse + Zebra and Summer + Winter (left) and Pix2Pix Edges - Shoes and Cityscapes (right). Note that "Original" represents the images generated by the uncompressed backbones.
Real
Original
Images
CAGC
GCC
CAGC
GCC
VEM
VEM
Figure 3: Qualitative results of StyleGAN2 using on the FFHQ dataset (left) and SAGAN on the CelebA dataset (right). Note that "Real Images" denotes data samples in CelebA while "Original" represents the images generated by the uncompressed StyleGAN2.
6 Conclusion
We presented novel GAN compression method via knowledge distillation based mutual informa- tion, which maximizes the lower bound of the mutual information by incorporating an energy-based variational distribution. Our mergy-based model is straightforward to be incorporated into various existing model compression algorithms for generative models. provides a more flexible variational distribution, which leads to theoretically tighter lower bounds and facilitates the consideration of the dependency between pixels. The proposed algorithm demonstrates outstanding performance in various GAN compression scenarios multiple datasets compared to the sfate-of-ther approaches.
Acknowledgments This work was partly supported by SAIT, Samsung Electronics Co., Ltd, the Bio Medical Technology Development Program of the National Research Foundation (NRF) funded by the Korea government (MSIT) [No. 2021 M3A944080772) and Institute of Information & communications Technology Planning Evaluation (ITTP) grant funded by the Korea government (MSIT) [No. 2021-0-01343, No. 2022-0-00959).
10

---

References
[1] Goodfellow, I., Pouget-Abadic, J., Mirza, M., Xu, B.. Warde-Farley, D., Ozair, S., Courville, A.,
Bengio, Y: Generative Adversarial Nets. In NIPS. (2014)
[2] Arjovsky, M., Chintala, S., Bottou, L.: Wasserstein Generative Adversarial Networks. In ICML. (2017)
[3] Miyato, T., Kataoka, T., Koyama, M., Yoshida, Y.: Spectral Normalization for Generative
Adversarial Networks. In ICLR. (2018)
[4] Zhang, H., Goodfellow, I,, Metaxas, D., Odena, A.: Self-Attention Generative Adversarial Networks. In ICML. (2019)
[5] Brock, A., Donahue, J., Simonyan, K.: Large Scale GAN Training for High Fidelity Natural Image Synthesís. In ICLR. (2019)
[6] Isola, Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-Image Translation with Conditional Adversarial Networks. In CVPR. (2017)
[7] Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired Image-to-Image Translation using Cycle- Consistent Adversarial Networks. In ICCV. (2017)
[8] Zhang, H., Xu, T., Li, H., Zhang, S., Wang, X., Huang. X., Metaxas, D.N.: StackGan++: Realistic Image Synthesis with Stacked Generative Adversarial Networks. TPAMI (2018)
[9] Park. T., Liu, M.Y., Wang. T.C., Zhu, J.Y.: Semantic Image Synthesis with Pratially-Adapiin Normalization. In CVPR. (2019)
[10] Vondrick, C., Pirsiavash, H., Torralba, A.: Generating Videos with Scene Dynamics. In NIPS. (2016)
[11] Tulyakov, S., Liu, M.Y., Yang, X., Kautz, J.: MoCoGAN; Decomposing Motion and Content for Video Generation. In CVPR. (2018)
[12] Xu, T., Wenliang, L.K., Munn, M., Acciaio, B.: COT-GAN: Generating Sequential Data via Causal Optimal Transport, In NeurIPS. (2020)
[13] Denton, E.L., Zaremba, W., Bruna, J., LeCun, Y., Fergus, R.: Exploiting Linear Structure within Convolutional Networks for Efficient Evaluation. In NIPS. (2014)
[14] Jaderberg, M., Vedaldi, A., Zisserman, A.: Speeding up Convolutional Neural Networks with Low Rank Expansions. In BMVC. (2014)
[15] Tai, C., Xiao, T., Zhang, Y., Wang, X., et al.: Convolutional Neural Networks with Low-Rank Regularization. In ICLR. (2016)
[16] Courbariaux, M., Bengio, Y., David, J.P:: Binaryconnect: Training Deep Neural Networks with Binary Weights during Propagations. In NIPS. (2015)
[17] Rastegari, M., Ordonez, V., Redmon, J., Farhadi, A.: XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks. In ECCV, (2016)
[18] Zhu, C., Han, S., Mao, H., Dally, WJJ.: Trained Ternary Quantization. In ICLR. (2017)
[19] Polino, A., Pascanu, R., Alistarh, D.: Model Compression via Distillation and Quantization. In ICLR. (2018)
[20] Han, S., Pool, J., Tran, J., Dally, W.: Learning both Weights and Connections for Efficient
Neural Network. In NIPS. (2015)
[21] Kang. M., Han, B.: Operation-aware soft channel pruning using differentiable masks. In ICML. (2020)
[22] Wen, W., Wu, C., Wang, Y., Chen, Y., Li, H.: Learning Structured Sparsity in Deep Neural Networks. In NIPS. (2016)

---

[23] Li, Hao and Kadav, Asim and Durdanovic, Igor and Samet, Hanan and Graf, Hans Peter: Pruning Filters for Efficient ConvNets. In ICLR. (2017)
[24] Liu, Z.. Li, J., Shen, Z., Huang. G., Yan, S., Zhang, C.: Learning Efficient Convolutional Networks through Network Slimming. In ICCV. (2017)
[25] He. Y., Liu, P., Wang, Z., Hu, Z., Yang, Y.: Filter Pruning via Geometric Median for Deep Convolutional Neural Networks Acceleration. In CVPR. (2019)
[26] Hinton, G., Vinyals, O., Dean, J.: Distilling the Knowledge in a Neural Network. arXiv preprint urXiv:1503.0253: (2015)
[27] Zagoruyko, S., Komodakis, N.:
Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer. ICLR. (2017)
[28] Passalis, N., Tefas, A.: Learning Deep Representations with Probabilistic Knowledge Transfer. In ECCV. (2018)
[29] Tian, Y., Krishnan, D., Isola, P:: Contrastive Representation Distillation. In ICLR. (2019)
[30] Chen, L.., Wang, D., Gan, 2., Liu, J., Henao, R., Carin, L.: Wasserstein Contrastive Representa- tion Distillation. In CVPR. (2021)
[31] Shu, H., Wang, Y., Jia, X., Han, K., Chen, H., Xu, C., Tian, Q., Xu, C.: Co-Fivolutionary Compression for Unpaired Image Translation. In ICCV. (2019)
[32] Fu, Y., Chen, W., Wang, H., Li, H., Lin, Y., Wang, Z.: AutoGAN-Distiler: Searching to Compress Generative Adversarial Networks. In ICML. (2020)
[33] Liu, Y., Shu, Z., Li, Y., Lin, Z., Perazzi, F., Kung, S.Y.: Content-Aware GAN Compression. In CVPR. (2021)
[34] Hu, H., Peng, R., Tai, Y.W., Tang, C.K.: Network trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architectures. arXiv preprint 1rXiv:1607.0322: (2016)
[35] Li, M., Lin, J., Ding, Y., Liu, Z., Zhu, J.Y., Han, S.: Gan Compression: Efficient Architectures for Interactive Conditional GANs. In CVPR. (2020)
[36] Wang, H., Gui, S., Yang, H., Liu, J., Wang, Z.: GAN Slimming: All-in-One GAN Compression by A Unified Optimization Framework. In ECCV. (2020)
[37] Li, S., Lin, M., Wang, Y., Chao, F., Mao, X., Xu, M., Wu, Y., Huang, F,, Shao, L., Ji, R.: Learning Efficient GANs for Image Translation via Differentiable Masks and co-Attention Distillation. arXiv preprint arXiv:201 1.111,..0100 1.08382 (2020)
[38] Jin, Q., Ren. J., Woodford, O.J., Wang, J., Yuan, G., Wang, Y., Tulyakov, S.: Teachers Do More Than Teach: Compressing Image-to- Maagge Models. In CVPR. (2021)
[39] Ren, Y., Wu, J., Xiao, X., Yang, J.: Online Multi-Granularity Distillation for GAN Compression. In ICCV. (2021)
[40] Johnson, J.. Alahi, A., Fei-Fei, L.: Perceptual Losses for Real-Time Style Transfer and Super-Resolution. In ECCV. (2016)
[41] Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.: The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. In CVPR. (2018)
[42] Barber, D., Agakov, F.: The im algorithm: A Variational Approach to Information Maximization. In NIPS. (2004)
[43] Mirza, M., Osindero, S.: Conditional generative adversarial nets.
arXiv preprint
11Xiv:1411.1784 (2014)
[44] Ledig, C., Theis, L.,, Huszár, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A., Tejani, A., Totz, J., Wang, Z., et al.: Photo-Realistic Single Image Super-Resoluce using a Generative Adversarial Network. In CVPR. (2017)
12

---

[45] Reed, S., Akata, Z., Yan, X., Logeswaran, L., Schiele, B., Lee, H.: Generative Adversarial Text Image Synthesis. In ICML. (2016)
[46] Pathak, D., Krahenbuhl, P., Donahue, J., Darrell, T., Efros, A.A.: Context Encoders: Feature Learning by Inpainting. In CVPR. (2016)
[47] Ba, J., Caruana, R.: Do Deep Nets really Need to be Deep? In NIPS, (2014)
[48]
Romero, A., Ballas, N., Kahou, S.E., Chassang, A., Gatta, C., Bengio, Y.: Fitnets: Hints for Thin Deep Nets. In ICLR. (2015)
[49] Ahn, S., Hu, S.X., Damianou, A., Lawrence, N.D., Dai, Z.: Variational Information Distillation for Knowledge Transfer. In CVPR. (2019)
[50] Tian, Y., Krishnan, D., Isola, P.: Contrasive representation distillation. In ICLR. (2020)
[51] Wang, Y., Xu, C., Qiu, J., Xu, C., Tao, D.: Towards evolutionary compression. In KDD. (2018)
[52] Cai, H., Gan, C., Wang, T., Zhang, Z., Han, S.: Once-for-All: Train One Network and Specialize it for Efficient Deployment. In ICLR. (2019)
[53] Cortes, C., Mohri, M., Rostamizadeh, A.: Algorithms for Learning Kernels based on Centered Alignment. JMLR (2012)
[54] Rudin, I.I., Osher, S., Fatemi, E.: Nonlinear Total Variation based Noise Removal Algorithms.
Physica D: nonlinear phenomena (1992)
[55] Li, S., Wu, J., Xiao, X., Chao, F., Mao, Ji, R.: Revisiting Discriminator in GAN Compression: Gemerator-diseriminntt Cooperative Compression Scheme. In NeurIPS. (2021)
[56] Wainwright, M.J., Simoncelli, E.P.: Scale Mixtures of Gaussians and the Statistics of Natural Images. In NIPS. (1999)
[57] Vemulapalli, R., Tuzel, O., Liu, M.Y.: Deep Gaussian Conditional Random Field Network: A Model-based Deep Network for Discriminative Denoising. In CVPR. (2016)
[58] Zhang, R., Isola, P., Efros, A.A.: Colorful Image Colorization. In ECCV. (2016)
[59] Welling, M., Teh, Y.W.: Bayesian Learning via Stochastic Gradient Langevin Dynamics. In ICML. (2011)
[60] Nijkamp, E., Hill, M., Zhu, S.C., Wu, Y.N.: Learning Con-Coovergent Con-Persistent Short- Run MCMC Toward Energy-Based Model. In NeurIPS, (2019)
[61] Xie, J., Lu, Y., Zhu, S.C., Wu, Y.: A Theory of Generative Convnet. In ICML. (2016)
[62] Hinton, G.E.: Training Products of Experts by Minimizing Contrastive Divergence. Neural computation (2002)
[63] Tieleman, T.: Training Restricted Boltzmann Machines using Approximations to the Likelihood Gradient. In ICML. (2008)
[64] Du, Y., Mordatch, I.: Implicit Generation and Modeling with Energy Based Models. In NeurlPS. (2019)
[65] Gao, R., Lu, Y., Zhou, J., Zhu, S.C., Wu, Y.N.: Learning Generative Convnets via Multi-grid Modeling and Sampling. In CVPR. (2018)
[66] Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., Aila, T.: Analyzing and Improving the Image Quality of StyleGAN. In CVPR. (2020)
[67] Yu, A., Grauman, K. Fine-Grained Visual Comparisons with Local Learning. In CVPR. (2014)
[68] Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., Schiele, B.: The Cityscapes Dataset for Semantic Urban Scene Understanding. CVPR. (2016)
13

---

[69] Ronneberger, O., Fischer, P., Brox, T.: U-Net: Convolutional Networks for Biomedical Image Segmentation. In MICCAL (2015)
[70] He, K., Zhang, X., Ren, S., Sun, J.: Deep Residual Learning for Image Recognition. CVPR. (2016)
[71] Karras, T., Laine, S., Aila, T.: A Style-based Generator Architecture for Generative Adversarial Networks. In CVPR. (2019)
[72] Liu, Z., Luo, P., Wang, X., Tang, X.: Deep Learning Face Attributes in the Wild. In ICCV. (2015)
[73] Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Hochreiter, S.: GANs Trained by Two
Time-Scale Update Rule Converge a Local Nash Equilibrium. In NIPS. (2017)
[74] Szegedy, C., Vanhoucke, V., loffe, S., Shlens, J., Wojna, Z.: Rethinking the Inception Architec- ture for Computer Vision. In CVPR. (2016)
[75] Yu, F., Koltun, V., Funkhouser, T.: Dilated residual networks. In CVPR. (2017)
[76] Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al.: PyTTrrc: An Imperative Style, High-Performance Deep Learning Library. In NeurIPS, (2019)
[77] Oord, v.v.d,, Li, Y., Vinyals, O.: Representation Learning with Contrastive Predictive Coding. arXiv preprint arXiv:1807.00331 (2018)
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes] See the abstract and Section
(b) Did you describe the limitations of your work? [Yes] See Section ??.
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] we have read the guidelines and ensured that our paper conforms to them.
2. If you are including theoretical results...
(a) Did you state the full set of assumptions all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes] We attach the code and instructions in the supplementary material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section 5
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [No] We did not report them.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [No] We did not include them.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] See Section
(b) Did you mention the license of the assets? [No] We did not mention the license.
(c) Did you include any new assets either in the supplemental material or as a URL? [No]
No we did not introduce any new assets.
(d) Did you discuss whether and how consent was obtained from people whose data you're
using/curating? [Yes] See Section
14

---

(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No] No we did not.
5. If you used crowdsoureing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "tZUEx2i0H3B", "reviewer": "Reviewer_cQSG", "summary": "Information-Theoretic Generative Model Compression with Variational Energy-based Model focuses on compressing generative and structured prediction neural networks, using a combination of teacher student distillation (either online or offline) and a variational energy based model which is optimized by MCMC. Combining a flexible variational distribution rather than alternative approaches taken in prior work, this method shows improvement on several image to image and image generation network compression tasks when combined with existing methods in the literature.\n", "questions": "Are there specific technical issues preventing the application of this method to more recent (perhaps patch based, latent, or otherwise) models e.g. VQ-GAN, ViT-VQ-GAN, or other large scale image modeling networks used in recent text-to-image generation? If there are limitations in terms of assumptions, this would also be a good thing to state in a limitations-type section.\n\nSimilarly to the previous question, it seems like most (but not all) of the experiments compare with a best method + VID and/or CRD - particularly between Table 1 2 and 3, on the best performing methods sometimes CRD, VID, or both are missing. Is there a direct reason for this, beyond resources/time? \"Filling the grid\" so to speak, or explaining why VID/CRD is not applicable in some cases, would help follow-on researchers who may tackle one of these specific areas in greater detail.", "limitations": "The limitations (listed in section A.4) are fairly minimal, and finding other limitations of the method compared to baselines (listing increased hyperparameters / tuning / MCMC related issues for example) would be beneficial. Given application to a face-related model (StyleGAN2) there is potential to analyze or focus on a broader study of where and how the compression modifies generations. There are relevant studies (for example \"Fairness for Image Generation with Uncertain Sensitive Attributes\", Jalal et. al. or \"Can Model Compression Improve NLP Fairness\", Xu and Hu in NLP) which can provide direction for looking at subsets of the overall data, to assess FID or other structural metrics to determine if one subset is disproportionately changed by the compression compared to the baseline. If this study is overly involved (to the level of needing its own study/paper), it is also appropriate to state as a limitation that the compression method has not been extensively tested with respect to increasing, reducing, or matching the bias of the original model, only analyzing characteristics such as parameter count and aggregate performance.", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nThere are a wide swath of generative models and compression methods applicable to such networks, the authors have done a commendable job in ablations against other relevant methods, as well as detailing both the background / core methodology of their method and several others. Though the proposed method (in terms of implementation) is quite complex, the description as well as coupled code and detailed supplemental materials are sufficient to fully understand the proposed method. This clarity should make the paper accessible to the broader community.\n\nOverall, I find the paper to be good quality, with significant interest to the model compression community and (with potential application to alternate methods) interest to the generative community at large.\n\nWeaknesses:\nWeaknesses in terms of experiments largely come in the form of \"directional improvement\", specifically for large, high performance models. The experiments here show a marked improvement in the small-to-medium scale, but application to more recent models could improve the impact of this work in the broader community - the lack of inference overhead, a large reduction in parameter count, and preservation or improvement of performance (in terms of FID) on the benchmarks are all critical aspects to running large scale text-to-image generators on commodity hardware.\n\nOne minor weakness is originality, as the proposed method follows closely in the footsteps of prior work from a methodology perspective, and sees gains in combination with other existing methods (rather than via replacement). This not a critical problem, but on the specific criteria of originality the contribution here feels \"moderate\", compared to other areas discussed in \"Strengths\".\n\nA secondary weakness in the experiments is scale - more experiments on the scale of StyleGAN2 (or similar size models / image domains) would be potentially be beneficial. There are significant performance gains in terms of FID on the smaller scale experiments, but all methods (including this one) result in lower FID performance on the largest task.\n\nWhile the proposed method does lower the gap between compressed and uncompressed (while having a drastic reduction in parameter count) compared to relevant benchmarks, it is still counter to the trend seen in several of the other experiments overall.\n\nHaving experiments on other domains or models (bedrooms as a potential intermediary? Large scale non-face models?) could show whether this effect is particular to StyleGAN2 (being a quite high performance model), or a broader trend on small scale vs large scale image models. Varied effects on small scale (potential performance *improvement* where parameter reduction is unneeded) and large scale (massive parameter reduction while minimizing the performance gap) experiments is a useful contribution, that is currently not spelled out. Though asking for more experiments is difficult given the extensive studies the authors have already provided, even one more large-scale experiment could potentially illuminate some properties of the proposed method. This bridges into some of the questions in the following section.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "J4a6pietHs", "reviewer": "Reviewer_6hcq", "summary": "\nSampling generative networks on devices with limited computational capabilities imposes limits on complexity of these networks. The concept of distillation provides one approach to obtaining networks whose performance gracefully degrades with network complexity. \n\nPaper proposes a new distillation technique for generative adversarial networks. The method uses a tractable lower bound on mutual information between student and teacher generative networks. \n\nAuthors introduce a family of methods which optimize variational lower bound on the mutual information between teacher and student network. The optimizer alternates between updating variational parameters, student network’s parameters and optionally teacher network’s parameters. This approach combines well with existing network compression algorithms.", "questions": "What is the interpretation of vertical arrows in Fig1?\n\nBackbone algorithms? Reference?\n\nWhat is the size of parameterization of the q(t|s) network for different tasks (architecture from supp inf Fig.4)? \nWhat is the computational cost of tacking VEM on top of other distillation/compression methods?\n\nIs VEM applicable in absence of a distillation method to piggy back on?", "limitations": "Due to presence of MCMC steps and need to track auxiliary Langevin dynamics variables there is an increased space and computational cost. The degree to which this cost can be decreased (or increased) and how that impacts the performance of the method is not quite clear. ", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The proposed method can be combined with existing compression algorithms. It improves performance measured by FID and mIoU for fixed number of parameters in a network compared to compression algorithms it augments. The paper is clear with few small corrections. Code is readily available. \n\n\nClarity: specify that p(t,s) = p(t|x) p(s|x)p(x) since mutual information stems from reuse of code between student and teacher.\nIt might be helpful to provide an example of perfect q(t|s) that keeps the bound tight -- p(x|s)p(t|x) -- and consists of decoder of students output and teacher's encoder.\n\nClarity:``student generator learns to maximize the variational lower bound'' might be more helpful rewritten as ``generator is trained by maximizing …''  The point here is that: Optimization algorithm optimizes, learning algorithm learns (if bug free and fed the right data), while the student generator ... generates samples. Crucially student generator does not learn how to maximize.\n\nVisuals in Figure 1 are a little difficult to understand. Arrows seem to have inconsistent meanings. Some arrows are simply meant as feeding forward outputs from one stage to another. Others are more abstract like the vertical arrows connecting teacher and student networks. Perhaps the intent was to refer to distillation methods embedded in the overall framework. However, the arrows themselves do not have a mathematical or algorithmic analog in the paper. This left me wondering if I missed some piece of the machinery. \n\nFID, MACs, mIoU and any other abbreviations should be defined and referenced.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "7Sb95teitJ0", "reviewer": "Reviewer_tn1T", "summary": "This paper proposes a knowledge distillation approach to accelerate GANs via maximizing mutual information between the teacher model and the student model. While mutual information (MI) between two continuous distribution is intractable, the paper alternatively maximize the lower bound of MI. The method is evaluated on both conditional and unconditional GANs with a noticeable numeric improvement. ", "questions": "See weakness.", "limitations": "Yes.", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. While VID has been proposed for classification model distillation, maximizing mutual information between teacher and student for GANs is more challenging and bears certain novelty.\n2. The motivation is clear and the technical details look correct.\n3. Experiments are carried out on multiple tasks and show good quantitative results.\n\nWeakness:\n1. My primary concern lies in the qualitative results. In fact, there is still not a good metric that really quantifies GAN's performance and thus the qualitative results are crucial to reflect the effectiveness of the approach. Looking at Fig. 2 of the paper, it's hard to tell whether OMGD or OMGD + VEM bears better quality. In addition, it seems that +VEM does not push the outputs from student to look similar to the teacher's. Why is that the case, as the mutual information is maximized during the distillation? It would be good to include figures or quantitative metrics (between teacher and student outputs, not FID) demonstrating the effectiveness of the distillation.\n2. The approach is only evaluated on GANs generating 256px images. However, prior method like CAGC has been applied for 1024px StyleGAN2. How would the method perform on high-resolution synthesis?\n3. The method uses an additional energy-based neural network and requires back-propagation to optimize its parameters, which is non-trival overhead. A computational time profiling on the additional module should be presented. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept