Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: Information-Theoretic Generative Model Compression with Variational Energy-based Model focuses on compressing generative and structured prediction neural networks, using a combination of teacher student distillation (either online or offline) and a variational energy based model which is optimized by MCMC. Combining a flexible variational distribution rather than alternative approaches taken in prior work, this method shows improvement on several image to image and image generation network compression tasks when combined with existing methods in the literature.

strengths_and_weaknesses: Strengths:
There are a wide swath of generative models and compression methods applicable to such networks, the authors have done a commendable job in ablations against other relevant methods, as well as detailing both the background / core methodology of their method and several others. Though the proposed method (in terms of implementation) is quite complex, the description as well as coupled code and detailed supplemental materials are sufficient to fully understand the proposed method. This clarity should make the paper accessible to the broader community.

Overall, I find the paper to be good quality, with significant interest to the model compression community and (with potential application to alternate methods) interest to the generative community at large.

Weaknesses:
Weaknesses in terms of experiments largely come in the form of "directional improvement", specifically for large, high performance models. The experiments here show a marked improvement in the small-to-medium scale, but application to more recent models could improve the impact of this work in the broader community - the lack of inference overhead, a large reduction in parameter count, and preservation or improvement of performance (in terms of FID) on the benchmarks are all critical aspects to running large scale text-to-image generators on commodity hardware.

One minor weakness is originality, as the proposed method follows closely in the footsteps of prior work from a methodology perspective, and sees gains in combination with other existing methods (rather than via replacement). This not a critical problem, but on the specific criteria of originality the contribution here feels "moderate", compared to other areas discussed in "Strengths".

A secondary weakness in the experiments is scale - more experiments on the scale of StyleGAN2 (or similar size models / image domains) would be potentially be beneficial. There are significant performance gains in terms of FID on the smaller scale experiments, but all methods (including this one) result in lower FID performance on the largest task.

While the proposed method does lower the gap between compressed and uncompressed (while having a drastic reduction in parameter count) compared to relevant benchmarks, it is still counter to the trend seen in several of the other experiments overall.

Having experiments on other domains or models (bedrooms as a potential intermediary? Large scale non-face models?) could show whether this effect is particular to StyleGAN2 (being a quite high performance model), or a broader trend on small scale vs large scale image models. Varied effects on small scale (potential performance *improvement* where parameter reduction is unneeded) and large scale (massive parameter reduction while minimizing the performance gap) experiments is a useful contribution, that is currently not spelled out. Though asking for more experiments is difficult given the extensive studies the authors have already provided, even one more large-scale experiment could potentially illuminate some properties of the proposed method. This bridges into some of the questions in the following section.
questions: Are there specific technical issues preventing the application of this method to more recent (perhaps patch based, latent, or otherwise) models e.g. VQ-GAN, ViT-VQ-GAN, or other large scale image modeling networks used in recent text-to-image generation? If there are limitations in terms of assumptions, this would also be a good thing to state in a limitations-type section.

Similarly to the previous question, it seems like most (but not all) of the experiments compare with a best method + VID and/or CRD - particularly between Table 1 2 and 3, on the best performing methods sometimes CRD, VID, or both are missing. Is there a direct reason for this, beyond resources/time? "Filling the grid" so to speak, or explaining why VID/CRD is not applicable in some cases, would help follow-on researchers who may tackle one of these specific areas in greater detail.
limitations: The limitations (listed in section A.4) are fairly minimal, and finding other limitations of the method compared to baselines (listing increased hyperparameters / tuning / MCMC related issues for example) would be beneficial. Given application to a face-related model (StyleGAN2) there is potential to analyze or focus on a broader study of where and how the compression modifies generations. There are relevant studies (for example "Fairness for Image Generation with Uncertain Sensitive Attributes", Jalal et. al. or "Can Model Compression Improve NLP Fairness", Xu and Hu in NLP) which can provide direction for looking at subsets of the overall data, to assess FID or other structural metrics to determine if one subset is disproportionately changed by the compression compared to the baseline. If this study is overly involved (to the level of needing its own study/paper), it is also appropriate to state as a limitation that the compression method has not been extensively tested with respect to increasing, reducing, or matching the bias of the original model, only analyzing characteristics such as parameter count and aggregate performance.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: 
Sampling generative networks on devices with limited computational capabilities imposes limits on complexity of these networks. The concept of distillation provides one approach to obtaining networks whose performance gracefully degrades with network complexity. 

Paper proposes a new distillation technique for generative adversarial networks. The method uses a tractable lower bound on mutual information between student and teacher generative networks. 

Authors introduce a family of methods which optimize variational lower bound on the mutual information between teacher and student network. The optimizer alternates between updating variational parameters, student network’s parameters and optionally teacher network’s parameters. This approach combines well with existing network compression algorithms.
strengths_and_weaknesses: The proposed method can be combined with existing compression algorithms. It improves performance measured by FID and mIoU for fixed number of parameters in a network compared to compression algorithms it augments. The paper is clear with few small corrections. Code is readily available. 


Clarity: specify that p(t,s) = p(t|x) p(s|x)p(x) since mutual information stems from reuse of code between student and teacher.
It might be helpful to provide an example of perfect q(t|s) that keeps the bound tight -- p(x|s)p(t|x) -- and consists of decoder of students output and teacher's encoder.

Clarity:``student generator learns to maximize the variational lower bound'' might be more helpful rewritten as ``generator is trained by maximizing …''  The point here is that: Optimization algorithm optimizes, learning algorithm learns (if bug free and fed the right data), while the student generator ... generates samples. Crucially student generator does not learn how to maximize.

Visuals in Figure 1 are a little difficult to understand. Arrows seem to have inconsistent meanings. Some arrows are simply meant as feeding forward outputs from one stage to another. Others are more abstract like the vertical arrows connecting teacher and student networks. Perhaps the intent was to refer to distillation methods embedded in the overall framework. However, the arrows themselves do not have a mathematical or algorithmic analog in the paper. This left me wondering if I missed some piece of the machinery. 

FID, MACs, mIoU and any other abbreviations should be defined and referenced.

questions: What is the interpretation of vertical arrows in Fig1?

Backbone algorithms? Reference?

What is the size of parameterization of the q(t|s) network for different tasks (architecture from supp inf Fig.4)? 
What is the computational cost of tacking VEM on top of other distillation/compression methods?

Is VEM applicable in absence of a distillation method to piggy back on?
limitations: Due to presence of MCMC steps and need to track auxiliary Langevin dynamics variables there is an increased space and computational cost. The degree to which this cost can be decreased (or increased) and how that impacts the performance of the method is not quite clear.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper proposes a knowledge distillation approach to accelerate GANs via maximizing mutual information between the teacher model and the student model. While mutual information (MI) between two continuous distribution is intractable, the paper alternatively maximize the lower bound of MI. The method is evaluated on both conditional and unconditional GANs with a noticeable numeric improvement. 
strengths_and_weaknesses: Strengths:
1. While VID has been proposed for classification model distillation, maximizing mutual information between teacher and student for GANs is more challenging and bears certain novelty.
2. The motivation is clear and the technical details look correct.
3. Experiments are carried out on multiple tasks and show good quantitative results.

Weakness:
1. My primary concern lies in the qualitative results. In fact, there is still not a good metric that really quantifies GAN's performance and thus the qualitative results are crucial to reflect the effectiveness of the approach. Looking at Fig. 2 of the paper, it's hard to tell whether OMGD or OMGD + VEM bears better quality. In addition, it seems that +VEM does not push the outputs from student to look similar to the teacher's. Why is that the case, as the mutual information is maximized during the distillation? It would be good to include figures or quantitative metrics (between teacher and student outputs, not FID) demonstrating the effectiveness of the distillation.
2. The approach is only evaluated on GANs generating 256px images. However, prior method like CAGC has been applied for 1024px StyleGAN2. How would the method perform on high-resolution synthesis?
3. The method uses an additional energy-based neural network and requires back-propagation to optimize its parameters, which is non-trival overhead. A computational time profiling on the additional module should be presented. 
questions: See weakness.
limitations: Yes.
review_title: null

Meta-review: metareview: This work concerns the compression of generative adversarial networks and other image generation networks, such as dense prediction/image to image networks. Where existing approaches to compress these models rely on matching pairs of outputs, this work optimizes the Barber-Agakov lower bound on the differential mutual information between teacher and student, parameterizing the bound using an energy-based model. Offline distillation alternately fixes the student network parameters and the EBM parameters while optimizing the other, while online distillation can be performed by also including the teacher parameters, holding two of the three fixed while optimizing the third in a "round robin" fashion. The gradient of the EBM partition function is estimated using Langevin dynamics for a fixed number of steps, with chains initialized at student outputs. Qualitative and quantitative results support the assertion that this represents an improvement over existing approaches. 

Reviewers found the paper overall quite clear, the method moderately original, the technical details sound, and the work well-situated in the context of other compression methods. Reviewer cQSG in particular praised the "commendable job in ablations against other relevant methods".  6hcq had several concerns around the clarity of the manuscript, which were addressed in rebuttal and hopefully can be incorporated into future versions. Several reviewers had concerns about the scale of experiments; the authors responded in rebuttal with megapixel experiments using StyleGAN2. The AC concurs with cQSG who remarked that these results significantly strengthen the paper.

Reviewer tn1T remained skeptical of the presented qualitative results, even after rebuttal; however, the authors were belatedly able to provide qualitative results on 1024x1024 images in the supplementary material. tn1T did not comment on these results, so I cannot infer how they feel about them, but they appear somewhat convincing to the AC: the samples presented from the VEM-trained do not show exhibit artifacts to the same degree.  There is, of course, the issue of cherry-picking, and I would encourage the authors to provide as many of these comparisons as possible at reduced size, highlighting artifacts where they appear but not dropping baseline samples which do not exhibit them (if such samples arise), and highlighting any noticeable artifacts in the images produced by their method.

The AC feels that this work is technically solid, and noting the endorsement of two reviewers and that the qualitative comparison demanded by tn1T was carried out but not acknowledged by tn1T, the case in favour of acceptance outweighs that in favour of rejection.
confidence: Less certain
Decision: Accept