Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: The problem tackled in the paper is the compression of generators in adversarially trained models. Considering the success of large generative models (like BigGAN) one may wonder if these models can be compressed after being trained to improve practical applicability. 

This paper is focused on the compression of image to image translation models and uses distillation on discriminator's outputs to achieve better results.

My decision is weak reject.

The message of the article is misleading. The whole goal of pruning a neural network is to remove filters from it, therefore, reducing the computation or, at least, storage space for the parameters. The attempt to remove filters was presented in the last figure, and it does not work as good as all other results presented in the paper.

The comparison in Figure 1 is arguably misleading as well. For example, one of the methods that were mentioned (LIT) does achieve a factor of 1.8 model compression, yet the comparison was not carried out directly with that method, but a modification proposed by the authors of this paper.

I would like to see more comparisons in terms of FLOPs or inference time between the baselines, SotA methods, and your proposed method. Weights pruning is simply one of the approaches for model compression, so you cannot ignore the alternatives.

Also, section 4 probably has to be rewritten, since some unorthodox notation is used. The authors should consider using some reference paper for the notations, like CycleGAN, that was mentioned in the paper. That will improve the clarity and readability of the used objectives.
review_title: Official Blind Review #4

final_score: rating: 6: Weak Accept
review_content: This paper proposes a method to compress GANs. The motivation is that the current compression methods work for other kinds of neural networks (classification, detection), but perform poorly in the GAN scenario. The authors present intuitive reasons for why this is the case. 

However, the motivation why we would like to compress GANs is unclear to us. The intro mentions: reducing memory requirements and improving their performance. Sure, compressing networks for object detection and classification on mobile devices is really useful. But GANs are mainly used for unsupervised density estimation, why put a GAN generator on a mobile device? But maybe we are missing something here. 

Their “self-supervised” method works by using the pre-trained discriminator network, while compressing only the generator. They show both qualitative and quantitative gains.

The paper is clear and well-written. It presents a way of pruning GAN generator network and although of limited novelty, it might be an interesting read as it provides extensive and convincing experiments in a clear manner. It does have several parts though which require additional clarification.

The idea of using the pre-trained discriminator network seems reasonable, but I am missing what the compression method for the generator network actually is (Section 4). From Table 2 I would assume it is pruning, in which case the paper’s contribution is very limited.

The authors claim that the “self-supervised” method generalizes well to new tasks and models. "Generalizes" seems a strong word here, since the procedure compresses only the generator network. A more appropriate way of putting it might be ‘can be applied to other tasks and models.'

In Section 4 the authors write: “Our main insight is found,” but then they describe the GAN method. What is the actual insight there?

The qualitative results in Figure 1 suggest that their “self-supervised” method is better than the other baselines. 

Scores from Table 2 also support the claims, but the table itself is not referenced anywhere in the text.

The analysis in Section 6 seems out of context with the rest of the paper. It is not clear how it relates to the “self-supervised” method.

Missing related work: 1st paragraph: compressing or distilling one network into another is much older than 2015, dating back to 1991 - see references in section 2 of the overview http://people.idsia.ch/~juergen/deep-learning-miraculous-year-1990-1991.html 
The GAN principle itself is also much older (1990) - see references in section 5 of the link above.  

General remarks:

In the first read of Section 3 it is not clear what [a], [b], [c] are.

It would be good to first refer to Table 1.

Table 1: why is there a “?” only on the “Fixed” column?

It would be good to have a larger font size in Figure 2, at least the size of the main text font.

In its current form, the pdf file has 100MBs (8MBs the main paper and the rest is the appendix). One could instead move the images from the appendix to a website and provide a link.

We might improve our rating provided the comments above were addressed in a satisfactory way in the rebuttal.


review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: In this paper, the authors tackle the task of compressing a network. While there
are many effective solutions so far regular computer vision tasks, as they demonstrate,
they fail catastrophically  when applied to generative adversarial networks(GANs).
They propose a modification to the classic distillation method, where a
"student" network tries to imitate the uncompressed one under the supervision of
a fully converged discriminator network. They perform evaluation on multiple
tasks from image synthesis to super-resolution. They also study the influence
of the compression factor on the quality of the generated images.

The task is well motivated and situated in the related literature. The first
section is very thorough and extremely efficient at describing the failure modes
of existing methods. On one side, the results demonstrated in the evaluation are
compelling, on the other side, the compression factor is only 50%, which is much
lower than seen in related work. However, as it is shown in section 3 the task
may be much harder for GANS than regular models so I still consider it a 
sizeable contribution.

There are a couple of points that require clarification. I personally found the
description of the method (Section 4) rather confusing. It is clear
what "discriminative loss" is as it is the one used in every GAN.
Unfortunately, I could not understand what "generative loss" means in the general
case. An example is given for StarGAN in equation (7) and I have a rough idea of
what to choose for Style Transfer, Domain Translation, Super Resolution and
Image translation. Though, it is unclear to me what to use in the case of
image synthesis. The experiments clearly show that it is possible so I think
it is necessary to show how this framework is concretely applied to each task at
hand.

During training, the discriminator only ever saw pictures from the true distribution
and the distribution generated by the generator (at each of its training steps).
If I understood the framework properly, here, the compressed generator is trained
from a random initialization. The distribution it outputs is therefore
completely unknown and potentially non overlapping with either of the true or
the generator ones. In that case it is hard to predict what the discriminator
would do on completely out of distribution samples. I seems reasonable to
conjecture that it might consider them "true" because it was never trained on
them. Could you provide an explanation of why it is not a problem in practice?
Do you have to try multiple initializations? Is the generative
loss enough to force the compressed discriminator to match the support
of the distribution of the dense generator?

I think this paper is novel, tackles a hard task and presents compelling results
(albeit using very mild compression ratios). It should be accepted if some
clarifications are made in section 3.

Minor Remarks:

- Figures 2, 9, 42 and 43 are unreadable when printed with a regular color office
  printer.
- It is unclear what it would take an extra 10% of the original number of epochs to train the compressed network. Why couldn't it be faster, or much longer?



review_title: Official Blind Review #2

Meta-review: The paper develops a new method for pruning generators of GANs. It has received a mixed set of reviews. Basically, the reviewers agree that the problem is interesting and appreciate that the authors have tried some baseline approaches and verified/demonstrated that they do not work. 

Where the reviewers diverge is on whether the authors have been successful with the new method. In the opinion of the first reviewer, there is little value in achieving low levels (e.g. 50%) of fine-grained sparsity, while the authors have not managed to achieve good performance with filter-level sparsity (as evidenced by Figure 7, Table 3 as well as figures in the appendices). The authors admit that the sparsity levels achieved with their approach cannot be turned into speed improvement without future work.

Furthermore, as pointed out by the first reviewer, the comparison with prior art, in particular with LIT method, which has been reported to successfully compress the same GAN, is missing and the results of LIT have been misrepresented. While the authors argue that their pruning is an "orthogonal technique", and can be applied on top of LIT, this is not verified in any way. In practice, combination of different compression techniques is known to be non-trivial, since they aim to explain the same types of redundancies.

Overall, while this paper comes close, the problems highlighted by the first reviewer have not been resolved convincingly enough for acceptance.
Decision: Reject