Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: GANs Can Play Lottery Tickets Too

Summary: 

This paper investigates model pruning and the Lottery Ticket Hypothesis in the context of GAN training. The authors apply a set of pruning techniques to SN-GANs on CIFAR and CycleGANs and examine whether pruning masks can be applied to the models at their initialization state (or their state 20% into training), taking into consideration the GAN-specific choice of whether to independently prune G or to prune G and D jointly. The results suggest that the sparsity masks obtained from pruned networks can be applied to networks at initialization up to a given sparsity level while recovering similar or better performance on the target task.


My take:


This is an empirical paper which focuses its efforts on experimental design and elucidating some details of pruning in GAN training. While I think the experiments are reasonably well-designed for their intention, my primary concern is that I am not convinced that there’s any reason that lottery tickets should matter in this context at all, given that the process (a) requires one to train a model to completion in order to obtain the masks, and then (b) require access to the trained model for a distillation loss. 

From my point of view, while it is interesting that one can use these masks on the networks in their init state, given that the actual goal of pruning is to obtain a slimmer model for inference (or to train models more quickly if pruning during training, which this technique does not permit), it is quite clear that standard pruning (pruning the pre-trained model rather than doing any sort of rewinding) is highly preferable in almost every way. That this result is downplayed in favor of highlighting that the authors have found lottery tickets is also concerning, albeit keeping with the general trend in the lottery ticket line of work which is more concerned with elucidating phenomenology than obtaining practical pruning or sparsity techniques. I feel strongly that the significance of this work will be diminished by this. On the other hand, I think that given that the intent of the paper is not as much to obtain the best possible sparsification technique as it is to investigate lottery tickets, this is perhaps okay, and I feel that the experimental design on this front is fairly strong and methodologically sound.

My other concern is that this paper’s clarity is poor. On the whole, the paper is not well presented, with poor or outright incorrect notation in many places. I struggled to read much of this paper due to this, and I feel that the paper’s impact would be strongly diminished due to this.

Overall, I would rate this paper about a 4.8. I will argue against acceptance, but I don’t think this is 100% a clear reject and depending on the opinions of the other reviewers I would not feel that accepting this paper was completely out of bounds.


Detailed and minor notes:



-Bibliography is inconsistently formatted: many of the references are missing information such as venue. If a paper is only on arXiv, the reference should indicate this. Note that one can use Scholar to automatically get the bibtex for a paper, and that one should do one’s best to get the most recent information on where a paper has landed. For example, the citation for the ResNet paper has no indication of the venue, when it was accepted by CVPR after being on arXiv first. 


-” SNGAN with ResNet (He et al., 2015) is one of the most popular noise-to-image GAN network and has state-of-the-art performance on several datasets like CIFAR10.”

This is not true, SNGAN is far from SOTA on CIFAR. In general I don’t think we should care much about SOTA or “a few points below SOTA,” especially on datasets like CIFAR, but if the authors are specifically going to use the term “state-of-the-art” it is not acceptable to omit related work from the past three years. Please consider consulting paperswithcode for at least a fairly broad overview of recent literature that reports results on these datasets.


-“let g(x; θg) be the output of the generator network G “.

The authors re-use the notation “x” (typically used to refer to a sample from the dataset) to refer to the random noise latent input to G, instead of the standard practice “z.” This might be acceptable if the authors did not immediately then re-use “x” to refer to the input to the discriminator, and then again as the inputs to both G and D in the image-to-image task. Please correct this misuse of notation.

The authors incorrectly use the notation “. in R^d1” in multiple places to refer to multiple different things which certainly have different dimensionalities, and additionally mismatch the notation when referring to variables which *should* have the same dimensionality (such as referring to m in {0,1}^d, the mask for a parameter theta in R^d1. This is further exacerbated as the subscript “d” is used to refer to the discriminator parameters in e.g. theta_d. Please correct this misuse of notation.

-Figure 9: I can’t see any difference between the source image and the output of the full model, but I can see checkerboard artifacts in the highly sparse models. I found this figure confusing and out of place.

-In the background and related work, I feel that the authors omit (or do not make clear) the critical detail of the Lottery Ticket Hypothesis, which is that these subnetworks *exist at initialization* assuming one uses the same initialization weights (i.e. initializes uses the same random seed). It is also important to note that the Lottery Ticket Hypothesis in this form does not hold past the very small-scale tasks on which it was originally tested, which is what necessitates the “rewinding” technique (and, in this reviewer’s opinion, greatly weakens the hypothesis).

-“We use a benchmark pruning approach named Standard Pruning”

Is there a reference for standard pruning, or is this novel? If this is something the authors did not introduce, appropriate citations should be included.


-”In summary, initialization benefits finding winning tickets”
I was not able to understand this statement; please consider rewording it.

-It is not surprising that resetting the generator but not resetting the Discriminator fails to train (IMPFG in Figure 4), as Gs and Ds are basically always intrinsically paired in this manner.

=========================================================================================

Edit post Rebuttal:

Thank you to the authors for their in-depth response and the effort they put into responding to this review, and my apologies for not engaging during the discussion phase. I'll respond to several points with the goal of furthering the discussion to try and avoid unfairly "having the final say" when the authors cannot respond.

-"The biggest potential impact of our work is that it provides empirical evidence that lottery tickets exist in GANs"
This reviewer's opinion is that this is not at all surprising. While GANs do have unusual and unique training dynamics on a number of levels, and interact interestingly with many typical building blocks, many of the aspects of neural-network based GAN models (prunability, the relationship between signal propagation and trainability, model capacity, etc) are largely indistinguishable from those of more "typical" neural nets. Nonetheless I do agree that, if one holds the lottery ticket line of work to be important or relevant (which I personally do not, but I will withhold my bias and not "legislate from the bench" here), its extension into the realm of GANs does first require verification of its existence in this regime. I apologize to the authors if it seems like they're pushing against a brick wall here because this reviewer is not a disciple of the lottery ticket hypothesis--please note that to the best of my ability this review is meant to be calibrated around this bias and instead focus on the strength of the manuscript.

-Transfer between tasks: I appreciate the authors response on this point; transfer of masks between tasks (rather than pretrained weights) does indeed represent a different modality of transfer learning, and one which may be well-suited to GANs which are known to be difficult to re-train or fine-tune in many situations (although there is a growing body of work on this topic outside of the pruning/lottery ticket context).

-"Lottery tickets without early weight rewinding techniques can still hold to small-scale tasks. "
Yes, this is precisely the problem--lottery tickets without early weight rewinding *do not, in general, tend to hold on large scale tasks*. This again returns to my baseline issue (one of many) with the hypothesis in the first place, that it has to be modified to work on even models like VGG.

-on checkerboard artifacts: This reviewer is very well calibrated to viewing GAN samples, and holds that the checkerboard artifacts are vastly more visible in the sparsified models. I would encourage the authors to, in future work,  consider why this might be (what about sparse networks leads to increased checkerboard artifacts?) rather than seek to claim that the unpruned models are also checkerboard-y.

-Additional experiments: I commend the authors for repeating their experiments on a range of architectures, and agree that the improvements for the sparsified models are in support of their argumentation and conclusions.

-Thank you to the authors for updating their notation and bibliography.

On the whole, I still feel that this paper is borderline. While the author's responses are fairly strong in context, re-reading the manuscript, I still do not feel that the paper (which is most of what matters here) is especially strong. I am upgrading my score to around a 5.5 (which I will simply round up to a 6 on OpenReview). I think this paper is true borderline--I won't argue for its rejection, but I don't feel especially strongly in favor of it and cannot champion it.
review_title: Tend to Reject

final_score: rating: 6: Marginally above acceptance threshold
review_content: In this paper, the ideas from lottery ticket hypothesis are evaluated in the case of SNGAN and CycleGAN. 

The evaluations aim to validate the following empirical results with respect to the lottery ticket hypothesis i.e. the iterative magnitude pruning and channel pruning ideas work, discriminator can also be pruned in addition to generator, it outperforms other compression methods such as GAN slimming and that instead of re-initializing the weights of the masked network to the initialization, it could be reset to that from an earlier epoch.

While most of the ideas evaluated have already been proposed for general model compression, in this paper they are specifically evaluated for GANs. 

The proposed work is useful in terms of the study and analysis of a particular approach from model compression to that of GANs, it would be useful to understand if the results are meaningful specially for the much larger GAN models such as Progressive GANs, BigGAN for high resolution images. This is because, compression is especially useful for the larger models and more so when one is generating high resolution images. Many of the early results on GANs trained on CIFAR 10/100 and SVHN do not hold for the larger GAN models. 

To summarize the pros and cons of the paper are as follows:
Pros:
1) Evaluates various aspects of the Lottery ticket hypothesis for specific GAN models
2) Demonstrates results for both image to image and generation from latent cases and shows improvements over GAN slimming
3) Provides useful analysis for the lottery ticket hypothesis by analysing specific claims that are relevant

Cons:
1) Larger GAN models and larger datasets are not evaluated. The approach could be more meaningful for those models and datasets
2) The approach evaluates a specific lottery ticket hypothesis and does not provide new insights into the actual hypothesis itself. That is, by applying it to GANs we could only verify most of the already valid claims that were earlier proposed
3) It is not clear that the ideas proposed would generalize to other GAN models. There are a huge number of GAN models proposed in literature and validating the ideas on all would be infeasible. It would however be useful to provide some more results on a few more GAN models, particularly addressing the first point.

---
The response from the authors satisfactorily addresses several points that I raised. However, I am not fully convinced that the LTH on GANs has provided significant new insights. However, based on the response I am inclined to raise my score to above acceptance threshold and tend towards acceptance.
review_title: Review

final_score: rating: 6: Marginally above acceptance threshold
review_content: In this paper, the authors provide an empirical study on lottery ticket hypothesis on GANs. To do this, the authors use two GAN models and two datasets: SNGAN/CycleGAN and CIFAR-10/horse2zebra. Extensive experiments show that matching subnetworks can be found using unstructured magnitude pruning and channel pruning and they are transferrable to other tasks. The performance of subnetworks found is competitive and even surpasses state-of-the-art performance.

This paper is well-written and addresses an interesting problem in GANs compression. This paper first studies to verify if lottery ticket hypothesis works on GANs. Although the concept of LTH is not new, empirical verification on GANs is of value to this community. The experimental set-up including methods, datasets, and claims is solid and empirical results for the claims are convincing.

I am not familiar with the original work of lottery ticket hypothesis, but it is easy to understand the concept of it without reading the related works except for the following:
1. Is the iterative magnitude pruning (IMP) equivalent to the unstructured pruning described in p3?
2. What is the difference between matching subnetworks and winning tickets?
For self-contained paper and readers unfamiliar with LTH, it would be better if these terms are clarified in the revision.

The authors mainly use FID scores and additionally compare Inception Scores. It would be better to see if experiments on recent metrics (e.g. precision/recall) show the same result.
review_title: Empirical study of lottery ticket hypothesis on GANs is worth to share in this community

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: **Summary**

The authors study the lottery ticket hypothesis  for generative adversarial networks. Specifically, they attempt to answer the following questions: the existence of winning tickets in GANs; the effect of discriminator pruning in finding such winning tickets; the effect of initialization during the rewinding steps; and finally if the subnetworks found transfer across datasets. They provide extensive empirical evidence using that ```winning' tickets exist in GANs. Further they show that iterative magnitude pruning and channel pruning successfully find such `winning' subnetworks. They analyse the effect of discriminator pruning and find that initialization during the rewind step matters more than the actual pruning of the discriminator. Finally, they show state-of-the-art results on GAN compression through channel pruning.

**Strengths**
1. The paper is well-written and has cited relevant related work. 

2. The work is well-motivated and novel. The authors answer some important questions about LTH based methods.

3. The experiments are extensive and help prove the authors' claims. I especially appreciate the attention to detail in the experiments, with various comparisions and ablation studies. 

**Weaknesses and Clarifications**

From the given qualitative results, there does seem to be a loss in the finer features (edges and textures) upon sparsification. However, the FID scores do not seem to reflect this. Could the authors provide more visual results to analyse this?


review_title: Review

Meta-review: Three reviewers recommended accept or weak accept. There are some concerns on the novelty of this approach since this work mainly validates that lottery tickets can be found on GANs, which seems like applying an existing idea to a new problem. Nevertheless, there are two reasons that such an effort is interesting: first, the implementation of this idea may be harder than it seems; second, it was not known a priori whether lottery tickets would exist for GANs due to the significantly different optimization problem (game instead of minimization). I think this work will be of interest to the community, and recommend acceptance. 
Decision: Accept (Poster)