Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposed to extend Lottery Ticket Hypothesis (LTH) for data-efficient GAN training. Specifically, the authors have found that sparse subnetworks lead to better sample qualities of Image GANs. In addition, they have proposed AdvAug that combines the ideas of feature augmentation and adversarial training.
main_review: [Strengths]
- The paper is largely well-written and easy to understand.
- Using sparse subnetwork to enhance data efficiency seems novel (to the best of my knowledge).

[Weaknesses]
- Limited novelty in terms of method. This paper does not present any new techniques in the method section. Lottery ticket hypothesis was explored in prior literature. Also, using adversarial training to improve robustness of deep networks are pretty well-known in computer vision.
- Unconvincing experiment results. Since the novelty of method is limited, the value of the paper hinges on lessons we can learn from the experiments. From Table 1, on 100% and 20% data, sparse network doesn't seem to be helping at all. However, on 10% data, sparse subnetworks are helpful. The results are not convincing to me and seem to be coincidental. I believe there is a lot of randomness going on here. The results are averaged over 3 "evaluation" runs, but not on 3 "training" runs. I suggest running experiments on 3 different 10%/90% splits for example. Also, in Tables such as Table 4, gaps between different entries are not big enough. Although there is no overlap between error bars, I believe these error bars are related to different evaluation runs (not training runs). From my experience, this kind of numerical difference can be easily reversed if another trial of GAN training is performed.
- No insight regarding why sparse subnetwork helps data-efficiency of GAN training. Also, does this finding can be extended to image recognition benchmarks?

[Typo]
- In line 164: Fig 2 outer circle should be Fig 1 outer circle
limitations_and_societal_impact: Limitations or potential negative societal impact are not discussed in depth in this paper. Including limitations as well as the position of this paper against prior literature will greatly enhance the quality of this paper.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors propose using iterative magnitude pruning, along with two novel augmentation strategies as a means to achieve few-shot GAN training. 

main_review: Novelty and research direction:

Finding ways to train GANs data-efficiently is a very intriguing premise. Attempting to use the lottery ticket hypotheses insights into achieving this is a very interesting research direction. The work uses previous work relating to ITP on GANs, on the few-shot domain, which is a fairly novel direction.

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Writing quality:

Overall the paper’s quality is decent but suffers from clumsy writing in various places, as well as some mild lack of conciseness and precision. 

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Summary of review:

The idea proposed is novel and very useful, however, the experiments undertaken do not sufficiently disentangle key variables that are necessary to draw a useful conclusion. Find the full details in the details section below (the experiments section is the one with the most value). 

Furthermore, the writing quality of the paper is at times clumsy, as well as mildly imprecise and inconcise. 

I cannot recommend this paper, in its current state for acceptance. 

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Details: 

1. Abstract:
The abstract is too large, and suffers from precision and clarity issues. Furthermore, the usage of citations in an abstractly is generally not recommended, at least in ML. Ideally an abstract should be 5-8 sentences long, unless there are special reasons for not doing so.

2. Introduction:

The very first sentence seems to be lacking precision since you imply that all modern advancements in GANs can be attributed to better data and nothing else. Also the usage of the verb ‘blessed’ here feels a bit out of place, but that’s just my taste. The same term is reused in line 55, which again feels out of place. 

The rest of the introduction seems to suffer from clumsy writing in a few places, but overall flows well and distills down the main idea well, but fails to clearly nail down the exact contributions the authors are claiming.

3. Related Work:

Very good related work section, that seems to have a few places of clumsy writing, but overall seems to be on point. 

Also, some notable papers relating to few-shot learning + GANs seem to be missing, i.e.  

 i. https://arxiv.org/abs/1711.04340

 ii. https://arxiv.org/abs/1711.04043

4. Methodology:

AdvAug description is inadequate, how is PGD applied? Optimized against what? The figure provides no information, neither does Equation 1. This is a majour point, and should not be in the appendix. I was able to figure out what was going on, but had majour trouble piecing it together without the appendix. 
The rest of the section seems to be OK. However, the last sentence in 3.2 does not make much sense to me, since it’s overly convoluted. 

5. Experiments:

Overall, the experiments conducted to showcase the usefulness of advAug were very interesting. 

That being said, I have some reservations regarding the experiments in Table 1 and 2. Is your pruning method causing the improved performance there? Or is it the mere fact that you are using fewer weights, which happen to create a regularization effect that produces better generalizing generators? For Table 1 to be better able to showcase your point, you need to run experiments where you randomly remove 36% and 67.24% of all weights and train on all the various data settings. This way you can see if your proposed methods offer any benefit, or whether it’s all due to a smaller model with what is probably a pretty tree-esque structure, forcing bottlenecks left and right, and inadvertently causing regularization. 

Furthermore, the experiments conducted in Table 5 seem a bit unfair. It seems that most benefit comes from the augmentations you used. Have you tried applying those to the baseline pretrained methods? That would tell you if your pruning method is useful or not. Furthermore, again, just like in my previous point, you need to ensure that any added performance comes from your pruning method and not from the reduced model size. You can verify that by randomly dropping say 48.80% of your weights, by masking them away, and training the rest of the model to compare a random pruning vs the guided pruning you are already showcasing in the paper.

Furthermore, it would have been cool to see how finding a lottery ticket GAN on the pretraining dataset, would transfer to the few-shot tasks. That could also push your model performance further. 

Overall, the experiments don’t do a good enough job at disentangling some important factors required to draw useful conclusions about the claims made by the authors.

6. Conclusion:

Line 322-324 states that the work is scientific in nature and that this somehow means that the work will not pose substantial risk of societal harm. I find this statement a bit odd. Does work that is scientific get a risk-free card? Isn’t science a very powerful tool that can both protect and destroy? I am a bit confused here. Did the authors mean that the work is far removed from a practical application that could be malicious because it is highly experimental? Please clarify.

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Improving the quality of this bar, to potentially meet my subjective NeurIPS bar

The work itself is interesting and novel. However, a few things need to be done to get this paper at a level where I’d be comfortable recommending acceptance.

1. Proof read the paper, and ensure the abstract is smaller (5-8 sentences), and make clarity and precision all around a priority. I find it useful to have colleagues from other institutions/countries review the work, as they are more objective and further removed from the work, that they can easily pick up any clarity/understanding issues.
2. For some very useful tips have a look at the below websites (which I use myself and my students all the time, and it’s really good stuff):

 a. https://cs.stanford.edu/people/widom/paper-writing.html 

 b. https://www.easterbrook.ca/steve/2010/01/how-to-write-a-scientific-abstract-in-six-easy-steps/ 

3. Ensure your introduction has clearly outlined contributions, perhaps use a bullet list to make them stand out to the reader even more.
4. Run experiments with randomly removed weights, to confirm that any gains come from your lottery ticket hypothesis method and not just a reduced model size and weird bottlenecks that inadvertently pop up in pruned networks.
5. Run experiments with your proposed augmentations applied to your baseline methods, this way you can ensure that your proposed LTH method + augmentations produce the best results, or to at least learn whether your gains are all due to the augmentation strategies used.
6. Try to find a LTH GAN on the pretraining datasets of other transfer methods, then fine tune on those with additional LTH steps on the few-shot domains. This might also help your method by leveraging data that other methods already use. 

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
limitations_and_societal_impact: Yes, they have, albeit in a clumsy manner, as I explain in the Details/Conclusion section above.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: Inspired by the lottery ticket observation, the authors decomposed GAN training into two stages: (i) finding sparse winning tickets in GANs by iterative pruning; (ii) training the found sparse tickets with various augmentations. Either sub-problem becomes more data-efficient to train, and they can re-use the same small set of real images.
main_review: This paper is a very strong empirical work on data-efficient GAN training. To my best knowledge, the authors are the first to show these sparse networks are much more efficient to train with smaller sets of data than their dense counterparts, and hence provide a blessing for few-shot regimes. 

From a fundamental perspective, it goes in parallel with the existing LTH literature along with the inference efficiency of networks. Although the lottery ticket phenomenon was discovered by other prior works, no existing work ever considered the influence of training data size on locating and training those tickets in general. It will inspire more re-thinking on the value of network sparsity, for not only GANs but also general deep NNs. 

The results are very strong across various datasets, some latest GAN models (including BigGAN and StyleGAN-V2), and the metrics reported.  Ablation studies are thorough and clear to follow.  In particular, the authors compared the two state-of-the-art methods DiffAug and ADA, and they were even able to include the very new RLC method (CVPR’21) into comparison, as well as their various combinations. I salute to the authors’ extensive efforts.

The authors’ technical approach is quite straightforward. But I feel to some extent, it is a pro rather than a con. Given the new angle (LTH for data-efficient training) as the main delivery of this paper, I like the authors could exploit this insight with very easy-to-follow approaches, and I value a simple method that works while bringing in new insights.

The writing clarity is good, but the AdvAug part seems a bit disconnected from the main theme of LTH.

limitations_and_societal_impact: It’ll be great if the authors can address the following concerns:

1. Table 1, Why 100% data was hurt by sparsity 67.24%, with or without AdvAug? Similarly in Table 3 CIFAR-10, Dense BigGAN outperformed sparse in all three conditions. It’s a bit confusing as the benefit of sparsity does not show to be fully consistent.

2. Table 4 StyleGAN-v2: neither AdvAug, nor DiffAug + AdvAug was reported, This looks weird, since previous BigGAN and SNGAN experiments mainly reported on the two.

3. Sparsity during training seems to benefit data-efficient learning, and LTH is one of the possible sparse training ways. Have you tried other ideas, such as SNIP or dynamic sparse training?

4. I can understand sparsity might constrain the hypothesis space and reduce the sample complexity. But why the sparsity structure pattern also matters in data efficiency (e.g., LTH largely outperforms random pruning)?

5. I don’t quite understand why network slimming performs so badly in Figure 7.

6. How about this algorithm applied to image-to-image translation GANs? I think this would be as interesting, such as developing some few-shot super-resolution models in some data-scarce scientific imaging domains.

7. Typos are non-sparse in the paper: “mini-max”, “and we then training them…”, and more. Proofreading is needed.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper addresses data-efficient GAN training.  It decomposes the task of the GAN training in low data regimes into two sub-problems: 1) finding the sparse subnetwork, so called lottery ticket, in the original GAN model, based on observations in [1,2], on a small set of training images; 2) then training the found sparse subnetwork, which is a more data-efficient solution in comparison to the dense one, on the same small training set.  In addition, the paper proposes a feature augmentation technique, which injects adversarial perturbations into intermediate features of both discriminator and generator. The paper shows that the sparse subnetwork can benefit from extra data- and feature-level augmentations.
main_review: Strengths:

-	The paper addresses a challenging and important problem of training GANs in low-data regimes. The extension of the lottery ticket hypothesis to training GANs in low data regimes is a novel and interesting idea.
-	The proposed feature augmentation helps to improve synthesis performance in both high and low data regimes for dense and sparse GAN networks. It’s also complementary to data augmentation techniques, such as ADA [3] or DiffAug[4].
-	The paper conducts experimental evaluation using different GAN architectures (BigGAN, SNGAN, StyleGAN-v2) across different datasets, showing the superior performance of sparse networks in comparison to their dense baselines in low data regimes.

Weaknesses:

-	It would be good to see the efficiency comparison between different sparse and dense networks during training and inference, e.g. what is the throughput of the networks? Number of parameters? Who long does it take to train the sparse subnetwork in comparison to its dense baseline?
-	In low-data regimes the difference between using the dense network and its sparse subnetwork in terms of FID is quite minor, ~1 FID point, e.g 13.59 vs. 13.05 FID on CIFAR100 in Table 4. It’s not obvious what is the benefit of the proposed approach as the performance improvement is marginal and finding lottery tickets requires costly train-prune-retrain process. Thus the paper would benefit from the efficiency analysis (see the point above).
-	It’s not clear form the paper if in low data regime a smaller network would perform better in comparison to the proposed method. A simple comparison of the found sparse subnetwork to the dense baseline with the reduced number of parameter (e.g. number of channels), equivalent to the number of parameters of the sparse subnetwork, would be interesting to see. It might happen that with little amount of training data a smaller network (in terms of parameters) will help to overcome overfitting and might perform even better.
-	The paper misses comparison with the related work on few-shot generation task: Liu et al., “Towards faster and stabilized gan training for high- fidelity few-shot image synthesis”, ICLR, 2021. This work reaches superior performance in comparison to the proposed approach in terms of FID, e.g. on Obama dataset 52.86 FID in Table 5 vs. 41.05 FID in Liu et al., on Panda 14.75 vs. 10.03 FID, on Animal Face – Cat 47.40 vs. 35.11 FID. Note that both approaches do not use any pre-training, thus the comparison is fair.

Typos: Line 164: Figure 2 -> Figure 1, Table 3 “10.14” should not be in bold.

limitations_and_societal_impact: The paper briefly discusses limitations and potential negative societal impact of the proposed approach.
review_title: null

Meta-review: This paper adopts the idea of pruning and data augmentation to improve the data efficiency of GANs. The rebuttal solves the reviewer's initial concerns about novelty and overfitting, and this paper is recommended for acceptance. 
Decision: Accept (Poster)