Data-Efficient GAN Training Beyond (Just) Augmentations: A Lottery Ticket Perspective
Authors: Tianlong Chen
Yu Cheng
Zhe Gan
Jingjing Liu
Zhangyang Wang
Venue: NeurIPS 2021 Poster (2021)
Keywords: Data-Efficient GAN Training
The Lottery Tickets Hypothesis

Abstract: Training generative adversarial networks (GANs) with limited real image data generally results in deteriorated performance and collapsed models. To conquer this challenge, we are inspired by the latest observation, that one can discover independently trainable and highly sparse subnetworks (a.k.a., lottery tickets) from GANs. Treating this as an inductive prior, we suggest a brand-new angle towards data-efficient GAN training: by first identifying the lottery ticket from the original GAN using the small training set of real images; and then focusing on training that sparse subnetwork by re-using the same set. We find our coordinated framework to offer orthogonal gains to existing real image data augmentation methods, and we additionally present a new feature-level augmentation that can be applied together with them. Comprehensive experiments endorse the effectiveness of our proposed framework, across various GAN architectures (SNGAN, BigGAN, and StyleGAN-V2) and diverse datasets (CIFAR-10, CIFAR-100, Tiny-ImageNet, ImageNet, and multiple few-shot generation datasets). Codes are available at: https://github.com/VITA-Group/Ultra-Data-Efficient-GAN-Training.

Data-Effciient GAN Training Beyond (Just) Augmentations: A Lottery Ticket Perspective
Tianlong Chen', Yu Cheng", Zhe JannJjigginng Liu Zhangyang Wang' University of Texas at Austin, Miirosooft Corporation, TTinghua University (t (tianlong. chen, atlaswang) @utexas edu, lyu. cheng, zhe, tthe.gan gan)mmicrosoot.ccmmmm com JJLiu0air. wwwuuaiir.tiinuuaa. edu.
Abstract
Training generative adversarial networks (GANs) with limited real image data generally results in deteriorated performance and collapsed models. To conquer this challenge, we are inspired by the latest observation, that one can discover independently trainable and highly sparse subnetworks (a.k.a., lottery tickets) from GANs. Treating this as an inductive prior, we suggest a brand-new angle towards data-efficient GAN training: by first identifying the lottery ticket from the original GAN using the small training set of real images; and then focusing on training that sparse subnetwork by re-using the same We find coordinated framework offer orthogonal gains to existing real image data augmentation methods, and we additionally present new feature-level augmentation that can be applied together with them. Comprehensive experiments endorse the effectiveness of our proposed framework, across various GAN architectures (SNGAN, BigGAN, and StyleGAN- V2) and diverse datasets (CIFAR- 10, CIFAR-100, Tiny-ImageNer ImageNet, and multiple few-shot generation datasets). Codes are available at: https: gig..... com/VITA- Group/ UUtraa Ultra-Data- Data-Effician Efficient- GAN- TENN-Trainng.
Introduction
The quantity, diversity, and high quality of natural im- ages available in the general domain have played an essential role in the achieved breakthroughs of Gener- ative Adversarial Networks (GANs) [2-11] over the past few years. However, it could become challeng- ing even infeasible for specific application domains to collect sufficiently large-scale dataset, due to var- ious constraints the imaging expense, subject type, image quality, privacy, copyright status, and more. That prohibits GANs' broader applications in these domains, c.g., for generating synthetic training data [12]. Examples of such domains include medical im- ages, images from scientific experiments, images of rare species, photos of specific person land- mark. Eliminating the need of immense datasets for GAN training is highly demanded for those scenar- ios. Naively training GAN with scarce samples leads overconfident discriminators that overfit the small training data [13-15, 1]; it usually ends up with train- ing divergence and drastic performance degradation (evidenced later in Figure 2).
48.80%
36.00%
(40
21
Stage
67.24%
1.00%
Stage I:[l
Baseline
73.79%
58%
Figure 1: FIDs on training BigGAN on 10% train- ing data from CIFAR- 100. Smaller distance to the origin indicates smaller FID/better performance. Compared to vanilla training baseline (*,, i.e., dense model or 0% sparsity), our method's Stage ()) finds highly sparse lottery tickets from the original BigGAN, with range of sparsity up 58%. Higher sparsity appears bring better data titteefffiiency. Stage Il further boosts the training of those found sparse subnetworks, by incorporat- ing existing data-level augmentation [1] and our newly proposed feature-level augmentation (o).
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

This paper addresses the above issue from a brand new perspective by decomposing the challenging GAN training in limited data regimes into two sequential sub-problems: (i) finding independent train- able subnetworks (i.e., lottery tickets in GANs) [16, 17]; then (ii) training the located subnetworks, which we show is more data-efficient by itself, and can further benefit from aggressive augmentations (both the input data and feature levels). Either sub-pr sub-problem becomes much less data-hungry to train, and the two sub-problems re-use the same small training set of real images. Although this paper focuses on tackling the data-efficient training of GANs, such coordinated framework might potentially be generalized to training other deep models with higher data efficiency too.
Our key enabling technique is to leverage the lottery ticket hypothesis (LTH) [18]. LTH shows the feasibility to locate highly sparse subnetworks (called "winning tickets") that are capable of training in isolation to match or even outperform the performance of original unpruned models. Recently, [17, 16] revealed the existence of winning ticket in GANs (called "GAN tickets"). However, none of the existing works discuss the influence of training data size on locating and training those tickets. Our work takes one step further, and shows that one can identify the same high-quality GAN tickets even in the data-scarce regime. The found GAN tickets also serve as sparse structural prior to solve the second sub-problent with less data, while maintaining an unimpaired trainability blessed by the LTH assumption [18]. Figure I (the outer circle's 's blue dots) evidences that we can identify sparse GAN tickets that achieve superior performance than full GANs in the data-scarce scenarios.
The new lottery ticket angle complements the existing augmentation techniques [19-21], and we further show that they can be organically combined to boost performance further'. When we train the identified lottery ticket, we demonstrate its training can benefit as well from the latest data-level augmentation strategies, ADA [15] and DiffAug [1]. Furthermore, we introduce a novel feature-lee augmentation that can be applied in parallel to data-level. It injects adversarial perturbations into GANs' intermediate features to implicitly regularize both discriminator and generator. Combining the new feature-level and existing data-level augmentations in training GAN tickets leads to more stabilized training dynamics, and establishes new state-of -hhe--rsst for data-efficient GAN training.
Extensive experiments are conducted on a variety of the latest GAN architectures and datasets, which consistently validate the effectiveness of our proposal. For example, our BigGAN tickets 36.00% and 67. .47 sparsity levels reach an (FID. IS) of (23. 14, 52.98) and (70.91, 7.03), on Net 64 x 64 and ImageNet 128 x 128, with 10% and 25% training data, respectively. On CIFAR-10 10 and CIFAR- 100, for SNGAN and BigGAN tickets at 67.24% ~ 86.58 sparsity, our results with only 10% training data can even surpass their dense counterparts. Impressively, our method can generate high-quality images on par with other GAN transfer learning approaches, by training on as few as 100 real samples and without using any pre-training.
2 Related Work
GANs and Data-Efficient GAN Training. GANs [22] have gained popularity in diverse computer vision scenarios. To stabilize GAN training and improve the visual fidelity and diversity of generated images, extensive studies have been conducted, such as sophisticated network architectures [23, 8, 24- 26], improved training recipes [27, 4, 28, 29], and more stable objectives [30-35]. [36, 37] utilize semi- and self-supervised learning to pursue label efficiency in GAN training.
Recently, how to train GANs without sufficient real images in the target domain sparkles new interests. There have been efforts on adapting a pre-trained GAN generator, including BSA [38], AdaFM [39], Elastic Weight Consolidation [40], and Few-Shot GAN [41-43]. However, those methods assume a large, related source domain as pre-training, based on which they further alleviate target domain data limitation by only tuning small subsets of weights. They are hence in a completely different track from our "stand-alone" data-efficiem training goal where no pre- pre-training is leveraged. [44, 45] select cote-sets of training data to speed up GAN training. A few recent attempts [1, 15] leverage differentiable or adaptive data augmentations to significantly improve GAN training in limited data regimes. Lately, [46] investigates a regularization approach, on constraining the distance between the current prediction of the real image and moving average variable that tracks the historical predictions of the generated image, that complements the data augmentation methods.
Lottery Ticket Hypothesis and GAN Tickets. [18] claims the existence of independently trainable sparse subnetworks that can match or even surpass the performance of dense networks. [47, 48] scale
We also tried to add augmentations in the lottery ticket finding stage, but did not observe visible impact.

---

up LTH by rewinding [49, 50J. uollow-up researches evidence LTH across broad fields, including visual recognition [18, 47, 51-601. natural language processing [48, 61, 50, 62-64], graph neural network [65], and reinforcement learning [61].
Recently, LTH has been extended GANs by [16, 17], who validated the existence of winning tickets in the min-max game beyond minimization. Compared with the aforementioned work, our work is the first study LTH in the data-scarce regime (for GANs, and in general). Besides finding highly compact yet same capable subnetworks, our work reveals LTH's power in saving training data an appealing perspective never being examined before.
Adversarial Training and Augmentations. Deep neural networks suffer from severe performance degradation [66, 67] when facing adversarial inputs [67-69]. To address this notorious vulnerability, various defense mechanisms [70-79] have been proposed. Among others, adversarial training- -baeed approaches achieve superior adversarial robustness [67-69], although at the price of sacrificing benign generalization [80, 70-75].
Several recent works investigate enhancing model (benign) generalization ability with adversarial training [81-86]. They adopt adversarially perturbed input images, embeddings, or intermediate features, into model training to ameliorate performance on the clean test sets. Specifically, the damaging effects of adversarial training could be controlled by extra batch normalization [81] or so. Different from those minimization problems that previous work has focused on, the two-player GAN optimization is more challenging. Generally, the adversarial competition between two players poses impediments to exploit extra adversarial information during training GANs.
3 Methodology
3.1 Revisiting GANs and the Overfitting Challenge
Generative adversarial networks (GANs) are dedicated to modeling the target distribution with the two-player game formulation of generator G and discriminator DD Specifically, the generator g takes a random sampled latent vector z (e.g., from a Gaussian distribution) as input and outputs the fake sample ((z). The discriminator 'D aims to distinguish generated fake samples G(=) from real samples æ. Alternative optimizations for the discriminator's 's loss Lp and the generator's 's loss Lg are adopted in the standard GAN training, which can be depicted as follows:
LD == (awpanaa( (f((=(())) + E (e
Cg == Fo
where loss functions fr(x), fa(()) have multiple choices, e.g., the non-saturating loss [3] with fD) fa( log(1 + e,), and the hinge loss [23] with fz(x)) max(0, 1 +xx and (x( Pasta x) and p(z) represent the data distribution of real samples and latent vectors. Lp is maximized to update D's parameters & (i.e., D(-) D(: o)), and Lg minimized update G's parameters
(i.e., G(-) = G(, ()).
20%
100%
Algorithm 1 Data-Efficient Iterative Magnitude Pruning Procedures
25.55
1800
15000
1: Input: Initial two masks my = 1l@o|la and
ma = 1llilo: Initialization weights 0g and do
2: Output: (9(-,00 @ mg), D(,, do 2 ma)}
3: repeat
26.6 1800
1500 2000c 2000
4: Training (G(-, Op @ my), 3 D(- do @ ma)) for epochs with limited training data
5: Pruning p = 20% of remaining weights in
Figure 2: The performance of SNGAN heavily de- grades with limited amount training data. Top two fig- ures show that training with 10% of CIFAR-10 data in- curs Fréchet Inception Distance (FID) explosion and In- ception Score (IS) drop. with the model (blue curves) collapsed. Bottom two figures present D's training and validation accuracies of correctly predicting generated images as fake samples.
Thenerg Saralces
both 9 and D
6: Updating the binary masks my and ma accordingly
7: Rewinding weights of G. D to Og and én
8: until masks reach the desired sparsity level
3

---

Training Failures of GANs under Limited Data. [15, 1] observe that GANs' performance has severely deteriorated when only limited training data is available. The discriminator tends to memorize and heavily overfit the small training samples, leaving a gap between the real sample's s and the generated sample's s distribution, As shown Figure 2, with only 10% CIFAR-I0 data available for training SNGAN [23], the training and validation accuracies of the discriminator 'D quickly saturate to nearly 100% (ideally close to 50%), which indicates D to become over-confident in distinguishing real and generated samples. It demonstrates that D simply memorizes the training data, and such overfitting leads to training collapses and deteriorated quality of generated images.
To address this dilemma, we suggest a new data-efficient GAN training workflow, decomposed into two stages: (i) finding winning tickets in CANs via Algorithm 1; then (ii) training the found GAN tickets, potentially with both data- and feature-level augmentations, via Algorithm 2. Blessed by LTH, the located GAN ticket shows improved generalization ability, and is further enhanced by augmentations that prevent 'D from becoming too confident.
3.2 Data-Efficient Lottery Ticket Finding from GANs
In this section, we provide the preliminaries and setups identifying data-efficient GAN tickets.
Subnetworks and winning tickets. A subnetwork of GAN is defined as {g(-, @ my): D(, e O m.)). where my {0, 1|11 and ma E {0, 11 are binary masks for the generator and discriminator respectively, and @ is the element-wise product. Let 8g and de be the initializa- tion weights of GANs. Following [18, 16], we define winning tickets of GAN subnetworks (G(,, 0o @ mg), D(, do @ m.)), that reach a matched or better performance compared to unpruned GANs when trained in isolation with similar training iterations.
Finding data-efficient winning tickets in GANs. To our best knowledge, we are the first to extend LTH to the limited data regimes. In this challenging scenario, only a small amount of training data accessible for the finding and training of GAN tickets. We unstructured magnitude pruning [87], e.g., Iterative Magnitude Pruning (IMP), to establish the sparse masks my and ma.
As shown in Algorithm 1, we first train the full GAN model for t epochs with limited training samples (e.g.. 100-shot), and then perform IMP to globally prune the weights with the lowest magnitude. Zero elements in the obtained masks my and m. index the pruned weights. Before repeating the process again, the weights of the sparse generator (-, 0 @ m2)) and discriminator D(,, d O ma) are rewound to the same initialization On and do. following the convention [18]. The pruning ratio p controls the portion of weights removed per round, and we fix p = 20% in all experiments.
Intuitively, identifying a special sparse mask (without requiring to train its weights well) should be an casier and hence more data-efficient task compared training the full network weights. That was verified by our observations in experiments too: when the training data volume reduces from 100% to 10% of the full training set, the quality of sparse mask remains to be stable, since achieves matched performance compared to its dense counterpart in both full and limited data re-training regimes.
3.3 Data-Level and Feature-level Augmentations for Training GAN Tickets
After locating the GAN ticket at certain sparsity, training it using the vanilla recipe could already attain significantly improved IS and FID compared to the full dense model trained in the same data-limited regime: see Figure outer circle for example.
Next, we discuss how our proposal can be applied together with augmentation- based approaches, for enhanced training of our found GAN tickets. Our natural choices include to plug-in the two recent state-of-the-art data augmentations, i.e., DiffAug [1] and ADA [15]. We further present a new adversarial feature-level augmentation (AdvAug), that can be jointly applied together with deta-level augmentations to gain an additional performance boost.
Revisiting adversarial training. Let (æ, y) denote the input image and its label. f(), x, y) is the loss function parameterized by d. Adversarial training [69] can be formulated as follows:
max f(0, x: + 5. 5. |611,
min E. v)
(I)
where 8 is crafted adversarial perturbation constrained within the lp norm ball that is centered at with radius e. 8 can be reliably generated by multi-step projected gradient descent (PGD) [69].

---

Updato
Update
(1(e)
(e(C)
LD
Co
D,(s)
P.
9/
(2(91(9) +5)
P.(0la)) +
Cadv
O(
Figure 3:
The pipeline of AdvAug for GANs. Left: Updating the discriminator D; Right: Updating the generator G. Purple arrows denote the path to generate adversarial feature perturbations.
Different from the above standard adversarial training, which adds perturbation on the image pixel space, AdvAug injects adversarial perturbations to intermediate feature embeddings of both & and D. A similar feature augmentation idea was proven to be helpful in NLP [82], and computer vision [86]. showing effectiveness to regularize the smoothness model's generalization. The AdvAug scheme is illustrated in Figure 3, and can be mathematically
the training landscape and enhance the trained
depicted as follows (A1 is a controlling hyperparameter).
min Lgg + X1 Lg 8.1. Ladv == max F (()))) +))]-- 8
(2)
Se
We choose A 1 in all experiments for simplicity. g = G2 o 91 denotes the generator, and between the Gi and G2 parts we inject AdvAug. Adversarial perturbations å generated by PGD [69], are applied to the intermediate feature space 9i (z). The details our feature-level augmentation on discriminator D is included in Appendix A1. The full algorithm of training GAN with both data- and feature-level augmentations is summarized in Algorithm
Algorithm 2 Training (Sparse) GAN with Data- and Feature-level Augmentations
Input: GAN (g(. fo), D(., (o Inputs æ and z Output: Trained GAN (9(: Or), (r)r
for f 1 to T do
# Training discriminator with data and feature augmentations
Augment input with DiffAug [1] or ADA [15]
Feed æ and (=)) to D
Generate ad adversarial augmented features in D (Equation. 5)
Update the discriminator ((-, Φi) (Equation.
# Training generator with data and feature augmentations
Sample and augment with DiffAug [1] or ADA [15]
Feed z to G. Generate adversarial augmented features in G (Equation. 3)
Update the discriminator ((: 0.) (Equation. 4)
end for
Note that AdvAug only affects the generated images through G intermediate features, and the classifier learning through D features. hence avoids to directly manipulate the real data distribution. One bonus of doing so is that it is potentially better alleviating the distribution leaking issue [15], i.e., GANs learn to mimic and generate the augmented distribution rather than the real one.
4 Experiments
In this section, we conduct comprehensive experiments on Tiny-ImageNet [88], ImageNet [89]. CIFAR-10 [90], and CIFAR- based on the unconditional SNGAN [23] and StyleGAN-V2 [6].
as well as the class-conditional BigGAN [2]. We adopt the common evaluation metrics, including Fréchet Inception Distance (FID) [91] and Inception Score (IS) [34]. Note that a smaller FID (L) and larger IS (1) indicate better performing GAN models. Furthermore,
evaluate our proposed method on few-shot generation both with and without pre-training in Section 4.3. Extensive ablation studies analyze effectiveness of each component in Section 4.4.
Implementation and Baseline Details. We follow the popular StudioGAN codebase [92], which contains high-quality re-inplementation of BigGAN and SNGAN on ImageNet and CIFAR. For example, our implemented BigGAN baseline performs much better, i.e., FID: 26.44 (ours) v.s. 39.78 (reported) on CIFAR-10, and FID: 36.58 (ours) v.s. 66.71 (reported) on CIFAR-100, than the recent reported baselines in [1], under 10% training data regimes, For detailed configuration, BigGAN takes
5

---

learning rates of (4, x 10 for G. of 2, x 10 for D, batch sizes of {256. 256, 64). 1 x 10th training iterations, and {1, 5) D steps per 9 step on ITane-,m,,,Ne,, ImageNet, CIFAR) datasets. SNGAN uses learning rates of 2 x 10 for 9 and D, batch sizes of 64, 5 x 10a training iterations, and five D steps per G step on CIFAR. For StyleGAN-V2 experiments, we use its popular PyTorch implementation", and keep the default configuration in [1] including image resolution (256 x 256), learning rates for DIG (2 X 10 10-4), batch size (5), and training iterations (1 x 10
Note that, same as the setting in [1], training iterations will be doubled when training GANs with DiffAug. We use implementations in the StudioGAN codebase for DiffAug [1], and the official implementation" for ADA [15]. AdvAug with PGD-1 and step size 0.01/0.001 is applied on CIFAR/(Tiny-)Immn ImageeNe datasets, which are tuned by a grid search in Section 4.4. All GANs are trained with 8 pieces of NVIDIA V100 32GB.
4.1 On the Effectiveness of Training with Winning Ticket and AdvAug
Table I: Timy-ImageNet 64 64 performance without the truncation trick [2]. FID and IS are measured using 10K samples; the official validation set is utilized as the reference distribution. BigGANs at 0.00% (full unpruned models), 36.00%, 67. 67.21% sparsity are found and trained with 100%, 20%, 10% data, respectively.
100% training data (full set)
20% training data
10% training data
Methods
FID (L)
IS (1)
FID (4)
IS (1)
FID (
IS (t)
BigGAN (0.00%)
21.54 0.03
18.33 + 0.15
59.77 0.05
7.81 0.20
84.53 0.08
5.45 0.23
AdvAug
21.07 + 0.03
18.92 + 0.09
58.55 0.05
8.46 0.19
81.72 + 0.05
6.32 1 0.18
BigGAN (36.00%) AdvAug
20.54 + 0.05
18.42 + 0.20
59.56 + 0.04
7.98 = 0.20
75.76 + 0.08
6.49 + 0.21
20.02 1 0.04
19.15 1 0.18
58.24 1 0.05
8.55 = 0.20
71.47 1 0.07
6.86 1 0.20
BigGAN (67.24%)
26.37 1: 0.03
16.38 1 0.15
59.02 0.03
8.17 0.18
73.23 1: 0.05
6.68 + 0.15
AAdvAug
25.59 + 0.03
17.62 + 0.16
57.60 0.04
8.94 0.18
70.91 + 0.05
7.03 + 0.16
We adopt the top-performing model BigGAN [2], and report experiments on both Tmay-It at 64 x 64 resolution and ImageNet at 128 128 resolution. We evaluate our proposal on Tiny- ImageNet with 10%, 20%, 100% data available, and ImageNet with 25% data available, with results summarized in Tables and 2, respectively. All our results are averaged over three independent evaluation runs (same hereinafter), and the best performance of cach column are highlighted.
The following observations can be drawn: First, sparse BigGAN tickets can achieve consistently improved performance over the full model (0.00%). Especially, with only 10% training data available, BigGAN tickets at 67. 27.24% sparsity obtain massive gains of 11.30 FID and 1.58 IS on Imag-Em Second, feature-level augmentation (AdvAug) consistently improves all training cases, from dense to sparse, and from full data to limited data. In particular, larger sparsity (e.g.. 67.24%) with less training data available (e.g.. 10%) tend to benefit more from applying AdvAug, which is aligned with our design principle. Third, while the full data regime (100% data) does not necessarily prefer the highest sparsity (moderate sparsity still benefits), the limited data regimes (10% or 20% data) see monotonically increasing gains the ticket sparsity goes higher. That is understandable since the former may need more model capacity to absorb full training data, while the latter case hinges on sparsity avoid overfitting their limited training data.
Table 2: ImageNet 128 x 128 performance without
Besides, we report another group of experiments of training SNGAN on CI using from 10% to 90% training data. The results are summa- rized in Figure 4. The conclusions we can draw are highly consistent with the above BigGAN case: (1) at the same data availability (from 10% to even 90%), training a sparse ticket is always preferred over training the dense model; (2) AdvAug is also consistently helpful in all cases; (3) for both spar- sity and AdvAug, they can contribute to larger gains when training data gets smaller.
the truncation trick [2]. FID and IS are measured using 50K samples; the validation set is utilized as the reference distribution. BigGANs at 0.00% and 36.00% sparsity levels are adopted, and only 25% training data are available in all training stages.
25% training data
Methods
FID (L)
IS (*)
BigGAN (0.00%
25.37 + 0.07
46.50 1: 0.40
+ AdvAug
23.95 + 0.06
47.95 + 0.32
BigGAN (36.00%)
24.03 + 0.08
50,07 1: 0.51
AdvAug
23.14 + 0.07
52.98 + 0.47
httpss//ggthhbbo/////ceiei////// com//uciidrrainn/SyyyeGAAAA V2- 12-pytorch. Note that the PyTorch version remains with a small performance gap compared to the TensorFlow implementation in [1]. http://fftbbb.oo//o////////ya..a
pytorch. The official Pytorch implementation in [15].

---

10% Training Data
20% Training Duta
30% Training Data
50% Training Data
70% Training Datt
90% Training Data
IS (1
7.0
7.07
7,3
781
7,4
1,89
74
SJK
8.17
6.12
Training
Training
30% Training
80% Training
Training Data
Training
41.43
76-13
SEER ILSS
19A8
12.17 27.0
1304 B4SE
1561
13.19 13.57 1581
Done
Dana Ane
Spans Tokra
Shate Te3ck Avo
Danee
Prese Axp
Stare Frass Alg
Tokets
Dane
Desse Alp
Stane Taich Avg
Dark
Spant Tadre
Spani 2eve
Daowe
Datse App
Spans
Doner
Briee Exbeer
Dane
Figure 4: IS (1) and FID (4) results of SNGAN with 10%, 20%, 30%, 50% 70%, 90% training data of CIFAR-10. Four settings are (valuated:()) Dense (unpruned SNGAN), (ii) Dense+Aug (we only apply AdvAug here), (iii) Sparse Tickets (pruned SNGAN), (iv) Sparse Tuc, where the top performing variants are highlighted with black boxes. SNGAN tickets with 20%, 36%, 36%, 67%, 49%, 36% sparsity levels are adopted accordingly. IS and FID measured using 10K samples; the validation set is utilized the reference.
Table 3: CIFAR-10 and CIFAR-100 results. FID (L) are measured using 10K samples; the validation set utilized as the reference distribution. Full dense models and sparse winning tickets of BigGAN are reported with 100%, 20%. 10%. respectively. Specifically, BigGAN tickets with 67 67.24% and 86.58 sparsity levels reported for the 100%, 20% and 10% training data regimes. Performance reported is averaged over three independent evaluation runs; all standard deviation less than 1%.
CIFAR-10
CIFAR-100
Methods
100% data
20% data
10% data
100% data
20% data
10% data
Dense BigGAN DiffAug [1]
8.57
17.38
26.44
11.83
22.13
36.58
8.09
13.04
17.40
10.60
18.32
25.69
DiffAAu AdvAug
7.70
12.19
14.40
8.96
17.94
23.94
Sparse BigGAN Tickets
8.26
16.03
25.41
11.73
21.05
30.96
DDiffAug
8.19
12.83
16.74
10.73
17.43
23.80
+ DDiffAug + AdvAug
8.15
12.02
14.38
10.14
17.19
22.37
4.2
Incorporating Our Proposal with Latest Data Augmentations and Regularization
Combining DiffAug- We first incorporate Dif- fAug [1], as representative of the latest data-level augmentation, into our proposal and show the com- plementary gains. We conduct experiments on the class-conditional BigGAN and unconditional SNGAN models with CIFAR-10 10 and CIFAR- 100, For BigGAN, we utilize 100%, 20%, 10% data locate GAN tickets; then we train them with data- level DiffAug. or with both DiffAug and feature-
Table 4: CIFAR-100 results with only 10% training data available. FID (L) of three evaluation measured using 10K samples; the validation set uti- lized as the reference distribution. Sparse StyleGAN- V2 tickets at 48.80% sparsity are adopted.
Methods
10% training data
Dense StyleGAN-V2
13.59 + 0.06
DDiffAug [1]
12.90 = 0.04
+ADA [15]
12.87 = 0.03
level AdvAug, as shown in Table 3. Consistent observations can be drawn: First, similar to our
+ADA + RLO [46]
13.01 = 0.02
Sparse StyleGAN-V2 Tickets
13.05 + 0.07
previous observations on AdvAug, DiffAug also
DiffAug [1]
12.53 = 0.03
shows to contribute more when the training data becomes more limited; Second, combining Dif- fAug and AdvAug improves over either alone,
ADA [15]
12.20 = 0.03
ADA + RLC [46]
12.48 0.04
ADA [15] AdvAug
12.11 = 0.05
and leads to the best results across all cases.
Combining Other Data Augmentations and Regularization. study to other recent data augmentation and regularization approaches, e.g., ADA [15] and RLC [46].
We then extend our combination
7

---

Experiments are conducted on CIFAR-I00 with StyleGAN-V2 backbone, and results are collected in Table 4. We observe that plugging in either ADA [15] or DiffAug [1] into our framework could improve sparse GAN winning tickets, and the gain is also enlarged when ADA is combined with AdvAug. Regard REC", " is less effective combined with other augmentations.
Taking above together, it has been clearly shown that our proposal is orthogonal to those existing efforts and is of independent merit. Moreover, combing them would lead more powerful pipelines for data-efficient GAN training.
4.3 Few-Shot Generation
It is laborious, and sometimes impossible collect large-scale dataset for certain images of interest. To tackle the few-shot image generation problem, [93] utilizes pre-training from external large-scale datasets and performs fine-tuning under limited data scenarios; [94], [38] and [95] partially fine-tune the GANs with part of the GAN model being frozen.
We compare these transfer learning approaches' with our data-efficient training scheme. Differently from them, ours is training from scratch and is free of any pre-training, while all transfer learning methods start from pre-trained StyleGAN-V2 V2 model the FFHQ face dataset [5].
Our comparison experiments are conducted using StyleGAN- V2 on the AnimalFace [96] dataset (160 cats and 389 dogs), and the 100-shot Obama, Grumpy Cat, and Panda datasets provided by [1]. As shown in Table 5, our method finds data-efficient GAN tickets at 48. 80% sparsity levels, that can be trained with only 100 training samples from scratch (without any pre-training) and show competitive performance to other transfer learning algorithms. Visualizations of style space interpolation and few-shot generation are provided in Figure 5 and 6.
Table 5: Few-shot generation. Following the setting in [1], we calculate the FID with 5K samples and the training dataset is adopted as the reference distribution. All transfer learning methods have their pre-trainings from FFHQ [5]. StyleGAN-V2 tickets 48.80% sparsity level are found and used in our method.
100-shot by [1]
AnimalFace
Methods
Pre-training?
Obama
Grumpy Cat
Panda
Cat
Dog
Scale/shift [38]
Yes
50.72
34.20
21.38
54.83
83.04
MineGAN [95]
Yes
50.63
35.54
14.84
54.45
93.03
TransferGAN [93]
Yes
48.73
34.06
23.20
52.61
82.38
FreezeD [94]
Yes
41.87
31.22
17.95
47.70
70.46
StyleGAN-V2 (0.00%)
No
89.18
61.97
90.96
95.75
164.54
+ +DiffAug AdvAug
No
54.11
35.46
15.94
54.02
72.47
StyleGAN-V2 Tickets (48 (48.80%)
No
73.92
56.81
82.45
85,92
153.90
+ DDiffAug AdvAug
No
52.86
31.02
14.75
47.40
68.28
Figure 5: Style interpolation visualizations of StyleGAN-V2 V2 tickets 48,80% with AdvAug only 100-shot Obama, Grumpy Cat, Panda, and AnimalFace datasets, respectively.
Figure 6: Few-shot generalization results of StyleGAN-V2 tickets (18.80%) with AdvAug only on 100-shot Obama, Grumpy Cat, Panda, and AnimalFace datasets, respectively.
'[46] advocates the hest-perfrmingg configuration Implementations from the codebase of [94].
ADA+RLC, with FID 13.01 on 10% data CIFAR- 1000
8

---

p.20%
Budike
OMP
Ncvokk Stezke
41
Strdes Freic
1.43
44
78
37
4.129
4
Figure 7: Performance of training GAN with 10% data of CIFAR-10. Remaining weight indicates the sparsity levels of identified GAN tickets. Left: FID of the data-efficient GAN tickets found by IMP with pruning ratios o = 10%, 20%, 40%. Middle: FID of the data-efficient GAN tickets trained with different settings of Adv Aug, including baseline without AdvAug, AdvAug on D or G only, and AdvAug on both & and D. Right: FID of trained SNGAN tickets found by IMP, Random Pruning, OMP, and Network Slimming [97].
0.1
2
0.05
3
.5
45
50
65
40
0
3
49
1
3
35
45
0.01
5
0.001
4
4 (d) Layer index in g
(a) Step size
(b) The number of steps
(c) Layer index in D
Figure 8: Ablation study on the location and strength of introducing AdvAug to data- GAN training. The step size and the numher of steps roughly indicate the strength of generated adversarial perturbations, e.g. a smaller step size or fewer steps for PGD means less aggressive perturbations [69]. FID (L) is reported.
4.4 Ablation and Analysis
Pruning Ratio p in the Ticket Finding. To understand the effect of the pruning ratio in IMP to the quality of data-efficient GAN tickets, we experiment on SNGAN with 10% data of CIFAR-10 10 and p 10,, 20%, 40% as the pruning ratio. As shown in Figure (Left), all three IMP settings find data-efficient winning tickets in GAN; IMP with a lower pruning ratio tends to identify higher-quality GAN tickets in terms of FID, while usually costs much more to reach the same level of sparsity as higher pruning ratios do.
Augment G or D: Either or Both. We apply AdvAug on G or D only, and AdvAug on both G and D. Only 10% data of CIFAR- 10 are available for finding GAN tickets and training with AdvAug. Results are summarized in Figure 7 (Middle). Either employing AdvAug on D G consistently obtains significant performance improvements (i.e., largely reducing the FID) over all sparsity levels, and augmenting both D and G further enhance the found data-efficient GAN tickets. Our results also show that sparser GAN tickets can benefit more from AdvAug, such as subnetworks with 20%, 36%, 33.22% sparsity.
Strength and Locations of Injecting AdvAug. To better interpret the influence of the strength and layer locations of injected adversarial feature perturbations, we comprehensively examine SNGAN on 10% training data of CIFAR-10 across different step sizes, the number of PGD steps, and locations (i.e., where to apply AdvAug). When studying one of the factors, we fix the other factors with the best setup. From Figure 8, several observations can be drawn:
Figure 8 (a) and (b) show that adopting AdvAug with step size 0.01/0.001 and PGD-1/3
assists the data-efficient GAN training, while Adv Aug with step size 0.05/0.1 1 and PGD-5/7 perform worse than the baseline without AdvAug (i.e., the setting with zero step size or zero step PGD in Figure 8). It reveals that overly strong AdvAug can hurt performance.
As shown in Figure 8 (c) and (d), augmenting the last layer of the discriminator DD and the first layer (i.e., the closest layer to the latent input vector) the generator G appears to be the best configuration for utilizing AdvAug. It seems that injecting adversarial perturbations into the "high-level" feature embeddings in general benefits more to mitigate the overfitting regimes.
issue in

---

In summary, we observe that applying AdvAug to the last layer of D and the first layer of G, with PGD-1 and step size 0.01, seems to be sweet-point configuration for data-efficient GAN training. which is hence adopted as our default setting.
Comparison with Baselines. Naive baselines, i.e.,
Table 6: Performance of SNGAN models aug- mented by Gaussian Noise or AdvAug on 100% and 10% training data.
random pruning, one-shot magnitude pruning (OMP) [87, 16], network slimming (NS [97], and random noise augmentation, are evaluated in Figure 7 (Right) and Table 6. Compared to random pruning, OMP and NS, IMP produces much better GAN tickets, especially at high sparsity levels (e.g.. 2 48.8%). Compared to augmenting features with random noise sampled from (0, 0.012), AdvAug also achieves larger performance gains on both 100% and 10% training data regimes.
Methods
1008 training data
10%
training data
FID
18
Baseline
8.29
15,69
5.24
44.22
Gaussian Noise
8.30
14.52
5.53
44.86
AdvAng
8.42
13.99
6.10
41.25
5 Conclusion and Discussion of Broader Impact
We introduce a novel perspective for data-efficient GAN training by leveraging lottery tickets, which augmentations can further enhance, including our newly introduced feature-level augmentation. Comprehensive experiments consistently demonstrate the effectiveness of our proposal, on diverse GAN architectures, objectives, and datasets. Note that although finding lottery tickets requires costly vrin--ruue-retrrin process, only data efficiency is of interest in this work. An intriguing future work would be to pursue data and resource efficiency (training and inference) together.
This research aims enhance GAN training in the limited data regimes. However, might amplify the existing societal risk of applying GANs. For example, the issue of image generation bias may be impacted or even amplified by the sparse structures, which we will verify future work. The data-efficient generation ability might also be leveraged by undesired applications such as DeepFake.
References
[1] Shengyu Zhao, Zhijian Liu, Ji Lin, Jun-Yan Yan Zhu, and Song Han. Differentiable augmentation for data-efficient gan training. In NeurlPS, 2020.
[2] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv: 1096, 2018
[3] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NeurIPS, 2014.
[4] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. arXiv preprint
2017.
[5] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In CVPR, 2019.
[6] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In CVPR, 2020.
[7] Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral Normaliza- tion for Generative Adversarial Networks. In ICLR, 2018.
[8] Takeru Miyato and Masanori Koyama. egans with projection discriminator. In ICLR, 2018.
[9] Orest Kupyn, Tetiana Martyniuk, Junru Wu, and Zhangyang Wang. Deblurgan-v2: Deblur- ring (fnders-of-magnitudee faster and better. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8878-8887, 2019.
[10] Xinyu Gong, Shiyu Chang, Yifan Jiang, and Zhangyang Wang. Autogan: Neural architecture search for generative adversarial networks. In Proceedings of the IEEE International Conference on Computer Vision, 2019.
10

---

[11] Yifan Jiang, Xinyu Gong, Ding Liu, Yu Cheng, Chen Fang, Xiaohui Shen, Jianchao Yang, Pan Zhou, and Zhangyang Wang. Enlightengan: Deep light enhancement without paired supervision. IEEE Transactions on Image Processing, 30:2340- 2349, 2021.
[12] Ashish Shrivastava, Tomas Pfister, Oncel Tuzel, Josh Susskind, Wenda Wang, and Russ Webb. Learning from simulated and unsupervised images through adversarial training. In CVPR, 2017.
[13] Martin Arjovsky and Léon Bottou. Towards principled methods for training generative adver- sarial networks. arXiv preprint
2017.
[14] Dan Zhang and Anna Khoreva. Pa-gan: Improving gan training by progressive augmentation. 2018.
[15] Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. arXiv preprint arXiv:2006.06670 2020.
[16] Xuxi Chen, Zhenyu Zhang, Yongduo Sui, and Tianlong Chen. Gans can play lottery tickets too. In ICLR, 2021.
[17] N. M. Kalibhat, Y. Balaji, and S. Feizi. Winning lottery tickets in deep generative models. arXiv preprint 20Xv:20000002350, 2020.
[18] Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In ICLR, 2019.
[19] Agoc-Trung Tran, Viet-Hung Tran, Ngoe-Bao Nguyen, Trung- Kien Nguyen, and Ngai-Man Cheung. Towards good practices for data augmentation in gan training. arXiv preprint 200x:20006.03388 2020.
[20] Han Zhang, Zizhao Zhang, Augustus Odena, and Honglak Lee. Consistency regularization for generative adversarial networks. arXiv preprint arXiv: 12027, 2019.
[21] Zhengli Zhao, Sameer Singh, Honglak Lee, Zizhao Zhang, Augustus Odena, and Han Zhang. Improved consistency regularization for gans. arXiv preprint arXiv:2002.04724, 2020.
[22] Jie Gui, Zhenan Sun, Yonggang Wen, Dacheng Tao, and Jieping Ye. A review generative adversarial networks: Algorithms, theory, and applications. arXiv preprint arXiv:2001.0697 2020.
[23] Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. arXiv preprint arXiv:1802.05955 2018.
[24] Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434 2015.
[25] Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. Self-attention generative adversarial networks. In International conference machine learning, pages 7363. PMLR, 2019.
[26] Hanting Chen, Yunhe Wang, Han Shu, Changyuan Wen, Chunjing Xu, Boxin Shi, Chao Xu, and Chang Xu. Distilling portable generative adversarial networks for image translation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 3585-3592 2020.
[27] Emily L Denton, Soumith Chintala, Rob Fergus, et al. Deep generative image models using a laplacian pyramid of adversarial networks. NeurlPS, 2015.
[28] Steven Liu, Tongzhou Wang, David Bau, Jun-Yan Zhu, and Antonio Torralba. Diverse image generation via self-conditioned gans. In CVPR, 2020.
[29] Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dimitris N Metaxas. Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks. In ICCV, 2017.
11

---

[30] Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In ICML, 2017.
[31] Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville. Improved training of wasserstein gans. In NeurIPS, 2017.
[32] Chun-Liang Li, Wei-Cheng Chang, Yu Cheng, Yiming Yang, and Barnabas Poczos. Mmd gan: Towards deeper understanding of moment matching network. In NeurlPS, volume 30, 2017.
[33] Lars Mescheder, Andreas Geiger, and Sebastian Nowozin. Which training methods for gans do actually converge? arXiv preprint arXiv: /80/v 0.040,4 2018.
[34] Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. arXiv preprint 160i, 03498, 2016.
[35] Tianyu Guo, Chang Xu, Jiajun Huang, Yunhe Wang. Boxin Shi, Chao Xu, and Dacheng Tao. On rositive-unlabeled classification in gan. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8385-8393, 2020.
[36] Ting Chen, Xiaohua Zhai, Marvin Ritter, Mario Lucic, and Neil Houlsby, Self-supervised gans via auxiliary rotation loss. In CVPR, 2019.
[37] Mario Lucic, Michael Tschannen, Marvin Ritter, Xiaohua Zhai, Olivier Bachem, and Sylvain Gelly. High-fidelity image generation with fewer labels. arXiv preprint veXiiv/903.000 2019.
[38] Atsuhiro Noguchi and Tatsuya Harada. Image generation from small datasets via batch statistics adaptation. In ICCV, 2019.
[39] Miaoyun Zhao, Yulai Cong, and Lawrence Carin. On leveraging pretrained gans for generation with limited data. In International Conference on Machine Learning, pages 1 11340-11351 PMLR, 2020.
[40] Yijun Li, Richard Zhang, Jingwan Lu, and Eli Shechtman, Few-shot image generation with elastic weight consolidation. arXiv preprint arXiv:2012.02788) 2020.
[41] Esther Robb, Wen-Sheng Chu, Abhishek Kumar, and Jia-Bin Huang. Few-shot adaptation of generative adversarial networks. arXiv preprint arXiv:2010. 19t:20000.//94,,,, 2020.
[42] Antreas Antoniou, Amos Storkey, and Harrison Edwards. Data augmentation generative adversarial networks. arXiv preprint arXiv: 1tte:11111.0400 2017.
[43] Victor Garcia and Joan Bruna. Few-shot learning with graph neural networks. arXiv preprint arXiv:1711.04043. 2017.
[44] Samarth Sinha, Han Zhang, Anirudh Goyal, Yoshua Bengio, Hugo Larochelle, and Augustus Odena. Small-gan: Speeding up gan training using core-sets. ICML, 2020.
[45] Terrance DeVries, Michal Drozdzal, and Graham W Taylor. Instance selection for gans. arXiv preprint arXiv:2007.15255 2020.
[46] Hung- Yu Tseng, Lu Jiang, Ce Liu, Ming-Hsuan Yang, and Weilong Yang. Regularizing generative adversarial networks under limited data. arXiv preprint arXiv:2104.03310. 2021.
[47] Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. Rethinking the value of network pruning. In ICLR, 2019.
[48] Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity deep neural networks. arXiv preprint
2019.
[49] Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M. Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. In ICML, 2020.
[50] Alex Renda, Jonathan Frankle, and Michael Carbin. Comparing rewinding and fine-tuning in neural network pruning. In ICLR, 2020.
12

---

[51] Chaoqi Wang, Guodong Zhang, and Roger Grosse. Picking winning tickets before training by preserving gradient flow. In ICLR, 2020.
[52] Utku Evci, Fabian Pedregosa, Aidan Gomez, and Erich Elsen. The difficulty of training sparse neural networks. arXiv preprint arXiv: 1906. 10732, 2019.
[53] Jonathan Frankle, David J. Schwab, and Ari S. Morcos. The early phase of neural network training. In ICLR, 2020.
[54] Pedro Savarese, Hugo Silva, and Michael Maire. Winning the lottery with continuous sparsifi- cation. In NeurIPS, 2020.
[55] Shihui Yin, Kyu-Hyoun Kim, Jinwook Oh, Naigang Wang, Mauricio Serrano, Jac-Sun Seo, and Jungwook Choi. The sooner the better: Investigating structure of early winning lottery tickets, 2020.
[56] Haoran You, Chaojian Li, Pengfei Xu, Yonggan Fu, Yue Wang, Xiaohan Chen, Richard G. Baraniuk, Zhangyang Wang, and Yingyan Lin. Drawing early-bird tickets: Toward more efficient training of deep networks. In ICLR, 2020.
[57] Tianlong Chen, Zhenyu Zhang, Sijia Liu, Shiyu Chang, and Zhangyang Wang. Long live the lottery: The existence of winning tickets in lifelong learning. In ICLR, 2021.
[58] Haoyu Ma, Tianlong Chen, Ting-Kuel Hu, Chenyu You, Xiaohui Xie, and Zhangyang Wang. Good students play big lottery better. arXiv, abs/2101.03255, 2021.
[59] Tianlong Chen, Jonathan Frankle, Shiyu Chang. Sijia Liu. Yang Zhang, Michael Carbin, and Zhangyang Wang. The lottery tickets hypothesis for supervised and self-supervised pre-training in computer vision models. arXiv, abs/2012.06908 2020.
[60] Zhenyu Zhang, Xuxi Chen, Tianlong Chen, and Zhangyang Wang. Efficient lottery ticket finding: Less data is more. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning. volume 139 of Proceedings of Machine Learning Research, pages 12380-12390. PMLR, 18-24 Jul 2021.
[61] Haonan Yu, Sergey Edunov, Yuandong Tian, and Ari S. Morcos. Playing the lottery with rewards and multiple languages: lottery tickets in rl and nip. In ICLR, 2020.
[62] Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Zhangyang Wang, and Michael Carbin. The lottery ticket hypothesis for pre-trained bert networks. arXiv preprins 20Xiv:0077.12223 2020.
[63] Shrey Desai, Hongyuan Zhan, and Ahmed Aly. Evaluating lottery tickets under distributional shifts. In Proceedings of the 2nd Workshop on Deep Learning Approaches for Low-Resource NLP, 2019.
[64] Xiaohan Chen, Yu Cheng, Shuohang Wang, Zhe Gan, Zhangyang Wang, and Jingjing Liu. Earlybert: Efficient bert training via early-bird lottery tickets. arXiv preprint 1:Xiv:2011.00000 2020.
[65] Tianlong Chen, Yongduo Sui, Xuxi Chen, Aston Zhang, and Zhangyang Wang. A unified lottery ticket hypothesis for graph neural networks, 2021.
[66] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfel- low, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199 2013.
[67] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversar- ial examples. arXiv preprint
2014.
[68] Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial machine learning at scale. arXiv preprint arXiv:1611.01236, 2016.
13

---

[69] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint (xXiv1706.060 2017.
[70] Hongyang Zhang. Yaodong Yu, Jiantao Jiao, Eric P Xing, Laurent El Ghaoui, and Michael I Jordan. Theoretically principled trade-off between robustness and accuracy. arXiv preprint arXiv:1901.0857,, 2019.
[71] Ludwig Schmidt, Shibani Santurkar, Dimitris Tsipras, Kunal Talwar, and Aleksander Madry. Adversarially robust generalization requires more data. In NeurIPS, 2018.
[72] Ke Sun, Zhanxing Zhu, and Zhouchen Lin. Towards understanding adversarial examples systematically: Exploring data size, task and model factors. arXiv preprint 17X1v:1902.11119, 2019.
[73] Preetum Nakkiran. Adversarial robustness may be at odds with simplicity. arXiv preprint arXiv:
2019.
[74] David Stutz, Matthias Hein, and Bernt Schiele. Disentangling adversarial robustness and generalization. In CVPR, 2019.
[75] Aditi Raghunathan, Sang Michael Xie, Fanny Yang, John C Duchi, and Percy Liang. Adversarial training can hurt generalization. In ICMLW, 2019.
[76] Ziyu Jiang, Tianlong Chen, Ting Chen, and Zhangyang Wang. Robust pre-training by adversarial contrastive learning. arXiv preprint ar arXiv:2010. 2020.
[77] Haotao Wang, Tianlong Chen, Shupeng Gui, Ting-Kuei Hu, Ji Liu, and Zhangyang Wang. Once-for-all adversarial training: In-situ tradeoff between robustness and accuracy for free. arXiv preprint arXiv:2010.11822 2020.
[78] Ting-Kuei Hu, Tianlong Chen, Haotao Wang, and Zhangyang Wang. Triple wins: Boosting accuracy, robustness and efficiency together by enabling input-adaptive inference. arXiv preprint 000:20002.0005. 2020.
[79] Tianlong Chen, Sijia Liu, Shiyu Chang, Yu Cheng, Lisa Amini, and Zhangyang Wang. Ad- versarial robustness: From self-supervised pre-training to fine-tuning In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 699-708. 2020.
[80] Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry, Robustness may be at odds with accuracy. arXiv preprint arXiv:1805.1211 2018
[81] Cihang Xie, Mingxing Tan, Boqing Gong, Jiang Wang, Alan L Yuille, and Quoc Le. Adver- sarial examples improve image recognition. In CVPR, 2020.
[82] Chen Zhu, Yu Cheng, Zhe Gan, Siqi Sun, Tom Goldstein, and Jingjing Liu. Freelb: Enhanced adversarial training for natural language understanding. In ICLR, 2020.
[83] Dilin Wang, Chengyue Gong, and Qiang Liu. Improving neural language modeling via adver- sarial training. arXiv preprint arXiv: ww...99060..... 2019.
[84] Zhe Gan, Yen-Chun Chen, Linjie Li, Chen Zhu, Yu Cheng, and Jingjing Liu. Large-scale adversarial training for vision-and-languag representation learning. In NeurIPS, 2020.
[85] Colin Wei and Tengyu Ma. Improved sample complexities for deep networks and robust. classification via an all-layer margin. arXiv preprint arXiv: 2019.
[86] Tianlong Chen, Yu Cheng, Zhe Gan, Jianfeng Wang. Lijuan Wang. Zhangyang Wang, and Jingjing Liu. Adversarial feature augmentation and normalization for visual recognition. arXiv preprint arXiv:2 /2/71, 2021.
[87] Song Han, Huizi Mao, and William Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffiman coding. In ICLR, 2016.
14

---

[88] Jiayu Wu, Qixiang Zhang, and Guoxi Xu. Tiny imagenet challenge. Technical report, Stanford University, 2017.
[89] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
[90] Alex Krizhevsky et al. Learning multiple layers of features from tiny images. 2009.
[91] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. In NeurIPS, 2017.
[92] Minguk Kang and Jaesik Park. Generation. NeurIPS, 2020.
ContraGAN: Contrastive Learning for Conditional Image
[93] Yaxing Wang, Chenshen Wu, Luis Herranz, Joost van de Weijer, Abel Gonzalez-Garcia, and Bogdan Raducanu. Transferring gans: generating images from limited data. ECCV, 2018.
[94] Sangwoo Mo, Minsu Cho, and Jinwoo Shin. Freeze discriminator: simple baseline for fine-tuning gans. arXiv preprint 2002:24002.1994,, 2020.
[95] Yaxing Wang, Abel Gonzalez-Garcia, David Berga, Luis Herranz, Fahad Shahbaz Khan, and Joost van de Weijer. Minegan: effective knowledge transfer from gans to target domains with few images. In CVPR, 2020.
[96] Zhangzhang Si and Bong-Chun Zhu. Learning hybrid image templates (hit) by information projection. TPAMI, 2011.
[97] Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learning efficient convolutional networks through network slimming. Proceedings of the IEEE International Conference on Computer Vision, pages
2017.
[98] Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. Big self-supervised models are strong semi-supervised learners. arXiv preprint arXiv:2006. 10029, 2020.
[99] Bingchen Liu, Yizhe Zhu, Kunpeng Song, and Ahmed Elgammal. Towards faster and stabilized gan training for high-fidelit fho-shot image synthesis. In International Conference on Learning Representations, 2020.
15


Reviews:
[{"review_id": "vd_mKCyU3HD", "reviewer": "Reviewer_6Tg4", "summary": "The authors propose using iterative magnitude pruning, along with two novel augmentation strategies as a means to achieve few-shot GAN training. \n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Novelty and research direction:\n\nFinding ways to train GANs data-efficiently is a very intriguing premise. Attempting to use the lottery ticket hypotheses insights into achieving this is a very interesting research direction. The work uses previous work relating to ITP on GANs, on the few-shot domain, which is a fairly novel direction.\n\n--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\nWriting quality:\n\nOverall the paper’s quality is decent but suffers from clumsy writing in various places, as well as some mild lack of conciseness and precision. \n\n--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\nSummary of review:\n\nThe idea proposed is novel and very useful, however, the experiments undertaken do not sufficiently disentangle key variables that are necessary to draw a useful conclusion. Find the full details in the details section below (the experiments section is the one with the most value). \n\nFurthermore, the writing quality of the paper is at times clumsy, as well as mildly imprecise and inconcise. \n\nI cannot recommend this paper, in its current state for acceptance. \n\n--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\n\nDetails: \n\n1. Abstract:\nThe abstract is too large, and suffers from precision and clarity issues. Furthermore, the usage of citations in an abstractly is generally not recommended, at least in ML. Ideally an abstract should be 5-8 sentences long, unless there are special reasons for not doing so.\n\n2. Introduction:\n\nThe very first sentence seems to be lacking precision since you imply that all modern advancements in GANs can be attributed to better data and nothing else. Also the usage of the verb ‘blessed’ here feels a bit out of place, but that’s just my taste. The same term is reused in line 55, which again feels out of place. \n\nThe rest of the introduction seems to suffer from clumsy writing in a few places, but overall flows well and distills down the main idea well, but fails to clearly nail down the exact contributions the authors are claiming.\n\n3. Related Work:\n\nVery good related work section, that seems to have a few places of clumsy writing, but overall seems to be on point. \n\nAlso, some notable papers relating to few-shot learning + GANs seem to be missing, i.e.  \n\n i. https://arxiv.org/abs/1711.04340\n\n ii. https://arxiv.org/abs/1711.04043\n\n4. Methodology:\n\nAdvAug description is inadequate, how is PGD applied? Optimized against what? The figure provides no information, neither does Equation 1. This is a majour point, and should not be in the appendix. I was able to figure out what was going on, but had majour trouble piecing it together without the appendix. \nThe rest of the section seems to be OK. However, the last sentence in 3.2 does not make much sense to me, since it’s overly convoluted. \n\n5. Experiments:\n\nOverall, the experiments conducted to showcase the usefulness of advAug were very interesting. \n\nThat being said, I have some reservations regarding the experiments in Table 1 and 2. Is your pruning method causing the improved performance there? Or is it the mere fact that you are using fewer weights, which happen to create a regularization effect that produces better generalizing generators? For Table 1 to be better able to showcase your point, you need to run experiments where you randomly remove 36% and 67.24% of all weights and train on all the various data settings. This way you can see if your proposed methods offer any benefit, or whether it’s all due to a smaller model with what is probably a pretty tree-esque structure, forcing bottlenecks left and right, and inadvertently causing regularization. \n\nFurthermore, the experiments conducted in Table 5 seem a bit unfair. It seems that most benefit comes from the augmentations you used. Have you tried applying those to the baseline pretrained methods? That would tell you if your pruning method is useful or not. Furthermore, again, just like in my previous point, you need to ensure that any added performance comes from your pruning method and not from the reduced model size. You can verify that by randomly dropping say 48.80% of your weights, by masking them away, and training the rest of the model to compare a random pruning vs the guided pruning you are already showcasing in the paper.\n\nFurthermore, it would have been cool to see how finding a lottery ticket GAN on the pretraining dataset, would transfer to the few-shot tasks. That could also push your model performance further. \n\nOverall, the experiments don’t do a good enough job at disentangling some important factors required to draw useful conclusions about the claims made by the authors.\n\n6. Conclusion:\n\nLine 322-324 states that the work is scientific in nature and that this somehow means that the work will not pose substantial risk of societal harm. I find this statement a bit odd. Does work that is scientific get a risk-free card? Isn’t science a very powerful tool that can both protect and destroy? I am a bit confused here. Did the authors mean that the work is far removed from a practical application that could be malicious because it is highly experimental? Please clarify.\n\n--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\n\nImproving the quality of this bar, to potentially meet my subjective NeurIPS bar\n\nThe work itself is interesting and novel. However, a few things need to be done to get this paper at a level where I’d be comfortable recommending acceptance.\n\n1. Proof read the paper, and ensure the abstract is smaller (5-8 sentences), and make clarity and precision all around a priority. I find it useful to have colleagues from other institutions/countries review the work, as they are more objective and further removed from the work, that they can easily pick up any clarity/understanding issues.\n2. For some very useful tips have a look at the below websites (which I use myself and my students all the time, and it’s really good stuff):\n\n a. https://cs.stanford.edu/people/widom/paper-writing.html \n\n b. https://www.easterbrook.ca/steve/2010/01/how-to-write-a-scientific-abstract-in-six-easy-steps/ \n\n3. Ensure your introduction has clearly outlined contributions, perhaps use a bullet list to make them stand out to the reader even more.\n4. Run experiments with randomly removed weights, to confirm that any gains come from your lottery ticket hypothesis method and not just a reduced model size and weird bottlenecks that inadvertently pop up in pruned networks.\n5. Run experiments with your proposed augmentations applied to your baseline methods, this way you can ensure that your proposed LTH method + augmentations produce the best results, or to at least learn whether your gains are all due to the augmentation strategies used.\n6. Try to find a LTH GAN on the pretraining datasets of other transfer methods, then fine tune on those with additional LTH steps on the few-shot domains. This might also help your method by leveraging data that other methods already use. \n\n--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, they have, albeit in a clumsy manner, as I explain in the Details/Conclusion section above.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "j5JvJiR4DsO", "reviewer": "Reviewer_npsk", "summary": "Inspired by the lottery ticket observation, the authors decomposed GAN training into two stages: (i) finding sparse winning tickets in GANs by iterative pruning; (ii) training the found sparse tickets with various augmentations. Either sub-problem becomes more data-efficient to train, and they can re-use the same small set of real images.", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is a very strong empirical work on data-efficient GAN training. To my best knowledge, the authors are the first to show these sparse networks are much more efficient to train with smaller sets of data than their dense counterparts, and hence provide a blessing for few-shot regimes. \n\nFrom a fundamental perspective, it goes in parallel with the existing LTH literature along with the inference efficiency of networks. Although the lottery ticket phenomenon was discovered by other prior works, no existing work ever considered the influence of training data size on locating and training those tickets in general. It will inspire more re-thinking on the value of network sparsity, for not only GANs but also general deep NNs. \n\nThe results are very strong across various datasets, some latest GAN models (including BigGAN and StyleGAN-V2), and the metrics reported.  Ablation studies are thorough and clear to follow.  In particular, the authors compared the two state-of-the-art methods DiffAug and ADA, and they were even able to include the very new RLC method (CVPR’21) into comparison, as well as their various combinations. I salute to the authors’ extensive efforts.\n\nThe authors’ technical approach is quite straightforward. But I feel to some extent, it is a pro rather than a con. Given the new angle (LTH for data-efficient training) as the main delivery of this paper, I like the authors could exploit this insight with very easy-to-follow approaches, and I value a simple method that works while bringing in new insights.\n\nThe writing clarity is good, but the AdvAug part seems a bit disconnected from the main theme of LTH.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "It’ll be great if the authors can address the following concerns:\n\n1. Table 1, Why 100% data was hurt by sparsity 67.24%, with or without AdvAug? Similarly in Table 3 CIFAR-10, Dense BigGAN outperformed sparse in all three conditions. It’s a bit confusing as the benefit of sparsity does not show to be fully consistent.\n\n2. Table 4 StyleGAN-v2: neither AdvAug, nor DiffAug + AdvAug was reported, This looks weird, since previous BigGAN and SNGAN experiments mainly reported on the two.\n\n3. Sparsity during training seems to benefit data-efficient learning, and LTH is one of the possible sparse training ways. Have you tried other ideas, such as SNIP or dynamic sparse training?\n\n4. I can understand sparsity might constrain the hypothesis space and reduce the sample complexity. But why the sparsity structure pattern also matters in data efficiency (e.g., LTH largely outperforms random pruning)?\n\n5. I don’t quite understand why network slimming performs so badly in Figure 7.\n\n6. How about this algorithm applied to image-to-image translation GANs? I think this would be as interesting, such as developing some few-shot super-resolution models in some data-scarce scientific imaging domains.\n\n7. Typos are non-sparse in the paper: “mini-max”, “and we then training them…”, and more. Proofreading is needed.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "RnHgJ4CNby", "reviewer": "Reviewer_3bu4", "summary": "This paper proposed to extend Lottery Ticket Hypothesis (LTH) for data-efficient GAN training. Specifically, the authors have found that sparse subnetworks lead to better sample qualities of Image GANs. In addition, they have proposed AdvAug that combines the ideas of feature augmentation and adversarial training.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "[Strengths]\n- The paper is largely well-written and easy to understand.\n- Using sparse subnetwork to enhance data efficiency seems novel (to the best of my knowledge).\n\n[Weaknesses]\n- Limited novelty in terms of method. This paper does not present any new techniques in the method section. Lottery ticket hypothesis was explored in prior literature. Also, using adversarial training to improve robustness of deep networks are pretty well-known in computer vision.\n- Unconvincing experiment results. Since the novelty of method is limited, the value of the paper hinges on lessons we can learn from the experiments. From Table 1, on 100% and 20% data, sparse network doesn't seem to be helping at all. However, on 10% data, sparse subnetworks are helpful. The results are not convincing to me and seem to be coincidental. I believe there is a lot of randomness going on here. The results are averaged over 3 \"evaluation\" runs, but not on 3 \"training\" runs. I suggest running experiments on 3 different 10%/90% splits for example. Also, in Tables such as Table 4, gaps between different entries are not big enough. Although there is no overlap between error bars, I believe these error bars are related to different evaluation runs (not training runs). From my experience, this kind of numerical difference can be easily reversed if another trial of GAN training is performed.\n- No insight regarding why sparse subnetwork helps data-efficiency of GAN training. Also, does this finding can be extended to image recognition benchmarks?\n\n[Typo]\n- In line 164: Fig 2 outer circle should be Fig 1 outer circle", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations or potential negative societal impact are not discussed in depth in this paper. Including limitations as well as the position of this paper against prior literature will greatly enhance the quality of this paper.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.5"}, "extra_text": {}}, {"review_id": "GhhVvaWA016", "reviewer": "Reviewer_Eqct", "summary": "The paper addresses data-efficient GAN training.  It decomposes the task of the GAN training in low data regimes into two sub-problems: 1) finding the sparse subnetwork, so called lottery ticket, in the original GAN model, based on observations in [1,2], on a small set of training images; 2) then training the found sparse subnetwork, which is a more data-efficient solution in comparison to the dense one, on the same small training set.  In addition, the paper proposes a feature augmentation technique, which injects adversarial perturbations into intermediate features of both discriminator and generator. The paper shows that the sparse subnetwork can benefit from extra data- and feature-level augmentations.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n-\tThe paper addresses a challenging and important problem of training GANs in low-data regimes. The extension of the lottery ticket hypothesis to training GANs in low data regimes is a novel and interesting idea.\n-\tThe proposed feature augmentation helps to improve synthesis performance in both high and low data regimes for dense and sparse GAN networks. It’s also complementary to data augmentation techniques, such as ADA [3] or DiffAug[4].\n-\tThe paper conducts experimental evaluation using different GAN architectures (BigGAN, SNGAN, StyleGAN-v2) across different datasets, showing the superior performance of sparse networks in comparison to their dense baselines in low data regimes.\n\nWeaknesses:\n\n-\tIt would be good to see the efficiency comparison between different sparse and dense networks during training and inference, e.g. what is the throughput of the networks? Number of parameters? Who long does it take to train the sparse subnetwork in comparison to its dense baseline?\n-\tIn low-data regimes the difference between using the dense network and its sparse subnetwork in terms of FID is quite minor, ~1 FID point, e.g 13.59 vs. 13.05 FID on CIFAR100 in Table 4. It’s not obvious what is the benefit of the proposed approach as the performance improvement is marginal and finding lottery tickets requires costly train-prune-retrain process. Thus the paper would benefit from the efficiency analysis (see the point above).\n-\tIt’s not clear form the paper if in low data regime a smaller network would perform better in comparison to the proposed method. A simple comparison of the found sparse subnetwork to the dense baseline with the reduced number of parameter (e.g. number of channels), equivalent to the number of parameters of the sparse subnetwork, would be interesting to see. It might happen that with little amount of training data a smaller network (in terms of parameters) will help to overcome overfitting and might perform even better.\n-\tThe paper misses comparison with the related work on few-shot generation task: Liu et al., “Towards faster and stabilized gan training for high- fidelity few-shot image synthesis”, ICLR, 2021. This work reaches superior performance in comparison to the proposed approach in terms of FID, e.g. on Obama dataset 52.86 FID in Table 5 vs. 41.05 FID in Liu et al., on Panda 14.75 vs. 10.03 FID, on Animal Face – Cat 47.40 vs. 35.11 FID. Note that both approaches do not use any pre-training, thus the comparison is fair.\n\nTypos: Line 164: Figure 2 -> Figure 1, Table 3 “10.14” should not be in bold.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper briefly discusses limitations and potential negative societal impact of the proposed approach.", "review_text": "", "extra_scores": {"ethical_concerns": "No ethical concerns.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3h"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper adopts the idea of pruning and data augmentation to improve the data efficiency of GANs. The rebuttal solves the reviewer's initial concerns about novelty and overfitting, and this paper is recommended for acceptance. 
Decision: Accept (Poster)