Data-Efficient GAN Training Beyond (Just) Augmentations: A Lottery Ticket Perspective

Abstract:
Training generative adversarial networks (GANs) with limited real image data generally results in deteriorated performance and collapsed models. To conquer this challenge, we are inspired by the latest observation, that one can discover independently trainable and highly sparse subnetworks (a.k.a., lottery tickets) from GANs. Treating this as an inductive prior, we suggest a brand-new angle towards data-efficient GAN training: by first identifying the lottery ticket from the original GAN using the small training set of real images; and then focusing on training that sparse subnetwork by re-using the same set. We find our coordinated framework to offer orthogonal gains to existing real image data augmentation methods, and we additionally present a new feature-level augmentation that can be applied together with them. Comprehensive experiments endorse the effectiveness of our proposed framework, across various GAN architectures (SNGAN, BigGAN, and StyleGANV2) and diverse datasets (CIFAR-10, CIFAR-100, Tiny-ImageNet, ImageNet, and multiple few-shot generation datasets). Codes are available at: https://github. com/VITA-Group/Ultra-Data-Efficient-GAN-Training.

text: Training generative adversarial networks (GANs) with limited real image data generally results in deteriorated performance and collapsed models. To conquer this challenge, we are inspired by the latest observation, that one can discover independently trainable and highly sparse subnetworks (a.k.a., lottery tickets) from GANs. Treating this as an inductive prior, we suggest a brand-new angle towards data-efficient GAN training: by first identifying the lottery ticket from the original GAN using the small training set of real images; and then focusing on training that sparse subnetwork by re-using the same set. We find our coordinated framework to offer orthogonal gains to existing real image data augmentation methods, and we additionally present a new feature-level augmentation that can be applied together with them. Comprehensive experiments endorse the effectiveness of our proposed framework, across various GAN architectures (SNGAN, BigGAN, and StyleGANV2) and diverse datasets (CIFAR-10, CIFAR-100, Tiny-ImageNet, ImageNet, and multiple few-shot generation datasets). Codes are available at: https://github. com/VITA-Group/Ultra-Data-Efficient-GAN-Training.
1 Introduction
The quantity, diversity, and high quality of natural images available in the general domain have played an essential role in the achieved breakthroughs of Generative Adversarial Networks (GANs) [2–11] over the past few years. However, it could become challenging or even infeasible for specific application domains to collect a sufficiently large-scale dataset, due to various constraints on the imaging expense, subject type, image quality, privacy, copyright status, and more. That prohibits GANs’ broader applications in these domains, e.g., for generating synthetic training data [12]. Examples of such domains include medical images, images from scientific experiments, images of rare species, or photos of a specific person or landmark. Eliminating the need of immense datasets for GAN training is highly demanded for those scenarios. Naively training GAN with scarce samples leads to overconfident discriminators that overfit the small training data [13–15, 1]; it usually ends up with training divergence and drastic performance degradation (evidenced later in Figure 2).
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
This paper addresses the above issue from a brand new perspective by decomposing the challenging GAN training in limited data regimes into two sequential sub-problems: (i) finding independent trainable subnetworks (i.e., lottery tickets in GANs) [16, 17]; then (ii) training the located subnetworks, which we show is more data-efficient by itself, and can further benefit from aggressive augmentations (both the input data and feature levels). Either sub-problem becomes much less data-hungry to train, and the two sub-problems re-use the same small training set of real images. Although this paper focuses on tackling the data-efficient training of GANs, such a coordinated framework might potentially be generalized to training other deep models with higher data efficiency too.
Our key enabling technique is to leverage the lottery ticket hypothesis (LTH) [18]. LTH shows the feasibility to locate highly sparse subnetworks (called “winning tickets”) that are capable of training in isolation to match or even outperform the performance of original unpruned models. Recently, [17, 16] revealed the existence of winning ticket in GANs (called “GAN tickets”). However, none of the existing works discuss the influence of training data size on locating and training those tickets. Our work takes one step further, and shows that one can identify the same high-quality GAN tickets even in the data-scarce regime. The found GAN tickets also serve as a sparse structural prior to solve the second sub-problem with less data, while maintaining an unimpaired trainability blessed by the LTH assumption [18]. Figure 1 (the outer circle’s blue dots) evidences that we can identify sparse GAN tickets that achieve superior performance than full GANs in the data-scarce scenarios.
The new lottery ticket angle complements the existing augmentation techniques [19–21], and we further show that they can be organically combined to boost performance further1. When we train the identified lottery ticket, we demonstrate its training can benefit as well from the latest data-level augmentation strategies, ADA [15] and DiffAug [1]. Furthermore, we introduce a novel feature-level augmentation that can be applied in parallel to data-level. It injects adversarial perturbations into GANs’ intermediate features to implicitly regularize both discriminator and generator. Combining the new feature-level and existing data-level augmentations in training GAN tickets leads to more stabilized training dynamics, and establishes new state-of-the-arts for data-efficient GAN training.
Extensive experiments are conducted on a variety of the latest GAN architectures and datasets, which consistently validate the effectiveness of our proposal. For example, our BigGAN tickets at 36.00% and 67.24% sparsity levels reach an (FID, IS) of (23.14, 52.98) and (70.91, 7.03), on Tiny-ImageNet 64× 64 and ImageNet 128× 128, with 10% and 25% training data, respectively. On CIFAR-10 and CIFAR-100, for SNGAN and BigGAN tickets at 67.24% ∼ 86.58% sparsity, our results with only 10% training data can even surpass their dense counterparts. Impressively, our method can generate high-quality images on par with other GAN transfer learning approaches, by training on as few as 100 real samples and without using any pre-training.

heading: 2 Related Work
text: GANs and Data-Efficient GAN Training. GANs [22] have gained popularity in diverse computer vision scenarios. To stabilize GAN training and improve the visual fidelity and diversity of generated images, extensive studies have been conducted, such as sophisticated network architectures [23, 8, 24– 26], improved training recipes [27, 4, 28, 29], and more stable objectives [30–35]. [36, 37] utilize semi- and self-supervised learning to pursue label efficiency in GAN training.
Recently, how to train GANs without sufficient real images in the target domain sparkles new interests. There have been efforts on adapting a pre-trained GAN generator, including BSA [38], AdaFM [39], Elastic Weight Consolidation [40], and Few-Shot GAN [41–43]. However, those methods assume a large, related source domain as pre-training, based on which they further alleviate target domain data limitation by only tuning small subsets of weights. They are hence in a completely different track from our “stand-alone” data-efficient training goal where no pre-training is leveraged. [44, 45] select core-sets of training data to speed up GAN training. A few recent attempts [1, 15] leverage differentiable or adaptive data augmentations to significantly improve GAN training in limited data regimes. Lately, [46] investigates a regularization approach, on constraining the distance between the current prediction of the real image and a moving average variable that tracks the historical predictions of the generated image, that complements the data augmentation methods.
Lottery Ticket Hypothesis and GAN Tickets. [18] claims the existence of independently trainable sparse subnetworks that can match or even surpass the performance of dense networks. [47, 48] scale
1We also tried to add augmentations in the lottery ticket finding stage, but did not observe visible impact.
up LTH by rewinding [49, 50]. Follow-up researches evidence LTH across broad fields, including visual recognition [18, 47, 51–60], natural language processing [48, 61, 50, 62–64], graph neural network [65], and reinforcement learning [61].
Recently, LTH has been extended to GANs by [16, 17], who validated the existence of winning tickets in the min-max game beyond minimization. Compared with the aforementioned work, our work is the first to study LTH in the data-scarce regime (for GANs, and in general). Besides finding highly compact yet same capable subnetworks, our work reveals LTH’s power in saving training data - an appealing perspective never being examined before.
Adversarial Training and Augmentations. Deep neural networks suffer from severe performance degradation [66, 67] when facing adversarial inputs [67–69]. To address this notorious vulnerability, various defense mechanisms [70–79] have been proposed. Among others, adversarial training-based approaches achieve superior adversarial robustness [67–69], although at the price of sacrificing benign generalization [80, 70–75].
Several recent works investigate enhancing model (benign) generalization ability with adversarial training [81–86]. They adopt adversarially perturbed input images, embeddings, or intermediate features, into model training to ameliorate performance on the clean test sets. Specifically, the damaging effects of adversarial training could be controlled by extra batch normalization [81] or so. Different from those minimization problems that previous work has focused on, the two-player GAN optimization is more challenging. Generally, the adversarial competition between two players poses impediments to exploit extra adversarial information during training GANs.

heading: 3 Methodology

heading: 3.1 Revisiting GANs and the Overfitting Challenge
text: Generative adversarial networks (GANs) are dedicated to modeling the target distribution with the two-player game formulation of a generator G and a discriminator D. Specifically, the generator G takes a random sampled latent vector z (e.g., from a Gaussian distribution) as input and outputs the fake sample G(z). The discriminator D aims to distinguish generated fake samples G(z) from real samples x. Alternative optimizations for the discriminator’s loss LD and the generator’s loss LG are adopted in the standard GAN training, which can be depicted as follows:
LD := Ex∼pdata(x)[fD(−D(x))] + Ez∼p(z)[fD(D(G(z)))] LG := Ez∼p(z)[fG(−D(G(z)))],
where loss functions fD(x), fG(x) have multiple choices, e.g., the non-saturating loss [3] with fD(x) = fG(x) = log(1 + e
x), and the hinge loss [23] with fD(x) = max(0, 1 +x) and fG(x) = x. pdata(x) and p(z) represent the data distribution of real samples and latent vectors. LD is maximized to update D’s parameters φ (i.e., D(·) := D(·, φ)), and LG is minimized to update G’s parameters θ (i.e., G(·) = G(·, θ)).
Algorithm 1 Data-Efficient Iterative Magnitude Pruning Procedures
1: Input: Initial two masks mg = 1‖θ‖0 and md = 1
‖φ‖0 ; Initialization weights θ0 and φ0 2: Output: {G(·, θ0 mg),D(·, φ0 md)} 3: repeat 4: Training {G(·, θ0 mg),D(·, φ0 md)} for t epochs with limited training data 5: Pruning ρ = 20% of remaining weights in both G and D 6: Updating the binary masks mg and md accordingly 7: Rewinding weights of G, D to θ0 and φ0 8: until masks reach the desired sparsity level
Training Failures of GANs under Limited Data. [15, 1] observe that GANs’ performance has severely deteriorated when only limited training data is available. The discriminator tends to memorize and heavily overfit the small training samples, leaving a gap between the real sample’s and the generated sample’s distribution. As shown in Figure 2, with only 10% CIFAR-10 data available for training SNGAN [23], the training and validation accuracies of the discriminator D quickly saturate to nearly 100% (ideally close to 50%), which indicates D to become over-confident in distinguishing real and generated samples. It demonstrates that D simply memorizes the training data, and such overfitting leads to training collapses and deteriorated quality of generated images.
To address this dilemma, we suggest a new data-efficient GAN training workflow, decomposed into two stages: (i) finding winning tickets in GANs via Algorithm 1; then (ii) training the found GAN tickets, potentially with both data- and feature-level augmentations, via Algorithm 2. Blessed by LTH, the located GAN ticket shows improved generalization ability, and is further enhanced by augmentations that prevent D from becoming too confident.

heading: 3.2 Data-Efficient Lottery Ticket Finding from GANs
text: In this section, we provide the preliminaries and setups to identifying data-efficient GAN tickets.
Subnetworks and winning tickets. A subnetwork of GAN is defined as {G(·, θ mg),D(·, φ md)}, where mg ∈ {0, 1}‖θ‖0 and md ∈ {0, 1}‖φ‖0 are binary masks for the generator and discriminator respectively, and is the element-wise product. Let θ0 and φ0 be the initialization weights of GANs. Following [18, 16], we define winning tickets of GAN as subnetworks {G(·, θ0 mg),D(·, φ0 md)}, that reach a matched or better performance compared to unpruned GANs when trained in isolation with similar training iterations.
Finding data-efficient winning tickets in GANs. To our best knowledge, we are the first to extend LTH to the limited data regimes. In this challenging scenario, only a small amount of training data are accessible for the finding and training of GAN tickets. We use unstructured magnitude pruning [87], e.g., Iterative Magnitude Pruning (IMP), to establish the sparse masks mg and md.
As shown in Algorithm 1, we first train the full GAN model for t epochs with limited training samples (e.g., 100-shot), and then perform IMP to globally prune the weights with the lowest magnitude. Zero elements in the obtained masks mg and md index the pruned weights. Before repeating the process again, the weights of the sparse generator G(·, θ mg) and discriminator D(·, φ md) are rewound to the same initialization θ0 and φ0, following the convention [18]. The pruning ratio ρ controls the portion of weights removed per round, and we fix ρ = 20% in all experiments.
Intuitively, identifying a special sparse mask (without requiring to train its weights well) should be an easier and hence more data-efficient task compared to training the full network weights. That was verified by our observations in experiments too: when the training data volume reduces from 100% to 10% of the full training set, the quality of sparse mask remains to be stable, since it achieves matched performance compared to its dense counterpart in both full and limited data re-training regimes.

heading: 3.3 Data-Level and Feature-level Augmentations for Training GAN Tickets
text: After locating the GAN ticket at certain sparsity, training it using the vanilla recipe could already attain significantly improved IS and FID compared to the full dense model trained in the same data-limited regime: see Figure 1 outer circle for example.
Next, we discuss how our proposal can be applied together with augmentation-based approaches, for enhanced training of our found GAN tickets. Our natural choices include to plug-in the two recent state-of-the-art data augmentations, i.e., DiffAug [1] and ADA [15]. We further present a new adversarial feature-level augmentation (AdvAug), that can be jointly applied together with data-level augmentations to gain an additional performance boost.
Revisiting adversarial training. Let (x,y) denote the input image and its label. f(ϑ,x,y) is the loss function parameterized by ϑ. Adversarial training [69] can be formulated as follows:
min ϑ
E(x,y) [
max ‖δ‖p≤ f(ϑ,x + δ,y)
] , (1)
where δ is crafted adversarial perturbation constrained within the `p norm ball that is centered at x with a radius . δ can be reliably generated by multi-step projected gradient descent (PGD) [69].
Different from the above standard adversarial training, which adds perturbation on the image pixel space, AdvAug injects adversarial perturbations to intermediate feature embeddings of both G and D. A similar feature augmentation idea was proven to be helpful in NLP [82], and computer vision [86], showing effectiveness to regularize the smoothness of the training landscape and enhance the trained model’s generalization. The AdvAug scheme is illustrated in Figure 3, and can be mathematically depicted as follows (λ1 is a controlling hyperparameter):
min θ LG + λ1 · LadvG s.t. LadvG := max ‖δ̂‖∞≤ Ez∼p(z)[fG(−D(G2(G1(z) + δ̂)))]. (2)
We choose λ1 = 1 in all experiments for simplicity. G = G2 ◦ G1 denotes the generator, and between the G1 and G2 parts we inject AdvAug. Adversarial perturbations δ̂ generated by PGD [69], are applied to the intermediate feature space G1(z). The details of our feature-level augmentation on the discriminator D is included in Appendix A1. The full algorithm of training GAN with both dataand feature-level augmentations is summarized in Algorithm 2.
Algorithm 2 Training (Sparse) GAN with Data- and Feature-level Augmentations Input: GAN {G(·, θ0),D(·, φ0)}; Inputs x and z Output: Trained GAN {G(·, θT),D(·, φT)} for t = 1 to T do
# Training discriminator with data and feature augmentations Augment input with DiffAug [1] or ADA [15] Feed x and G(z) to D Generate adversarial augmented features in D ( Equation. 5) Update the discriminator D(·, φt) (Equation. 6) # Training generator with data and feature augmentations Sample and augment z with DiffAug [1] or ADA [15] Feed z to G. Generate adversarial augmented features in G ( Equation. 3) Update the discriminator G(·, θt) (Equation. 4)
end for
Note that AdvAug only affects the generated images through G intermediate features, and the classifier learning through D features. It hence avoids to directly manipulate the real data distribution. One bonus of doing so is that it is potentially better at alleviating the distribution leaking issue [15], i.e., GANs learn to mimic and generate the augmented distribution rather than the real one.

heading: 4 Experiments
text: In this section, we conduct comprehensive experiments on Tiny-ImageNet [88], ImageNet [89], CIFAR-10 [90], and CIFAR-100 based on the unconditional SNGAN [23] and StyleGAN-V2 [6], as well as the class-conditional BigGAN [2]. We adopt the common evaluation metrics, including Fréchet Inception Distance (FID) [91] and Inception Score (IS) [34]. Note that a smaller FID (↓) and a larger IS (↑) indicate better performing GAN models. Furthermore, we evaluate our proposed method on few-shot generation both with and without pre-training in Section 4.3. Extensive ablation studies analyze effectiveness of each component in Section 4.4.
Implementation and Baseline Details. We follow the popular StudioGAN codebase [92], which contains high-quality re-implementation of BigGAN and SNGAN on ImageNet and CIFAR. For example, our implemented BigGAN baseline performs much better, i.e., FID: 26.44 (ours) v.s. 39.78 (reported) on CIFAR-10, and FID: 36.58 (ours) v.s. 66.71 (reported) on CIFAR-100, than the recent reported baselines in [1], under 10% training data regimes. For detailed configuration, BigGAN takes
learning rates of {4, 2, 2} × 10−4 for G, of {1, 5, 2} × 10−4 for D, batch sizes of {256, 256, 64}, 1× 105 training iterations, and {1, 2, 5} D steps per G step on {Tiny-ImageNet, ImageNet, CIFAR} datasets. SNGAN uses learning rates of 2× 10−4 for G and D, batch sizes of 64, 5× 104 training iterations, and five D steps per G step on CIFAR. For StyleGAN-V2 experiments, we use its popular PyTorch implementation2, and keep the default configuration in [1] including image resolution (256× 256), learning rates for D/G (2× 10−4), batch size (5), and training iterations (1× 105). Note that, same as the setting in [1], training iterations will be doubled when training GANs with DiffAug. We use implementations in the StudioGAN codebase for DiffAug [1], and the official implementation3 for ADA [15]. AdvAug with PGD-1 and step size 0.01/0.001 is applied on CIFAR/(Tiny-)ImageNet datasets, which are tuned by a grid search in Section 4.4. All GANs are trained with 8 pieces of NVIDIA V100 32GB.

heading: 4.1 On the Effectiveness of Training with Winning Ticket and AdvAug
text: We adopt the top-performing model BigGAN [2], and report experiments on both Tiny-ImageNet at 64 × 64 resolution and ImageNet at 128 × 128 resolution. We evaluate our proposal on TinyImageNet with 10%, 20%, 100% data available, and ImageNet with 25% data available, with results summarized in Tables 1 and 2, respectively. All our results are averaged over three independent evaluation runs (same hereinafter), and the best performance of each column are highlighted.
The following observations can be drawn: First, sparse BigGAN tickets can achieve consistently improved performance over the full model (0.00%). Especially, with only 10% training data available, BigGAN tickets at 67.24% sparsity obtain massive gains of 11.30 FID and 1.58 IS on Tiny-ImageNet. Second, feature-level augmentation (AdvAug) consistently improves all training cases, from dense to sparse, and from full data to limited data. In particular, larger sparsity (e.g., 67.24%) with less training data available (e.g., 10%) tend to benefit more from applying AdvAug, which is aligned with our design principle. Third, while the full data regime (100% data) does not necessarily prefer the highest sparsity (moderate sparsity still benefits), the limited data regimes (10% or 20% data) see monotonically increasing gains as the ticket sparsity goes higher. That is understandable since the former may need more model capacity to absorb full training data, while the latter case hinges on sparsity to avoid overfitting their limited training data.
Besides, we report another group of experiments of training SNGAN on CIFAR-10, using from 10% to 90% training data. The results are summarized in Figure 4. The conclusions we can draw are highly consistent with the above BigGAN case: (1) at the same data availability (from 10% to even 90%), training a sparse ticket is always preferred over training the dense model; (2) AdvAug is also consistently helpful in all cases; (3) for both sparsity and AdvAug, they can contribute to larger gains when training data gets smaller.
2https://github.com/lucidrains/StyleGAN-V2-pytorch. Note that the PyTorch version remains with a small performance gap compared to the TensorFlow implementation in [1].
3https://github.com/NVlabs/StyleGAN-V2-ada-pytorch. The official Pytorch implementation in [15].

heading: 10% Training Data

heading: 10% Training Data

heading: 4.2 Incorporating Our Proposal with Latest Data Augmentations and Regularization
text: Combining DiffAug. We first incorporate DiffAug [1], as a representative of the latest data-level augmentation, into our proposal and show the complementary gains. We conduct experiments on the class-conditional BigGAN and unconditional SNGAN models with CIFAR-10 and CIFAR-100. For BigGAN, we utilize 100%, 20%, 10% data to locate GAN tickets; then we train them with datalevel DiffAug, or with both DiffAug and featurelevel AdvAug, as shown in Table 3. Consistent observations can be drawn: First, similar to our previous observations on AdvAug, DiffAug also shows to contribute more when the training data becomes more limited; Second, combining DiffAug and AdvAug improves over either alone, and leads to the best results across all cases.
Combining Other Data Augmentations and Regularization. We then extend our combination study to other recent data augmentation and regularization approaches, e.g., ADA [15] and RLC [46].
Experiments are conducted on CIFAR-100 with StyleGAN-V2 backbone, and results are collected in Table 4. We observe that plugging in either ADA [15] or DiffAug [1] into our framework could improve sparse GAN winning tickets, and the gain is also enlarged when ADA is combined with AdvAug. Regard to RLC4, it is less effective combined with other augmentations.
Taking above together, it has been clearly shown that our proposal is orthogonal to those existing efforts and is of independent merit. Moreover, combing them would lead to more powerful pipelines for data-efficient GAN training.

heading: 4.3 Few-Shot Generation
text: It is laborious, and sometimes impossible to collect a large-scale dataset for certain images of interest. To tackle the few-shot image generation problem, [93] utilizes pre-training from external large-scale datasets and performs fine-tuning under limited data scenarios; [94], [38] and [95] partially fine-tune the GANs with part of the GAN model being frozen.
We compare these transfer learning approaches5 with our data-efficient training scheme. Differently from them, ours is training from scratch and is free of any pre-training, while all transfer learning methods start from a pre-trained StyleGAN-V2 model on the FFHQ face dataset [5].
Our comparison experiments are conducted using StyleGAN-V2 on the AnimalFace [96] dataset (160 cats and 389 dogs), and the 100-shot Obama, Grumpy Cat, and Panda datasets provided by [1]. As shown in Table 5, our method finds data-efficient GAN tickets at 48.80% sparsity levels, that can be trained with only 100 training samples from scratch (without any pre-training) and show competitive performance to other transfer learning algorithms. Visualizations of style space interpolation and few-shot generation are provided in Figure 5 and 6.
Table 5: Few-shot generation. Following the setting in [1], we calculate the FID with 5K samples and the training dataset is adopted as the reference distribution. All transfer learning methods have their pre-trainings from FFHQ [5]. StyleGAN-V2 tickets at 48.80% sparsity level are found and used in our method.
Methods Pre-training? 100-shot by [1] AnimalFace
Obama Grumpy Cat Panda Cat Dog
Scale/shift [38] Yes 50.72 34.20 21.38 54.83 83.04 MineGAN [95] Yes 50.63 35.54 14.84 54.45 93.03 TransferGAN [93] Yes 48.73 34.06 23.20 52.61 82.38 FreezeD [94] Yes 41.87 31.22 17.95 47.70 70.46
StyleGAN-V2 (0.00%) No 89.18 61.97 90.96 95.75 164.54 + DiffAug + AdvAug No 54.11 35.46 15.94 54.02 72.47
StyleGAN-V2 Tickets (48.80%) No 73.92 56.81 82.45 85.92 153.90 + DiffAug + AdvAug No 52.86 31.02 14.75 47.40 68.28
Figure 5: Style interpolation visualizations of StyleGAN-V2 tickets (48.80%) with AdvAug only on 100-shot Obama, Grumpy Cat, Panda, and AnimalFace datasets, respectively.
Figure 6: Few-shot generalization results of StyleGAN-V2 tickets (48.80%) with AdvAug only on 100-shot Obama, Grumpy Cat, Panda, and AnimalFace datasets, respectively.
4[46] advocates the best-performing configuration is ADA+RLC, with FID 13.01 on 10% data of CIFAR-100. 5Implementations are from the codebase of [94].

heading: 4.4 Ablation and Analysis
text: Pruning Ratio ρ in the Ticket Finding. To understand the effect of the pruning ratio in IMP to the quality of data-efficient GAN tickets, we experiment on SNGAN with 10% data of CIFAR-10 and ρ = 10%, 20%, 40% as the pruning ratio. As shown in Figure 7 (Left), all three IMP settings find data-efficient winning tickets in GAN; IMP with a lower pruning ratio tends to identify higher-quality GAN tickets in terms of FID, while it usually costs much more to reach the same level of sparsity as higher pruning ratios do.
Augment G or D: Either or Both. We apply AdvAug on G or D only, and AdvAug on both G and D. Only 10% data of CIFAR-10 are available for finding GAN tickets and training with AdvAug. Results are summarized in Figure 7 (Middle). Either employing AdvAug on D or G consistently obtains significant performance improvements (i.e., largely reducing the FID) over all sparsity levels, and augmenting both D and G further enhance the found data-efficient GAN tickets. Our results also show that sparser GAN tickets can benefit more from AdvAug, such as subnetworks with 20%, 36%, 83.22% sparsity.
Strength and Locations of Injecting AdvAug. To better interpret the influence of the strength and layer locations of injected adversarial feature perturbations, we comprehensively examine SNGAN on 10% training data of CIFAR-10 across different step sizes, the number of PGD steps, and locations (i.e., where to apply AdvAug). When studying one of the factors, we fix the other factors with the best setup. From Figure 8, several observations can be drawn:
• Figure 8 (a) and (b) show that adopting AdvAug with step size 0.01/0.001 and PGD-1/3 assists the data-efficient GAN training, while AdvAug with step size 0.05/0.1 and PGD-5/7 perform worse than the baseline without AdvAug (i.e., the setting with zero step size or zero step PGD in Figure 8). It reveals that overly strong AdvAug can hurt performance.
• As shown in Figure 8 (c) and (d), augmenting the last layer of the discriminator D and the first layer (i.e., the closest layer to the latent input vector) of the generator G appears to be the best configuration for utilizing AdvAug. It seems that injecting adversarial perturbations into the “high-level” feature embeddings in general benefits more to mitigate the overfitting issue in data-limited regimes.
In summary, we observe that applying AdvAug to the last layer of D and the first layer of G, with PGD-1 and step size 0.01, seems to be a sweet-point configuration for data-efficient GAN training, which is hence adopted as our default setting.
Comparison with Baselines. Naive baselines, i.e., random pruning, one-shot magnitude pruning (OMP) [87, 16], network slimming (NS [97], and random noise augmentation, are evaluated in Figure 7 (Right) and Table 6. Compared to random pruning, OMP and NS, IMP produces much better GAN tickets, especially at high sparsity levels (e.g., ≥ 48.8%). Compared to augmenting features with random noise
sampled from N (0, 0.012), AdvAug also achieves larger performance gains on both 100% and 10% training data regimes.

heading: 5 Conclusion and Discussion of Broader Impact
text: We introduce a novel perspective for data-efficient GAN training by leveraging lottery tickets, which augmentations can further enhance, including our newly introduced feature-level augmentation. Comprehensive experiments consistently demonstrate the effectiveness of our proposal, on diverse GAN architectures, objectives, and datasets. Note that although finding lottery tickets requires a costly train-prune-retrain process, only data efficiency is of interest in this work. An intriguing future work would be to pursue data and resource efficiency (training and inference) together.
This research aims to enhance GAN training in the limited data regimes. However, it might amplify the existing societal risk of applying GANs. For example, the issue of image generation bias may be impacted or even amplified by the sparse structures, which we will verify in future work. The data-efficient generation ability might also be leveraged by undesired applications such as DeepFake.

Review summaries:
This paper proposed to extend Lottery Ticket Hypothesis (LTH) for data-efficient GAN training. Specifically, the authors have found that sparse subnetworks lead to better sample qualities of Image GANs. In addition, they have proposed AdvAug that combines the ideas of feature augmentation and adversarial training.

The paper addresses data-efficient GAN training.  It decomposes the task of the GAN training in low data regimes into two sub-problems: 1) finding the sparse subnetwork, so called lottery ticket, in the original GAN model, based on observations in [1,2], on a small set of training images; 2) then training the found sparse subnetwork, which is a more data-efficient solution in comparison to the dense one, on the same small training set.  In addition, the paper proposes a feature augmentation technique, which injects adversarial perturbations into intermediate features of both discriminator and generator. The paper shows that the sparse subnetwork can benefit from extra data- and feature-level augmentations.