Deceive D: Adaptive Pseudo Augmentation for GAN Training with Limited Data
Authors: Liming Jiang
Bo Dai
Wayne Wu
Chen Change Loy
Venue: NeurIPS 2021 Poster (2021)
Keywords: image and video synthesis
generative adversarial networks
limited data
discriminator overfitting

Abstract: Generative adversarial networks (GANs) typically require ample data for training in order to synthesize high-fidelity images. Recent studies have shown that training GANs with limited data remains formidable due to discriminator overfitting, the underlying cause that impedes the generator's convergence. This paper introduces a novel strategy called Adaptive Pseudo Augmentation (APA) to encourage healthy competition between the generator and the discriminator. As an alternative method to existing approaches that rely on standard data augmentations or model regularization, APA alleviates overfitting by employing the generator itself to augment the real data distribution with generated images, which deceives the discriminator adaptively. Extensive experiments demonstrate the effectiveness of APA in improving synthesis quality in the low-data regime. We provide a theoretical analysis to examine the convergence and rationality of our new training strategy. APA is simple and effective. It can be added seamlessly to powerful contemporary GANs, such as StyleGAN2, with negligible computational cost. Code: https://github.com/EndlessSora/DeceiveD.

Deceive D: Adaptive Pseudo Augmentation for GAN Training with Limited Data
Liming Jiang'
Bo Dai
Wayne Wu²
Chen Change Loy!
S-LLa,, Nanyang Technological University (1iming002, da.d ecloy)@ntu.ee
2 Sensee Time Research
wuvenyan@sensetingg com
Abstract
Generative adversarial networks (GANs) typically require ample data for train- ing in order to synthesize high-fidelity images. Recent studies have shown that training GANs with limited data remains formidable due discriminator overfit- ting, the underlying cause that impedes the generator's convergence. This paper introduces novel strategy called Adaptive Pseudo Augmentation (APA) to en- courage healthy competition between the generator and the discriminator. As an alternative method to existing approaches that rely on standard data augmentations model regularization, APA alleviates overfitting by employing the generator itself to augment real data distribution with generated images, which deceives the discriminator adaptively. Extensive experiments demonstrate the effectiveness APA in improving synthesis quality in the low-data regime. We provide a theoretical analysis examine the convergence and rationality of our training strategy. APA is simple and effective. can be added scamlessly to powerful contemporary GANs, such StyleGAN2, with negligible computational cost. Code: htttps://githut: wwtp:/coeeeeenneeeeieeeee httm//ddlessSor/eaaaeellllll
Introduction
While state-of-the-art GANs like StyleGAN2 20) are constantly pushing forward the fidelity and resolution of synthesized images, they usually require large amount of training data to fully unleash their power. Training GANs with insufficient data tend to generate poor-quality images, as shown in Figure Collecting sufficient data samples for these GANs is sometimes infeasible, especially in domains where data are sparse and privacy-senstivve. To ease the practical deployment of powerful GANs, it is necessary devise new strategies for training GANs with limited data while preserving the quality of synthesis.
Recent studies have shown that the overfitting of the discriminator is the critical reason that impedes effective GAN training on limited data 2 18 50 42
rendering severe instability of training dynamics. Specifically, when the discriminator starts overfit, the distributions of its outputs for real and generated samples gradually diverge from each other 18 42). and its feedback to the generator becomes less informative. Consequently, the generator converges the quality of synthesized images. Recent solutions to this problem include the use of standard data augmentations. either conventional or differentiable, real and generated images [18] 50 52, 41
inferior point, compromising
applying an additional model regularization term 42]. Addressing the discriminator overfitting is still an open problem. We are interested in finding an alternative way the aforementioned approaches.
In this paper, we present a simple yet effective way to regularize a discriminator without introducing any external augmentations regularization terms. We call our method Adaptive Pseudo Augmen-
tation (APA). contrast to previous standard data augmentations [8 50) 52 41
we exploit the
Corresponding author.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

StyleGAN2
APA (Ours)
FFHQ-5k (5,000 img)
Anime AFHQ-Cat-Sk (5,153 img)
15,000 img)
Figure 1: StyleGAN2 20 synthesized results (no truncation) deteriorate given the limited amount of training data (256 x 256), i... FFHQ [19] (a subset of 5, 000 images, 7% of full data), AFHQ- Cat [8] (5, 153 images, which small by itself), and Danbooru2019 Portraits (Anime) 4 (a subset of 5, 000 images, ~ 2% of full data). The proposed Adaptive Pseudo Augmentation (APA) effectively
ameliorates the degraded performance of StyleGAN2 on limited data.
generator in GAN itself to provide the augmentation, more natural way regularize the overfitting of the discriminator. Compared to the model regularization, our approach is more adaptive to fit different settings and training status without manual tuning. Specifically, APA takes the fake/pseudo samples synthesized by the generator and moderately feeds them into the limited real data. Such pseudo data adaptively presented to the discriminator "real" instances. The goal of this pseudo augmentation for the real data is not to enlarge the real dataset but to suppress the discriminator's confidence in distinguishing real and fake distributions. The deceit is introduced adaptively, which is moderated by deception probability according to the degree of overfitting. To quantify overfitting, study series of plausible heuristics derived from the discriminator
output logits.
The main contribution of this work a novel adaptive pseudo augmentation method for training GANs with limited data. This approach deceives the discriminator adaptively and mitigates the problem of discriminator overfitting. The proposed APA can be readily added to existing GAN training with negligible computational cost. We conduct extensive experiments demonstrate the effectiveness of APA for state-of-the-an GAN training with limited data. The results are comparable better than other types of solutions [8 42). APA is also complementary methods based on standard data augmentations for gaining a further performance boost. Besides, theoretically connect APA with minimizing the JS divergence (3] between the smoothed data distribution and generated distribution, proving its convergence and rationality. We hope that our approach could extend the breadth and potential solutions GAN training with limited data.
existing
Related Work
Generative adversarial networks. Generative adversarial networks (GANs) 28
35 47 adopt
an adversarial training scheme, where a generator keeps refining its capability in synthesizing images compete with a discriminator (i.e., a binary classifier) until the discriminator fails to classify
the generated samples as fakes. GANs are known to suffer from training instability (9, 38 2 27 Various approaches have been proposed stabilize the training and improve the quality of synthesis by minimizing different f-divergences of the real and fake distributions [30]. The saturated form
two distributions. LSGAN 26 and EBGAN 49 correspond to the optimizations of x² -dvvergeen [34
of vanilla GAN [ theoretically proven to minimize the JS divergence (23 between
and the total variation respectively. another note, WGAN designed for minimizing the Wasserstein distance.

---

Phaer 7Uk Orae 70k
- 2k
FID 73k
II0- 7
Ora 7k
sign 70k
120
Signs of discriminttr outpurs
1.0
Discrmiintoo cutous
0.
60
2
40
+
%
10005 15500 20000
2500
Training progress (ximg)
Training progress (kimg)
Training progress (kimg) (c)
The overfitting of discriminator in GANs when limited training data available. The three subplots report statistics of training snapshots of two StyleGAN2 (20) models on FFHQ [19] (256 x 256). "70k" indicates the full dataset, and "7k" means subset of 000 images (10% data). The "kimg" denotes thousands of real images shown to the discriminator. (a) Discriminator raw output logits. (b) Signs of discriminator outputs. (c) Training convergence measured by FID (TT).
Figure 2:
State-of-the-ar methods, such as PGGAN 17. BigGAN [S), StyleGAN [19] and StyleGAN2 20).
employ large-scale training with contemporary techniques, achieving photorealistic results. These
methods have been extended to various tasks, including face generation 17 19
20
image editing [
7 33
semantic image synthesis [44 32 25). image-to-image translation 13 53
g 16
15 36].
style transfer [24 12 22]. and GAN inversion
31 36 46). Despite the remarkable success, the
performance of GANs relies heavily on the amount of training data.
Training GANs with limited data. The significance and difficulty of training GANs with limited data have been attracting attention from many researchers recently. The issue of data insufficiency tends to cause overfitting in the discriminator 45 50 which in turn deteriorates the stability of training dynamics in GANs, compromising the quality of generated images.
Many recent studies 48 51 41 [52 50 [8] propose apply standard data augmentations for GAN training to enrich the diversity of the dataset to mitigate the overfitting of the discriminator. For instance, Diff Dugf'etgmett (DA) (50] adopts the same differentiable augmentation to both real and fake images for the generator and the discriminator without manipulating the target distribution. Adaptive discriminator augmentation (ADA) [8] shares similar idea with DA, while further devises an adaptive approach that controls the strength of data augmentations adaptively. In this work, we extend the study of such an adaptive approach.
Another type of solution is model regularization. Previous efforts on regularizing GANs include adding noise to the inputs of the discriminator 2 40 14), gradient penalty (27] 10 37). one-sided label smoothing (38), spectral normalization (29, label noise [6. etc. These methods are designed
for stabilizing training or preventing mode collapse 3 The essence of their goals could be considered similar to our method since training GANs in the low-data regime exhibits similar behaviors as previously observed in early GANs with sufficient data. Additional discussions these techniques are provided in the supplementary material. Under the limited data setting, a very recent study proposes an Lerreuularization term 42 regulate the discriminator predictions using two exponential moving average variables that track the discriminator outputs throughout training.
Our work explores alternative solution from different perspective, which is also complementary to previous approaches based on standard data augmentations.
3 Methodology
In GAN's adversarial training, the goal of the generator G is to deceive the discriminator D and maximize the probability that D makes a wrong judgment. Therefore, G keeps refining its generated samples to better deceive D over time. When the training only accesses limited amount of data, one would observe that D turns out to be overly confident and hardly makes any mistake, causing its feedback to G to become meaningless.
In Figure 2 we show the training "snapshots" of two StyleGAN2 20) models on the FFHQ
dataset [19 The settings of the two models differ only by the amount of data available to them for
3

---

limited real data
Doa
Duse
real x
Discriminator outputs
4. b.
overfitting
adaptive
heuristic
Training progress (kimg)
fake (zz)
D(x)
2 (2)z(Z)
D
D(G(Z))
Figure 3: Adaptive pseudo augmentation (APA) for GAN training with limited data. We employ GAN to augment itself using the generated images to deceive the discriminator adaptively. Specifically, APA feeds the images synthesized by the generator into the limited real data moderately, and these fakes are presented as "real" instances to the discriminator. Such deccits are introduced adaptively using an overfitting heuristic À defined by the discriminator raw output logits. The sugmentationn/decptioo probability p can be adaptively controlled throughout training.
training, As can be observed, both training processes start smoothly, and distributions of discriminator outputs for the real and generated images overlap at the early stage. As the training progresses, the discriminator, which only has access to limited data (7k images, 10% of full data), experiences diverged predictions much more rapidly, and the average sign boundary turns out to be more apparent. This divergence in prediction shows that becomes increasingly confident in classifying real and fake samples. At the late stage of training, D can even judge all the input samples correctly with high confidence. Meanwhile, the evaluation FID [1] scores (lower is better) deteriorate, consistent with the divergence of D's predictions. The phenomena above demonstrate how a discriminator gets overfitted quickly with limited data. As can be seen from the FID curves, the overfitting directly influences training dynamics and the convergence of G, leading poor generation performance.
3.1 Adaptive Pseudo Augmentation
The generator itself naturally possesses the capability to counteract discriminator overfitting. To harness this capability, our method employs a GAN to augment itself using the generated samples (see feed the samples generated by G into the limited real data form pseudo-real set. The fake images in this set will be adaptively presented to D, pretending themselves as real data. The goal here is to deceive D with the pseudo-real set and consequently suppressing its confidence in distinguishing real and fake images.
Figure 3 Specifically,
Blindly presenting the fake images as reals to D may weaken the fundamental ability of D in adversarial training. In our approach, the deceit is introduced adaptively to avoid any potential adverse effects. To moderate the deception, perform pseudo augmentation based probability. p E [0. 1). that quantifies the deception strength. Specifically, the pseudo augmentation will be applied with the probability p be skipped with the probability 1 1-p.
We note that the overfitting state of D is dynamic throughout the training (see Figure 2 is intuitive to let the deception probability y be adjusted adaptively based on the degree of overfitting. Ideally, should be adjusted without manual tuning regardless of data scales and properties. To achieve this goal, inspired by ADA (18), apply overfitting heuristic À that quantifies the degree of D's overfitting. We extend the control scheme ADA and provide three plausible variants:
E (sign (Droa)) (sign (Drak))
>r = E (sign (Dreat), Ay F (sign (Drake))
Xry
(1)
2
where Dreal and Diake are defined
Dreal logit (D (z)), Diake logit (D (G (z))),
(2)

---

where logit denotes the logit function. As shown in Figure 2 the X in Eq. estimates the portion of real images that obtain positive logit predictions by D, and that of generated images is captured by Ay. Besides, fr indicates half of the distance between the signs of the real and fake logits. For all these heuristics, our main experiments and study other variants in the ablation study.
represents no overfitting, and
means complete overfitting. We use X, in
The strategy of using À to adjust p is follows. We set threshold (in most cases of our experiments, = 0.6) and initialize to be zero. If A signifies too much/little overfitting regarding t (i.e., larger/smaller than t). the probability p will be increased/decreesedd by one fixed step. Using this step size, p can increase from zero to one in 500k images shown to D. We adjust once every four iterations and clamp p from below zero after each adjustment. this way, the strength of pseudo augmentation can be adaptively controlled based on the degree of overfitting (see Figure
3.2 Theoretical Analysis
Let p= (2) be the prior the input noise variable. The mapping from the latent space the image space is denoted as G (z). For sample x, D (x) represents the estimated probability x coming from the real data. To examine the rationality of APA, we analyze it in non-parametrie setting (9) where a model is represented with infinite capacity by exploring its convergence in the space of probability density functions. Ideally, the estimated probability distribution P9 defined by G should perfectly model the real data distribution Pdata without bias if given enough capability and training time.
Since the deception strength p is adaptively adjusted, to facilitate this theoretical analysis, make mild assumption that a is the expected strength that approximates the effect of dynamic adjustment during the entire training procedure. Since p E [0, 1), have 0 S Puax 1, where Pmax is the maximum of deception strength throughout training. Hence, the value function V (G, D) for the minimax two- woo-player game of APA can be reformulated as:
min max V (G, D) = (1 a:) (gpppsaaa()) [log D (x)] + a (x) [log D (G G D
(3)
+ (ewps(s) [log (1 - (GG
First, let us consider the optimal discriminator 9 for any given generator.
Proposition 1. If the generator G is fixed, the optimal discriminator D for APA is:
(1 a) Pdata (z) + a Pg (z)
De (x) (1 (x) Paata (x) + (1 + aa) Pg
(4)
Proof. Applying APA, given any generator G, the training objective of the discriminator is to maximize the value function V (G, in Eq. (3)
V VG, D) = (1 aa) / Paata (z) log D (x) +
f p- (2) Do (gg (=))dzz + / (z) D (=)))ds
(11 -0)) pdata (x) log D (x) + a log D (x) + (z) log (1 - D (x)))
= / [((1 - a) paata (x) + taap Ps (z)) log D (z) Pa (z) log (1 D (x)))
(S)
For any (m, n) RR (0.00) the function (y) log (v) + n log (1 (1-y) achieves its maximum in the range [0, at m m+n" Besides, the discriminator D is defined only inside of supp (Pdata) U supp (py), where supp is the set-theoretic support. Therefore, we conclude the proof for Proposition 1.
We have got the optimal discriminator D2 (z) in Eq. (4) that maximizes the value function V (G, D) given any fixed generator G. The goal of generator G in the adversarial training is to minimize the
value function V (G. D) in Eq. (3) when D achieves the optimum. Since the training objective of D can be interpreted maximizing the log-likelihood for the conditional probability P (Y y/
where Y estimates that x comes from Pdata (i.e.) = 1) or from Pg (i.e., 0), we reformulate virtual training criterion
9
C (C) (1 a) Buropaata [log D' (Ir) Exop, [log Da (G)) + Bewr, Dlo Da (=)))
(6)
= (1 - a) Eawpatta [log Dā (I)) + a Expp, [log Da (4)] + Exsp, [log (1 Dē (=)))
5

---

Then, let us consider the global minimum of C (G) trained with the proposed APA. Proposition 2. Applying APA, the global minimum the virtual training criterion C (G) is still achieved if and only if Pg = Pdata, where C (G) = log 4.
Proof. 1) Pg Pdata. we have D (z) 2 according to Eq. By inspecting Eq. ( at
D2 (z) = 2. we get C* (G) =
a) log + a log 2/2 + log = log + Log 2 = log 4.
2) To verify C* (G) is the global minimum of C (G). and it can only be achieved when Pg = Pdata as in the derivation of Eq. (5). we obtain:
-f ((1 aa) Pdata (z) - a Py (x)) log De (r)dr + / P9 (x) log (1 D2 (x))
C (G)
(7)
Observe that
loog 4 (1 a) (/ea)e/rppp Bempant log 2] Bamp, |- log 2| Bunp, [- loogg 2]
=- / ((1 (t) Pdata (x) + Pg (2:) log dr | Pg (x) log dr
(8)
Subtracting Eq. (8) from Eq.
/ ((1 a) paata (x) Pa (x)) 2og Do (x) + (z) (1 D3 (x))dz
(G) log 4
By substituting Eq. into Eq. ( we achieve:
(1 a) Pdata + (1 + a) Pg
C (G) - log 4 KLD ((1 a:) Pdata a Pg)
2
(10)
(1 - a)) Pdata + (1 + a)
+ KLD
Pg
2
where KLD is the Kullback-Leible Leibler (KL) divergence. Moreover, Eq. (00) further implies that the generation process of G by APA can be regarded as minimizing the Jensen-Shannon (JS) divergence between the smoothed data distribution and the generated distribution:
(G) log + 2- JSD (((1 ax) Pasta + a Pτ) llp))
(11)
For the two distributions P and Q, their JS divergence JSD (PIIQ) 2 0 and JSD (PII)) 0 if and only if P = Q. Therefore, for 0 S a < Pmax < 1, we obtain that C (G) = log the global minimum of C' (G), and the only solution is (1 a) Pdata + Ps Pg. i.e., Pg Pdata Q.E.D.
Given the proof in 9 if G and 1) have enough capacity to reach their optimum, Proposition indicates that the generated distribution Py can ideally converge to the real data distribution Paata- So far, we have proved the convergence of G trained with our proposed APA, which can perfectly model the real data distribution given sufficient capability and training time. Besides, the JS divergence term between the smoothed data distribution and the generated distribution Eq. (IT) implies that the judgment of D may be moderated to alleviate overfitting. These conclusions explain the rationality of the proposed APA for training GANs with limited data.
Experiments
Datasets. We use four datasets in our main experiments: Flickr-Faces-HQ (FFHQ) [19 with 70. 000 human face images, AFHQ-Cat [8] with 5, 153 cat faces, Caltech-UCSD Birds-200-2011 (CUB) [43 with 11, 788 images of birds, and Danbooru2019 Portraits (Anime) [4] with 302, 652 anime portraits. We exploit some of their artificially limited subsets under different settings. All the images are resized to moderate resolution of 256 x 256 using high-quality Lanczos filter 21 to reduce the energy consumption for large-scale GAN training while preserving image quality. Additional dataset details, including the data source and license information, can be found in supplementary material.
Evaluation metrics. We follow the standard evaluation protocol Specifically, we use the Fréchet Inception Distance (FID, lower is better) [i1, which quantifies the distance berween distributions for the real and generated images. FID evaluates the realness of synthesized images. Following [|] [8] calculate FID for the models trained with limited
for the quantitative evaluation.

---

StyleGAN2
APA (Ours)
StyleGAN2
APA (Ours)
AFHQ-Cot-5k (5,153 img, 100% data)
FFHQ-SK (5,000 img, ~7% dato)
StyleGAN2
APA (Ours)
StyleGAN2
APA (Ours)
Anime-5k (5,000 img, ~2% data)
CUB-12K (11,788 img, 100% dota)
Figure The proposed APA improves StyleGAN2 (20) synthesized results (256 x 256, no truncation) on various datasets with limited data amounts. We randomly select subsets to confine the size of large datasets (i.e., FFHQ-5k [19] and Anime-5k [ and directly use small datasets (i.e., AFHQ-Cat-
5k ( and CUB-12k 43 whose data amount is already insufficient for StyleGAN2.
Table 1: The FID (lower better) and IS (higher is better) scores (256 256) of our method compared state-of-the-art StyleGAN2 various datasets with limited data amounts.
AFlIQ-Cat-5k
FFHQ-SKK
Anime-5k
CUB 12k
Method
F
IST
FID
IS
FID
IS
FID
IS
StyleGAN2 [20]
7.737
1.825
37,830
4,018
23.778
2.289
23.437
5.812
APA (Ours)
4.876
2.156
13.249
4.487
13.089
2.330
12.889
5,869
datasets using 50k generated images and all the real images in the original datasets. We also apply the Inception Score (IS, higher is better) [38]. IS considers the clarity and diversity of generated images.
Implementation details. We choose the state-of-the StyleGAN2 (20) the backbone to verify the effectiveness of APA on limited data. We use the default setups of APA provided in Section 3.1 unless specified otherwise. For fair and controllable comparison, and run the experiments from scratch using official code. All the models are trained on 8 NVIDIA Tesla V100 GPUs. Please refer to the supplementary material for more implementation details and additional benchmark results (e.g., performance on BigGAN 5)).
reimplement all baselines
4.1 The Effectiveness of APA
Effectiveness on various datasets. The comparative results of StyleGAN2 on various datasets with limited data amounts are shown in Figure 4 The quality of images synthesized by StyleGAN2 deteriorates under limited data. Ripple artifacts appear the cat faces and human faces, and the facial features of the anime faces are misplaced. On the bird dataset with heavy background clutter, the generated images completely distorted albeit trained with more data. The proposed APA significantly ameliorates image quality on all these datasets, producing much more photorealistic results. The quantitative evaluation results reported in Table Applying APA contributes to performance boost of FID and IS in all cases, suggesting that the synthesized images are with higher quality and diversity different datasets.
Effectiveness given different data amounts. The comparative results on subsets of FFHQ [T9], with varying amounts of data, shown in Figure The corresponding quantitative test results

---

1k (*1.4% data)
5k (*7% data)
7k (10% data)
70k (100% data)
APA (Ours) StyleGAN2
Figure 5: The effectiveness of APA to improve StyleGAN2 (0) synthesized results (256 256, no truncation) on the subsets of FFHQ 19] with different data amounts.
Table 2: The FID (lower is better) and IS (higher is better) scores (256 256) of our method on with different data amounts.
StyleGAN2 trained using the subsets of FFHQ [9
(~ 1.4%)
5k
7%)
(10%)
70k (100%)
Method
F
FID
IS
FID
IS
FID
IS
StyleGAN2 [20
86,407
2,806
37,830
4.018
27.738
4.264
3.862
5.243
APA (Ours)
45.192
4.130
13.249
4.487
10.800
4.860
3.678
5.336
presented in Table 2 APA improves the image quality and metric performance in all cases. Notably, the quality synthesized images by APA on 5k/7k data visually close to StyleGAN2 results on the full dataset while with order of magnitude fewer training samples. As for the quantitative results, the score APA on Ik data better than that StyleGAN2 on 5k data, and both metrics of APA on 5k data outperform StyleGAN2 results on 7k data. APA can improve StyleGAN2 performance on the full dataset, further indicating its potential.
Overfitting and convergence analysis. As shown in Figure 6. the divergence of StyleGAN2 dis- criminator predictions can be effectively restricted on FFHQ-7k (10% of full data) by applying the proposed APA. The curves of APA on FFHQ-7k become closer to that of StyleGAN2 on FFHQ-70k, suggesting the effectiveness of APA in curbing the overfitting of the discriminator. Besides, APA improves the training convergence of StyleGAN2 overfitting and convergence analysis can be found in the supplementary material.
limited data, shown by the FID curves. More
4.2 Comparison with Other Solutions for GAN Training with Limited Data
Performance and compatibility. We compare the proposed APA with representative approaches designed for the low-data regime, including ADA [8] and LC-reuularization (LC-Reg) (42, which perform standard data augmentations and model regularization, respectively. The results are reported As a single method, APA outperforms previous solutions in most cases, effectively
Table 3
Den
Decriminator outputs
Decriminator outpus
100
00
20
3000
Training progress (<img)
10039 20333
20080
23800
5500
Training progress (ximg)
10030
19333
20002
25000
2000
Training progress (kimg)
10003
35000
20000 25000
(a)
Figure 6: The overfitting and convergence status of APA compared to StyleGAN2 (SG2) on FFHQ 19 (256 256). (a) The discriminator raw output logits of StyleGAN2 the full (70k) or limited (7k) datasets. (b) The discriminator raw output logits of StyleGAN2 and APA on the limited (7k) dataset. (c) The training convergence shown by FID.

---

Table 3: The FID (lower is better) and IS (higher is better) scores (256 x 256) of our method compared other state-of-the-art solutions designed for GAN training with limited data on StyleGAN2. The bold number indicates the best value, and the underline marks the second best.
AFHQ-Cat-5k
FFHQ-5k
FFHQ- 70k (full)
Method
FID 4
IS f
FID 1
1
FID 1
1
StyleGAN2 20
7.737
1.825
37.830
4.018
3.862
5.243
ADA [8
6,053
2.119
11.409
4.721
4.018
5.329
LC-Reg 42
6,699
1.943
35.148
3.926
3.933
5.312
APA (Ours)
4.876
2.156
13.249
4.487
3.678
5,336
ADA + APA (Ours)
4.377
2.169
8.379
4.849
3.811
5.321
Table 4: The FID (lower is better) and IS (higher is better) scores (256 256) of our method compared
to previous techniques for regularizing GANs on StyleGAN2 trained with FFHQ-5k [9).
Metric
StyleGAN2 120
Instance noise [40]
One-sided 38
APA (Ours)
FID
37.830
40.981
33.978
13.249
IS
4.018
4.231
4.029
4.487
improving the StyleGAN2 baseline on both the limited and full datasets. Although ADA [18
achieves slightly better results than our method on FFHQ-5K, yields a worse FID) score with StyleGAN2 on the full dataset. Applying LC-Reg 42 needs careful manual tuning, and its own effect on limited data not apparent compared to other methods.
It is noteworthy that APA is also complementary to existing methods based on standard data augmen- tations, e.g., ADA [8]. As can be observed in Table 3 APA can further boost the performance of StyleGAN2 with ADA 18] given limited training data, suggesting the compatibility of our approach with standard data augmentations. Combining ADA [8] and APA on FFHQ full data outperforms StyleGAN2 but is slightly inferior to applying APA solely. The degraded performance is mainly affected by ADA [8]. which we empirically found might slightly harm the performance when the training data is sufficient. Overall, these methods are not dedicated to improving performance under sufficient data. Nevertheless, as a beneficial side effect, the proposed APA may have this potential. More comparative results are included in our supplementary material.
Training cost. We compare the computational cost of APA against ADA 18 using the same basic official codebase. There is no parameter or memory increment for both methods. As for the time consumption, we test the training cost on 8 NVIDIA Tesla V100 GPUs. On FFHQ-5K (256 x 256), the average training time of the StyleGAN2 (20 baseline is (4.740 + 0.100) sec/kimg (i.e., seconds per thousand of images shown to the discriminator). The cost of our method is negligible. slightly increasing this value to (4.789 + 0.078) sec/kimg. As reference, the value for ADA [18] is (5.327 + 0.116) sec/kimg, spending additional time for applying external augmentations.
4.3 Comparison with Previous Techniques for Regularizing GANs
As mentioned in Section 2. APA is closely related to previous techniques for regularizing GANs. The
comparative results on APA and some representative conventional techniques, i.e., instance noise [40]
and one-sided label smoothing (LS) (38), are shown in Table 4 Applying instance noise [40] may not
boost the performance of StyleGAN2 much under limited data. One-sided label smoothing (LS) (38]
(with the real label of 0.9) outperforms StyleGAN2 but still has huge performance gap with
method. This further suggests the effectiveness and usefulness of APA.
4.4 Ablation Studies
Ablation studies on variants of APA. We study three key elements of APA, i.e. the overfitting heuristic X, the deception strength and the deception strategy. The version used in our main experiments is denoted as the "main" version, where the "main" version applies the deception strategy that is analogous to one-sided label flipping. As reported in T when using other variants of we suggested in Eq. ① (i.e. = X5 and À = > the models achieve comparable performance as the main version. The FID score becomes even better for À Xr indicating the flexibility of our provided heuristics to extend and modify APA. More interestingly, even fixed moderate deception probability (c.g., p 0.5) can still work much better
Ar, and p adjusted adaptively. Besides,

---

(256 x 256). We study three key elements of APA, i.e. the overfitting heuristic X, the deception strength p. and the deception strategy. The "main" denotes the main version used in our previous experiments (i.e., X Xr, is adaptively adjusted, and the deception strategy is analogous to sided label flipping).
Table 5: Ablation studies on variants of APA FFHQ-5K 19
Metric
StyleGAN2
main
A5
À A=A., X-f
p (fix
two-sided
FID
37.830
13.249
13.470
12.679
14.632
15,440
IS f
4.018
4.487
4,420
4.412
4.403
4.167
Table 6: Ablation studies on the threshold t on FFHQ-5K [19] (256 x 256). The "main" denotes
the main version used in our previous experiments (i.e., t = 0.6).
Metric
StyleGAN2
t= 0.4
f 0.6 (main)
== 0.8
FID IS f
37.830
13.687
13.249
13.984
4.018
4.418
4.487
4.395
than original StyleGAN2 on limited data, albeit slightly inferior to the adaptively adjusted p. This implies the importance of the pseudo augmentation, and the adaptive control scheme further boost performance without manual tuning. As for the deception strategy, we empirically observe that two-sided label flipping can still outperform StyleGAN2 but is inferior the main version.
Ablation studies on the threshold t. We further provide the ablation studies on the threshold value in T The version used in our main experiments is denoted as the "main" version, where t = 0.6. can be seen that the models with different values of achieve comparable results, outperforming the StyleGAN2 baseline. On FFHQ-5K (256 256), ł = 0.6 could be more plausible choice. For further explanation, we use = 0.6 as the default value since works well in most cases. In practice, the value of t could be further adjusted to achieve chosen when one has fewer data. This means the deception strength can be adjusted to increase more rapidly since the discriminator is more prone to overfitting when the data amount is fewer.
better results. Empirically, smaller t can be
5 Discussion
We have shown the effectiveness of the proposed adaptive pseudo augmentation (APA) for state- of-the-art GAN training with limited data empirically. With negligible computational cost, APA achieves comparable or even better performance than other types of solutions on various datasets. APA is also complementary existing methods based on standard data augmentations.
Limitations. Despite promising results, the quality synthesized images by APA on the datasets with extremely limited data amount (e.g.. hundreds of images) can still be improved. Besides, certain datasets such as FFHQ-5k, applying APA solely may be slightly inferior to approaches based standard data augmentations. Since we do not apply any external augmentations, these two limitations are both due to the insufficiency of the dataset's intrinsic diversity. These limitations may be approached in the future in two ways: 1) Incorporating better standard data augmentations to APA. 2) Exploring the issue of data insufficiency from the generator aspect, e.g., using a multi-modal generator [39 to enhance diversity. In addition, we only theoretically verified the convergence and rationality of APA. In future work, the theoretical analysis on the effectiveness of APA could be further explored.
Broader impact. On the one hand, the effectiveness of APA with negligible computational cost will benefit the practical deployment of GANs, especially in the low-data regime. APA may also extend the breadth and potential of solutions to training GANs with limited data and benefit downstream tasks, such conditional synthesis. On the other hand, APA may also bring potential concerns on its capability to ease the higher-quality fake media synthesis using only limited data, as technology is usually a double-edged sword. However, we believe that these concerns can be resolved by developing better media forensics methods and datasets as countermeasures.
Acknowledgments and funding disclosure. This study is supported under the RIE2020 Industry Alignment Fund - Industry Collaboration Projects (IAF-ICP) Funding Initiative, as well as cash and in-kind contribution from the industry partner(s).
10

---

References
[1] GAN,
GAN/ Accessed: 2019-08-16.
[2] Martin Arjovsky and Léon Bottou. Towards principled methods for training generative adversarial networks. In ICLR, 2017.
[3] Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In ICML, 2017.
[4] Gwern Branwen, Anonymous, and Danbooru Community, Danbooru2019 Portraits: A large-scale anime head illustration dataset. https://wwx wwwrg.wwrng..ommmm 2021-04-10.
Accessed:
[5] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In ICLR,, 2018.
[6] Soumith Chintala. How to train a GAN? NeurlPS workshop on adversarial training. https://ssii 2016. Accessed: 2021-08-03.
Yunjey Choi, Minje Choi, Munyoung Kim, Jung-Woo Woo Ha, Sunghun Kim, and Jacgul Choo. StarGAN: Unified generative adversarial networks for multi-domain image-to-image translation. CVPR, 2018.
[8] Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung-Wood fung-Woo Ha. StarGAN v2: Diverse image synthesis for multiple domains. In CVPR, 2020.
[9] lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farke Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NeurlPS, 2014.
[10] Ishaan Gulrajani. Faruk Ahmed, Martin Arjovsky. Vincent Dumoulin, and Aaron Courville. Improved training of wasserstein gans. In NeurlPS, 2017.
[11] Martin Heusel, Hubert Ramsauer. Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by two time-scale update rulc converge to local nash equilibrium. In Neur/PS, 2017.
[12] Xun Huang, Ming- Yu Liu, Serge Belongie, and Jan Kautz. Multimodal unsupervised image-to-image translation. In ECCV, 2018.
[13] Phillip Isola, Jun- Yan Zhu, Tinghui Zhou, and Alcxci A Efros, taage-to-image translation with conditional adversarial networks. In CVPR, 2017.
[14] Simon Jenni and Paolo Favaro. On stabilizing generative adversarial training with noise. CVPR, 2019.
[15] Liming Jiang. Bo Dai. Wayne Wu, and Chen Change Loy. Focal frequency loss for image reconstruction and synthesis. ICCV, 2021.
[16] Liming Jiang, Changxu Zhang, Mingyang Huang, Chunxiao Liu, Jianping Shi, and Chen Change Loy. TSIT: simple and versatile framework for image-to-image translation. In ECCV, 2020.
[17] Tero Karras. Timo Aila. Samuli Laine, and Jaakko Lehtinen. Progressive growing of GANs for improved quality, stability, and variation. In ICLR, 2018
[18] Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. In NeurlPS, 2020.
[19] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In CVPR, 2019.
[20] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of StyleGAN. In CVPR, 2020.
[21] Cornelius Lanezos. An iteration method for the solution of the eigenvalue problem of linear differential and integral operators. United States Governm. Press Office Los Angeles, CA. 1950.
[22] Hsin- Ying Lee, Hung- Yu Tseng, Jia-Bin Huang, Maneesh Singh, and Ming-Hsuan Yang. Diverse image- to-image translation via disentangled representations. In ECCV, 2018
[23] Jianhua Lin. Divergence measures based the Shannon entropy. TIT, 37:145-151, 1991.
[24] Ming- Yu Liu, Thomas Breuel, and Jan Kautz. Unsupervised image-to taggae translation networks. In NewrIPS, 2017.
11

---

[25] Xihui Liu, Guojun Yin, Jing Shao, Xiaogang Wang, and Hongsheng Li. Learning to predict layout-to-image conditional convolutions for semantic image synthesis. In NeurlPS, 2019.
[26] Xudong Mao, Qing Li, Haoran Xie, Raymond YK Lau, Zhen Wang, and Stephen Paul Smolley. Least squares generative adversarial networks. In ICCV, 2017.
[27] Lars Mescheder, Andreas Geiger, and Sebastian Nowozin. Which training methods for gans do actually converge? In ICML,, 2018.
[28] Mehdi Mirza and Simon Osindero. 1rXiv:1411.7784 2014.
Conditional generative adversarial nets. arXiv preprint,
[29] Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In ICLR, 2018
[30] Sebastian Nowozin, Botond Cseke, and Ryota Tomioka, f-GAN: Training generative neural samplers using variational divergence minimization. In NeurIPS, 2016.
[31] Xingang Pan, Xiaohang Zhan, Bo Dai, Dahua Lin, Chen Change Loy, and Ping Luo. Exploiting deep generative prior for versatile image restoration and manipulation. In ECCV, 2020.
[32] Taesung Park, Ming- Liu, Ting-Chun Wang, and Jun-Yan Zhu, Semantic image synthesis with spatially- adaptive normalization. In CVPR, 2019.
[33] Patashnik, Zongze Wu, Bli Shechtman, Daniel Cohen- Or, and Dani Lischinski. StyleCLIP: Text-driven manipulation of stylegan imagery. In ICCV, 2021.
[34] Karl Pcarson. X. on the criterion that given system of deviations from the probable in the case of correlated system of variables is such that it can be reasonably supposed to have arisen from random sampling. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science, 50:157- 175, 1900.
[35] Alec Radford, Luke Metz. and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint, arXiv:1511.06473 2015.
[36] Elad Richardson. Yuval Alaluf, Or Patashnik, Yotam Nitzan, Yaniv Azar, Stav Shapiro, and Daniel Cohen-On Or. Encoding in style: a stylegan encoder for image-to- to to-mage translation. In CVPR, 2021.
[37] Kevin Roth, Aurelien Lucchi, Scbastian Nowozin, and Thomas Hofmann. Stabilizing training of generative adversarial networks through regularization. In NewrIPS, 2017.
[38] Tim Salimans, Ian Goodfellow, Wojeiech Zaremha, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training GANs. In NeurlPS, 2016.
[39] Omry Sendik, Dani Lischinski, and Daniel Cohen-Or. Unsupervised multi-modal styled content generation. arXiv preprint, arXiv:2001.08 006400, 2020.
[40] Casper Kaac Sonderby, Jose Caballero, Lucas Theis, Wenzhe Shi, and Ferene Huszár. Amortised map inference for image rusou-ncn.. In ICLR, 2017.
[41] Ngoc-Trung Tran, Viet-Hung Tran, Ngoc-Bao Nguyen, Trung-Kien Nguyen, and Ngai-Man Cheung. On data augmentation for gan training. TIP, 30::::::812200 1888888-1897, 2021.
[42] Hung-Yu Yu Tseng, La Jiang, Ce Liu, Ming-Hsuan Yang, and Weilong Yang. Regularizing generative adversarial networks under limited data. CVPR, 2021.
[43] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The Caltech-UCSD 20rds-200-2011 Dataset. 2011.
[44] Ting-Chun Wang, Ming- Yu Liu, Jun-Yan Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. High- resolution image synthesis and semantic manipulation with conditional GANs. In CVPR, 2018.
[45] Ryan Webster, Julien Rabin, Loic Simon, and Frédéric Jurie. Detecting overfitting of deep generative networks via latent recovery. In CVPR, 2019.
[46] Weihao Xia. Yulun Zhang, Yujiu Yang, Jing-Hao Xue, Bolei Zhou, and Ming- Yang. GAN inversion: A survey, arXiv preprint, arXiv: 2101.05278. 2021.
[47] Yuanbo Xiangli, Yubin Deng. Bo Dai, Chen Change Loy, and Dahua Lin. Real or not real, that is the question. In ICLR, 2020.
12

---

[48] Han Zhang, Zizhao Zhang. Augustus Odena, and Honglak Lee. Consistency regularization for generative adversarial networks. In ICLR, 2020.
[49] Junbo Zhao, Michael Mathieu, and Yann LeCun. Energy-hased generative adversarial networks. In ICLR, 2017.
[50] Shengyu Zhao, Zhijian Liu, Ji Lin, Jun-Yan Zhu, and Song Han. Differentiable augmentation for data- efficient gan training. In NeurIPS, 2020.
[51] Zhengli Zhao, Sameer Singh, Honglak Lee, Zizhao Zhang, Augustus Odena, and Han Zhang. Improved consistency regularization for GANs. In AAAI, 2021.
[52] Zhengli Zhao, Zizhao Zhang, Ting Chen, Sameer Singh, and Han Zhang. Image augmentations for GAN training. arXiv preprint, arXiv:2006.02599,, 2020.
[53] Yan Zhu, Taesung Park, Phillip Isola, and Alexci A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In ICCV, 2017.
13


Reviews:
[{"review_id": "oniArssDvt", "reviewer": "Reviewer_GodP", "summary": "This paper tries to solve discriminator overfitting problem.\n\nThe authors propose adaptive pseudo augmentation (APA).\n* APA employs the generator itself to augment the real data distribution with fake images.\n  * i.e., Fake images are presented as \"real\" instances to the discriminator.\n* Adaptiveness comes from measuring overfittingness of the discriminator.\n* APA has similar theoretical properties to the original GAN, but with $(1-\\alpha)p_\\text{data} + \\alpha p_g = p_g$.\n\nAPA improves systnesis quality in the limited-data regime.\n* on FFHQ, AFHQ-Cat, CUB, Danbooru\nThe authors provide a theoretical analysis of APA, similar to the original GAN.\n", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality.\n* (+) This paper proposes a novel way to augment the real distribution by adding fake samples.\n  * Existing methods transforms images to augment the real and/or fake distributions.\n\nQuality.\n* (+) APA has sound theoretical groundings.\n* (+) APA improves StyleGAN2 in all cases (number of real samples).\n* (+) APA is roughly robust to choice of hyperparameters.\n* (-) Analyses are provided only on FFHQ 7k. Showing the similar trend on 1k and 5k would be better.\n* (-) Benchmark datasets are not as diverse as ADA.\n* (-) ADA is compared only on AFHQ-Cat, FFHQ-5k on FFHQ-70k.\n\nClarity.\n* (+) Everything is clear to understand.\n\nSignificance.\n* (+) APA is generally effective on different settings, even with existing standard augmentation methods.\n* (+) APA brings negligible computational cost.\n* (+) APA is simple.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes to both.", "review_text": "", "extra_scores": {"ethical_concerns": "n/a", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.5"}, "extra_text": {}}, {"review_id": "lXo9jg0WN2L", "reviewer": "Reviewer_Kz3j", "summary": "This paper introduces a data augmentation strategy for training GANs, which proves to be especially effective in the low data regime. The essence of the idea is to replace real images by images produced by the generator, using an adaptive replacement probability in the discriminator training step. By classifying some fake images as real, discriminator overfitting is prevented, which normally leads to very poor training performance and instability when very little data is available. The proposed method (named APA) improves over or can be combined with another recent data augmentation scheme for the low data regime (such as ADA).", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n-\tThe proposed method (APA) is simple and can be easily applied to other data modalities, besides images, in contrast to ADA which requires image-specific transformations. At the same time, it performs better most of the time and can be applied in combination with ADA. \n-\tThe proposed approach is quite effective and helps to stabilize the training under limited training data.\n-\tThe paper is sound and easy to read.\n\n\nWeaknesses:\n\n-  The core idea of the proposed method is adding label noise to the discriminator (switching the label of fake images to real), which is a well-known trick used for improving training stability of GANs [https://github.com/soumith/ganhacks]. One way was to flip the labels of the real images, adding pseudo-labels to the fake images, or to flip the labels of both fake and real with the fixed ratio. In the proposed approach, the labels are flipped for the fake images only. It’d be beneficial to see if the comparison can be provided between one-sided flipping techniques (adding pseudo-labels to either real or fake images) and a two-sided label flipping (adding pseudo-labels to both real or fake images). Also the paper should state clear that adding label noise is a well known technique to improve the stability of the GAN training.\n\n-\tThere are also other techniques that can “cripple” discriminator, such as one-sided label smoothing [37], instance noise [39] or progressive augmentation of the discriminator [PA, Zhang et al., NeurIPS’2019]. As the proposed APA approach is quite close to those techniques, I think the comparison with them is required. \n\n-\tThe clarity of  Section 3.1 (Method) can be improved: the description of the adjustment of t is a little vague. Take the sentence “If lambda signifies too much/little overfitting regarding t, the probability p will be increased/decreased by one fixed step.”. In particular, how much is too much overfitting regarding t? What exactly is the step size? It seems like the step size is chosen such that p increases to 1 after exactly 500K images or it is increased in every single step? \n\n-\tIt would be beneficial to see how the APA method performs under low amount of training data, when the number samples is less than 1k images. \n\n-\tTable 3: What happens if you combine ADA and APA on the full dataset?\n\n-\tIn Section 4.3, it would be helpful to see an ablation on the hyper-parameter t. How was threshold t=0.6 chosen?\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "yes, the limitations are addressed in the paper", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5h"}, "extra_text": {}}, {"review_id": "RixU-rzIBt", "reviewer": "Reviewer_e1RP", "summary": "The submission tackles the problem of training GANs with limited data.\nIt proposes to do so by adaptively regularizing GAN discriminators by replacing some real data by generated samples when the discriminator overfits, accoding to an overfitting heuristic.\n\nA study of the proposed objectives show that the method keep the classical GAN optima while alleviating overfitting of the discriminator.\n\nThe method is tested on various datasets with limited data, and compared, using a StyleGAN-2 backbone, against other state-of-the-art methods for training with limited data.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nOriginality:\nIt might be worth mentioning that feeding the discriminator with generated data with the wrong label (label flipping) was a fairly well-known trick in 2016, discussed for example at the NeurIPS 2016 GAN Workshop [1, 2]. Connexions [3] had been drawn with label smoothing [4] and instance noise [5]. Using adaptive heuristics is also discussed in the same workshop presentation, and had been found unpractical at the time.\nOf course, the proposed work is no less original, as modern GAN training dynamics have little to do with early implementations, they did not consider training in the low-data regime, and in any case, a detailed, peer-reviewed, report on label-flipping was missing. Still, discussing those connexions might be interesting.\n\nQuality: \nThe submission is technically strong. The proposed heuristic seems very ad hoc, but the method is demonstrated to be effective in a variety of settings, including when combined with other methods. Experiments are thorough, with multiple datasets and numbers of samples.\nIt is however tested only with the StyleGAN-2. As it is perhaps the most popular GAN variant, the study is still very interesting and relevant but could be even stronger if tried with BigGAN as well.\n\nClarity:\nThe submission is clear and easy to understand.\n\nSignificance:\nThe tackled problem is important (training with limited data) and the proposed method is a simple and very effective method that is likely to be of use to the community.\n\n----------\n[1] How to train a GAN?\nSoumith Chintala. NIPS 2016 Workshop on Adversarial Training\nhttps://sites.google.com/site/nips2016adversarial/\n\n[2] soumith. https://github.com/soumith/ganhacks#6-use-soft-and-noisy-labels\n\n[3] Instance Noise: A trick for stabilising GAN training\nFerenc Huszár. inFERENCe.\nhttps://www.inference.vc/instance-noise-a-trick-for-stabilising-gan-training/\n\n[4] Improved Techniques for Training GANs.\nTim Salimans, Ian J. Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, Xi Chen. NeurIPS 2016.\n\n[5] Amortised MAP Inference for Image Super-resolution.\n Casper Kaae Sønderby, Jose Caballero, Lucas Theis, Wenzhe Shi, Ferenc Huszár. ICLR 2017.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "One limitation that is not discussed is about the possible inadequacy of a theoretical analysis at optimality to draw conclusions about the effectiveness of the regularisation during training. This approach, however, is fairly common in works about GANs.\n\nOtherwise, limitations and societal impacts are adequately discussed.\n\n--------------------------------------------------------\nPost-rebuttal:\nI'd like to thank the authors for providing the additional clarification and additional results in their rebuttal.\nAfter reading the different reviews and responses, I keep my positive score. The method is simple and effective, backed with strong empirical evidence.\nI also concur with reviewer Kz3j's suggestion to clearly state and discuss the relation of the proposed method with related techniques (label smoothing, instance noise...), as well as the proximity between the objectives of training in low-data regime and reducing GAN instability.\nIn that regard, the results provided in the response to reviewer Kz3j and the discussions with reviewer GodP are already very interesting additions.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "E4lK6mkgPv", "reviewer": "Reviewer_zQEC", "summary": "This paper proposes a new method for GAN training with limited data, named adaptive pseudo augmentation (APA). The key idea originates from the empirical observation that, when trained with limited data, the GAN discriminator tends to overfit easily, leading to poor generation performance. Therefore, the authors propose to adaptively add generated samples to the training data, so as to counteract the overfitting during training. Accordingly, they present the APA that is developed based on the output of the discriminator. Empirical experiments demonstrate the effectiveness of the proposed APA.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality. The presented APA is believed new and sound. More importantly, APA is quite easy to implement in practice. Related work is adequately cited and discussed. However, in most of the experiments, APA is only compared with StyleGAN2 that is not tailored for a generation with limited data. The ADA results should be reported, for example in Table 2, to highlight the advantages of the presented APA.\n\nQuality. The presented APA is sound and seems to be a simple but powerful technique. The claims are well supported empirically. However, theoretically, it would be better if the authors discuss why applying APA leads to better performance than the vanilla GAN? After all, the optimum is the same.\n\nClarity. The paper is clearly written and quite easy to follow.\n\nSignificance. The presented APA is important for GAN training with limited data; it's likely to be reused by other researchers.\n\nOther comments:\n\nIn the experiments, why did you use Lanczos filter to do the preprocessing? Please elaborate on it.\n\n*********************************\nAfter rebuttal\n*********************************\nThanks for the detailed responses. Please address the concerns in the comments and revise the paper accordingly. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a very simple method of adaptively feeding generated instances into real data to prevent the overfitting of GAN Discriminator in the situation where the training data is limited. It is somewhat incremental in terms of novelty, considering existing label-noise techniques, and the algorithm in deciding the feeding probability is also somewhat heuristic. The theory also shows a trivial extension from original GAN paper. Nevertheless, all reviewers, including this AC, unanimously supported the acceptance of this paper because of its simplicity and strong empirical success. In the final version, please be sure to add comparison/discussion with the related technique mentioned by reviewers.
Decision: Accept (Poster)