Augmentation-Aware Self-Supervision for Data-Efficient GAN Training
Authors: Liang Hou
Qi Cao
Yige Yuan
Songtao Zhao
Chongyang Ma
Siyuan Pan
Pengfei Wan
Zhongyuan Wang
Huawei Shen
Xueqi Cheng
Venue: NeurIPS 2023 poster (2023)
Keywords: generative adversarial networks
limited data
self-supervised learning
TL;DR: This paper proposes an augmentation-aware self-supervised discriminator to improve the data efficiency of training GANs.

Abstract: Training generative adversarial networks (GANs) with limited data is challenging because the discriminator is prone to overfitting. Previously proposed differentiable augmentation demonstrates improved data efficiency of training GANs. However, the augmentation implicitly introduces undesired invariance to augmentation for the discriminator since it ignores the change of semantics in the label space caused by data transformation, which may limit the representation learning ability of the discriminator and ultimately affect the generative modeling performance of the generator. To mitigate the negative impact of invariance while inheriting the benefits of data augmentation, we propose a novel augmentation-aware self-supervised discriminator that predicts the augmentation parameter of the augmented data. Particularly, the prediction targets of real data and generated data are required to be distinguished since they are different during training. We further encourage the generator to adversarially learn from the self-supervised discriminator by generating augmentation-predictable real and not fake data. This formulation connects the learning objective of the generator and the arithmetic $-$ harmonic mean divergence under certain assumptions. We compare our method with state-of-the-art (SOTA) methods using the class-conditional BigGAN and unconditional StyleGAN2 architectures on data-limited CIFAR-10, CIFAR-100, FFHQ, LSUN-Cat, and five low-shot datasets. Experimental results demonstrate significant improvements of our method over SOTA methods in training data-efficient GANs.

Augmentation-Aware Self-Supervision for Data-Efficient GAN Training
Liang Hou 13.4, Qi Cao', Yige Yuan 1.3, Songtao Zhao', Chongyang Ma²,
Siyuan Pan', Pengfei Wan Zhongyuan Wang'. Huawei Shen Xueqi Cheng
CAS Key Laboratory of Al Safety and Security, Institute of Computing Technology, Chinese Academy of Sciences 2CAS Key Laboratory of Network Data Science and Technology.
Institute of Computing Technology, Chinese Academy of Sciences
University of Chinese Academy of Sciences "Kuaishou Technology 1 Lianghou960gmail. com
Abstract
Training generative adversarial networks (GANs) with limited data is challenging because the discriminator is prone to overfitting. Previously proposed differentiable augmentation demonstrates improved data efficiency of training GANs. However, the augmentation implicitly introduces undesired invariance augmentation for the discriminator since ignores the change of semantics in the label space caused by data transformation, which may limit the representation learning ability of the discriminator and ultimately affect the generative modeling performance generator. To mitigate the negative impact of invariance while inheriting the benefits of data augmentation, we propose a novel augmentation-awaat discriminator that predicts the augmentation parameter of the augmented data. Particularly, the prediction targets of real data and generated data are required to be distinguished since they are different during training. We further encourage the generator to adversarially learn from the self-supervised discriminator by generating nugmentation-predicee predictable real and not fake data. This formulation connects the learning objective of the generator and the arithmetic harmonic mean divergence under certain assumptions. We compare our method with state- of of-the-art (SOTA) methods using the class-conde conditional BigGAN and unconditional StyleGAN2 architectures on data-limited CIFAR-10, CIFAR-100, FFHQ. LSUN- Cat, and five low-shot datasets. Experimental results demonstrate significant improvements of our method over SOTA methods in training data-efficient GANs.!
the
self-supervised
1 Introduction
Generative adversarial networks (GANs) [10] have achieved great progress in synthesizing diverse and high-quality images in recent years [2, 17, 19, 20, 13]. However, the generation quality of GANs depends heavily on the amount of training data [47, 18]. In general, the decrease of training samples usually yields sharp decline in both fidelity and diversity of the generated images [39, 48]. This issue hinders the wide application of GANs due to the fact of insufficient data real-world applications. For instance, is valuable to imitate the style of artist whose paintings are limited. GANS typically consist of generator that is designed to generate new data and a discriminator that guides the generator to recover the real data distribution. The major challenge of training GANs under limited data is that the discriminator is prone to overfitting [47, 18], and therefore lacks generalization teach the generator learn the underlying real data distribution.
Our code is available at https://githube cww/liang--//..oo..o..a.iaooalaa http://aaangnuuuuuugeeaeeee gaan
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

In order to alleviate the overfitting issue, recent researches have suggested variety of approaches, mainly from the perspectives of training data [18], loss functions [40], and network architectures [28]. Among them, data augmentation- and extensibility. Specifically, DigfAut [47] introduced differentiable augmentation techniques for GANs, in which both real and generated data are augmented to supplement the training set of the discriminator. However, this straightforward augmentation method overlooks augmentation-relan semantic information, as it solely augments the domain of the discriminator while neglecting the range. Such a practice might introduces an inductive bias that potentially forces the discriminator to remain invariant to different augmentations [24], which could limit the representation learning of the discriminator and subsequently affect the generation performance of the generator [12].
methods have gained widespread attention due to its simplicity
In this paper, we propose a novel augmentation -waare self- supervised discriminator that predicts the augmentation parameter of augmented data with the original data as reference to address the above problem. Meanwhile, the self-supprvised discriminator is required to be distinguished between the real data and the generated data since their distributions are different during training, especially in the early stage. The proposed discriminator can benefit the generator in two ways, implicitly and explicitly. On one hand, the self-supervised discriminator can transfer the learned augmentation-awann knowledge to the original discriminator through parameter sharing. On the other hand, we allow the generator to learn adversarially from the self-supervised discriminator by generating augmentation- predictable real and not fake data (Equation (6)). We also theoretically analyzed the connection between this objective function and the minimization of a robust f-divergence divergence (the arithmetic harmonic mean divergence [37]). In experiments, we show that the proposed method compares favorably to the data augmentation counterparts and other state-of-the-art (SOTA) methods on common data-limited benchmarks (CIFAR-10 [21], CIFAR-100 [21], FFHQ [17], LSUN-Cat [45], and five low-shot image generation datasets [36]) based on the class-condittonal BigGAN [2] and unconditional StyleGAN2 [19] architectures. In addition, we carried out extensive experiments
to demonstrate the effectiveness of the objective function design, the adaptability to stronger data selection our method.
augmentations, and the robustness of
2 Related Work
In this section, we provide overview of existing work related to training GANs in data-limited scenarios. We also discuss methodologies incorporating self-supervised learning techniques.
2.1 GANs under Limited Training Data
Recently, researchers have become interested in freeing training GANs from the need to collect large amounts of data for adaptability in real-world scenarios. Previous studies typically fall into two main categories. The first one involves adopting a pre-trained GAN model to the target domain by fine-tuning partial parameters [41, 31, 42, 30]. However, it requires external training data, and the adoption performance depends heavily on the correlation between the source and target domains.
The other one focuses on training GANs from seratch with elaborated data-efficient training strategies. DiffAugment [47] utilized differentiable augmentation to supplement the training set to prevent discriminator from overfitting in limited data regimes. Concurrently, ADA [18] introduced adaptive data augmentation with a richer set of augmentation categories. APA [16] adaptively augmented the
real data with the most plausible generated data. LeCam- [40] proposed adaptive regularization for the discriminator and showed a connection to the Le Cam divergence [23]. [3] discovered
that sparse sub-network (lottery tickets) [5] and feature-level adversarial augmentation could offer orthogonal gains to data augmentation methods. InsGen [43] improved the data efficiency of training GANs by incorporating instance discrimination tasks to the discriminator. MaskedGAN [14] employed masking in the spatial and spectral domains to alleviate the discriminator overfitting issue. GenCo [7] discriminated samples from multiple views with weight-discrepancy and data-discrypancy mechanisms. FreGAN [44] focused on discriminating between real and fake samples in the high- frequency domain. DigGAN [8] constrains the discriminator gradient gap between real and generated data. FastGAN [28] designed a lightweight generator architecture and observed that a self-supervised discriminator could enhance low-shot generation performance. Our method falls into the second category, supplementing data augmentation--asse based GANs and can be also applied to other methods.

---

Original data
Wcolor (0.2, 0.5) Wiromelation = (0.1, 0.8)
cutout (0.4,06)
Figure 1: Examples of images with different kinds of differentiable augmentation (including the original unaugmented one) and their re-scaled corresponding augmentation parameters w [0, 1]
2.2 GANs with Self-Supervised Learning
Self-supervised learning techniques excel at learning meaningful representations without human- annotated labels by solving pretext tasks. Transformation-base self-supervised learning methods such as rotation recognition [9] have been incorporated into GANs to address catastrophic forgetting in discriminators [4, 38, 12]. Various other self-supervised tasks have also been explored, including jigsaw puzzle solving [1], latent transformation detection [33], and mutual information maximiza-
tion [25]. Moreover, ContraD [15] decouples the representation learning and discrimination of the discriminator, utilizing contrastive learning for representation learning and a discriminator head for distinguishing real from fake upon the contrastive representations. In contrast to ours, CR- FR-GAN [46] and ICR-GAL GAN proposed consistency regularization for the discriminator, which corresponds to an explicit pnprentttioonnnvaiiaa of the discriminator. both our proposed method and SSGAN-LA [12] belong to adversarial self-supervised learning, they differ in the type of self-supervised signals and model inputs. SSGAN-LA is limited categorical self-supervision [9], which incompati- ble with popular augmentation based GANs like DiffAAgment [47]. Our method is applicable for continuous self-supervision and integrates seamlessly with DiffAugment. Furthermore, continuous self-supervision have magnitude relationship and thus can provide more refined gradient feedback for the model to overcome overfitting in data-limited scenarios. Additionally, unlike SSGAN-LA, our method does not constrain the invertibility of data transformations (Theorem 1) because it additionally take the original sample as input for the self-supervised discriminator (Equation (5)).
Preliminaries
In this section, we introduce the necessary concepts and preliminaries for completeness of the paper.
3.1 Generative Adversarial Networks
Generative adversarial networks (GANs) [10] typically contain a generator G 2 - X that maps low-dimensional latent code z E Z endowed with a tractable prior p(z). c.g., multivariate normal distribution (O, I). high-dimensional data point x X. which induces a generated data distribution (density) (x) = Jz p(z))(x- G(z) ddz with the Dirac delta distribution 8(-), and also contain a discriminator D X + that is required to distinguish between the real data sampled from the underlying data distribution (density) Paata and the generated ones. The generator attempts to fool the discriminator to eventually recover the real data distribution, i.e., PG(x) Paata (x). Formally, the loss functions for the discriminator and the generator can be formulated as follows:
Ln EEx (x) (x (D()) + (M
(1)
LG = (g
(2)
Different real-valued functions f. h, and g correspond different variants of GANs [32]. For example,
the minimax GAN [10] can be constructed by setting f(x)
(og((((z)) and h(r) = -9(x)
loog a (a)) with the sigmoid function (æ) 1/ exp(-xx) In this study, we follow the practices of DiffAugment [47] to adopt the hinge loss [26], i.e., f(x) = h((=x) max(0, x)
and g(x)) = -r, for experiments based on BigGAN [2] and the log loss [10], ie.. f(x) = g(x)
log (r (z) and h(x)) = log(1 (z)),, for experiments based on StyleGAN2 [19].
3

---

CIFAR-10
CIFAR-100
0.55
BigGAN+DifAAu
0.8
SegSelf-BigGAN
Accuracy
0.5
0.75
0.45
0.7
100% data 20% data
10% data
100% data 20% data
10% data
Figure 2: Comparison of representation learning ability of discriminator between BigGAN + Dif- fAugment and our AugSelf- BigGAN on CIFAR-10 and CIFAR-100 using linear logistic regression.
AugSelf-BigGAN
3.2 Differentiable Augmentation for GANs
Diff'Augment [47] introduces differentiable augmentation T X S2 - X' parameterized by a randomly-sampled parameter w E Z with a prior p(w) for data-efficient GAN training. The parameter w determines exactly how to transfer a sample x to an augmented one & & Å' for the discriminator. After manually re-scaling (for E 1])), the parameters of all three kinds of differentiable augmentation used in DiffAugment for 2D images can be expressed as follows:
color: Wenlor = (Abrightness, Asnturation Acontrast) E [0, 1]3: translation: Wtranslation = (Itranslation Btranslation) E [0, 1]2: cutout: Weutout (Zoffeet Voffset) E 1/2.
Figure illustrates the augmentation operations and their parameters. Formally, the loss functions for the discriminator and the generator of GANs with DiffAugment are defined as follows:
Cda Ex-pins (x), (x) (D w))) wo) DDT
(3)
Cda Ez-o(t) (DTT)
where w can represent any combination of these parameters. We choose all augmentations by default, which means augmentation color, translation, and cutout are adopted for each image sequentially.
Method
Data augmentation for GANs allows the discriminator distinguish single sample from multiple perspectives by transforming it into various augmented samples according to different augmentation parameters. However, it overlooks the differences in augmentation intensity, such as color contrast. and translation magnitude, leading the discriminator to implicitly maintain invariance to these varying intensities. The invariance may limit the representation learning ability of the discriminator because it loses augmentation-relatee information (e.g., color and position) [24]. Figure 2 confirms the impact of this point on the discriminator representation learning task [4]. We argue that a discriminator that captures comprehensive representations contributes to better convergence of the generator [35, 22]. Moreover, data augmentation may lead to augmentation leaking in generated data, when using specific data augmentations such random 90-degree rotations [18, 12]. Therefore, our goal is to eliminate the unnecessary potential inductive bias (invariance to augmentations) for the discriminator while preserving the benefits of data augmentation for training data-cfficient GANs.
To achieve this goal, we propose a novel augmentation-aware self-supervised discriminator D & X 2 UΩ- that predicts the augmentation parameter and authenticity of the augmented data given the original data as reference. Distinguishing between the real data and the generated data with different self-supervision because they are different during training, especially in the early stage. Specifically, the predictive targets of real data and generated data are represented as w E Ω and - E EQ- respectively. They are constructed from the augmentation parameter w with different transformations, i.e., + -- Since the augmentation parameter is continuous

---

+/-
T(G(z): t
hp
w+ w
Z
G
((z))
X
Figure 3: Diagram of AugSelf-GAN. The original augmentation-base discriminator is D(T())) (()(T())) The augmentation-an aware self-supervised discriminator D(T()),)) (()((()))
((-)), where is our newly introduced linear layer with negligible additional parameters.
vector, we use mean squared error loss to regress it. The proposed method combines continuous self-supervised signals with real-vs-fake discrimination signals, thus can be considered as soft-label augmentation [12]. Comparison with self seff-supervision that does not distinguish between real and fake is referred Table in Appendix C. Notice that the predictive targets (augmentations) can be subset of performed augmentations (see Table 7 in Appendix C for comparison). Mathematically, the loss function for the augmentation-ann aware self superviud discriminator formulated as the following:
-
Ex,w [ID(T(x:W),x) - - Ez,. [IDTT(G())))) G(z), w
(5)
In our inplementations, the proposed self-supervised discriminator δ goo shares the backbone o X' > Rd with the original discriminator D v except the output linear layer y RRd - Ω UΩ- This purameter-sharing design not only improves the representation learning ability of the original discriminator but also saves the number of parameters in our model compared to the base model, c.g. 0.04% more parameters in BigGAN and 0.01% in StyleGAN2. More specifically, the self-supervised discriminator predicts the target based on the difference between learned representations of the augmented data and the original data, i.e., D(T(x;w) x) = p(o(T(x; w)) - o(x)) (see Table in Appendix C for comparison with other architectures). The philosophy behind our design is that the backbone a should capture rich (which necessitates the design of a simple head yp) and linear (inspiring us to perform subtraction the features) representations.
In order for the generator to directly benefit from the self-supervision of data augmentation, establish novel adversarial game between the augmentation-ave aware self-supervised discriminator and the generator with the objective function for the generator defined follows:
C = E2,w [I)(T(G()):)), G(x)) - w Ez, [ITTTGG());)) G(z)) w (6)
[
The objective function is actually the combination of the non non-saturating loss (regarding the generated data as real, ming _ (il ( (G(2)),, G(z)) - + and the saturating loss (reversely optimiz- ing the objective function of the discriminator, maxe Wz,w [[](())( w), G(z)) w - (see Table in Appendix C for ablation). Intuitively, the non- nat-saaung loss encourages the generator to produce augmentation- eggmeetttioo--prdiitabllee predictable data, facilitating fidelity but reducing diversity. Conversely, the saturating loss strives for the generator to avoid generating augmentation citteenttttoo--rdittaaell predictable data, pro- moting diversity the cost of fidelity. We will elucidate in Section how this formalization assists the generator in matching the fidelity and diversity of real data, ultimately leading to an accurate approximation of the target data distribution.
The total objective functions for the original discriminator, the augmentation-awwar self-supervised discriminator, and the generator of our proposed method, named AugSelf-GAN are given by:
min Lda + Xa L
D
min eda >gg L
(8)
5

---

8
Figure 4: Comparison of the func- tion f in different f-divergences. The f-divergence between two prob- ability distributions p(x) and g(x) defined as Dy (p((xx)lg(x)) with con-
KL
6
rLL.
f(p(x) /g(x)))))
JS
4
LC
AHM
Jx (p(x))xx vex function f: R200- + R satisfying f(1) = 0. The and y-axis denote the input and the value of the function in the f-divergence. The function f of the AHM divergence yields the most robust value for large inputs.
2
2
2
3
4
5
p(x)//(xx)
where the hyper-parameters are set as Xa A = 1 in experiments by default unless otherwise speci- fied (see Figure 6 for empirical studies). Details of objective functions are referred to Appendix
Theoretical Analysis
In this section, analyze the connection between the theoretical learning objective of AugSelf-GAN and the arithmetic harmonic mean (AHM) divergence [37] under certain assumptions.
Proposition 1. For any generator G and given unlimited capacity in the function space, the optimal augmentution-awwrr self-supervised discriminator Dr hus the form of:
(*x S Pásta (x, w, x) dw + PG(x, w, x)w dw
Pdaza (x, 8) + (x,
The proofs of all theoretical results (including the following ones) are deferred in Appendix A. Theorem Assume that w = - = c is constant, under the optimal self-supeeiied discrimina- tor D* optimizing the self-supe for the generator G is equivalent to:
min 4c MAn(Paata (x, x) (xx))xxx))),
(10)
where c = |cli is constant and MAH is the arithmetic harmonic mean divergence [37], of which the minimum is achieved if and only if px x) Pdata x) = px(() = Pdata(x)..
Theorem reveals that the generator of AugSelf-GAN theoretically still satisfies generative modeling, i.e., accurately learning the target data distribution, under certain assumptions. Although AugSelf- GAN does not obey the strict assumption, we note that this is not rare in the literature." Under this assumption, AugSelf-Cu GAN can be regarded as a multi- dimensional extension of LS-GAN [29] in terms of the loss function, while excluding that of the generator. Additionally, our analysis offers an alternative theoretically grounded generator loss function for the LS-GAN family.
Corollary 1. The following equality and inequality hold for the AHM divergence:
MAH MMH(paata (x, x)]lpa(Xx (x, x)) + MAH(PO(X, x) 8) (x, x)) = (ppara (x, x) x))
MAH (pdeza Xx, &) &)plpo((x x)) = 1 W (paatall (x, Xo x)) <
where A is the Le Cam (LC) divergence [23], and W is the harmonic mean divergence [37].
Corollary reveals an inequality MAH (pasta (x, ()pppp (x, x)) & &Dpaata x)]lpG(x, x)) because of non-negativity of AHM divergence MAn(PG(X, {)]lPdata (x, x)) O. Figure plots the function in AHM divergence and other common f-divergences shows better robustness of the function f than others for extremely large inputs p(x)/q(x) (x) D* (x)). which is likely for the optimal discriminator D* in data-limited scenarios.
the GAN literature. The AHM divergence
"Goodfellow al. [10] analyzed that saturated GAN optimizes the Jensen Shannon (JS) divergence, but in fact it uses non-saturated loss. LeCam-Gan GAN [40] showed connection between
Cam (LC) divergence [23] objective function based on fixed regularization, but in practice uses exponential moving average. The theoretical analysis of LS S-- is actually inconsistent with its generator loss function.

---

Table IS and FID comparisons of and CIFAR-100 with full and limited data. The best result is bold and the second best is underlined.
with state-of-the methods on CIFAR-10
100% training data
20% training data
10% training data
Method
IS (t)
FID (L)
IS (t)
FID (4)
IS (1)
FID (L)
BigGAN [2]
9.07
9.59
8.52
21.58
7.09
39.78
DiffAugment [47]
9.16
8.70
8.65
14.04
8.09
22.40
CR-GAN [46]
9.17
8.49
8.61
12.84
8.49
18.70
CIFAR-10
LeCam-GAN [40]
9.43
8.28
8.83
12.56
8.57
17.68
DigGAN [8]
9.28
8.49
8.89
13.01
8.32
17.87
Tickets [3]
8.19
12.83
16.74
MaskedGAN [14]
8.41
12.51
15,89
GenCo [7]
7.98
12.61
18.10+.06
AugSelf-BigGAN
9,43
7.68
8.98
10.97+.09
8.76
15.68
AugSelf-BigGAN+
9.27
7.54
9.08 1.04
9.95
8.79
12.76
BigGAN [2]
10.71
12.87
8.58
33.11
6.74
66.71
DiffAugment [47]
10.66
12.00
9.47
22.14
8.38
33.70
CR-GAN [46]
10.81
11.25
9.12
20.28
8.70
26.90
CIFAR-100
LeCam-GAN [40]
11.05
11.20
9.81
18.03
9.27
27.63
DigGAN [8]
11.45
11.63
9.54
19.79
8.98
24.59
Tickets [3]
10.73
17.43
23.80
MaskedGAN [14]
11.65
18.334.09
24.02...1
GenCo [7]
10.92
18.444.04
25.22
AugSelf- BigGAN
11.19
9.88
10.25
16.11
9.78
21.30+.15
BegSelf-BiigAAN
11.12
10.09
10.14
15.334.20
9.93 06
18.64+.09
Table 2: FID comparison of AugSelf-StylcGANN2 with competing methods FFHQ and LSUN- -Cat with limited training samples. The best result is highlighted in bold and the second best is underlined.
FFHQ
LSUN-Cat
Method
30K
10K
5K
1K
30K
10K
5K
1K
StyleGAN2 [19]
6.16
14.75
26.60
62.16
10.12
17.93
34.69
182.85
+ ADA [18]
5.46
8.13
10.96
21.29
10.50
13.13
16.95
43.25
+ DiffAugment [47]
5.05
7.86
10.45
25.66
9.68
12.07
16.11
42.26
AugSelf-StyleGAN2
4.95
6.98
9.69
23.38
9.22
11.98
14.86
36.76
AugSelf-StyleGAN2+
5.82
6.65
9.15
20.39
9.43
12.00
14.12
26.52
6 Experiments
We implement AugSelf-GAN based on DiffAugment [47], keeping the backbones and settings unchanged for fair comparisons with prior work under two evaluation metrics, IS [34] and FID [11]. The mean and standard deviation (if reported) are obtained with five evaluation runs at the best FID checkpoint. Each of experiments in this work was conducted
32GB NVIDIA V100 GPU.
6.1 Comparison with State-of-the-Art Methods
CIFAR-10 and CIFAR-100. Table reports the results on CIFAR-10 and CIFAR-100 [21]. These experiments are based on the BigGAN architecture [2]. Our method significantly outperforms the direct baseline DiffAugment [47] and yields the best generation performance in terms of FID and IS compared with SOTA methods. Notably, our method achieves further improvement when using stronger augmentation (see Table 5), i.e., AugSelf-BigGANg (translationf and cutout?).
FFHQ and LSUN-Cat. Table reports the FID results on FFHQ [17] and LSUN-Cat [45]. The seper-pprramter is X, 0.2. AugSelf-Gaa performs substantially better than baselines with the same network backbone. Also, stronger augmentation, i.e., SugSelf-StyleGANNY (translationf and cutout?), further improves the performance when training data is very limited.

---

Obama
Grumpy
Panda
AnimalFace Cat
AnimalFace Dog
DA
AS
Figure 5: Qualitative comparison between DiffAug (DA) and AugSelf (AS) on low-shot generation.
Table 3: FID comparison of AugSelf-Stpp Styeee--tyyyl with state-of-the-arr methods with and without pre-training on five low-shot datasets. The best result is in bold and the second best is underlined.
100-shot
AnimalFaces
Method
Pre-training?
Obama
Grumpy cat
Panda
Cat
Dog
Scale/shift [31]
Yes
50.72
34.20
21.38
54.83
83.04
MineGAN [42]
Yes
50.63
34.54
14.84
54.45
93.03
TransferGAN [41]
Yes
48.73
34.06
23.20
52.61
82.38
FreezeD [30]
Yes
41.87
31.22
17.95
47.70
70.46
StyleGAN2 [19]
No
80.20
48.90
34.27
71.71
130.19
+ AdvAug [3]
No
52.86
31.02
14.75
47.40
68.28
+ ADA [18]
No
45.69
26.62
12.90
40.77
56.83
APA [16]
No
42.97
28.10
19.21
42.60
81.16
+ DiffAugment [47]
No 46.87
27.08
12.06
42.44
58.85
AugSeff-StyleGANN
No
26.00
19.81
8.36
30.53
48.19
Low-shot image generation. Table shows the FID scores on five common low-shot image generation benchmarks [36] (Obama, Grumpy cat, Panda, AnimalFace cat, and AnimalFace dog). The baselines are divided into two categories according to whether they were pre-trained. Due its training stability, we can train AugSelf-StyleGANn
for 5k generator update steps ensure convergence. The hyper- parameters are As X 0.1 on Grumpy cat and AnimalFace cat, and the self-supervision color on all datasets. Impressively, SugSelf-StyleGANN methods by large margin all low-shot datasets, approaching SOTA FID scores to our knowledge. Figure visually compares the generated images of DiffAugment and AugSelf-GAN revealing the latter as superior in generating images with more diversity and fewer artifacts.
surpasses competing
6.2 Analysis of AugSelf-GAN
Fixed supervision. Table 4 reports the results of AugSelf-GAN on CIFAR-10 and CIFAR- 100 compared to the setup using fixed self-supervision, i.e., w = -- 1, which corresponds to the assumption in Theorem 1. AugSelf-GAN outperforms the fixed self-supervision terms of both IS and FID in all training data regimes. The reason is somewhat intuitive, as fixed self-supervision does not constitute self-supervised learning for the model, therefore it cannot enable the discriminator to learn semantic information related data augmentation to optimize the generator. Interestingly, the fixed one beats the baseline, which may be attributed to the multi-input [27] and regression loss [29].
Table 4: FID comparison with fixed self-supervision. The best results are highlighted in bold.
100% training data
20% training data
10% training data
Method
IS (t)
FID (L)
IS (t)
FID (L)
IS (t)
FID (L)
C10
Fixed (c 1)
9.25
8.01
8.70+.12
12.58
8.534.05
17.66
AugSelf-GAM GAN
9.43
7.68
8.98+.09
10.974.09
8.76
15.68
C100
Fixed (c = 1) 10.67
12.02+
9,944.06
17.70...7
9.50
22.84
AugSelf-GAN GAN
11.19+.00
9.88
10.25
16.11
9.784.08
21.30

---

Table 5: Study on stronger augmentation. The best is in bold and the second best is underlined.
100% training data
20% training data
10% training data
Method
IS (1)
FID (1)
IS (t)
FID (4)
IS (t)
FID (4)
DiffAugment
9.29
8.48
8.84
15.14
8.80
20.60
C10
trans.f cut
9.28
8.42.1.18
8.78
14.28
8.69
20.93
AugSelf-GAN
9.43
1.68
8.98
10.97..09
8.76.1.05
15.68 1..6
trans.f cut.
9.27
7.54
9.08+.04
9.95
8.79+.04
12.76
DiffAugment
11.02
11.49.1.1
9.45
24.98 1.48
8.50
34.92
C100
+ trans. cut
11,10=.08
+1.28
9.58+.05
24. 24.104.66
8.594.04
35,32+46
AugSelf-GAN
11.19
9.88+.07
10.25+.06
16. 16.15
9.78++08
21.30
+ trans.t cut
11.12
10.09+.05
10.14+.11
15.33
9.934.06
18.64
CIFAR-10
CIFAR-100
20
10% data
20% data
30
100% data
FID
15
20
10
10
0
0.1
0.2
0.5
1
2
5
10
0
0.1
0.2
0.5
2
5
10
A
Figure 6: FID curves with varying hyper-parameters À CIFAR-100. The hyper-paramest
10) on CIFAR-10 and corresponds to baseline BigGAN + DiffAugment.
Aa
X E 0.
Stronger augmentation. Translation and cutout actually erase parts of image information, which help prevent the discriminator from overfitting, but could suffer from underfitting if excessive. Our self-supervised task enables the discriminator to be aware of different levels of translation and cutout, which helps alleviate underfitting and allows to explore stronger translation and cutout. Table compares AugSelf-GAN with DiffAugment in this setting. Overall, when data limited, AugSelf-GAN can further benefit from stronger translation and cutout and achieve new SOTA FID results, while DiffAugment cannot. This implicitly indicates that our method enables the model under strong data augmentation.
learn meaningful features to overcome underfitting,
Hyper-parameters. Figure 6 plots the FID results of AugSelf-GAN with different hyper-parametee = Ao = Ag ranging from [0, 10) CIFAR-IO and CIFAR-100. Notice that À 0 corresponds to the baseline BigGAN DiffAugment. AugSelf-BigGAN BigGAN performs the best when À is near 1. It is worth noting that AugSelf-BigGAN outperforms the baseline even for À 10 with 10% and 20% training data, demonstrating superior robustness with respect to the hyper-parameter X.
7 Conclusion
This paper proposes a data-efficient GAN training method by utilizing augmentation parameters as self-supervision. Specifically, a novel self-supervised discriminator proposed for predicting the augmentation parameters and data authenticity of augmented (real and generated) data simultaneously. given the original data. Meanwhile, the generator is encouraged generate real rather than fake data of which augmentation parameters can be recognized by the self-supervised discriminator after augmentation. Theoretical analysis reveals a connection between the optimization objective of generator and the arithmetic harmonic mean divergence under certain assumptions. Experiments on data-limited benchmarks demonstrate superior qualitative and quantitative performance of the proposed method compared to previous methods.

---

Limitations. In our experiments, we observed less significant improvement of AugSelf-GAl under sufficient training data. Furthermore, its effectiveness depends on the specific data augmentation used. In some cases, inappropriate data augmentation may limit the performance gain.
Broader impacts. This work aims at improving GANs under limited training data. While this may result in negative societal impacts, such as lowering the threshold of generating fake content or exacerbating bias and discrimination due to data issues, we believe that these risks can be mitigated. By establishing ethical guidelines for users and exploring fake content detection techniques, one can prevent these undesirable outcomes. Furthermore, this work contributes to the overall development of GANs and even generative models, ultimately promoting their potential benefits for society.
Acknowledgements
We thank the anonymous reviewers for their valuable and constructive feedback. This work is funded by the National Natural Science Foundation of China under Grant Nos. 62272125, 62102402, U21B2046. Huawei Shen also supported by Beijing Academy of Artificial Intelligence (BAAI).
References
[1] Gulcin Baykal, Furkan Ozcelik, and Gozde Unal. Exploring deshufflegans in self-supervised generative adversarial networks. Puttern Recognition, 2022. ISSN 0031-3203.
[2] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In International Conference on Learning Representations, 2019.
[3] Tianlong Chen, Yu Cheng, Zhe Gan, Jingjing Liu, and Zhangyang Wang. Data-efficient gan training beyond (just) augmentations: A lottery ticket perspective. In Advances in Neural Information Processing Systems, 2021.
[4] Ting Chen, Xiaohua Zhai, Marvin Ritter, Mario Lucic, and Neil Houlsby. Self-supervised gans via auxiliary rotation loss. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[5] Xuxi Chen, Zhenyu Zhang, Yongduo Sui, and Tianlong Chen. (GAN)s can play lottery tickets too. In International Conference on Learning Representationss 2021.
[6] Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung-Woo Ha. Stargan v2: Diverse image synthesis for multiple domains. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2020.
[7] Kaiwen Cui, Jiaxing Huang, Zhipeng Luo, Gongjie Zhang, Fangneng Zhan, and Shijian Lu. Genco: Generative co-training for generative adversarial networks with limited data. Proceedings of the AAAI Conference on Artificial Intelligence, 2022. doi: 10. 10.16//aaai.vv v36i.. 19928.
[8] Tiantian Fang, Ruoyu Sun, and Alex Schwing. Diggan: Discriminator gradient gap regulariza- tion for gan training with limited data. In Advances in Neural Information Processing Systems, 2022.
[9] Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. In International Conference on Learning Representations, 2018.
[10] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems, 2014.
[11] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In Advances in Neural Information Processing Systems, 2017.
[12) Liang Hou, Huawei Shen, Qi Cao, and Xueqi Cheng. Self-supervised gans with label augmen- tation. In Advances in Neural Information Processing Systems, 2021.
10

---

[13] Liang Hou, Qi Cao, Huawei Shen, Siyuan Pan, Xiaoshuang Li, and Xueqi Cheng. Condi- tional GANs with auxiliary discriminative classifier. Proceedings of the 39th International Conference Machine Learning. Proceedings of Machine learning Research, 2022.
[14] Jiaxing Huang, Kaiwen Cui, Dayan Guan, Aoran Xiao, Fangneng Zhan, Shijian Lu, Shengcai Liao, and Eric Xing. Masked generative adversarial networks are data-efficient generation learners. In Advances in Neural Information Processing Systems, 2022.
[15] Jongheon Jeong and Jinwoo Shin. Training [gan]s with stronger augmentations via contrastive discriminator. In International Conference on Learning Representations, 2021.
[16] Liming Jiang, Bo Dai, Wayne Wu, and Chen Change Loy. Deceive d: Adaptive pseudo augmentation for gan training with limited data. In Advances in Neural Information Processing Systems, 2021.
[17] Tero Karras, Samuli Laine, and Timo Aila, A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[18) Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. In Advances in Neural Information Processing Systems, 2020.
[19] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[20] Tero Karras, Miik Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Alias-free generative adversarial networks. In Advances in Neural Information Processing Systems, 2021.
[21] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Master's thesis, Department of Computer Science, University of Toronto, 2009.
[22] Nupur Kumari, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu Ensembling off-the-shelf models for gan training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[23] Lucien Le Cam. Asymptotic methods in statistical decision theory. 2012.
[24] Hankook Lee, Kibok Lee, Kimin Lee, Honglak Lee, and Jinwoo Shin. Improving transferability of representations via augmentation-awai self-supervision. In Advances in Neural Information Processing Systems, 2021.
[25] Kwot Sin Lee, Ngoc- Trung Tran, and Ngai-Man Cheung. Infomax-gan: Improved adversarial image generation via information maximization and contrastive learning. In Proceedings the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2021.
[26] Jae Hyun Lim and Jong Chul Ye. Geometric gan. arXiv preprint arxi:: 1705.0287.. 2017.
[27] Zinan Lin, Ashish Khetan, Giulia Fanti, and Sewoong Oh. Pacgan: The power of two samples in generative adversarial networks. In Advances in Neural Information Processing Systems, 2018.
(28) Bingchen Liu, Yizhe Zhu, Kunpeng Song, and Ahmed Elgammal. Towards faster and stabilized (gan) training for high-fieelity few-shot image synthesis. In International Conference on Learning Representations, 2021.
[29] Xudong Mao, Qing Li, Haoran Xie, Raymond Y.K. Lau, Zhen Wang, and Stephen Paul Smolley, Least squares generative adversarial networks. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2017.
[30] Sangwoo Mo, Minsu Cho, and Jínwoo Shin. Freeze the discriminator: a simple baseline for fine-tuning gans, arXiv preprint 20X1v.2002.10964, 2020.

---

[31] Atsuhiro Noguchi and Tatsuya Harada. Image generation from small datasets via batch statistics adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019.
[32] Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-gan:
Training generative neural sam- plers using variational divergence minimization. In Advances in Neural Information Processing Systems, 2016.
[33] Parth Patel, Nupur Kumari, Mayank Singh, and Balaji Krishnamurthy, Lt-gan: Self-supervised gan with latent transformation detection. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2021.
[34] Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, Xi Chen, and Xi Chen. Improved techniques for training gans. In Advances in Neural Information Processing Systems, 2016.
[35] Axel Sauer, Kashyap Chitta, Jens Müller, and Andreas Geiger. Projected gans converge faster. In Advances Neural Information Processing Systems, 2021.
[36] Zhangzhang Si and Song-Chun Zhu. Learning hybrid image templates (hit) by information projection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2011.
[37] Inder Jeet Taneja. On mean divergence measures. Advances in Inequalities from probability theory and statistics. Nova, USA, 2008
[38] Ngoc-Trung Tran, Viet-Hung Tran, Bao-Ngoc Nguyen, Linxiao Yang, and Ngai-Man (Man) Cheung. Self-super gan: Analysis and improvement with multi-class minimax game. In Advances in Neural Information Processing Systems, 2019.
[39] Ngoc-Trung Tran, Viet-Hung Tran, Ngoc-Bao Nguyen, Trung-Kien Nguyen, and Ngai-Man Cheung. On data augmentation for gan training. IEEE Transactions on Image Processing, 2021. doi: 10.1 111.//0..222..34999.....
[40] Hung- Yu Tseng, Lu Jiang, Ce Liu, Ming-Hsuad Yang, and Weilong Yang. Regularizing generative adversarial networks under limited data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[41] Yaxing Wang, Chenshen Wu, Luis Herranz, Joost van de Weijer, Abel Gonzalez-Garcia, and Bogdan Raducanu. Transferring gans: generating images from limited data. In Proceedings of the European Conference on Computer Vision (ECCV), 2018.
[42] Yaxing Wang, Abel Gonzalez-Garcin David Berga, Luis Herranz, Fahad Shahbaz Khan, and Joost van de Weijer. Minegan: Effective knowledge transfer from gans to target domains with few images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[43] Ceyuan Yang, Yujun Shen, Yinghao Xu, and Bolei Zhou. Data-efficient instance generation from instance discrimination. In Advances in Neural Information Processing Systems, 2021.
[44] mengping yang, Zhe Wang, Ziqiu Chi, and Yanbing Zhang. Fregan: Exploiting frequency components for training gans under limited data. In Advances in Neural Information Processing Systems, 2022.
[45] Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint 150.0033006333555 2015.
[46] Han Zhang, Zizhao Zhang, Augustus Odena, and Honglak Lee. Consistency regularization for generative adversarial networks. In International Conference Learning Representations, 2020.
[47] Shengyu Zhao, Zhijian Liu, Ji Lin, Jun-Yan Zhu, and Song Han. Differentiable augmentation for data-efficient gan training. In Advances in Neural Information Processing Systems, 2020.
[48] Zhengli Zhao, Zizhao Zhang, Ting Chen, Sameer Singh, and Han Zhang. Image augmentations for gan training. arXiv preprint 22X5v22006.02555 2020.
12

---

A Proofs
Proposition 1. For any generator G and given unlimited capacity in the function space, the optimal augmenttation-aware self-supervised discriminator D has the form of:
x* = P w, x)w dw + PG(x, w, x)w dw
(9)
Pdata (x, 8) + Pa (x, x)
Proof. The objective function of the self-supervised discriminator can be written follows:
L = Ex,w [D((((; w 1 + Ez.w [ID(T(GZ)) G()) - w w- [13
Pdata (x, x) [ x) w + pa(x, x) (({, w dxdwdd.
Minimizing the integral objective is equivalent minimizing the objective on each data point:
(x, (xxx))/- [pests w. (() x) w+ + pa((,, w. x) x) - w w 11 dw.
Getting its derivative with respect to the self-supervised discriminator and let it equals to O, we have the optimal self-supervised discriminator as:
OCB
-f
= f |daaas (x, w. (Dx( w*) pc(x, w, x))((xxx))))) w w-] dw
0D(2,x)
J Pdaaaa (x, w, xt) dw + PG (x, w, x)w dw
= (xx
Pdata (x, x) + (x, x)
Theorem 1. Assume that w - is constant, under the optimal self-supervised discrimina- tor D*, optimizing the supf-supe for the generator G is equivalent to:
min 4c MAn(Paate (x, x) (x))p((xx))),
(10)
where c =|c| is constant and MAn is the arithmetic harmonic mean divergence [37], of which the minimum is achieved if and only if pa x) = Pdata(x, x) = ppp()) = (x)
Proof.
The objective function of the self-supervised task for the generator can be written as follows:
Bz
[T(T(G(Z):w) G(()) w
-11 Ez,w
[D(T(G()):))
G(z) w 113]
|/ʃ pG(x w:
|1x(x x) w 11 |10(x, x)
dxdwdx
/f p x)
c C(Pdata (x, x) - Pa(x, x))
c(pdata (x, x) - (x, x))
- c12
+ cll2 dxd&
Petata x, x) + PG (x, x)
Pdata x, x) + pG (x, x)
e. /f PG pxxxxx x)
Pdata (x, x) PG (x, x)
Pdata (x, x) PG (x, x)
11
+ +1/
dxdx
Pdata (x x) + PG (x, x)
Pduta (x, x) + PG [x, x)
// x))
2pc x)
2paata (x, x)
112
12
dxdx
Pdata Xx x) + PG (x, x)
Pdata (x, x) PG (x x)
4e. /f po(xx x)
Px x)2 2 - Pdaca (x, x)2
dxdx
(Pratal (x, x) + pG (x, x))2
[/PO(x,. x)
Pa x) - Pelata (x, x)
4cc
dxdx
Palata (x. x) + PG (x, x)
4c MAH(Pdatal (x, x) [lpp x)),
where = ||c||3 is constant scalar for the constant vector c -t = -w- And we have the optimal
generator PG xx x)
Pdata (x, x) = (x(x) Pdata (x) to minimize the AHM divergence.
13

---

Table 6: IS and FID of
with different self-supervised tasks (SS: Equations (19)
and (21); SS+: Equations (20) and (21); ASS: Equations (5) and (6)) on CIFAR-10 10 and CIFAR-100 with full and limited training data. The best bold and the second best is underlined.
100% training data
20% training data
10% training data
Method
IS (1)
FID (4)
IS (T)
FID (J)
IS (t)
FID (L)
Baseline
9.29 0.02
g.48
8.84
15. 15.14.0047
8.80
20.60
CIFAR- 10
SS (Ag 0)
9.28=0.0m
8.30
8.86+0009
13. 13.9640.19
8.62+0.09
20.94_0.42
SS
=.29
8.24
8.98+0.04
13.42+0036
8,6 +.69
19.40=0.28
SS+ (Ag 0)
9.25=0.06
8.1 8.11+0.18
8
13.8640.11
8.76440.6
19.52-0.24
SS+
8.26
8.85 10.05
13.81 1.81 1:027
8.63
19.85 20.18
ASS (dg 0)
9.12 0.05
8.90
8.82
13.65
8.52
18. 0.33
ASS
9.43 00314
7.68 10.06
8.98 +0.09
10.97
8. 8.76
15.68_0.2
Baseline
11.02
1 11,49+0.21
9.45+
24.98
8.50
34.92
SS (Ag =
11.04
11.48
9.81 +0.07
21.17+0.19
9.1 +0.09
30.48
CIFAR-
SS
10.16
11.04
9.99 Ln.09
20.72
9.23
31.54 1.12
SS+ (dg = 0)
10.84 10.84
11.48
9.95 10.07
21.22
9.14
31.02 1.00
SS+
10.94 10.10
10.94
9.88
22.7210.37
9.23
31.40
ASS (2, = =)
10.82
11.29+0.12
9.96+0.11
18.90+041
9.45
25.77
ASS
11.19
9.88
10.25
16.11
9.78
21.30
Corollary 1. The following equality and inequality hold for the AHM divergence:
MAH (Pasta (x, x)Ilpa x)) + MAH(Pa(X, [lllPsstaxx, x)) (ppata x) &))
MAH (Pásta (x, x)) x)) W (Pdata (X, (x S
where A the Le Cam (LC) divergence [23], and W is the harmonic mean divergence [37].
Proof. We first prove the first corollary:
MAn(Paata (x, x) (x, x))
< MAH (x, x) PC
(x. (x, x)) (AH (x, &) &)]lpaataaxx 8))
= /f (x( x)
PG xx x) Pdnta x)
dxdx + /I Pilata (x, x) Pdata (x, 8) pa(xp (x, x)
dxdx
Ddata xx x) + PG (x, x)
Pc(x, (x X) + Paata (x, x)
// pa (x, x)
Pe xx, x) Pdata (x,
Pdata (x, x) (x x) Pdaca (x,,(xxxx)) Xx x)
dxdx
Pa (x x) Pdata (x, x) dxdo
Pdaaa x, x) DG (x,X)
= (po x) Prtata (x, x3
dxd&
Pdata (x, 8) + PC (x, &)
A (Paataa (x, x) ||a (x, x))
where 0 < A(paata (x, & |lloprr (x, x)) S is the Le Cam (LC) divergence [23].
The following proves the second corollary:
0 < МАH (x, x) ())))>>l(xxxxm x))
lʃ pa (x, x) pa (x(xx x)
Pdata (x, x)
dxdx
Pdata (x, + PG (x, x)
// (x)(x
2pdata (x,
dxdx
Pdata (x, 8)
per -ppoxxx x)
- /I 2pdata (x, x)pG x, x)
dxdx
Pdata (x, x) + PGG (x,
1 - W (pdata (x, &) x)][pe(( (x, x)) < 1,
where 0 < ((Patta (x, x))lpo(x (x) S 1 is the well known harmonic mean divergence [37].
14

---

B Loss Functions
AugSelf-GAN adopts all kinds of augmentations of DiffAuument as the self-supervised signals by default, thus the self-supervised loss functions of the discriminator and the generator (Equations (5) and (6)) actually each comprise three sub self-supervised loss functions. Specifically, for AugSelf- GAN that uses color, translation, and cutout as self- self-supervision, the objective functions are:
min
D Aa
(Eroler
Ctranslation Desamolation
(11)
Deutoua
D Denttr Derandlation Donton
min Lda + Xg (R + LG
Ciranslation + +GG Ccutout).
(12)
G
where Coolor
Ctranslation
Ceutout Deutout
GG LG Ceranalation and Ce are defined as:
Devtae
Dieauslation
Coolor
= Ex,,
D Deolor (T (x(;:w), w color
+ m
D Dootor (T(G(Z);w) G(()) w
(13)
color
C otranslation
Ex
Dtranstation x) w translation
Derauelation
+ Ws
D Diranstation (T((()))) G(z))
(14)
translation
Leutout
Ex,w
Deutout (T(x;w), weutoutll
+ Ezzw
D Doutout (T(G(z):w) G(z)) cutout
(15)
LG
Coolor
En
D color (T(G(Z):w), G(z)) color
Exgw
D Doctor (T(G());w), G(z)) color
(16)
G
atranslation
En,w
(rraastation G(z)) w
cranslation
Eszw
Deranstation (T(G()):w) (rranslation G(z)) cranslation
(17)
G
centout
Deutout (T(G(z); G(z)) cutout
Ez w
Deutout (T(G(z): G(z)) cutout
(18)
where Declor = Pcolor G Φ. D Deranslation Ptranslation o 6, and D Deutout = Poutout o Φ share the backbone but differ in the heads Peolorr *trrassationn Pentout, respectively. For the simplicity of notations, we write Equations (5) and (6) in the main text as the objective function.
C
Ablation Studies
Self-supervised tasks. We introduce two non-adversarial self-supervised tasks for comparison. The first version is that the discriminator only learns self-supervision on real data, defined as:
S Exx-poss (o)pp)) [IT(T( (), -wll
(19)
The second version is that the discriminator learns self-supervised tasks both real and generated data simultancously, given by:
C = Ex [x)( x) w
Ez.w (IbCT(GG)):
(20)
For both versions, the generator is encouraged produce augmentation--roognzzablee data, follows:
c (e=p(( ()) w11
(21)
According to the self-supervised task, we denote different approaches as SS (Fquations (19) and (21)), SS+ (Fquations (20) and (21)), and ASS (Equations (5) and (6), short for adversarial self-supervised learning, i.c., the proposed AugSeIf-GAN).
15

---

Table IS and FID of
with different self self-superiise signals on CIFAR-IO and
CIFAR- 100 with full and limited training data. The self supervised signal to be predicted is marked by the symbol V. Notice that all methods adopt color, translation, and cutout as data augmentation.
Self- Self-Supervised Signals
100% training data
20% training data
10% training data
color
trans.
cutout
IS (1)
FID (4)
IS (t)
FID (C)
IS (1)
FID (1)
x
X
X
9.29
8.48
8.84
15.14
8.80
20.60
X
X
/
9.30
7.48
8.94
11.73
8.73
16.66
CIFAR-10
X
V
X
9.
7.51
8.95
11.82
8.80
16.27
J
V
9.
7.41
8.87
11.19
8.63
16.30
X
X
9.
7.57+
8.99
11.38
8.72+
16,50
X
V
9.41
7.51
8.92.1.12
11.20
8.83
15,55
J
X
9.42
7.43
8.91
11.19
8.58
15.17
V
9.43
7.68
8.98
10.97 1.09
8.76.4.0
15.68 4:26
X
X
11.02 07
11.49
9.45
24.98 48
8.50
34.92
/
11.16
10.03
10.18
19,45 4.62
9,46+1112
25.99
X
CIFAR-100
X
11.35 4.10
9.88+.0
10.01
18.39
9.29
26,50
V
11.26
9.75+.05
10.35
16,88
9,92+
23.09
X
11,20
10.01
9.90
18.162:62
9,39
21,48
V
11.17
10.12.
10.14
17.11
9.94
24.52
X
11.26.1.16
9.65
10.21
17.32
9.92
22.94
V
11.194.09
9.88
10.25
16.11
9.78
21.30
Table 8: IS and FID of AugSelf-BigGAN with different architectures and fusions of ip CIFAR-10 and CIFAR-100 100 with 10% training data. The best result is bold and the second best is underlined.
Input can be concatenation [o(x), ox)) Iddd subtraction o(x) <(x), and augmentation @(x).
CIFAR-10 10% data
CIFAR- 100 10% data
Architecture
Input
IS (t)
FID (4)
IS (t)
FID (L)
two-layer MLP
concatenation
8.54:10.07
8.65 18.47:40.37
3..15=0.47
two-layer MIP
subtraction
8.62-10.07
16.944-0.49
8.88
28.63-00.6
two-layer MLP
augmentation 8.78+10.05
19.06:10.81
8.73+0.10
29.40=0.91
linear layer
concatenation 8.85:10.11
17.52:0.200
9.3710.09
25.22-1.225
linear layer
subtraction
8.76:10.05
15.6810.26
9.78+0.08
21.30
linear layer
augmentation
8.66110.09
20.29410.36
9.9210.10
24.47-0.58
bilinear layer
8.57=0.04
25.27:40.16
9.03+00.33
26.72=0.17
Table 6 reports the comparison between methods with different self-supervised tasks. We also conducted generator-free self-supervised learning experiments by setting the hyper-parameter as >= 0 on each kind of self-supervised task. According to the FID score, ASS is significantly superior to SS and SS+. Even without the self-supervised task of the generator, ASS (2g = 0) outperforms SS and SS+ that include generator self-supervised tasks in limited (10% and 20%) data, and the introduction of generator self-supervised tasks further expands this advantage.
Self-supervised signals. We empirically analyze the role of different augmentation parameters as self-supervised signals for AugSelf-GAN, All methods employ three types of data augmentations (color, translation, and cutout), with the difference lying in the predicted self-supervised signals. According to the FID scores in Table 7, predicting any augmentation parameters can significantly improve the final generative performance compared to the baseline. Although there is no significant difference among them, we still choose to predict all augmentation parameters as the default setting for AugSelf-GAs
This experiment demonstrates that the self-supervised task itself plays decisive role in training AugSelf-GAN rather than the specific predicted augmentations. Therefore, we believe that our method is generalizable and can be extended to other advanced data augmentations.
Network architectures. We investigate the impact of different network architectures (two-layer multi-layer perceptron (MLP), linear layer, and bilinear layer) and input approaches (concatenation [d(x), (x)), subtraction x(x) Φ(x). and augmented samples only o(x)) AugSelf-GAN, As
16

---

Table IS and FID of AugSelf-Bigg BigGAN with different loss functions on CIFAR-10 and CIFAR-100 with full and limited training data. The best result is bold and the second best is underlined.
100% training data
20% training data
10% training data
Method
IS (1)
FID (4)
IS (1)
FID (L)
IS (t)
FID (4)
Aa = 0, Ag \== = 1,
9.29
8.48
8.84
15.14
8.80
20.60
CIFAR-10
9.12...05
8.90
8.82..m
13.65
8.52
18.14
saturating
9.27
8.13
8.83+.04
12.764.29
8.42
18.43
non-saturating
9.37+.07
7.60+.08 .08
8.91+.09
11.44+.116
8.63+.10
15.86+.26
combination
9.43
7.68+.06
8.98
10.97
8.76
15.68
CIFAR- 100
0, 11.02+.07
11.49
9.45
24.98+.48
8.50
34.92+.63
Xd 1,,,,,-0
10.82+.10
11.29
9.964.11
18.90+.41
9,45
25.77
saturating
10.90.4.07
10.53
9.71
18.78+.43
9.30+.09
25.92
non-saturating
11.22.+.14
9.90+.09
10.07
16.12.::6
9.81
21.99
combination
11.19+.09
9.88
10.25
16.11+:25
9.78,+008
21.30
reported in Table 8, we found that more complicated architectures of the predict head y such as two- layer MLP with input concatenation (x, w) x) = MLP([o(T(xx, w)), o(x))) or bilinear layer W E Rdxd) works worse than the simple linear layer with (el(((((T w)) - ((x))), which is the design of AugSelf-GAN, The reason might be that complicated architectures of the head ip actually discourage the backbone ø from capturing rich and linear representations, resulting in poor generalization of discriminators.
(D(T(x;w), x) = input subtraction (T(T(x;;))
Generator loss functions. We study the effects of AugSelf-GANs with different generator loss functions. As shown in Table 9, The typer-parameters a = 0, X, = 0 represent the baseline BigGAN + DiffAugment. The generation performance improved with only augmentation-ave self-supervision for the discriminator (Aa 1, A = 0). The saturating version of self-supervised generator loss shows no significant further improvement. The non-saturating version yields compa- rable performance with the combination one (AugSelf-GAN) and they both substantially surpass the others. The reason that the non-saturating loss directly encourages the generator to generate piggnentation-predicaall real data, providing more informative guidance than the saturating one.
D Additional Results
100% training data
20% training data
10% training data
CIFAR-10 10
Aptlo Higilia
RubtS
CIFAR- --FAR-100 100
Figure 7: FID plots comparison of AugSelf-GAN and baselines on CIFAR-10 and CIFAR-100.
17

---

Table 10: FID comparison on AFHQ. The number within the parentheses represents the percentage of improvement (1) in AugSelf-StyleGAN2 compared to the baseline StyleGAN2 + DiffAugment.
Method
Cat
Dog
Wild
StyleGAN2 [19]
5.13
19.4
3,48
+ DiffAuument [47]
3.49
8.75
2.69
AugSeff-StyleGAN2
3.23 (J 7.45%)
8.17 (L 6.63%)
2.48 (L 7.81%%
StyleGAN2 DiffAugment
AugSelf-StyleGAN2
Cat
Dog
PIEM
Figure Visual comparison between StyleGAAN+DDffAuugmenn and AugSelf-StyleGANg on AFHQ.
Training stability. Figure shows the FID plots during training of our method and baselines CIFAR-10 10 and CIFAR-100. Overall, AugSelf-BigGAN demonstrates a more stable convergence compared to the baselines, while AugSelf-BigGANg goes even further.
AFHQ We also conduct experiments on the AFHQ dataset [6] to compare our proposed method with DiffAugment by employing StyleGAN2 the backbone model. The hyperparameters
set as A 0.1, Aa = 0.5, and A2 = 0.2 on Cat, Dog, and Wild domains, respectively, and X, = 0.1 all three domains. Quantitative experimental results, as shown in Table 10, suggest that AugSelf-StyleGAN yields approximately a 7% improvements in FID across all three domains when compared Diff DiffAugment. Figure 8 presents qualitative comparison between these methods, demonstrating significant improvement in image quality for our method.
FFHQ and LSUN-Cat Figures 9 and 10 present the randomly generated images produced by StyleGAN2 + Diff'Augment and AugSelf-StyleGAAN2+ on the FFHQ [17] and LSUN-Cat [45] datasets with varying amounts of training data, respectively. Our MngSeffSttleGANNC demonstrates significant superiority over StyleGAN2 DiffAugment in terms of visual quality of generated images.
18

---

StyleGAN2 + DiffAuumentt
AugSelf-StylGGAN2+
1k training samples
5k training samples
10k training samples
30k training samples
Figure 9: Comparison of random generated images between StyleGAN2 + DiffAugment and AugSelf- StyleGAN2+ on FFHQ with different amounts of training data.
19

---

StyleGAN2 + DiffAuument
AugSelf-StylcGANN
1k training samples
5k training samples
tola
10k training samples
30k training samples
Figure 10: Comparison of randomly generated images between StyleGAN2 + DiffAugment and AugSelf-StyleGAAN2 on LSUN-Cat with different amounts of training data.
20


Reviews:
[{"review_id": "Qj19JEfCnF", "reviewer": "Reviewer_xecU", "summary": "This paper deals with the challenge of data-efficient GAN training. It is built upon DiffAugment and attempts to mitigate its issue of undesired augmentation invariance. The proposed method introduces an additional linear layer that predicts the augmentation parameter from the features extracted by the backbone of the discriminator.", "questions": "Can you provide more explanation about how the theoretical analysis that “reveals a connection between the optimization objective of the generator and the arithmetic harmonic mean divergence” is relevant to the proposed method?", "limitations": "The authors have discussed the limitations of their work. Please see the Weaknesses stated above to see other limitations I find.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper’s target task is clearly stated and holds significance.\n    \n- The proposed method is simple and exhibits consistent effectiveness across various scenarios.", "weaknesses": "- The motivation of the proposed method is unclear and lacks of evidence. The authors discuss the issue of DiffAugment in lines 36-42, citing [23] to argue the importance of augmentation sensitivity. However, it is questionable whether the motivations discussed in [23] are directly applicable to this work since the former aims at enhancing the transferability of the pretrained features. It is comprehensible that augmentation invariance can be harmful for the transferability of representation learning, and Figure 4 and 5 in [23] demonstrate intuitively that augmentation awareness helps the model learn position- and color-sensitive representations. However, in the case of image generation, it is hard for me to understand why the discriminator should be transformation sensitive.\n    \n\n- One of the major benchmark dataset, ImageNet, is not considered in this work, which is a common choice in GAN literature [1, 2, 3]. Given that the proposed method is mainly an improvement of DiffAugment, it would be vital to show the comparison on the primary dataset used in the original paper.\n    \n- The paper could benefit from a  qualitative comparison with the baselines by making it easier to intuitively grasp the effectiveness of the proposed method.\n    \n\n[1] Brock et al. Large scale GAN training for high fidelity natural image synthesis. 2019.\n\n[2] Lucic et al. High-Fidelity Image Generation With Fewer Labels. 2019.\n\n[3] Zhao et al. Differentiable augmentation for data-efficient gan training. 2020.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank Reviewer xecU for the valuable feedback. The comments and suggestions have greatly helped us in improving the quality of our work and ensuring its scientific rigor.\n\n---\n\n**The motivation of the proposed method is unclear and lacks of evidence.**\n\nWe apologize for any confusion caused by our explanation of the motivation behind our method. Please allow us to provide additional clarification.\n\nThe sensitivity of the discriminator to different levels of data augmentation enables it to learn rich data representation such as color and object position information, thus enhancing the discriminator's representation learning ability. This has been supported by Table 2, where AugSelf-GAN exhibits stronger discriminator representation learning ability than DiffAugment. As previous works have shown [1,2,3,4], the stronger representation learning ability of the discriminator leads to better feedback to the generator, resulting in better fidelity and diversity of generated samples, which is consistent with our experimental results. In summary, we enhance the representation learning ability of the discriminator through perceptual augmentation-based self-supervised tasks, resulting in higher quality feedback to the generator and thus improved performance.\n\nWe appreciate your feedback and will revise our manuscript to make our motivation and evidence clearer.\n\n---\n\n**One of the major benchmark dataset, ImageNet, is not considered in this work.**\n\nWe appreciate your suggestion to include ImageNet experiments in our research. While we understand that ImageNet is a common choice in GAN literature, we did not conduct ImageNet experiments initially due to resource limitations. Our code is mainly based on the open-source data-efficient-gans repository that builds on top of DiffAugment. The ImageNet on BigGAN experiments require TPUs, which we are not able to access.\n\nNonetheless, we conducted high-resolution (256x256) experiments using StyleGAN2 and are currently in the process of conducting ImageNet experiments. Regrettably, because training BigGAN on ImageNet requires eight GPUs running for approximately a month to obtain good results, we are unable to report the results within the short rebuttal period. We are, however, working to reproduce the ImageNet experiments on GPUs and will add the results to an updated version of the paper once they are available.\n\n---\n\n**The paper could benefit from a qualitative comparison with the baselines.**\n\nWe have uploaded a PDF file to show the qualitative comparison of AugSelf-GAN with the baselines on CIFAR-10, Low-shot, FFHQ and LSUN-Cat (Figure 12, 13, 14), and will include it in the updated version of our paper. Please review the new PDF file, hopefully it will address your concerns.\n\n---\n\n**How the theoretical analysis is relevant to the proposed method?**\n\nOur theoretical analysis starts from the objective function of the proposed method and is based on an assumption (that the self-supervised signal $\\omega^+=-\\omega^-$ is constant). As GANs solve a bi-level optimization problem, we first analyze the mathematical form of the optimal self-supervised discriminator and then substitute it into the objective function of the generator, resulting in the generator optimizing the arithmetic harmonic mean divergence. Although this assumption is different from our actual approach (where $\\omega^+=-\\omega^-$ is not constant), it is not uncommon in the field, as seen in non-saturating GAN [5] and LeCam-GAN [6]. Furthermore, we claim to reveal a connection rather than equivalence between the optimization objective of the generator and the arithmetic harmonic mean divergence. In our experiments, we demonstrate the effectiveness of dynamic self-supervised signals compared to fixed/constant ones.\n\n---\n\n[1] Chen, et al. Self-Supervised GANs via Auxiliary Rotation Loss. CVPR 2019\n\n[2] Jeong, et al. Training GANs with Stronger Augmentations via Contrastive Discriminator. ICLR 2021\n\n[3] Sauer, et al. Projected GANs Converge Faster. NeurIPS 2021\n\n[4] Kumari, et al. Ensembling off-the-shelf Models for GAN Training. CVPR 2022\n\n[5] Goodfellow, et al. Generative Adversarial Nets. NIPS 2014\n\n[6] Tseng, et al. Regularizing Generative Adversarial Networks under Limited Data. CVPR 2021"}, {"review_id": "IvpNCKEivp", "reviewer": "Reviewer_CpCq", "summary": "This paper addresses the challenge of training GANs with limited data, where the discriminator is prone to overfitting. Previous approaches, such as differentiable augmentation, have improved data efficiency but introduce undesired invariance to augmentation for the discriminator. To overcome this limitation while retaining the benefits of data augmentation, the authors propose an augmentation-aware self-supervised discriminator. This discriminator predicts the augmentation parameter of the augmented data, distinguishing between the prediction targets of real and generated data during training. The generator is encouraged to learn from the self-supervised discriminator by generating augmentation-predictable real data. This formulation connects the learning objective of the generator and the arithmetic-harmonic mean divergence. ", "questions": "1. Is there a way to determine which augmentations are beneficial to the training?\n\n2. What are the major differences between this work and  SSGAN-LA ?\n\n3. SSGAN-LA can addresses catastrophic forgetting problem. Is AugSelf also able to tackle this challenge?", "limitations": "The limitations are elaborated in the prior sections. In summary, the limited novelty is the major concern. ", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "This paper proposes an augmentation-aware self-supervised discriminator. The authors include both experimental results and theoretical analysis to show the effectiveness of the proposed method. According to the results, AugSelf can improve the performance than SOTA methods.", "weaknesses": "The major concern is the limited novelty. For example, SSGAN-LA works on augmenting the GAN labels (real or fake) via self-supervision of data transformation. The only difference is the form of self-supervised signals. Though the authors try to highlight the differences between these two works, these two methods still share many similarities. In the experiments, the paper also doesn't include a direct comparison with the closely related work SSGAN-LA. \n\nFurthermore, the method relies on the used augmentation method. How to select the appropriate augmentation is a problem. The paper doesn't include guidelines of how to select these augmentations. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank Reviewer CqCq for providing valuable feedback. The comments and suggestions have greatly helped us in improving the quality of our work and ensuring its scientific rigor.\n\n---\n\n**Limited novelty compared to SSGAN-LA and no direct comparison with SSGAN-LA.**\n\nAugSelf-GAN and SSGAN-LA have significant differences in the problems they aim to solve, their specific methods, and theoretical analysis. Firstly, in terms of the problem they aim to solve, SSGAN-LA focuses on improving the training stability of unsupervised GANs and correcting the theoretical learning objective deviation of existing self-supervised GANs. On the other hand, our AugSelf-GAN aims to enhance the performance of GANs in data-limited scenarios, specifically the efficiency of utilizing training data.\n\nIn terms of methods, SSGAN-LA employs a discrete adversarial self-supervised task, while AugSelf-GAN utilizes a continuous adversarial self-supervised task. It should be noted that the transition from discrete to continuous self-supervised adversarial tasks is not easy. Specifically, the adversarial idea requires us to distinguish self-supervised labels of true and false data. In the discrete case, SSGAN-LA uses class augmentation to distinguish the categories of true and false data, while in the continuous case, there is no similar concept. Therefore, we must design a brand-new adversarial task that conforms to the continuous label. We innovatively choose the origin-symmetric scheme ($\\omega^+=-\\omega^-$) for distinguishing continuous labels of true and false data and provide corresponding theoretical explanations. \n\nThis theoretical explanation reveals the correlation between the proposed method and a robust divergence, explaining the training stability of the method under data constraints. This advantage is not present in SSGAN-LA, whose theoretically optimized divergence is the reversed KL divergence that is inferior to the AHM divergence in this particular scenario. \n\nAdditionally, in terms of the specific implementation of the discriminator, SSGAN-LA does not accept original data as input and thus requires the data augmentation to be reversible, while AugSelf-GAN accepts both original and augmented data as input, making it unrestricted in data augmentation. We believe that these significant differences demonstrate the novelty of this work compared to SSGAN-LA.\n\nThe table below compares AugSelf-GAN and SSGAN-LA on the CIFAR-10 and CIFAR-100. According to the IS and FID scores, AugSelf-GAN outperforms SSGAN-LA in most settings, especially when the data is small. We appreciate your constructive suggestions, and will add this comparative experiment in the updated version of our paper.\n\n| IS, FID | CIFAR-10 | CIFAR-10-0.2 | CIFAR-10-0.1 | CIFAR-100 | CIFAR-100-0.2 | CIFAR-100-0.1 |\n| --- | --- | --- | --- | --- | --- | --- |\n| SSGAN-LA | 9.30, 8.05 | 8.84, 11.16 | 8.30, 15.08 | 11.13, **9.77** | **10.52**, 15.91 | 9.81, 23.17 |\n| AugSelf-GAN | **9.43**, 7.68 | 8.98, 10.97 | 8.76, 15.68 | **11.19**, 9.88 | 10.25, 16.11 | 9.78, 21.30 |\n| AugSelf-GAN+ | 9.27, **7.54** | **9.08**, **9.95** | **8.79**, **12.76** | 11.12, 10.09 | 10.14, **15.33** | **9.93**, **18.64** |\n\n---\n\n**Is there a way to determine which augmentations are beneficial to the training?**\n\nAccording to Table 7 in the appendix, we found that any type of data augmentation (color, translation, cutout), as well as their combinations, are beneficial for GAN training compared to the baseline method DiffAugment. We believe that this is one of the advantages of AugSelf-GAN, which is effective and robust in selecting various types of data augmentation. For simplicity, we default to using all data augmentations of baseline methods (e.g. DiffAugment) as self-supervised signals. As for the strength of data augmentation, experiments in Table 5 show that fewer training data require stronger data augmentation, and vice versa. We appreciate your valuable feedback and will clarify this point in the updated version of our paper.\n\n---\n\n**Is AugSelf also able to tackle catastrophic forgetting problem?**\n\nThank you for your constructive question. We have conducted additional experiments and have provided the results in Figure 11 (d) and (h) of the new PDF file. AugSelf-BigGAN(+) achieved significantly better accuracy on the representation learning task in the discriminator compared to BigGAN and BigGAN+DiffAugment. The accuracy curve of AugSelf-BigGAN(+) tended to be stable after a rapid increase, unlike BigGAN which showed a significant decrease with increased training iterations. This indicates that the AugSelf method effectively addresses the catastrophic forgetting problem in the discriminator."}, {"review_id": "7YIn1h7czU", "reviewer": "Reviewer_b4Yw", "summary": "This work proposed an augmentation aware training framework to improve the synthesis of GANs under the low data regimes. The discriminator is also required to regress the augmentation parameters, preventing undesired invariance introduced by augmentations. Theoretical analysis also shows the connection between the given objective and the arithmetic-harmonic mean divergence. Experimental results alos demonstrate the effectiveness of the proposed approach under various benchmarks. ", "questions": "Please see Weaknesses", "limitations": "Please see Weaknesses", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- This paper is well-organized and easy to follow. \n- Making discriminator aware of the appiled augmentation is inspiring since some undesired invariance is always introduced more or less. - - Experiments are quite solid, including results on multiple benchmarks and insightful analysis (Sec. 6.2).\n- Section 5 also presents the theoretical analysis, showing the connection with arithmetic\u0000harmonic mean divergence, which may gives more deep understanding for audience. ", "weaknesses": "- There might lack the comparisons against prior SOTA. For instance, InsGEN and Vision-aided GAN achieves appealing performances for FFHQ and while Table.2 did not presents these. Obviously, the consistent gains could be obtained compared to the baseline StyleGANs. However, SOTA performances could make this work stronger.\n- Experiments on AFHQ are missing. I believe AFHQ is also a good benchmark to evaluate the approaches for limited data synthesis. \n- Performances on the sufficient data for FFHQ and LSUN-Cat would further give a whole picture of how well the proposed method performs. \n- Could this augmentation aware scheme be combined with more complex augmentations like ADA?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank Reviewer b4Yw for the valuable feedback. The comments and suggestions have greatly helped us in improving the quality of our work and ensuring its scientific rigor.\n\n---\n\n**There might lack the comparisons against prior SOTA (InsGEN and Vision-aided GAN).**\n\nThank you very much for your suggestion. We would like to further clarify the motivation behind our work to demonstrate the rationale of our experimental design in the initial version of our paper.\n\nFirst, we do not consider Vision-aided GAN as a baseline as it introduces additional pre-trained models and data, and is a fine-tuning work based on our baseline, DiffAugment. Therefore, this comparison would be unfair. InsGen is an improvement on ADA, which utilizes richer data augmentation techniques than DiffAugment with a total of 18 transformations. Therefore, InsGen may achieve additional gains due to more data augmentation. Of course, our AugSelf-GAN can also be applied to ADA's 18 transformations, which will include discrete 90-degree rotation data augmentation. However, this would partially include SSGAN-LA, which is not our intended result.\n\nDiffAugment is our direct baseline, and our goal is to surpass it. We have verified this through a large number of fair experiments. After this, we once again thank you for your suggestion and acknowledge that SOTA's performance will make this work even stronger. Therefore, we will consider this as our future work, exploring extending AugSelf-GAN to more powerful data augmentations and compare it with SOTA methods.\n\n---\n\n**Experiments on AFHQ are missing.**\n\nWe appreciate your thoughtful suggestion to evaluate our proposed method on AFHQ, which is a suitable dataset for limited data scenario. We are currently conducting experiments on AFHQ, but due to limited training resources, it is taking us some time to obtain reliable results. However, we assure you that we will include the results in the updated version of the paper.\n\n---\n\n**Performances on the sufficient data for FFHQ and LSUN-Cat.**\n\nThank you for your suggestion regarding the evaluation on the full FFHQ and LSUN Cat datasets. However, we apologize that conducting such experiments and obtaining results during the short rebuttal phase would be difficult for us. We will continue to conduct experiments on full FFHQ and LSUN Cat data and report the results. We chose the current data setting in experiments because our research focuses on training GANs with **limited data.** Therefore, we follow exactly the same data setting as our baseline DiffAugment.\n\n---\n\n**Could this augmentation aware scheme be combined with more complex augmentations like ADA?**\n\nThank you for your constructive question, which enriched our experiments and improved our work. We believe our method can indeed be combined with ADA. One major difference between ADA and DiffAugment is that ADA uses adaptive data augmentation probabilities. In fact, ADA incorporates the same augmentations as DiffAugment, such as brightness, contrast, saturation, translation, and cutout. We still use these as self-supervised signals for our AugSelf framework to conduct new experiments based on ADA on unconditional CIFAR10, and the results are as follows, showing that our method outperforms ADA. \n\n| FID | uncond CIFAR-10 |\n| --- | --- |\n| ADA | 3.11 |\n| AugSelf | **3.02** |\n\nWe will continue to conduct more experiments, including more data augmentation as self supervision, to verify the applicability of our method in stronger data augmentation methods."}, {"review_id": "2Dk6fVz7ys", "reviewer": "Reviewer_nM6o", "summary": "This paper proposes using a discriminator to predict the augmentation applied to the following sample, in addition to the standard real/fake discrimination objective. The authors demonstrate that using this objective training the GAN leads to minimizing to the Arithmetic-Harmonic Mean divergence objective. Further, the method shows improved results through FID in various data efficient setups of CIFAR-10, CIFAR-100, Few-Shot datasets etc.", "questions": "1) Missing Details: The major missing detail is about how to determine the value of w for various augmentations introduced in the paper. What is the general principle which is followed to determine this? This is also one of the major weaknesses, if there is no general way.\n\t\n2) Missing Evaluation using CLIP FID. Recently it has been argued in multiple papers [R1,R2] that CLIP based FID is a better metric for evaluating GANS. Hence, I request authors to provide the same in rebuttal.\n\n[R1]: The Role of ImageNet Classes in Fr\\'echet Inception Distance\n[R2]: NoisyTwins: Class-consistent and Diverse Image Generation through StyleGANs\n\n\nPost Rebuttal Update: The authors responses were helpful and some of my major concerns have been resolved. I will maintain my borderline accept rating.", "limitations": "Yes, the authors have discussed limitations", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1)\tThe proposed objective of predicting the augmentation through discriminator is novel to my knowledge.\n2)\tThe paper is well written and clear.\n3)\tThe technique of self-supervision of predicting the augmentation type seems to show significant improvement across baselines.", "weaknesses": "1) Reasoning for Effectiveness of Proposed Method: The authors argue that discriminator being invariant to augmentations is wrong. However, that seems intuitively sensible however the theory presented does not properly justify the reasoning for it’s effectiveness. A possible direction for this to think upon could be how the discriminator distribution expands to augmentation.\n\t\n2) Convergence Plot: As the method seems to improve performance, an analysis of convergence of FID could be provided for clearing whether the proposed method delays collapse or prevents it all together. In L136 the authors mentioned about better convergence but do not provide FID plot.\n\nMinor:\n1) Fig. 2 is not described in text.\n2) L136 misses citation.\n3) Rephrase L49-51, as it's unclear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank Reviewer nM6o for providing valuable feedback. The comments and suggestions have greatly helped us in improving the quality of our work and ensuring its scientific rigor.\n\n---\n\n**Reasoning for Effectiveness of Proposed Method**\n\nWe provided evidence of limitations to the representation learning of the discriminator of DiffAugment in Table 2, which motivated us to propose self-supervised task aiming to enhance this capability, and our experimental results have shown its effectiveness. We will modify the description of the existing limitations of DiffAugment to emphasizing that the discriminator lacks perception to different data augmentations, which limits the representation learning capability. Additionally, our theoretical analysis shows the robustness and superiority of the loss function of AugSelf-GAN in limited data settings.\n\n---\n\n**Convergence Plot of FID.**\n\nWe appreciate your constructive suggestion and have uploaded an extra PDF file that includes the FID convergence plots for AugSelf-BigGAN(+), DiffAugment, and BigGAN on CIFAR-10 and CIFAR-100 under different training data regimes for better analysis of convergence and performance comparison. As shown in Figure 11 (a,b,c,e,f,g), AugSelf-BigGAN(+) is significantly superior to the baselines DiffAugment and BigGAN on the FID convergence curve.\n\n---\n\n**Fig. 2 is not described in text.**\n\nFig. 2 is intended to demonstrate the representation learning capability of the discriminator. Specifically, we used the penultimate layer of the discriminator as a feature extractor and trained a downstream linear classifier on training set to classify the extracted feature on the validation set. We will include a corresponding description in the main text of the updated version.\n\n---\n\n**L136 misses citation.**\n\nWe will add citations to SSGAN [1] and SSGAN-LA [2] in L136 of the updated version to help clarify the motivation and impact of the experiment.\n\n---\n\n**Rephrase L49-51, as it's unclear.**\n\nWe will modify the description of L49-51 to make it more precise. The sentence aims to explain Equation 6, where the generated data from the generator through augmentation needs to be accurately identified by the self-supervised discriminator as real augmented data, instead of fake augmented data. We will strengthen the clarity of this statement by providing a reference to Equation 6 in the updated version.\n\n---\n\n**Missing detail is about how to determine the value of w for various augmentations. What is the general principle which is followed to determine this?**\n\nIn general, when applying data augmentation, the augmentation parameter $\\omega$ follows a certain range $\\Omega \\in \\mathbb{R}^d$ which is randomly sampled. In our work, we simply rescale the $\\Omega$ space to $[0,1]^d$ to obtain the rescaled values of $\\omega$. In our early experiments, we found that modifying the range of the rescaled space had little effect on the results, therefore we adopted the current approach to ensure the usability and generality of our method. We will clarify this in the updated version of our paper.\n\n---\n\n**Missing Evaluation using CLIP FID.**\n\nThank you for your valuable suggestion. We have followed your advice and evaluated augself(+) and diffaugment, using CLIP-FID on FFHQ and low-shot image generation. The experimental results, as shown in the table below, demonstrate that augself(+) outperforms diffaugment on most of the datasets.\n\n| CLIP-FID | FFHQ-30k | FFHQ-10k | FFHQ-5k | FFHQ-1k | 100-shot-obama | 100-shot-grumpy_cat | 100-shot-panda | AnimalFace-cat | AnimalFace-dog |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| DiffAugment | 3.67 | 5.64 | 7.53 | 17.47 | 15.98 | 16.04 | 15.46 | 22.68 | **22.69** |\n| AugSelf | **3.61** | 5.31 | 7.08 | **14.48** | **9.54** | **10.19** | **11.84** | **19.68** | 25.95 |\n| AugSelf+ | 4.01 | **4.87** | **7.03** | **14.48** |  |  |  |  |  |\n\n---\n\n[1] Chen, et al. Self-Supervised GANs via Auxiliary Rotation Loss. CVPR 2019\n\n[2] Hou, et al. Self-Supervised GANs with Label Augmentation. NeurIPS 2021"}, {"review_id": "7Kne01rRVR", "reviewer": "Reviewer_NBv7", "summary": "This paper introduces AugSelf-GAN, an enhanced approach for self-supervised GANs with label augmentation (SSGAN-LA). The authors present two major contributions. Firstly, they propose a discriminator that predicts continuous augmentation parameters, instead of classifying discrete augmentation methods. This allows the discriminator to learn a more fine-grained representation of the data distribution and to better distinguish between real and fake images. Secondly, the authors conduct a theoretical analysis that establishes a connection between the proposed method and the arithmetic-harmonic mean divergence. Experimental results demonstrate that AugSelf-GAN enhances GAN performance (as measured by FID and IS) in scenarios of limited data settings.", "questions": "Please see the above weakness.", "limitations": "Yes, the authors have discussed the limitations of the proposed method, as well as the potential negative impact of this work.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The proposed method demonstrates significant improvements in quantitative results despite its simplicity. \n- This paper offers comprehensive empirical and theoretical evidence to support the effectiveness of the proposed approach.", "weaknesses": "- L138–139: The authors claim that SSGAN-LA and ADA result in augmentation leaking. However, no evidence, such as citations or empirical results, is provided. I think this claim should be justified since it is the main motivation of the paper.\n\n- The proposed method has limited novelty in terms of technical details. It can be seen as a variant of SSGAN-LA, with the main difference being that the discriminator predicts continuous augmentation parameters instead of discrete ones. \n\n- The paper lacks a comparison with some previous methods, such as ContraD and SSGAN-LA. These methods are also designed to train GANs with data augmentation. It would be interesting (indeed necessary) to see how the proposed method compares to them.\n\n- The authors argue that the representation learning capability of the discriminator could be improved by the proposed method. However, there are not enough experiments to support this claim. The experiments are conducted on CIFAR10/100 only (Fig. 2), and there are no comparisons with the methods that claim similar advantages (i.e., ContraD).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank Reviewer NBv7 for the valuable feedback. The comments and suggestions have greatly helped us in improving the quality of our work and ensuring its scientific rigor.\n\n---\n\n**Augmentation leaking in SSGAN-LA and ADA?**\n\nIn L138-139, we did not aim to claim that SSGAN-LA and ADA suffer from the problem of augmentation leaking; instead, we cited them to demonstrate how straightforward data augmentation may have this issue. We will clarify this statement more precisely in the updated version.\n\n---\n\n**The proposed method has limited novelty in terms of technical details.**\n\nThere are significant differences in the proposed methods and theoretical analysis between AugSelf-GAN and SSGAN-LA. SSGAN-LA employs a discrete adversarial self-supervised task, while AugSelf-GAN utilizes a continuous adversarial self-supervised task. It should be noted that the transition from discrete to continuous self-supervised adversarial tasks is not easy. Specifically, the adversarial idea requires us to distinguish self-supervised labels of real and generated data. In the discrete case, SSGAN-LA uses label augmentation to distinguish the class-label of real and fake data, while in the continuous case, there is no similar concept. Therefore, we must design a brand-new adversarial task that conforms to the continuous self-supervision. There may be multiple schemes for distinguishing continuous labels of real and fake data, but we innovatively choose the origin-symmetric approach and provide corresponding theoretical explanations. This theoretical explanation reveals the connection between the proposed method and a robust divergence, the arithmetic-harmonic mean (AHM) divergence, explaining the superiority of the method under limited data. This advantage is not present in SSGAN-LA, whose theoretically optimized divergence is the reversed KL divergence that is inferior to the AHM divergence in the studied scenario. Additionally, in terms of the specific implementation of the discriminator, SSGAN-LA does not accept original data as input and thus requires the data augmentation to be reversible, while AugSelf-GAN accepts both original and augmented data as input, making it unrestricted in data augmentation. We believe that the significant differences in method and theoretical analysis demonstrate the technical novelty of this work.\n\n---\n\n**The paper lacks a comparison with some previous methods, such as ContraD and SSGAN-LA.**\n\nThe following Table reports the comparison of AugSelf-GAN(+) with ContraD and SSGAN-LA on CIFAR-10 and CIFAR-100, showing that AugSelf-GAN(+) can outperform the best in limited-data settings in terms of FID, and we will also report it in the updated version of our paper. \n\n| IS, FID | CIFAR-10 | CIFAR-10-0.2 | CIFAR-10-0.1 | CIFAR-100 | CIFAR-100-0.2 | CIFAR-100-0.1 |\n| --- | --- | --- | --- | --- | --- | --- |\n| ContraD + StyleGAN2 | **9.47**, 9.80 | - | - | 10.00, 14.10 | - | - |\n| SSGAN-LA | 9.30, 8.05 | 8.84, 11.16 | 8.30, 15.08 | 11.13, **9.77** | **10.52**, 15.91 | 9.81, 23.17 |\n| AugSelf-GAN | 9.43, 7.68 | 8.98, 10.97 | 8.76, 15.68 | **11.19**, 9.88 | 10.25, 16.11 | 9.78, 21.30 |\n| AugSelf-GAN+ | 9.27, **7.54** | **9.08**, **9.95** | **8.79**, **12.76** | 11.12, 10.09 | 10.14, **15.33** | **9.93**, **18.64** |\n\n---\n\nT**here are not enough experiments to support the improved representation learning capability of the discriminator of the proposed method.**\n\nConducting representation learning experiments requires datasets with labels. In our experiments, CIFAR-10 and CIFAR-100 were the only datasets with labels available. We appreciate your valuable suggestions and have added Tiny ImageNet experiments. The experimental results are as follows, which also verify that this method can effectively improve the representation learning ability of the discriminator. We also report the IS and FID results, and will report the results in the updated version of our paper.\n\n| Acc | TI | TI-0.2 | TI-0.1 |\n| --- | --- | --- | --- |\n| BigGAN + DiffAugment | 0.3244 | 0.3073 | 0.3006 |\n| AugSelf-BigGAN | **0.4171** | **0.3577** | **0.3293** |\n\n| IS | TI | TI-0.2 | TI-0.1 |\n| --- | --- | --- | --- |\n| BigGAN + DiffAugment | 16.23+-0.22 | 13.83+-0.19 | 10.18+-0.08 |\n| AugSelf-BigGAN | **16.51+-0.19** | **14.81+-0.06** | **12.78+-0.19** |\n\n| FID | TI | TI-0.2 | TI-0.1 |\n| --- | --- | --- | --- |\n| BigGAN + DiffAugment | 17.11+-0.04 | 27.85+-0.16 | 41.96+-0.14 |\n| AugSelf-BigGAN | **15.82+-0.23** | **21.61+-0.27** | **29.47+-0.03** |\n\n---\n\n**There are no comparisons with the methods that claim similar advantages.**\n\nWe did not compare to ContraD in the submitted version as ContraD does not address the issue of data-limited GAN training, which is our studied problem, and it does not use BigGAN as the backbone on CIFAR-10/100, unlike the setting we followed with DiffAugment. Conducting code migration and experiments for this comparison would require significant time and resource, and we apologize that we could not provide comparison results in this short rebuttal phase. However, we will make the effort to migrate the code and conduct experiments to provide comparison results in the updated version of our paper."}]

Author rebuttal:
We would like to sincerely thank all the reviewers for their valuable time and constructive feedback that have helped improve the quality of our research.

We have carefully considered all the concerns and questions raised by the individual reviewers and provided detailed responses in our rebuttal. We hope that the reviewers and the AC will take our responses into account while evaluating our manuscript.

To further demonstrate the effectiveness of our proposed method, we have uploaded a PDF file containing several figures. Figure 11 shows the convergence comparison between our method and the baselines on FID plots (a,b,c,e,f,g) and Acc plots (d,h) to validate the training convergence and ability of our method to overcome catastrophic forgetting, respectively. Figures 12, 13, and 14 exhibit qualitative comparisons between our proposed method AugSelf and DiffAugment on CIFAR-10, CIFAR-100, Low-shot image generation, FFHQ, and LSUN-Cat with limited data.

Finally, we would like to express our gratitude to everyone for their efforts and contributions to this conference. We hope that our paper can be fully discussed in the next stage.

Meta-review: 5x BA. This paper proposes an augmentation-aware self-supervised discriminator for low-shot GAN training, so as to mitigate the discriminator overfitting and improve the generation quality. The reviewers all lean to accept the paper due to its (1) clear presentation, (2) effective empirical results, and (3) comprehensive theoretical analysis. The rebuttal has addressed their concerns.
Decision: Accept (poster)