Self-Supervised GANs with Label Augmentation
Authors: Liang Hou
Huawei Shen
Qi Cao
Xueqi Cheng
Venue: NeurIPS 2021 Poster (2021)
Keywords: generative adversarial networks
self-supervised learning
label augmentation

Abstract: Recently, transformation-based self-supervised learning has been applied to generative adversarial networks (GANs) to mitigate catastrophic forgetting in the discriminator by introducing a stationary learning environment. However, the separate self-supervised tasks in existing self-supervised GANs cause a goal inconsistent with generative modeling due to the fact that their self-supervised classifiers are agnostic to the generator distribution. To address this problem, we propose a novel self-supervised GAN that unifies the GAN task with the self-supervised task by augmenting the GAN labels (real or fake) via self-supervision of data transformation. Specifically, the original discriminator and self-supervised classifier are unified into a label-augmented discriminator that predicts the augmented labels to be aware of both the generator distribution and the data distribution under every transformation, and then provide the discrepancy between them to optimize the generator. Theoretically, we prove that the optimal generator could converge to replicate the real data distribution. Empirically, we show that the proposed method significantly outperforms previous self-supervised and data augmentation GANs on both generative modeling and representation learning across benchmark datasets.

Self-Supervised GANs with Label Augmentation
Liang Hou! Huawei Shen 13, Qi Cao', Xueqi Cheng?
Data Intelligence System Research Center, Institute of Computing Technology, Chinese Academy of Sciences "CAS Key Laboratory of Network Data Science and Technology, Institute of Computing Technology, Chinese Academy of Sciences UUniversity of Chinese Academy of Sciences fhouliang17,, shenhuawei, caoqi, cxqh@ict.acc cn
Abstract
Recently, transformation--aase self-super learning has been applied to gen- erative adversarial networks (GANs) mitigate catastrophic forgetting in the discriminator by introducing a stationary learning environment. However, the separate sele-super tasks existing self-supervised GANs cause goal incon- sistent with generative modeling due to the fact that their self-supervised classifiers are agnostic to the generator distribution. To address this problem, propose a novel self-supervised GAN that unifies the GAN task with the self-supervised task by augmenting the GAN labels (real or fake) via superviiini of data transfor- mation. Specifically, the original discriminator and self-superviser classifier are unified into label-augmented discriminator that predicts the augmented labels be aware of both the generator distribution and the data distribution under every transformation, and then provide the discrepancy between them to optimize the generator. Theoretically. we prove that the optimal generator could converge to replicate the real data distribution. Empirically, we show that the proposed method significantly outperforms previous self-supervised and data augmentation GANs on both generative modeling and representation learning across benchmark datasets.
1 Introduction
Generative adversarial networks (GANs) 14] have developed rapidly in synthesizing realistic images in recent years 4 23 24] Conventional GANs consist of generator and discriminator, which are trained adversarially in a minimax game. The discriminator attempts to distinguish the real data from the fake ones that are synthesized by the generator, while the generator aims to confuse the discriminator to reproduce the real data distribution. However, the continuous evolution of the generator results in non-stationary learning environment for the discriminator. In such non-stationary environment, the single discriminative signal (real or fake) provides unstable and Therefore, the discriminator may encounter 25 0 [55]. leading to the training instability of GANs and thereby mode of the generator in learning high-dimensional, complex data distributions.
limited information the discriminator
37 70 57
catastrophic forgetting
collapse [52 61
33
36
To alleviate the forgetting problem of the discriminator, transformation-based self-supervised tasks such rotation recognition [3 have been applied to GANs o Self-supervised tasks designed to predict the constructed pscudo-labels of data, establishing stationary learning environment as the distribution of training data for this task does not change during training. The discriminator enhanced by the stationary self-supervision is able to learn stable representations to resist the forgetting problem, and thus can provide continuously informative gradients obtain a well-performed generator. After obtaining empirical success, self-supervised GAN was theoretically analyzed and partially improved by a new self supervised task [57]. However, these superviedd tasks cause goal inconsistent with
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

generative modeling that aims to learn the real data distribution. In addition, the that trade-off between the original GAN task and the additional self-supervised task bring extra workload for tuning on different datasets. To sum up, we still lack proper approach to overcome catastrophic forgetting and improve the training stability of GANs via self-supervised learning.
In this paper, we first point out that the reason for the presence of undesired goal in existing self- supervised GANs is that their self-supervised classifiers that train the generator are unfortunately agnostic to the generator distribution. In other words, their classifiers cannol provide the discrepancy between the real and generated data distributions to teach the generator to approximate the real data distribution. This issue originates from that these classifiers are trained to only recognize the pseudo- label of the transformed real data but not the transformed generated data. Inspired by this observation, we realize that the self-suuer classifier should be trained to discriminatively recognize the corresponding pseudo-labels of the transformed generated data and the transformed real data. In this case, the classifier includes the task of the original discriminator, thus the original discriminator could even be omitted. Specifically, we augment the GAN labels (real or fake) via supersuuevn of data transformation to construct augmented labels for novel discriminator, which attempts to recognize the performed transformation of transformed real and fake data while simultaneously distinguishing between real and fake. By construction, this formulates a single multi-class classification task for the aabelaaugmeted discriminator, which is different from a single binary classification task in standard GANs or two separate classification tasks in existing self- sulf-supeeeieed GANs. As a result, our method does not require any hyper parameter to trade-off between the original GAN objective and the additional self-supervised objective, unlike existing self- sule-vupe GANs. Moreover, prove that the optimal label-augmented discriminator is aware of both the generator distribution and the data distribution under different transformations, and thus can provide the discrepancy between them to optimize the generator. Theoretically, the generator can replicate the real data distribution at the Nash equilibrium under mild assumptions that can be easily satisfied, successfully solving the convergence problem of previous self-supervised GANs.
hyper-parameter
The proposed method seamlessly combines two unsupervised learning paradigms, generative adver- sarial networks and self-supe learning. to achieve stable generative modeling of the generator and superior representation learning of the discriminator by taking advantage of cach other. In the case of generative modeling. the self-supervised subtask resists the discriminator catastrophic forgetring, so that the discriminator can provide continuously informative feedback to the generator to learn the real data distribution. In the case of representation learning, the augmented-label prediction task encourages the discriminator to simultaneously capture both semantic information brought by the real/fake-distinguishel signals and the self-supervised signals to learn comprehensive data representations. Experimental results on various datasets demonstrate the superiority of the proposed method compared to competitive methods on both generative modeling and representation learning.
2 Preliminaries
2.1 Generative Adversarial Networks
Learning the underlying data distribution Pa. which is hard to specify but easy to sample from, lies at the heart of generative modeling. In the branch of deep latent variable generative models, the generator G Z + X' maps a latent code E Z endowed with a tractable prior Pa (e.g.. (V(0, I)) to a data point æ E X. and thereby induces a generator distribution P3 G#P.. To learn the data distribution through the generator (i.e., P, Pa). one can minimize the Sensen-Shannol (JS) divergence between the data distribution and the generator distribution (i.c., ming Das (Pall To estimate this divergence, generative adversarial networks (GANs) [4] leverage discriminator D : X + {0. }} that distinguishes the real data from the generated ones. Formally, the objective function for the discriminator and generator of the standard minimax GAN [14] is defined as follows:
min max V(G, D) = Ex-Po [log D(1 + E [log D(O G D
(1)
where D(1 = 1- D(0l2) [0. 1] indicates the probability that a data x is distinguished as real by the discriminator. While theoretical guarantees hold in standard GANs, the non-stationary learning environment caused by the evolution of the generator contributes to catastrophic forgetting of the discriminator 6 which means that the discriminator forgets the previously learned knowledge by over-fitting the current data and may therefore hurt the generation performance of the generator.

---

2.2 Self-Supervised GAN
To mitigate catastrophic forgetting the discriminator, self-supervised GAN (SSGAN)
6 introduced
an auxiliary self-supervised rotation recognition task to classifier C' & = T(x) (1,2, K}
that shares parameters with the discriminator, and forced the generator to collaborate with the classifier on the rotation recognition task. Formally, the objective functions of SSGAN are given by:
max V(G, D) + Xa EESPA.T (lo (x)))
(2)
D.C min V(G, P) >g. BB--P, Dog (())] G
where Aa and Ag are two perra--arameters that balance the original GAN task and the additional self-supervised task, T= = {Th K (i.e., 7= {0", 90", 180" 270") ). Throughout this paper, data transformations are uniformly sampled by default (i.e., ((.e..((TT) K v E T), unless otherwise specified.
k=1 is the set of deterministic data transformations such as rotations
Theorem 1 (57)). Given the optimal classifier C (k|z)
of SSGAN, at the equilibrium task for the generator is equivalent to:
Pd
ΣE 7 (a)
point, maximizing the
max G
Σ
E
Pp log
P
2
Th
(4)
L Pd
where p P the transformation Tg with density of Pg (ā) f - (z))))(2)))d or p (ø) J j
indicate the distribution of transformed generated or real data E X under
Th (x())))pd())d
All proofs of theoremssproposiiionn including this one (for completeness) in this paper are referred
to Appendix A Theorem reveals that the self-supervised task of SSGAN enforces the generator to produce only potation-detectall images other than the whole real images (57), causing a goal inconsistent with generative modeling that faithfully learns the real data distribution. Consequently, SSGAN still requires the original GAN task, which forms a multi-task learning framework that needs hyper-parameters to trade off the two separate tasks.
2.3 SSGAN with Multi-Class Minimax Game
To improve the self-supervised task in SSGAN, SSGAN-MS 5 proposed multi-class minimax self-supervised task by adding the fake class 0 to label-extend classifier C+ X + {0, ... K}.
where the generator and the classifier compete on the self-supervised task. Formally, the objective functions of SSGAN-MS are defined as follows:
max V(G, + Aa (Be-Pa.T flo C- (k]T((<==)) Ez- T (O/Ta)))))
(5)
D,C,
P min VGG, D) - ->g- (Be-P,.Te-T (M Byn T - flog C+ (x))) G
(6)
Theorem 2. Given the optimal classifier (k
Pa (2)
(1 of SSGAN-MS, at
P
Σ (
the equilibrium point, maximizing the self-supervised task for the generator equivalent ic
1 Σ
min DKL(P") (1PT
E
2ND log
K
E
k=
where PT, Pf represent the mixture distribution of transformed generated or real data ā E X with
density of p (*) = E ((Tx))))))))) (f) or (+) = KK P (#).
'Note that our Theorem
2 corrects the wrong version in the SSGAN-MS paper 57 where the authors
p (8)
Σ pn Tx(a)
P Tx(a)
mistakenly regard
(2)
P9
3t (t(a)
in their proof. Please Appendix A.2 for details.
3

---

Standard
Data Augmentation
Multi-task Learning
Joint Learning
!
k
k/
(k, I)
share
share
D
Dx
C
DLA
T
TR
T&
Tx
X
X
X
GAN
DAGAN
DAGAN-MD
SSGAN
SSGAN-MS
SSGAN-LA
Figure 1: Schematics of the diseriminators and classifiers of the competitive methods. We systemati- cally divide these methods into four categories: Standard (GAN), Data Augmentation (DAGAN and DAGAN-MD), Multi-task Learning (SSGAN and SSGAN-MS), and Joint Learning (SSGAN-LA).
Our Theorem 2 states that the improved supervuee task for the generator of SSGAN-MA -MS, under the optimal classifier, retains the inconsistent goal in SSGAN that enforces the generator to produce typical rotation-detectable images rather than the entire real images. In addition, minimizing (PL IP'')) cannot guarantee the generator to recover the real data distribution under the used data transformation setting (see Theorem 4 Due to the existence of these issues, SSGAN-MS also requires the original GAN task to approximate the real data distribution. We regard the learning paradigm of SSGAN-MS and SSGAN as the multi-task learning framework (see Figure
3
Method
We note that the self-supervised classifiers of existing self-supervised GANs are agnostic to the density of the generator distribution Pg (zx (or p (ā)) (see (k(ā) in Theorem and C1 (k]a) in Theorem D. In other words, their classifiers do not know how far away the current generator distribution P, is Trom the data distribution Pa, making them unable provide any informative guidance to match the generator distribution with the data distribution. Consequently, existing self-suuprrised GANs suffer from the problem of involving an undesired goal for the generator when learning from the generator distribution -agnosiii simply discarding the sulf-supervised task for the generator. However, the generator cannot take full advantage of the direct benefits of self-supervised learning in this case (see Appendix D
supervised classifiers. One naive solution for this problem is
In this work, our goal is to make full use of self-supervision to overcome catastrophic forgetting of the discriminator and enhance generation performance of the generator without introducing any undesired goal. We argue that the generator-agnoste issue of previous self-supervised classifiers originates from that those classifiers are trained to recognize the corresponding angles for only the rotated real images but not the rotated fake images. Motivated by the this understanding, we propose to allow the self-supervised classifier to discriminatively recognize the corresponding angles of the rotated real and fake images such that it can be aware of the generator distribution and the data distribution like the discriminator. In particular, the new classifier includes the role of the original discriminator, therefore the discriminator could be omitted though the convergence speed might be slowed down (see Appendix for performance with the original discriminator). Specifically, we augment the original GAN labels (real or fake) via the self-supervision of data transformation to construct augmented
labels and develop a novel label-augmented discriminator DI.A X + {1, 2, ... K} x {0, 1} (2K classes) to predict the augmented labels. By construction, this formulation forms single multi-class classification task for the label-augmented discriminator, which is different from single binary classification in standard GANs and two separate classification tasks in existing self-supervised GANs (see Figure M]. Formally, the objective function for the label- augmenned discriminator of the proposed self- self-supervised GANs with label augmentation, dubbed SSGAN-LA, is defined follows:
max MeaxEe>a, (log DLA (k, [[T)( + E DLA
DLA (k, (fT)(
(8)
where DLA (k. TTTx (resp. DL.A (k. T (x))) outputs the probability that a transformed data Te (x) is jointly classified as the augmented label of real (resp. fake) and the k-th transformation.

---

Proposition 1. For any fixed generator. given a data î E X that drawn, frawn,from mixture distribution of transformed data, the optimal label- augeeened discrininator of SSGAN-LA has the form of:
Pd
Pg
(+)
Di (k, 1|z)
DLA (k, 0,5)
(9)
T (3)) Σ Ek=1 1P (3) (p)
Σ (P Ta
(6)
Pa Ta
(ā))
[ shows that the optimal label-augmented discriminator of SSGAN-LA is able to aware the densities of both real and generated data distributions under every transformation (i.e., Pd Th (i)
and p (z)), and thus can provide the discrepancy (i.e., p/ (a)/p) (ï) = D* / D* 0|ž))
between the data distribution and the generator distribution under every transformation to optimize
the generator. Consequently, we now propose to optimize the generator of SSGAN-LA by optimizing
such discrepancy using the objective function formulated as follows:
max Baa PT.T DLA (k, 1[Tx(x))] E DLA (k, (T))
(10)
Theorem 3. The objective function for the generator of SSGAN-LA, given the optimal label- augmented discriminator, boils down to:
min
DeL. (P'H1|1 k=1
(II)
The global minimum is achieved and only Pf Pa when 37p E T is invertible transformation.
Theorem 3 shows that the generator of SSGAN-LA is encouraged to approximate the real data distribution under different transformations measured by the reverse KL divergence without any goal inconsistent with generative modeling, unlike previous self-supervised GANs. In particular, the generator is accordingly guaranteed to replicate the real data distribution when existing transformation is invertible such as the identity and 90-degree rotation transformations. In summary, SSGAN-LA seamlessly incorporates self-supervised learning into the original GAN objective to improve the training stability on the premise of faithfully learning of the real data distribution.
Compared with standard GANs, the transformation prediction task, especially the real samples, establishes stationary self-supervised learning environments in SSGAN-LA to prevent the discrimi- nator from catastrophic forgetting. In addition, the multiple discriminative signals under different transformations are capable of providing diverse feedback to obtain a well-performed generator. Compared with previous self-supervised GANs, in addition to the unbiased learning objective for the generator, the discriminator of SSGAN-LA has the ability to learn better representations. In particular, the discriminator utilizes not only the real data but also the generated data as augmented data obtain ability, and distinguishes between real data and generated data from multiple perspectives via different transformations to obtain enhanced discrimination ability. These two abilities enable the discriminator to learn more meaningful representations than existing self supervupe GANs, which could turn promote the generation performance of the generator.
robust mannsformation-rcogginn
4
The Importance of Self-Supervision
The proposed SSGAN-LA might benefit from two potential aspects, i.e., augmented data and the corresponding self self-supervision. To verify the importance of the latter, we introduce two ablation models that do not explicitly receive feedback from the self-supervision as competitors.
4.1 Data Augmentation GAN
We first consider a naive data augmentation GAN (DAGAN) that completely ignores self-supervision. Formally, the objective function of DAGAN is formulated as follows:
min max Eg [log ((I[T,(=)) + BE DeNPPITiNNToo (og ((O)T
(12)
D
This formulation is mathematically equivalent to DiffAugment (66 and IDA 559 We regard this approach as an ablation model of SSGAN-LA as trains with only the GAN labels but ignores the self-supervised labels. Next, show the indispensability of self-supervisiin.
5

---

Theorem 4. At the equilibrium point of DAGAN, the optimal generator implies PT = PI However. if (T, o) forms a group and Ts E T is uniformly sampled, then the probability that the optimal generator replicates the real data distribution is P(pg = Pa = P O.
Theorem 4 suggests that the formulation of DAGAN is insufficient to recover the real data distribu- tion for the generator as (109. ,90", 180", 270"). },o) forms a cyclic group and the transformation is uniformly sampled by default. Intuitively, if transformations are not identified by their corresponding self-supervision and can be represented by others, then the transformed data will be indistinguishable from the original data so that they may be leaked to the generator. In more detail, the generator would converge to an arbitrary mixture distribution of transformed real data (see Appendix [A.5) Note that all theoretical results in this paper are not limited to rotation but many other types of data transformation, e.g., RBG permutation and patch shufffing. Nonetheless, we follow the practice of SSGAN to use rotation the data transformation for fairness. In summary, data augmentation GANs may even suffer from the convergence problem without utilizing self-supervision.
4.2 DAGAN with Multiple Discriminators
To verify the importance of self-supervision while eliminating the convergence problem of DAGAN, we introduce another data augmentation GAN that has the same convergence point as SSGAN-LA. DAG 59 proposed data augmentation GAN with multiple discriminators (Dx) k=1 (we refer it as DAGAN-MD throughout this paper), which are indexed by the self-supervision and share all layers but the head with each other, to solve the convergence issue of DAGAN, Formally, the objective function of DAGAN-MD is formulated as:
min max EENPA.DeNT log (x(1 + G (De)
flg Dx (D.00)/)))))
(13)
We regard DAGAN-MD as an ablation model of SSGAN-LA since DAGAN-MD takes the self- supervised signal input while SSGAN-LA views target (see Figure D. It is proved that the generator of DAGAN-MD could approximate the real data distribution at the optimum [59] However, the lack of leveraging self-supervised signals supervision will disadvantage the performance of DAGAN-MD in two ways. On one hand, the discriminators cannot learn stable representations to resist catastrophic forgetting without learning from self- eef-suupe tasks. On the other hand, the discriminators cannot learn high-level representations contained in the specific semantics of different transformations. Specifically, these diseriminators presumably enforce invariance to the shared layers [23]. which could hurt the performance of representation learning as transformations modify the semantics of data. As we emphasized above, the more useful information learned by the discriminator, the better guidance provided to the generator, and the better optimization result reached by the generator. Accordingly, the performance of generative modeling would be affected.
5 Experiments
Our code is available at atbctpe//ggghuu.oocm/ooouuaigags
5.1 SSGAN-LA Faithfully Learns the Real Data Distribution
We experiment on a synthetic dataset to intuitively verify whether SSGAN, SSGAM-MS, and SSGAN-LA can accurately match the real data distribution. The real data distribution is one- dimensional normal distribution Pa = (0, 1). The set of transformations is T = {T K= with the k-th transformation of (xx := 2(k 1) (I; is the identity transformation). We assume that the distributions of real data through different transformations have non-negligible overlaps. The generator and discriminator networks are implemented by multi-layer perceptrons with hidden size of 10 and non- -on-eiityy of Tanh. Figure 2 shows the density estimation of the target and generated data estimated by kernel density estimation 44 through sampling 10k data. SSGAN learns a biased distribution as its self-supe task forces the generator to produce Transformation-llatt
data. SSGAN-MS slightly mitigates this issue, but still cannot accurately approximate the real data distribution. The proposed SSGAN-LA successfully replicates the data distribution and achieves the best maximum mean discrepancy (MMD) [5] results as reported in brackets. In general, experimental results on synthetic data accord well with theoretical results of self-supervised methods.

---

(a) Real Data
(b) SSGAN (0.4367) (c) SSGAN-MS (0.2551) (d) SSGAN-LA (0.0035)
Figure 2: Learned distribution one-dimensional synthetic data. The blue line left-most the untransformed real or generated data distribution. The numbers in brackets are the MMD scores.
(a) DAGAN
(h) SSGAN-LA
Figure Generated samples on CelebA. DAGAN suffers from the augmentation-leakin issue.
5.2 SSGAN-LA Resolves the Augmentation-Leaking Issue
Figure shows images randomly generated by DAGAN and SSGAN-LA the CelebA dataset Apparently, the generator of DAGAN converges to generate the rotated images, validating Theorem that DAGAN cannot guarantee recover the original data distribution under the used ootation-based data augmentation setting. The proposed SSGAN-LA resolves this augmentation-leakinn problem attributed to explicitly identifying different rotated images by predicting the self-supervised pseudo signals (i.e., rotation angles).
34
5.3 Comparison of Sample Quality
We conduct experiments three real-world datasets: CIFAR-10 26
STL-10 H and Tiny-
ImageNet [7 We implement all methods based on unconditional BigGAN 4 without accessing human-annotated labels. We defer the detailed experimental settings into Appendix B due to the limited space. To evaluate the performance of methods on sample quality, we adopt two widely-used evaluation metries: Fréchet Inception Distance (FID) [i7] and Inception Score (IS) 49 We follow the practice of BigGAN to randomly sample 50k images to calculate the IS and FID scores.
As shown in Table D. SSGAN and SSGAN-MS surpass GAN due to the auxiliary self-supervised tasks that encourage discriminators learn stable representations to resist the forgetting issue. As DAGAN suffers from the augmentation-leaking problem (22] in the default setting, up-weight the probability of the identity transformation for DAGAN (named DAGAN+) to avoid the undesirable sugmentation-leaking problem. D and DAGAN-MD have made considerable improvements over GAN on CIFAR-10 and STL-10 but a substantial degradation on Enn-ImmgeNee, indicating
Table 1: FID (L) and IS (t) comparison of methods on CIFAR-10, STL-10, and Tiny-Image
Dataset
Metrie
GAN
SSGAN
SSGAN-MS
DAGAN+ DAGAN-MD SSGAN-LA
FID (1)
10.83
7.52
7.08
10.01
7.80
5.87
CIFAR-10
(t)
8.42
8.29
8.45
8.10
8.31
8.57
FID (L)
20.15
16.81
16.16
16.42
16.16
14.58
STL-10
IS (t)
10.25
10.36
10.40
10.27
10.31
10.61
FID (L)
31.01
30.09
25.76
52.54
39.57
23.69
Tiny- ImageNet
IS (1)
9.80
10.21
10.57
7.55
8.44
11.14

---

Table Accuracy (1) comparison of methods on CIFAR-IO, CIFAR- 100, and Tiny Iny-ImmeeNet based on learned representations extracted from each residual block of the discriminator.
Dataset
Block
GAN
SSGAN
SSGAN
DAGAN+
DAGAN-MD
SSGAN-LA
Block1
0.641
0.636
0.640
0.639
0.633
0.637
Block2
0.729
0.722
0.728
0.726
0.720
0.741
CIFAR-10
Block3
0.765
0.769
0.770
0.764
0.746
0.782
Block4
0.769
0.784
0.787
0.772
0.768
0.803
Block1
0.323
0.320
0.322
0.323
0.324
0.318
Block2
0.456
0.452
0.461
0.460
0.443
0,468
CIFAR-100
Block3
0.494
0.493
0.502
0.493
0.478
0.511
Block4
0.504
0.513
0.521
0.519
0.492
0.543
Block1
0.142
0.138
0.132
0.155
0.135
0.129
Block2
0.221
0.224
0.214
0.184
0.184
0.209
Tiny-ImageNet
Block3
0.251
0.277
0.280
0.211
0.231
0,283
Block4
0.325
0.346
0.344
0.263
0.279
0.349
Block5
0.141
0.313
0.341
0.110
0.224
0.351
that only data augmentation without self- self-supervised signals cannot bring consistent improvements. The proposed SSGAN-LA significantly outperforms existing SSGANs and DAGANs by inheriting their advantages while overcoming their shortcomings. Compared with existing SSGANs, SSGAN- LA inherits the benefits of self-supervised tasks while avoiding the undesired learning objective. Compared with DAGANs, SSGAN-LA inherits the benefits of augmented data and utilizes self- supervised signals to enhance the representation learning ability of the discriminator that could provide more valuable guidance to the generator.
5.4 Comparison of Representation Quality
In order to check whether the diseriminator learns meaningful representations, we train a 10-way logistic regression classifier on CIFAR-1O, 100-way CIFAR-I00 and 200-way on IingeenmaeNet, respectively, using the learned representation of real data extracted from residual blocks in the discriminator. Specifically, we train the classifier with batch size of 128 for 50 epochs. The optimizer Adam with a learning rate 0.05 and decayed by at both epoch 30 and epoch 40, following the practice of [6]. The linear model is trained on the training set and tested on the validation set of the corresponding datasets. Classification accuracy is the evaluation metric.
As shown in Table SSGAN and SSGAN-MS obtain improvements on GAN, confirming that the self-supervised tasks facilitate learning more meaningful representations. DAGAN-MD per- forms substantially worse than the self- DAGAN in some cases, validating that the multiple discriminators of DAGAN-MD tend weaken the representation learning ability in the shared blocks. In other words, the shared blocks are limited invariant features, which may lose useful information about data. Compared with all competitive baselines, the proposed SSGAN-LA achieves the best accuracy results on most blocks, especially the deep blocks. These results verify that the discriminator of SSGAN-LA learns meaningful high-level representations. We argue that the powerful representation ability of the discriminator could in turn promote the performance of generative modeling of the generator.
counterparts and even worse than GAN and
5.5 SSGAN-LA Overcomes Catastrophic Forgetting
Figure g plots the accuracy (calculated following Section 5.4 curves of different methods during GAN training CIFAR-10 With the increase of GAN training iterations, the accuracy results of GAN and DAGAN-MD first increased and then decreased, in- dicating that their discriminators both suffer from catastrophic forgetting some extent. The proposed SSGAN-LA achieves con- secutive increases in terms accuracy and consistently surpasses other competitive methods, verifying that utilizing self supervisson can effectively overcome the catastrophic forgetting issue.
GAi
Figure 4: Accuracy curves dur- ing GAN training on CIFAR-10.

---

Table FID and IS comparison under full and limited data regimes on CIFAR-10 10 and CIFAR 100.
CIFAR-10
CIFAR-100
Metric
Method
100%
20%
10%
100%
20%
10%
BigGAN 4
DiffAugment [66
8.70
14.04
22.40
12.00
22.14
33.70
FID
+ +SSGAN-LA
8.05
11.16
15.08
9.77
15.91
23.17
BigGAN 4 + DiffAugment 666
9.16
8.65
8.09
10.66
9.47
8.38
IS
+ +SSGAN-LA
9.30
8.84
8.30
11.13
10.52
9.81
5.6 SSGAN-LA Improves the Data Efficiency of GANs
In this section, we experiment on the full and limited data regimes on CIFAR-10 10 and CIFAR-100 compared with the ofate-of-the-eee data augmentation method (DiffAugment 66 for training GANs. We adopt the codebase of DiffAAgment and follow its practice to randomly sample 10k data for calculating the FID and IS scores for fair comparison. As shown in Table 3 SSGAN-LA significantly improves the data efficiency of GANs and to our knowledge achieves the new state-of-the-arr FII) and IS results under the full and limited data regimes CIFAR-10 and CIFAR- based the BigGAN 4 backbone. We argue that the reason is that the proposed label-augmented discriminator less easy over-fitting than the normal discriminator by solving a more challenging task.
Related Work
Since generative adversarial networks (GANs) 14! appearance, numerous variants focus on im- proving the generation performance from perspectives of network architecture 48
63
5
23
24
divergence minimization 44
16
BC
5
21
learning procedure [37 [5] 18] weight normaliza
tion 38 50 67 model regularization 46
62
54 and multiple discriminators 4l
12
40
[o M.
Recently, self -ulf-suuprviiion has been applied to GANs to improve the training stability. sulf-supervised learning is family of unsupervised representation learning methods that solve pretext tasks. Various self-supervised pretext tasks on image representation learning have been proposed, e.g., predicting context [1]. solving jigsaw puzzle [42]. and recognizing rotation K introduced auxiliary rotation prediction task 1 onto the discriminator to alleviate catastrophic forgetting. [57] 58 ana-
lyzed the drawback of the self-supervised task 6 and proposed improved self-supervised task. 351 combined self- and semi-supervised learning to surpass fully supervised GANs. K introduced a deshuffling task to the discriminator that solves the jigsaw puzzle. (19) required the discriminator to examine the structural consistency of real images. 45 extended the self supervirrision into the latent space by detecting the latent transformation. A explored contrastive learning and mutual informa- tion maximization to simultancously tackle the catastrophic forgetting and mode collapse issues.
64
presented transformation-baat based consistency regularization, which has improved in [8
69
The
similar label augmentation technique (2 for discriminators is also proposed in DADA 655 and (0)
However, the former (DADA) aimed at data augmentation for few-shot classification, and the latter focused on self-labeling in semi-supervised learning. Most importantly, they are both different from ours with the loss function for the generator, which is the key for the Theorem that guarantees the generator to faithfully learn the real data distribution.
Conclusion
In this paper, we present novel formulation of transformation-basee self-supervised GANs that is able to faithfully learn the real data distribution. Specifically, we unify the original GAN task with the self-supervised task into a single multi-class classification task for the discriminator (forming a label-augmented discriminator) by augmenting the original GAN labels (real or fake) via self- supervision of data transformation. Consequently, the generator is encouraged minimize the reversed Bellback-Leibbes divergence between the real data distribution and the generated data distribution under different transformations provided by the optimal label-augmented discriminator. We provide both theoretical analysis and empirical evidence to support the superiority of the proposed method compared with previous self-supervised GANs and data augmentation GANs, the future, we will explore and exploit the proposed approach to semi- and fully supervised GANs.

---

Acknowledgments and Disclosure of Funding
This work is funded by the National Natural Science Foundation of China under Grant Nos. 62102402, 91746301, and National Key R&D Program of China (2020AAA0105200) Huawei Shen also supported by Beijing Academy of Artificial Intelligence (BAAI) under the grant number BAA12019QN0304 and K.C. Wong Education Foundation.
References
[1] I. Albuquerque, J. Monteiro, T. Doan, B. Considine, T. Falk, and I. Mitliagkas. Multi-objective training of generative adversarial networks with multiple discriminators. In Proceedings of the 36th International Conference on Machine Learning, 2019.
[2] M. Arjovsky, S. Chintala, and L. Bottou. Wasserstein generative adversarial networks. In Proceedings the 34th International Conference Machine Learning, 2017.
[3] G. Baykal and G. Unal. Deshufflegan: A self-supervised gan to improve structure learning. In 2020 IEEE International Conference on Image Processing (ICIP), pages 708-712. IEEE, 2020.
[4] A. Brock, J. Donahue, and K. Simonyan. Large scale GAN training for high fidelity natural image synthesis. In International Conference on Learning Representations, 2019.
[5] T. Chen, M. Lucic, N. Houlsby, and S. Gelly. On self modulation for generative adversarial networks. International Conference on Learning Representations, 2019.
[6] T. Chen, X. Zhai. M. Ritter, M. Lucic, and N. Houlsby. Self-supervised gans via auxiliary rotation loss. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[7] X. Chen, Y. Duan, R. Houthooft, J. Schulman, Sutskever, and P. Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. Advances in neural information processing systems, 2016.
[8] A. Coates, Ng, and H. Lee. An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 215-223. JMLR Workshop and Conference Proceedings, 2011.
[9] H. de Vries, F. Strub, Mary, H. Larochelle, O. Pietquin, and A. C. Courville. Modulating early visual processing by language. In Advances in Neural Information Processing Systems 30. 2017.
[10] T. Doan, J. Monteiro, I. Albuquerque, B. Mazoure, A. Durand, J. Pineau, and R. D. Hjelm. On-line adaptative curriculum learning for gans. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 3470-3477, 2019.
[11] C. Doersch, A. Gupta, and A. A. Efros. Unsupervised visual representation learning by context prediction. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2015.
[12] I. Dunugkar, I. Gemp, and S. Mahadevan. Generative multi-adversarial networks. arXiv preprint 161 2016.
[13] S. Gidaris, P. Singh, and N. Komodakis. Unsupervised representation learning by predicting image rotations. In International Conference Learning Representations, 2018.
[14] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and
Y. Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems
27. 2014.
[15] A. Gretton, K. M. Borgwardt, M. J. Rasch, B. Schölkopf, and A. Smola. A kernel two-sample test. The Journal of Machine Learning Research, 2012.
[16] Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin, and A. C. Courville. Improved training of wasserstein gans. In Advances in Neural Information Processing Systems 30, 2017.
10

---

[17] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter. Gans trained by a two time-scale update rule converge local nash equilibrium. In Advances in Neural Information Processing Systems 30. 2017.
[18] L. Hou, Z. Yuan, L. Huang, H. Shen, X. Cheng, and C. Wang. Slimmable generative adversarial networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 7746-7753, 2021.
[19] R. Huang, W. Xu, T.-.. Lee, A. Cherian, Y. Wang, and T. Marks. Fx-gan: Self-supervised gan learning via feature exchange. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), March 2020.
[20] S. loffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. F. Bach and D. Blei, editors, Proceedings of the 32nd International Conference Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 448-456, Lille, France, 07-09 Jul 2015. PMLR.
[21] A. Conference Machine Learning. 2020.
On relativistic f-divergences. Proceedings of the 37th International
[22] T. Karras, M. Aittala, J. Hellsten, S. Laine, J. Lehtinen, and T. Aila. Training generative adversarial networks with limited data. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 12104-12114. Curran Associates, Inc., 2020.
[23] Karras, S. Laine, and T. Aila. A style-based generator architecture for generative adversarial networks. In The IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2019.
[24] T. Karras, S. Laine, M. Aittala, J. Hellsten, J. Lehtinen, and T. Aila. Analyzing and improving the image quality of stylegan. In JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[25] J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, et al. Overcoming catastrophic forgetting in
J. Quan, T. Ramalho, A. neural networks. Proceedings of the national academy of sciences, 2017.
[26] A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[27] Y. Le and X. Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.
[28] H. Lee, S. J. Hwang, and J. Shin. Self-supervised label augmentation via input transformations. In Proceedings of the 37th International Conference on Machine Learning, 2020.
[29] K. S. Lee, N.-T. Tran, and N.-M. Cheung. Infomax-gan Improved adversarial image generation via information maximization and contrastive learning. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 3942-3952, January 2021.
[30] C.-L. Li, W.-C. Chang, Y. Cheng, Y. Yang, and B. Poczos. Mmd gan: Towards deeper understanding of moment matching network. In Advances in Neural Information Processing Systems, 2017.
[31] F. Liese and Vajda. On divergences and informations in statistics and information theory. IEEE Transactions on Information Theory, 52(10):4394-4112 2006.
[32] J. H. Lim and J. C. Ye. Geometric gan. arXiv preprint arXiv: ww.Xiv:/70500000 770.022894 2017.
[33] Z. Lin, A. Khetan, G. Fanti, and S. Oh. Pacgan: The power of two samples in generative adversarial networks. In Advances in Neural Information Processing Systems, 2018
[34] Z. Liu, P. Luo, X. Wang, and X. Tang. Deep learning face attributes in the wild. Proceedings of International Conference Computer Vision (ICCV), December 2015.
[35] M. Lučic, M. Tschannen, M. Ritter, X. Zhai, O. Bachem, and S. Gelly. High-fidelity image generation with fewer labels. In Proceedings of the 36th International Conference on Machine Learning, 2019.
11

---

[36] Q. Mao. H.-Y. Lee, H.-Y. Tseng, S. Ma, and M.-H. Yang. Mode seeking generative adversarial networks for diverse image synthesis. In Proceedings the Vision and Puttern Recognition (CVPR), 2019.
the IEEE/CVF Conference on Computer
[37] L. Metz, B. Poole, D. Pfau, and J. Sohl-Dickstein. Unrolled generative adversarial networks. arXiv preprint
2016.
[38] T. Miyato, T. Kataoka, M. Koyama, and Y. Yoshida. Spectral normalization for generative adversarial networks. In International Conference on Learning Representations, 2018.
[39] T. Miyato and M. Koyama. eGANs with projection discriminator, In International Conference on Learning Representations, 2018.
[40] B. Neyshabur, S. Bhojanapalli, and A. Chakrabarti. Stabilizing gan training with multiple random projections. arXiv preprint arXiv 2017.
[41] T. Nguyen, T. Le, H. Vu, and D. Phung. Dual discriminator generative adversarial nets. In Advances in Neural Information Processing Systems, 2017.
[42] M. Noroozi and P. Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In European Conference on Computer Vision, 2016.
[43] S. Nowozin, B. Cseke, and R. Tomioka. f-gan: Training generative neural samplers using variational divergence minimization. In Advances in Neural Information Processing Systems 29. 2016.
[44] E. Parzen. On estimation of a probability density function and mode. The annals of mathematical statistics, 1962.
[45] P. Patel, N. Kumari, M. Singh, and B. Krishnamurthy, Lt-gan: Self-supervised gan with latent transformation detection. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 3189-3198, January 2021.
[46] H. Petzka, A. Fischer, and D. Lukovnikov. On the regularization of wasserstein GANs. In International Conference on Learning Representations, 2018.
[47] Y. Qiao and N. Minematsu. A study on invariance of f-divergence and its application to speech recognition. IEEE Transactions on Signal Processing, 58(7):3884-3890, 2010.
[48] A. Radford, Metz, and S. Chintala. Unsupervised representation learning with deep convolu- tional generative adversarial networks. arXiv preprint arXiv:1511.066 2015.
[49] T. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford, X. Chen, and X. Chen. Improved techniques for training gans. In Advances in Neural Information Processing Systems 29. 2016.
[50] A. Sanyal, H. Torr, and K. Dokania. Stable rank normalization for improved generalization in neural networks and gans. In International Conference on Learning Representations, 2020.
[51] F. Schaefer, H. Zheng, and A. Anandkumar. Implicit competitive regularization in GANs. In Proceedings of the 37th International Conference on Machine Learning, 2020.
[52] A. Srivastava, L. Valkov, C. Russell, M. U. Gutmann, and C. Sutton. Veegan: Reducing mode collapse in gans using implicit variational learning. In Advances in Neural Information Processing Systems, 2017.
[53] C. Tao, L. Chen, R. Henao, J. Feng, and L. C. Duke. Chi-square generative adversarial network. In Proceedings of the 35th International Conference on Machine Learning, 2018
[54] D. Terjék. Adversarial lipschitz regularization. In International Conference on Learning Representations, 2020.
[55] H. Thanh-Tung and T. Tran. Catastrophic forgetting and mode collapse in gans. In 2020 International Joint Conference Neural Networks (IJCNN, pages 1-10. IEEE, 2020.
12

---

[56] D. Tran, R. Ranganath, and D. M. Blei. Deep and hierarchical implicit models. arXiv preprint arXiv:1702.08896 2017.
[57] N.-T. Tran, V.-H. Tran, B.-N. Nguyen, L. Yang, and N.-M. M. Cheung. Self-supervised gan: Analysis and improvement with multi-class minimax game. In Advances in Neural Information Processing Systems, 2019.
[58] N.-T. Tran, V.-H. Tran, N.-B. Nguyen, and N.-M. Cheung. An improved self-supervised gan via adversarial training. arXiv preprint arXiv:/908 190.00699,, 2019.
[59] N.-T. Tran, V.-H. Tran. N.-B. Nguyen, T.-K. Nguyen, and N.-M. Cheung. On data augmentation for gan training. IEEE Transactions on Image Processing, 30: 30:1882-1897, 2021.
[60] T. Watanabe and P. Favaro. A unified generative adversarial network training via self-labeling and self-attention. In M. Meila and T. Zhang, editors, Proceedings of the 38th International Conference on Machine Learning. volume 139 of Proceedings of Machine Learning Research, pages 11024-1 1034. PMLR, 18-24 Jul 2021.
[61] C. Xiao, P. Zhong, and C. Zheng. Bourgan: Generative networks with metric embeddings. In Advances in Neural Information Processing Systems, 2018.
[62] S. Yamaguchi and M. Koyama. Distributional concavity regularization for GANs. In Interna- lional Conference on Learning Representations, 2019.
[63] H. Zhang, I. Goodfellow, D. Metaxas, and A. Odena. Self-attention generative adversarial networks. Proceedings of the 36th International Conference on Machine Learning, 2019.
[64] H. Zhang, Z. Zhang, A. Odena, and H. Lee. Consistency regularization for generative adversarial networks. In International Conference on Learning Representations, 2020.
[65] X. Zhang, Z. Wang, D. Liu, and Q. Ling. Dada: Deep adversarial data augmentation for extremely low data regime classification. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 2807-2811, IEEE, 2019.
[66] S. Zhao, Z. Liu, J. Lin, J.-Y. Zhu, and S. Han. Differentiable augmentation for data-efficient gan training. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, A Advances in Neural Information Processing Systems, volume 33, pages 7559-7570. Curran Associates, Inc., 2020.
[67] Y. Zhao, C. Li, P. Yu, J. Gao, and C. Chen. Feature quantization improves GAN training. In Proceedings of the 37th International Conference on Machine Learning, 2020.
[68] Z. Zhao, Singh, H. Lee, Z. Zhang, A. Odena, and H. Zhang. Improved consistency regular- ization for gans. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pages 11033-11041 2021.
[69] Z. Zhao, Z. Zhang, T. Chen, S. Singh, and H. Zhang. Image augmentations for gan training. arXiv preprint arXiv:2006.0 20Xx::2006.02555, 2020.
[70] Z. Zhou, J. Liang. Y. Song, L. Yu, H. Wang, W. Zhang, Y. Yu, and Z. Zhang. Lipschitz generative adversarial nets. In Proceedings of the 36th International Conference on Machine Learning, 2019.
13


Reviews:
[{"review_id": "w1IHQ64xniW", "reviewer": "Reviewer_f1xR", "summary": "The authors propose a label augmentation GAN that combines the discriminator and the classifier in self-supervised learning GANs and show the priority on a variety of generative modeling tasks.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "originality: The idea is novel. Combination of the classifier and the discriminator sounds natural and is an improvement over self-supervised GANs.\n\nquality: The theorems look correct and the experiments seems solid. The only thing is that the numbers reported by the authors in Table seems to be different from BigGAN and SSGAN-MS. For example, BigGAN authors reportThe author may need to explain why. For example, the BigGAN authors report IS 9.22 and FID 14.73 which are different from the authors' IS 8.42 and FID 10.83. It may be an architecture difference but why not use the same architecture as BigGAN.\n\nclarity: The paper is clearly written. The authors may cite [45] for theorem 2 and change \"discriminator\" to \"classifier\" in line 108. It seems that \"classifier\" and \"generator\" are competing against each other in the context.\n\nsignificance: The only concern I have about the paper is the significance. The method seems to be novel but not significant. It seems to be a natural extension of self-supervised GANs. The paper seems to contain some theorems but there are still many loose points. It's unclear to me why self-supervised GAN can learn \"stable\" and \"high-level\" representations. I may be careless and skip some important details. Could the authors help me find which theorem in the paper gives the guarantee that the representation learned is \"stable\" and \"high-level\"?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors do not mention negative societal impact. One negative impact can be some improper use of generative models, such as DeepFake.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "ppD0YrrHh8H", "reviewer": "Reviewer_qAsa", "summary": "The paper presents an approach for training GANs with transformation-based self-supervision. It unifies the original GAN task with the self-supervised task into a single multi-class classification task for the discriminator by augmenting the real/fake labels with the self-supervised labels. The authors demonstrate that the proposed approach outperforms some existing transformation-based self-supervised and data augmentation GANs using experiments on CIFAR-10, STL-10 and Tiny-ImageNet. They also provide a theoretical analysis of their method. ", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**:\n\n\n- The proposed approach is intuitive, and outperforms some of the existing self-supervised and augmentation GANs. \n\n\n**Weaknesses**:\n\n\n- There are other works not included in comparisons. Karras et al. [A] present a method for training GANs with limited data, which achieves an FID of 2.92 on CIFAR-10. This is better than the FID of 5.87 reported in this paper. Another relevant work is [53] (differentiable augmentation GAN) which is not included in comparisons. The authors mention that \"we do not aim to compare with state-of-the-art data augmentation GANs in this work\" (L 179). This is unclear to me as they already provide a comparison with another augmentation method (DAGAN). Both self-supervised and augmentation GAN approaches aim to improve the final generation quality, so comparison with methods in both of these domains is expected. \n\n\n-  Contributions of the paper are modest for this venue. The proposed approach is quite similar to SSGAN(-MS), with some relatively minor differences such as applying self-supervision to real samples as well. Theoretical analysis in the paper is also similar to the original GAN paper. \n\n\n- Resolution and complexity of the datasets used in the paper are limited. The maximum resolution used is 64x64 for Tiny ImageNet. [A, 53] show results on higher resolution datasets.    \n\n\n[A] Training Generative Adversarial Networks with Limited Data; Karras et al.; NeurIPS 2020", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "Yiob8nzbSJJ", "reviewer": "Reviewer_VBjE", "summary": "In the context of using self-supervision signals in the GAN training framework, this paper proposes to use a multi-class classification approach in the discriminator paired with the task of predicting the rotation of an image.  This is proposed instead of using two separate discriminators: one for the rotation prediction and the other for the real/fake classification.\nThe idea supposes a small and simple change to existing methods that, although in terms of novelty it is limited, it is justified theoretically and quantitative empirical results support the idea.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is generally well written and clear. In my opinion, the experimental section is sufficient and it is tested on enough datasets. The idea is simple but well motivated, and it improves over the self-supervised baselines. \n\nAlthough the paper states that it does not aim to compare against state of the art data augmentation techniques, it is a very important comparison to do. If data augmentation works better than self-supervision in GANs, why use self-supervision to begin with? In this case, it is not clear that the data augmentations can directly be combined with the self-supervised rotation signals. \n\nMoreover, the section regarding data augmentations in the paper (and the baseline comparisons) revolve around rotations, which is not the best data augmentation strategy one could use. Moreover, the paper talks about transformation leaking, when in [1], experimental results show that, as long as the non-transformed (0 degrees) image is seen enough times, leakage should not occur. Why was this not considered in this paper, whether it is in the discussion part and/or the experimental results? Could the authors elaborate on this? My concern is that the data augmentation baselines might be at a disadvantage and therefore, be an unfair comparison. \n\nAs an additional thought, the idea of using rotations as self-supervision signals in the self-supervision community is arguably outdated. There is other self-supervision tasks or transformations to predict in the discriminator that could be borrowed from the self-supervised learning community to apply them to GAN training, and perhaps be more competitive with the extensive data augmentation proposed in [1], for example. \n\nL.90: if I understand correctly, the generator wants to fool the rotation classifier as much as if wants to fool the discriminator, so \nI suggest changing the term “collaborate” for another more appropriate word. \n\n[1] Zhao, S., Liu, Z., Lin, J., Zhu, J. Y., & Han, S. (2020). Differentiable augmentation for data-efficient gan training. arXiv preprint arXiv:2006.10738.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have not addressed the societal impact. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "HTrJz54-cjN", "reviewer": "Reviewer_NbmS", "summary": "\nThe authors argue that GAN discriminators experience catastrophic forgetting due to instability during training (the exact source of the instability the authors refer to is not clear). The authors claim that self-supervised methods, that provide an auxiliary loss (via an auxiliary self-supervised task),for GAN training can help prevent catastrophic forgetting, however they point out two weaknesses with the current methods:\n(1) The auxiliary self-supervised losses do not consider the data synthesised by the generator.\n(2) They have additional hyper-parameters that need to be fine-tuned to balance losses.\n\nIn this paper the authors train the self-supervised classifier on the transformed generate data as well as the transformed real data. Rather than having a separate classifier the discriminator predict the label, this avoids the need for balance hyper-parameters.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Originality**\n\n- The authors extend self supervised GANs to learn a join distribution p(real, label) rather than learning each independently.\n\n**Quality**\n\n- It’s good that the authors compare to other data augmentation methods to distinguish the benefits of self-supervised training from those of data augmentation.\n\n- Figure 2: It would be helpful if all figures shared the same y axis this would highlight better that benefits of SSGAN-LA over other the baselines.\n\n- Synthetic dataset: in general (and also according to this paper) when training a generative model the objective is to learn a distribution that matches the training data distribution. Given this, I’m unsure why experiments in section 5.1 are trying to recover the distribution of the augmented data? From my understanding of the paper the goal should be to recover the original training data distribution, the data augmentation used in self-supervised learning should only improve the model’s ability to recover that distribution. Ultimately, these results suggest that the SSGAN-LA model learns to generate samples from outside the original training data distribution? Some clarification on this would be really helpful, thank you.\n\nFurther in Section 5.2.1, qualitative results: the authors suggest the fact the DAGAN synthesises transformed data samples is a weakness, adding additional confusion to the motivation behind results in Figure 2.\n\n- The authors compare SSGAN-LA to a large and relevant number of baselines models on three natural image datasets.\n\n- Table 1: The authors suggest SSGAN-LA outperforms all baselines on all tasks, reporting FID and IS in all cases. It’s not clear that SSGAN-LA is statistically significantly better than DAGAN-MD on the STL-10 task, may the authors please clarify? How reproducible are these results? How many runs did you do?\n\n- In some cases the authors have put answers to the Checklist in the checklist rather than in the paper. For example the hardware used for training and the implementation details. These should be in the main paper.\n\n- 5.2.2 Image classification, Table 2: the authors show that the discriminator learns representations useful for classification. The SSGAN-LA appears to show some improvement, however the authors do not provide stdev. so it is hard to tell if these improvements are statistically significant. For example, CIFAR-100 SSGAN-MS performance is very close to SSGAN-LA.\n\n**Clarity**\n\n- Line 23: What do the authors mean by a “non-stationary learning environment”? Is there a more precise definition? Are you referring to the learning dynamics or the distribution of synthesised data?\n\n- Line 31: What do you mean by “stable representations”? Are these representations that help to improve stability during training? Or representations that don’t change significantly between training steps? \n\n- Line 48 - 54: From the introduction it’s not clear what the discriminator will predict. Line 48-49 suggests it predicts only the label, but 52-53 suggests it predicts “real” or “fake”\n\n- Line 100: What is meant by “enforces the generator to produce rotation-detectable image”? Do you mean images, whose orientation can be easily predicted (by a classifier)? \n\n- Figure 1: What is SNGAN?\n\n- Proposition 1: To have notation more consistent with the GAN literature,  it would be good to super-script D with a * e.g. D*(k, 1|y) = … .\n\n- Line 159 - 161: The authors claim “Compared with standard GANs, the transformation prediction especially on real samples of SSGAN LA establishes stationary self-supervised signals to prevent the discriminator from the forgetting problem.” - what is means by stationary self-supervised signals? At this point it is still not clear how training signals from the self-supervised tasks help to prevent catastrophic forgetting?\n\n - Does Theorem 5 apply to DAGAN-MD,  SSGAN-LA or both?\n\n- On a first parse it was not clear that the role of the  “The importance of Self-supervision” section was to introduce baselines? It may make sense to refer to this section as “Related Work” or “Related Work: The importance of Self-supervision” or to include is in the “Experiments” section. \n\n**Significance**\n\n- The approach presented in the paper is an incremental improvement on existing SS-GAN architectures and the improvements.\n\n- There is no clear theoretical contribution in the main body of the paper.\n\n- The link between the proposed approach and catastrophic forgetting are not made clear in the paper. \n\n- The authors do show some improvements on image synthesis, and show the application of the learned representations to image classification.\n\n- The proposed approach does not require a hyper-parameter to trade off between the self-supervision loss and the discriminator loss.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors do not discuss the limitations of their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This was a borderline paper with a good amount of discussion both between the reviewers and reviewers and authors. I'll try to summarize some of the pros and cons of this work:

* novel yet simple extension to SSGAN. Simplicity is good, but some reviewers have argued that contribution is incremental.
* the theoretical contribution is not in the main body of the paper (this is easily fixed)
* the link with catastrophic forgetting should be made more obvious in the paper itself.
* the results are on small resolution sets only (some reviewers argued that access to resources is not an excuse, but I respectfully disagree).

All in all, I am inclined to accept this work based on the fact that it's a simple, principled, theoretically grounded change that seems to work well in comparable empirical settings on lower-resolution image sets. And I think self-supervised GANs are an interesting practically useful line of research, where novel insights may be useful to many researchers.

 
Decision: Accept (Poster)