Self-Supervised GANs with Label Augmentation

1 Introduction . Generative adversarial networks ( GANs ) [ 14 ] have developed rapidly in synthesizing realistic images in recent years [ 4 , 23 , 24 ] . Conventional GANs consist of a generator and a discriminator , which are trained adversarially in a minimax game . The discriminator attempts to distinguish the real data from the fake ones that are synthesized by the generator , while the generator aims to confuse the discriminator to reproduce the real data distribution . However , the continuous evolution of the generator results in a non-stationary learning environment for the discriminator . In such a non-stationary environment , the single discriminative signal ( real or fake ) provides unstable and limited information to the discriminator [ 37 , 7 , 70 , 57 ] . Therefore , the discriminator may encounter catastrophic forgetting [ 25 , 6 , 55 ] , leading to the training instability of GANs and thereby mode collapse [ 52 , 61 , 33 , 36 ] of the generator in learning high-dimensional , complex data distributions . To alleviate the forgetting problem of the discriminator , transformation-based self-supervised tasks such as rotation recognition [ 13 ] have been applied to GANs [ 6 ] . Self-supervised tasks are designed to predict the constructed pseudo-labels of data , establishing a stationary learning environment as the distribution of training data for this task does not change during training . The discriminator enhanced by the stationary self-supervision is able to learn stable representations to resist the forgetting problem , and thus can provide continuously informative gradients to obtain a well-performed generator . After obtaining empirical success , self-supervised GAN was theoretically analyzed and partially improved by a new self-supervised task [ 57 ] . However , these self-supervised tasks cause a goal inconsistent with 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . generative modeling that aims to learn the real data distribution . In addition , the hyper-parameters that trade-off between the original GAN task and the additional self-supervised task bring extra workload for tuning on different datasets . To sum up , we still lack a proper approach to overcome catastrophic forgetting and improve the training stability of GANs via self-supervised learning . In this paper , we first point out that the reason for the presence of undesired goal in existing selfsupervised GANs is that their self-supervised classifiers that train the generator are unfortunately agnostic to the generator distribution . In other words , their classifiers can not provide the discrepancy between the real and generated data distributions to teach the generator to approximate the real data distribution . This issue originates from that these classifiers are trained to only recognize the pseudolabel of the transformed real data but not the transformed generated data . Inspired by this observation , we realize that the self-supervised classifier should be trained to discriminatively recognize the corresponding pseudo-labels of the transformed generated data and the transformed real data . In this case , the classifier includes the task of the original discriminator , thus the original discriminator could even be omitted . Specifically , we augment the GAN labels ( real or fake ) via self-supervision of data transformation to construct augmented labels for a novel discriminator , which attempts to recognize the performed transformation of transformed real and fake data while simultaneously distinguishing between real and fake . By construction , this formulates a single multi-class classification task for the label-augmented discriminator , which is different from a single binary classification task in standard GANs or two separate classification tasks in existing self-supervised GANs . As a result , our method does not require any hyper-parameter to trade-off between the original GAN objective and the additional self-supervised objective , unlike existing self-supervised GANs . Moreover , we prove that the optimal label-augmented discriminator is aware of both the generator distribution and the data distribution under different transformations , and thus can provide the discrepancy between them to optimize the generator . Theoretically , the generator can replicate the real data distribution at the Nash equilibrium under mild assumptions that can be easily satisfied , successfully solving the convergence problem of previous self-supervised GANs . The proposed method seamlessly combines two unsupervised learning paradigms , generative adversarial networks and self-supervised learning , to achieve stable generative modeling of the generator and superior representation learning of the discriminator by taking advantage of each other . In the case of generative modeling , the self-supervised subtask resists the discriminator catastrophic forgetting , so that the discriminator can provide continuously informative feedback to the generator to learn the real data distribution . In the case of representation learning , the augmented-label prediction task encourages the discriminator to simultaneously capture both semantic information brought by the real/fake-distinguished signals and the self-supervised signals to learn comprehensive data representations . Experimental results on various datasets demonstrate the superiority of the proposed method compared to competitive methods on both generative modeling and representation learning . 2 Preliminaries . 2.1 Generative Adversarial Networks . Learning the underlying data distribution Pd , which is hard to specify but easy to sample from , lies at the heart of generative modeling . In the branch of deep latent variable generative models , the generator G : Z ! X maps a latent code z 2 Z endowed with a tractable prior Pz ( e.g. , N ( 0 , I ) ) to a data point x 2 X , and thereby induces a generator distribution Pg = G # Pz . To learn the data distribution through the generator ( i.e. , Pg = Pd ) , one can minimize the Jensen–Shannon ( JS ) divergence between the data distribution and the generator distribution ( i.e. , minG DJS ( PdkPg ) ) . To estimate this divergence , generative adversarial networks ( GANs ) [ 14 ] leverage a discriminator D : X ! { 0 , 1 } that distinguishes the real data from the generated ones . Formally , the objective function for the discriminator and generator of the standard minimax GAN [ 14 ] is defined as follows : min G max D V ( G , D ) = Ex⇠Pd [ logD ( 1|x ) ] + Ex⇠Pg [ logD ( 0|x ) ] , ( 1 ) where D ( 1|x ) = 1 D ( 0|x ) 2 [ 0 , 1 ] indicates the probability that a data x is distinguished as real by the discriminator . While theoretical guarantees hold in standard GANs , the non-stationary learning environment caused by the evolution of the generator contributes to catastrophic forgetting of the discriminator [ 6 ] , which means that the discriminator forgets the previously learned knowledge by over-fitting the current data and may therefore hurt the generation performance of the generator . 2.2 Self-Supervised GAN . To mitigate catastrophic forgetting of the discriminator , self-supervised GAN ( SSGAN ) [ 6 ] introduced an auxiliary self-supervised rotation recognition task to a classifier C : X̃ = T ( X ) ! { 1 , 2 , · · · , K } that shares parameters with the discriminator , and forced the generator to collaborate with the classifier on the rotation recognition task . Formally , the objective functions of SSGAN are given by : max D , C V ( G , D ) + d · Ex⇠Pd , Tk⇠T [ logC ( k|Tk ( x ) ) ] , ( 2 ) min G V ( G , D ) g · Ex⇠Pg , Tk⇠T [ logC ( k|Tk ( x ) ) ] , ( 3 ) where d and g are two hyper-parameters that balance the original GAN task and the additional self-supervised task , T = { Tk } Kk=1 is the set of deterministic data transformations such as rotations ( i.e. , T = { 0 , 90 , 180 , 270 } ) . Throughout this paper , data transformations are uniformly sampled by default ( i.e. , p ( Tk ) = 1K , 8Tk 2 T ) , unless otherwise specified . Theorem 1 ( [ 57 ] ) . Given the optimal classifier C⇤ ( k|x̃ ) = p Tk d ( x̃ ) PK k=1 p Tk d ( x̃ ) of SSGAN , at the equilibrium point , maximizing the self-supervised task for the generator is equivalent to : max G 1 K KX k=1 '' E x̃⇠PTkg log pTkd ( x̃ ) PK k=1 p Tk d ( x̃ ) ! # , ( 4 ) where PTkg , P Tk d indicate the distribution of transformed generated or real data x̃ 2 X̃ under the transformation Tk with density of pTkg ( x̃ ) = R ( x̃ Tk ( x ) ) pg ( x ) dx or pTkd ( x̃ ) = R ( x̃ Tk ( x ) ) pd ( x ) dx . All proofs of theorems/propositions including this one ( for completeness ) in this paper are referred to Appendix A. Theorem 1 reveals that the self-supervised task of SSGAN enforces the generator to produce only rotation-detectable images other than the whole real images [ 57 ] , causing a goal inconsistent with generative modeling that faithfully learns the real data distribution . Consequently , SSGAN still requires the original GAN task , which forms a multi-task learning framework that needs hyper-parameters to trade off the two separate tasks . 2.3 SSGAN with Multi-Class Minimax Game . To improve the self-supervised task in SSGAN , SSGAN-MS [ 57 ] proposed a multi-class minimax self-supervised task by adding the fake class 0 to a label-extend classifier C+ : X̃ ! { 0 , 1 , 2 , · · · , K } , where the generator and the classifier compete on the self-supervised task . Formally , the objective functions of SSGAN-MS are defined as follows : max D , C+ V ( G , D ) + d · ( Ex⇠Pd , Tk⇠T [ logC+ ( k|Tk ( x ) ) ] + Ex⇠Pg , Tk⇠T [ logC+ ( 0|Tk ( x ) ) ] ) , ( 5 ) min G V ( G , D ) g · ( Ex⇠Pg , Tk⇠T [ logC+ ( k|Tk ( x ) ) ] Ex⇠Pg , Tk⇠T [ logC+ ( 0|Tk ( x ) ) ] ) . ( 6 ) Theorem 2 . Given the optimal classifier C⇤ + ( k|x̃ ) = p T d ( x̃ ) pTg ( x̃ ) p Tk d ( x̃ ) PK k=1 p Tk d ( x̃ ) C⇤ + ( 0|x̃ ) of SSGAN-MS , at the equilibrium point , maximizing the self-supervised task for the generator is equivalent to 1 : min G DKL ( PTg kPTd ) 1 K KX k=1 '' E x̃⇠PTkg log pTkd ( x̃ ) PK k=1 p Tk d ( x̃ ) ! # , ( 7 ) where PTg , PTd represent the mixture distribution of transformed generated or real data x̃ 2 X̃ with density of pTg ( x̃ ) = PK k=1 p ( Tk ) p Tk g ( x̃ ) or p T d ( x̃ ) = PK k=1 p ( Tk ) p Tk d ( x̃ ) . 1Note that our Theorem 2 corrects the wrong version in the SSGAN-MS paper [ 57 ] , where the authors mistakenly regard p T d ( x̃ ) pTg ( x̃ ) = PK k=1 p ( Tk ) p Tk d ( x̃ ) PK k=1 p ( Tk ) p Tk g ( x̃ ) as p Tk d ( x̃ ) p Tk g ( x̃ ) in their proof . Please see Appendix A.2 for details . Our Theorem 2 states that the improved self-supervised task for the generator of SSGAN-MS , under the optimal classifier , retains the inconsistent goal in SSGAN that enforces the generator to produce typical rotation-detectable images rather than the entire real images . In addition , minimizing DKL ( PTg kPTd ) can not guarantee the generator to recover the real data distribution under the used data transformation setting ( see Theorem 4 ) . Due to the existence of these issues , SSGAN-MS also requires the original GAN task to approximate the real data distribution . We regard the learning paradigm of SSGAN-MS and SSGAN as the multi-task learning framework ( see Figure 1 ) .

Summary:  The authors argue that GAN discriminators experience catastrophic forgetting due to instability during training (the exact source of the instability the authors refer to is not clear). The authors claim that self-supervised methods, that provide an auxiliary loss (via an auxiliary self-supervised task),for GAN training can help prevent catastrophic forgetting, however they point out two weaknesses with the current methods: (1) The auxiliary self-supervised losses do not consider the data synthesised by the generator. (2) They have additional hyper-parameters that need to be fine-tuned to balance losses.  In this paper the authors train the self-supervised classifier on the transformed generate data as well as the transformed real data. Rather than having a separate classifier the discriminator predict the label, this avoids the need for balance hyper-parameters.