Self-Supervised GANs with Label Augmentation

1 Introduction . Generative adversarial networks ( GANs ) [ 14 ] have developed rapidly in synthesizing realistic images in recent years [ 4 , 23 , 24 ] . Conventional GANs consist of a generator and a discriminator , which are trained adversarially in a minimax game . The discriminator attempts to distinguish the real data from the fake ones that are synthesized by the generator , while the generator aims to confuse the discriminator to reproduce the real data distribution . However , the continuous evolution of the generator results in a non-stationary learning environment for the discriminator . In such a non-stationary environment , the single discriminative signal ( real or fake ) provides unstable and limited information to the discriminator [ 37 , 7 , 70 , 57 ] . Therefore , the discriminator may encounter catastrophic forgetting [ 25 , 6 , 55 ] , leading to the training instability of GANs and thereby mode collapse [ 52 , 61 , 33 , 36 ] of the generator in learning high-dimensional , complex data distributions . To alleviate the forgetting problem of the discriminator , transformation-based self-supervised tasks such as rotation recognition [ 13 ] have been applied to GANs [ 6 ] . Self-supervised tasks are designed to predict the constructed pseudo-labels of data , establishing a stationary learning environment as the distribution of training data for this task does not change during training . The discriminator enhanced by the stationary self-supervision is able to learn stable representations to resist the forgetting problem , and thus can provide continuously informative gradients to obtain a well-performed generator . After obtaining empirical success , self-supervised GAN was theoretically analyzed and partially improved by a new self-supervised task [ 57 ] . However , these self-supervised tasks cause a goal inconsistent with 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . generative modeling that aims to learn the real data distribution . In addition , the hyper-parameters that trade-off between the original GAN task and the additional self-supervised task bring extra workload for tuning on different datasets . To sum up , we still lack a proper approach to overcome catastrophic forgetting and improve the training stability of GANs via self-supervised learning . In this paper , we first point out that the reason for the presence of undesired goal in existing selfsupervised GANs is that their self-supervised classifiers that train the generator are unfortunately agnostic to the generator distribution . In other words , their classifiers can not provide the discrepancy between the real and generated data distributions to teach the generator to approximate the real data distribution . This issue originates from that these classifiers are trained to only recognize the pseudolabel of the transformed real data but not the transformed generated data . Inspired by this observation , we realize that the self-supervised classifier should be trained to discriminatively recognize the corresponding pseudo-labels of the transformed generated data and the transformed real data . In this case , the classifier includes the task of the original discriminator , thus the original discriminator could even be omitted . Specifically , we augment the GAN labels ( real or fake ) via self-supervision of data transformation to construct augmented labels for a novel discriminator , which attempts to recognize the performed transformation of transformed real and fake data while simultaneously distinguishing between real and fake . By construction , this formulates a single multi-class classification task for the label-augmented discriminator , which is different from a single binary classification task in standard GANs or two separate classification tasks in existing self-supervised GANs . As a result , our method does not require any hyper-parameter to trade-off between the original GAN objective and the additional self-supervised objective , unlike existing self-supervised GANs . Moreover , we prove that the optimal label-augmented discriminator is aware of both the generator distribution and the data distribution under different transformations , and thus can provide the discrepancy between them to optimize the generator . Theoretically , the generator can replicate the real data distribution at the Nash equilibrium under mild assumptions that can be easily satisfied , successfully solving the convergence problem of previous self-supervised GANs . The proposed method seamlessly combines two unsupervised learning paradigms , generative adversarial networks and self-supervised learning , to achieve stable generative modeling of the generator and superior representation learning of the discriminator by taking advantage of each other . In the case of generative modeling , the self-supervised subtask resists the discriminator catastrophic forgetting , so that the discriminator can provide continuously informative feedback to the generator to learn the real data distribution . In the case of representation learning , the augmented-label prediction task encourages the discriminator to simultaneously capture both semantic information brought by the real/fake-distinguished signals and the self-supervised signals to learn comprehensive data representations . Experimental results on various datasets demonstrate the superiority of the proposed method compared to competitive methods on both generative modeling and representation learning . 2 Preliminaries . 2.1 Generative Adversarial Networks . Learning the underlying data distribution Pd , which is hard to specify but easy to sample from , lies at the heart of generative modeling . In the branch of deep latent variable generative models , the generator G : Z ! X maps a latent code z 2 Z endowed with a tractable prior Pz ( e.g. , N ( 0 , I ) ) to a data point x 2 X , and thereby induces a generator distribution Pg = G # Pz . To learn the data distribution through the generator ( i.e. , Pg = Pd ) , one can minimize the Jensen–Shannon ( JS ) divergence between the data distribution and the generator distribution ( i.e. , minG DJS ( PdkPg ) ) . To estimate this divergence , generative adversarial networks ( GANs ) [ 14 ] leverage a discriminator D : X ! { 0 , 1 } that distinguishes the real data from the generated ones . Formally , the objective function for the discriminator and generator of the standard minimax GAN [ 14 ] is defined as follows : min G max D V ( G , D ) = Ex⇠Pd [ logD ( 1|x ) ] + Ex⇠Pg [ logD ( 0|x ) ] , ( 1 ) where D ( 1|x ) = 1 D ( 0|x ) 2 [ 0 , 1 ] indicates the probability that a data x is distinguished as real by the discriminator . While theoretical guarantees hold in standard GANs , the non-stationary learning environment caused by the evolution of the generator contributes to catastrophic forgetting of the discriminator [ 6 ] , which means that the discriminator forgets the previously learned knowledge by over-fitting the current data and may therefore hurt the generation performance of the generator . 2.2 Self-Supervised GAN . To mitigate catastrophic forgetting of the discriminator , self-supervised GAN ( SSGAN ) [ 6 ] introduced an auxiliary self-supervised rotation recognition task to a classifier C : X̃ = T ( X ) ! { 1 , 2 , · · · , K } that shares parameters with the discriminator , and forced the generator to collaborate with the classifier on the rotation recognition task . Formally , the objective functions of SSGAN are given by : max D , C V ( G , D ) + d · Ex⇠Pd , Tk⇠T [ logC ( k|Tk ( x ) ) ] , ( 2 ) min G V ( G , D ) g · Ex⇠Pg , Tk⇠T [ logC ( k|Tk ( x ) ) ] , ( 3 ) where d and g are two hyper-parameters that balance the original GAN task and the additional self-supervised task , T = { Tk } Kk=1 is the set of deterministic data transformations such as rotations ( i.e. , T = { 0 , 90 , 180 , 270 } ) . Throughout this paper , data transformations are uniformly sampled by default ( i.e. , p ( Tk ) = 1K , 8Tk 2 T ) , unless otherwise specified . Theorem 1 ( [ 57 ] ) . Given the optimal classifier C⇤ ( k|x̃ ) = p Tk d ( x̃ ) PK k=1 p Tk d ( x̃ ) of SSGAN , at the equilibrium point , maximizing the self-supervised task for the generator is equivalent to : max G 1 K KX k=1 '' E x̃⇠PTkg log pTkd ( x̃ ) PK k=1 p Tk d ( x̃ ) ! # , ( 4 ) where PTkg , P Tk d indicate the distribution of transformed generated or real data x̃ 2 X̃ under the transformation Tk with density of pTkg ( x̃ ) = R ( x̃ Tk ( x ) ) pg ( x ) dx or pTkd ( x̃ ) = R ( x̃ Tk ( x ) ) pd ( x ) dx . All proofs of theorems/propositions including this one ( for completeness ) in this paper are referred to Appendix A. Theorem 1 reveals that the self-supervised task of SSGAN enforces the generator to produce only rotation-detectable images other than the whole real images [ 57 ] , causing a goal inconsistent with generative modeling that faithfully learns the real data distribution . Consequently , SSGAN still requires the original GAN task , which forms a multi-task learning framework that needs hyper-parameters to trade off the two separate tasks . 2.3 SSGAN with Multi-Class Minimax Game . To improve the self-supervised task in SSGAN , SSGAN-MS [ 57 ] proposed a multi-class minimax self-supervised task by adding the fake class 0 to a label-extend classifier C+ : X̃ ! { 0 , 1 , 2 , · · · , K } , where the generator and the classifier compete on the self-supervised task . Formally , the objective functions of SSGAN-MS are defined as follows : max D , C+ V ( G , D ) + d · ( Ex⇠Pd , Tk⇠T [ logC+ ( k|Tk ( x ) ) ] + Ex⇠Pg , Tk⇠T [ logC+ ( 0|Tk ( x ) ) ] ) , ( 5 ) min G V ( G , D ) g · ( Ex⇠Pg , Tk⇠T [ logC+ ( k|Tk ( x ) ) ] Ex⇠Pg , Tk⇠T [ logC+ ( 0|Tk ( x ) ) ] ) . ( 6 ) Theorem 2 . Given the optimal classifier C⇤ + ( k|x̃ ) = p T d ( x̃ ) pTg ( x̃ ) p Tk d ( x̃ ) PK k=1 p Tk d ( x̃ ) C⇤ + ( 0|x̃ ) of SSGAN-MS , at the equilibrium point , maximizing the self-supervised task for the generator is equivalent to 1 : min G DKL ( PTg kPTd ) 1 K KX k=1 '' E x̃⇠PTkg log pTkd ( x̃ ) PK k=1 p Tk d ( x̃ ) ! # , ( 7 ) where PTg , PTd represent the mixture distribution of transformed generated or real data x̃ 2 X̃ with density of pTg ( x̃ ) = PK k=1 p ( Tk ) p Tk g ( x̃ ) or p T d ( x̃ ) = PK k=1 p ( Tk ) p Tk d ( x̃ ) . 1Note that our Theorem 2 corrects the wrong version in the SSGAN-MS paper [ 57 ] , where the authors mistakenly regard p T d ( x̃ ) pTg ( x̃ ) = PK k=1 p ( Tk ) p Tk d ( x̃ ) PK k=1 p ( Tk ) p Tk g ( x̃ ) as p Tk d ( x̃ ) p Tk g ( x̃ ) in their proof . Please see Appendix A.2 for details . Our Theorem 2 states that the improved self-supervised task for the generator of SSGAN-MS , under the optimal classifier , retains the inconsistent goal in SSGAN that enforces the generator to produce typical rotation-detectable images rather than the entire real images . In addition , minimizing DKL ( PTg kPTd ) can not guarantee the generator to recover the real data distribution under the used data transformation setting ( see Theorem 4 ) . Due to the existence of these issues , SSGAN-MS also requires the original GAN task to approximate the real data distribution . We regard the learning paradigm of SSGAN-MS and SSGAN as the multi-task learning framework ( see Figure 1 ) .

Summary: In the context of using self-supervision signals in the GAN training framework, this paper proposes to use a multi-class classification approach in the discriminator paired with the task of predicting the rotation of an image.  This is proposed instead of using two separate discriminators: one for the rotation prediction and the other for the real/fake classification. The idea supposes a small and simple change to existing methods that, although in terms of novelty it is limited, it is justified theoretically and quantitative empirical results support the idea.