Deceive D: Adaptive Pseudo Augmentation for GAN Training with Limited Data

1 Introduction . While state-of-the-art GANs like StyleGAN2 [ 20 ] are constantly pushing forward the fidelity and resolution of synthesized images , they usually require a large amount of training data to fully unleash their power . Training GANs with insufficient data tend to generate poor-quality images , as shown in Figure 1 . Collecting sufficient data samples for these GANs is sometimes infeasible , especially in domains where data are sparse and privacy-sensitive . To ease the practical deployment of powerful GANs , it is necessary to devise new strategies for training GANs with limited data while preserving the quality of synthesis . Recent studies have shown that the overfitting of the discriminator is the critical reason that impedes effective GAN training on limited data [ 2 , 18 , 50 , 42 ] , rendering severe instability of training dynamics . Specifically , when the discriminator starts to overfit , the distributions of its outputs for real and generated samples gradually diverge from each other [ 18 , 42 ] , and its feedback to the generator becomes less informative . Consequently , the generator converges to an inferior point , compromising the quality of synthesized images . Recent solutions to this problem include the use of standard data augmentations , either conventional or differentiable , to real and generated images [ 18 , 50 , 52 , 41 ] or applying an additional model regularization term [ 42 ] . Addressing the discriminator overfitting is still an open problem . We are interested in finding an alternative way to the aforementioned approaches . In this paper , we present a simple yet effective way to regularize a discriminator without introducing any external augmentations or regularization terms . We call our method Adaptive Pseudo Augmentation ( APA ) . In contrast to previous standard data augmentations [ 18 , 50 , 52 , 41 ] , we exploit the ∗Corresponding author . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . generator in a GAN itself to provide the augmentation , a more natural way to regularize the overfitting of the discriminator . Compared to the model regularization , our approach is more adaptive to fit different settings and training status without manual tuning . Specifically , APA takes the fake/pseudo samples synthesized by the generator and moderately feeds them into the limited real data . Such pseudo data are adaptively presented to the discriminator as “ real ” instances . The goal of this pseudo augmentation for the real data is not to enlarge the real dataset but to suppress the discriminator ’ s confidence in distinguishing real and fake distributions . The deceit is introduced adaptively , which is moderated by a deception probability according to the degree of overfitting . To quantify overfitting , we study a series of plausible heuristics derived from the discriminator raw output logits . The main contribution of this work is a novel adaptive pseudo augmentation method for training GANs with limited data . This approach deceives the discriminator adaptively and mitigates the problem of discriminator overfitting . The proposed APA can be readily added to existing GAN training with negligible computational cost . We conduct extensive experiments to demonstrate the effectiveness of APA for state-of-the-art GAN training with limited data . The results are comparable or even better than other types of solutions [ 18 , 42 ] . APA is also complementary to existing methods based on standard data augmentations for gaining a further performance boost . Besides , we theoretically connect APA with minimizing the JS divergence [ 23 ] between the smoothed data distribution and generated distribution , proving its convergence and rationality . We hope that our approach could extend the breadth and potential of solutions to GAN training with limited data . 2 Related Work . Generative adversarial networks . Generative adversarial networks ( GANs ) [ 9 , 28 , 35 , 47 ] adopt an adversarial training scheme , where a generator keeps refining its capability in synthesizing images to compete with a discriminator ( i.e. , a binary classifier ) until the discriminator fails to classify the generated samples as fakes . GANs are known to suffer from training instability [ 9 , 38 , 2 , 27 ] . Various approaches have been proposed to stabilize the training and improve the quality of synthesis by minimizing different f -divergences of the real and fake distributions [ 30 ] . The saturated form of vanilla GAN [ 9 ] is theoretically proven to minimize the JS divergence [ 23 ] between the two distributions . LSGAN [ 26 ] and EBGAN [ 49 ] correspond to the optimizations of χ2-divergence [ 34 ] and the total variation [ 3 ] , respectively . On another note , WGAN [ 3 ] is designed for minimizing the Wasserstein distance . State-of-the-art methods , such as PGGAN [ 17 ] , BigGAN [ 5 ] , StyleGAN [ 19 ] , and StyleGAN2 [ 20 ] , employ large-scale training with contemporary techniques , achieving photorealistic results . These methods have been extended to various tasks , including face generation [ 17 , 19 , 20 ] , image editing [ 1 , 7 , 33 ] , semantic image synthesis [ 44 , 32 , 25 ] , image-to-image translation [ 13 , 53 , 8 , 16 , 15 , 36 ] , style transfer [ 24 , 12 , 22 ] , and GAN inversion [ 31 , 36 , 46 ] . Despite the remarkable success , the performance of GANs relies heavily on the amount of training data . Training GANs with limited data . The significance and difficulty of training GANs with limited data have been attracting attention from many researchers recently . The issue of data insufficiency tends to cause overfitting in the discriminator [ 45 , 18 , 50 ] , which in turn deteriorates the stability of training dynamics in GANs , compromising the quality of generated images . Many recent studies [ 48 , 51 , 41 , 52 , 50 , 18 ] propose to apply standard data augmentations for GAN training to enrich the diversity of the dataset to mitigate the overfitting of the discriminator . For instance , DiffAugment ( DA ) [ 50 ] adopts the same differentiable augmentation to both real and fake images for the generator and the discriminator without manipulating the target distribution . Adaptive discriminator augmentation ( ADA ) [ 18 ] shares a similar idea with DA , while it further devises an adaptive approach that controls the strength of data augmentations adaptively . In this work , we extend the study of such an adaptive approach . Another type of solution is model regularization . Previous efforts on regularizing GANs include adding noise to the inputs of the discriminator [ 2 , 40 , 14 ] , gradient penalty [ 27 , 10 , 37 ] , one-sided label smoothing [ 38 ] , spectral normalization [ 29 ] , label noise [ 6 ] , etc . These methods are designed for stabilizing training or preventing mode collapse [ 38 ] . The essence of their goals could be considered similar to our method since training GANs in the low-data regime exhibits similar behaviors as previously observed in early GANs with sufficient data . Additional discussions on these techniques are provided in the supplementary material . Under the limited data setting , a very recent study proposes an LC-regularization term [ 42 ] to regulate the discriminator predictions using two exponential moving average variables that track the discriminator outputs throughout training . Our work explores an alternative solution from a different perspective , which is also complementary to previous approaches based on standard data augmentations . 3 Methodology . In GAN ’ s adversarial training , the goal of the generator G is to deceive the discriminator D and maximize the probability that D makes a wrong judgment . Therefore , G keeps refining its generated samples to better deceive D over time . When the training only accesses a limited amount of data , one would observe that D turns out to be overly confident and hardly makes any mistake , causing its feedback to G to become meaningless . In Figure 2 , we show the training “ snapshots ” of two StyleGAN2 [ 20 ] models on the FFHQ dataset [ 19 ] . The settings of the two models differ only by the amount of data available to them for training . As can be observed , both training processes start smoothly , and distributions of discriminator outputs for the real and generated images overlap at the early stage . As the training progresses , the discriminator , which only has access to limited data ( 7k images , 10 % of full data ) , experiences diverged predictions much more rapidly , and the average sign boundary turns out to be more apparent . This divergence in prediction shows that D becomes increasingly confident in classifying real and fake samples . At the late stage of training , D can even judge all the input samples correctly with high confidence . Meanwhile , the evaluation FID [ 11 ] scores ( lower is better ) deteriorate , consistent with the divergence of D ’ s predictions . The phenomena above demonstrate how a discriminator gets overfitted quickly with limited data . As can be seen from the FID curves , the overfitting directly influences training dynamics and the convergence of G , leading to poor generation performance . 3.1 Adaptive Pseudo Augmentation . The generator itself naturally possesses the capability to counteract discriminator overfitting . To harness this capability , our method employs a GAN to augment itself using the generated samples ( see Figure 3 ) . Specifically , we feed the samples generated by G into the limited real data to form a pseudo-real set . The fake images in this set will be adaptively presented to D , pretending themselves as real data . The goal here is to deceive D with the pseudo-real set and consequently suppressing its confidence in distinguishing real and fake images . Blindly presenting the fake images as reals to D may weaken the fundamental ability of D in adversarial training . In our approach , the deceit is introduced adaptively to avoid any potential adverse effects . To moderate the deception , we perform pseudo augmentation based on a probability , p ∈ [ 0 , 1 ) , that quantifies the deception strength . Specifically , the pseudo augmentation will be applied with the probability p or be skipped with the probability 1− p. We note that the overfitting state of D is dynamic throughout the training ( see Figure 2 ) . It is intuitive to let the deception probability p be adjusted adaptively based on the degree of overfitting . Ideally , p should be adjusted without manual tuning regardless of data scales and properties . To achieve this goal , inspired by ADA [ 18 ] , we apply an overfitting heuristic λ that quantifies the degree of D ’ s overfitting . We extend the control scheme of ADA and provide three plausible variants : λr = E ( sign ( Dreal ) ) , λf = −E ( sign ( Dfake ) ) , λrf = E ( sign ( Dreal ) ) − E ( sign ( Dfake ) ) 2 , ( 1 ) where Dreal and Dfake are defined as Dreal = logit ( D ( x ) ) , Dfake = logit ( D ( G ( z ) ) ) , ( 2 ) where logit denotes the logit function . As shown in Figure 2 , the λr in Eq . ( 1 ) estimates the portion of real images that obtain positive logit predictions by D , and that of generated images is captured by λf . Besides , λrf indicates half of the distance between the signs of the real and fake logits . For all these heuristics , λ = 0 represents no overfitting , and λ = 1 means complete overfitting . We use λr in our main experiments and study other variants in the ablation study . The strategy of using λ to adjust p is as follows . We set a threshold t ( in most cases of our experiments , t = 0.6 ) and initialize p to be zero . If λ signifies too much/little overfitting regarding t ( i.e. , larger/smaller than t ) , the probability p will be increased/decreased by one fixed step . Using this step size , p can increase from zero to one in 500k images shown to D. We adjust p once every four iterations and clamp p from below to zero after each adjustment . In this way , the strength of pseudo augmentation can be adaptively controlled based on the degree of overfitting ( see Figure 3 ) .

Summary: The submission tackles the problem of training GANs with limited data. It proposes to do so by adaptively regularizing GAN discriminators by replacing some real data by generated samples when the discriminator overfits, accoding to an overfitting heuristic.  A study of the proposed objectives show that the method keep the classical GAN optima while alleviating overfitting of the discriminator.  The method is tested on various datasets with limited data, and compared, using a StyleGAN-2 backbone, against other state-of-the-art methods for training with limited data.