Deceive D: Adaptive Pseudo Augmentation for GAN Training with Limited Data

1 Introduction . While state-of-the-art GANs like StyleGAN2 [ 20 ] are constantly pushing forward the fidelity and resolution of synthesized images , they usually require a large amount of training data to fully unleash their power . Training GANs with insufficient data tend to generate poor-quality images , as shown in Figure 1 . Collecting sufficient data samples for these GANs is sometimes infeasible , especially in domains where data are sparse and privacy-sensitive . To ease the practical deployment of powerful GANs , it is necessary to devise new strategies for training GANs with limited data while preserving the quality of synthesis . Recent studies have shown that the overfitting of the discriminator is the critical reason that impedes effective GAN training on limited data [ 2 , 18 , 50 , 42 ] , rendering severe instability of training dynamics . Specifically , when the discriminator starts to overfit , the distributions of its outputs for real and generated samples gradually diverge from each other [ 18 , 42 ] , and its feedback to the generator becomes less informative . Consequently , the generator converges to an inferior point , compromising the quality of synthesized images . Recent solutions to this problem include the use of standard data augmentations , either conventional or differentiable , to real and generated images [ 18 , 50 , 52 , 41 ] or applying an additional model regularization term [ 42 ] . Addressing the discriminator overfitting is still an open problem . We are interested in finding an alternative way to the aforementioned approaches . In this paper , we present a simple yet effective way to regularize a discriminator without introducing any external augmentations or regularization terms . We call our method Adaptive Pseudo Augmentation ( APA ) . In contrast to previous standard data augmentations [ 18 , 50 , 52 , 41 ] , we exploit the ∗Corresponding author . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . generator in a GAN itself to provide the augmentation , a more natural way to regularize the overfitting of the discriminator . Compared to the model regularization , our approach is more adaptive to fit different settings and training status without manual tuning . Specifically , APA takes the fake/pseudo samples synthesized by the generator and moderately feeds them into the limited real data . Such pseudo data are adaptively presented to the discriminator as “ real ” instances . The goal of this pseudo augmentation for the real data is not to enlarge the real dataset but to suppress the discriminator ’ s confidence in distinguishing real and fake distributions . The deceit is introduced adaptively , which is moderated by a deception probability according to the degree of overfitting . To quantify overfitting , we study a series of plausible heuristics derived from the discriminator raw output logits . The main contribution of this work is a novel adaptive pseudo augmentation method for training GANs with limited data . This approach deceives the discriminator adaptively and mitigates the problem of discriminator overfitting . The proposed APA can be readily added to existing GAN training with negligible computational cost . We conduct extensive experiments to demonstrate the effectiveness of APA for state-of-the-art GAN training with limited data . The results are comparable or even better than other types of solutions [ 18 , 42 ] . APA is also complementary to existing methods based on standard data augmentations for gaining a further performance boost . Besides , we theoretically connect APA with minimizing the JS divergence [ 23 ] between the smoothed data distribution and generated distribution , proving its convergence and rationality . We hope that our approach could extend the breadth and potential of solutions to GAN training with limited data . 2 Related Work . Generative adversarial networks . Generative adversarial networks ( GANs ) [ 9 , 28 , 35 , 47 ] adopt an adversarial training scheme , where a generator keeps refining its capability in synthesizing images to compete with a discriminator ( i.e. , a binary classifier ) until the discriminator fails to classify the generated samples as fakes . GANs are known to suffer from training instability [ 9 , 38 , 2 , 27 ] . Various approaches have been proposed to stabilize the training and improve the quality of synthesis by minimizing different f -divergences of the real and fake distributions [ 30 ] . The saturated form of vanilla GAN [ 9 ] is theoretically proven to minimize the JS divergence [ 23 ] between the two distributions . LSGAN [ 26 ] and EBGAN [ 49 ] correspond to the optimizations of χ2-divergence [ 34 ] and the total variation [ 3 ] , respectively . On another note , WGAN [ 3 ] is designed for minimizing the Wasserstein distance . State-of-the-art methods , such as PGGAN [ 17 ] , BigGAN [ 5 ] , StyleGAN [ 19 ] , and StyleGAN2 [ 20 ] , employ large-scale training with contemporary techniques , achieving photorealistic results . These methods have been extended to various tasks , including face generation [ 17 , 19 , 20 ] , image editing [ 1 , 7 , 33 ] , semantic image synthesis [ 44 , 32 , 25 ] , image-to-image translation [ 13 , 53 , 8 , 16 , 15 , 36 ] , style transfer [ 24 , 12 , 22 ] , and GAN inversion [ 31 , 36 , 46 ] . Despite the remarkable success , the performance of GANs relies heavily on the amount of training data . Training GANs with limited data . The significance and difficulty of training GANs with limited data have been attracting attention from many researchers recently . The issue of data insufficiency tends to cause overfitting in the discriminator [ 45 , 18 , 50 ] , which in turn deteriorates the stability of training dynamics in GANs , compromising the quality of generated images . Many recent studies [ 48 , 51 , 41 , 52 , 50 , 18 ] propose to apply standard data augmentations for GAN training to enrich the diversity of the dataset to mitigate the overfitting of the discriminator . For instance , DiffAugment ( DA ) [ 50 ] adopts the same differentiable augmentation to both real and fake images for the generator and the discriminator without manipulating the target distribution . Adaptive discriminator augmentation ( ADA ) [ 18 ] shares a similar idea with DA , while it further devises an adaptive approach that controls the strength of data augmentations adaptively . In this work , we extend the study of such an adaptive approach . Another type of solution is model regularization . Previous efforts on regularizing GANs include adding noise to the inputs of the discriminator [ 2 , 40 , 14 ] , gradient penalty [ 27 , 10 , 37 ] , one-sided label smoothing [ 38 ] , spectral normalization [ 29 ] , label noise [ 6 ] , etc . These methods are designed for stabilizing training or preventing mode collapse [ 38 ] . The essence of their goals could be considered similar to our method since training GANs in the low-data regime exhibits similar behaviors as previously observed in early GANs with sufficient data . Additional discussions on these techniques are provided in the supplementary material . Under the limited data setting , a very recent study proposes an LC-regularization term [ 42 ] to regulate the discriminator predictions using two exponential moving average variables that track the discriminator outputs throughout training . Our work explores an alternative solution from a different perspective , which is also complementary to previous approaches based on standard data augmentations . 3 Methodology . In GAN ’ s adversarial training , the goal of the generator G is to deceive the discriminator D and maximize the probability that D makes a wrong judgment . Therefore , G keeps refining its generated samples to better deceive D over time . When the training only accesses a limited amount of data , one would observe that D turns out to be overly confident and hardly makes any mistake , causing its feedback to G to become meaningless . In Figure 2 , we show the training “ snapshots ” of two StyleGAN2 [ 20 ] models on the FFHQ dataset [ 19 ] . The settings of the two models differ only by the amount of data available to them for training . As can be observed , both training processes start smoothly , and distributions of discriminator outputs for the real and generated images overlap at the early stage . As the training progresses , the discriminator , which only has access to limited data ( 7k images , 10 % of full data ) , experiences diverged predictions much more rapidly , and the average sign boundary turns out to be more apparent . This divergence in prediction shows that D becomes increasingly confident in classifying real and fake samples . At the late stage of training , D can even judge all the input samples correctly with high confidence . Meanwhile , the evaluation FID [ 11 ] scores ( lower is better ) deteriorate , consistent with the divergence of D ’ s predictions . The phenomena above demonstrate how a discriminator gets overfitted quickly with limited data . As can be seen from the FID curves , the overfitting directly influences training dynamics and the convergence of G , leading to poor generation performance . 3.1 Adaptive Pseudo Augmentation . The generator itself naturally possesses the capability to counteract discriminator overfitting . To harness this capability , our method employs a GAN to augment itself using the generated samples ( see Figure 3 ) . Specifically , we feed the samples generated by G into the limited real data to form a pseudo-real set . The fake images in this set will be adaptively presented to D , pretending themselves as real data . The goal here is to deceive D with the pseudo-real set and consequently suppressing its confidence in distinguishing real and fake images . Blindly presenting the fake images as reals to D may weaken the fundamental ability of D in adversarial training . In our approach , the deceit is introduced adaptively to avoid any potential adverse effects . To moderate the deception , we perform pseudo augmentation based on a probability , p ∈ [ 0 , 1 ) , that quantifies the deception strength . Specifically , the pseudo augmentation will be applied with the probability p or be skipped with the probability 1− p. We note that the overfitting state of D is dynamic throughout the training ( see Figure 2 ) . It is intuitive to let the deception probability p be adjusted adaptively based on the degree of overfitting . Ideally , p should be adjusted without manual tuning regardless of data scales and properties . To achieve this goal , inspired by ADA [ 18 ] , we apply an overfitting heuristic λ that quantifies the degree of D ’ s overfitting . We extend the control scheme of ADA and provide three plausible variants : λr = E ( sign ( Dreal ) ) , λf = −E ( sign ( Dfake ) ) , λrf = E ( sign ( Dreal ) ) − E ( sign ( Dfake ) ) 2 , ( 1 ) where Dreal and Dfake are defined as Dreal = logit ( D ( x ) ) , Dfake = logit ( D ( G ( z ) ) ) , ( 2 ) where logit denotes the logit function . As shown in Figure 2 , the λr in Eq . ( 1 ) estimates the portion of real images that obtain positive logit predictions by D , and that of generated images is captured by λf . Besides , λrf indicates half of the distance between the signs of the real and fake logits . For all these heuristics , λ = 0 represents no overfitting , and λ = 1 means complete overfitting . We use λr in our main experiments and study other variants in the ablation study . The strategy of using λ to adjust p is as follows . We set a threshold t ( in most cases of our experiments , t = 0.6 ) and initialize p to be zero . If λ signifies too much/little overfitting regarding t ( i.e. , larger/smaller than t ) , the probability p will be increased/decreased by one fixed step . Using this step size , p can increase from zero to one in 500k images shown to D. We adjust p once every four iterations and clamp p from below to zero after each adjustment . In this way , the strength of pseudo augmentation can be adaptively controlled based on the degree of overfitting ( see Figure 3 ) .

Summary: This paper tries to solve discriminator overfitting problem.  The authors propose adaptive pseudo augmentation (APA). * APA employs the generator itself to augment the real data distribution with fake images.   * i.e., Fake images are presented as "real" instances to the discriminator. * Adaptiveness comes from measuring overfittingness of the discriminator. * APA has similar theoretical properties to the original GAN, but with $(1-\alpha)p_\text{data} + \alpha p_g = p_g$.  APA improves systnesis quality in the limited-data regime. * on FFHQ, AFHQ-Cat, CUB, Danbooru The authors provide a theoretical analysis of APA, similar to the original GAN.