Self-Diagnosing GAN: Diagnosing Underrepresented Samples in Generative Adversarial Networks

Despite remarkable performance in producing realistic samples , Generative Adversarial Networks ( GANs ) often produce low-quality samples near low-density regions of the data manifold , e.g. , samples of minor groups . Many techniques have been developed to improve the quality of generated samples , either by postprocessing generated samples or by pre-processing the empirical data distribution , but at the cost of reduced diversity . To promote diversity in sample generation without degrading the overall quality , we propose a simple yet effective method to diagnose and emphasize underrepresented samples during training of a GAN . The main idea is to use the statistics of the discrepancy between the data distribution and the model distribution at each data instance . Based on the observation that the underrepresented samples have a high average discrepancy or high variability in discrepancy , we propose a method to emphasize those samples during training of a GAN . Our experimental results demonstrate that the proposed method improves GAN performance on various datasets , and it is especially effective in improving the quality and diversity of sample generation for minor groups . 1 Introduction . Generative Adversarial Networks ( GANs ) have achieved remarkable performance in producing realistic samples for complex generation tasks , including image/video synthesis [ 5 , 22 ] , style transfer [ 45 , 14 ] , and data augmentation [ 29 ] . However , GANs often fail to cover sparse regions of data manifold [ 16 , 9 ] , leading to the underrepresentation of minor groups in the dataset [ 43 ] . In particular , GANs generate samples of minor groups with low fidelity or even fail to generate such samples , exhibiting the mode collapse [ 43 ] . Many of previous techniques have focused on improving the overall sample quality of GANs , either by pre-processing the training dataset or by post-processing generated samples . The pre-processing aims to remove instances that can not be well-represented by GANs even before the training starts and gains fidelity on the focused samples [ 9 ] . A similar idea has been used to truncate the latent space by resampling or moving samples that fall outside of some acceptable range during training [ 16 , 6 ] . ∗Equal contribution . †This work was done as a student at KAIST . ‡Corresponding author . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Post-processing , on the other hand , is a technique that can be applied after the training to remove low-quality generated samples by rejection sampling [ 3 , 37 ] . All these approaches are effective in increasing the overall fidelity of samples from GANs , but reducing the diversity as a trade-off , and may exacerbate biases against the minor groups in sample generation . In this work , we aim to improve diversity in sample generation without degrading the overall quality , with a special focus on coverage and quality improvement for minor groups . Toward this , we design methods to detect and emphasize underrepresented samples in training of GANs . Due to the lack of explicit labels available , detecting minor-subgroup samples is especially challenging for unsupervised learning . Therefore , we first develop two new metrics , which can be easily calculated from a discriminator output of GANs , to detect underrepresented samples . The main idea is to measure the statistics ( mean and variance ) of the estimated discrepancy between the data distribution and model distribution at each data instance over multiple epochs of the training . The mean discrepancy indicates how close the data distribution is to the model distribution at each data over the training , while the variance in discrepancy measures how such discrepancy fluctuates across the training . We provide theoretical and empirical evidence that the mean discrepancy can effectively detect underrepresented samples , especially near collapsed modes , while the variance in discrepancy can detect minor data instances , which GANs suffer from modeling . Based on these observations , we propose a novel method to emphasize underrepresented samples during the training of GANs by score-based weighted sampling , where the score is defined as a weighted sum of the two metrics we devised . We validate our method with thorough experiments over controlled and real datasets and demonstrate the efficacy of the proposed sampling method in improving not only the overall quality ( both fidelity and diversity combined ) of sample generation but also the coverage and quality for semantic features of minor subgroups . Our contributions can be summarized as follows . • We propose two new metrics , which can be simply computed from the discriminator , to diagnose GAN training and to detect underrepresented samples . By theoretical analysis and controlled experiments , we demonstrate that the proposed metrics are effective in detecting underrepresented minor samples . • We propose an algorithm that can effectively emphasize underrepresented data by scorebased weighted sampling during the training of GANs . Our experiments on controlled and real datasets show that our method improves diverse performance metrics on several GAN variants and enhances the coverage and quality of minor group generation . Our code is publicly available at https : //github.com/grayhong/self-diagnosing-gan . 2 Related Work . Promoting data coverage in GANs Due to the unstable nature of the min-max game between a generator and a discriminator , GANs often suffer from mode collapse and produce samples with poor diversity . Several approaches have been proposed to promote better data coverage by modifying architectures [ 20 , 23 ] , loss functions [ 2 , 1 ] or adding regularizations [ 8 , 4 , 35 ] . While effective in promoting overall data coverage , these approaches do not provide special care on minor modes and often fail to recover them when the minority ratio for certain feature is extremely low . We provide a method to promote data coverage for minor features even when the minority ratio is significantly low . There exists another line of works to improve data coverage by designing hybrid generative models [ 32 , 27 , 43 ] , which combines the idea of reconstructive models ( e.g . variational autoencoder ) to GANs , to take advantages of the reconstructive models in recovering diverse modes . This hybrid method , however , requires relatively high computational overhead to guarantee data coverage for all ( or partial ) real modes by optimizing reconstruction error in feature domain . Our method directly detects and emphasizes underrepresented samples so that the computational overhead is much lower . Improving GAN performance by diagnosing samples There have been promising attempts to improve GAN training by using the discriminator outputs to estimate the discrepancy between the data distribution and implicit model distribution . DRS [ 3 ] proposes the density ratio estimate based on the discriminator output to apply rejection sampling to filter generated samples . GOLD [ 26 ] uses the similar estimate to re-weight fake samples to emphasize underrepresented fake samples . In [ 10 ] and [ 12 ] , on the other hand , an external classifier is used to improve the density ratio estimates . There also exist some approaches to use discriminator outputs to select or weight “ useful ” fake samples during training . Top-k training [ 30 ] updates the generator by using only top-k fake samples with the largest discriminator outputs . In [ 31 ] and [ 40 ] , discriminator-based importance re-weighting schemes for fake samples are developed , and in [ 39 ] , latent samples are optimized to improve the fidelity . Our method uses the discrepancy estimate proposed in [ 3 ] , but its empirical mean and variance over multiple epochs , to extract more reliable and useful information to detect underrepresented minor group samples . We provide theoretical evidence of why not only the mean but also the variance of discrepancy estimate is effective in detecting underrepresented samples . Our method detects and emphasizes underrepresented real samples , not the fake samples . This difference is significant in promoting the data coverage of minor groups , since when fake samples already fail to cover minor modes , emphasizing a subset of fake samples can not improve the data coverage for missed modes . 3 Two Metrics to Detect Underrepresented Samples During GAN Training . 3.1 Measuring the discrepancy of GANs . GAN training aims to train a generator with an implicit model distribution pg ( x ) that closely matches the data distribution pdata ( x ) . The discrepancy between pdata ( x ) and pg ( x ) can be measured by the log density ratio log ( pdata ( x ) /pg ( x ) ) , but it can not be directly calculated in GANs , since pdata ( x ) is unknown and pg ( x ) is implicit . Instead , the analysis in the original GAN paper [ 11 ] can be used to define an estimate on the density ratio by using the discriminator output as explained in [ 3 ] . The original GAN solves the min-max optimization minGmaxD V ( D , G ) for the loss V ( D , G ) = Ex∼pdata [ logD ( x ) ] +Ez∼pz [ log ( 1−D ( G ( z ) ) ) ] . For any fixed generatorG , the optimal discriminator yields D∗ ( x ) = pdata ( x ) pdata ( x ) +pg ( x ) and this allows us to define the Log-Density-Ratio estimate ( LDR ) by LDR ( x ) : = log ( D ( x ) / ( 1−D ( x ) ) ) . ( 1 ) When D ( x ) = D∗ ( x ) , the LDR ( x ) is equal to the log density ratio log ( pdata ( x ) /pg ( x ) ) . When LDR ( x ) > 0 , the data point x is underrepresented in the model , i.e. , pdata ( x ) > pg ( x ) , while when LDR ( x ) < 0 , the data is overrepresented , i.e. , pdata ( x ) < pg ( x ) . Thus , we can leverage the value of LDR ( x ) of each instance x to give feedback to improve the generator if the estimation is valid . Some prior works have used the LDR estimate to improve GAN training . As an example , GOLD [ 26 ] uses LDR ( x ) to evaluate the quality of the fake samples and re-weights the underrepresented fake samples when training the generator for conditional GANs . However , we later show that reweighting fake samples is less effective than re-weighting real samples in improving diversity in sample generation . We also empirically show that LDR ( x ) is an unstable metric to use . More detailed arguments are available in the Appendix §A . As a remedy , we propose to use statistics of LDR ( x ) , which are much more stable and informative metrics , to detect underrepresented data regions during the training . The main intuition is to use training dynamics–the behavior of a model as training progresses–to diagnose the learning behavior of each sample . In supervised learning , training dynamics have been widely studied to detect “ hard-tolearn ” samples [ 7 , 33 , 38 ] . However , in learning generative models , the metrics to diagnose training dynamics are not clear since there is no explicit reference to measure the accuracy of the model . Here we define metrics that estimate the mean and variance of the discrepancy of GANs , LDRM ( LDR Mean ) and LDRV ( LDR Variance ) , at each sample x across the training steps T = { ts , ... , te } : LDRM ( x ; T ) = 1 |T | ∑ k∈T LDR ( x ) k , LDRV ( x ; T ) = 1 |T | − 1 ∑ k∈T [ LDR ( x ) k − LDRM ( x ; T ) ] 2 , ( 2 ) where LDR ( x ) k is the recorded LDR estimate ( 1 ) in the k-th training step . LDRM ( x ) measures how close pdata ( x ) is to pg ( x ) over the training at sample point x , while LDRV ( x ) measures how such discrepancy fluctuates across training . Intuitively , samples that have been well-learned and generalized will have consistently small LDR ( x ) since D ( x ) ≈ 1/2 ( i.e. , pdata ( x ) ≈ pg ( x ) ) , thus will exhibit low LDRM and LDRV , while underrepresented “ hard-to-learn ” samples will show high LDRM or LDRV values . In the rest of this section , we thoroughly study the characteristics of data instances with high LDRM or high LDRV .

Summary: The paper proposes a scoring to measure to what extend an image in a dataset is characterized by an under-represented visual appearance. GANs fail to accurately model these under-represented modes of the data distribution or drop them completely. Here, the score is based on the discriminator predictions on real images, and is used as a basis for score-weighted sampling of real data during training, to show under-represented real images more often. The method is shown to improve mode coverage and to result in better image quality/diversity than the considered baselines.