On Predicting Generalization using GANs

1 INTRODUCTION . Why do vastly overparametrized neural networks achieve impressive generalization performance across many domains , with very limited capacity control during training ? Despite some promising initial research , the mechanism behind generalization remains poorly understood . A host of papers have tried to adapt classical generalization theory to prove upper bounds of the following form on the difference between training and test error : test− train ≤ √ C |S| + tiny term ( 1 ) where S is the training dataset and C is a so-called complexity measure , typically involving some function of the training dataset as well as the trained net parameters ( e.g. , a geometric norm ) . Current upper bounds of this type are loose , and even vacuous . There is evidence that such classically derived bounds may be too loose ( Dziugaite & Roy , 2017 ) or that they may not correlate well with generalization ( Jiang et al. , 2019 ) . This has motivated a more principled empirical study of the effectiveness of generalization bounds . The general idea is to use machine learning to determine which network characteristics promote good generalization in practice and which do not —in other words , treat various deep net characteristics/norms/margins etc . as inputs to a machine learning model that uses them to predict the generalization error achieved by the net . This could help direct theorists to new types of complexity measures and motivate new theories . A recently started competition of Predicting Generalization in Deep Learning ( Jiang et al. , 2020 ) ( PGDL ) seeks to increase interest in such investigations , in an effort to uncover new and promising network characteristics and/or measures of network complexity that correlate with good generalization . As required in Jiang et al . ( 2020 ) , a complexity measure should depend on the trained model , optimizer , and training set , but not the held out test data . The first PGDL competition in 2020 did uncover quite a few measures that seemed to be predictive of good generalization but had not been identified by theory work . In this paper , we explore a very simple baseline for predicting generalization that had hitherto not received attention : train a Generative Adversarial Network ( GAN ) on the training dataset , and use performance on the synthetic data produced by the GAN to predict generalization . At first sight GANs may not appear to be an obvious choice for this task , due to their well known limitations . For instance , while the goal of GANs training is to find a generator that fools the discriminator net —in the sense that the discriminator has low probability of spotting a difference between GAN samples and the samples from the true distribution—in practice the discriminator is often able to discriminate very well at the end , demonstrating that it was not fooled . Also , GANs ’ generators are known to exhibit mode collapse i.e. , the generated distribution is a tiny subset of the true distribution . There is theory and experiments suggesting this may be difficult to avoid ( Arora et al. , 2018 ; Santurkar et al. , 2018 ; Bau et al. , 2019 ) . Given all these negative results about GANs , the surprising finding in the current paper is that GANs do allow for good estimates of test error ( and generalization error ) . This is verified for families of well-known GANs and datasets including primarily CIFAR-10/100 , Tiny ImageNet and wellknown deep neural classifier architectures . In particular , in sections 3.1 and 3.2 we evaluate on the PGDL and DEMOGEN benchmarks of predicting generalization and present strong results . In Section 4 and 5 , we also investigate reasons behind the surprising efficacy of GANs in predicting generalization as well as the effects of using data augmentation during GAN training . 2 RELATED WORK . Generalization Bounds Traditional approaches to predict generalization construct a generalization bound based on some notion of capacity such as parameter count , VC dimension , Rademacher complexity , etc . Neyshabur et al . ( 2018 ) provide a tighter generalization bound that decreases with increasing number of hidden units . Dziugaite & Roy ( 2017 ) reveal a way to compute nonvacuous PAC-Bayes generalization bounds . Recently , bounds based on knowledge distillation have also come to light ( Hsu et al. , 2020 ) . Despite progress in these approaches , a study conducted by Jiang et al . ( 2019 ) with extensive hyper-parameter search showed that current generalization bounds may not be effective , and the root cause of generalization remains elusive . Given the arduous nature of constructing such bounds , an interest in complexity measures has arisen . Predicting Generalization in Deep Learning The PGDL competition ( Jiang et al. , 2020 ) was held in NeuRIPS 2020 in an effort to encourage the discovery of empirical generalization measures following the seminal work of Jiang et al . ( 2018 ) . The winner of the PGDL competition Natekar & Sharma ( 2020 ) investigated properties of representations in intermediate layers to predict generalization . Kashyap et al . ( 2021 ) , the second place winner , experiment with robustness to flips , random erasing , random saturation , and other such natural augmentations . Afterwards , Jiang et al . ( 2021 ) interestingly find that generalization can be predicted by running SGD on the same architecture multiple times , and measuring the disagreement ratio between the different resulting networks on an unlabeled test set . There are also some pitfalls in predicting generalization , as highlighted by Dziugaite et al . ( 2020 ) . They find that distributional robustness over a family of environments is more applicable to neural networks than straight averaging . Limitations of GANs Arora et al . ( 2017 ) prove the lack of diversity enforcing in GAN ’ s training objective , and Arora et al . ( 2018 ) introduce the Birthday Paradox test to conclude that many popular GAN models in practice do learn a distribution with relatively small support . Santurkar et al . ( 2018 ) and Bau et al . ( 2019 ) investigate the extent to which GAN generated samples can match the distributional statistics of the original training data , and find that they have significant shortcomings . Ravuri & Vinyals ( 2019 ) find that GAN data is of limited use in training ResNet models , and find that neither inception score nor FID are predictive of generalization performance . Notably , despite the small support , Arora et al . ( 2018 ) reveal that GANs generate distinct images from their nearest neighbours in the training set . Later , Webster et al . ( 2019 ) use latent recovery to conclude more carefully that GANs do not memorize the training set . Theoretical Justification for Using GANs Arora et al . ( 2017 ) construct a generator that passes training against any bounded capacity discriminator but can only generate a small number of distinct data points either from the true data distribution or simply from the training set . For predicting generalization , it is crucial for the generator not to memorize training data . While Arora et al . ( 2017 ) do not answer why GANs do not memorize training data , a recent empirical study by Huang et al . ( 2021 ) demonstrates the difficulty of recovering input data by inverting gradients . Their work may cast light on how the generator could learn to generate data distinct from the training set when trained with gradient feedbacks from the discriminator . However , we are not aware of any theory that fully explains GANs ’ strength for predicting generalization despite limitations . 3 PREDICTING TEST PERFORMANCE USING GAN SAMPLES . We now precisely define what it means to predict test performance in our setting . We denote by Strain , Stest and Ssyn the training set , test set and the synthetic dataset generated by GANs . Given a classifier f trained on the training set Strain , we aim to predict its classification accuracy g ( f ) : = 1 |Stest| ∑ ( x , y ) ∈Stest 1 [ f ( x ) = y ] on a test set Stest . Our proposal is to train a conditional GAN model on the very same training set Strain , and then sample from the generator a synthetic dataset Ssyn of labeled examples . In the end , we simply use f ’ s accuracy on the synthetic dataset as our prediction for its test accuracy . Algorithm 1 formally describes this procedure . Algorithm 1 Predicting test performance Require : target classifier f , training set Strain , GAN training algorithm A 1 . Train a conditional GAN model using Strain : G , D = A ( Strain ) where G , D are the generator and discriminator networks . 2 . Generate a synthetic dataset by sampling from the generator G : Ssyn = { ( x̃1 , ỹ1 ) , . . . , ( x̃N , ỹN ) } where x̃i , ỹi = G ( zi , ỹi ) . The zi ’ s are drawn i.i.d . from G ’ s default input distribution . N and ỹi ’ are chosen so as to match statistics of the training set . Output : the synthetic accuracy ĝ ( f ) : = 1|Ssyn| ∑ ( x̃ , ỹ ) ∈Ssyn 1 [ f ( x̃ ) = ỹ ] as the prediction Remark . Any N ≥ |Strain| is a safe choice to ensure that ĝ ( f ) concentrates1around its mean Ez , ỹ [ 1 [ f ( G ( z , ỹ ) ) = ỹ ] ] and its deviation has negligible influence on the performance . 1the deviation is only O ( 1/ √ N ) by standard concentration bounds We demonstrate in Figure 1 that the test accuracy g ( f ) consistently resides in a small neighborhood of ĝ ( f ) for a diverse class of deep neural net classifiers trained on different datasets . For the choice of GAN architecture , we adopt the pre-trained BigGAN+DiffAug models ( Brock et al. , 2019 ; Zhao et al. , 2020 ) from the StudioGAN library2 . We evaluate on pretrained deep neural net classifiers with architectures ranging from VGG- ( 11 , 13 , 19 ) , ResNet- ( 18 , 34 , 50 ) , DenseNet ( 121 , 169 ) , ShuffleNet , PNASNet and MobileNet trained on CIFAR-10 and Tiny ImageNet with hyper-parameter setting ( learning rate , weight decay factor , learning rate schedule ) uniformly sampled from the grid { 10−1 , 10−2 } × { 5× 10−4 , 10−3 } × { CosineAnnealing , ExponentialDecay } . We use SGD with momentum 0.9 and batch size 128 and data augmention of horizontal flips for training all classifiers . 3.1 EVALUATION ON PGDL COMPETITION . We evaluate our proposed method on the tasks of NeurIPS 2020 Competition on Predicting Generalization of Deep Learning . The PGDL competition consists of 8 tasks3 , each containing a set of pre-trained deep net classifiers of similar architectures but with different hyper-parameter settings , as well as the training data . The tasks of PGDL are based on a wide range of datasets inlcuding CIFAR-10 , SVHN , CINIC-10 , Oxford Flowers , Oxford Pets and Fashion MNIST . For every task , the goal is to compute a scalar prediction for each classifier based on its parameters and the training data that correlates as much as possible with the actual generalization errors of the classifiers measured on a test set . The correlation score is measured by the so-called Conditional Mutual Information , which is designed to indicate whether the computed predictions contain all the information about the generalization errors such that knowing the hyper-parameters does not provide additional information , see ( Jiang et al. , 2020 ) for details . Since the goal of PGDL is to predict generalization gap instead of test accuracy , our proposed solution is a slight adaptation of Algorithm 1 . For each task , we first train a GAN model on the provided training dataset and sample from it a labeled synthetic dataset . Then for each classifier within the task , instead of the raw synthetic accuracy , we use as prediction the gap between training and synthetic accuracies . For all tasks , we use BigGAN+diffAug with the default4 implementation for CIFAR-10 from the StudioGAN library . We found that a subset of them ( Task 6 with Oxford Flowers and Task 7 with Oxford Pets ) were not well-suited for standard GAN training without much tweaking due to a small training set and highly uneven class sizes . We focused only on the subset of the tasks where GAN training worked out of the box , specifically Task 1 , 2 , 4 , 5 , 8 , and 9 . 2available at https : //github.com/POSTECH-CVLab/PyTorch-StudioGAN 3namely Task 1 , 2 , 4 , 5 , 6 , 7 , 8 , 9 , without 3 . 4Task 8 uses the single-channel 28×28 Fashion-MNIST dataset . For GAN training , we symmetrically zeropad the training images to 32 × 32 and convert to RGB by replicating the channel . To compute the synthetic accuracy , we convert the generated images back to single-channel and perform a center cropping . We report the results in Table 1 , where we observe that on tasks involving either CIFAR-10 or VGGlike classifiers , our proposed solution out-performs all of the solutions from the top 3 teams of the competition by a large margin . One potential reason may be that the default hyper-parameters of the BigGAN model have been engineered towards CIFAR-10 like datasets and VGG/ResNet-like discriminator networks . It is worth mentioning that we conducted absolutely zero hyper-parameter search for all tasks here . Especially for CIFAR-10 tasks , we directly take the pre-trained models from the StudioGAN library . It is likely that we may achieve even better scores with fine-tuned GAN hyper-parameters and optimized training procedure for each PGDL task , and we leave it to future work .

Summary: This paper investigates the task of using GAN’s samples as a substitute for the true test distribution for estimating the generalization. Experiments show that samples from GAN are a powerful substitute for the test set and the predictive power of the synthetic accuracy outperforms all prior methods include the winner of the 2020 PGDL competition, despite all the empirical and theoretical evidence that GANs are not good at approximating the true data distribution. The paper then performs a preliminary investigation on why samples from GANs are a good substitute for the test set and found that using a certain notion of perceptual similarity, samples from GAN’s are much closer to the test distribution than the training distribution despite being trained on the training data.