On Predicting Generalization using GANs

1 INTRODUCTION . Why do vastly overparametrized neural networks achieve impressive generalization performance across many domains , with very limited capacity control during training ? Despite some promising initial research , the mechanism behind generalization remains poorly understood . A host of papers have tried to adapt classical generalization theory to prove upper bounds of the following form on the difference between training and test error : test− train ≤ √ C |S| + tiny term ( 1 ) where S is the training dataset and C is a so-called complexity measure , typically involving some function of the training dataset as well as the trained net parameters ( e.g. , a geometric norm ) . Current upper bounds of this type are loose , and even vacuous . There is evidence that such classically derived bounds may be too loose ( Dziugaite & Roy , 2017 ) or that they may not correlate well with generalization ( Jiang et al. , 2019 ) . This has motivated a more principled empirical study of the effectiveness of generalization bounds . The general idea is to use machine learning to determine which network characteristics promote good generalization in practice and which do not —in other words , treat various deep net characteristics/norms/margins etc . as inputs to a machine learning model that uses them to predict the generalization error achieved by the net . This could help direct theorists to new types of complexity measures and motivate new theories . A recently started competition of Predicting Generalization in Deep Learning ( Jiang et al. , 2020 ) ( PGDL ) seeks to increase interest in such investigations , in an effort to uncover new and promising network characteristics and/or measures of network complexity that correlate with good generalization . As required in Jiang et al . ( 2020 ) , a complexity measure should depend on the trained model , optimizer , and training set , but not the held out test data . The first PGDL competition in 2020 did uncover quite a few measures that seemed to be predictive of good generalization but had not been identified by theory work . In this paper , we explore a very simple baseline for predicting generalization that had hitherto not received attention : train a Generative Adversarial Network ( GAN ) on the training dataset , and use performance on the synthetic data produced by the GAN to predict generalization . At first sight GANs may not appear to be an obvious choice for this task , due to their well known limitations . For instance , while the goal of GANs training is to find a generator that fools the discriminator net —in the sense that the discriminator has low probability of spotting a difference between GAN samples and the samples from the true distribution—in practice the discriminator is often able to discriminate very well at the end , demonstrating that it was not fooled . Also , GANs ’ generators are known to exhibit mode collapse i.e. , the generated distribution is a tiny subset of the true distribution . There is theory and experiments suggesting this may be difficult to avoid ( Arora et al. , 2018 ; Santurkar et al. , 2018 ; Bau et al. , 2019 ) . Given all these negative results about GANs , the surprising finding in the current paper is that GANs do allow for good estimates of test error ( and generalization error ) . This is verified for families of well-known GANs and datasets including primarily CIFAR-10/100 , Tiny ImageNet and wellknown deep neural classifier architectures . In particular , in sections 3.1 and 3.2 we evaluate on the PGDL and DEMOGEN benchmarks of predicting generalization and present strong results . In Section 4 and 5 , we also investigate reasons behind the surprising efficacy of GANs in predicting generalization as well as the effects of using data augmentation during GAN training . 2 RELATED WORK . Generalization Bounds Traditional approaches to predict generalization construct a generalization bound based on some notion of capacity such as parameter count , VC dimension , Rademacher complexity , etc . Neyshabur et al . ( 2018 ) provide a tighter generalization bound that decreases with increasing number of hidden units . Dziugaite & Roy ( 2017 ) reveal a way to compute nonvacuous PAC-Bayes generalization bounds . Recently , bounds based on knowledge distillation have also come to light ( Hsu et al. , 2020 ) . Despite progress in these approaches , a study conducted by Jiang et al . ( 2019 ) with extensive hyper-parameter search showed that current generalization bounds may not be effective , and the root cause of generalization remains elusive . Given the arduous nature of constructing such bounds , an interest in complexity measures has arisen . Predicting Generalization in Deep Learning The PGDL competition ( Jiang et al. , 2020 ) was held in NeuRIPS 2020 in an effort to encourage the discovery of empirical generalization measures following the seminal work of Jiang et al . ( 2018 ) . The winner of the PGDL competition Natekar & Sharma ( 2020 ) investigated properties of representations in intermediate layers to predict generalization . Kashyap et al . ( 2021 ) , the second place winner , experiment with robustness to flips , random erasing , random saturation , and other such natural augmentations . Afterwards , Jiang et al . ( 2021 ) interestingly find that generalization can be predicted by running SGD on the same architecture multiple times , and measuring the disagreement ratio between the different resulting networks on an unlabeled test set . There are also some pitfalls in predicting generalization , as highlighted by Dziugaite et al . ( 2020 ) . They find that distributional robustness over a family of environments is more applicable to neural networks than straight averaging . Limitations of GANs Arora et al . ( 2017 ) prove the lack of diversity enforcing in GAN ’ s training objective , and Arora et al . ( 2018 ) introduce the Birthday Paradox test to conclude that many popular GAN models in practice do learn a distribution with relatively small support . Santurkar et al . ( 2018 ) and Bau et al . ( 2019 ) investigate the extent to which GAN generated samples can match the distributional statistics of the original training data , and find that they have significant shortcomings . Ravuri & Vinyals ( 2019 ) find that GAN data is of limited use in training ResNet models , and find that neither inception score nor FID are predictive of generalization performance . Notably , despite the small support , Arora et al . ( 2018 ) reveal that GANs generate distinct images from their nearest neighbours in the training set . Later , Webster et al . ( 2019 ) use latent recovery to conclude more carefully that GANs do not memorize the training set . Theoretical Justification for Using GANs Arora et al . ( 2017 ) construct a generator that passes training against any bounded capacity discriminator but can only generate a small number of distinct data points either from the true data distribution or simply from the training set . For predicting generalization , it is crucial for the generator not to memorize training data . While Arora et al . ( 2017 ) do not answer why GANs do not memorize training data , a recent empirical study by Huang et al . ( 2021 ) demonstrates the difficulty of recovering input data by inverting gradients . Their work may cast light on how the generator could learn to generate data distinct from the training set when trained with gradient feedbacks from the discriminator . However , we are not aware of any theory that fully explains GANs ’ strength for predicting generalization despite limitations . 3 PREDICTING TEST PERFORMANCE USING GAN SAMPLES . We now precisely define what it means to predict test performance in our setting . We denote by Strain , Stest and Ssyn the training set , test set and the synthetic dataset generated by GANs . Given a classifier f trained on the training set Strain , we aim to predict its classification accuracy g ( f ) : = 1 |Stest| ∑ ( x , y ) ∈Stest 1 [ f ( x ) = y ] on a test set Stest . Our proposal is to train a conditional GAN model on the very same training set Strain , and then sample from the generator a synthetic dataset Ssyn of labeled examples . In the end , we simply use f ’ s accuracy on the synthetic dataset as our prediction for its test accuracy . Algorithm 1 formally describes this procedure . Algorithm 1 Predicting test performance Require : target classifier f , training set Strain , GAN training algorithm A 1 . Train a conditional GAN model using Strain : G , D = A ( Strain ) where G , D are the generator and discriminator networks . 2 . Generate a synthetic dataset by sampling from the generator G : Ssyn = { ( x̃1 , ỹ1 ) , . . . , ( x̃N , ỹN ) } where x̃i , ỹi = G ( zi , ỹi ) . The zi ’ s are drawn i.i.d . from G ’ s default input distribution . N and ỹi ’ are chosen so as to match statistics of the training set . Output : the synthetic accuracy ĝ ( f ) : = 1|Ssyn| ∑ ( x̃ , ỹ ) ∈Ssyn 1 [ f ( x̃ ) = ỹ ] as the prediction Remark . Any N ≥ |Strain| is a safe choice to ensure that ĝ ( f ) concentrates1around its mean Ez , ỹ [ 1 [ f ( G ( z , ỹ ) ) = ỹ ] ] and its deviation has negligible influence on the performance . 1the deviation is only O ( 1/ √ N ) by standard concentration bounds We demonstrate in Figure 1 that the test accuracy g ( f ) consistently resides in a small neighborhood of ĝ ( f ) for a diverse class of deep neural net classifiers trained on different datasets . For the choice of GAN architecture , we adopt the pre-trained BigGAN+DiffAug models ( Brock et al. , 2019 ; Zhao et al. , 2020 ) from the StudioGAN library2 . We evaluate on pretrained deep neural net classifiers with architectures ranging from VGG- ( 11 , 13 , 19 ) , ResNet- ( 18 , 34 , 50 ) , DenseNet ( 121 , 169 ) , ShuffleNet , PNASNet and MobileNet trained on CIFAR-10 and Tiny ImageNet with hyper-parameter setting ( learning rate , weight decay factor , learning rate schedule ) uniformly sampled from the grid { 10−1 , 10−2 } × { 5× 10−4 , 10−3 } × { CosineAnnealing , ExponentialDecay } . We use SGD with momentum 0.9 and batch size 128 and data augmention of horizontal flips for training all classifiers . 3.1 EVALUATION ON PGDL COMPETITION . We evaluate our proposed method on the tasks of NeurIPS 2020 Competition on Predicting Generalization of Deep Learning . The PGDL competition consists of 8 tasks3 , each containing a set of pre-trained deep net classifiers of similar architectures but with different hyper-parameter settings , as well as the training data . The tasks of PGDL are based on a wide range of datasets inlcuding CIFAR-10 , SVHN , CINIC-10 , Oxford Flowers , Oxford Pets and Fashion MNIST . For every task , the goal is to compute a scalar prediction for each classifier based on its parameters and the training data that correlates as much as possible with the actual generalization errors of the classifiers measured on a test set . The correlation score is measured by the so-called Conditional Mutual Information , which is designed to indicate whether the computed predictions contain all the information about the generalization errors such that knowing the hyper-parameters does not provide additional information , see ( Jiang et al. , 2020 ) for details . Since the goal of PGDL is to predict generalization gap instead of test accuracy , our proposed solution is a slight adaptation of Algorithm 1 . For each task , we first train a GAN model on the provided training dataset and sample from it a labeled synthetic dataset . Then for each classifier within the task , instead of the raw synthetic accuracy , we use as prediction the gap between training and synthetic accuracies . For all tasks , we use BigGAN+diffAug with the default4 implementation for CIFAR-10 from the StudioGAN library . We found that a subset of them ( Task 6 with Oxford Flowers and Task 7 with Oxford Pets ) were not well-suited for standard GAN training without much tweaking due to a small training set and highly uneven class sizes . We focused only on the subset of the tasks where GAN training worked out of the box , specifically Task 1 , 2 , 4 , 5 , 8 , and 9 . 2available at https : //github.com/POSTECH-CVLab/PyTorch-StudioGAN 3namely Task 1 , 2 , 4 , 5 , 6 , 7 , 8 , 9 , without 3 . 4Task 8 uses the single-channel 28×28 Fashion-MNIST dataset . For GAN training , we symmetrically zeropad the training images to 32 × 32 and convert to RGB by replicating the channel . To compute the synthetic accuracy , we convert the generated images back to single-channel and perform a center cropping . We report the results in Table 1 , where we observe that on tasks involving either CIFAR-10 or VGGlike classifiers , our proposed solution out-performs all of the solutions from the top 3 teams of the competition by a large margin . One potential reason may be that the default hyper-parameters of the BigGAN model have been engineered towards CIFAR-10 like datasets and VGG/ResNet-like discriminator networks . It is worth mentioning that we conducted absolutely zero hyper-parameter search for all tasks here . Especially for CIFAR-10 tasks , we directly take the pre-trained models from the StudioGAN library . It is likely that we may achieve even better scores with fine-tuned GAN hyper-parameters and optimized training procedure for each PGDL task , and we leave it to future work .

Summary: This work studies generalization prediction with synthetic data produced by GANs. Specifically the training accuracy gap between synthetic and given data is proposed as a measure for generalization. The authors show strong improvement over other PGDL and DEMOGEN results. A further empirical finding is made on the greater similarity between test and synthetic samples than test and train samples, as measured by FID. A variety of GAN architectures are explored.