Year: 2021

Abstract:
Research on generalization bounds for deep networks seeks to give ways to predict test error using just the training dataset and the network parameters. While generalization bounds can give many insights about architecture design, training algorithms etc., what they do not currently do is yield good predictions for actual test error. A recently introduced Predicting Generalization in Deep Learning competition (Jiang et al., 2020) aims to encourage discovery of methods to better predict test error. The current paper investigates a simple idea: can test error be predicted using synthetic data, produced using a Generative Adversarial Network (GAN) that was trained on the same training dataset? Upon investigating several GAN models and architectures, we find that this turns out to be the case. In fact, using GANs pre-trained on standard datasets, the test error can be predicted without requiring any additional hyper-parameter tuning. This result is surprising because GANs have well-known limitations (e.g. mode collapse) and are known to not learn the data distribution accurately. Yet the generated samples are good enough to substitute for test data. Several additional experiments are presented to explore reasons why GANs do well at this task. In addition to a new approach for predicting generalization, the counter-intuitive phenomena presented in our work may also call for a better understanding of GANs’ strengths and limitations.

heading: 1 INTRODUCTION
text: Why do vastly overparametrized neural networks achieve impressive generalization performance across many domains, with very limited capacity control during training? Despite some promising initial research, the mechanism behind generalization remains poorly understood. A host of papers have tried to adapt classical generalization theory to prove upper bounds of the following form on the difference between training and test error:
test− train ≤
√ C
|S| + tiny term (1)
where S is the training dataset and C is a so-called complexity measure, typically involving some function of the training dataset as well as the trained net parameters (e.g., a geometric norm). Current upper bounds of this type are loose, and even vacuous. There is evidence that such classically derived bounds may be too loose (Dziugaite & Roy, 2017) or that they may not correlate well with generalization (Jiang et al., 2019). This has motivated a more principled empirical study of the effectiveness of generalization bounds. The general idea is to use machine learning to determine which network characteristics promote good generalization in practice and which do not —in other words, treat various deep net characteristics/norms/margins etc. as inputs to a machine learning model that uses them to predict the generalization error achieved by the net. This could help direct theorists to new types of complexity measures and motivate new theories.
A recently started competition of Predicting Generalization in Deep Learning (Jiang et al., 2020) (PGDL) seeks to increase interest in such investigations, in an effort to uncover new and promising network characteristics and/or measures of network complexity that correlate with good generalization. As required in Jiang et al. (2020), a complexity measure should depend on the trained model, optimizer, and training set, but not the held out test data. The first PGDL competition in 2020 did uncover quite a few measures that seemed to be predictive of good generalization but had not been identified by theory work.
In this paper, we explore a very simple baseline for predicting generalization that had hitherto not received attention: train a Generative Adversarial Network (GAN) on the training dataset, and use performance on the synthetic data produced by the GAN to predict generalization. At first sight GANs may not appear to be an obvious choice for this task, due to their well known limitations. For instance, while the goal of GANs training is to find a generator that fools the discriminator net —in the sense that the discriminator has low probability of spotting a difference between GAN samples and the samples from the true distribution—in practice the discriminator is often able to discriminate very well at the end, demonstrating that it was not fooled. Also, GANs’ generators are known to exhibit mode collapse i.e., the generated distribution is a tiny subset of the true distribution. There is theory and experiments suggesting this may be difficult to avoid (Arora et al., 2018; Santurkar et al., 2018; Bau et al., 2019).
Given all these negative results about GANs, the surprising finding in the current paper is that GANs do allow for good estimates of test error (and generalization error). This is verified for families of well-known GANs and datasets including primarily CIFAR-10/100, Tiny ImageNet and wellknown deep neural classifier architectures. In particular, in sections 3.1 and 3.2 we evaluate on the PGDL and DEMOGEN benchmarks of predicting generalization and present strong results. In Section 4 and 5, we also investigate reasons behind the surprising efficacy of GANs in predicting generalization as well as the effects of using data augmentation during GAN training.

heading: 2 RELATED WORK
text: Generalization Bounds Traditional approaches to predict generalization construct a generalization bound based on some notion of capacity such as parameter count, VC dimension, Rademacher complexity, etc. Neyshabur et al. (2018) provide a tighter generalization bound that decreases with increasing number of hidden units. Dziugaite & Roy (2017) reveal a way to compute nonvacuous PAC-Bayes generalization bounds. Recently, bounds based on knowledge distillation have also come to light (Hsu et al., 2020). Despite progress in these approaches, a study conducted by Jiang et al. (2019) with extensive hyper-parameter search showed that current generalization bounds may not be effective, and the root cause of generalization remains elusive. Given the arduous nature of constructing such bounds, an interest in complexity measures has arisen.
Predicting Generalization in Deep Learning The PGDL competition (Jiang et al., 2020) was held in NeuRIPS 2020 in an effort to encourage the discovery of empirical generalization measures following the seminal work of Jiang et al. (2018). The winner of the PGDL competition Natekar & Sharma (2020) investigated properties of representations in intermediate layers to predict generalization. Kashyap et al. (2021), the second place winner, experiment with robustness to flips, random erasing, random saturation, and other such natural augmentations. Afterwards, Jiang et al. (2021) interestingly find that generalization can be predicted by running SGD on the same architecture multiple times, and measuring the disagreement ratio between the different resulting networks on an unlabeled test set. There are also some pitfalls in predicting generalization, as highlighted by Dziugaite et al. (2020). They find that distributional robustness over a family of environments is more applicable to neural networks than straight averaging.
Limitations of GANs Arora et al. (2017) prove the lack of diversity enforcing in GAN’s training objective, and Arora et al. (2018) introduce the Birthday Paradox test to conclude that many popular GAN models in practice do learn a distribution with relatively small support. Santurkar et al. (2018) and Bau et al. (2019) investigate the extent to which GAN generated samples can match the distributional statistics of the original training data, and find that they have significant shortcomings. Ravuri & Vinyals (2019) find that GAN data is of limited use in training ResNet models, and find that neither inception score nor FID are predictive of generalization performance. Notably, despite the small support, Arora et al. (2018) reveal that GANs generate distinct images from their nearest neighbours in the training set. Later, Webster et al. (2019) use latent recovery to conclude more carefully that GANs do not memorize the training set.
Theoretical Justification for Using GANs Arora et al. (2017) construct a generator that passes training against any bounded capacity discriminator but can only generate a small number of distinct data points either from the true data distribution or simply from the training set. For predicting
generalization, it is crucial for the generator not to memorize training data. While Arora et al. (2017) do not answer why GANs do not memorize training data, a recent empirical study by Huang et al. (2021) demonstrates the difficulty of recovering input data by inverting gradients. Their work may cast light on how the generator could learn to generate data distinct from the training set when trained with gradient feedbacks from the discriminator. However, we are not aware of any theory that fully explains GANs’ strength for predicting generalization despite limitations.

heading: 3 PREDICTING TEST PERFORMANCE USING GAN SAMPLES
text: We now precisely define what it means to predict test performance in our setting. We denote by Strain, Stest and Ssyn the training set, test set and the synthetic dataset generated by GANs. Given a classifier f trained on the training set Strain, we aim to predict its classification accuracy g(f) :=
1 |Stest| ∑ (x,y)∈Stest 1 [f(x) = y] on a test set Stest. Our proposal is to train a conditional GAN model on the very same training set Strain, and then sample from the generator a synthetic dataset Ssyn of labeled examples. In the end, we simply use f ’s accuracy on the synthetic dataset as our prediction for its test accuracy. Algorithm 1 formally describes this procedure.
Algorithm 1 Predicting test performance Require: target classifier f , training set Strain, GAN training algorithm A
1. Train a conditional GAN model using Strain:
G,D = A(Strain) where G,D are the generator and discriminator networks.
2. Generate a synthetic dataset by sampling from the generator G:
Ssyn = {(x̃1, ỹ1), . . . , (x̃N , ỹN )} where x̃i, ỹi = G(zi, ỹi).
The zi’s are drawn i.i.d. from G’s default input distribution. N and ỹi’ are chosen so as to match statistics of the training set.
Output: the synthetic accuracy ĝ(f) := 1|Ssyn| ∑ (x̃,ỹ)∈Ssyn 1 [f(x̃) = ỹ] as the prediction
Remark. Any N ≥ |Strain| is a safe choice to ensure that ĝ(f) concentrates1around its mean Ez,ỹ [1 [f(G(z, ỹ)) = ỹ]] and its deviation has negligible influence on the performance.
1the deviation is only O ( 1/ √ N ) by standard concentration bounds
We demonstrate in Figure 1 that the test accuracy g(f) consistently resides in a small neighborhood of ĝ(f) for a diverse class of deep neural net classifiers trained on different datasets. For the choice of GAN architecture, we adopt the pre-trained BigGAN+DiffAug models (Brock et al., 2019; Zhao et al., 2020) from the StudioGAN library2. We evaluate on pretrained deep neural net classifiers with architectures ranging from VGG-(11, 13, 19), ResNet-(18, 34, 50), DenseNet(121, 169), ShuffleNet, PNASNet and MobileNet trained on CIFAR-10 and Tiny ImageNet with hyper-parameter setting (learning rate, weight decay factor, learning rate schedule) uniformly sampled from the grid { 10−1, 10−2 } × { 5× 10−4, 10−3 } × {CosineAnnealing, ExponentialDecay}. We use SGD with momentum 0.9 and batch size 128 and data augmention of horizontal flips for training all classifiers.

heading: 3.1 EVALUATION ON PGDL COMPETITION
text: We evaluate our proposed method on the tasks of NeurIPS 2020 Competition on Predicting Generalization of Deep Learning. The PGDL competition consists of 8 tasks3, each containing a set of pre-trained deep net classifiers of similar architectures but with different hyper-parameter settings, as well as the training data. The tasks of PGDL are based on a wide range of datasets inlcuding CIFAR-10, SVHN, CINIC-10, Oxford Flowers, Oxford Pets and Fashion MNIST. For every task, the goal is to compute a scalar prediction for each classifier based on its parameters and the training data that correlates as much as possible with the actual generalization errors of the classifiers measured on a test set. The correlation score is measured by the so-called Conditional Mutual Information, which is designed to indicate whether the computed predictions contain all the information about the generalization errors such that knowing the hyper-parameters does not provide additional information, see (Jiang et al., 2020) for details.
Since the goal of PGDL is to predict generalization gap instead of test accuracy, our proposed solution is a slight adaptation of Algorithm 1. For each task, we first train a GAN model on the provided training dataset and sample from it a labeled synthetic dataset. Then for each classifier within the task, instead of the raw synthetic accuracy, we use as prediction the gap between training and synthetic accuracies. For all tasks, we use BigGAN+diffAug with the default4 implementation for CIFAR-10 from the StudioGAN library. We found that a subset of them (Task 6 with Oxford Flowers and Task 7 with Oxford Pets) were not well-suited for standard GAN training without much tweaking due to a small training set and highly uneven class sizes. We focused only on the subset of the tasks where GAN training worked out of the box, specifically Task 1, 2, 4, 5, 8, and 9.
2available at https://github.com/POSTECH-CVLab/PyTorch-StudioGAN 3namely Task 1, 2, 4, 5, 6, 7, 8, 9, without 3. 4Task 8 uses the single-channel 28×28 Fashion-MNIST dataset. For GAN training, we symmetrically zeropad the training images to 32 × 32 and convert to RGB by replicating the channel. To compute the synthetic accuracy, we convert the generated images back to single-channel and perform a center cropping.
We report the results in Table 1, where we observe that on tasks involving either CIFAR-10 or VGGlike classifiers, our proposed solution out-performs all of the solutions from the top 3 teams of the competition by a large margin. One potential reason may be that the default hyper-parameters of the BigGAN model have been engineered towards CIFAR-10 like datasets and VGG/ResNet-like discriminator networks. It is worth mentioning that we conducted absolutely zero hyper-parameter search for all tasks here. Especially for CIFAR-10 tasks, we directly take the pre-trained models from the StudioGAN library. It is likely that we may achieve even better scores with fine-tuned GAN hyper-parameters and optimized training procedure for each PGDL task, and we leave it to future work.

heading: 3.2 EVALUATION ON DEMOGEN BENCHMARK
text: The seminal work of Jiang et al. (2018) constructed the Deep Model Generalization benchmark (DEMOGEN) that consists of 216 ResNet-32 (He et al., 2016) models and 216 Network-inNetworks (Lin et al., 2013) models trained on CIFAR-10 plus another 324 ResNet-32 models trained on CIFAR-100, along with their training and test performances. These models are configured with different normalization techniques (group-norm v.s. batch-norm), network widths, learning rates, weight-decay factors, and notably whether to train with or without data augmentation.
The goal of DEMOGEN, different from PGDL, is to numerically predict generalization gap (or equivalently test accuracy) with the additional knowledge of the test accuracies of a few pretrained classifiers. Following the setup of Jiang et al. (2018), we use a simple linear predictor g(f) = a · ĝ(f) + b with two scalar variables a, b ∈ R instead of directly using ĝ(f) for the best performance. Here a, b are estimated using least squares regression on a separate pool of pre-trained classifiers with known test accuracies. Jiang et al. (2018) proposes to use variants of coefficient of determination (R2) of the resulted predictor as evaluation metrics. Specifically, the adjusted R2 and k-fold R2 are adopted (see Jiang et al. (2018) for concrete definitions).
In Table 2, we compare the scores of our method with several state-of-the-art GAN models. Specifically, we again take the pre-trained GAN models from the StudioGAN library to generate synthetic datasets for the CIFAR-10 tasks. For the CIFAR-100 task, we use the same hyper-parameters as the CIFAR-10 models to train a plain BigGAN and a BigGAN+DiffAug model from scratch. Again, each synthetic dataset is sampled only once. Overall, predicting using GAN samples achieves remarkably higher scores than all methods proposed in Jiang et al. (2018). We also visualize the resulted linear fit for each task in Figure 2 where we observe the strong linear correlation. Note that the resulting linear fits are close the ideally desired y = x fit. In fact, using directly ĝ(f) as the prediction would also lead to extremely competitive R2 values 0.928, 0.995 and 0.975 respectively.

heading: 4 GAN SAMPLES ARE CLOSER TO TEST SET THAN TRAINING SET
text: In this section, we demonstrate a surprising property of the synthetic examples produced by GANs that may provide insights into their observed ability to predict generalization: synthetic datasets generated by GANs are ‘closer’ to test set than training set in the feature space of well-trained deep net classifiers, even though GAN models had (indirect) access to the training set but no access to the test set. Here ‘well-trained’ classifiers are the ones that almost perfectly classify the training examples, i.e. its training accuracy is > 97%. To define closeness properly, we modify the widely used standard Fréchet Distance proposed by Heusel et al. (2017) to account for labeled examples. We name our modified version as the Class-conditional Fréchet Distance.

heading: 4.1 CLASS-CONDITIONAL FRÉCHET DISTANCE
text: The goal is to measure how close the labeled synthetic dataset is to the test set in the eyes of a certain pre-trained classifier. However, FID does not capture any label information by default. Our proposed class-conditional Fréchet Distance can be viewed as the sum over each class of the Fréchet Distance between test and synthetic examples in a certain feature space. The original FID measures a certain distance between the feature embeddings of real examples and synthetic examples from GANs, in the feature space of an Inception network pre-trained on ImageNet. For our purposes, we modify the notion of FID to use any feature extractor h. Specifically, we define the distance between two sets of examples S and S′ w.r.t. a feature extractor h as
Fh (S, S′) := ‖µS(h)− µS′(h)‖22 +Tr [ CovS (h) + CovS′ (h)− 2 [CovS (h) CovS′ (h)]1/2 ] where we have the empirical mean of h’s output µS(h) := 1|S| ∑ (x,y)∈S h(x) and the empirical
covariance CovS(h) := 1|S| ∑ (x,y)∈S (h(x)− µS(h)) (h(x)− µS(h)) > over the examples from S, and µS′(h),CovS′(h) are computed analogously. Furthermore, [·]1/2 is the matrix square root and Tr [·] is the trace function.
For labeled datasets S and S̃, we denote by C the collection of all possible labels. For each label c ∈ C, we let Sc and S̃c denote the the subsets of examples with label c in S and S̃ respectively. This leads to the class-conditional Fréchet Distance:
dh
( S, S̃ ) := ∑ c∈C Fh ( Sc, S̃c ) In Figure 3, we evaluate dh on CIFAR-10 among the training set Strain, the test set Stest and the synthetic set Ssyn, with h being the feature map immediately before the fully-connected layer of each pre-trained deep net used in the middle sub-figure of Figure 2. The synthetic dataset is generated by the same BigGAN+DiffAug model used in Figure 2 as well. We observe that for all pre-trained nets, dh(Ssyn, Stest) < dh(Strain, Stest) by a big margin, which provides a partial explanation for why GAN synthesized samples may be a good replace for the test set. Furthermore, for every well-trained net, dh(Ssyn, Stest) < dh(Ssyn, Strain) as well. This is particularly surprising because Ssyn ends up being ‘closer’ to the unseen Stest rather than Strain, despite the GAN model being trained to minimize certain distance between Ssyn and Strain as part of the training objective.

heading: 4.2 CLASS-CONDITIONAL FRÉCHET DISTANCE ACROSS CLASSIFIER ARCHITECTURES
text: It is natural to wonder whether we can attribute the phenomenon of GAN samples being closer to test set to the architectural similarity between the evaluated classifiers and the discriminator network
used in the GAN model — they are all derivatives from the ResNet family. Thus we investigate the values of dh for the collection of classifiers with diverse architectures used in Figure 1, which contains network architectures that may be remote from ResNet such as VGG, MobileNet, PNASNet etc. The result is demonstrated in Figure 4, where we observe the surprising phenomenon that the single copy of the synthetic dataset is regarded closer to the test set by all classifiers in the collection. This suggests that certain GAN models are capable of matching, to a degree, the feature distribution universally for a diverse set of deep neural classifiers.

heading: 5 EXPLORATION: EFFECTS OF DATA AUGMENTATION
text: In this section, we discuss certain intriguing effects of data augmentation techniques used during GAN training on the performance of the resulted synthetic datasets for predicting generalization. We use the CIFAR-100 task from DEMOGEN as a demonstration where the effects are the most pronounced. The task consists of 324 ResNet classifiers, only half of which are trained with data augmentation, specifically random crop and horizontal flipping.
Given the success of complexity measures that verify ‘robustness to augmentation’ in the PGDL competition, one would imagine applying data augmentations to the real examples during GAN training could be beneficial for prediciting generalization, since the generator would be encouraged to match the distribution of the augmented images. However, we show in Figure 5 this is not the case here.
Specifically, Figure 5a indicates that, when evaluated on a synthetic dataset from a GAN model trained with augmentation only applied to real examples, the ResNets trained without any data augmentation exhibit a test accuracy v.s. synthetic accuracy curve that substantially deviates from y = x, while the ones trained with data augmentations are relatively unaffected. In comparison, once data augmentation is removed from GAN’s training procedure, this deviation disappears, as captured in Figure 5b. This could be because the ResNets trained without data augmentation are sensitive to the perturbations learned by the generator. Overall, the best result is achieved by deploying the differential augmentation technique that applies the same differentiable augmentation to both real and fake images and enables the gradients to be propagated through the augmentation back to the generator so as to regularize the discriminator without manipulating the target distribution.

heading: 6 DISCUSSION AND CONCLUSIONS
text: Competitions such as PGDL are an exciting way to inject new ideas into generalization theory. The interaction of GANs and good generalization needs further exploration. It is conceivable that the quality of the GAN-based prediction is correlated with the niceness of architectures in some way. It would not be surprising that it works for popular architectures but does not work for all. We leave it as future work to determine the limits of using GANs for predicting generalization.
Not only does our finding presented in this paper offer a new approach for predicting generalization, but it also sheds light on a possibly bigger question —what can GANs learn despite known limitations? To our best knowledge, we are not aware of any existing theory that completely explains the counter-intuitive empirical phenomena presented here. Such a theoretical understanding will necessarily require innovative techniques and thus is perceivably non-trivial. However, we believe it is an important future direction because it will potentially suggest more use cases of GANs and will help
improve existing models in a principled way. We hope our work inspires new endeavors along this line.

Review summaries:
This paper investigates the task of using GAN’s samples as a substitute for the true test distribution for estimating the generalization. Experiments show that samples from GAN are a powerful substitute for the test set and the predictive power of the synthetic accuracy outperforms all prior methods include the winner of the 2020 PGDL competition, despite all the empirical and theoretical evidence that GANs are not good at approximating the true data distribution. The paper then performs a preliminary investigation on why samples from GANs are a good substitute for the test set and found that using a certain notion of perceptual similarity, samples from GAN’s are much closer to the test distribution than the training distribution despite being trained on the training data.

This work studies generalization prediction with synthetic data produced by GANs. Specifically the training accuracy gap between synthetic and given data is proposed as a measure for generalization. The authors show strong improvement over other PGDL and DEMOGEN results. A further empirical finding is made on the greater similarity between test and synthetic samples than test and train samples, as measured by FID. A variety of GAN architectures are explored. 

The paper considers the problem of predicting the test performance of neural networks, given only the training set and model hyperparameters. While there have been many methods proposed for this, the paper considers the simple baseline of generating synthetic test samples from a GAN and computing error using these samples. This baseline appears to outperform all or almost all existing methods on a variety of tasks, including a recent competition on predicting generalization.