Fantastic Generalization Measures and Where to Find Them

1 INTRODUCTION . Deep neural networks have seen tremendous success in a number of applications , but why ( and how well ) these models generalize is still a mystery ( Neyshabur et al. , 2014 ; Zhang et al. , 2016 ; Recht et al. , 2019 ) . It is crucial to better understand the reason behind the generalization of modern deep learning models ; such an understanding has multiple benefits , including providing guarantees for safety-critical scenarios and the design of better models . A number of papers have attempted to understand the generalization phenomenon in deep models from a theoretical perspective e.g . ( Neyshabur et al. , 2015b ; Bartlett et al. , 2017 ; Neyshabur et al. , 2018a ; Golowich et al. , 2017 ; Arora et al. , 2018 ; Nagarajan and Kolter , 2019a ; Wei and Ma , 2019a ; Long and Sedghi , 2019 ) . The most direct and principled approach for studying generalization in deep learning is to prove a generalization bound ; typically an upper bound on the test error based on some quantity that can be calculated on the training set . Unfortunately , finding tight bounds has proven to be an arduous undertaking . While encouragingly Dziugaite and Roy ( 2017 ) showed that PAC-Bayesian bounds can be optimized to achieve a reasonably tight generalization bound , current bounds are still not tight enough to accurately capture the generalization behavior . Others have proposed more direct empirical ways to characterize generalization of deep networks without attempting to deriving bounds ( Keskar et al. , 2016 ; Liang et al. , 2017 ) . However , as pointed by Dziugaite and Roy ( 2017 ) , empirical correlation does not necessarily translate to a casual relationship between a measure and generalization . A core component in ( theoretical or empirical ) analysis of generalization is the notion of complexity measure ; a quantity that monotonically relates to some aspect of generalization . More specifically , lower complexity should often imply smaller generalization gap . A complexity measure may depend on the properties of the trained model , optimizer , and possibly training data , but should not have access to a validation set . Theoretically motivated complexity measures such as VC-dimension , norm of parameters , etc. , are often featured as the major components of generalization bounds , where the monotonic relationship between the measures and generalization is mathematically established . In contrast , empirically motivated complexity measures such as sharpness ( Keskar et al. , 2016 ) are justified by experimentation and observation . In this work , we do not need to distinguish between theoretically vs empirically motivated measures , and simply refer to both as complexity measures . ∗Contributed equally . †Work done as part of the Google AI Residency Program . Despite the prominent role of complexity measures in studying generalization , the empirical evaluation of these measures is usually limited to a few models , often on toy problems . A measure can only be considered reliable as a predictor of generalization gap if it is tested extensively on many models at a realistic problem size . To this end , we carefully selected a wide range of complexity measures from the literature . Some of the measures are motivated by generalization bounds such as those related to VC-dimension , norm or margin based bounds , and PAC-Bayesian bounds . We further selected a variety of empirical measures such as sharpness ( Keskar et al. , 2016 ) , Fisher-Rao norm ( Liang et al. , 2017 ) and path norms ( Neyshabur et al. , 2017 ) . In this study , we trained more than 10,000 models over two image classification datasets , namely , CIFAR-10 ( Krizhevsky et al. , 2014 ) and Street View House Numbers ( SVHN ) Netzer et al . ( 2011 ) . In order to create a wide range of generalization behaviors , we carefully varied hyperparameters that are believed to influence generalization . We also selected multiple optimization algorithms and looked at different stopping criteria for training convergence . Details of all our measures and hyperparameter selections are provided in Appendix D. Training under all combination of hyperparameters and optimization resulted in a large pool of models . For any such model , we considered 40 complexity measures . The key findings that arise from our large scale study are summarized below : 1 . It is easy for some complexity measures to capture spurious correlations that do not reflect causal insights about generalization ; to mitigate that we propose a more rigorous method to study them . 2 . Many norm-based measures not only perform poorly , but negatively correlate with generalization specifically when the optimization procedure injects some stochasticity . In particular , the generalization bound based on the product of spectral norms of the layers ( similar to that of Bartlett et al . ( 2017 ) ) has very strong negative correlation with generalization . 3 . Sharpness-based measures like PAC-Bayesian bounds ( McAllester , 1999 ) and the measure proposed by Keskar et al . ( 2016 ) perform best and seem promising candidates for further research . 4 . Measures related to the optimization procedures such as the gradient noise and the speed of the optimization can be predictive of generalization . Our findings on the relative success of sharpness-based and optimization-based complexity measures for predicting the generalization gap can provoke further study of these measures . 1.1 RELATED WORK . The theoretically motivated measures that we consider in this work belong to a few different families : PAC-Bayes ( McAllester , 1999 ; Dziugaite and Roy , 2017 ; Neyshabur et al. , 2017 ) ; VC-dimension ( Vapnik and Chervonenkis , 1971 ) ; and norm-based bounds ( Neyshabur et al. , 2015b ; Bartlett et al. , 2017 ; Neyshabur et al. , 2018a ) . The empirically motivated measures from prior literature that we consider are based on sharpness measure ( Keskar et al. , 2016 ) ; Fisher-Rao measure ( Liang et al. , 2017 ) ; distance of trained weights from initialization ( Nagarajan and Kolter , 2019b ) and path norm ( Neyshabur et al. , 2015a ) . Finally , we consider some optimization based measures based on the speed of the optimization algorithm as motivated by the work of ( Hardt et al. , 2015 ) and ( Wilson et al. , 2017a ) , and the magnitude of the gradient noise as motivated by the work of ( Chaudhari and Soatto , 2018 ) and ( Smith and Le , 2017 ) . A few papers have explored a large scale study of generalization in deep networks . Neyshabur et al . ( 2017 ) perform a small scale study of the generalization of PAC-Bayes , sharpness and a few different norms , and the generalization analysis is restricted to correlation . Jiang et al . ( 2018 ) studied the role of margin as a predictor of the generalization gap . However , they used a significantly more restricted set of models ( e.g . no depth variations ) , the experiments were not controlled for potential undesired correlation ( e.g . the models can have vastly different training error ) and some measures contained parameters that must be learned from the set of models . Novak et al . ( 2018 ) conducted large scale study of neural networks but they only looked at correlation of a few measures to generalization . In contrast , we study thousands of models , and perform controlled experiments to avoid undesired artificial correlations . Some of our analysis techniques are inspired by Neal ( 2019 ) who proposed the idea of studying generalization in deep models via causal graphs , but did not provide any details or empirical results connected to that idea . Our work focuses on measures that can be computed on a single model and compares a large number of bounds and measures across a much wider range of models in a carefully controlled fashion . 2 GENERALIZATION : WHAT IS THE GOAL AND HOW TO EVALUATE ? . Generalization is arguably the most fundamental and yet mysterious aspect of machine learning . The core question in generalization is what causes the triplet of a model , optimization algorithm , and data properties1 , to generalize well beyond the training set . There are many hypotheses concerning this question , but what is the right way to compare these hypotheses ? The core component of each hypothesis is complexity measure that monotonically relates to some aspect of generalization . Here we briefly discuss some potential approaches to compare different complexity measures : • Tightness of Generalization Bounds . Proving generalization bounds is very useful to establish the causal relationship between a complexity measure and the generalization error . However , almost all existing bounds are vacuous on current deep learning tasks ( combination of models and datasets ) , and therefore , one can not rely on their proof as an evidence on the causal relationship between a complexity measure and generalization currently2 . • Regularizing the Complexity Measure . One may evaluate a complexity measure by adding it as a regularizer and directly optimizing it , but this could fail due to two reasons . The complexity measure could change the loss landscape in non-trivial ways and make the optimization more difficult . In such cases , if the optimization fails to optimize the measure , no conclusion can be made about the causality . Another , and perhaps more critical , problem is the existence of implicit regularization of the optimization algorithm . This makes it hard to run a controlled experiment since one can not simply turn off the implicit regularization ; therefore , if optimizing a measure does not improve generalization it could be simply due to the fact that it is regularizing the model in the same way as the optimization is regularizing it implicitly . • Correlation with Generalization Evaluating measures based on correlation with generalization is very useful but it can also provide a misleading picture . To check the correlation , we should vary architectures and optimization algorithms to produce a set of models . If the set is generated in an artificial way and is not representative of the typical setting , the conclusions might be deceiving and might not generalize to typical cases . One such example is training with different portions of random labels which artificially changes the dataset . Another pitfall is drawing conclusion from changing one or two hyper-parameters ( e.g changing the width or batch-size and checking if a measure would correlate with generalization ) . In these cases , the hyper-parameter could be the true cause of both change in the measure and change in the generalization , but the measure itself has no causal relationship with generalization . Therefore , one needs to be very careful with experimental design to avoid unwanted correlations . In this work we focus on the third approach . While acknowledging all limitations of a correlation analysis , we try to improve the procedure and capture some of the causal effects as much as possible through careful design of controlled experiments . Further , to evaluate the effectiveness of complexity measures as accurately as possible , we analyze them over sufficiently trained models with a wide range of variations in hyperparameters . For practical reasons , these models must reach convergence within a reasonable time budget . Details of the notations used are outlined in Appendix A . 2.1 TRAINING MODELS ACROSS HYPERPARAMETER SPACE . In order to create models with different generalization behavior , we consider various hyperparameter types , which are known or believed to influence generalization ( e.g . batch size , dropout rate , etc. ) . Formally , denote each hyperparameter by θi taking values from the set Θi , for i = 1 , . . . , n and n denoting the total number of hyperparameter types3 . For each value of hyperparameters θ , ( θ1 , θ2 , . . . , θn ) ∈ Θ , where Θ , Θ1 × Θ2 × · · · × Θn , we train the architecture until the training loss ( cross-entropy value ) reaches a given threshold . See the Appendix C.2 for a discussion on the choice of the stopping criterion . Doing this for each hyper-parameter configuration θ ∈ Θ , we obtain a total of |Θ| models . The space Θ reflects our prior knowledge about a reasonable hyperparameter 1For example , it is expected that images share certain structures that allows some models ( which leverage these biases ) to generalize . 2See Dziugaite and Roy ( 2017 ) for an example of non-vacuous generalization bound and related discussions . 3In our analysis we use n = 7 hyperparameters : batch size , dropout probability , learning rate , network depth , weight decay coefficient , network width , optimizer . space , both in terms of their types and values . Regarding the latter , one could , for example , create Θi by grid sampling of a reasonable number of points within a reasonable range of values for θi .

Summary: The paper addresses the well-known generalization problem for deep networks: why do these models successfully generalize even though they contain more parameters than the size of the training set? Recent works on this problem have proposed various measures of model complexity (that is, different from the number of parameters) with the claim that they (at least partially) capture the "true" capacity of deep networks in order to answer this question. The authors report the results of a large-scale, systematic study of the empirical effect of these complexity measures on the observed generalization behavior of deep networks.