Assessing Generalization of SGD via Disagreement

1 INTRODUCTION . Consider the following intriguing observation made in Nakkiran & Bansal ( 2020 ) . Train two networks of the same architecture to zero training error on two independently drawn datasets S1 and S2 of the same size . Both networks would achieve a test error ( or equivalently , a generalization gap ) of about the same value , denoted by . Now , take a fresh unlabeled dataset U and measure the rate of disagreement of the predicted label between these two networks on U . Based on the triangle inequality , one can quickly surmise that this disagreement rate could lie anywhere between 0 and 2 . However , across various training set sizes and for various models like neural networks , kernel SVMs and decision trees , Nakkiran & Bansal ( 2020 ) ( or N & B ’ 20 in short ) report that the disagreement rate not only linearly correlates with the test error , but nearly equals ( see first two plots in Fig 1 ) . What brings about this unusual equality ? Resolving this open question from N & B ’ 20 could help us identify fundamental patterns in how neural networks make errors . That might further shed insight into generalization and other poorly understood empirical phenomena in deep learning . In this work , we first identify a stronger observation . Consider two neural networks trained with the same hyperparameters and the same dataset , but with different random seeds ( this could take the form e.g. , of the data being presented in different random orders and/or by using a different random initialization of the network weights ) . We would expect the disagreement rate in this setting to be much smaller than in N & B ’ 20 , since both models see the same data . Yet , this is not the case : we observe on the SVHN ( Netzer et al. , 2011 ) , CIFAR-10/100 ( Krizhevsky et al. , 2009 ) datasets , and for variants of Residual Networks ( He et al. , 2016 ) and Convolutional Networks ( Lin et al. , 2013 ) , that the disagreement rate is still approximately equal to the test error ( see last two plots in Fig 1 ) , only slightly deviating from the behavior in N & B ’ 20 . In fact , while N & B ’ 20 show that the disagreement rate captures significant changes in test error with varying training set sizes , we highlight a much stronger behavior : the disagreement rate is able to capture even minute variations in the test error under varying hyperparameters like width , depth and batch size . Furthermore , we show that under certain training conditions , these properties even hold on many kinds of out-of-distribution data in the PACS dataset ( Li et al. , 2017 ) , albeit not on all kinds . The above observations not only raise deeper conceptual questions about the behavior of deep networks but also crucially yield a practical benefit . In particular , our disagreement rate does not require fresh labeled data ( unlike the rate in N & B ’ 20 ) and rather only requires fresh unlabeled data . Hence , ours is a more meaningful and practical estimator of test accuracy ( albeit , a slightly less accurate estimate at that ) . Indeed , unsupervised accuracy estimation is valuable for real-time evaluation of models when test labels are costly or unavailable to due to privacy considerations ( Donmez et al. , 2010 ; Jaffe et al. , 2015 ) . While there are also many other measures that correlate with generalization without access to even unlabeled test data ( Jiang et al. , 2018 ; Yak et al. , 2019 ; Jiang et al. , 2020b ; a ; Natekar & Sharma , 2020 ; Unterthiner et al. , 2020 ) , these require ( a ) computing intricate proportionality constants and ( b ) knowledge of the neural network weights/representations . Disagreement , however , provides a direct estimate , and works even with a black-box model , which makes it practically viable when the inner details of the model are unavailable due to privacy concerns . In the second part of our work , we theoretically investigate these observations . Informally stated , we prove that if the ensemble learned from different stochastic runs of the training algorithm ( e.g. , across different random seeds ) is well-calibrated ( i.e. , the predicted probabilities are neither overconfident nor under-confident ) , then the disagreement rate equals the test error ( in expectation over the training stochasticity ) . Indeed , such kinds of SGD-trained deep network ensembles are known to be naturally calibrated in practice ( Lakshminarayanan et al. , 2017 ) . While we do not prove why calibration holds in practice , the fact the condition in our theorem is empirically satisfied implies that our theory offers a valuable insight into the practical generalization properties of deep networks . Overall , our work establishes a new connection between generalization and calibration via the idea of disagreement . This has both theoretical and practical implications in understanding generalization and the effect of stochasticity in SGD . To summarize , our contributions are as follows : 1 . We prove that for any stochastic learning algorithm , if the algorithm leads to a well-calibrated ensemble , then the ensemble satisfies the Generalization Disagreement Equality1 ( GDE ) in expectation over the stochasticity . Notably , our theory is general and makes no restrictions on the hypothesis class , the algorithm , the source of stochasticity , or the test distributions ( which may be different from the training distribution ) . 2 . We empirically show that for Residual Networks ( He et al. , 2016 ) , convolutional neural networks ( Lin et al. , 2013 ) and fully connected networks , and on CIFAR-10/100 ( Krizhevsky et al. , 2009 ) and SVHN ( Netzer et al. , 2011 ) , GDE is nearly satisfied , even on pairs of networks trained on the same data with different random seeds . This yields a simple method that in practice accurately estimates the test error using unlabeled data in these settings . We also empirically show that the corresponding ensembles are well-calibrated ( according to our particular definition of calibration ) in practice . We do not , however , theoretically prove why this calibration holds . 3 . We present preliminary observations showing that GDE is approximately satisfied even for certain distribution shifts within the PACS ( Li et al. , 2017 ) dataset . This implies that the disagreement rate can be a promising estimator even for out-of-distribution accuracy . 4 . We empirically find that different sources of stochasticity in SGD are almost equally effective in terms of their effect on GDE and calibration of deep models trained with SGD . We also explore the effect of pre-training on these phenomena . 1Nakkiran & Bansal ( 2020 ) refer to this as the Agreement Property , but we use the term Generalization Disagreement Equality to be more explicit and to avoid confusion regarding certain technical differences . 2 RELATED WORKS . Understanding and predicting generalization . Conventionally , generalization in deep learning has been studied through the lens of PAC-learning ( Vapnik , 1971 ; Valiant , 1984 ) . Under this framework , generalization is roughly equivalent to bounding the size of the search space of a learning algorithm . Representative works in this large area of research include Neyshabur et al . ( 2014 ; 2017 ; 2018 ) ; Dziugaite & Roy ( 2017 ) ; Bartlett et al . ( 2017 ) ; Nagarajan & Kolter ( 2019b ; c ) ; Krishnan et al . ( 2019 ) . Several works have questioned whether these approaches are truly making progress toward understanding generalization in overparameterized settings ( Belkin et al. , 2018 ; Nagarajan & Kolter , 2019a ; Jiang et al. , 2020b ; Dziugaite et al. , 2020 ) . Subsequently , recent works have proposed unconventional ways to derive generalization bounds ( Negrea et al. , 2020 ; Zhou et al. , 2020 ; Garg et al. , 2021 ) . Indeed , even our disagreement-based estimate marks a significant departure from complexity-based approaches to generalization bounds . Of particular relevance here is Garg et al . ( 2021 ) who also leverage unlabeled data . Their bound requires modifying the original training set and then performing a careful early stopping , and is thus inapplicable to ( and becomes vacuous for ) interpolating models . While our estimate applies to the original training process , our guarantee applies only if we know a priori that the training procedure results in well-calibrated ensembles . There have been a few other works in unsupervised accuracy estimation such as Donmez et al . ( 2010 ) ; Platanios et al . ( 2017 ) ; Jaffe et al . ( 2015 ) ; Steinhardt & Liang ( 2016 ) ; ElSahar & Gallé ( 2019 ) ; Schelter et al . ( 2020 ) ; Chuang et al . ( 2020 ) . Since these works focus on out-of-distribution and/or specialized learning settings , they often require extra information about the tasks or specialized training algorithms . In a concurrent work , Chen et al . ( 2021 ) make similar discoveries regarding estimating accuracy via agreement by developing a sophisticated ensemble-learning algorithm and make connections to calibration . Overall , their focus is more algorithmic , while our focus is primarily on understanding the nature of this phenomenon . Furthermore , in comparison to all these works which focus on out-of-distribution settings , since we focus on the in-distribution setting , we are able to identify a much simpler approach that works with vanilla-trained blackbox models . Calibration . Calibration of a statistical model is the property that the probability obtained by the model reflects the true likelihood of the ground truth ( Murphy & Epstein , 1967 ; Dawid , 1982 ) . A well-calibrated model provides an accurate confidence on its prediction which is paramount for high-stake decision making and interpretability . In the context of deep learning , several works ( Guo et al. , 2017 ; Lakshminarayanan et al. , 2017 ; Fort et al. , 2019 ; Wu & Gales , 2021 ; Bai et al. , 2021 ; Mukhoti et al. , 2021 ) have found that while individual neural networks are usually over-confident about their predictions , ensembles of several independently and stochastically trained models tend to be naturally well-calibrated . In particular , there are two types of ensembles that have typically been studied : ( a ) ensembles where each member is trained by independently sampling training data ( with replacement ) , also called bagging ( Breiman , 1996 ) and ( b ) ensembles where each member is trained on the same data , but with different random seeds ( e.g. , different random initialization and data ordering ) , also called deep ensembles ( Lakshminarayanan et al. , 2017 ) . The latter typically achieves better accuracy and calibration ( Nixon et al. , 2020 ) . On the theoretical side , Allen-Zhu & Li ( 2020 ) have studied why deep ensembles outperform individual models in terms of accuracy . Other works studied post-processing methods of calibration ( Kumar et al. , 2019 ) , established relationships to confidence intervals ( Gupta et al. , 2020 ) , and derived upper bounds on calibration error either in terms of sample complexity or in terms of the accuracy ( Bai et al. , 2021 ; Ji et al. , 2021 ; Liu et al. , 2019 ; Jung et al. , 2020 ; Shabat et al. , 2020 ) . The discussion in our paper complements the above works in multiple ways . First , most works within the machine learning literature focus on top-class calibration , which is concerned only with the confidence level of the top predicted class for each point . The theory in our work , however , requires looking at the confidence level of the model aggregated over all the classes . We then empirically show that SGD ensembles are well-calibrated even in this class-aggregated sense . Furthermore , we carefully investigate what sources of stochasticity result in well-calibrated ensembles . Finally , we provide an exact formal relationship between generalization and calibration via the notion of disagreement , which is fundamentally different from existing theoretical calibration bounds . Empirical phenomena in deep learning . Broadly , our work falls in the area of research on identifying & understanding empirical phenomena in deep learning ( Sedghi et al. , 2019 ) , especially in the context of overparameterized models that interpolate . Some example phenomena include the generalization puzzle ( Zhang et al. , 2017 ; Neyshabur et al. , 2014 ) , double descent ( Belkin et al. , 2019 ; Nakkiran et al. , 2020 ) , and simplicity bias ( Kalimeris et al. , 2019 ; Arpit et al. , 2017 ) . As stated earlier , we build on N & B ’ 20 ’ s empirical observation of the Generalization Disagreement Equality ( GDE ) in pairs of models trained on independently drawn datasets . Here we provide a detailed discussion of how our results are distinct from and/or complement their other relevant findings . First , N & B ’ 20 provide a proof of GDE specific to 1-Nearest Neighbor models trained on two independent datasets . Our result does not restrict the hypothesis class , the algorithm or its stochasticity . Second , N & B ’ 20 identify a notion they term as “ feature calibration ” which can be thought of as a generalized version of calibration . However , the instantiations of feature calibration that they empirically study are significantly different from the standard notion we study . Furthermore , they treat GDE and feature calibration as independent phenomena . Conversely , we show that calibration in the standard sense implies GDE . Finally , in their Appendix D.7.1 , N & B ’ 20 do report studies of ensembles where the members are trained on the same data . But this is in an altogether independent context — the GDE-related experiments in N & B ’ 20 are all reported only on ensembles of members trained on different data . Hence , overall , their empirical results do not imply our GDE results .

Summary: This paper builds upon and extends from Nakkiran/Bansal (2020). First, it shows empirically that models learned from two independent runs of SGD on the same training set have their prediction disagreement highly correlated (and nearly equal) to the test error of the models.  It then attempts a theoretical explanation for this phenomenon. Under two notions of "calibration", the paper proves that when the learning algorithm is well-calibrated, a "generalization disagreement equality" holds (i.e., disagreement in prediction is equal to the test error in expectation).  The paper performs further experiments and provides empirical evidence suggesting that SGD is well calibrated.  The paper also proves that the gap between the test error and the prediction disagreement is upper bounded by a calibration error.