The Deep Bootstrap Framework: Good Online Learners are Good Offline Generalizers

1 INTRODUCTION . The goal of a generalization theory in supervised learning is to understand when and why trained models have small test error . The classical framework of generalization decomposes the test error of a model ft as : TestError ( ft ) = TrainError ( ft ) + [ TestError ( ft ) − TrainError ( ft ) ] ︸ ︷︷ ︸ Generalization gap ( 1 ) and studies each part separately ( e.g . Vapnik and Chervonenkis ( 1971 ) ; Blumer et al . ( 1989 ) ; ShalevShwartz and Ben-David ( 2014 ) ) . Many works have applied this framework to study generalization of deep networks ( e.g . Bartlett ( 1997 ) ; Bartlett et al . ( 1999 ) ; Bartlett and Mendelson ( 2002 ) ; Anthony and Bartlett ( 2009 ) ; Neyshabur et al . ( 2015b ) ; Dziugaite and Roy ( 2017 ) ; Bartlett et al . ( 2017 ) ; Neyshabur et al . ( 2017 ) ; Harvey et al . ( 2017 ) ; Golowich et al . ( 2018 ) ; Arora et al . ( 2018 ; 2019 ) ; Allen-Zhu et al . ( 2019 ) ; Long and Sedghi ( 2019 ) ; Wei and Ma ( 2019 ) ) . However , there are at least two obstacles to understanding generalization of modern neural networks via the classical approach . 1 . Modern methods can interpolate , reaching TrainError ≈ 0 , while still performing well . In these settings , the decomposition of Equation ( 1 ) does not actually reduce test error into two different subproblems : it amounts to writing TestError = 0 + TestError . That is , understanding the generalization gap here is exactly equivalent to understanding the test error itself . 2 . Most if not all techniques for understanding the generalization gap ( e.g . uniform convergence , VC-dimension , regularization , stability , margins ) remain vacuous ( Zhang et al. , 2017 ; Belkin et al. , 2018a ; b ; Nagarajan and Kolter , 2019 ) and not predictive ( Nagarajan and Kolter , 2019 ; Jiang et al. , 2019 ; Dziugaite et al. , 2020 ) for modern networks . In this work , we propose an alternate approach to understanding generalization to help overcome these obstacles . The key idea is to consider an alternate decomposition : TestError ( ft ) = TestError ( f iidt ) ︸ ︷︷ ︸ A : Online Learning + [ TestError ( ft ) − TestError ( f iidt ) ] ︸ ︷︷ ︸ B : Bootstrap error ( 2 ) where ft is the neural-network after t optimization steps ( the “ Real World ” ) , and f iidt is a network trained identically to ft , but using fresh samples from the distribution in each mini-batch step ( the “ Ideal World ” ) . That is , f iidt is the result of optimizing on the population loss for t steps , while ft is the result of optimizing on the empirical loss as usual ( we define this more formally later ) . This leads to a different decoupling of concerns , and proposes an alternate research agenda to understand generalization . To understand generalization in the bootstrap framework , it is sufficient to understand : ( A ) Online Learning : How quickly models optimize on the population loss , in the infinite-data regime ( the Ideal World ) . ( B ) Finite-Sample Deviations : How closely models behave in the finite-data vs. infinite-data regime ( the bootstrap error ) . Although neither of these points are theoretically understood for deep networks , they are closely related to rich areas in optimization and statistics , whose tools have not been brought fully to bear on the problem of generalization . The first part ( A ) is purely a question in online stochastic optimization : We have access to a stochastic gradient oracle for a population loss function , and we are interested in how quickly an online optimization algorithm ( e.g . SGD , Adam ) reaches small population loss . This problem is well-studied in the online learning literature for convex functions ( Bubeck , 2011 ; Hazan , 2019 ; Shalev-Shwartz et al. , 2011 ) , and is an active area of research in nonconvex settings ( Jin et al. , 2017 ; Lee et al. , 2016 ; Jain and Kar , 2017 ; Gao et al. , 2018 ; Yang et al. , 2018 ; Maillard and Munos , 2010 ) . In the context of neural networks , optimization is usually studied on the empirical loss landscape ( Arora et al. , 2019 ; Allen-Zhu et al. , 2019 ) , but we propose studying optimization on the population loss landscape directly . This highlights a key difference in our approach : we never compare test and train quantities— we only consider test quantities . The second part ( B ) involves approximating fresh samples with “ reused ” samples , and reasoning about behavior of certain functions under this approximation . This is closely related to the nonparametric bootstrap in statistics ( Efron , 1979 ; Efron and Tibshirani , 1986 ) , where sampling from the population distribution is approximated by sampling with replacement from an empirical distribution . Bootstrapped estimators are widely used in applied statistics , and their theoretical properties are known in certain cases ( e.g . Hastie et al . ( 2009 ) ; James et al . ( 2013 ) ; Efron and Hastie ( 2016 ) ; Van der Vaart ( 2000 ) ) . Although current bootstrap theory does not apply to neural networks , it is conceivable that these tools could eventually be extended to our setting . Experimental Validation . Beyond the theoretical motivation , our main experimental claim is that the bootstrap decomposition is actually useful : in realistic settings , the bootstrap error is often small , and the performance of real classifiers is largely captured by their performance in the Ideal World . Figure 1 shows one example of this , as a preview of our more extensive experiments in Section 4 . We plot the test error of a ResNet ( He et al. , 2016a ) , an MLP , and a Vision Transformer ( Dosovitskiy et al. , 2020 ) on a CIFAR-10-like task , over increasing minibatch SGD iterations . The Real World is trained on 50K samples for 100 epochs . The Ideal World is trained on 5 million samples with a single pass . Notice that the bootstrap error is small for all architectures , although the generalization gap can be large . In particular , the convnet generalizes better than the MLP on finite data , but this is “ because ” it optimizes faster on the population loss with infinite data . See Appendix D.1 for details . Our Contributions .. • Framework : We propose the Deep Bootstrap framework for understanding generalization in deep learning , which connects offline generalization to online optimization . ( Section 2 ) . • Validation : We give evidence that the bootstrap error is small in realistic settings for supervised image classification , by conducting extensive experiments on large-scale tasks ( including variants of CIFAR-10 and ImageNet ) for many architectures ( Section 4 ) . Thus , The generalization of models is largely determined by their optimization speed in online and offline learning . • Implications : We highlight how our framework can unify and yield insight into important phenomena in deep learning , including implicit bias , model selection , data-augmentation and pretraining ( Section 5 ) . In particular : Good models and training procedures are those which ( 1 ) optimize quickly in the Ideal World and ( 2 ) do not optimize too quickly in the Real World . Additional Related Work . The bootstrap error is also related to algorithmic stability ( e.g . Bousquet and Elisseeff ( 2001 ) ; Hardt et al . ( 2016 ) ) , since both quantities involve replacing samples with fresh samples . However , stability-based generalization bounds can not tightly bound the bootstrap error , since there are many settings where the generalization gap is high , but bootstrap error is low . 2 THE DEEP BOOSTRAP . Here we more formally describe the Deep Bootstrap framework and our main claims . LetF denote a learning algorithm , including architecture and optimizer . We consider optimizers which can be used in online learning , such as stochastic gradient descent and variants . Let TrainF ( D , n , t ) denote training in the “ Real World ” : using the architecture and optimizer specified by F , on a train set of n samples from distribution D , for t optimizer steps . Let TrainF ( D , ∞ , t ) denote this same optimizer operating on the population loss ( the “ Ideal World ” ) . Note that these two procedures use identical architectures , learning-rate schedules , mini-batch size , etc – the only difference is , the Ideal World optimizer sees a fresh minibatch of samples in each optimization step , while the Real World reuses samples in minibatches . Let the Real and Ideal World trained models be : Real World : ft ← TrainF ( D , n , t ) Ideal World : f iidt ← TrainF ( D , ∞ , t ) We now claim that for all t until the Real World converges , these two models ft , f iidt have similar test performance . In our main claims , we differ slightly from the presentation in the Introduction in that we consider the “ soft-error ” of classifiers instead of their hard-errors . The soft-accuracy of classifiers is defined as the softmax probability on the correct label , and ( soft-error ) : = 1 − ( soft-accuracy ) . Equivalently , this is the expected error of temperature-1 samples from the softmax distribution . Formally , define ε as the bootstrap error – the gap in soft-error between Real and Ideal worlds at time t : TestSoftErrorD ( ft ) = TestSoftErrorD ( f iidt ) + ε ( n , D , F , t ) ( 3 ) Our main experimental claim is that the bootstrap error ε is uniformly small in realistic settings . Claim 1 ( Bootstrap Error Bound , informal ) For choices of ( n , D , F ) corresponding to realistic settings in deep learning for supervised image classification , the bootstrap error ε ( n , D , F , t ) is small for all t ≤ T0 . The “ stopping time ” T0 is defined as the time when the Real World reaches small training error ( we use 1 % ) – that is , when Real World training has essentially converged . The restriction on t ≤ T0 is necessary , since as t→∞ the Ideal World will continue to improve , but the Real World will at some point essentially stop changing ( when train error ≈ 0 ) . However , we claim that these worlds are close for “ as long as we can hope ” — as long as the Real World optimizer is still moving significantly . Error vs. Soft-Error . We chose to measure soft-error instead of hard-error in our framework for both empirical and theoretically-motivated reasons . Empirically , we found that the bootstrap gap is often smaller with respect to soft-errors . Theoretically , we want to define the bootstrap gap such that it converges to 0 as data and model size are scaled to infinity . Specifically , if we consider an overparameterized scaling limit where the Real World models always interpolate the train data , then Distributional Generalization ( Nakkiran and Bansal , 2020 ) implies that the bootstrap gap for test error will not converge to 0 on distributions with non-zero Bayes risk . Roughly , this is because the Ideal World classifier will converge to the Bayes optimal one ( argmaxy p ( y|x ) ) , while the Real World interpolating classifier will converge to a sampler from p ( y|x ) . Considering soft-errors instead of errors nullifies this issue . We elaborate further on the differences between the worlds in Section 6 . See also Appendix C for relations to the nonparametric bootstrap ( Efron , 1979 ) .

Summary: The authors propose a bootstrap framework for understanding generalization in deep learning.  In particular, instead of the usual decomposition of test error as training error plus the generalization gap, the bootstrap framework decomposes the empirical test error as online error plus the bootstrap error (the gap between the population and empirical error).  The authors then demonstrate empirically on variants of CIFAR10 and a subset of ImageNet that the bootstrap error is small on several common architectures.  Hence, the empirical test error is controlled by the online error (i.e. a rapid decrease in the error in the online setting leads to low test error).  The authors then provide empirical evidence to demonstrate that same techniques perform well in both over and under-parameterized regimes.