Generalization Through the Lens of Leave-One-Out Error

1 INTRODUCTION . Neural networks have achieved astonishing performance across many learning tasks such as in computer vision ( He et al. , 2016 ) , natural language processing ( Devlin et al. , 2019 ) and graph learning ( Kipf & Welling , 2017 ) among many others . Despite the large overparametrized nature of these models , they have shown a surprising ability to generalize well to unseen data . The theoretical understanding of this generalization ability has been an active area of research where contributions have been made using diverse analytical tools ( Arora et al. , 2018 ; Bartlett et al. , 2019 ; 2017 ; Neyshabur et al. , 2015 ; 2018 ) . Yet , the theoretical bounds derived by these approaches typically suffer from one or several of the following limitations : i ) they are known to be loose or even vacuous ( Jiang et al. , 2019 ) , ii ) they only apply to randomized networks ( Dziugaite & Roy , 2018 ; 2017 ) , and iii ) they rely on the concept of uniform convergence which was shown to be non-robust against adversarial perturbations ( Nagarajan & Kolter , 2019 ) . In this work , we revisit the concept of leave-one-out error ( LOO ) and demonstrate its surprising ability to predict generalization for deep neural networks in the so-called kernel regime . This object is an important statistical estimator of the generalization performance of an algorithm that is theoretically motivated by a connection to the concept of uniform stability ( Pontil , 2002 ) . It has also recently been advocated by Nagarajan & Kolter ( 2019 ) as a potential substitute for uniform convergence bounds . Despite being popular in classical statistics ( Cawley & Talbot , 2003 ; Vehtari et al. , 2016 ; Fukunaga & Hummels , 1989 ; Zhang , 2003 ) , LOO has largely been overlooked in the context of deep learning , likely due to its high computational cost . However , recent advances from the neural tangent kernel perspective ( Jacot et al. , 2018 ) render the LOO error all of a sudden tractable thanks to the availability of a closed-form expression due to Stone ( 1974 , Eq . 3.13 ) . While the role of the LOO error as an estimator of the generalization performance is debated in the statistics community ( Zhang & Yang , 2015 ; Bengio & Grandvalet , 2004 ; Breiman , 1996 ; Kearns & Ron , 1999 ) , the recent work by Patil et al . ( 2021 ) established a consistency result in the case of ridge regression , ensuring that LOO converges to the generalization error in the large sample limit , under mild assumptions on the data distribution . Inspired by these recent advances , in this work , we investigate the use of LOO as a generalization measure for deep neural networks in the kernel regime . We find that LOO is a surprisingly rich descriptor of the generalization error , capturing a wide range of phenomena such as random label fitting , double descent and transfer learning . Specifically , we make the following contributions : • We extend the LOO expression for the multi-class setting to the case of zero regularization and derive a new closed-form formula for the resulting LOO accuracy . • We investigate both the LOO loss and accuracy in the context of deep learning through the lens of the NTK , and demonstrate empirically that they both capture the generalization ability of networks in the kernel regime in a variety of settings . • We showcase the utility of LOO for practical networks by accurately predicting their transfer learning performance . • We build on the mathematically convenient form of the LOO loss to derive some novel insights into double descent and the role of regularization . Our work is structured as follows . We give an overview of related works in Section 2 . In Section 3 we introduce the setting along with the LOO error and showcase our extensions to the multi-class case and LOO accuracy . Then , in Section 4 we analyze the predictive power of LOO in various settings and present our theoretical results on double descent . We discuss our findings and future work in Section 5 . We release the code for our numerical experiments on Github1 . 2 RELATED WORK . Originally introduced by Lachenbruch ( 1967 ) ; Mosteller & Tukey ( 1968 ) , the leave-one-out error is a standard tool in the field of statistics that has been used to study and improve a variety of models , ranging from support vector machines ( Weston , 1999 ) , Fisher discriminant analysis ( Cawley & Talbot , 2003 ) , linear regression ( Hastie et al. , 2020 ; Patil et al. , 2021 ) to non-parametric density estimation ( Fukunaga & Hummels , 1989 ) . Various theoretical perspectives have justified the use of the LOO error , including for instance the framework of uniform stability ( Bousquet & Elisseeff , 2002 ) or generalization bounds for KNN models ( Devroye & Wagner , 1979 ) . Elisseeff et al . ( 2003 ) focused on the stability of a learning algorithm to demonstrate a formal link between LOO error and generalization . The vanilla definition of the LOO requires access to a large number of trained models which is typically computationally expensive . Crucially , Stone ( 1974 ) derived an elegant closed-form formula for kernels , completely removing the need for training multiple models . In the context of deep learning , the LOO error remains largely unexplored . Shekkizhar & Ortega ( 2020 ) propose to fit polytope interpolators to a given neural network and estimate its resulting LOO error . Alaa & van der Schaar ( 2020 ) rely on a Jackknife estimator , applied in a LOO fashion to obtain better confidence intervals for Bayesian neural networks . Finally , we want to highlight the concurrent work of Zhang & Zhang ( 2021 ) who analyze the related concept of influence functions in the context of deep models . Recent works have established a connection between kernel regression and deep learning . Surprisingly , it is shown that an infinitely-wide , fully-connected network behaves like a kernel both at initialization ( Neal , 1996 ; Lee et al. , 2018 ; de G. Matthews et al. , 2018 ) as well as during gradient flow training ( Jacot et al. , 2018 ) . Follow-up works have extended this result to the non-asymptotic setting ( Arora et al. , 2019a ; Lee et al. , 2019 ) , proving that wide enough networks trained with small learning rates are in the so-called kernel regime , i.e . essentially evolving like their corresponding tangent kernel . Moreover , the analysis has also been adapted to various architectures ( Arora et al. , 2019a ; Huang et al. , 2020 ; Du et al. , 2019 ; Hron et al. , 2020 ; Yang , 2020 ) as well as discrete gradient descent ( Lee et al. , 2019 ) . The direct connection to the field of kernel regression enables the usage of the closed-form expression for LOO error in the context of deep learning , which to the best of our knowledge has not been explored yet . 1https : //github.com/gregorbachmann/LeaveOneOut 3 SETTING AND BACKGROUND . In the following , we establish the notations required to describe the problem of interest and the setting we consider . We study the standard learning setting , where we are given a dataset of inputtarget pairs S = { ( x1 , y1 ) , . . . , ( xn , yn ) } where each ( xi , yi ) i.i.d.∼ D for i = 1 , . . . , n is distributed according to some probability measure D. We refer to xi ∈ X ⊂ Rd as the input and to y ∈ Y ⊂ RC as the target . We will often use matrix notations to obtain more compact formulations , thus summarizing all inputs and targets as matrices X ∈ Rn×d and Y ∈ Rn×C . We will mainly be concerned with the task of multiclass classification , where targets yi are encoded as one-hot vectors . We consider a function space F and an associated loss function Lf : X × Y −→ R that measures how well a given function f ∈ F performs at predicting the targets . More concretely , we define the regularized empirical error as L̂λS : F −→ R , f 7→ 1 n n∑ i=1 Lf ( xi , yi ) + λΩ ( f ) , where λ > 0 is a regularization parameter and Ω : F −→ R is a functional . For any vector v ∈ RC , let v∗ = argmaxk≤C vk . Using this notation , we let af ( x , y ) : = 1 { f∗ ( x ) =y∗ } . We then define the empirical accuracy ÂS as the average number of instances for which f ∈ F provides the correct prediction , i.e . ÂS : F −→ [ 0 , 1 ] , f 7→ 1 n n∑ i=1 af ( xi , yi ) Finally , we introduce a learning algorithm QF : ( X × Y ) n −→ F , that given a function class F and a training set S ∈ ( X × Y ) n , chooses a function f ∈ F . We will exclusively be concerned with learning through empirical risk minimization , i.e . QλF ( S ) : = f̂λS : = argminf∈F L̂λS ( f ) , and use the shortcut f̂S : = f̂λ=0S . In practice however , the most important measures are given by the generalization loss and accuracy of the model , defined as L : F −→ R , f 7→ E ( x , y ) ∼D [ Lf ( x , y ) ] , A : F −→ [ 0 , 1 ] , f 7→ P ( x , y ) ∼D ( f∗ ( x ) = y∗ ) . A central goal in machine learning is to control the difference between the empirical error L̂S ( f ) and the generalization error L ( f ) . In this work , we mainly focus on the mean squared loss , Lf ( x , y ) = 1 2 C∑ k=1 ( fk ( x ) − yk ) 2 , and also treat classification as a regression task , as commonly done in the literature ( Lee et al. , 2018 ; Chen et al. , 2020 ; Arora et al. , 2019a ; Hui & Belkin , 2021 ; Bachmann et al. , 2021 ) . Finally , we consider the function spaces induced by kernels and neural networks , presented in the following . 3.1 KERNEL LEARNING . A kernel K : X × X −→ R is a symmetric , positive semi-definite function , i.e . K ( x , x′ ) = K ( x′ , x ) ∀ x , x′ ∈ X and for any set { x1 , . . . , xn } ⊂ X , the matrix K ∈ Rn×n with entries Kij = K ( xi , xj ) is positive semi-definite . It can be shown that a kernel induces a reproducing kernel Hilbert space , a function space equipped with an inner product 〈· , ·〉H , containing elements H = cl ( { f : fk ( · ) = n∑ i=1 αikK ( · , xi ) for α ∈ Rn×C , xi ∈ X } ) where cl is the closure operator . Although H is an infinite dimensional space , it turns out that the minimizer f̂λS can be obtained in closed form for mean squared loss , given by f̂λS ( x ) = K T x ( K + λ1n ) −1 Y λ−→0−−−→ f̂S ( x ) : = KTxK†Y where the regularization functional is induced by the inner product of the space , Ω ( f ) = ||f ||2H , K ∈ Rn×n has entriesKij = K ( xi , xj ) andKx ∈ Rn has entries ( Kx ) i = K ( x , xi ) . A† denotes the pseudo-inverse of A ∈ Rn×n . We will analyze both the regularized ( f̂λS ) and unregularized model ( f̂S ) and their associated LOO errors in the following . For a more in-depth discussion of kernels , we refer the reader to Hofmann et al . ( 2008 ) ; Paulsen & Raghupathi ( 2016 ) .

Summary: The manuscript considers the use leave-one-out error to analyse the generalization ability of one-versus-all multi-class classification models that can be expressed as multiple-output ridge regression models. As examples of such models, the authors review certain variations of neural networks. Specific attention is given on the double descent phenomenon showing that the traditionally considered U-shaped curve demonstrating the trade-off between underfitting and overfitting areas can be extended over the interpolation point after which the generalization ability again tends to improve. The spiking effect of the generalization error is derived from the closed form solutions of the leave-one-out computations for ridge regression based methods.