Short optimization paths lead to good generalization

1 INTRODUCTION . From the perspective of statistical learning theory , the goal of machine learning is to find a predictive function that can give accurate predictions on new data . For supervised learning problems , empirical risk minimization ( ERM ) is a common practice to achieve this goal . The idea of ERM is to minimize a cost function on observed data by an optimization algorithm . Therefore , a fundamental question is whether an optimization algorithm produces a solution with good generalization . Recent works have shed light on providing theoretical explanations to this question from different angles . One line of works considered the case when gradient methods converge to minimal norm solutions ( Bartlett et al. , 2020 ; Tsigler & Bartlett , 2020 ; Liang & Rakhlin , 2020 ; Liang et al. , 2020 ) on kernel regression , and then analyzed the generalization of those minimal norm solutions . However , the phenomenon of norm minimization has been known to happen only for the quadratic loss with an appropriate initialization . Another line of works focused on overparameterized models , e.g. , neural networks under the Neural Tangent Kernel ( NTK ) regime ( Allen-Zhu et al. , 2019 ; Arora et al. , 2019 ; Cao & Gu , 2020 ; Ji & Telgarsky , 2020 ; Chen et al. , 2021 ) , proving that overparameterized neural networks trained by ( stochastic ) gradient descent ( ( S ) GD ) have good generalization performance on certain target functions ( for example , polynomial functions ) . Although existing works have made significant progress on the interplay of optimization and generalization , they focused on studying specific models , such as the NTK and models possessing minimal norm solutions . In this paper , instead , we study general loss function conditions that induce interesting connections between optimization and generalization . We start with a simple observation , as shown in Figure 1 , that under a generic random initialization , the generalization performance for both linear regression model and random feature regression model is closely related to the length of the optimization path1 after convergence . In particular , short optimization path are associated with good generalization . Here , by length we mean the trajectory or path length of the parameter evolution during training . This is not the “ length of time ” used for training , as is usually analyzed in early-stopping type of algorithms . Thus , the generalization error concerns the weights of model trained to completion by empirical risk minimization . This empirical investigation motivates us to use the trajectory length to connect optimization and generalization . Intuitively , the length of the optimization path can be viewed as a kind of capacity control , and a 1For discrete iterations , we use the sum of the distance between every two consecutive iterations to represent the length of the optimization path ( See details in Appendix A.1 ) . short path signifies low “ complexity ” . In other words , a general condition that guarantees a short optimization path can be used to induce good generalization performance . Thus , inspired by the theory in ( Bolte et al. , 2007 ) that Łojasiewicz gradient inequality ( LGI ) induces an explicit bound for the gradient flow path , we consider Uniform-LGI ( Definition 1 ) . This is a modified version of LGI and plays a critical role in obtaining a length estimate for the gradient flow trajectory . Once the length is estimated , we can use our length-based generalization bounds to show the generalization performance . Contributions . We summarize our contributions as follows : • We focus on the gradient flow algorithm and propose a framework for combining optimization and generalization . This framework is based on the length of the gradient flow trajectory , and its key component is the Uniform-LGI property . • We then prove that under appropriate conditions , gradient flow returns a global minimum and gives an explicit length-estimate for the gradient flow trajectory ( Theorem 1 ) . If this length-estimate has an uniform bound for a certain initialization method , then we can give a length-based generalization bound ( Theorem 2 ) . • We further show applications of Theorem 1 and Theorem 2 to obtain generalization bounds on underdetermined ℓp linear regression ( Theorem 3 ) , kernel regression ( Theorem 4 ) , and overparameterized two-layer ReLU neural networks ( Theorem 5 ) . These bounds match or expand the type of scenarios where we can rigorously establish the phenomenon of benign overfitting . 2 RELATED WORKS . Optimization . Theoretically analyzing the training process of most machine learning models is a challenging problem as the loss landscapes are highly non-convex . One approach to studying non-convex optimization problems is to use the Polyak-Łojasiewicz ( PL ) condition ( Polyak , 1963 ) , which characterizes the local geometry of loss landscapes and ensures the existence of global minima . It is shown in ( Karimi et al. , 2016 ) that GD admits linear convergence for a class of optimization objective functions under the PL condition . In this paper , we consider the Łojasiewicz gradient inequality ( Lojasiewicz , 1965 ) , an extended version of the PL condition that can be applied to more cases . In particular , we modify the original LGI to get length estimates of the optimization paths , which will be used to derive length-based generalization bounds . Generalization . Traditional VC dimension-based generalization bounds depend on the number of parameters and will be vacuous for huge models such as overparameterized neural networks . To overcome this limitation , several non-vacuous generalization bounds are proposed . For example , the norm/margin-based generalization bounds ( Neyshabur et al. , 2015 ; Bartlett et al. , 2017 ; Golowich et al. , 2018 ; Neyshabur et al. , 2019 ) , and the PAC-Bayes-based bounds ( Dziugaite & Roy , 2017 ; Neyshabur et al. , 2018 ; Zhou et al. , 2019 ; Rivasplata et al. , 2020 ) . However , these bounds tend to focus less on optimization , e.g. , norm-based generalization bounds may not discuss how small-norm solutions are obtained through practical training . In this paper , we connect the optimization and generalization by deriving generalization bounds based on the optimization trajectory length . Interface between optimization and generalization . Implicit bias builds the bridge between optimization and generalization , which has been widely studied to explain the generalization ability of machine learning models . Recent works ( Soudry et al. , 2018a ; b ; Nacson et al. , 2019a ; b ; Lyu & Li , 2020 ) showed that linear classifiers or deep neural networks trained by GD/SGD maximizes the margin of the separating hyperplanes and therefore generalizes well . Other works ( Arora et al. , 2019 ; Zou & Gu , 2019 ; Cao & Gu , 2020 ; Ji & Telgarsky , 2020 ; Chen et al. , 2021 ) concentrated on overparameterized neural networks , showing that GD/SGD trajectories fall within the NTK regime , where the minimizer has good generalization due to the small “ complexity ” of the parameter space . In this work , we focus on specific conditions on loss functions under which we can connect optimization and generalization based on the lengths of optimization paths . 3 MAIN RESULTS . In this section , we aim to address optimization and generalization together by proving results based on the Uniform-LGI property . We begin with describing notations and the problem setting in Section 3.1 . Then we give an explicit length estimate for the gradient flow trajectory in Section 3.2 . Lastly , we show a length-based generalization bound in Section 3.3 . 3.1 SETUP AND NOTATIONS . Consider a supervised learning problem on a hypothesis space F = { f ( w , · ) : Rd −→ R | w ∈ W } , where W is a parameter set in Euclidean space . Given a loss function ℓ : R× R −→ R , and a training set S = { ( xi , yi ) } ni=1 ⊆ Rd × R with n independent and identically distributed ( i.i.d . ) samples from a joint distribution D , the goal of ERM is to optimize the empirical loss function Ln ( w ) on S : argmin w Ln ( w ) : = 1 n n∑ i=1 ℓ ( f ( w , xi ) , yi ) . We assume that each input vector and label have bounded norms . Specifically , we assume that , for any ( x , y ) following D , ∥x∥ = 1 , |y| ≤ 1 . This can be achieved by data normalization . To simplify the analysis , we optimize the empirical loss by gradient flow2 : dw ( t ) dt = −∇Ln ( w ( t ) ) , t ∈ [ 0 , +∞ ) , ( 1 ) where w ( t ) is the parameter value at time t , w ( 0 ) is the initialization parameter set , and ∇Ln ( w ) is the gradient of Ln ( w ) with respect to w. Given the gradient flow , we may define the length of the gradient flow curve from w ( 0 ) to w ( t ) as len ( w ( 0 ) , t ) : = ∫ t 0 ∥∥∥dw ( s ) ds ∥∥∥ ds . Notations . We use ∥·∥ to denote the ℓ2 norm of a vector or the spectral norm of a matrix , and use ∥·∥F to denote the Fronenius norm of a matrix . For a set S ∈ Rn , we use ∂S to denote its boundary . We use d ( S1 , S2 ) to represent the Euclidean distance between two sets S1 , S2 , which is defined as d ( S1 , S2 ) = { inf ∥s1 − s2∥ : s1 ∈ S1 , s2 ∈ S2 } . For two vectors , we use ⟨ , ⟩ to denote their inner product . Let λmin ( A ) and λmax ( A ) be the smallest and largest eigenvalues of a symmetric matrix A . For any positive integer n , we denote [ n ] = { 1 , 2 , . . . , n } . We use O ( · ) to stand for Big-O notation . The classical LGI gives a lower bound of the gradient of a differentiable function based on its value above its minimum . Many functions , e.g. , real analytic functions and subanalytic functions , satisfy this property , at least locally ( Bolte et al. , 2007 ) . Here , we require a global version of this inequality as a condition to control the gradient flow trajectory . Let us define this notion below . Definition 1 ( Uniform-LGI ) . A function L ( w ) satisfies Uniform-LGI on a set S with two constants c > 0 and θ ∈ [ 12 , 1 ) , if ∥∇L ( w ) ∥ ≥ c ( L ( w ) −minv∈W L ( v ) ) θ , ∀w ∈ S. 2Here we assume the existence of the gradient flow and the limit limt−→∞ w ( t ) . In the special case when θ = 1/2 and c = √ 2µ , the Uniform-LGI corresponds to the µ-PL condition ( Karimi et al. , 2016 ) . Here we give examples of functions satisfying the Uniform-LGI but not the PL condition : L ( w ) = w2k , k ∈ Z+ . When k ≥ 2 , L ( w ) satisfies the Uniform-LGI on R with c = 2k and θ = 1− 1/2k .

Summary: The paper studies generalization and optimization of kernel-based and one-hidden layer neural network models. The convergence guarantee is shown for a Uniform-LGI loss function in Theorem 1. Then, in Theorem 2 the paper proves a generalization guarantee assuming a particular parametric model in Equation (3). Finally, the paper applies these results to p-norm regression, kernel regression, and one hidden layer neural network learning problems.