Short optimization paths lead to good generalization

1 INTRODUCTION . From the perspective of statistical learning theory , the goal of machine learning is to find a predictive function that can give accurate predictions on new data . For supervised learning problems , empirical risk minimization ( ERM ) is a common practice to achieve this goal . The idea of ERM is to minimize a cost function on observed data by an optimization algorithm . Therefore , a fundamental question is whether an optimization algorithm produces a solution with good generalization . Recent works have shed light on providing theoretical explanations to this question from different angles . One line of works considered the case when gradient methods converge to minimal norm solutions ( Bartlett et al. , 2020 ; Tsigler & Bartlett , 2020 ; Liang & Rakhlin , 2020 ; Liang et al. , 2020 ) on kernel regression , and then analyzed the generalization of those minimal norm solutions . However , the phenomenon of norm minimization has been known to happen only for the quadratic loss with an appropriate initialization . Another line of works focused on overparameterized models , e.g. , neural networks under the Neural Tangent Kernel ( NTK ) regime ( Allen-Zhu et al. , 2019 ; Arora et al. , 2019 ; Cao & Gu , 2020 ; Ji & Telgarsky , 2020 ; Chen et al. , 2021 ) , proving that overparameterized neural networks trained by ( stochastic ) gradient descent ( ( S ) GD ) have good generalization performance on certain target functions ( for example , polynomial functions ) . Although existing works have made significant progress on the interplay of optimization and generalization , they focused on studying specific models , such as the NTK and models possessing minimal norm solutions . In this paper , instead , we study general loss function conditions that induce interesting connections between optimization and generalization . We start with a simple observation , as shown in Figure 1 , that under a generic random initialization , the generalization performance for both linear regression model and random feature regression model is closely related to the length of the optimization path1 after convergence . In particular , short optimization path are associated with good generalization . Here , by length we mean the trajectory or path length of the parameter evolution during training . This is not the “ length of time ” used for training , as is usually analyzed in early-stopping type of algorithms . Thus , the generalization error concerns the weights of model trained to completion by empirical risk minimization . This empirical investigation motivates us to use the trajectory length to connect optimization and generalization . Intuitively , the length of the optimization path can be viewed as a kind of capacity control , and a 1For discrete iterations , we use the sum of the distance between every two consecutive iterations to represent the length of the optimization path ( See details in Appendix A.1 ) . short path signifies low “ complexity ” . In other words , a general condition that guarantees a short optimization path can be used to induce good generalization performance . Thus , inspired by the theory in ( Bolte et al. , 2007 ) that Łojasiewicz gradient inequality ( LGI ) induces an explicit bound for the gradient flow path , we consider Uniform-LGI ( Definition 1 ) . This is a modified version of LGI and plays a critical role in obtaining a length estimate for the gradient flow trajectory . Once the length is estimated , we can use our length-based generalization bounds to show the generalization performance . Contributions . We summarize our contributions as follows : • We focus on the gradient flow algorithm and propose a framework for combining optimization and generalization . This framework is based on the length of the gradient flow trajectory , and its key component is the Uniform-LGI property . • We then prove that under appropriate conditions , gradient flow returns a global minimum and gives an explicit length-estimate for the gradient flow trajectory ( Theorem 1 ) . If this length-estimate has an uniform bound for a certain initialization method , then we can give a length-based generalization bound ( Theorem 2 ) . • We further show applications of Theorem 1 and Theorem 2 to obtain generalization bounds on underdetermined ℓp linear regression ( Theorem 3 ) , kernel regression ( Theorem 4 ) , and overparameterized two-layer ReLU neural networks ( Theorem 5 ) . These bounds match or expand the type of scenarios where we can rigorously establish the phenomenon of benign overfitting . 2 RELATED WORKS . Optimization . Theoretically analyzing the training process of most machine learning models is a challenging problem as the loss landscapes are highly non-convex . One approach to studying non-convex optimization problems is to use the Polyak-Łojasiewicz ( PL ) condition ( Polyak , 1963 ) , which characterizes the local geometry of loss landscapes and ensures the existence of global minima . It is shown in ( Karimi et al. , 2016 ) that GD admits linear convergence for a class of optimization objective functions under the PL condition . In this paper , we consider the Łojasiewicz gradient inequality ( Lojasiewicz , 1965 ) , an extended version of the PL condition that can be applied to more cases . In particular , we modify the original LGI to get length estimates of the optimization paths , which will be used to derive length-based generalization bounds . Generalization . Traditional VC dimension-based generalization bounds depend on the number of parameters and will be vacuous for huge models such as overparameterized neural networks . To overcome this limitation , several non-vacuous generalization bounds are proposed . For example , the norm/margin-based generalization bounds ( Neyshabur et al. , 2015 ; Bartlett et al. , 2017 ; Golowich et al. , 2018 ; Neyshabur et al. , 2019 ) , and the PAC-Bayes-based bounds ( Dziugaite & Roy , 2017 ; Neyshabur et al. , 2018 ; Zhou et al. , 2019 ; Rivasplata et al. , 2020 ) . However , these bounds tend to focus less on optimization , e.g. , norm-based generalization bounds may not discuss how small-norm solutions are obtained through practical training . In this paper , we connect the optimization and generalization by deriving generalization bounds based on the optimization trajectory length . Interface between optimization and generalization . Implicit bias builds the bridge between optimization and generalization , which has been widely studied to explain the generalization ability of machine learning models . Recent works ( Soudry et al. , 2018a ; b ; Nacson et al. , 2019a ; b ; Lyu & Li , 2020 ) showed that linear classifiers or deep neural networks trained by GD/SGD maximizes the margin of the separating hyperplanes and therefore generalizes well . Other works ( Arora et al. , 2019 ; Zou & Gu , 2019 ; Cao & Gu , 2020 ; Ji & Telgarsky , 2020 ; Chen et al. , 2021 ) concentrated on overparameterized neural networks , showing that GD/SGD trajectories fall within the NTK regime , where the minimizer has good generalization due to the small “ complexity ” of the parameter space . In this work , we focus on specific conditions on loss functions under which we can connect optimization and generalization based on the lengths of optimization paths . 3 MAIN RESULTS . In this section , we aim to address optimization and generalization together by proving results based on the Uniform-LGI property . We begin with describing notations and the problem setting in Section 3.1 . Then we give an explicit length estimate for the gradient flow trajectory in Section 3.2 . Lastly , we show a length-based generalization bound in Section 3.3 . 3.1 SETUP AND NOTATIONS . Consider a supervised learning problem on a hypothesis space F = { f ( w , · ) : Rd −→ R | w ∈ W } , where W is a parameter set in Euclidean space . Given a loss function ℓ : R× R −→ R , and a training set S = { ( xi , yi ) } ni=1 ⊆ Rd × R with n independent and identically distributed ( i.i.d . ) samples from a joint distribution D , the goal of ERM is to optimize the empirical loss function Ln ( w ) on S : argmin w Ln ( w ) : = 1 n n∑ i=1 ℓ ( f ( w , xi ) , yi ) . We assume that each input vector and label have bounded norms . Specifically , we assume that , for any ( x , y ) following D , ∥x∥ = 1 , |y| ≤ 1 . This can be achieved by data normalization . To simplify the analysis , we optimize the empirical loss by gradient flow2 : dw ( t ) dt = −∇Ln ( w ( t ) ) , t ∈ [ 0 , +∞ ) , ( 1 ) where w ( t ) is the parameter value at time t , w ( 0 ) is the initialization parameter set , and ∇Ln ( w ) is the gradient of Ln ( w ) with respect to w. Given the gradient flow , we may define the length of the gradient flow curve from w ( 0 ) to w ( t ) as len ( w ( 0 ) , t ) : = ∫ t 0 ∥∥∥dw ( s ) ds ∥∥∥ ds . Notations . We use ∥·∥ to denote the ℓ2 norm of a vector or the spectral norm of a matrix , and use ∥·∥F to denote the Fronenius norm of a matrix . For a set S ∈ Rn , we use ∂S to denote its boundary . We use d ( S1 , S2 ) to represent the Euclidean distance between two sets S1 , S2 , which is defined as d ( S1 , S2 ) = { inf ∥s1 − s2∥ : s1 ∈ S1 , s2 ∈ S2 } . For two vectors , we use ⟨ , ⟩ to denote their inner product . Let λmin ( A ) and λmax ( A ) be the smallest and largest eigenvalues of a symmetric matrix A . For any positive integer n , we denote [ n ] = { 1 , 2 , . . . , n } . We use O ( · ) to stand for Big-O notation . The classical LGI gives a lower bound of the gradient of a differentiable function based on its value above its minimum . Many functions , e.g. , real analytic functions and subanalytic functions , satisfy this property , at least locally ( Bolte et al. , 2007 ) . Here , we require a global version of this inequality as a condition to control the gradient flow trajectory . Let us define this notion below . Definition 1 ( Uniform-LGI ) . A function L ( w ) satisfies Uniform-LGI on a set S with two constants c > 0 and θ ∈ [ 12 , 1 ) , if ∥∇L ( w ) ∥ ≥ c ( L ( w ) −minv∈W L ( v ) ) θ , ∀w ∈ S. 2Here we assume the existence of the gradient flow and the limit limt−→∞ w ( t ) . In the special case when θ = 1/2 and c = √ 2µ , the Uniform-LGI corresponds to the µ-PL condition ( Karimi et al. , 2016 ) . Here we give examples of functions satisfying the Uniform-LGI but not the PL condition : L ( w ) = w2k , k ∈ Z+ . When k ≥ 2 , L ( w ) satisfies the Uniform-LGI on R with c = 2k and θ = 1− 1/2k .

Summary: The authors study the connection between optimization and generalization for gradient flow (GF) on loss functions that satisfy a global version of the Lojasiewicz gradient inequality. Under this assumption, they prove convergence of GF to a global minimum and they find an upper bound on the optimization length — measured as the integral of the $\ell_2-$norm of the gradient from time $t=0$ to the final time. This upper bound depends on the specific choice of the loss function and on the number of samples. With an additional assumption on the hypothesis class (encompassing, e.g., linear shallow networks, two-layer networks) the first result is used to derive an upper bound on the generalization gap. The bound depends on the choice of the loss function, its initial value, and the length of the optimisation path. This leads to the main result that shorter optimization paths induce smaller generalization gap. The authors apply this result to three models (underdetermined $\ell_p$ linear regression, kernel regression, and overparametrized two-layer networks with ReLU activation) with a given target function. They compute non-asymptotic expressions for the generalization bounds at fixed ratio between sample size and ambient dimension, and show that in these cases the bounds are non-vacuous when the dimension increases.