Directional Bias Helps Stochastic Gradient Descent to Generalize in Nonparametric Model

1 INTRODUCTION . In this paper , we study the Stochastic Gradient Descent ( SGD ) algorithm in a nonparametric regression model . Among nonparametric models , one popular choice in both statistics and machine learning communities is the kernel model that is generated by a Reproducing Kernel Hilbert Space ( RKHS ) . When fitting kernel models to the data , SGD is computationally efficient as compared to Gradient Descent ( GD ) ( Ma et al. , 2018 ) . This motivates us to analyze the properties of SGD in the kernel model , especially for SGD with a nontrivial step size for a practical reason . In particular , we aim to provide a fundamental explanation of why SGD estimators generalize well . Our work is inspired by Wu et al . ( 2021 ) , which shows that directional bias has a significant impact on the generalization property in the linear regression model . We adopt a similar concept of the directional bias , but generalize them to the nonparametric model . Our result is a non-trivial extension of their approach , due to the difference in our problem setting and our SGD algorithm design . To the best of our knowledge , we are the first to show the directional bias phenomenon of SGD and analyze how it helps generalization in nonparametric regression . Our contributions are two folded . First , we study the directional bias of ( S ) GD in a nonparametric regression model . On the one hand , nonparametric regression is well studied in both statistics and machine learning . On the other hand , the directional bias is a relatively new concept ( Wu et al. , 2021 ) of an algorithm that affects the statistics properties , and there is no thorough understanding of the directional bias of ( S ) GD algorithms for the nonparametric regression model . Note that our result is closely related to those in Belkin et al . ( 2018 ) ; Liang & Rakhlin ( 2020 ) : they prove that SGD and GD algorithms both converge to the minimum norm interpolate , where the same properties are discovered for SGD and GD ; whereas we investigate the solution paths before their convergence , and show that SGD and GD have different solution paths that lead to different properties . Our result helps to explain why the SGD generalizes better than GD . Our second contribution is to unify the conditions to show the directional bias of ( S ) GD sequences in nonparametric models . The main condition is the diagonal dominant gram matrix , which covers a large class of kernel functions and allows us to study their properties . Moreover , our SGD is different from those in Wu et al . ( 2021 ) : they define SGD in epochs while we define SGD in steps . The fundamental difference in the SGD algorithm requires us to develop different techniques for analyzing the SGD sequence and showing its directional bias , which has not yet been covered in the current literature . Main Theorems of this paper can be divided into two parts , briefly summarized as follows : First is the directional bias of SGD . Theorem 5 shows that for a two-stage SGD with a moderate step size in the first stage and a small step size in the second stage , an early-stopped estimator has a directional bias towards the eigenvector that corresponds to the largest eigenvalue of the gram matrix . Later we refer to this direction as the direction towards the largest eigenvector to simplify the statement . As a comparison , Theorem 7 shows that GD with both moderate and small step sizes has a directional bias towards the eigenvector that corresponds to the smallest eigenvalue of the gram matrix ( denote it as the direction towards the smallest eigenvector ) . From which , we conclude that SGD and GD have different direction biases in kernel regression . Second is the implication of directional bias . The implication is very useful since it quantifies the effect of the directional bias on the generalization error . Theorem 9 considers a general problem of quadratic loss . It shows that the estimator biased towards the largest eigenvector of the Hessian ( which is the gram matrix in our nonparametric regression ) can have the smallest parameter estimation error , when compared with other estimators of the same loss . With this high-level idea of directional bias helps generalization , Theorem 11 compares the generalization error of SGD and GD in our problem setting . In particular , it upper bounds the generalization error of SGD and lower bounds the generalization error of GD . By directly comparing the error bounds , we guarantee that the generalization error of the SGD estimator is smaller than that of the GD estimator with high probability . We also point out that our result might shed new light on deep learning ( Belkin et al. , 2018 ) . By the state-of-the-art mathematical theory of Neural Networks ( NN ) , kernel and/or nonparametric methods can approximate the functional space of neural networks , see for example the NTK theory ( Jacot et al. , 2018 ) , and the Radon bounded variance space description for ReLU NN ( Parhi & Nowak , 2021 ) . Our technique might allow one to characterize the SGD solution path and show the generalization property in those problem settings . Paper organization . The rest of the paper is as follows : In Section 2 , we review some relevant literature ; In Section 3 , we give the formulation of the nonparametric regression and define the optimization problem . We also formalize the algorithm that is considered in this work and make assumptions to analyze the algorithms ; In Section 4 , we state our main theory on the directional bias of SGD/GD in nonparametric regression , where we include both the directional bias result and the implication of the directional bias for generalization . Experiments are conducted to support our theory ; In Section 5 , we discuss the finding in this paper , and propose some future research topic . All the proof , experiment details , and more experiments are deferred to the appendix due to page limits . 2 LITERATURE REVIEW . In this section , we review some relevant works . For better understanding , we split into two subsections : Subsection 2.1 reviews the background of RKHS ; Subsection 2.2 presents the state-of-the-art technique for analyzing the directional bias of the ( S ) GD algorithm . 2.1 RKHS . Kernel methods are among the core algorithms in machine learning and statistics ( Bartlett et al. , 2021 ) . As proposed by Wahba ( 1990 ) , the kernel method and RKHS serve as a unified framework of a group of nonparametric models , which extends the spline method . Later , kernel models become an important component in nonparametric models . In machine learning , the kernel-based method is always referred to as the “ kernel trick ” . By lifting the x variable to a high dimensional space via the kernel method , we can explore possibly nonlinear relationships between variables . Moreover , to play the kernel trick , one can directly calculate the kernel function using original features . This is computationally efficient since we avoid calculating high dimension or infinite dimension features . For applications of kernel method in machine learning , one can see kernel regression for image processing ( Takeda et al. , 2007 ) , for text mining ( Greene & Cunningham , 2006 ) , and for tasks in bioinformatics ( Saigo et al. , 2004 ) . Regarding deep learning , the kernel method is also important because it has implications for deep learning models . On the one hand , the kernel methods have similar benign overfitting behavior to neural network due to the implicit regularization and/or implicit bias phenomenon that we will review in the next subsection ( Belkin et al. , 2018 ) . On the other hand , the RKHS itself is closely related to Neural Networks via the Neural Tangent Kernel theory ( Jacot et al. , 2018 ) . This all indicates that to understand deep learning , one should first study kernel methods . 2.2 DIRECTIONAL BIAS . This paper analyzes the directional bias of SGD for the nonparametric regression . Directional bias , also referred to as implicit bias , of an algorithm refers to that its solution path is biased towards a certain direction . It works as that the algorithm prefers some directions over the others even though they may have the same objective function value . Since the algorithm selects a direction by itself , instead of explicitly required by any constraint , people use the term “ implicit ” . It is worth noting that the implicit regularization is related to implicit bias . The implicit regularization refers to that the converged point of an algorithm is like a regularized estimator , even if the objective function is not explicitly regularized . One can also interpret implicit regularization as the “ final result ” of implicit bias . In the recent work by Derezinski et al . ( 2020 ) , implicit regularization is used to develop an exact bound for double descent in linear regression . In this way , implicit regularization/bias serves as a way to explain some deep learning phenomenons that could not be addressed by the classical empirical risk minimization ( ERM ) framework . Therefore , it is important to study directional bias . State-of-the-art study on the directional bias of first-order algorithms can be divided into two categories by the technique they use : The first category is the ( stochastic ) gradient flow method , by taking an infinitesimal step size in ( S ) GD , the parameter dynamic follows a ( stochastic ) differential equation . Studying the solution path and the stationary point of the underlying differential equation helps to reveal the property of the parameter estimation . We list some works that use the first method to show the directional bias result of the ( stochastic ) gradient descent . Liu et al . ( 2018 ) analyze the Momentum SGD ( MSGD ) with infinitesimal step size , and show that the solution path escapes from the saddle for a nonconvex objective function . It is worth noting that in their case , the associated stochastic differential equation defines a complicated stochastic process , thus they replace it with an appropriate diffusion process , and the analysis is done based on such diffusion approximation . If one analyzes a stochastic gradient flow and finds it intractable , one may consider using the technique of diffusion approximation . Ali et al . ( 2020 ) shows the stochastic gradient flow for the linear regression problem minw ‖Xw−y‖22 has a solution path close to the solution path of Ridge regression ; Blanc et al . ( 2020 ) shows the stochastic gradient flow for a general loss function has a solution path close to the solution path of gradient flow on the objective function plus some extra penalty terms , and they explicitly identify the penalty terms ; Smith et al . ( 2021 ) go one more step from the infinitesimal step size to small step size , and characterize the effect of small step size as an extra penalty term in the gradient flow . Another category is analyzing the discrete ( S ) GD sequence . This technique in general just requires a moderate step size such that the algorithm converges ( or nearly converges ) , thus it is more meaningful from a practical perspective . We also find some directional bias work that is based on this technique . Vaskevicius et al . ( 2019 ) ; Zhao et al . ( 2019 ) ; Fan et al . ( 2021 ) analyze Hadamard reparameterized GD in sparse regression . They divide the true parameter into strong , weak , and 0 parts , and for each part , they carefully develop the stepwise error bound for each step of GD . They finally show that an early-stopped estimator along the solution path achieves the minimax optimal error rate for sparse regression , which indicates that the solution path is in the direction that biased towards a sparse solution . Recently , Wu et al . ( 2021 ) show that for overparameterized linear regression , SGD with moderate step size converges to the minimum norm interpolant in the direction that corresponds to the largest eigenvalue of the design matrix , while GD converges in the direction that corresponds to the smallest eigenvalue . For Neural Networks in the ‘ lazy training ’ regime , ? shows that GD also converges in the direction of the smallest eigenvalue of the Neural Tangent Kernel . Their result further reveal the mechanism of the directional bias as : GD fits the direction of a larger eigenvalue faster at the beginning of the training , left the smaller eigenvalue direction unfitted ; later the direction of smaller eigenvalue is fitted , resulting in that the estimator goes in this direction .

Summary: The paper studies the trajectory of GD and SGD for kernel regression, and the connection to generalization. The paper shows that, under some stepsize choices, SGD has a directional bias towards sharp eigendirections of the kernel matrix. The authors link this feature to generalization using the properties of quadratics.