Directional Bias Helps Stochastic Gradient Descent to Generalize in Nonparametric Model

1 INTRODUCTION . In this paper , we study the Stochastic Gradient Descent ( SGD ) algorithm in a nonparametric regression model . Among nonparametric models , one popular choice in both statistics and machine learning communities is the kernel model that is generated by a Reproducing Kernel Hilbert Space ( RKHS ) . When fitting kernel models to the data , SGD is computationally efficient as compared to Gradient Descent ( GD ) ( Ma et al. , 2018 ) . This motivates us to analyze the properties of SGD in the kernel model , especially for SGD with a nontrivial step size for a practical reason . In particular , we aim to provide a fundamental explanation of why SGD estimators generalize well . Our work is inspired by Wu et al . ( 2021 ) , which shows that directional bias has a significant impact on the generalization property in the linear regression model . We adopt a similar concept of the directional bias , but generalize them to the nonparametric model . Our result is a non-trivial extension of their approach , due to the difference in our problem setting and our SGD algorithm design . To the best of our knowledge , we are the first to show the directional bias phenomenon of SGD and analyze how it helps generalization in nonparametric regression . Our contributions are two folded . First , we study the directional bias of ( S ) GD in a nonparametric regression model . On the one hand , nonparametric regression is well studied in both statistics and machine learning . On the other hand , the directional bias is a relatively new concept ( Wu et al. , 2021 ) of an algorithm that affects the statistics properties , and there is no thorough understanding of the directional bias of ( S ) GD algorithms for the nonparametric regression model . Note that our result is closely related to those in Belkin et al . ( 2018 ) ; Liang & Rakhlin ( 2020 ) : they prove that SGD and GD algorithms both converge to the minimum norm interpolate , where the same properties are discovered for SGD and GD ; whereas we investigate the solution paths before their convergence , and show that SGD and GD have different solution paths that lead to different properties . Our result helps to explain why the SGD generalizes better than GD . Our second contribution is to unify the conditions to show the directional bias of ( S ) GD sequences in nonparametric models . The main condition is the diagonal dominant gram matrix , which covers a large class of kernel functions and allows us to study their properties . Moreover , our SGD is different from those in Wu et al . ( 2021 ) : they define SGD in epochs while we define SGD in steps . The fundamental difference in the SGD algorithm requires us to develop different techniques for analyzing the SGD sequence and showing its directional bias , which has not yet been covered in the current literature . Main Theorems of this paper can be divided into two parts , briefly summarized as follows : First is the directional bias of SGD . Theorem 5 shows that for a two-stage SGD with a moderate step size in the first stage and a small step size in the second stage , an early-stopped estimator has a directional bias towards the eigenvector that corresponds to the largest eigenvalue of the gram matrix . Later we refer to this direction as the direction towards the largest eigenvector to simplify the statement . As a comparison , Theorem 7 shows that GD with both moderate and small step sizes has a directional bias towards the eigenvector that corresponds to the smallest eigenvalue of the gram matrix ( denote it as the direction towards the smallest eigenvector ) . From which , we conclude that SGD and GD have different direction biases in kernel regression . Second is the implication of directional bias . The implication is very useful since it quantifies the effect of the directional bias on the generalization error . Theorem 9 considers a general problem of quadratic loss . It shows that the estimator biased towards the largest eigenvector of the Hessian ( which is the gram matrix in our nonparametric regression ) can have the smallest parameter estimation error , when compared with other estimators of the same loss . With this high-level idea of directional bias helps generalization , Theorem 11 compares the generalization error of SGD and GD in our problem setting . In particular , it upper bounds the generalization error of SGD and lower bounds the generalization error of GD . By directly comparing the error bounds , we guarantee that the generalization error of the SGD estimator is smaller than that of the GD estimator with high probability . We also point out that our result might shed new light on deep learning ( Belkin et al. , 2018 ) . By the state-of-the-art mathematical theory of Neural Networks ( NN ) , kernel and/or nonparametric methods can approximate the functional space of neural networks , see for example the NTK theory ( Jacot et al. , 2018 ) , and the Radon bounded variance space description for ReLU NN ( Parhi & Nowak , 2021 ) . Our technique might allow one to characterize the SGD solution path and show the generalization property in those problem settings . Paper organization . The rest of the paper is as follows : In Section 2 , we review some relevant literature ; In Section 3 , we give the formulation of the nonparametric regression and define the optimization problem . We also formalize the algorithm that is considered in this work and make assumptions to analyze the algorithms ; In Section 4 , we state our main theory on the directional bias of SGD/GD in nonparametric regression , where we include both the directional bias result and the implication of the directional bias for generalization . Experiments are conducted to support our theory ; In Section 5 , we discuss the finding in this paper , and propose some future research topic . All the proof , experiment details , and more experiments are deferred to the appendix due to page limits . 2 LITERATURE REVIEW . In this section , we review some relevant works . For better understanding , we split into two subsections : Subsection 2.1 reviews the background of RKHS ; Subsection 2.2 presents the state-of-the-art technique for analyzing the directional bias of the ( S ) GD algorithm . 2.1 RKHS . Kernel methods are among the core algorithms in machine learning and statistics ( Bartlett et al. , 2021 ) . As proposed by Wahba ( 1990 ) , the kernel method and RKHS serve as a unified framework of a group of nonparametric models , which extends the spline method . Later , kernel models become an important component in nonparametric models . In machine learning , the kernel-based method is always referred to as the “ kernel trick ” . By lifting the x variable to a high dimensional space via the kernel method , we can explore possibly nonlinear relationships between variables . Moreover , to play the kernel trick , one can directly calculate the kernel function using original features . This is computationally efficient since we avoid calculating high dimension or infinite dimension features . For applications of kernel method in machine learning , one can see kernel regression for image processing ( Takeda et al. , 2007 ) , for text mining ( Greene & Cunningham , 2006 ) , and for tasks in bioinformatics ( Saigo et al. , 2004 ) . Regarding deep learning , the kernel method is also important because it has implications for deep learning models . On the one hand , the kernel methods have similar benign overfitting behavior to neural network due to the implicit regularization and/or implicit bias phenomenon that we will review in the next subsection ( Belkin et al. , 2018 ) . On the other hand , the RKHS itself is closely related to Neural Networks via the Neural Tangent Kernel theory ( Jacot et al. , 2018 ) . This all indicates that to understand deep learning , one should first study kernel methods . 2.2 DIRECTIONAL BIAS . This paper analyzes the directional bias of SGD for the nonparametric regression . Directional bias , also referred to as implicit bias , of an algorithm refers to that its solution path is biased towards a certain direction . It works as that the algorithm prefers some directions over the others even though they may have the same objective function value . Since the algorithm selects a direction by itself , instead of explicitly required by any constraint , people use the term “ implicit ” . It is worth noting that the implicit regularization is related to implicit bias . The implicit regularization refers to that the converged point of an algorithm is like a regularized estimator , even if the objective function is not explicitly regularized . One can also interpret implicit regularization as the “ final result ” of implicit bias . In the recent work by Derezinski et al . ( 2020 ) , implicit regularization is used to develop an exact bound for double descent in linear regression . In this way , implicit regularization/bias serves as a way to explain some deep learning phenomenons that could not be addressed by the classical empirical risk minimization ( ERM ) framework . Therefore , it is important to study directional bias . State-of-the-art study on the directional bias of first-order algorithms can be divided into two categories by the technique they use : The first category is the ( stochastic ) gradient flow method , by taking an infinitesimal step size in ( S ) GD , the parameter dynamic follows a ( stochastic ) differential equation . Studying the solution path and the stationary point of the underlying differential equation helps to reveal the property of the parameter estimation . We list some works that use the first method to show the directional bias result of the ( stochastic ) gradient descent . Liu et al . ( 2018 ) analyze the Momentum SGD ( MSGD ) with infinitesimal step size , and show that the solution path escapes from the saddle for a nonconvex objective function . It is worth noting that in their case , the associated stochastic differential equation defines a complicated stochastic process , thus they replace it with an appropriate diffusion process , and the analysis is done based on such diffusion approximation . If one analyzes a stochastic gradient flow and finds it intractable , one may consider using the technique of diffusion approximation . Ali et al . ( 2020 ) shows the stochastic gradient flow for the linear regression problem minw ‖Xw−y‖22 has a solution path close to the solution path of Ridge regression ; Blanc et al . ( 2020 ) shows the stochastic gradient flow for a general loss function has a solution path close to the solution path of gradient flow on the objective function plus some extra penalty terms , and they explicitly identify the penalty terms ; Smith et al . ( 2021 ) go one more step from the infinitesimal step size to small step size , and characterize the effect of small step size as an extra penalty term in the gradient flow . Another category is analyzing the discrete ( S ) GD sequence . This technique in general just requires a moderate step size such that the algorithm converges ( or nearly converges ) , thus it is more meaningful from a practical perspective . We also find some directional bias work that is based on this technique . Vaskevicius et al . ( 2019 ) ; Zhao et al . ( 2019 ) ; Fan et al . ( 2021 ) analyze Hadamard reparameterized GD in sparse regression . They divide the true parameter into strong , weak , and 0 parts , and for each part , they carefully develop the stepwise error bound for each step of GD . They finally show that an early-stopped estimator along the solution path achieves the minimax optimal error rate for sparse regression , which indicates that the solution path is in the direction that biased towards a sparse solution . Recently , Wu et al . ( 2021 ) show that for overparameterized linear regression , SGD with moderate step size converges to the minimum norm interpolant in the direction that corresponds to the largest eigenvalue of the design matrix , while GD converges in the direction that corresponds to the smallest eigenvalue . For Neural Networks in the ‘ lazy training ’ regime , ? shows that GD also converges in the direction of the smallest eigenvalue of the Neural Tangent Kernel . Their result further reveal the mechanism of the directional bias as : GD fits the direction of a larger eigenvalue faster at the beginning of the training , left the smaller eigenvalue direction unfitted ; later the direction of smaller eigenvalue is fitted , resulting in that the estimator goes in this direction .

Summary: This paper studies the directional bias of SGD in kernel regression. In particular, this paper shows that when using moderate or small step size, GD converges along the direction corresponding to the smallest eigenvalue of the covariance matrix. In contrast, when provided with a moderate initial learning rate with annealing, SGD converges along the direction corresponding to the largest eigenvalue. Consequently, the authors show that such directional bias of SGD can result in an estimator that is closer to the ground truth, which further leads to better generalization.