Learning a Single Neuron with Bias Using Gradient Descent

We theoretically study the fundamental problem of learning a single neuron with a bias term ( x 7→ σ ( 〈w , x〉+ b ) ) in the realizable setting with the ReLU activation , using gradient descent . Perhaps surprisingly , we show that this is a significantly different and more challenging problem than the bias-less case ( which was the focus of previous works on single neurons ) , both in terms of the optimization geometry as well as the ability of gradient methods to succeed in some scenarios . We provide a detailed study of this problem , characterizing the critical points of the objective , demonstrating failure cases , and providing positive convergence guarantees under different sets of assumptions . To prove our results , we develop some tools which may be of independent interest , and improve previous results on learning single neurons . 1 Introduction . Learning a single ReLU neuron with gradient descent is a fundamental primitive in the theory of deep learning , and has been extensively studied in recent years . Indeed , in order to understand the success of gradient descent on complicated neural networks , it seems reasonable to expect a satisfying analysis of convergence on a single neuron . Although many previous works studied the problem of learning a single neuron with gradient descent , none of them considered this problem with an explicit bias term . In this work , we study the common setting of learning a single neuron with respect to the squared loss , using gradient descent . We focus on the realizable setting , where the inputs are drawn from a distribution D on Rd+1 , and are labeled by a single target neuron of the form x 7→ σ ( 〈v , x〉 ) , where σ : R → R is some non-linear activation function . To capture the bias term , we assume that the distribution D is such that its first d components are drawn from some distribution D̃ on Rd , and the last component is a constant 1 . Thus , the input x can be decomposed as ( x̃ , 1 ) with x̃ ∼ D̃ , the vector v can be decomposed as ( ṽ , bv ) , where ṽ ∈ Rd and bv ∈ R , and the target neuron computes a function of the form x 7→ σ ( 〈ṽ , x̃〉+ bv ) . Similarly , we can define the learned neuron as x 7→ σ ( 〈w̃ , x̃〉+ bw ) , where w = ( w̃ , bw ) . Overall , we can write the objective function we wish to ∗Equal contribution 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . optimize as follows : F ( w ) : = E x∼D [ 1 2 ( σ ( w > x ) − σ ( v > x ) ) 2 ] ( 1 ) = E x̃∼D̃ [ 1 2 ( σ ( w̃ > x̃ + bw ) − σ ( ṽ > x̃ + bv ) ) 2 ] . ( 2 ) Throughout the paper we consider the commonly used ReLU activation function : σ ( x ) = max { 0 , x } . Although the problem of learning a single neuron is well studied ( e.g . Soltanolkotabi [ 2017 ] , Yehudai and Shamir [ 2020 ] , Frei et al . [ 2020 ] , Du et al . [ 2017 ] , Kalan et al . [ 2019 ] , Tan and Vershynin [ 2019 ] , Mei et al . [ 2018 ] , Oymak and Soltanolkotabi [ 2018 ] ) , none of the previous works considered the problem with an additional bias term . Moreover , previous works on learning a single neuron with gradient methods have certain assumptions on the input distribution D , which do not apply when dealing with a bias term ( for example , a certain `` spread '' in all directions , which does not apply when D is supported on { 1 } in the last coordinate ) . Since neural networks with bias terms are the common practice , it is natural to ask how adding a bias term affects the optimization landscape and the convergence of gradient descent . Although one might conjecture that this is just a small modification to the problem , we in fact show that the effect of adding a bias term is very significant , both in terms of the optimization landscape and in terms of which gradient descent strategies can or can not work . Our main contributions are as follows : • We start in Section 3 with some negative results , which demonstrate how adding a bias term makes the problem more difficult . In particular , we show that with a bias term , gradient descent or gradient flow2 can sometimes fail with probability close to half over the initialization , even when the input distribution is uniform over a ball . In contrast , Yehudai and Shamir [ 2020 ] show that without a bias term , for the same input distribution , gradient flow converges to the global minimum with probability 1 . • In Section 4 we give a full characterization of the critical points of the loss function . We show that adding a bias term changes the optimization landscape significantly : In previous works ( cf . Yehudai and Shamir [ 2020 ] ) it has been shown that under mild assumptions on the input distribution , the only critical points are w = v ( i.e. , the global minimum ) and w = 0 . We prove that when we have a bias term , the set of critical points has a positive measure , and that there is a cone of local minima where the loss function is flat . • In Sections 5 and 6 we show that gradient descent converges to the global minimum at a linear rate , under some assumptions on the input distribution and on the initialization . We give two positive convergence results , where each result is under different assumptions , and thus the results complement each other . We also use different techniques for proving each of the results : The analysis in Section 6 follows from some geometric arguments and extends the technique from Yehudai and Shamir [ 2020 ] , Frei et al . [ 2020 ] . The analysis in Section 5 introduces a novel technique , not used in previous works on learning a single neuron , and has a more algebraic nature . Moreover , that analysis implies that under mild assumptions , gradient descent with random initialization converges to the global minimum with probability 1− eΩ ( d ) . • The best known result for learning a single neuron without bias using gradient descent for an input distribution that is not spherically symmetric , establishes convergence to the global minimum with probability close to 12 over the random initialization [ Yehudai and Shamir , 2020 , Frei et al. , 2020 ] . With our novel proof technique presented in Section 5 this result can be improved to probability at least 1− e−Ω ( d ) ( see Remark 5.7 ) . Related work Although there are no previous works on learning a single neuron with an explicit bias term , there are many works that consider the problem of a single neuron under different settings and assumptions . Several papers showed that the problem of learning a single neuron can be solved under minimal assumptions using algorithms which are not gradient-based ( such as gradient descent or SGD ) . These 2I.e. , gradient descent with infinitesimal step size . algorithms include the Isotron proposed by Kalai and Sastry [ 2009 ] and the GLMtron proposed by Kakade et al . [ 2011 ] . The GLMtron algorithm is also analyzed in Diakonikolas et al . [ 2020 ] . These algorithms allow learning a single neuron with bias . We note that these are non-standard algorithms , whereas we focus on the standard gradient descent algorithm . An efficient algorithm for learning a single neuron with error parameter = Ω ( 1/ log ( d ) ) was also obtained in Goel et al . [ 2017 ] . In Mei et al . [ 2018 ] the authors study the empirical risk of the single neuron problem . However , their analysis does not include the ReLU activation , or adding a bias term . A related analysis is also given in Oymak and Soltanolkotabi [ 2018 ] , where the ReLU activation is not considered . Several papers showed convergence guarantees for the single neuron problem with ReLU activation under certain distributional assumptions , although none of these assumptions allows for a bias term . Notably , Tian [ 2017 ] , Soltanolkotabi [ 2017 ] , Kalan et al . [ 2019 ] , Brutzkus and Globerson [ 2017 ] showed convergence guarantees for gradient methods when the inputs have a standard Gaussian distribution , without a bias term . Du et al . [ 2017 ] showed that under a certain subspace eigenvalue assumption a single neuron can be learned with SGD , although this assumption does not allow adding a bias term . Yehudai and Shamir [ 2020 ] , Frei et al . [ 2020 ] use an assumption about the input distribution being sufficiently `` spread '' in all directions , which does not allow for a bias term ( since that requires an input distribution supported on { 1 } in the last coordinate ) . Yehudai and Shamir [ 2020 ] showed a convergence result under the realizable setting , while Frei et al . [ 2020 ] considered the agnostic and noisy settings . In Tan and Vershynin [ 2019 ] convergence guarantees are given for the absolute value activation , and a specific distribution which does not allow a bias term . Less directly related , Vardi and Shamir [ 2020 ] studied the problem of implicit regularization in the single neuron setting . In Yehudai and Shamir [ 2019 ] , Kamath et al . [ 2020 ] it is shown that approximating a single neuron using random features ( or kernel methods ) is not tractable in high dimensions . We note that these results explicitly require that the single neuron which is being approximated will have a bias term . Thus , our work complements these works by showing that the problem of learning a single neuron with bias is also learnable using gradient descent ( under certain assumptions ) . Agnostically learning a single neuron with non gradient-based algorithms and hardness of a agnostically learning a single neuron were studied in Diakonikolas et al . [ 2020 ] , Goel et al . [ 2019 , 2020 ] . 2 Preliminaries . Notations . We use bold-faced letters to denote vectors , e.g. , x = ( x1 , . . . , xd ) . For u ∈ Rd we denote by ‖u‖ the Euclidean norm . We denote ū = u‖u‖ , namely , the unit vector in the direction of u . For 1 ≤ i ≤ j ≤ d we denote ui : j = ( ui , . . . , uj ) ∈ Rj−i+1 . We denote by 1 ( · ) the indicator function , for example 1 ( t ≥ 5 ) equals 1 if t ≥ 5 and 0 otherwise . We denote by U ( [ −r , r ] ) the uniform distribution over the interval [ −r , r ] in R , and byN ( 0 , Σ ) the multivariate normal distribution with mean 0 and covariance matrix Σ . Given two vectors w , v we let θ ( w , v ) = arccos ( 〈w , v〉 ‖w‖‖v‖ ) = arccos ( 〈w̄ , v̄〉 ) ∈ [ 0 , π ] . For a vector u ∈ Rd+1 we often denote by ũ ∈ Rd the first d components of u , and denote by bu ∈ R its last component . Gradient methods . In this paper we focus on the following two standard gradient methods for optimizing our objective F ( w ) from Eq . ( 2 ) : • Gradient descent : We initialize at some w0 ∈ Rd+1 , and set a fixed learning rate η > 0 . At each iteration t ≥ 0 we have : wt+1 = wt − η∇F ( wt ) . • Gradient Flow : We initialize at some w ( 0 ) ∈ Rd+1 , and for every time t ≥ 0 , we set w ( t ) to be the solution of the differential equation ẇ = −∇F ( w ( t ) ) . This can be thought of as a continuous form of gradient descent , where the learning rate is infinitesimally small . The gradient of the objective in Eq . ( 1 ) is : ∇F ( w ) = E x∼D [ ( σ ( w > x ) − σ ( v > x ) ) · σ′ ( w > x ) x ] . ( 3 ) Since σ is the ReLU function , it is differentiable everywhere except for 0 . Practical implementations of gradient methods define σ′ ( 0 ) to be some constant in [ 0 , 1 ] . Following this convention , the gradient used by these methods still correspond to Eq . ( 3 ) . We note that the exact value of σ′ ( 0 ) has no effect on our results .

Summary: The work considers learning a single ReLU with the bias term using gradient based methods (and more specifically with gradient flow and gradient descent - which falls in the infinite data regime). Whereas previous works mostly considered this problem without the bias term, it (the bias term) is essential in a lot of practical applications. The paper considers gradient descent/flow with respect to standard squared error, which is practically relevant.   First, it is shown (Section 3) that learning a single ReLU with bias via. gradient based methods is much harder than without by constructing specific negative examples. Then, under natural assumptions on the data distribution, the stationary points of the loss function is characterized and it is shown that it is a cone in Section 4 (rather than a finite set as is the case when there is no bias). The  subsequent sections explore several assumptions under which convergence can be guaranteed with high probability and with random initialization.