Efficiently testing local optimality and escaping saddles for ReLU networks

1 INTRODUCTION . Empirical success of deep neural networks has sparked great interest in the theory of deep models . From an optimization viewpoint , the biggest mystery is that deep neural networks are successfully trained by gradient-based algorithms despite their nonconvexity . On the other hand , it has been known that training neural networks to global optimality is NP-hard ( Blum & Rivest , 1988 ) . It is also known that even checking local optimality of nonconvex problems can be NP-hard ( Murty & Kabadi , 1987 ) . Bridging this gap between theory and practice is a very active area of research , and there have been many attempts to understand why optimization works well for neural networks , by studying the loss surface ( Baldi & Hornik , 1989 ; Yu & Chen , 1995 ; Kawaguchi , 2016 ; Soudry & Carmon , 2016 ; Nguyen & Hein , 2017 ; 2018 ; Safran & Shamir , 2018 ; Laurent & Brecht , 2018 ; Yun et al. , 2019 ; 2018 ; Zhou & Liang , 2018 ; Wu et al. , 2018 ; Shamir , 2018 ) and the role of ( stochastic ) gradientbased methods ( Tian , 2017 ; Brutzkus & Globerson , 2017 ; Zhong et al. , 2017 ; Soltanolkotabi , 2017 ; Li & Yuan , 2017 ; Zhang et al. , 2018 ; Brutzkus et al. , 2018 ; Wang et al. , 2018 ; Li & Liang , 2018 ; Du et al. , 2018a ; b ; c ; Allen-Zhu et al. , 2018 ; Zou et al. , 2018 ; Zhou et al. , 2019 ) . One of the most important beneficial features of convex optimization is the existence of an optimality test ( e.g. , norm of the gradient is smaller than a certain threshold ) for termination , which gives us a certificate of ( approximate ) optimality . In contrast , many practitioners in deep learning rely on running first-order methods for a fixed number of epochs , without good termination criteria for the optimization problem . This means that the solutions that we obtain at the end of training are not necessarily global or even local minima . Yun et al . ( 2018 ; 2019 ) showed efficient and simple global optimality tests for deep linear neural networks , but such optimality tests can not be extended to general nonlinear neural networks , mainly due to nonlinearity in activation functions . Besides nonlinearity , in case of ReLU networks significant additional challenges in the analysis arise due to nondifferentiability , and obtaining a precise understanding of the nondifferentiable points is still elusive . ReLU activation function h ( t ) = max { t , 0 } is nondifferentiable at t = 0 . This means that , for example , the function f ( w , b ) : = ( h ( wTx + b ) − 1 ) 2 is nondifferentiable for any ( w , b ) satisfyingwTx+b = 0 . See Figure 1 for an illustration of how the empirical risk of a ReLU network looks like . Although the plotted function does not exactly match the definition of empirical risk we study in this paper , the figures help us understand that the empirical risk is continuous but piecewise differentiable , with affine hyperplanes on which the function is nondifferentiable . Such nondifferentiable points lie in a set of measure zero , so one may be tempted to overlook them as “ non-generic. ” However , when studying critical points we can not do so , as they are precisely such “ non-generic ” points . For example , Laurent & Brecht ( 2018 ) study one-hidden-layer ReLU networks with hinge loss and note that except for piecewise constant regions , local minima always occur on nonsmooth boundaries . Probably due to difficulty in analysis , there have not been other works that handle such nonsmooth points of losses and prove results that work for all points . Some theorems ( Soudry & Carmon , 2016 ; Nguyen & Hein , 2018 ) hold “ almost surely ” ; some assume differentiability or make statements only for differentiable points ( Nguyen & Hein , 2017 ; Yun et al. , 2019 ) ; others analyze population risk , in which case the nondifferentiability disappears after taking expectation ( Tian , 2017 ; Brutzkus & Globerson , 2017 ; Du et al. , 2018b ; Safran & Shamir , 2018 ; Wu et al. , 2018 ) . 1.1 SUMMARY OF OUR RESULTS . In this paper , we take a step towards understanding nondifferentiable points of the empirical risk of one-hidden-layer ReLU ( -like ) networks . Specifically , we provide a theoretical algorithm that tests second-order stationarity for any point of the loss surface . It takes an input point and returns : ( a ) The point is a local minimum ; or ( b ) The point is a second-order stationary point ( SOSP ) ; or ( c ) A descent direction in which the function value strictly decreases . Therefore , we can test whether a given point is a SOSP . If not , the test extracts a guaranteed direction of descent that helps continue minimization . With a proper numerical implementation of our algorithm ( although we leave it for future work ) , one can run a first-order method until it gets stuck near a point , and run our algorithm to test for optimality/second-order stationarity . If the point is an SOSP , we can terminate without further computation over many epochs ; if the point has a descent direction , our algorithm will return a descent direction and we can continue on optimizing . Note that the descent direction may come from the second-order information ; our algorithm even allows us to escape nonsmooth second-order saddle points . This idea of mixing first and second-order methods has been explored in differentiable problems ( see , for example , Carmon et al . ( 2016 ) ; Reddi et al . ( 2017 ) and references therein ) , but not for nondifferentiable ReLU networks . The key computational challenge in constructing our algorithm for nondifferentiable points is posed by data points that causes input 0 to the ReLU hidden node ( s ) . Such data point bisects the parameter space into two halfspaces with different “ slopes ” of the loss surface , so one runs into nondifferen- tiability . We define these data points to be boundary data points . For example , in Figure 1b , if the input to our algorithm is ( w , v ) = ( −2/3 , 1/3 ) , then there are two boundary data points : “ blue ” and “ red. ” If there are M such boundary data points , then in the worst case the parameter space divides into 2M regions , or equivalently , there are 2M “ pieces ” of the function that surround the input point . Of course , naively testing each region will be very inefficient ; in our algorithm , we overcome this issue by a clever use of polyhedral geometry . Another challenge comes from the second-order test , which involves solving nonconvex QPs . Although QP is NP-hard in general ( Pardalos & Vavasis , 1991 ) , we prove that the QPs in our algorithm are still solved efficiently in most cases . We further describe the challenges and key ideas in Section 2.1 . Notation . For a vector v , [ v ] i denotes its i-th component , and ‖v‖H : = √ vTHv denotes a seminorm where H is a positive semidefinite matrix . Given a matrix A , we let [ A ] i , j , [ A ] i , · , and [ A ] · , j be A ’ s ( i , j ) -th entry , the i-th row , and the j-th column , respectively . 2 PROBLEM SETTING AND KEY IDEAS . We consider a one-hidden-layer neural network with input dimension dx , hidden layer width dh , and output dimension dy . We are given m pairs of data points and labels ( xi , yi ) mi=1 , where xi ∈ Rdx and yi ∈ Rdy . Given an input vector x , the output of the network is defined as Y ( x ) : = W2h ( W1x+ b1 ) + b2 , where W2 ∈ Rdy×dh , b2 ∈ Rdy , W1 ∈ Rdh×dx , and b1 ∈ Rdh are the network parameters . The activation function h is “ ReLU-like , ” meaning h ( t ) : = max { s+t , 0 } + min { s−t , 0 } , where s+ > 0 , s− ≥ 0 and s+ 6= s− . Note that ReLU and Leaky-ReLU are members of this class . In training neural networks , we are interested in minimizing the empirical risk R ( ( Wj , bj ) 2 j=1 ) = ∑m i=1 ` ( Y ( xi ) , yi ) = ∑m i=1 ` ( W2h ( W1xi + b1 ) + b2 , yi ) , over the parameters ( Wj , bj ) 2j=1 , where ` ( w , y ) : Rdy × Rdy 7→ R is the loss function . We make the following assumptions on the loss function and the training dataset : Assumption 1 . The loss function ` ( w , y ) is twice differentiable and convex in w. Assumption 2 . No dx + 1 data points lie on the same affine hyperplane . Assumption 1 is satisfied by many standard loss functions such as squared error loss and crossentropy loss . Assumption 2 means , if dx = 2 for example , no three data points are on the same line . Since real-world datasets contain noise , this assumption is also quite mild . 2.1 CHALLENGES AND KEY IDEAS . In this section , we explain the difficulties at nondifferentiable points and ideas on overcoming them . Our algorithm is built from first principles , rather than advanced tools from nonsmooth analysis . Bisection by boundary data points . Since the activation function h is nondifferentiable at 0 , the behavior of data points at the “ boundary ” is decisive . Consider a simple example dh = 1 , so W1 is a row vector . IfW1xi+b1 6= 0 , then the sign of ( W1+∆1 ) xi+ ( b1+δ1 ) for any small perturbations ∆1 and δ1 stays invariant . In contrast , when there is a point xi on the “ boundary , ” i.e. , W1xi + b1 = 0 , then the slope depends on the direction of perturbation , leading to nondifferentiability . As mentioned earlier , we refer to such data points as boundary data points . When ∆1xi + δ1 ≥ 0 , h ( ( W1 +∆1 ) xi+ ( b1 +δ1 ) ) = h ( ∆1xi+δ1 ) = s+ ( ∆1xi+δ1 ) = h ( W1xi+b1 ) +s+ ( ∆1xi+δ1 ) , and similarly , the slope is s− for ∆1xi + δ1 ≤ 0 . This means that the “ gradient ” ( as well as higher order derivatives ) of R depends on direction of ( ∆1 , δ1 ) . Thus , every boundary data point xi bisects the space of perturbations ( ∆j , δj ) 2j=1 into two halfspaces by introducing a hyperplane through the origin . The situation is even worse if we have M boundary data points : they lead to a worst case of 2M regions . Does it mean that we need to test all 2M regions separately ? We show that there is a way to get around this issue , but before that , we first describe how to test local minimality or stationarity for each region . Second-order local optimality conditions . We can expand R ( ( Wj +∆j , bj +δj ) 2j=1 ) and obtain the following Taylor-like expansion for small enough perturbations ( see Lemma 2 for details ) R ( z + η ) = R ( z ) + g ( z , η ) T η + 12η TH ( z , η ) η + o ( ‖η‖2 ) , ( 1 ) where z is a vectorized version of all parameters ( Wj , bj ) 2j=1 and η is the corresponding vector of perturbations ( ∆j , δj ) 2j=1 . Notice now that in ( 1 ) , at nondifferentiable points the usual Taylor expansion does not exist , but the corresponding “ gradient ” g ( · ) and “ Hessian ” H ( · ) now depend on the direction of perturbation η . Also , the space of η is divided into at most 2M regions , and g ( z , η ) andH ( z , η ) are piecewise-constant functions of η whose “ pieces ” correspond to the regions . One could view this problem as 2M constrained optimization problems and try to solve for KKT conditions at z ; however , we provide an approach that is developed from first principles and solves all 2M problems efficiently . Given this expansion ( 1 ) and the observation that derivatives stay invariant with respect to scaling of η , one can note that ( a ) g ( z , η ) T η ≥ 0 for all η , and ( b ) ηTH ( z , η ) η ≥ 0 for all η such that g ( z , η ) T η = 0 are necessary conditions for local optimality of z , thus z is a “ SOSP ” ( see Definition 2.2 ) . The conditions become sufficient if ( b ) is replaced with ηTH ( z , η ) η > 0 for all η 6= 0 such that g ( z , η ) T η = 0 . In fact , this is a generalized version of second-order necessary ( or sufficient ) conditions , i.e. , ∇f = 0 and∇2f 0 ( or ∇2f 0 ) , for twice differentiable f . Efficiently testing SOSP for exponentially many regions . Motivated from the second-order expansion ( 1 ) and necessary/sufficient conditions , our algorithm consists of three steps : ( a ) Testing first-order stationarity ( in the Clarke sense , see Definition 2.1 ) , ( b ) Testing g ( z , η ) T η ≥ 0 for all η , ( c ) Testing ηTH ( z , η ) η ≥ 0 for { η | g ( z , η ) T η = 0 } . The tests are executed from Step ( a ) to ( c ) . Whenever a test fails , we get a strict descent direction η , and the algorithm returns η and terminates . Below , we briefly outline each step and discuss how we can efficiently perform the tests . We first check first-order stationarity because it makes Step ( b ) easier . Step ( a ) is done by solving one convex QP per each hidden node . For Step ( b ) , we formulate linear programs ( LPs ) per each 2M region , so that checking whether all LPs have minimum cost of zero is equivalent to checking g ( z , η ) T η ≥ 0 for all η . Here , the feasible sets of LPs are pointed polyhedral cones , whereby it suffices to check only the extreme rays of the cones . It turns out that there are only 2M extreme rays , each shared by 2M−1 cones , so testing g ( z , η ) T η ≥ 0 can be done with only O ( M ) inequality/equality tests instead of solving exponentially many LPs . In Step ( b ) , we also record the flat extreme rays , which are defined to be the extreme rays with g ( z , η ) T η = 0 , for later use in Step ( c ) . In Step ( c ) , we test if the second-order perturbation ηTH ( · ) η can be negative , for directions where g ( z , η ) T η = 0 . Due to the constraint g ( z , η ) T η = 0 , the second-order test requires solving constrained nonconvex QPs . In case where there is no flat extreme ray , we need to solve only one equality constrained QP ( ECQP ) . If there exist flat extreme rays , a few more inequality constrained QPs ( ICQPs ) are solved . Despite NP-hardness of general QPs ( Pardalos & Vavasis , 1991 ) , we prove that the specific form of QPs in our algorithm are still tractable in most cases . More specifically , we prove that projected gradient descent on ECQPs converges/diverges exponentially fast , and each step takesO ( p2 ) time ( p is the number of parameters ) . In case of ICQPs , it takesO ( p3 +L32L ) time to solve the QP , where L is the number of boundary data points that have flat extreme rays ( L ≤M ) . Here , we can see that if L is small enough , the ICQP can still be solved in polynomial time in p. At the end of the paper , we provide empirical evidences that the number of flat extreme rays is zero or very few , meaning that in most cases we can solve the QP efficiently .

Summary: The paper proposes a method to check if a given point is a stationary point or not (if not, it provides a descent direction), and then classify stationary points as either local min or second-order stationary. The method works for a specific non-differentiable loss.  In the worst case, there can be exponentially many flat directions to check (2^L), but usually this is no the case.