The Hidden Convex Optimization Landscape of Regularized Two-Layer ReLU Networks: an Exact Characterization of Optimal Solutions

1 INTRODUCTION . Let X ∈ Rn×d and y ∈ Rn be the data matrix and the label vector . Given a number of neurons m > 1 and a regularization parameter β > 0 , we consider the regularized optimization problem P∗m = min θ∈Θm { Lβ ( θ ) : = ` ( m∑ i=1 σ ( Xui ) αi ) + β 2 m∑ i=1 ( ‖ui‖22 + α2i ) } . ( 1 ) where Θm = Rd×m × Rm , θ = ( U , α ) , ui is the i-th column of U ∈ Rd×m and αi is the i-th coefficient of α ∈ Rm . Here we focus on the ReLU activation , i.e. , σ ( z ) = max { z , 0 } and absorb the label y ∈ Rn in the loss function ` : Rn → R , which is assumed to be convex ( e.g. , logistic , hinge , squared loss ) . The model ∑m i=1 σ ( Xui ) αi in ( 1 ) can be easily extended to the one with bias term by adding a column of 1 ’ s into the data X . We refer to an element θ ∈ Θm as a neural network and to each pair ( ui , αi ) as a neuron . We denote the set of optimal neural network as Θ∗m = { θ ∈ Θm | Lβ ( θ ) = P∗m } . ( 2 ) We denote the best training loss achievable by a neural as P∗ = infm > 1 P∗m . The ReLU activation induces a natural partition of the parameter space . We denote D1 , . . . , Dp as all possible values of diag ( 1 ( Xu > 0 ) ) . We introduce the corresponding convex cones Ci = { u ∈ Rd| ( 2Di − I ) Xu > 0 } for i ∈ [ p ] where we denote [ p ] = { 1 , . . . , p } . From this partition we have the local linearization σ ( Xu ) = DiXu , for u ∈ Ci . ( 3 ) We let Di+p = −Di for i ∈ [ p ] and Ci+p = Ci for i ∈ [ p ] . Such a partition of the parameter space has regained attention in the recent literature . In fact , Pilanci & Ergen ( 2020 ) recently showed that an optimal neural network θ∗ ∈ Θm for any m > 2p can be constructed based on a solution of the convex optimization problem P∗c : = min W∈W { Lcβ ( W ) : = ` ( 2p∑ i=1 DiXwi ) + β · 2p∑ i=1 ‖wi‖2 } , ( 4 ) where we introduced the convex feasible setW : = { W = ( w1 , . . . , w2p ) | wi ∈ Ci } . In a nutshell , this equivalence can be intuitively explained as follows . From the constraint wi ∈ Ci , we obtain the local linearization σ ( Xwi ) = DiXwi for i ∈ [ p ] and σ ( Xwi ) = −DiXwi for i ∈ [ p+ 1 , 2p ] . By choosing neurons ( ui , αi ) such that wi = |αi|ui , αi > 0 for i ∈ [ p ] and αi 6 0 for i > p + 1 , we further obtain by positive homogeneity of the ReLU that 2p∑ i=1 DiXwi = 2p∑ i=1 σ ( Xui ) αi . ( 5 ) From the fact that the cones C1 , . . . , Cp cover the entire space , Pilanci & Ergen ( 2020 ) establish the equality P∗c = P∗ , and show that an optimal neural network can be constructed from an optimal solution w∗1 , . . . , w ∗ p. In this work , we explore in more depth the mapping from the optimal set of solutionsW∗ of the convex program ( 4 ) to the set of optimal neural networks Θ∗m . In fact , our main contribution is to show how to construct the set Θ∗m givenW∗ through simple mappings and transformations . Throughout our analysis , we unveil some novel necessary conditions for a neural network to be optimal and we illustrate the relevance of these conditions by relating them to usual necessary conditions for optimality ( e.g. , Clarke stationarity ) . 1.1 PRIOR AND RELATED WORK . Literature on the theory of deep learning is now extremely vast , and we mention here only a few classical results and recent works which focus on the properties of the neural network training loss , from an optimization perspective ( e.g. , the convergence of first-order methods , absence of spurious minima in the loss landscape , and , convex re-formulations of neural network training ) , and in the case of single-layer with ReLU-type activations . Several recent works considered over-parameterized neural networks in the infinite-width limit . In particular , it is known that in this regime , gradient descent converges to an optimal solution , see ( Jacot et al. , 2018 ; Du et al. , 2018 ; Allen-Zhu et al. , 2018 ; Nguyen , 2021 ) . Further analysis in ( Chizat & Bach , 2018 ) showed that almost no hidden neurons move from their initial values to actively learn useful features , so that this regime resembles that of kernel training and the infinite-width limit infuses convexity . Wang & Lin ( 2021 ) showed that with an explicit regularizer based on the scaled variation norm , overparametrization is generally harmless to two-layer ReLU networks . However , experiments in ( Arora et al. , 2016 ) suggest that this kernel approximation is unable to fully explain the success of non-convex neural network models . Convexity arguments in neural networks were proposed in the recent literature ( Bengio et al. , 2006 ; Bach , 2017 ) . However , existing works , except ( Pilanci & Ergen , 2020 ) , are restricted to infinitely wide networks . In turn , Bengio et al . ( 2006 ) and Bach ( 2017 ) consider greedy neuron-wise optimization strategies for the infinite-dimensional optimization problem , which requires solving non-convex problems at every step to train a shallow neural network . In contrast , in our work , we reveal the hidden convex optimization landscape for any finite number of hidden neurons . Besides the convexity properties of infinitely wide networks , many works derived lower bounds on the hidden layer size to guarantee the absence of spurious minima . Venturi et al . ( 2019 ) showed that the un-regularized ( i.e . β = 0 ) objective Lβ has no spurious local minima provided that the number of neurons satisfies m > n , and a similar result was shown in ( Livni et al. , 2014 ) . Similar results were derived for deep networks . For instance , Soudry & Carmon ( 2016 ) showed that under a dropout-like noise assumption , there exist no differentiable spurious minima if the product of the dimensions of the layer weights exceeds n and this result matches the classical lower bound ( Baum , 1988 ) on the minimal width of a neural network to implement any dichotomy for inputs in general position . In a similar vein , Nguyen & Hein ( 2017 ) showed that no spurious minima occur provided that one of the layer ’ s inner width exceeds n and under additional non-degeneracy conditions . For activations other than the ReLU ( e.g. , linear , quadratic , polynomial ) , similar lower bounds were derived in ( Venturi et al. , 2019 ; Du & Lee , 2018 ; Soltanolkotabi et al. , 2018 ) . These analyses are typically based on the idea that when m & n then it is very likely that the features σ ( Xu1 ) , . . . , σ ( Xum ) form a basis of Rn so that the training problem reduces to finding a linear model with weights α1 , . . . , αm which perfectly fits the labels . For the hinge loss and linear separable data , ( Wang et al. , 2019 ) show that the modified stochastic gradient descent method can achieve global optimality despite the presence of spurious local minima and saddle points . The training landscape of neural networks is of great interest for theoretical analysis in the optimization of neural networks . An important perspective is to analyze the landscape via paths through the parameter space , see ( Vidal et al. , 2017 ) . Indeed , in ( Haeffele & Vidal , 2015 ; 2017 ; Sharifnassab et al. , 2019 ) , it is shown that there exists a non-increasing path in objective value from every point to the global minimum with mild assumption on the layer width . The existence of such paths also indicates that the level sets of the training loss are connected ( Freeman & Bruna , 2016 ; Venturi et al. , 2019 ; Nguyen , 2019 ; Nguyen et al. , 2021 ) and there is no bad local valley ( Nguyen & Hein , 2017 ) . However , with regularization , the training problem is more challenging . Intuitively , it reduces the set of optimal solutions to those with small norms . Without regularization ( i.e. , β = 0 ) , it should be noted that the set of optimal solutions always contains infinitely many points . For instance , with ReLU activations , it holds that if θ∗ = { ( u∗i , α∗i ) } mi=1 is an optimal neural network , then any re-scaling of θ∗ in the form { ( u ∗ i γi , γi α ∗ i ) } mi=1 ( with γ1 , . . . , γm > 0 ) has the same objective value and is thus optimal . With regularization , this manifold is reduced to a single point . It is then natural to expect that this minimal size of the hidden layer must increase . Further , the aforementioned analyses do not extend since regularization also penalizes the norms of the ui ’ s , and one can not simply generate such a basis of Rn based on the features σ ( Xu1 ) , . . . , σ ( Xum ) by random sampling and then overfitting the labels . Recently , Pilanci & Ergen ( 2020 ) ; Ergen & Pilanci ( 2020 ) show that two-layer ReLU neural networks can be optimized exactly via finite-dimensional convex programs with complexity polynomial in the number of samples and hidden neurons . As indicated in Pilanci & Ergen ( 2020 ) , the worst-case complexity is exponential in the dimension of the training samples unless P = NP . 1.2 SUMMARY OF OUR CONTRIBUTIONS . In Section 2 , we introduce the notions of minimal neural networks and nearly minimal neural networks . These two notions are closely related to the plateau and the edge of the plateau of the loss landscape . In Section 3 , we show that any minimal neural network θ can be represented , via an explicit map , in the convex feasible spaceW as a point W ( θ ) such that Lβ ( θ ) > Lcβ ( W ( θ ) ) , and vice-versa . This structural result provides a mathematically rich perspective to characterize optimal neural networks through the lens of convexity . We then provide an exact characterization of the set of all global optima of the nonconvex problem , which include all nearly minimal neural networks generated via the optimal solutions of the convex program . In Section 4 , we show that any Clarke stationary point θ with respect to Lβ is a nearly minimal neural network . This provides a preliminary structure on the solutions found by stochastic gradient descent ( SGD ) , as it has been recently shown ( see , for instance , Corollary 5.11 in Davis et al . ( 2020 ) ) that the limit points of SGD applied to neural network optimization are Clarke stationary . More importantly , we show that Clarke stationary point θ with respect to Lβ also corresponds to a global minimum of a subsampled convex problem . We also provide a polynomial-time algorithm ( in the sample size n and the hidden-layer size m ) in order to test whether a neural network is globally optimal . In Section 5 , we show that any neural network is path-connected to a succinct representation ( with at most n + 1 non-zero neurons ) and this path is with constant objective value . Then , from the convex perspective of two-layer ReLU neural networks , we provide an explicit path of non-increasing loss between θ and θ′ , where θ′ is the global optimum of the non-convex training problem . This establishes that the training loss Lβ has no spurious local minima , provided that the number of neurons is sufficiently large .

Summary: In this paper, the authors study the training of two-layer ReLU networks with weight decay.  A previous paper (Pilanci and Ergen 2020) introduced a convex optimization problem that corresponds to this non-convex case.  In the present paper, the authors prove that all optimal solutions of the nonconvex formulation can be found via optimal solutions of this convex formulation.  (Whereas the Pilanci and Ergen paper only constructed a single solution).  The authors additionally show that a Clarke stationary point of the nonconvex objective corresponds to a global minimum of a subsampled version of the convex problem, and they provide a polynomial time algorithm to test if a neural network is globally optimal.  Finally, the authors prove that the nonconvex loss landscape has no spurious local minima provided the number of neurons is large enough.