The Hidden Convex Optimization Landscape of Regularized Two-Layer ReLU Networks: an Exact Characterization of Optimal Solutions

1 INTRODUCTION . Let X ∈ Rn×d and y ∈ Rn be the data matrix and the label vector . Given a number of neurons m > 1 and a regularization parameter β > 0 , we consider the regularized optimization problem P∗m = min θ∈Θm { Lβ ( θ ) : = ` ( m∑ i=1 σ ( Xui ) αi ) + β 2 m∑ i=1 ( ‖ui‖22 + α2i ) } . ( 1 ) where Θm = Rd×m × Rm , θ = ( U , α ) , ui is the i-th column of U ∈ Rd×m and αi is the i-th coefficient of α ∈ Rm . Here we focus on the ReLU activation , i.e. , σ ( z ) = max { z , 0 } and absorb the label y ∈ Rn in the loss function ` : Rn → R , which is assumed to be convex ( e.g. , logistic , hinge , squared loss ) . The model ∑m i=1 σ ( Xui ) αi in ( 1 ) can be easily extended to the one with bias term by adding a column of 1 ’ s into the data X . We refer to an element θ ∈ Θm as a neural network and to each pair ( ui , αi ) as a neuron . We denote the set of optimal neural network as Θ∗m = { θ ∈ Θm | Lβ ( θ ) = P∗m } . ( 2 ) We denote the best training loss achievable by a neural as P∗ = infm > 1 P∗m . The ReLU activation induces a natural partition of the parameter space . We denote D1 , . . . , Dp as all possible values of diag ( 1 ( Xu > 0 ) ) . We introduce the corresponding convex cones Ci = { u ∈ Rd| ( 2Di − I ) Xu > 0 } for i ∈ [ p ] where we denote [ p ] = { 1 , . . . , p } . From this partition we have the local linearization σ ( Xu ) = DiXu , for u ∈ Ci . ( 3 ) We let Di+p = −Di for i ∈ [ p ] and Ci+p = Ci for i ∈ [ p ] . Such a partition of the parameter space has regained attention in the recent literature . In fact , Pilanci & Ergen ( 2020 ) recently showed that an optimal neural network θ∗ ∈ Θm for any m > 2p can be constructed based on a solution of the convex optimization problem P∗c : = min W∈W { Lcβ ( W ) : = ` ( 2p∑ i=1 DiXwi ) + β · 2p∑ i=1 ‖wi‖2 } , ( 4 ) where we introduced the convex feasible setW : = { W = ( w1 , . . . , w2p ) | wi ∈ Ci } . In a nutshell , this equivalence can be intuitively explained as follows . From the constraint wi ∈ Ci , we obtain the local linearization σ ( Xwi ) = DiXwi for i ∈ [ p ] and σ ( Xwi ) = −DiXwi for i ∈ [ p+ 1 , 2p ] . By choosing neurons ( ui , αi ) such that wi = |αi|ui , αi > 0 for i ∈ [ p ] and αi 6 0 for i > p + 1 , we further obtain by positive homogeneity of the ReLU that 2p∑ i=1 DiXwi = 2p∑ i=1 σ ( Xui ) αi . ( 5 ) From the fact that the cones C1 , . . . , Cp cover the entire space , Pilanci & Ergen ( 2020 ) establish the equality P∗c = P∗ , and show that an optimal neural network can be constructed from an optimal solution w∗1 , . . . , w ∗ p. In this work , we explore in more depth the mapping from the optimal set of solutionsW∗ of the convex program ( 4 ) to the set of optimal neural networks Θ∗m . In fact , our main contribution is to show how to construct the set Θ∗m givenW∗ through simple mappings and transformations . Throughout our analysis , we unveil some novel necessary conditions for a neural network to be optimal and we illustrate the relevance of these conditions by relating them to usual necessary conditions for optimality ( e.g. , Clarke stationarity ) . 1.1 PRIOR AND RELATED WORK . Literature on the theory of deep learning is now extremely vast , and we mention here only a few classical results and recent works which focus on the properties of the neural network training loss , from an optimization perspective ( e.g. , the convergence of first-order methods , absence of spurious minima in the loss landscape , and , convex re-formulations of neural network training ) , and in the case of single-layer with ReLU-type activations . Several recent works considered over-parameterized neural networks in the infinite-width limit . In particular , it is known that in this regime , gradient descent converges to an optimal solution , see ( Jacot et al. , 2018 ; Du et al. , 2018 ; Allen-Zhu et al. , 2018 ; Nguyen , 2021 ) . Further analysis in ( Chizat & Bach , 2018 ) showed that almost no hidden neurons move from their initial values to actively learn useful features , so that this regime resembles that of kernel training and the infinite-width limit infuses convexity . Wang & Lin ( 2021 ) showed that with an explicit regularizer based on the scaled variation norm , overparametrization is generally harmless to two-layer ReLU networks . However , experiments in ( Arora et al. , 2016 ) suggest that this kernel approximation is unable to fully explain the success of non-convex neural network models . Convexity arguments in neural networks were proposed in the recent literature ( Bengio et al. , 2006 ; Bach , 2017 ) . However , existing works , except ( Pilanci & Ergen , 2020 ) , are restricted to infinitely wide networks . In turn , Bengio et al . ( 2006 ) and Bach ( 2017 ) consider greedy neuron-wise optimization strategies for the infinite-dimensional optimization problem , which requires solving non-convex problems at every step to train a shallow neural network . In contrast , in our work , we reveal the hidden convex optimization landscape for any finite number of hidden neurons . Besides the convexity properties of infinitely wide networks , many works derived lower bounds on the hidden layer size to guarantee the absence of spurious minima . Venturi et al . ( 2019 ) showed that the un-regularized ( i.e . β = 0 ) objective Lβ has no spurious local minima provided that the number of neurons satisfies m > n , and a similar result was shown in ( Livni et al. , 2014 ) . Similar results were derived for deep networks . For instance , Soudry & Carmon ( 2016 ) showed that under a dropout-like noise assumption , there exist no differentiable spurious minima if the product of the dimensions of the layer weights exceeds n and this result matches the classical lower bound ( Baum , 1988 ) on the minimal width of a neural network to implement any dichotomy for inputs in general position . In a similar vein , Nguyen & Hein ( 2017 ) showed that no spurious minima occur provided that one of the layer ’ s inner width exceeds n and under additional non-degeneracy conditions . For activations other than the ReLU ( e.g. , linear , quadratic , polynomial ) , similar lower bounds were derived in ( Venturi et al. , 2019 ; Du & Lee , 2018 ; Soltanolkotabi et al. , 2018 ) . These analyses are typically based on the idea that when m & n then it is very likely that the features σ ( Xu1 ) , . . . , σ ( Xum ) form a basis of Rn so that the training problem reduces to finding a linear model with weights α1 , . . . , αm which perfectly fits the labels . For the hinge loss and linear separable data , ( Wang et al. , 2019 ) show that the modified stochastic gradient descent method can achieve global optimality despite the presence of spurious local minima and saddle points . The training landscape of neural networks is of great interest for theoretical analysis in the optimization of neural networks . An important perspective is to analyze the landscape via paths through the parameter space , see ( Vidal et al. , 2017 ) . Indeed , in ( Haeffele & Vidal , 2015 ; 2017 ; Sharifnassab et al. , 2019 ) , it is shown that there exists a non-increasing path in objective value from every point to the global minimum with mild assumption on the layer width . The existence of such paths also indicates that the level sets of the training loss are connected ( Freeman & Bruna , 2016 ; Venturi et al. , 2019 ; Nguyen , 2019 ; Nguyen et al. , 2021 ) and there is no bad local valley ( Nguyen & Hein , 2017 ) . However , with regularization , the training problem is more challenging . Intuitively , it reduces the set of optimal solutions to those with small norms . Without regularization ( i.e. , β = 0 ) , it should be noted that the set of optimal solutions always contains infinitely many points . For instance , with ReLU activations , it holds that if θ∗ = { ( u∗i , α∗i ) } mi=1 is an optimal neural network , then any re-scaling of θ∗ in the form { ( u ∗ i γi , γi α ∗ i ) } mi=1 ( with γ1 , . . . , γm > 0 ) has the same objective value and is thus optimal . With regularization , this manifold is reduced to a single point . It is then natural to expect that this minimal size of the hidden layer must increase . Further , the aforementioned analyses do not extend since regularization also penalizes the norms of the ui ’ s , and one can not simply generate such a basis of Rn based on the features σ ( Xu1 ) , . . . , σ ( Xum ) by random sampling and then overfitting the labels . Recently , Pilanci & Ergen ( 2020 ) ; Ergen & Pilanci ( 2020 ) show that two-layer ReLU neural networks can be optimized exactly via finite-dimensional convex programs with complexity polynomial in the number of samples and hidden neurons . As indicated in Pilanci & Ergen ( 2020 ) , the worst-case complexity is exponential in the dimension of the training samples unless P = NP . 1.2 SUMMARY OF OUR CONTRIBUTIONS . In Section 2 , we introduce the notions of minimal neural networks and nearly minimal neural networks . These two notions are closely related to the plateau and the edge of the plateau of the loss landscape . In Section 3 , we show that any minimal neural network θ can be represented , via an explicit map , in the convex feasible spaceW as a point W ( θ ) such that Lβ ( θ ) > Lcβ ( W ( θ ) ) , and vice-versa . This structural result provides a mathematically rich perspective to characterize optimal neural networks through the lens of convexity . We then provide an exact characterization of the set of all global optima of the nonconvex problem , which include all nearly minimal neural networks generated via the optimal solutions of the convex program . In Section 4 , we show that any Clarke stationary point θ with respect to Lβ is a nearly minimal neural network . This provides a preliminary structure on the solutions found by stochastic gradient descent ( SGD ) , as it has been recently shown ( see , for instance , Corollary 5.11 in Davis et al . ( 2020 ) ) that the limit points of SGD applied to neural network optimization are Clarke stationary . More importantly , we show that Clarke stationary point θ with respect to Lβ also corresponds to a global minimum of a subsampled convex problem . We also provide a polynomial-time algorithm ( in the sample size n and the hidden-layer size m ) in order to test whether a neural network is globally optimal . In Section 5 , we show that any neural network is path-connected to a succinct representation ( with at most n + 1 non-zero neurons ) and this path is with constant objective value . Then , from the convex perspective of two-layer ReLU neural networks , we provide an explicit path of non-increasing loss between θ and θ′ , where θ′ is the global optimum of the non-convex training problem . This establishes that the training loss Lβ has no spurious local minima , provided that the number of neurons is sufficiently large .

Summary: Along the line of Pilanci & Ergen (2020), this submission deals learning two-layer ReLU neural networks through convex optimization. It introduces a number of new notions such as (nearly) minimal neural networks, developing a set of interesting tools, and draws connections between the minimal neural networks and the convex optimization landscape. This paper provides a rich framework along with new analyses and solutions for learning two-layer ReLU networks through convex cone optimization.