Analytic Study of Families of Spurious Minima in Two-Layer ReLU Neural Networks: A Tale of Symmetry II

One of the outstanding conundrums of deep learning concerns the ability of simple gradient-based methods to successfully train neural networks despite the nonconvexity of the associated optimization problems . Indeed , generic nonconvex optimization landscapes can exhibit wide and flat basins of attraction around poor local minima which may lead to a complete failure of such methods . The nature by which nonconvex problems associated with neural networks deviate from generic ones is currently not well-understood . In particular , much of the dynamics of gradient-based methods follows from the curvature of the loss landscape around local minima . It is therefore vital to study the local geometry of spurious ( i.e. , non-global local ) and global minima in order to understand the mysterious mechanism which drives gradient-based methods towards minima of high quality . However , already establishing the very existence of spurious minima seems to be beyond reach of existing analytic tools ; let alone rigorously arguing about their height , curvature and structure—the aim of this work . In this paper , we focus on two-layer ReLU neural networks of the form k∑ i=1 αiϕ ( 〈wi , x〉 ) , W ∈M ( k , d ) , α ∈ Rk , ( 1 ) where ϕ ( z ) = max { 0 , z } is the ReLU activation function acting entry-wise , M ( k , d ) denotes the space of k×dmatrices andwi denotes the ith row ofW . We are primarily interested in characterizing various optimization-related obstructions for local search method , independently of the expressive power of two-layer ReLU networks . Thus , data is understood to be fully realizable . Concretely , we assume that there are d inputs which are drawn from the standard multivariate Gaussian distribution 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . and are labeled by a planted target network . We consider directly optimizing the expected squared loss , which results in the following highly nonconvex optimization problem : L ( W , α ) : = 1 2 Ex∼N ( 0 , Id ) [ ( k∑ i=1 αiϕ ( 〈wi , x〉 ) − d∑ i=1 βiϕ ( 〈vi , x〉 ) ) 2 ] , ( 2 ) where k denotes the number of hidden neurons , W ∈ M ( k , d ) and αi ∈ Rk are the optimization variables , and vi and βi are fixed parameters . This setting , in which the data distribution is regulated rather than being allowed to admit worst-case behavior , has drawn a considerable amount of interest in recent years [ 66 , 17 , 19 , 40 , 61 , 11 , 24 , 54 , 4 , 1 ] , in part due to the growing number of evidences which indicate that any explanation for the empirical success of deep learning ( DL ) must take into account the intricate interplay between the network architecture , the input distribution and the label distribution ( cf . [ 11 , 9 , 59 ] and references therein for hardness results of optimization and learnability under partial sets of assumptions ) . Moreover , as demonstrated later in the paper , in spite of its apparent simplicity , nonconvex problem ( 2 ) shares a few important characteristics with full-scale neural networks , such as low-dimensional minima and extremely skewed Hessian spectrum . Learning problem ( 2 ) has also been studied in the statistical physics community starting from the 80 ’ [ 23 , 57 , 37 , 62 , 18 , 8 , 52 , 53 , 51 ] under the student-teacher ( ST ) framework , in which one aims to adjust a student network so as to fit the output of a teacher network . The ST framework offers a clean venue for analyzing optimization-related aspects of neural network models in the spirit of physical reductionism . The success of DL models in the past decade has reinitiated a surge of interest in this framework , e.g. , [ 7 , 28 , 42 , 47 ] . However , despite the long tradition in the statistical physics community and the wide effort put nowadays by the machine learning community , the perplexing geometry of problem ( 2 ) still seems to be out of reach of existing analytic tools in regimes encountered in practice . In this paper , we present a novel set of symmetry-based tools which allows us , for the first time , to analytically characterize important aspects of the associated highly nonconvex landscape for a finite number of inputs and neurons . Our contributions , in order of appearance , can be stated as follows : • We demonstrate that , empirically , and in a well-defined sense , minima in two-layer ReLU neural networks break the symmetry of the target weight matrix . Although ReLU networks have been studied for many years , this phenomenon of symmetry breaking seems to have gone largely unnoticed . • We show that symmetry breaking makes it possible to derive analytic expressions for families of spurious minima in the form of fractional power series in terms of d and k. Crucially , in contrast to existing approaches which employ various limiting processes , e.g. , [ 7 , 28 , 42 , 47 , 29 , 43 , 13 , 33 , 14 ] , our method operates in the natural regime where d and k are finite . • We develop a novel technique which yields an analytic characterization of the Hessian spectrum of minima to O ( d−1/2 ) -order for d ≥ k , and determine the exact fractional value of d at which critical points turn from saddles into spurious minima—a key ingredient in understanding how over-parameterization annihilates spurious minima . • Based on the unique access to high-dimensional spectral information , we closely examine a number of hypotheses in the machine learning literature pertaining to curvature , optimization and generalization . In particular , we prove that the Hessian spectrum at minima concentrates near small positive constants , with the exception of Θ ( d ) eigenvalues which grow linearly with d. Although this phenomenon of extremely skewed spectrum has been observed many times [ 10 , 38 , 55 , 56 ] , to our knowledge , this is the first time it has been established rigorously for two-layer ReLU networks . In addition , our analysis shows that the Hessian spectra of spurious and global minima are identical to O ( d−1/2 ) -terms , and further implies that the inductive bias of stochastic gradient descent ( SGD ) , provably , can not be exclusively explained in terms of local curvature [ 32 , 36 , 34 , 63 , 64 , 12 , 16 ] . The results presented in the paper are threefold : identifying a symmetry breaking principle for two-layer ReLU networks , analytically characterizating spurious minima , and computing the Hessian spectrum . The next three sections are organized accordingly , along with a literature survey of related work . The last section is devoted for a high-level description of the novel symmetry-based technique used in this work . Proofs , formalities and lengthy technical details are deferred to the appendix . 1 Symmetry breaking in two-layer ReLU neural networks . Optimization problem ( 2 ) exhibits a very rich symmetry structure . Indeed , the loss function L is invariant to left- and right-multiplication of W by permutation matrices ( see Section A.1 for a formal proof ) , i.e. , L ( W , α ) = L ( PπWP > ρ , α ) for all ( π , ρ ) ∈ Sk × Sd , where Sm generally denotes the symmetric group of degree m , and ( Pπ ) ij = { 1 i = π ( j ) , 0 o.w. . It is therefore natural to ask how the critical points of L reflect this symmetry . For example , the identity matrix Id , one of the global minimizers of L for the case in which V = Id , α = β = 1d and k = d , is invariant under simultaneous left- and right-multiplication by any permutation matrix . Indeed , PπIdP > π = PπP > π = Id for any π ∈ Sd . ( Modulo group conjugation , this holds for any global minimizer of L. See [ 6 , Proposition 4.14. ] . ) This simple observation is conveniently stated using the concept of the isotropy group . Given a weight matrix W ∈M ( k , d ) , we let Iso ( W ) : = { ( π , ρ ) | ( π , ρ ) ∈ Sk × Sd , PπWP > ρ = W } , ( 3 ) Thus , we have Iso ( Id ) = ∆Sd , where ∆ maps any subgroup H ⊆ Sd to its diagonal counterpart ∆H : = { ( h , h ) | h ∈ H } ⊆ Sd × Sd . Empirically , and somewhat miraculously , spurious minima and saddles of L tend to be highly symmetric in the ( formal ) sense that their isotropy groups are ( conjugated to ) large subgroups of ∆Sd , the isotropy of the global minima ( see Figure 2 ) . Thus , the principle of symmetry breaking can be concisely phrased as follows : Spurious minima break the symmetry of global minima . The principle extends to more general target networks ; when the isotropy of the target weight matrix V changes , the symmetry of spurious minima belonging to the respective optimization problem . In Section A , we provide a series of experiments which empirically corroborates symmetry breaking for optimization problem ( 2 ) . The principle of symmetry breaking for ReLU networks was first studied in [ 3 , 6 ] , and was later extended to various tensor decomposition problems in [ 2 ] . Here , we analyze two-layer ReLU networks where both layers are trainable , an architecture which has received a considerable amount of attention in recent years , e.g. , [ 66 , 17 , 19 , 40 , 61 , 11 , 24 ] . We note in passing that , in its broader sense , the principle of symmetry breaking has been observed many times in various scientific fields , e.g. , Higgs-Landau theory , equivariant bifurcation theory and replica symmetry-breaking ( see , e.g. , [ 46 , 30 , 21 , 15 ] ) . Two-layer ReLU networks seems to form a rather unexpected instance of this principle . One intriguing quality of nonconvex landscapes in which the symmetry breaking principle applies is that local minima lie in fixed low-dimensional spaces ( see Section 4 ) . A similar phenomenon of hidden low-dimensional structure has been observed in various learning problems in DL with real datasets [ 39 , 31 ] , and is believed by some to be an important factor of learnability in nonconvex settings . In the context of this work , the hidden low-dimensionality of spurious minima turns out to be a key ingredient to our analytic study , as we now present . 2 Power series representation of families of spurious minima . Although the problem of fitting neural networks under the ST framework have been studied for more than 30 years , the symmetry breaking principle exhibited by optimization problem ( 2 ) seems to have gone largely unnoticed . Early studies adopted tools ( at times , heuristic , e.g. , [ 45 , 44 ] ) from statistical physics to analyze phase transitions and generalization errors [ 23 , 57 , 37 , 62 ] . Later work focused on the dynamics of SGD and studied the evolution of the generalization error along the optimization process through a set of carefully-derived ODEs [ 18 , 8 , 52 , 53 , 51 ] . Following the empirical success of DL in the past decade , this line of work has recently drawn a renewed interest , e.g , [ 7 , 28 , 47 , 29 ] , which puts past analyses on rigorous grounds and addresses a broader class of architectures and activation functions . The methods used in this long line of works operate in the thermodynamic limit—where the number of inputs is taken to infinity . Thus , the formal validity of the results to finite width networks is currently limited . Other common approaches for analyzing problem ( 2 ) are based on : mean-field [ 43 ] , optimal control [ 13 ] , NTK [ 33 ] and compositional kernels [ 14 ] . These approaches offer , in essence , convex surrogates which apply in strict parameter regimes ( including algorithmic parameters , such as learning rates ) . Similarly to the thermodynamic limit approach , the convex surrogates are obtained by limiting processes—this time by taking the number of hidden neurons to infinite . A growing number of works has severely limited , if not invalidated , the explanatory power of these approaches for network widths encountered in practice [ 65 , 26 ] . In sharp contrast to the approaches discussed above , our technique directly addresses the associated highly nonconvex optimization landscape in the natural regime where the number of inputs and neurons is finite , and provides analytic , rather than heuristic , information . This is obtained by exploiting the presence of symmetry breaking phenomena whereby fractional power series representation for families of spurious minima are derived . Below , we provide analytic expressions for families of minima of different isotropy ( see Definition 3 and Figure 2 ) , along with their respective objective value . For brevity , expressions are given to O ( d−3/2 ) -order . In the appendix , we list additional O ( d−5/2 ) -order terms which are required for computing the Hessian spectrum . The invariance properties of optimization problem ( 2 ) imply that additional families of minima can be obtained by permuting the rows and the columns of a given family . The number of new distinct minima generated using such transformations ( i.e. , the minima multiplicity ) depends on the very structure of the family of minima under consideration , and is stated in Theorem 1 . Lastly , for any λ > 0 and i ∈ [ d ] , the value of the ReLU network ( 1 ) remains fixed under ( wi , αi ) 7→ ( λwi , αi/λ ) . This degree-of-freedom is expressed by using the slack variables λi > 0 , i ∈ [ d ] below . Theorem 1 . Optimization problem ( 2 ) with k = d , V = Id and β = 1d possesses the following families of minima for d ≥ 9 : 1 . Two families of ∆Sd-minima of multiplicity d ! with W ( d ) = Diag ( λ1 , . . . , λd ) Ad ( a1 , a2 ) and α ( d ) = ( λ−11 , . . . , λ −1 d ) , where λi > 0 , Ad ( a1 , a2 ) : = a1 a2 . . . a2 a2 a1 a2 . . . a2 ... ... a2 . . . a2 a1 ∈M ( d , d ) , Diag ( · ) maps a given vector to the diagonal entries of a diagonal matrix , and ( a ) a1 = 1 and a2 = 0 ( a global minimizer ) , in which case L ( W ( d ) , α ( d ) ) = 0 . ( b ) a1 = −1 + 2d +O ( d −3 2 ) , a2 = 2 d +O ( d −3 2 ) , in which case L ( W ( d ) , α ( d ) ) = − 1 π + 1 2 − 4 3π √ d + − 12 − 2π2 + 3π d +O ( d −3 2 ) . ( 4 ) 2 . Two families of ∆ ( Sd−1 × S1 ) -minima of multiplicity d · d ! with W ( d ) = Diag ( λ1 , . . . , λd ) ( Ad−1 ( a1 , a2 ) a3Id−1,1 a4I1 , d−1 a5 ) , and α ( d ) = ( λ−11 , . . . , λ −1 d ) , where λi > 0 and ( a ) a1 = 1 + O ( d −3 2 ) , a2 = O ( d −3 2 ) , a3 = 2 d + O ( d −3 2 ) , a4 = 4 πd + O ( d −3 2 ) and a5 = −1 + 8 π2 +2+ 8π d +O ( d −3 2 ) , in which case L ( W ( d ) , α ( d ) ) = 1 2 − 2π2 d +O ( d −3 2 ) . ( 5 ) ( b ) a1 = −1 + 2d +O ( d −3 2 ) , a2 = 2 d +O ( d −3 2 ) , a3 = O ( d −3 2 ) , a4 = 2− 4π d +O ( d −3 2 ) and a5 = 1 + 8 ( −1+π ) π2d +O ( d −3 2 ) , in which case L ( W ( d ) , α ( d ) ) = − 1 π + 1 2 − 4 3π √ d + −1− 4π2 + 5π d +O ( d −3 2 ) . ( 6 ) 3 . Two families of ∆ ( Sd−2 × S2 ) -minima of multiplicity d ! ( d 2 ) with W ( d ) = Diag ( λ1 , . . . , λd ) ( Ad−2 ( a1 , a2 ) a3Id−2,2 a4I2 , d−2 A2 ( a5 , a6 ) ) , and α ( d ) = ( λ−11 , . . . , λ −1 d ) , where λi > 0 and ( a ) a1 = 1 +O ( d −3 2 ) , a2 = 0 +O ( d −3 2 ) , a3 = 2 d +O ( d −3 2 ) , a4 = 4 πd +O ( d −3 2 ) , a5 = −1 + 8 π2 +2+ 8π d +O ( d −3 2 ) and a6 = 2 ( −12π+16+π3+4π2 ) π2d ( 2+π ) +O ( d −3 2 ) , in which case L ( W ( d ) , α ( d ) ) = −4 + π 2 π2d +O ( d −3 2 ) . ( 7 ) ( b ) a1 = −1 + 2d + O ( d −3 2 ) , a2 = 2 d + O ( d −3 2 ) , a3 = 0 + O ( d −3 2 ) , a4 = 2− 4π d + O ( d −3 2 ) , a5 = 1 + 8 ( −1+π ) π2d +O ( d −3 2 ) and a6 = 4 ( −π2−8+6π ) π2d ( 2+π ) +O ( d −3 2 ) , in which case L ( W ( d ) , α ( d ) ) = − 1 π + 1 2 − 4 3π √ d + − 32 − 6π2 + 7π d +O ( d −3 2 ) . ( 8 ) 4 . One family of ∆ ( Sd−3 × S3 ) -minima of multiplicity d ! ( d 3 ) with W ( d ) = Diag ( λ1 , . . . , λd ) ( Ad−3 ( a1 , a2 ) a3Id−3,3 a4I3 , d−3 A2 ( a5 , a6 ) ) , and α ( d ) = ( λ−11 , . . . , λ −1 d ) , where λi > 0 and a1 = 1 + O ( d −3 2 ) , a2 = O ( d −3 2 ) , a3 = 2 d + O ( d −3 2 ) , a4 = 4 πd+O ( d −3 2 ) , a5 = −1+ 8 π2 +2+ 8π d +O ( d −3 2 ) and a6 = 2 ( −12π+16+π3+4π2 ) π2d ( 2+π ) + O ( d −3 2 ) , in which case L ( W ( d ) , α ( d ) ) = 3 2 − 6π2 d +O ( d −3 2 ) . ( 9 ) The idea of the proof of Theorem 1 is given in Section 4.1 , and the multiplicity computation , a simple application of the orbit-stabilizer theorem , in Section B . We note that the same technique can be used for other choices of target networks , mutatis mutandis . The power series stated in Theorem 1 also represent spurious minima for the under-parameterized d > k-case . This is obtained by appropriately padding the entries of the weight matrices by zeros ( see [ 6 , section 4.3 ] for details ) . The derivation of the respective Hessian spectrum is then an immediate application of [ 4 , section E.1 ] to the spectral analysis given in Theorem 2 below ( for the case where d = k ) . By contrast , studying the over-parameterized regime for which d < k requires qualitatively different tools and is outside the scope of this work . The analysis above reveals that not all local minima are alike : while the objective value of some minima decays like Θ ( 1/d ) , in other cases the objective value converges to the positive constant 1 2 − 1π . In particular , except for the global minima case 1a , all minima described above are spurious . The difference between the two types of behavior seems to lie in the limiting values of the entries of W ( d ) ; in the former , the diagonal mainly consists of ones , whereas in the latter mainly minus ones . This is consistent with the fact that all the teacher ’ s diagonal entries are ones . For reasons which are yet to be understood , empirically , the bias induced by Xavier initialization [ 27 ] seems to favor the class of minima for which the objective value decays as Θ ( 1/d ) . This is not to say that under Xavier initialization the expected value upon convergence must decrease with d. The objective value decays to zero at different rates ( see cases 2a , 3a , 4 above ) and depends on the probability to converge to a given type of minima . ( A possible proxy to the latter is the minima multiplicity stated in Theorem 1 . )

Summary: Under a student-teacher setup with unit Gaussian input, this submission studies the Hessian of a finite-width two-layer ReLU network at critical points of the (population) MSE loss, and characterizes the spectrum and loss at different minima. The starting observation is a symmetry-breaking property based on which a power series description of the minima can be derived, which then leads to an analytic description of the Hessian (up to a dimension-dependent residual). The analysis reveals interesting non-asymptotic properties of the landscape of two-layer network.