Analytic Study of Families of Spurious Minima in Two-Layer ReLU Neural Networks: A Tale of Symmetry II

One of the outstanding conundrums of deep learning concerns the ability of simple gradient-based methods to successfully train neural networks despite the nonconvexity of the associated optimization problems . Indeed , generic nonconvex optimization landscapes can exhibit wide and flat basins of attraction around poor local minima which may lead to a complete failure of such methods . The nature by which nonconvex problems associated with neural networks deviate from generic ones is currently not well-understood . In particular , much of the dynamics of gradient-based methods follows from the curvature of the loss landscape around local minima . It is therefore vital to study the local geometry of spurious ( i.e. , non-global local ) and global minima in order to understand the mysterious mechanism which drives gradient-based methods towards minima of high quality . However , already establishing the very existence of spurious minima seems to be beyond reach of existing analytic tools ; let alone rigorously arguing about their height , curvature and structure—the aim of this work . In this paper , we focus on two-layer ReLU neural networks of the form k∑ i=1 αiϕ ( 〈wi , x〉 ) , W ∈M ( k , d ) , α ∈ Rk , ( 1 ) where ϕ ( z ) = max { 0 , z } is the ReLU activation function acting entry-wise , M ( k , d ) denotes the space of k×dmatrices andwi denotes the ith row ofW . We are primarily interested in characterizing various optimization-related obstructions for local search method , independently of the expressive power of two-layer ReLU networks . Thus , data is understood to be fully realizable . Concretely , we assume that there are d inputs which are drawn from the standard multivariate Gaussian distribution 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . and are labeled by a planted target network . We consider directly optimizing the expected squared loss , which results in the following highly nonconvex optimization problem : L ( W , α ) : = 1 2 Ex∼N ( 0 , Id ) [ ( k∑ i=1 αiϕ ( 〈wi , x〉 ) − d∑ i=1 βiϕ ( 〈vi , x〉 ) ) 2 ] , ( 2 ) where k denotes the number of hidden neurons , W ∈ M ( k , d ) and αi ∈ Rk are the optimization variables , and vi and βi are fixed parameters . This setting , in which the data distribution is regulated rather than being allowed to admit worst-case behavior , has drawn a considerable amount of interest in recent years [ 66 , 17 , 19 , 40 , 61 , 11 , 24 , 54 , 4 , 1 ] , in part due to the growing number of evidences which indicate that any explanation for the empirical success of deep learning ( DL ) must take into account the intricate interplay between the network architecture , the input distribution and the label distribution ( cf . [ 11 , 9 , 59 ] and references therein for hardness results of optimization and learnability under partial sets of assumptions ) . Moreover , as demonstrated later in the paper , in spite of its apparent simplicity , nonconvex problem ( 2 ) shares a few important characteristics with full-scale neural networks , such as low-dimensional minima and extremely skewed Hessian spectrum . Learning problem ( 2 ) has also been studied in the statistical physics community starting from the 80 ’ [ 23 , 57 , 37 , 62 , 18 , 8 , 52 , 53 , 51 ] under the student-teacher ( ST ) framework , in which one aims to adjust a student network so as to fit the output of a teacher network . The ST framework offers a clean venue for analyzing optimization-related aspects of neural network models in the spirit of physical reductionism . The success of DL models in the past decade has reinitiated a surge of interest in this framework , e.g. , [ 7 , 28 , 42 , 47 ] . However , despite the long tradition in the statistical physics community and the wide effort put nowadays by the machine learning community , the perplexing geometry of problem ( 2 ) still seems to be out of reach of existing analytic tools in regimes encountered in practice . In this paper , we present a novel set of symmetry-based tools which allows us , for the first time , to analytically characterize important aspects of the associated highly nonconvex landscape for a finite number of inputs and neurons . Our contributions , in order of appearance , can be stated as follows : • We demonstrate that , empirically , and in a well-defined sense , minima in two-layer ReLU neural networks break the symmetry of the target weight matrix . Although ReLU networks have been studied for many years , this phenomenon of symmetry breaking seems to have gone largely unnoticed . • We show that symmetry breaking makes it possible to derive analytic expressions for families of spurious minima in the form of fractional power series in terms of d and k. Crucially , in contrast to existing approaches which employ various limiting processes , e.g. , [ 7 , 28 , 42 , 47 , 29 , 43 , 13 , 33 , 14 ] , our method operates in the natural regime where d and k are finite . • We develop a novel technique which yields an analytic characterization of the Hessian spectrum of minima to O ( d−1/2 ) -order for d ≥ k , and determine the exact fractional value of d at which critical points turn from saddles into spurious minima—a key ingredient in understanding how over-parameterization annihilates spurious minima . • Based on the unique access to high-dimensional spectral information , we closely examine a number of hypotheses in the machine learning literature pertaining to curvature , optimization and generalization . In particular , we prove that the Hessian spectrum at minima concentrates near small positive constants , with the exception of Θ ( d ) eigenvalues which grow linearly with d. Although this phenomenon of extremely skewed spectrum has been observed many times [ 10 , 38 , 55 , 56 ] , to our knowledge , this is the first time it has been established rigorously for two-layer ReLU networks . In addition , our analysis shows that the Hessian spectra of spurious and global minima are identical to O ( d−1/2 ) -terms , and further implies that the inductive bias of stochastic gradient descent ( SGD ) , provably , can not be exclusively explained in terms of local curvature [ 32 , 36 , 34 , 63 , 64 , 12 , 16 ] . The results presented in the paper are threefold : identifying a symmetry breaking principle for two-layer ReLU networks , analytically characterizating spurious minima , and computing the Hessian spectrum . The next three sections are organized accordingly , along with a literature survey of related work . The last section is devoted for a high-level description of the novel symmetry-based technique used in this work . Proofs , formalities and lengthy technical details are deferred to the appendix . 1 Symmetry breaking in two-layer ReLU neural networks . Optimization problem ( 2 ) exhibits a very rich symmetry structure . Indeed , the loss function L is invariant to left- and right-multiplication of W by permutation matrices ( see Section A.1 for a formal proof ) , i.e. , L ( W , α ) = L ( PπWP > ρ , α ) for all ( π , ρ ) ∈ Sk × Sd , where Sm generally denotes the symmetric group of degree m , and ( Pπ ) ij = { 1 i = π ( j ) , 0 o.w. . It is therefore natural to ask how the critical points of L reflect this symmetry . For example , the identity matrix Id , one of the global minimizers of L for the case in which V = Id , α = β = 1d and k = d , is invariant under simultaneous left- and right-multiplication by any permutation matrix . Indeed , PπIdP > π = PπP > π = Id for any π ∈ Sd . ( Modulo group conjugation , this holds for any global minimizer of L. See [ 6 , Proposition 4.14. ] . ) This simple observation is conveniently stated using the concept of the isotropy group . Given a weight matrix W ∈M ( k , d ) , we let Iso ( W ) : = { ( π , ρ ) | ( π , ρ ) ∈ Sk × Sd , PπWP > ρ = W } , ( 3 ) Thus , we have Iso ( Id ) = ∆Sd , where ∆ maps any subgroup H ⊆ Sd to its diagonal counterpart ∆H : = { ( h , h ) | h ∈ H } ⊆ Sd × Sd . Empirically , and somewhat miraculously , spurious minima and saddles of L tend to be highly symmetric in the ( formal ) sense that their isotropy groups are ( conjugated to ) large subgroups of ∆Sd , the isotropy of the global minima ( see Figure 2 ) . Thus , the principle of symmetry breaking can be concisely phrased as follows : Spurious minima break the symmetry of global minima . The principle extends to more general target networks ; when the isotropy of the target weight matrix V changes , the symmetry of spurious minima belonging to the respective optimization problem . In Section A , we provide a series of experiments which empirically corroborates symmetry breaking for optimization problem ( 2 ) . The principle of symmetry breaking for ReLU networks was first studied in [ 3 , 6 ] , and was later extended to various tensor decomposition problems in [ 2 ] . Here , we analyze two-layer ReLU networks where both layers are trainable , an architecture which has received a considerable amount of attention in recent years , e.g. , [ 66 , 17 , 19 , 40 , 61 , 11 , 24 ] . We note in passing that , in its broader sense , the principle of symmetry breaking has been observed many times in various scientific fields , e.g. , Higgs-Landau theory , equivariant bifurcation theory and replica symmetry-breaking ( see , e.g. , [ 46 , 30 , 21 , 15 ] ) . Two-layer ReLU networks seems to form a rather unexpected instance of this principle . One intriguing quality of nonconvex landscapes in which the symmetry breaking principle applies is that local minima lie in fixed low-dimensional spaces ( see Section 4 ) . A similar phenomenon of hidden low-dimensional structure has been observed in various learning problems in DL with real datasets [ 39 , 31 ] , and is believed by some to be an important factor of learnability in nonconvex settings . In the context of this work , the hidden low-dimensionality of spurious minima turns out to be a key ingredient to our analytic study , as we now present . 2 Power series representation of families of spurious minima . Although the problem of fitting neural networks under the ST framework have been studied for more than 30 years , the symmetry breaking principle exhibited by optimization problem ( 2 ) seems to have gone largely unnoticed . Early studies adopted tools ( at times , heuristic , e.g. , [ 45 , 44 ] ) from statistical physics to analyze phase transitions and generalization errors [ 23 , 57 , 37 , 62 ] . Later work focused on the dynamics of SGD and studied the evolution of the generalization error along the optimization process through a set of carefully-derived ODEs [ 18 , 8 , 52 , 53 , 51 ] . Following the empirical success of DL in the past decade , this line of work has recently drawn a renewed interest , e.g , [ 7 , 28 , 47 , 29 ] , which puts past analyses on rigorous grounds and addresses a broader class of architectures and activation functions . The methods used in this long line of works operate in the thermodynamic limit—where the number of inputs is taken to infinity . Thus , the formal validity of the results to finite width networks is currently limited . Other common approaches for analyzing problem ( 2 ) are based on : mean-field [ 43 ] , optimal control [ 13 ] , NTK [ 33 ] and compositional kernels [ 14 ] . These approaches offer , in essence , convex surrogates which apply in strict parameter regimes ( including algorithmic parameters , such as learning rates ) . Similarly to the thermodynamic limit approach , the convex surrogates are obtained by limiting processes—this time by taking the number of hidden neurons to infinite . A growing number of works has severely limited , if not invalidated , the explanatory power of these approaches for network widths encountered in practice [ 65 , 26 ] . In sharp contrast to the approaches discussed above , our technique directly addresses the associated highly nonconvex optimization landscape in the natural regime where the number of inputs and neurons is finite , and provides analytic , rather than heuristic , information . This is obtained by exploiting the presence of symmetry breaking phenomena whereby fractional power series representation for families of spurious minima are derived . Below , we provide analytic expressions for families of minima of different isotropy ( see Definition 3 and Figure 2 ) , along with their respective objective value . For brevity , expressions are given to O ( d−3/2 ) -order . In the appendix , we list additional O ( d−5/2 ) -order terms which are required for computing the Hessian spectrum . The invariance properties of optimization problem ( 2 ) imply that additional families of minima can be obtained by permuting the rows and the columns of a given family . The number of new distinct minima generated using such transformations ( i.e. , the minima multiplicity ) depends on the very structure of the family of minima under consideration , and is stated in Theorem 1 . Lastly , for any λ > 0 and i ∈ [ d ] , the value of the ReLU network ( 1 ) remains fixed under ( wi , αi ) 7→ ( λwi , αi/λ ) . This degree-of-freedom is expressed by using the slack variables λi > 0 , i ∈ [ d ] below . Theorem 1 . Optimization problem ( 2 ) with k = d , V = Id and β = 1d possesses the following families of minima for d ≥ 9 : 1 . Two families of ∆Sd-minima of multiplicity d ! with W ( d ) = Diag ( λ1 , . . . , λd ) Ad ( a1 , a2 ) and α ( d ) = ( λ−11 , . . . , λ −1 d ) , where λi > 0 , Ad ( a1 , a2 ) : = a1 a2 . . . a2 a2 a1 a2 . . . a2 ... ... a2 . . . a2 a1 ∈M ( d , d ) , Diag ( · ) maps a given vector to the diagonal entries of a diagonal matrix , and ( a ) a1 = 1 and a2 = 0 ( a global minimizer ) , in which case L ( W ( d ) , α ( d ) ) = 0 . ( b ) a1 = −1 + 2d +O ( d −3 2 ) , a2 = 2 d +O ( d −3 2 ) , in which case L ( W ( d ) , α ( d ) ) = − 1 π + 1 2 − 4 3π √ d + − 12 − 2π2 + 3π d +O ( d −3 2 ) . ( 4 ) 2 . Two families of ∆ ( Sd−1 × S1 ) -minima of multiplicity d · d ! with W ( d ) = Diag ( λ1 , . . . , λd ) ( Ad−1 ( a1 , a2 ) a3Id−1,1 a4I1 , d−1 a5 ) , and α ( d ) = ( λ−11 , . . . , λ −1 d ) , where λi > 0 and ( a ) a1 = 1 + O ( d −3 2 ) , a2 = O ( d −3 2 ) , a3 = 2 d + O ( d −3 2 ) , a4 = 4 πd + O ( d −3 2 ) and a5 = −1 + 8 π2 +2+ 8π d +O ( d −3 2 ) , in which case L ( W ( d ) , α ( d ) ) = 1 2 − 2π2 d +O ( d −3 2 ) . ( 5 ) ( b ) a1 = −1 + 2d +O ( d −3 2 ) , a2 = 2 d +O ( d −3 2 ) , a3 = O ( d −3 2 ) , a4 = 2− 4π d +O ( d −3 2 ) and a5 = 1 + 8 ( −1+π ) π2d +O ( d −3 2 ) , in which case L ( W ( d ) , α ( d ) ) = − 1 π + 1 2 − 4 3π √ d + −1− 4π2 + 5π d +O ( d −3 2 ) . ( 6 ) 3 . Two families of ∆ ( Sd−2 × S2 ) -minima of multiplicity d ! ( d 2 ) with W ( d ) = Diag ( λ1 , . . . , λd ) ( Ad−2 ( a1 , a2 ) a3Id−2,2 a4I2 , d−2 A2 ( a5 , a6 ) ) , and α ( d ) = ( λ−11 , . . . , λ −1 d ) , where λi > 0 and ( a ) a1 = 1 +O ( d −3 2 ) , a2 = 0 +O ( d −3 2 ) , a3 = 2 d +O ( d −3 2 ) , a4 = 4 πd +O ( d −3 2 ) , a5 = −1 + 8 π2 +2+ 8π d +O ( d −3 2 ) and a6 = 2 ( −12π+16+π3+4π2 ) π2d ( 2+π ) +O ( d −3 2 ) , in which case L ( W ( d ) , α ( d ) ) = −4 + π 2 π2d +O ( d −3 2 ) . ( 7 ) ( b ) a1 = −1 + 2d + O ( d −3 2 ) , a2 = 2 d + O ( d −3 2 ) , a3 = 0 + O ( d −3 2 ) , a4 = 2− 4π d + O ( d −3 2 ) , a5 = 1 + 8 ( −1+π ) π2d +O ( d −3 2 ) and a6 = 4 ( −π2−8+6π ) π2d ( 2+π ) +O ( d −3 2 ) , in which case L ( W ( d ) , α ( d ) ) = − 1 π + 1 2 − 4 3π √ d + − 32 − 6π2 + 7π d +O ( d −3 2 ) . ( 8 ) 4 . One family of ∆ ( Sd−3 × S3 ) -minima of multiplicity d ! ( d 3 ) with W ( d ) = Diag ( λ1 , . . . , λd ) ( Ad−3 ( a1 , a2 ) a3Id−3,3 a4I3 , d−3 A2 ( a5 , a6 ) ) , and α ( d ) = ( λ−11 , . . . , λ −1 d ) , where λi > 0 and a1 = 1 + O ( d −3 2 ) , a2 = O ( d −3 2 ) , a3 = 2 d + O ( d −3 2 ) , a4 = 4 πd+O ( d −3 2 ) , a5 = −1+ 8 π2 +2+ 8π d +O ( d −3 2 ) and a6 = 2 ( −12π+16+π3+4π2 ) π2d ( 2+π ) + O ( d −3 2 ) , in which case L ( W ( d ) , α ( d ) ) = 3 2 − 6π2 d +O ( d −3 2 ) . ( 9 ) The idea of the proof of Theorem 1 is given in Section 4.1 , and the multiplicity computation , a simple application of the orbit-stabilizer theorem , in Section B . We note that the same technique can be used for other choices of target networks , mutatis mutandis . The power series stated in Theorem 1 also represent spurious minima for the under-parameterized d > k-case . This is obtained by appropriately padding the entries of the weight matrices by zeros ( see [ 6 , section 4.3 ] for details ) . The derivation of the respective Hessian spectrum is then an immediate application of [ 4 , section E.1 ] to the spectral analysis given in Theorem 2 below ( for the case where d = k ) . By contrast , studying the over-parameterized regime for which d < k requires qualitatively different tools and is outside the scope of this work . The analysis above reveals that not all local minima are alike : while the objective value of some minima decays like Θ ( 1/d ) , in other cases the objective value converges to the positive constant 1 2 − 1π . In particular , except for the global minima case 1a , all minima described above are spurious . The difference between the two types of behavior seems to lie in the limiting values of the entries of W ( d ) ; in the former , the diagonal mainly consists of ones , whereas in the latter mainly minus ones . This is consistent with the fact that all the teacher ’ s diagonal entries are ones . For reasons which are yet to be understood , empirically , the bias induced by Xavier initialization [ 27 ] seems to favor the class of minima for which the objective value decays as Θ ( 1/d ) . This is not to say that under Xavier initialization the expected value upon convergence must decrease with d. The objective value decays to zero at different rates ( see cases 2a , 3a , 4 above ) and depends on the probability to converge to a given type of minima . ( A possible proxy to the latter is the minima multiplicity stated in Theorem 1 . )

Summary: The paper theoretically study a one hidden layer neural network with ReLU activation trained using Gaussian data, and analytically derive the values of the different classes of local minima of this objective, and the spectrum of the Hessian at these local minima. The results show that as the degree of over-parameterization is increased, the local minima transform into saddle points in the loss landscape, and further find the exact fractional dimension at which this transition occurs.   Note: I am not familiar with the mathematical tools used in this paper. So my comments pertain to the utility of the findings and claims made in the paper.