Hessian Eigenspectra of More Realistic Nonlinear Models

1 Introduction . The Hessian is ubiquitous in applied mathematics , statistics , and machine learning ( ML ) . Given a ( loss ) functionL ( w ) of some parameters w ∈ Rp , the Hessian H ( w ) ∈ Rp×p is defined as the second derivative of the loss function with respect to the model parameter , i.e. , H ( w ) = ∂L ( w ) / ( ∂w∂wT ) . When a ML model is being trained , it is common to parameterize that model by w , and then train that model by minimizing some ( smooth ) loss function L ( w ) , with the associated Hessian H ( w ) , e.g. , by backpropagating the error to improve w [ 26 ] . Alternatively , once a ML model is trained , the Hessian ( and the related Fisher information matrix [ 66 , 68 ] ) can be examined to identify outliers , perform diagnostics , and/or engage in other sorts of model validation [ 31 , 79 , 62 ] . For convex problems , the Hessian H ( w ) provides detailed information on how to adjust the gradient to achieve improved convergence , e.g. , in Newton-like methods . For non-convex problems , the properties of the local loss “ landscape ” around a given point w in the parameter space is of central ∗Work done at ICSI and Department of Statistics , University of California , Berkeley , USA . †We refer the readers to an extended version of this article [ 43 ] for detailed proofs and more discussions . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . significance [ 18 , 36 , 13 , 39 , 77 , 78 , 79 ] . In this case , most obviously , the signs of the smallest and largest Hessian eigenvalue can be used to test whether a given w is a local maximum , local minimum , or a saddle point . More subtly , the Hessian eigenvalue distribution characterizes the local curvature of the loss function and provides direct access to , for instance , the fraction of negative Hessian eigenvalues that determines the number of ( local ) descent directions , a quantity that is directly connected to the rates of convergence of various optimization algorithms [ 33 ] . For theoretical analysis of neural network ( NN ) models , Hessian eigenspectra are often assumed to follow well-known random matrix distributions such the Marc̆enko–Pastur law [ 45 ] or the Wigner ’ s semicircle law [ 70 ] . This enables one to use Random Matrix Theory ( RMT ) , but it involves ( for NNs , at least ) making relatively strong simplifying assumptions ( e.g. , the Hessian can be decomposed as the sum of the two freely independent matrices , the residual error , data feature , and weights are all composed of i.i.d . zero mean normal random variables ) [ 57 , 58 , 15 ] . A somewhat more realistic setup involves using a so-called spiked model ( or a spiked covariance model ) [ 3 , 5 , 41 ] . In this case , the matrix follows a signal-plus-noise model and consists of full rank random noise matrix and low rank statistical information structure.3 The “ signal ” eigenvalues are generally larger than the noisy “ bulk ” eigenvalues ; and the maximum eigenvalues , when isolated from the bulk , are referred to as the “ spikes. ” A substantial theory-practice gap exists , however . In both toy examples [ 27 ] and state-of-the-art NN models [ 77 , 78 , 79 , 80 , 64 , 20 ] , the strong simplifying assumptions are far from satisfactory . ( A similar theory-practice gap has been observed for other NN matrices to which RMT has been applied , perhaps most notably weight matrices [ 47 , 48 , 46 ] . ) A more precise understanding of the Hessian eigenspectra ( and its dependence on the input data structure , the underlying response model and model parameters , as well as the loss function ) for more realistic models is needed . 1.1 Our approach . In this article , we address these issues , in a setting that is simple enough to be analytically tractable but complex enough to shed light on realistic large-scale models . We consider a family of generalized generalized linear models ( G-GLMs ) that extends the popular generalized linear model ( GLM ) [ 19 , 31 ] ; and we show that , even for such simple models , the key simplifying assumptions used in previous theoretical analyses of Hessian can be very inexact . In particular , apart from a few special cases ( including linear least squares and logistic regression with homogeneous features ) , most Hessians of G-GLMs are not close to the Marc̆enko–Pastur and/or the semicircle law . Instead , the corresponding Hessian depends on the input feature structure , the underlying response model , and the loss function , in a more involved fashion that can be precisely characterized by the proposed analysis . The G-GLM describes a generalized linear relation between the input feature xi ∈ Rp and the corresponding response yi , in the sense that there exists some parameters w∗ ∈ Rp such that for given wT∗xi , the response yi is independently drawn from yi ∼ f ( y | wT∗xi ) ( 1 ) for some conditional density function f ( · | · ) . This extends the classical GLM such as logistic model : P ( y = 1 | wT∗x ) = ( 1 + e−w T ∗x ) −1 , y ∈ { −1 , 1 } , ( 2 ) and covers a large family of models in applications in statistics and ML . Other examples include : ( i ) the ( noisy ) nonlinear factor model where y ∼ N ( g ( wT∗x ) , σ2 ) for some nonlinear g : R→ R and σ > 0 [ 14 ] ; ( ii ) the ( noiseless ) phase retrieval model with y = ( wT∗x ) 2 , in which case one wishes to reconstruct w∗ from its ( squared ) magnitude measurements [ 21 ] ; and ( iii ) the single-layer NN model y = σ ( wT∗x ) for some nonlinear activation function σ ( t ) such as the tanh-sigmoid σ ( t ) = tanh ( t ) . For a given training set { ( xi , yi ) } ni=1 of size n , the standard approach to obtain/recover the parameter w∗ ∈ Rp of a G-GLM is to solve the following optimization problem min w L ( w ) = min w 1 n n∑ i=1 ` ( yi , w Txi ) , ( 3 ) for some loss function ` ( y , h ) : R × R → R , e.g. , the negative log-likelihood of the observation model within the maximum likelihood estimation framework [ 31 ] such as the logistic loss ` ( y , h ) = 3Since the same informative pattern is repeated in each row or column of the matrix . ln ( 1 + e−yh ) in the case of logistic model ( 2 ) . In many applications , however , the optimization problem in ( 3 ) may not be convex , for example to achieve superior robustness and/or accuracy [ 49 , 71 , 10 ] , and can be NP-hard in general ( the noiseless phase retrieval model y = ( wT∗x ) 2 with the square loss ` ( y , h ) = ( y−h2 ) 2 as an example [ 9 ] ) . As we shall see , in such non-convex G-GLMs , the dominant Hessian eigenvector can be shown , in some cases , to positively correlate with the sought-for parameter w∗ and therefore be used as the initialization of gradient descent methods [ 9 , 37 , 34 ] . This particularly motivates our study of the possible isolated Hessian eigenvalue-eigenvector pairs . 1.2 Our main contributions . The main contribution of this work is the exact characterization of Hessian eigenspectra for the family of G-GLMs , in the high-dimensional regime where the feature dimension p and the sample size n are both large and comparable . Precisely , we establish : 1. the limiting eigenvalue distribution of the Hessian matrix ( Theorem 1 ) ; and 2. the behavior of ( possible ) isolated eigenvalue-eigenvector pairs ( Theorem 2 and 3 ) , as a function of the dimension ratio c = lim p/n , feature statistics , the loss function ` in ( 3 ) , and the underlying response model in ( 1 ) . Our results are based on a technical result of independent interest : 3. a deterministic equivalent ( Theorem 4 ) of the random resolvent Q ( z ) = ( H− zIp ) −1 , for z ∈ C not an eigenvalue of H , of the generalized sample covariance:4 H ≡ H ( w ) = 1n ∑n i=1 ` ′′ ( yi , w Txi ) xix T i ≡ 1nXDX T , ( 4 ) for X = [ x1 , . . . , xn ] ∈ Rp×n , D ≡ diag { ` ′′ ( yi , wTxi ) } ni=1 ∈ Rn×n , and ` ′′ ( y , h ) ≡ ∂2 ` ( y , h ) /∂h2 , as n , p → ∞ with p/n → c ∈ ( 0 , ∞ ) , under the setting of generic Gaussian feature xi ∼ N ( µ , C ) , for µ ∈ Rp and positive definite covariance C ∈ Rp×p . We also demonstrate our results empirically by showing that : 4. for a given response model ( 1 ) , the Hessian eigenvalue distribution depends on the choice of loss function and the data/feature statistics in an intrinsic manner , e.g. , bounded versus unbounded support and single- versus multi-bulk in Fig 2 ; and 5. there may exist two qualitatively different spikes—one due to data signal µ and the other due to w∗ or w and thus the underlying model—which may appear on different sides of the main bulk , and their associated phase transition behaviors are characterized ( Fig 4 versus 5 ) . To have a more clear picture of our contribution , we compare , in Fig 1a and 1b , the Hessian eigenvalues for the logistic model ( 2 ) with the logistic loss ` ( y , h ) = ln ( 1 + e−yh ) ( which is the loss of choice within the maximum likelihood framework ) , for different choices of w in the parameter space . A nontrivial interplay between the response model , feature statistics and the parameter w is reflected by the range of the Hessian eigenvalue support and an additional right-hand spike in Fig 1b , as confirmed by our theory . For phase retrieval model y = ( wT∗x ) 2 with square loss ` ( y , h ) = ( y − h2 ) 2/4 , the non-convex nature of the problem is reflected by a ( relatively large ) fraction of negative Hessian eigenvalues in Fig 1c . We also note that the top eigenvector ( that corresponds to the largest eigenvalue ) contains structural information of the underlying model , in the sense that it is positively correlated with w∗ , as predicted by our theory . This is indeed connected to the Hessian-based initialization scheme widely used in non-convex problems [ 8 , 37 , 34 , 40 , 2 , 51 ] . We conclude by emphasizing that , by focusing on the simple yet fundamental G-GLM , we obtain results that improve upon and are different than previous efforts in the following aspects : ( i ) We provide precise asymptotic characterizations of the Hessian eigenspectra that go beyond , e.g. , [ 7 ] , where only Hessian lower bounds are given in the case of logistic model with logistic loss : our methodology and theoretical results hold much more generally for the family of G-GLM with arbitrary loss . As illustrating examples , we discuss linear least squares in Sec 3.1 , logistic model with different choices of loss function in Fig 2 , phase retrieval model in Figure 1c , and more in [ 43 , Sec 4 ] . ( ii ) We extend the results in [ 57 , 53 , 44 , 50 ] to G-GLMs by considering generic data statistics and loss function , whereas in previous efforts only much more homogeneous models are 4Here we consider w independent of xi , yi . With some additional effort , our results extend to the case where w takes some simple explicit function form of X and y , e.g. , w is the least square solution to ( X , y ) . discussed , and sometimes under unrealistic assumption , e.g. , the Hessian can be decomposed as the sum of two freely independent matrices , and the residual error , data feature , and weights are all composed of i.i.d . zero mean normal random variables [ 57 , 58 ] . ( iii ) Instead of focusing solely on the main eigenvalue bulk as in [ 57 , 58 ] , our results also shed novel light on the isolated eigenvalues ( above and/or below the bulk ) that are empirically observed in the Hessian of modern NNs [ 60 , 22 , 42 , 53 , 52 ] , as well as on the associated eigenvectors that are shown closely connected to NN training dynamics [ 28 ] . Also , relative to [ 57 , 58 ] , we show qualitatively different behaviors for the Hessian eigenspectra , e.g. , bounded versus unbounded support , single- versus multi-bulk as in Figure 2 . To our knowledge , these are not covered in the existing Hessian literature . 0.1 0.2 0.3 0.4 0 2 4 6 E ig en va lu es Empirical Theory ( a ) Logistic , w = w∗ = µ 0.2 0.4 0 2 4 0.2 0.4 0 2 4 ( b ) Logistic , indep . w −10 0 0 0.1 0.2 −10 0 0 0.1 0.2 −10 0 0 0.1 0.2 ( c ) Phase retrieval model Top eigenvector Theory ( d ) Eigenvector in Fig 1c Figure 1 : Illustration of our main results : eigenspectral properties of the Hessian of G-GLMs with p = 800 , n = 6 000 and C = Ip . Fig 1a versus 1b : absence versus presence of a right-hand side spike for different choices of w , logistic model ( 2 ) with logistic loss , and w∗ = µ ∼ N ( 0 , Ip/p ) . Fig 1c versus 1d : the Hessian eigenspectra have a rather different shape ( as opposed to the Marc̆enkoPastur-like in Fig 1a and 1b ) for the ( non-convex ) phase retrieval model ( 1c ) and the top eigenvector is known in this case to be a ( noisy ) estimate of w∗ ( 1d ) , as confirmed by our theory . With square loss ` ( y , h ) = ( y − h2 ) 2/4 , w∗ = [ −2 · 1p/2 ; 2 · 1p/2 ] / √ p , w ∼ N ( 0 , Ip/p ) and µ = 0 .

Summary: This paper studies the spectrum of empirical cost functions used with data from generalized generalized linear models (G-GLM). Such models depend on some parameters $w^* \in \mathbb{R}^p$ that link the input features $x_i \in \mathbb{R}^p$ to the responses $y_i \in \mathbb{R}$ by the relation $y_i \sim  f(y |  (w^*)^T x_i)$ where $f(\cdot | \cdot)$ is some conditional density function. Given a training set $(x_i, y_i)$ for $i=1, \dots, n$ a popular approach to estimate $w_*$ is to minimize the cost function $ L(w) = \sum_{i=1}^n \ell(y_i, x_i^T w) $ for some loss $\ell$ that is carefully chosen.  This framework is quite general and encompass many classical models such as the standard linear model, the logistic model or the phase retrieval problem.   The authors studies this model in the setting where - $n,p \to \infty$, while $n/p$ converges to some positive number. - $x_i \sim_{iid} \mathcal{N}(\mu, C)$ - $\|w\|, \|w_*\|, \| \mu \|, \|C\| = O(1)$   Under these assumptions, they provide a precise assymptotic caracterization of the eigenvalues and eigenvectors of the Hessian $ H = \frac{\partial^2 L}{\partial w \partial w}(w). $ More precisely:  1. They show that the empirical distribution of the eigenvalues of $H$ converges weakly almost surely to a probability measure $\mu$ defined by an equation on its Stieltjes transform. 2. When $\mu$ has a bounded support, they caracterize precisely the assymptotic postitions of potential isolated eigenvalues. When such isolated eigenvalues exists, the authors also compute the limiting angle between the correspondeing eigenvectors and the vectors $w,w_*,\mu$. 3. The authors illustrate their results on popular models and losses, showing the existence of interesting phenomena. In particular, they show that isolated eigenvalues can not only appears due to 'signal' in the data (large $\|\mu\|$) but also appear because of the model itself.   The main results are obtained through an so-called 'deterministic equivalent', that allows to study precisely the resolvant of the Hessian $H$.