Analytic Insights into Structure and Rank of Neural Network Hessian Maps

1 Introduction . Since the very infancy of neural networks , the Hessian matrix has been a central object of study . This is because the Hessian captures pairwise interactions of parameters via second-order derivatives of the loss function . As a result , the Hessian was productively employed , for instance , in ( quasiNewton ) optimization methods [ 1 , 2 ] , model design and pruning [ 3–5 ] , generalization [ 6 ] , network calibration [ 7 ] , automatically tuning hyper-parameters [ 8 ] . But , from the outset the main practical challenge has been its size , scaling quadratically with the model dimensionality . This makes the problem severe for today ’ s DNNs which have millions or even billions of parameters [ 9 , 10 ] . Consequently , most prior work has focused on designing scalable Hessian approximations , which either take the route of Hessian-vector products ( R-operator ) [ 11–13 ] or employ positive definite approximations by appealing to the Fisher information matrix . Additional approximations — without exception — are needed on top , such as diagonal approximations [ 3 , 14 ] in the former or K-FAC [ 15– 17 ] , restricted to layerwise or arbitrary blocks on the diagonal [ 18–20 ] , in the latter . ∗Detailed list of contributions are : Sidak first discovered that the Hessian rank formula , in an early form , holds experimentally to high fidelity , thus kick-starting the project . Sidak came up with the proof technique and proved Theorems 3 , 5 , 9 , 12 . Sidak wrote essentially the entire paper and noted the rank-deficiency interpretation . Gregor proved Lemma 8 , assisted in a part of Theorem 3 , empirically observed the eventual formula for the Hessian rank , and essentially ran all the experiments for the final submission and made the corresponding figures . Correspondence to sidak.singh @ inf.ethz.ch . 1The most recent version of our paper can be found at https : //arxiv.org/abs/2106.16225 and the code corresponding to the experiments is located at https : //github.com/dalab/hessian-rank . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . The goal of this paper is to advance the analytical understanding of the Hessian map of a neural network . We pursue the fundamental question of how the model architecture induces structural properties of the Hessian . In particular , we analyze the dimension of its range ( i.e. , the rank ) and identify the sources of rank deficiency . Understanding the range of the Hessian map , in turn , delivers insights into the important aspect of how gradients change between iterations . A reason why such an important direction currently remains sidelined is that non-linearities in a neural network result in an increased dependence on the data distribution , making a suitable theoretical analysis seem intractable . Following Saxe et al . [ 21 ] , Kawaguchi [ 22 ] , who delivered useful general insights on neural networks by looking at the linear case , we take a step back and rigorously characterize the range of the Hessian map and determine the resultant rank deficiency for deep linear networks . The key result of our paper is an exact formula along with tight upper bound on the rank of the Hessian — which effectively depend on the sum of hidden-layer widths . This stands opposed to the total number of parameters which are proportional to the sum of squared layer widths , thus implying a significant redundancy in the parameterization of neural networks ( see Fig . 1 ) . The exact quantification of the Hessian rank gives a precise yet interpretable ballpark on the inherent complexity of neural networks since rank naturally measures the effective number of parameters ( as best illustrated by the case of a quadratic minimization problem ) . This relationship is further reinforced by connection to the classical complexity measure of Gull [ 23 ] , MacKay [ 24 ] , which is equivalent to rank for a sufficiently small constant controlling the prior . Therefore , this sheds a novel perspective on the nature and degree of overparameterization in neural networks , and opens up interesting avenues for future investigation . Contributions . The main contributions of our paper can be summarized as follows : ( i ) Section 3 : We characterize the structure of the Hessian range by exhibiting it via matrix derivatives . ( ii ) Section 4 : We prove tight upper bounds and provide exact formulas on the Hessian rank that are neatly interpretable . To the best of our knowledge , this is the very first time that such formulas and bounds are made available for neural networks . ( iii ) Section 5.2 : In the non-linear case , we show that our ( linear ) rank formulae faithfully capture the numerical rank . ( iv ) Section 6 : We demonstrate via experiments and theory that such rank bounds also hold throughout the course of training . ( v ) Section 7.1 : In the non-linear case , we also provide a pessimistic yet non-trivial bound , which provably establishes degeneracy of the Hessian at the minimum . ( vi ) Section 7.2 , 5.1 : We extend our rank results to the case of bias and investigate the effects of architectural components ( such as width , depth , bias ) on rank . ( vii ) Appendix S8 : As a by-product , our analysis also reveals interesting properties of the Hessian spectrum , and we prove the presence of additional redundancies due to repeated eigenvalue plateaus . Related work . The study of the Hessian , in recent times , has been re-invigorated by the empirical observations of Sagun et al . [ 25 , 26 ] who noted a high degree of degeneracy experimentally , and characterized the spectrum as being composed of a bulk around zero and few outlier eigenvalues . Since then , works such as [ 27 , 28 ] have scaled the empirical analysis to bigger networks via efficient spectral density calculations and better explained the observations in [ 26 ] . However , a drawback is that due to their empirical nature , it remains hard to uncover the exact level of degeneracy via these methods [ 29 ] , and neither are the exact factors that affect the rank deficiency discerned . Another approach has been to fit models from random matrix theory to match the observed Hessian spectrum in neural networks [ 29 , 30 ] , however , such methods do not properly capture the sharp peak at zero observed in empirical spectra [ 31 ] . Yet others have investigated the properties of the spectrum in the asymptotic regime [ 32 , 33 ] , but the bounds , if any , are quite coarse ( see Fig . 4 ) . Moreover , a common issue underlying most of these approaches is that they are blind to the layerwise compositional structure of neural networks , and are often motivated by a black-box decomposition of the Hessian . In regards to indicating the parameter redundancy in neural networks , we are far from being the first work . Prior empirical studies have long reported similar observations , like in the form of , post-training pruning [ 3 , 4 , 19 , 20 , 34 ] or inherently contained sub-networks ( Lottery Ticket Hypothesis [ 35 ] ) or intrinsic dimension of the loss landscape [ 36 ] . Recently , [ 37 ] have experimentally argued for the effective dimensionality from [ 23 , 24 ] to be a good predictor of double descent [ 38 ] . Nevertheless , determining the precise extent of redundancy — and showing it provably — as well as the structural reasons behind it have remained illusory in such approaches . 2 Setup and Formalism . General notation . We use the shorthand , Wk : l , to refer to the matrix product chain Wk · · ·Wl , when k > l. When k < l , Wk : l will stand for the transposed product chain Wk ⊤ · · ·Wl⊤ . Besides , ⊗ denotes the Kronecker product of two matrices , vecr indicates the row-wise vectorization of matrices , rk refers to the rank of a matrix , and Ik is the identity matrix of size k × k. Deep Neural Networks ( DNNs ) . A feedforward DNN is a composition of maps , i.e. , F = FL ◦ · · · ◦ F 1 , where the l-th layer map F l : RMl−1 → RMl , with input dimension d : = M0 , output dimension K : = ML , total number of hidden neurons M : = ∑L−1 l=1 Ml . Each layer map is parameterized by a weight matrix Wl and applies an elementwise activation function σl . So we have , F l = σl ◦Wl with Wl ∈ RMl×Ml−1 and σl : R→ R .. For the sake of tractability , we will often investigate linear DNNs where σl = id , and so F ( x ) = WL:1x . For compactness , we also represent the entire set of parameters by θ : = { W1 , · · · , WL } , and where emphasis requires , we will subscript the DNN map F with it and write Fθ . Next , assume that we are given a dataset S = { ( xi , yi ) } Ni=1 of N input-output pairs , drawn i.i.d from an underlying distribution px , y . Our focus will be on the squared loss ( MSE ) , ℓx , y ( θ ) = 12∥y − ŷ∥ 2 and its residual δx , y : = ŷ − y = ∂ℓx , y∂ŷ , where ŷ = Fθ ( x ) is the DNN prediction . The population loss , L is : L ( θ ) = E px , y [ ℓx , y ( θ ) ] . Finally , we will analyze the Hessian matrix HL = ∂2L ∂θ ∂θ . Backpropagation in matrix derivatives . As all parameters are collected into matrices , we often work with matrix-matrix derivatives by vectorizing row-wise in the numerator ( Jacobian ) layout , i.e. , ∂Y ∂X : = ∂ vecr ( Y ) ∂ vecr ( X ) ⊤ , see [ 39 ] and Appendix S2.5 . Alongside this , we use the following rule : ∂AWB ∂W = A⊗ B⊤ , e.g . ∂F ∂Wk = WL : k+1 ⊗ x⊤W1 : k−1 ∈ RK×MkMk−1 . ( 1 ) By the usual chain rule ( backpropagation ) one has for a linear DNN , at a sample ( x , y ) : ∂ℓx , y ∂Wk = [ Wk+1 : Lδx , y ] ︸ ︷︷ ︸ backward ∈RMk · [ Wk−1:1x ] ⊤ ︸ ︷︷ ︸ forward ∈RMk−1 = Wk+1 : L [ WL:1 xx⊤ − y x⊤ ] W1 : k−1 . ( 2 ) The above gradient with respect to Wk is of first order in Wk itself and second order in the other weight matrices . Lastly , let us setup the following shorthand , Ω : = E [ δx , y x⊤ ] = E [ WL:1xx⊤ − yx⊤ ] , Σxx : = E [ xx⊤ ] , and Σyx : = E [ yx⊤ ] , which we will use throughout the paper . 3 Hessian Maps of Linear DNNs . 3.1 Hessian structure . The Hessian map has a natural block structure defined by the layers and their dimensionality . In order to leverage this structure , we directly take the derivative of the loss gradient in a matrix-by-matrix fashion . First , consider the k-th diagonal block of the Hessian , which is independent of y and is given by , HkkL : = ∂2L ∂Wk∂Wk = Wk+1 : LWL : k+1 ⊗ Wk−1:1 Σxx W1 : k−1 . This follows from the matrix-derivative rule in Eq . ( 1 ) along with the Eq . ( 2 ) and taking expectation . The calculation of the off-diagonal Hessian blocks ( kl-th block of size Mk Mk−1 × Ml Ml−1 ) involves the product rule . Note that the two occurrences of a weight matrix Wl , in Eq . ( 2 ) , are once non-transposed ( in WL:1xx⊤ ) and once transposed ( in Wk+1 : L or W1 : k−1 respectively for k < l or k > l ) . For simplicity , let us express these parts without adding them and directly write the other Hessian contribution with respect to the transposed matrix , giving : ∀ k ̸= l , H̃klL : = ∂2L ∂Wl∂Wk = Wk+1 : LWL : l+1 ⊗ Wk−1:1 Σxx W1 : l−1 . ( 3 ) ∀ k < l ĤklL : = ∂2L ∂Wl⊤∂Wk = Wk+1 : l−1 ⊗ Wk−1:1 Ω⊤ WL : l+1 . ( 4 ) ∀ k > l ĤklL : = ∂2L ∂Wl⊤∂Wk = Wk+1 : L ΩW1 : l−1 ⊗ Wk−1 : l+1 . ( 5 ) Equivalence to Gauss-Newton Decomposition . A common approach is to look at the Hessian map from the perspective of the Hessian chain rule , where we have that , HL = Ho +Hf , with Ho = E px , y [ ∇θF ( x ) ⊤ [ ∂2ℓx , y ] ∇θF ( x ) ] , and Hf = E px , y [ K∑ c=1 [ ∂ℓx , y ] c ∇2θ Fc ( x ) ] . For the MSE loss , the Gauss-Newton decomposition is in fact equivalent to what we discussed before ( see details in Appendix S1 ) , where Ho contains the blocks HkkL and H̃ kl L , while Hf consists of ĤklL ( although with the non-transposed matrix ) . Henceforth , we will refer to the first term Ho as the outer-product Hessian , while we coin the second-term Hf as the functional Hessian .

Summary: This paper studies the rank of Hessian for deep linear networks. Theoretically, with squared loss, and assuming the weight matrices are full-rank, this paper provides rank upper bound for outer-product, functional and full Hessian, and empirically verifies that the bounds are almost tight. This paper further shows empirically that the numerical rank with nonlinear networks can still be captured by the linear rank bound, and that the Hessian rank decreases during training. Finally, Hessian degeneracy results for 1-hidden-layer networks and extension to the case with bias are also provided.