user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Towards understanding the true loss surface of deep neural networks using random matrix theory and iterative spectral methods

Paper content:
The geometric properties of loss surfaces, such as the local flatness of a solution, are associated with generalization in deep learning.
The Hessian is often used to understand these geometric properties.
We investigate the differences between the eigenvalues of the neural network Hessian evaluated over the empirical dataset, the Empirical Hessian, and the eigenvalues of the Hessian under the data generating distribution, which we term the True Hessian.
Under mild assumptions, we use random matrix theory to show that the True Hessian has eigenvalues of smaller absolute value than the Empirical Hessian.
We support these results for different SGD schedules on both a 110-Layer ResNet and VGG-16.
To perform these experiments we propose a framework for spectral visualization, based on GPU accelerated stochastic Lanczos quadrature.
This approach is an order of magnitude faster than state-of-the-art methods for spectral visualization, and can be generically used to investigate the spectral properties of matrices in deep learning.
The extraordinary success of deep learning in computer vision and natural language processing has been accompanied by an explosion of theoretical (Choromanska et al., 2015a; b; Pennington & Bahri, 2017) and empirical interest in their loss surfaces, typically through the study of the Hessian and its eigenspectrum (Ghorbani et al., 2019; Li et al., 2017; Sagun et al., 2016; Wu et al., 2017) .
Exploratory work on the Hessian, and its evolution during training (e.g., Jastrzębski et al., 2018) , attempts to understand why optimization procedures such as SGD can discover good solutions for training neural networks, given complex non-convex loss surfaces.
For example, the ratio of the largest to smallest eigenvalues, known as the condition number, determines the convergence rate for first-order optimization methods on convex objectives (Nesterov, 2013) .
The presence of negative eigenvalues indicates non-convexity even at a local scale.
Hessian analysis has also been a primary tool in further explaining the difference in generalization of solutions obtained, where under Bayesian complexity frameworks, flatter minima, which require less information to store, generalize better than sharp minima (Hochreiter & Schmidhuber, 1997) .
Further work has considered how large batch vs small batch stochastic gradient descent (SGD) alters the sharpness of solutions (Keskar et al., 2016) , with smaller batches leading to convergence to flatter solutions, leading to better generalization.
These geometrical insights have led to generalization procedures, such as taking the Cesàro mean of the weights along the SGD trajectory , and algorithms that optimize the model to select for local flatness (Chaudhari et al., 2016) .
Flat regions of weight space are more robust under adversarial attack (Yao et al., 2018) .
Moreover, the Hessian defines the curvature of the posterior over weights in the Laplace approximation for Bayesian neural networks (MacKay, 1992; 2003) , and thus crucially determines its performance.
In this paper we use random matrix theory to analyze the spectral differences between the Empirical Hessian, evaluated via a finite data sample (hence related to the empirical risk) and what we term the True Hessian, given under the expectation of the true data generating distribution.
1 1 We consider loss surfaces that correspond to risk surfaces in statistical learning theory terminology.
In particular, we show that the differences in extremal eigenvalues between the True Hessian and the Empirical Hessian depend on the ratio of model parameters to dataset size and the variance per element of the Hessian.
Moreover, we show that that the Empirical Hessian spectrum, relative to that of the True Hessian, is broadened; i.e. the largest eigenvalues are larger and the smallest smaller.
We support this theory with experiments on the CIFAR-10 and CIFAR-100 datasets for different learning rate schedules using a large modern neural network, the 110 Layer PreResNet.
It is not currently known if key results, such as (1) the flatness or sharpness of good and bad optima, (2) local non-convexity at the end of training, or (3) rank degeneracy hold for the True Hessian in the same way as for the Empirical Hessian.
We hence provide an investigation of these foundational questions.
The geometric properties of loss landscapes in deep learning have a profound effect on generalization performance.
We introduced the True Hessian to investigate the difference between the landscapes for the true and empirical loss surfaces.
We derived analytic forms for the perturbation between the extremal eigenvalues of the True and Empirical Hessians, modelling the difference between the two as a Gaussian Orthogonal Ensemble.
Moreover, we developed a method for fast eigenvalue computation and visualization, which we used in conjunction with data augmentation to approximate the True Hessian spectrum.
We show both theoretically and empirically that the True Hessian has smaller variation in eigenvalues and that its extremal eigenvalues are smaller in magnitude than the Empirical Hessian.
We also show under our framework that we expect the Empirical Hessian to have a greater negative spectral density than the True Hessian and our experiments support this conclusion.
This result may provide some insight as to why first order (curvature blind) methods perform so well on neural networks.
Reported non-convexity and pathological curvature is far worse for the empirical risk than the true risk, which is what we wish to descend.
The shape of the true risk is particularly crucial for understanding how to develop effective procedures for Bayesian deep learning.
With a Bayesian approach, we not only want to find a single point that optimizes a risk, but rather to integrate over a loss surface to form a Bayesian model average.
The geometric properties of the loss surface, rather than the specific location of optima, therefore greatly influences the predictive distribution in a Bayesian procedure.
Furthermore, the posterior representation for neural network weights with popular approaches such as the Laplace approximation has curvature directly defined by the Hessian.
In future work, one could also replace the GOE noise matrix ε(w) with a positive semi-definite white Wishart kernel in order to derive results for the empirical Gauss-Newton and Fisher information matrices, which are by definition positive semi-definite and are commonly employed in second order deep learning (Martens & Grosse, 2015) .
Our approach to efficient eigenvalue computation and visualization can be used as a general-purpose tool to empirically investigate spectral properties of large matrices in deep learning, such as the Fisher information matrix.
Following the notation of (Bun et al., 2017 ) the resolvent of a matrix H is defined as
with z = x + iη ∈ C. The normalised trace operator of the resolvent, in the N → ∞ limit
is known as the Stieltjes transform of ρ.
The functional inverse of the Siteltjes transform, is denoted the blue function B(S(z)) = z.
The R transform is defined as
crucially for our calculations, it is known that the R transform of the Wigner ensemble is
Consider an n × n symettric matrix M n , whose entries are given by
The Matrix M n is known as a real symmetric Wigner matrix.
Theorem 2.
Let {M n } ∞ n=1 be a sequence of Wigner matrices, and for each n denote X n = M n / √ n.
Then µ Xn , converges weakly, almost surely to the semi circle distribution,
the property of freeness for non commutative random matrices can be considered analogously to the moment factorisation property of independent random variables.
The normalized trace operator, which is equal to the first moment of the spectral density
We say matrices A&B for which ψ(A) = ψ(B) = 0 4 are free if they satisfy for any integers n 1 ..
n k
E DERIVATION
The Stijeles transform of Wigners semi circle law, can be written as (Tao, 2012)
from the definition of the Blue transform, we hence have
Computing the R transform of the rank 1 matrix H true , with largest non-trivial eigenvalue β, on the effect of the spectrum of a matrix A, using the Stieltjes transform we easily find following (Bun et al., 2017) that
We can use perturbation theory similar to in equation equation 22 to find the blue transform which to leading order gives
setting ω = S M (z)
using the ansatz of
we find that S 0 (z) = S (w) (z) and using that B M (z) = 1/g (z) , we conclude that
and hence
and hence in the large N limit the correction only survives if S (w) (z) = 1/β
clearly for β → −β we have

assistant: text: Understanding the neural network Hessian eigenvalues under the data generating distribution.