Using Objective Bayesian Methods to Determine the Optimal Degree of Curvature within the Loss Landscape

1 INTRODUCTION . When training a neural network we aim to find a parametrization which minimizes the variance of the data around the model ’ s conditional mean value . A statistic which is reflective of this variance is known as a loss function and can be seen as creating a landscape mapping a model parametrization to a corresponding loss value . Thus , higher points in the landscape reflect higher loss values and a worse model parametrization . The saliency of other features of the loss landscape on the model performance are relatively less clear and in some cases are points of contention within the field . One such point is whether the width of a basin in the landscape surrounding a local minimum ( we will also refer to this as the width of the minimum ) is reflective of the ability of a model parametrization at the minimum to generalize to unseen data . It is a common notion that the wider the minimum in the landscape , as measured by the Hessian matrix of the loss function ( Keskar et al. , 2016 ; Dinh et al. , 2017 ) , the better the model parametrization will generalize . The intuition behind such a belief is simply that , wider minima reflect that a model will experience less deviation in its loss metric as a result of minor deviations of its parameter values . As a result the model is more robust than if it were to be parametrized by a very specific parameter set found at a sharp minimum . In this work we aim to demonstrate that the width of a minimum is a key feature of the loss landscape and provides significant information on the progress of the training of a model . We deviate , however , from the views of the field that the widest minima provide the best generalizability by reflecting that there is instead an optimal width or curvature around the parametrization with the best generalizability which is not necessarily the widest point in the landscape . To this end Section 2 provides the necessary background information that we will utilize in developing our theories which are presented in Section 3 . Section 4 and Section 5 then provide empirical evidence in support of the theoretical findings with Section 4 describing the methods employed to test the theories . Section 5 then provides and discusses the results of these empirical tests . Finally we conclude in Section 6 with our closing remarks . The contributions of this work are threefold . Firstly we evaluate the concept of Energy-Entropy competition of neural networks ( Zhang et al. , 2018 ) in the context of the bias-variance trade-off ( Geman et al. , 1992 ) and reflect that a correlation exists between energy and entropy as opposed to a competition or trade-off as was first presented . Secondly we utilize the Fisher Information of the loss landscape in the area of a minimum to reflect that an optimal level of curvature exists within the landscape which does not necessarily occur at the point in the landscape with the least curvature . Further to this , we provide a novel view on the overfitting of models to their training data using the loss landscape . Finally , the Fisher Information is utilized in defining the objective Bayesian prior known as the Jeffreys prior and we show that the test error of the model reaches its minimum value at the point in the landscape which corresponds to the Jeffreys prior . In addition , we show that at this point in the landscape the dimension of principal curvature of the model is at its maximum entropy . In doing so we also reflect the noise invariance of the dimension of principal curvature . 2 BACKGROUND . 2.1 FISHER INFORMATION AND UNINFORMATIVE ( JEFFREYS ) PRIORS . The Fisher Information ( which we denote by ζ ( θ ) ) is a metric dependent on the model parametrization which measures the amount of information that a sufficient statistic based on the observable data , such as the variance of the data around the model predictions ( Jaynes , 2003 ) , carries about an unknown parameter θ . In the case of a Gaussian model , the Fisher Information is equal to the Expected Hessian of the log-likelihood of the Gaussian . The necessary regulatory conditions for this equality to be true apply to the entire exponential family of distributions , however , in our case it is sufficient for this to hold for the Gaussian distribution ( Ly et al. , 2017 ) . One of the key properties of the Fisher Information Matrix is that its determinant is invariant under reparametrizations of a trained model . Thus , when the parameters used in modelling a distribution are changed , the Fisher Information in each dimension will change , however , the determinant or volume of information remains unchanged between the model parametrizations . The invariance property of the Fisher Information was the reason for its utilization in Jeffreys ( 1946 ) who sought to create a Bayesian prior with such an invariance property . The resultant prior is known as the Jeffreys prior and is shown in Equation 1 , whereH ( θ ) denotes both the Hessian and Expected Hessian matrices ( we will treat the Expected Hessian and Hessian interchangeably for the remainder of this work , as is common in the literature ( Zhang et al. , 2018 ; Karakida et al. , 2019 ) ) . P ( θ ) ∝ √ det ζ ( θ ) = √ detH ( θ ) ( 1 ) As has been shown in Jaynes ( 1968 ; 2003 ) the utility of the Jeffreys prior is not limited to the invariance property , as the Jeffreys prior is an example of an uninformative or objective prior , and as a result informs the posterior distribution as little as possible . The Jeffreys prior is thus used to reflect complete prior ignorance about the correct model parametrization , resulting in a posterior distribution with parameters completely determined by the observed data . A key perspective of this property is that the Jeffreys prior , thus , imposes a uniform distribution over the function space of the model , not the parameter space . This is due to the density of the prior being inversely proportional to the Hessian at a certain parametrization , and as such places higher density on parametrizations with a unique function approximation and low parameter variance . This would result in an even distribution over the function approximations and as a result the choice between function approximations is left to the model learning from the data . The relationship between the Hessian and the parameter variance of a model is discussed further in Section 2.2 below . 2.2 THE BIAS-VARIANCE DILEMMA , ENERGY-ENTROPY COMPETITION AND MINIMUM DESCRIPTION LENGTH . It is a well-known fact that a model learning to equate its conditional mean precisely to the values found in the training data is not always beneficial to the performance of the model on unseen data . In particular when we observe a decrease in training error but increase in validation or test error we say that the model is overfitting the training data ( Hawkins , 2004 ) . In Geman et al . ( 1992 ) it is shown empirically that to decrease the variance of the data around a model ’ s predictions ( reduce the training error ) it is necessary for the variance in the model parameters to increase . Further , Geman et al . ( 1992 ) reflect that a large parameter variance corresponds to the overfitting of the model to the training data . This trade-off between the bias of the model and the variance of its parameters is known as the Bias-Variance Dilemma ( Sammut & Webb , 2011 ) . In Geman et al . ( 1992 ) the only means presented to mitigate this dilemma is to obtain more training data . We see , however , that neural networks are capable of learning complex tasks with limited data and even generalize in spite of the Bias-Variance Dilemma . In Zhang et al . ( 2018 ) it is argued that the success of neural networks is due to a bias of Gradient Descent towards wider minima in the loss landscape . To reflect this , Zhang et al . ( 2018 ) derive a Bayesian posterior distribution for the parameters of a model given the training data . To derive this distribution , Zhang et al . ( 2018 ) utilize a Gaussian likelihood with conjugate Gaussian prior , which we generalize in Equation 2 by allowing any prior distribution exp ( h ( θ ) ) which results in a proper posterior distribution . The exponential term of this generalized prior h ( θ ) is seen as some function of θ while f ( xi , θ ) denotes the function approximation by the neural network . σ2i is the variance of the output corresponding to data point xi , and finally yi is the true output for a particular xi in the training data . The derivation of Equation 2 can be found in Appendix A.1 . P ( θ|X ) = 1 Z exp [ − ( P∑ i=1 ( yi − f ( xi , θ ) ) 2 2σ2i − h ( θ ) + 1 2 log det ( H ( θ ) ) ) ] ( 2 ) From Equation 2 we see that to maximize the probability of a parametrization we must simultaneously maximize the model likelihood ( by minimizing the first term in the exponential ) , the prior probability of the parametrization and the model entropy . The model entropy is reflected by the final term in the exponential and is inversely proportional to the Hessian of the loss landscape at the parametrization . Using the posterior distribution Zhang et al . ( 2018 ) note that maximizing the model likelihood is not the only factor which should be used in determining the model parametrization , and in some cases it may be beneficial to trade-off some training error for an increase in model entropy , which the authors called Energy-Entropy competition . Zhang et al . ( 2018 ) state that the bias of Gradient Descent towards wider minima , with smaller Hessian values , results in the model naturally maximizing entropy , aiding in its generalizability . We see , however , from the Bias-Variance Dilemma that by reducing the bias of the model on the training data , and increasing its likelihood , that the model entropy will naturally also increase due to the higher variance in the parameter values . With the perspective of both the Bias-Variance Dilemma and Energy-Entropy competition we see that wide points in the loss landscape have been related to both overfitting and improved generalizability of a model parametrization . Thus , from one perspective we aim for sharp minima within the landscape and from the other we should aim for wide minima . The issue of the width of a minimum is further confounded by Dinh et al . ( 2017 ) which states that the width of a minimum is not a consistent indicator of the ability of a model to generalize . The impact of the width of a minimum in the landscape is still an open question and one which we try address in this work . The Minimum Description Length ( MDL ) Principle is an information theoretic principle which states that the optimal model for a set of data provides the best compression of the data ( Rissanen , 1978 ) . In other words the optimal model is the simplest model which incurs the least training error . This principle is another example of the trade-off between model complexity and minimizing the model bias . Due to its assertion that the optimal model is the simplest unbiased one the MDL Principle is the mathematical formulation of Occam ’ s Razor , and is expressed by Equation 3 , which reflects that to compress the data D optimally we must find the parametrization with the net minimum entropy in the parameter space L ( θ ) and in the data given the parametrization L ( D|θ ) . L ( D ) = min θ∈ϑ ( L ( θ ) + L ( D|θ ) ) ( 3 ) For the exponential family of likelihood distributions the Jeffreys prior is used to enforce the MDL property on the posterior distribution and results in a minimax optimal posterior , which is to say that the maximal risk of the model parametrization is minimal out of all unbiased parametrizations ( Lehmann & Casella , 2006 ) . The minimax optimality property , thus , provides the lowest upper bound of the risk for all model parametrizations . Thus the MDL property is related to the BiasVariance Dilemma and MDL posterior distributions aim to avoid overfitting . A final necessary principle which encompasses all of the topics above is the Likelihood Principle ( Jaynes , 2003 ) , which states that within the context of a specified model , the likelihood distribu- tion L ( D|θ ) trained from data D contains all the information about the model parameters θ that is contained in D .

Summary: This paper targets at a deep learning theory contribution based on information geometry. This contribution is tightly based on Zhang et al. (2018) and explains the generalization of deep learning from a Bayesian perspective. The main contribution the authors claimed is an optimal degree of curvature exist which gives the best generalization guarantees, which is in contrast to the commonly perceived "the wider the better".