Using Objective Bayesian Methods to Determine the Optimal Degree of Curvature within the Loss Landscape
Authors: Devon Jarvis
Richard Klein
Benjamin Rosman
Venue:  (2020)
Keywords: Objective Bayes
Information Geometry
Artificial Neural Networks

Abstract: The efficacy of the width of the basin of attraction surrounding a minimum in parameter space as an indicator for the generalizability of a model parametrization is a point of contention surrounding the training of artificial neural networks, with the dominant view being that wider areas in the landscape reflect better generalizability by the trained model. In this work, however, we aim to show that this is only true for a noiseless system and in general the trend of the model towards wide areas in the landscape reflect the propensity of the model to overfit the training data. Utilizing the objective Bayesian (Jeffreys) prior we instead propose a different determinant of the optimal width within the parameter landscape determined solely by the curvature of the landscape. In doing so we utilize the decomposition of the landscape into the dimensions of principal curvature and find the first principal curvature dimension of the parameter space to be independent of noise within the training data.

Under review as a conference paper at ICLR 2020
USING OBJECTIVE BAYESIAN METHODS TO DETER- MINE THE OPTIMAL DEGREE OF CURVATURE WITHIN THE LOss LANDSCAPE
Anonymous authors Paper under double-blee blind review
ABSTRACT
The efficacy of the width of the basin attraction surrounding a minimum parameter space an indicator for the generalizability of model parametriza- tion is point of contention surrounding the training of artificial neural networks, with the dominant view being that wider areas in the landscape reflect better gen- cralizability by the trained model. In this work, however, we aim to show that this is only true for noiseless system and in general the trend of the model to- wards wide areas in the landscape reflect the propensity of the model to overfit the training data. Utilizing the objective Bayesian (Jeffreys) prior we instead pro- pose different determinant of the optimal width within the parameter landscape determined solely by the curvature of the landscape. In doing we utilize the de- composition of the landscape into the dimensions of principal curvature and find the first principal curvature dimension of the parameter space to be independent of noise within the training data.
INTRODUCTION
When training a neural network we aim find parametrization which minimizes the variance of the data around the model's 's conditional mean value. A statistic which is reflective of this variance is known as loss function and can seen as creating a landscape mapping a model parametrization to corresponding loss value. Thus, higher points in the landscape reflect higher loss values and a worse model parametrization. The saliency of other features of the loss landscape on the model performance are relatively less clear and in some cases are points of contention within the field. One such point is whether the width of a basin in the landscape surrounding a local minimum (we will also refer to this as the width of the minimum) is reflective of the ability of model parametrization at the minimum to generalize to unseen data. a common notion that the wider the minimum in the landscape, measured by the Hessian matrix of the loss function (Keskar et al., 2016 Dinh et al. 2017). the better the model parametrization will generalize. The intuition behind such a belief simply that, wider minima reflect that model will experience less deviation in its loss metric as a result of minor deviations of its parameter values. As a result the model is more robust than if it were to be parametrized by a very specific parameter set found at a sharp minimum.
In this work we aim to demonstrate that the width of a minimum is a key feature of the loss land- scape and provides significant information on the progress of the training of model. We deviate, however, from the views of the field that the widest minima provide the best generalizability by re- flecting that there is instead an optimal width or curvature around the parametrization with the best generalizability which is not necessarily the widest point in the landscape. To this end Section 2 provides the necessary background information that we will utilize in developing our theories which presented in Section 3 Section 4 and Section 5 then provide empirical evidence in support of the theoretical findings with Section 4 describing the methods employed to test the theories. Section 5 then provides and discusses the results of these empirical tests. Finally conclude in Section with our closing remarks.
The contributions of this work are threefold. Firstly we evaluate the concept of Energy-Entropy competition of neural networks (Zhang et al.) 2018) in the context of the bias-variance trade-off (Geman et al. 1992) and reflect that correlation exists between energy and entropy as opposed to

---

Under review as a conference paper at ICLR 2020
competition trade-off as was first presented. Secondly we utilize the Fisher Information of the loss landscape in the area of a minimum to reflect that an optimal level of curvature exists within the landscape which does not necessarily occur at the point in the landscape with the least curvature. Further to this, we provide a novel view on the overfitting of models to their training data using the loss landscape. Finally, the Fisher Information is utilized in defining the objective Bayesian prior known as the Jeffreys prior and we show that the test error of the model reaches its minimum value the point in the landscape which corresponds to the Jeffreys prior. In addition, we show that at this point in the landscape the dimension of principal curvature of the model is at its maximum entropy. In doing so we also reflect the noise invariance of the dimension of principal curvature.
2
BACKGROUND
2.1 FISHER INFORMATION AND UNINFORMATIVE (JEFFREYS) PRIORS
The Fisher Information (which we denote by ((0) a metric dependent on the model parametriza- tion which measures the amount of information that sufficient statistic based on the observable data, such as the variance of the data around the model predictions (Jaynes] 2003), carries about an unknown parameter 0. In the case of a Gaussian model, the Fisher Information is equal to the Expected Hessian of the log- log-likelihood of the Gaussian. The necessary regulatory conditions for this equality to be true apply to the entire exponential family of distributions, however, in case it is sufficient for this to hold for the Gaussian distribution (Ly et al. 2017). One of the key properties of the Fisher Information Matrix is that its determinant is invariant under reparametrizzaions of trained model. Thus, when the parameters used in modelling a distribution are changed, the Fisher Information in each dimension will change, however, the determinant or volume of information remains unchanged between the model parametrizations.
The invariance property of the Fisher Information was the reason for its utilizzation in Jeffreys (946) who sought to create Bayesian prior with such an invariance property. The resultant prior is known as the Jeffreys prior and is shown in Equation| [ where (0) denotes both the Hessian and Expected Hessian matrices (we will treat the Expected Hessian and Hessian interchangeably for the remainder of this work, as is common in the literature (Zhang et al. 2018 Karakida et al., 2019)).
P(O) x det ((0) v det (0)
(1)
As has been shown in Jaynes (1968] 2003) the utility of the Jeffreys prior is not limited to the invariance property, as the Jeffreys prior is an example of an uninformative or objective prior, and as a result informs the posterior distribution as little as possible. The Jeffreys prior is thus used to reflect complete prior ignorance about the correct model parametrization, resulting in a posterior distribution with parameters completely determined by the observed data. A key perspective of this property is that the Jeffreys prior, thus, imposes a uniform distribution over the function space of the model, not the parameter space. This is due to the density of the prior being inversely proportional to the Hessian at a certain parametrization, and as such places higher density on parametrizations with a unique function approximation and low parameter variance. This would result in an even distribution over the function approximations and as a result the choice between function approximations is left to the model learning from the data. The relationship between the Hessian and the parameter variance of a model is discussed further in Section below.
2.2 THE BIAS- VARIANCE DILEMMA, ENERGY-ENTROPY COMPETITION AND MINIMUM DESCRIPTION LENGTH
a well-known fact that a model learning to equate its conditional mean precisely to the values found in the training data is not always beneficial to the performance of the model on unseen data. In particular when we observe a decrease in training error but increase in validation or test error we say that the model is overfitting the training data (Hawkins) 2004). In Geman et al. (1922) it is shown empirically that to decrease the variance of the data around a model's 's predictions (reduce the training error) it is necessary for the variance in the model parameters to increase. Further, Geman jet al. (1992) reflect that a large parameter variance corresponds to the overfitting of the model to the training data. This trade-off between the bias of the model and the variance of its parameters is
2

---

Under review as a conference paper at ICLR 2020
known the Bias- Variance Dilemma (Sammut & Webb 2011). In Geman et al. (1992) the only means presented to mitigate this dilemma is to obtain more training data.
We see, however, that neural networks are capable of learning complex tasks with limited data and even generalize in spite of the Bias- Variance Dilemma. In Zhang et al. (2018) it is argued that the success of neural networks is due to a bias of Gradient Descent towards wider minima in the loss landscape. To reflect this, Zhang et al. (2018) derive a Bayesian posterior distribution for the parameters of model given the training data. To derive this distribution, Zhang al. (2018) utilize Gaussian likelihood with conjugate Gaussian prior, which we generalize in Equation 2 by allowing any prior distribution exp(h(0)) which results in proper posterior distribution. The exponential term of this generalized prior h(0) is seen as some function of 0 while fz; 0) denotes the function approximation by the neural network, of is the variance of the output corresponding to data point Ti, and finally is the true output for a particular z; in the training data. The derivation of Equation 2 can be found Appendix A
(vs f(z,, 20?
P(O Z exp
Σ
h(0) 22 log del (H(())
(2)
From Equation we see that to maximize the probability of a parametrization we must simultane- ously maximize the model likelihood (by minimizing the first term the exponential), the prior probability of the parametrization and the model entropy. The model entropy is reflected by the final term in the exponential and is inversely proportional to the Hessian of the loss landscape at the parametrization. Using the posterior distribution Zhang et al. (2018) note that maximizing the model likelihood is not the only factor which should be used in determining the model parametriza- tion, and in some cases it may be beneficial to trade-off some training error for increase in model entropy, which the authors called of Gradient Descent towards wider minima, with smaller Hessian values, results in the model nat- urally maximizing entropy, aiding in its generalizability. We see, however, from the Variine Dilemma that by reducing the bias of the model on the training data, and increasing its likelihood, that the model entropy will naturally also increase due to the higher variance in the parameter values.
competition. Zhang et al. (2018) state that the bias
With the perspective of both the Varrinar Dilemma and Energy-Entropy competition we see that wide points in the loss landscape have been related to both overfitting and improved generaliz- ability of a model parametrization. Thus, from one perspective we aim for sharp minima within the landscape and from the other we should aim for wide minima. The issue of the width of minimum further confounded by Dinh et al. (2017) which states that the width of a minimum is not a con- sistent indicator of the ability of a model to generalize. The impact of the width of a minimum in the landscape is still an open question and one which we try address in this work.
The Minimum Description Length (MDL) Principle is an information theoretic principle which states that the optimal model for a set of data provides the best compression of the data (Rissa-) nen. 1978). In other words the optimal model is the simplest model which incurs the least training error. This principle is another example of the trade-off between model complexity and minimiz- ing the model bias. Due its assertion that the optimal model is the simplest unbiased one the MDL. Principle is the mathematical formulation of Occam's 's Razor, and is expressed by Equation 3 which reflects that to compress the data D optimally we must find the parametrization with the net minimum entropy in the parameter space L()) and in the data given the parametrization L(D ().
L(D) = -min (L(0) + L(D(0)) 8E0
(3)
For the exponential family of likelihood distributions the Jeffreys prior is used to enforce the MDL. property on the posterior distribution and results in minimax optimal posterior, which is to say that the maximal risk of the model parametrization is minimal out of all unbiased parametrizations (Lehmann & Casella 2006). The minimax optimality property, thus, provides the lowest upper bound of the risk for all model parametrizations. Thus the MDL property is related to the Bias- Variance Dilemma and MDL posterior distributions aim to avoid overfitting.
A final necessary principle which encompasses all of the topics above is the Likelihood Principle Jaynes 2003), which states that within the context of a specified model, the likelihood distribu-
3

---

Under review as a conference paper at ICLR 2020
tion L(D(0) trained from data D contains all the information about the model parameters @ that is contained in D.
2.3 PRINCIPAL CURVATURE
The Jeffreys prior, and by extension the Fisher Information, finds further utility in its use as a right Haar measure for the parameter space of a normal distribution (Berger 2013). The Haar measure is used to assign an invariant volume to a subset of locally compact topological groups and, thus, forms an integral over the compact groups. In the case of the parameter space of the normal distribution the topological groups will be of parametrizations with similar function approximations and, thus, similar loss metric within the basin surrounding local minimum in the loss landscape. Further, we note that the parameter space of a probabilistic model forms a statistical manifold and by extension a Riemannian manifold (Rao) 1945). The metric tensor for statistical manifolds is the Fisher Infor- mation metric (Skovgaard 1984), defined as the expected value of the individual elements of the Fisher Information matrix, which forms the tangent space of such manifolds. As stated Section 2.1 in the case of Gaussian parameter spaces the Fisher Information Matrix can be equally derived as the Hessian matrix of the loss function relative to the model parameters. This is significant as the Hessian matrix is used in the area of critical point on Riemannian manifold for obtaining the shape operator (Spivak) 11970,, and as a result the principal curvatures at the point (Portcous) [200I). In the case of a Gaussian parameter space the shape operator is the Gaussian curvature detined as the determinant of the Hessian matrix (0)) (Koenderink & Van Doom 1992). The princi-
pal curvatures are defined as the eigenvalues of the Hessian matrix and decompose the manifold
into orthogonal dimensions of curvature, with the first eigenvector reflecting the dimension of most curvature.
It is important to note that while the parameter space of a statistical model forms a Riemannian man- ifold, when parametrized by an weeel-determiied model such neural network, the parameter space will not be Riemannian but rather mimi-Riemannian, due to the fact that the Fisher Informa- tion metric will no longer be defined over the entire manifold. Such undefined points for the metric are a result of singular metric tensor at the model parametrization and occur due to the covariance of parameters within the model. Covariant parameters necessarily occur with the addition of hidden layers to the model and result in dimensions on the manifold in which the parameters may be varied without altering the behaviour of the model. This results in dimensions of no curvature along the manifold. As seen in Section [S] this is not destructive point for the training procedure, however, we must necessarily remain cognisant of such covariant dimensions along the statistical manifold.
MODEL ENTROPY, THE LOSS LANDSCAPE AND GENERALIZATION
The aim of training a neural network is to find the most probable parametrization for a model as determined by the posterior probability reflected by Equation ? This is achieved by maximizing the combination of the likelihood, prior probability and entropy of the model parametrization. The likelihood we increase normally by decreasing the variance of the training data around the model predictions. The entropy term we have no direct control over as the landscape is completely deter- mined by the data, the sufficient statistic being used to determine the parameters (which is the loss metric) and the hypothesis (the model architecture being trained). So the only component of the posterior left to be determined is the prior. As with most work in Bayesian statistics this is the most difficult part and must be treated with great care. There are presently two common approaches to setting this prior distribution, the first of which being to not specify one, or more precisely use an implicit uniform prior (Chaudhari & Soatto 2018) and, thus, use maximum likelihood estimation to determine the parameter values. The second common approach is to utilize a conjugate Gaussian with a mean of for the prior. practice this method takes the form of L.2 regularization, also known as weight decay (Krogh & Hertz 1992). with h(()) x ||0||2 in Equation 2 Neither approach has proven to be sufficient consistently for deterring models from overfitting, without introducing a form of bias, due to their unjustified assumptions about the correct model parametrization. This is relatively clear in the conjugate Gaussian prior approach which assumes a mean of 0 for the pa- rameter values. In case where we have explicit prior knowledge that such mean and distribution is in-fact correct for the model parameters then this would be a correct approach, however, in al- most every case we are completely ignorant the values of the true parametrization and, thus, we
4

---

Under review as a conference paper at ICLR 2020
would be biasing models to some degree by using this prior. It is necessary when developing an unbiased model that this absence of prior knowledge be reflected in the training procedure.
The source of error from the uniform prior is slightly more nuanced, initial intuition would suggest that giving equal probability to all values a parameter could take is a correct means of reflecting prior ignorance about the parameter values. However, this method fails accurately reflect the probabilities that unlikely parametrizations may be correct and in a sense exhibits a kind of confirmation bias. As discussed in Section 2.2 the model which is optimal is the one which has the lowest variance of the data around its predictions L(D(0) while maintaining low variance in the model parameter space L(0). In addition the bias-variance trade-off (Geman et al., 1992) state that to decrease data variance we must increase parameter variance reflecting the trade-off between T.()) and ((D in the MDL equation. Further, we related this to the Energy-Entropp competition concept (Zhang et al. 2018). where it was also stated that neural networks are biased towards wide minima. Hence, by placing equal prior probability on all areas of the landscape we see that the use of the uniform prior will result in posterior distribution over the model parameters which places excessive density on high variance areas of the landscape while at the same time places too little probability on very specific, low variance parametrizations of the model. This results in the development of a sub-optimal model which favours wider minima within the loss landscape and as result excessively reduces the variance of the data around its predicted values.
Thus we conjecture that a correct prior for a model would be the Jeffreys prior shown in Equation and Equation 4
P(0) x v det ((0) v det (())
(4)
As result we see in Equation 2 that such prior would give the equation for h(0)
1
h(0) 2 log det(H (0))
(5)
We note that, with the use of the Jeffreys prior, the prior and entropy term in the posterior formulation cancel out, leaving the likelihood term as the only factor determining the posterior probability, as can be see in Equation 6
P)O Z exp
Σ
P
(yi - ((x1,)))²
log det (H(()) + log det ((()))
1 2
207
2
(6)
1 exp
f(z 0))²
Z
20
is, thus, possible to utilize the loss landscape in the area of a minimum to determine the degree of certainty may have in our model parametrization being the true parametrization and result determine the necessary Jeffreys prior probability. This is due to the fact that higher entropy means wider minima which reflects higher parameter variance and the necessity to be less certain of the parametrization in that area. The opposite is true for an increase in certainty in our parametrization sharp minimum. Furthermore, this would mean we objectively setting our prior based on the model behaviour given the observed data and sufficient statistic. Note, we do not say we determine our prior based on the hypothesis as the determinant of the Fisher Information/Heessian is invariant under reparametrization to be modelled by an alternate set of parameters 6' the dimensions and volume of the landscape will adjust such that /det (0)) the Jeffreys prior is that, while it places the full parameter determination on the data, it does not necessarily result in a posterior distribution which has extracted all information from the data. Infor- mation which provides insufficient decrease in data variance to warrant the increase in variance in the model parameters will not be utilized as the model naturally "distrusts" this information by providing relatively lower prior probability to the more entropic parametrization found in the wider basin. This is where we see the utility of the Jeffreys prior with regard to the MDL property reflected by Equation as it balances both model complexity L(0) while fitting the data L(D
This means that in the area of a minimum, by transforming the hypothesis
/ det ((()) (#)) (Fisher, 1922). A necessary distinction regarding
The primary power of the Jeffreys prior comes from the use of the Fisher Information. Naturally as the model fits the data and captures information, the amount of information left in the data which
5

---

Under review as a conference paper at ICLR 2020
remains uncaptured by the model decreases. This is observed as a decrease in the Fisher Informa- tion. We see, however, that the model entropy increases as the Hessian matrix, and by extension the Fisher Information, decreases, which is again in agreement with the htas-variancs trade-off. The consequence of this observation is that to capture all the information from a sufficient statistic deter- mined by the training data we must utilize increasingly complex models, capable of modelling finer details found in the data. The utility of such fine details to the model performance exhibits dimin- ishing returns to a point where the perturbation of a parameter capturing these details will not result in any significant deviation in the model behaviour. Simply, as more information is shared between parameters, the individual importance of a parameter decreases. This is in contrast to an under- parametrized model where the parameters capture as much of the most important information from the sufficient statistic as possible and rely heavily on this information determining its behaviour. In light of the Fisher Information we see the Maximum Likelihood Principle further reflects that the use of the uniform prior biases our models towards maximum entropy within the loss landscape by extracting all information from the training data at the expense of higher model complexity.
must be noted that the propensity of neural networks to extract all information from the training data is not inherently negative quality of the models. Quite the opposite, it is reflective of the power and capability of the models which are designed to learn the variances within data and utilize this information in determining their behaviour. As a result, however, the efficacy of these models is directly related to the reliability of the data on which they are trained and for all the information found in the training data be present and reflective of the entire population of data being modelled. This is seldom the case as training data is inevitably noisy, either due to noise from sampling and capturing of the data, or due to confounding aspects of the task domain which on average do not affect the population data distribution but do provide source of structured noise when their effect observed on the training data. Minimizing the Fisher Information metric and fulfilling the Maxi- mum Likelihood Principle in such cases would reflect that the information found in the noise of the training data was utilized determining the model parameter values, which is clearly undesirable and is known as the model over-fitting the training data (Hawkins 2004).
We, thus, see that the notion of the widest possible minima in the loss landscape providing the best generalization performance is only true in noiseless environment. The view, however, that wide areas in the landscape generalize better is true as this width in the landscape would merely reflect that the model has captured more information from the data than parametrization found in a steeper portion of the landscape. Naturally this would provide better test error performance by the model if it has captured the information found in the training data which is reflective of the information within the population data distribution being modelled. We conclude that the width of the landscape in which model finds itself is demonstrably important and that there a precise width in the landscape which provides the model with the best possible test error performance. This point would be exactly where the model prior is equal to the Jeffreys prior as determined by the Fisher Information of the loss landscape. This is due to the aversion of the Jeffreys prior to any information which does not justify the increase model entropy by a superior decrease in data variance or prediction error, while remaining objective in the sense that the prior is completely determined by the loss landscape and has as little effect on the parameter posterior distribution as possible. Thus, noise in the training data which only serves perturb and hinder the learning of the model without providing sufficient benefit to how the training data is fit will not be learned by the model.
4 METHODS
From the above discussion it is clear that it is not enough to merely reduce the variance of the data around the model prediction as it is possible for the model to reduce this variance to an excess degree. We, thus, require a metric for the difference between the model and true distributions which is only minimized when the two distributions are identical. This is not the KL--divergence as this metric merely reflects the density of a distribution B which lies outside of another distribution A. It is, thus, possible to minimize the KL-divergence while the distributions are not identical by having distribution A surround or encapsulate distribution B. We will, thus, use the Jeffreys divergence as the difference metric between the two distributions, as the Jeffreys divergence is uniquely minimized when the two distributions are identical. The Jeffreys divergence is shown below in Equation / and is merely the sum of the KL-ddieegeenee for the true parameter distribution T(0) compared to the

---

Under review as a conference paper at ICLR 2020
model parameter distribution P) and the opposite Ki-divergence. We show the derivation of the Jeffreys Divergence from this summation in Section A.2 of the Appendix.
()((()))((0X)) = - ((())))P(00))))))) -n -00)000)00)0 (
(7)
Naturally Equation 7 is intractable due to the necessity to integrate over all parametrizations. How- ever, as discussed in Section E the use of maximum likelihood estimation results in an excess of density high variance parametrizations in the posterior parameter distribution in Equation 2 It was thus sufficient to evaluate the Jeffreys Divergence at a single point in parameter space and ob- serve the relative densities at that point, providing distance metric opposed divergence metric. We will, thus, refer to the Jeffreys Distance for the remainder of this work, with the formula shown in Equation 8
(((T())))((X (T(0) ((O))))(::(()) -
In ())())
(8)
The necessity of distance metric being positive semi-definite is upheld by this metric as it is clear when (T(0) P(0 X)) < 0 then (InT(0) In ln[((
Likewise when (T(0)
P))[X) > 0 then (lnT(0) X)) > 0. This a benefit of the symmetrical property of the Jeffreys Divergence which the KL-divergence does not possess. Thus, substituting the model posterior formula from Equation 2 as well as the true model distribution in Equation 9 into the
logarithmic terms of Equation g we obtain the formulation shown in Equation 10
P (y: f(x,, ))2
T(O) = 7
exp
20
(InT(0)- - InP(((X)) =
Σ
P
(y:: - 02)2
+
P
(y: f(x;
20;
20²
1=1
1=1
-h(0) + log (Hdt(H(())) + 2" -2) (10)
1
2
Assuming now that 0 8* and thus fx,, f(x,, (*), as would be the case the end of an unbiased training procedure, we see that the two variance terms in Equation I0 will cancel out. Further, we see that the only means of obtaining O value for the expression is to use the Jeffreys prior causing h(0) to cancel with the entropy term & log det (O)). as discussed above in Section 3, with Equation [5 and Equation 66 Finally we see that using the Jeffreys prior would result in the posterior model distribution shown in the last line of Equation 6 If fz, 0) fII,, 8*) then it is clear that = Z is the necessary corresponding normalizing constant, and, thus, these terms will also cancel out in Equation TO A similar argument
be made for the probabilities component of the Jeffreys distance (T(#) P))[X)) in Equation R, whereby we equate the two likelihood terms, then the use of the Jeffreys prior ensures that the posterior distributions do not differ resulting in Jeffreys Distance of O.
From Equation 10 we that the use of the Jeffreys prior while minimizing the error of the model directs the model to a parametrization which results in the model likelihood being equal to that of the true underlying distribution likelihood. This allowed model found the most probable parametrization in the landscape corresponding to the use of the Jeffreys prior, as being the parametrization which equated the model likelihood and true distribution likelihood values. In light of the bias-variane trade-off this would also mean that if the model reduces its bias to a greater extent that would have moved to a point of excess parameter variance and model entropy well of an excessively low Fisher Information metric. As expressed in Section 3 this indicative of the use noise by the model determining its parameter values. As a result the model would have overfit the training data. We, thus, reach the intuitive conclusion that once model parametrization becomes a more likely distribution to have generated the training data than the underlying true distribution itself, that the model will have began to overfit the training data.
determine the point at which the
7

---

Under review as a conference paper at ICLR 2020
The first empirical result presented in Section 5 is, thus, distribution reflecting the proximity of the training step at which minimum test error was reached to the training step at which the model likelihood is equal to that of the true underlying distribution likelihood (we will refer to this as the likelihoods intersecting) over a number of network training procedures. The aim of this experiment is to empirically confirm that the Jeffrey Prior parametrization provides the minimum test error for a model. For this experiment it is necessary to possess a ground truth on the likelihood of the true data generating distribution on the training data. Unfortunately such ground truth information is not available on real-world data sets. As a result it was necessary for our experiments to use synthetic data which was generated by a ground truth network, which we shall refer to as the "True" network. A "Training" network will then learn to model this ground truth network on noisy training data.
Hence, the procedure for this first experiment is as follows. We create a randomly generated True network with depth between 5 and 15 layers. The widths of the model layers are randomly sam- pled between 5 and 100 neurons. The layers are sorted in descending order of width, as is common for network architectures used in practice and results in the wider, earlier layers extracting fea- tures for the later layers. We then prepend the 100 neuron input layer and append the neuron output layer. All layers except the last are sigmoidal. This is the model used to generate data. We then randomly initialize our training network. The number of layers in this network is ran- domly chosen from the range of large network to overfit the data. The widths of this network's 's layers are sampled from the range of [TrueN NereeNeworkee ks Layee 100). This is again to ensure the model is over-varaman over-parametrized. The True networks parameter values as well as the Training networks initial values are sampled uniformly from 1.0, 1.0] with random positional bias added the True network parameters in the range of 00.5, 0.5]. This bias is to ensure the Training network starts with a significant degree of error. Finally, we utilize randomly sampled values between [0.0. 1.0] as input to the models, with training batch size of 50 data points and test batch size of 500. This data is input to the True network and we obtain the corresponding data labels as output. Lastly we add Gaussian noise to the Training data only (while the Test data remains clean) with mean of 0 and variance of 0.2. The Training network is then trained to model the True network using this data and we observe the points where the likelihoods intersect and where the test error is minimized. This process was repeated for 935 separate training procedures. The distribution of the distances between the likelihoods inter- secting and the minimum test error are shown below in Section 5
+ 5, 25) ensure
obtaín sufficiently
Having observed the relationship between test eтror and the relative likelihoods of the training and true networks, we then decompose the parameter landscape into its dimensions of principal curva- ture to reflect the impact of noise in the training data on the landscape by first training on noiseless data until a certain training error is achieved (error < 0.4). which point the noise added to the data. We observe and interpret the resulting impact of the addition of the noise on the curvature of the landscape. Secondly we aim to observe the relationship between the dimensions of principal cur- vature and the generalizability of model parametrization by observing the entropy sub-manifold of the original statistical manifold (Chen 2019), defined over the primary dimensions of principal curvature, relative to the test error and likelihood values of a model parametrization.
Riemannian
Due to the necessity to calculate a full Hessian matrix ! for these experiments a smaller network was utilized, than in the first experimental procedure. The procedure is the same as described above for the first experimental procedure, however, in this case the training model was composed of 4 hidden layers with widths of 25, 20, 12 and 7 neurons respectively. As shown empirically in Karakida et al. 2019), however, the behaviour of the Eigen-decompe Eigen-decomposition of the Fisher Information is independent of the size of the network architecture, and, hence, this smaller network is sufficient for observing the effect of noise on the dimensions of principle curvature of the model. Hence, the Hessian matrix calculated has dimensionality of 831 x 831 elements. The training data was again obtained from a true network. In this case different permutations of 25- binary strings were input to this model which returned the corresponding scalar using one hidden layer of 5 neurons and a linear output layer.
'The Hessian matrix was calculated using the Jax library (Bradbury et al. 2018
8

---

Under review as a conference paper at ICLR 2020
Dolitzrinn Foed ieteaceeera Aeound Minmunn
Diszbutiin Drfenine bowwen Jehays Pasaraticaaon Text Drot Menmure
-80
-90
-200
-255
235
202
4029
-234
+033
-006
0.20
0.56
0.30
032
029
Figure 1: Distributions of the number of parameter update steps (left) and the difference in test error (right) between the Jeffreys prior parametrization and the minimum test error from 935 individual training procedures (Kernel Density Estimate shown in red on the left).
RESULTS
As stated in Section 4 the first empirical result aimed determine if the point in the landscape which is most probable under the posterior distribution resulting from the use of the Jeffreys prior possesses the optimal test error performance. The results of this first experiment are presented in Figure ! where the left image reflects the distribution of the number of parameter update steps between where the Jeffreys prior parametrization occurs (where we observe the intersection of the likelihoods) and where minimum test error occurs. We see in this image that the highest density is placed around 0, with a vast majority of training procedures having the Jeffreys prior parametrization coincide exactly with the point of minimum test error. This supports the assertion that the Jeffreys prior results in the parametrization with the best generalization performance. We do, however, observe small uniform spread of density the right of 0 in this image. We observed that this is due to the test eтror oscillating once the Jeffreys prior parametrization is reached. This is merely result of our inability fine-tune the learning rate for the individual training procedures of the randomly generated training networks, with significantly different architectures. To reflect the fact that the test error for the Jeffreys prior parametrization in the trainings where oscillations occur is negligibly different from the minimum test error we present the right image of Figure| In this image we plot the density of the difference in test error of the Jeffreys prior parametrization compared to the minimum test eтror of the 935 separate training procedures. To make the error independent of the size of the regression values being modelled we divide the error by the mean of the regression y-values (generally this value is around 2.0 for the respective training procedures). We observe that all trainings the discrepancy in test error less than 0.1, with almost all discrepancies being less than 0.05. These error discrepancies negligible and, thus, these results empirically confirm hypothesis that the Jeffreys prior parametrization corresponds to the minimum test error.
The results of injecting noise into the training data only once the model has been sufficiently trained on clean data can be see in Figure 2 In this figure we present the Eigenvalues of 3 of the 5 principal curvatures of the loss landscape. Thus, each value reflects the inverse of the variance of the model in the dimension of the corresponding Eigenvector and a lower value reflects a higher entropy in the given dimension. From these results we can that the injection of noise results in sudden increase in the entropy of the lower principal curvature dimensions but has no effect on the first dimension of principal curvature. As the Fisher Information is reflected by the entropy this would mean that the low principal curvature dimensions capture more information at the injection of the noise into the data, while the information captured by the first principal curvature dimension remains smooth throughout the training procedure. This would reflect a noise invariance of the principal curvature of the landscape and as a result we can see that the dimension of principal curvature in the landscape is exclusively responsible for the capturing of the true primary data information. In light this noise invariance we observed the entropy of a sub-manifold corresponding only to the dimensions of high curvature (Eigenvalues > 1) during the training of model. The results of this experiment are shown in Figure 3 In agreement with Figure [ we see that in the area of the Jeffreys prior parametrization (intersection of likelihood values), the test emrr reaches its minimum

---

Under review as a conference paper at ICLR 2020
1st Principal Curvature
3rd Principal Curvature
5th Principal Curvature
124
Midel Constur Numarr
trecpel Canstrn
Cunsture
curvature
Curvature
29
122
256
209
250
243
24
226
350
206
200
306
50
330
159
200
25.
290
Number of Training Stops
Number Training Stops
Number Tratring Stees
Figure 2: Impact of the addition of noise on 3 of the Principal Curvatures of the Loss Landscape. Vertical lines signifies point of noise injection.
Mean Squared Error Values During Training
Liceithood Model and True Patarmetrization
Model Entropy on Princioal Sub-manifoid
Fas Oro
0.02
0003-
Likeihoodd
0043
Modil iracpy miripaa iddannii knn Minhum bikwey
-35
Data cb-tbaae
-350
Deallty Lhatress
0063
Number of Training Stops
Number of Training Stops
Number of Trairing Stots
The Jeffreys prior parametrization is found in the area of parameter space minimizing test error and maximizing entropy on the principal sub-manifold.
Figure 3:
value. A number of insights can be gained from the third image in Figure 3. Firstly, that the Fisher Information matrix and the Fisher Information Metric, are non-singular and positive semi-definite on this sub-manifold This reflects that the dimensions responsible for capturing true information in the data are convex, with positive Gaussian curvature and that it is sufficient for the model merely minimize this well-behaved region of parameter space. We, further, observe that the Jeffreys prior parametrization maximizes the entropy this sub-manifold, reflecting that it still captures all true information from the data. We see the green portion of the entropy metric as being the area where the entropy is within 0.003 of its maximum value. The fact that the entropy begins decreasing later in the training is reflective of the model forgetting true information while learning the noise once it starts overfitting. The fact that the entropy stagnates past the Jeffreys prior parametrization is due to the fact that the lower dimensions of principal curvatures in the sub-manifold were minorly sensitive to noise and that in this region the model is beginning to learn noise and forget true information at the same rate. We have, thus, demonstrated that maximizing entropy is beneficial in the absence of noise. However, when noise is present in the data, maximum entropy corresponds to overfitting.
6 CONCLUSION
We see that the notion of the width of the loss landscape being an indicator of a robust parametriza- tion correct, however, this is conditional upon the model being developed in noiseless domain or, more significantly along dimension of parameter space which is independent to the noise of the domain. With the aid of the Fisher Information perspective of the geometry of the landscape we see that the higher entropic points in the landscape directly reflect the absence of further in- formation upon which the parameter values may be determined. Thus, we see the propensity of maximum likelihood models towards such high entropic points as a reflection of their propensity to utilize all information in determining the parameter values, including noise. Thus, we make the final conclusion that the point of maximum entropy in the loss landscape does not possess the best generalization performance and corresponds to the overfitting of the model to the training data. In- stead the optimal point in the landscape occurs at maximum entropy in the dimension of principal curvature which corresponds to the most probable parametrization found by a Bayesian posterior distribution resulting from the use of the Jeffreys prior.
10

---

Under review as a conference paper at ICLR 2020
REFERENCES
Madhu Advani, Subhaneil Lahiri, and Surya Ganguli. Statistical mechanics of complex neural systems and high dimensional data. Journal of Statistical Mechanics: Theory and Experiment, 2013(03):P03014, 2013.
James O Berger. Statistical decision theory and Bayesian analysis. Springer Science & Business Media, 2013.
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, and Skye Wanderman-Milne. JAX: composable transformations of Python+NumPy programs, 2018. URL BRL.ttt: /ttt:://g wttpbb//oooooogooooooooooooo
Pratik Chaudhari and Stefano Soatto. Stochastic gradient descent performs variational inference, converges limit cycles for deep networks. In 2018 Information Theory and Applications Work- shop (ITA), pp. 1-10. IEEE, 2018.
Bang-yen Chen. Geometry of submanifolds. Courier Dover Publications, 2019.
Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. Sharp minima can generalize for deep nets. In Proceedings of the 34th International Conference Machine ceoniing-Volume 70, pp. 1019-1028. JMLR. org, 2017.
Ronald A Fisher. On the mathematical foundations of theoretical statisties. Philosophical Transac- fions of the Royal Society London. Series A, Containing Papers of Mathematical or Physical Character, 222(594-604):309--36 1922.
Stuart Geman, Elie Bienenstock, and René Doursat. Neural networks and the bias/variance dilemma. Neural computation, 4(1):1-58 1992.
Douglas Hawkins. The problem of overfitting. Journal of chemical information and computer sciences, 44(1):1-12, 2004.
Edwin T Jaynes. Prior probabilities. IEEE Transactions on systems science and cybernetics, 4(3): 227-241, 1968.
Edwin Jaynes. Prubability theory: The logic of science. Cambridge university press, 2003.
Harold Jeffreys. An invariant form for the prior probability in estimation problems. Proceedings of the Royal Society of London. Series A. Mathematical and Physical Sciences, 186(1007):453 461, 1946.
Ryo Karakida, Shotaro Akaho, and Shun-ichi Amari. Pathological spectra of the fisher information metric and its variants in deep neural networks, arXiv preprint
2019.
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Pe- ter Tang. On larce-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint 1r09 (1t.:16699.0866. 2016.
Jan Koenderink and Andrea Van Doorn. Surface shape and curvature scales. Image and vision computing, 10(8):557-564, 1992.
Anders Krogh and John A. Hertz. A simple weight decay can improve generalization. In J. E. Moody, J. Hanson, and Lippmann (eds.), Advances Neural Information Processing Sys- tems 4, pp. 950-957. Morgan- Kaufmann, 1992. URL http::ttt//ppet hwtp://appaaaaaa nips. cc/paper/ 563 a a-si-iiii simpll.m... e
can improve -eeereeeeeeereeee www.liii.iiiiiiiii pdd
Erich Lehmann and George Casella. Theory of point estimation. Springer Science Business Media, 2006.
Alexander Ly, Maarten Marsman, Josine Verhagen, Raoul PPP Grasman, and Eric-Jan Wagenmak- ers. tutorial on fisher information. Journal of Mathematical Psychology, 80:40-55, 2017.
Ian Porteous. Geometric differentiation: for the intelligence of curves and surfaces. Cambridge University Press, 2001.
11

---

Under review as a conference paper at ICLR 2020
C.R. Rao. Information and the accuracy attainable in the estimation of statistical parameters. Bulletin of Calcutta Mathematical Society, 37:81-91, 1945.
Jorma Rissanen. Modeling by shortest data description. Automatica, 14(5):465-471, 1978.
Claude Sammut and Geoffry Webb. Encyclopedia of machine learning. Springer Science &
Business Media, 2011.
Lene Theil Skovgaard. riemannian geometry of the multivariate normal model. Scandinavian Journal of Statistics, pp. 211 1-23,, 1984.
Michael D Spivak. comprehensive introduction to differential geometry. Publish or perish, 1970.
Yao Zhang, Andrew M Saxe, Madhu S Advani, and Alpha A Lee. Energy-entropy competition and the effectiivnness of stochastic gradient descent machine learning. Molecular Physics, 116 (21-22)33114-3223, 2018.
A APPENDIX
A.1 DERIVATION OF THE NEURAL NETWORK BAYESIAN POSTERIOR DISTRIBUTION
The derivation presented below is adapted from Zhang al. (2018).
We aim derive a Bayesian posterior probability for the parameters of an artificial neural network conditional upon the training data. To this end we will assume a Gaussian likelihood distribution, as well as a general exponential prior. By this definition, this prior may improper, however, without loss of generality we will assume that only priors which result in proper posterior distributions are utilized. Finally, we will assume that the data points are independent and identically distributed (i.id). Thus we utilize the following likelihood and prior distributions respectively:
-
P
(yi f(xi, 0))²
P(X(0) x exp
20
P(0) = exp)(0)
We, thus, obtain the posterior distribution, where Z is the normalizing constant (also known as partition function):
P
(ys
((2,,0)² 20 + h(0)
P(O Z exp
As stated Zhang et al. (2018), machine learning problems where stochastic gradient descent has been successfully applied have asymptotic data-to-parameet ratio of P/N O(1) as P where denotes the number of data points being used to determine parameters. This is referred to as the high dimensional limit (Advani et al. 2013). In this limit it is justified to perform Laplace A Approximation:
/ P(O
Σ
1
exp
EP
(yi (x)
20
M(0)
Z
/ det ((,,)
where 0g is a parametrization for local minimum of given loss function and H H(0g) denotes the Hessian matrix of the model parameters at this minimum. We are, thus, summing over the various local minima within the loss landscape in the above formulation.
12

---

Under review as a conference paper at ICLR 2020
Finally, rewriting the denominator in exponential form:
exp -- log det(H((0)))
det H(O,)
We obtain:
/
P(
2
1 Z
Σ
(3s-
04)2 20
h(0g) log (ogde))((Oa))) )
exp
With the point density for the model parametrization at single minimum in parameter space:
X(
Z exp
1
Σ
(yi
f(a;, 0n))²
h(0g) log det det(H H ((a)) )
20²
A.2 DERIVATION OF JEFFREYS DIVERGENCE
- =D (TO(((())() D
T(0m P(O[X) do 10-1 / |P(OIX)m P(O
T(O) d0
T(O)
((0) - InT(()) dA / P(OIX) - )n de
= (T0)((((0X))+ -
(-T(0) + - (-T(()) ((0))))
/ (-T(6) ((P(I) in)(
-((110) (I) - dd
13


Reviews:
[{"review_id": "SJxwSVtcKr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper targets at a deep learning theory contribution based on information geometry. This contribution is tightly based on Zhang et al. (2018) and explains the generalization of deep learning from a Bayesian perspective. The main contribution the authors claimed is an optimal degree of curvature exist which gives the best generalization guarantees, which is in contrast to the commonly perceived \"the wider the better\".\n\nFirst of all, the writing (including language etc) is of poor quality, to the extent that the submission is very difficult to read and can be rejected merely based on this, with unusual expressions, missing  punctuations, super long sentenses, and wongly used words. The reviewer won't list example here because they are everywhere.\n\nWhat is even worse is the conceptral errors and defected derivations. For example, in eq.(1), the authors equate the Fisher information matrix (which is an expected Hessian) to the Hessian matrix, this is subject to conditions which must be clearly given right before/after the equation. As their results are largely based on the correctness of eq.(2), let's examine the derivations in appendix A.1.  In the first equation in A.1, what is the subindex \"j\"?  \"Utilizing Laplace Approximation of the integral\": such approximations have conditions that must be clearly stated. It is not clear how one can get the last approximation in page 12 from the previous equations. In summary, their eq.(2) is a loose approximation which is subject to a set of conditions (that are not given), and the derivation is of poor quality.\n\nAs a theoreiritical contirbution, the authors did not manage to converge to some simple and clear statements (theorems or equvalent). Instead, the contribution is largely *explanatory*. It is hard to observe anything new, given the poor writing and organization. The first 4 pages are mainly introductions of previous works.\n\nThe authors used information geometry and minimum description length to explain the generalization of deep learning. This is a small area. It is hard to miss closely related works by simple searching. Instead, the authors only cited Rissanen (1978). On the other hand, as the authors used the spectrum properties of the Fisher information matrix, there are some recent works by Amari which can be cited.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I did not assess the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "HJx3a8J0FB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper conjectures that the so called Jeffreys prior over the parameters of a neural network is the prior leading to the best generalization performance. The authors test this conjecture on an artificial task using a small neural network, and investigate the sensitivity of the results to noise.\n\nI like the general idea of the paper and appreciate the very detailed exposition placing it in the context of other works. In particular, I enjoyed the summary showing the sometimes conflicting evidence for better generalization in either broader or sharper minima, and how it relates to the Jeffreys prior.\n\nHowever, as I understood the paper, the main claim in page 5 Equation 4 “Thus we conjecture that a correct prior for a model would be:” is an *assertion* that Jeffreys prior is the correct prior to use over the parameter space of neural networks. While it is a possibility, the amount of empirical evidence presented does not (at least to me) provide strong enough justification.\n\nOn page 7, you say “This model was a neural network composed of one, 5 neuron, hidden layer which utilized a sigmoid activation function in its hidden layer and a linear activation in its scalar output layer.“, describing your experiment. I don't think this experiment is sufficiently large to convince me.\n\nFurthermore, in Figure 1 values outside the optical cluster at 0.0 appear nonetheless. I am not sure how to judge the amount of spread I see, and what effect they have on the performance of the network.\n\nIn general I would like to see experiments on datasets and with architectures that are at least somewhat close to what people use in practice (at least in terms of the size of the task and the capacity of the net). That would give me more confidence that your conjecture is true. While I appreciate your detailed theoretical exposition, I think the amount of empirical evidence you provide is insufficient to back the claims. Considering the explicit instruction to judge papers exceeding 8 pages with a higher standard, I believe that the lack of a greater amount of empirical evidence is a significant deficiency of your otherwise very interesting work.\n\nI encourage you to expand this paper and resubmit to another venue -- I believe it has a great potential.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HJlP-ZkhFr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper argues that the widest minimum in the loss landscape is not the best in terms of generalization. The authors provide theoretical arguments and claim that there exists an optimal width beyond which generalization can be poor. Synthetic simulations are presented to support these claims.\n\nThe authors employ Fisher Information to characterize the optimal width or the curvature around the minimum. The fact that the determinant of the Fisher Information Matrix is invariant to parametrization, under certain conditions, serves as the motivation to design an objective Bayesian prior called Jeffrey's prior. \n\nThe motivation and the theoretical arguments are interesting, but the paper lacks in presentation and sufficient empirical evidence is also lacking to get fully convinced by the claims. \n\nThe authors should discuss the architecture design choices used for the synthetic data-generating model. Why are the last 3 layers of the larger model comprise of linear mappings?\n\nFig 1 is not clear. What does n=23 signify in the caption? More discussion is needed to describe \"intersection of the likelihood values\", \"Difference in update Step\" and \"density is placed around 0\" in section 5.\n\n\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: There has been significant discussion in the literature on the effect of the properties of the curvature of minima on generalization in deep learning.  This paper aims to shed some light on that discussion through the lens of theoretical analysis and the use of a Bayesian Jeffrey's prior.  It seems clear that the reviewers appreciated the work and found the analysis insightful.  However, a major issue cited by the reviewers is a lack of compelling empirical evidence that the claims of the paper are true.  The authors run experiments on very small networks and reviewers felt that the results of these experiments were unlikely to extrapolate to large scale modern models and problems.  One reviewer was concerned about the quality of the exposition in terms of the writing and language and care in terminology.  Unfortunately, this paper falls below the bar for acceptance, but it seems likely that stronger empirical results and a careful treatment of the writing would make this a much stronger paper for future submission.
Decision: Reject