A Reparametrization-Invariant Sharpness Measure Based on Information Geometry
Authors: Cheongjae Jang
Sungyoon Lee
Frank C. Park
Yung-Kyun Noh
Venue: NeurIPS 2022 Accept (2022)
Keywords: Deep learning
Information geometry
Sharpness measure
Reparametrization invariance
Generalization
the Fisher information matrix

Abstract: It has been observed that the generalization performance of neural networks correlates with the sharpness of their loss landscape. Dinh et al. (2017) have observed that existing formulations of sharpness measures fail to be invariant with respect to scaling and reparametrization. While some scale-invariant measures have recently been proposed, reparametrization-invariant measures are still lacking. Moreover, they often do not provide any theoretical insights into generalization performance nor lead to practical use to improve the performance. Based on an information geometric analysis of the neural network parameter space, in this paper we propose a reparametrization-invariant sharpness measure that captures the change in loss with respect to changes in the probability distribution modeled by neural networks, rather than with respect to changes in the parameter values. We reveal some theoretical connections of our measure to generalization performance. In particular, experiments confirm that using our measure as a regularizer in neural network training significantly improves performance.

A Reparametrization-nvarial Sharpness Measure Based on Information Geometry
Cheongjae Jang Hanyang University cj angehanyang. ac. kr
Sungyoon Lee Korea Institute for Advanced Study sungyoonlee0kias re.kr
Frank C. Park
Yung-Kyun Noh Hanyang University
Scoul National University
Saige Research
Korea Institute for Advanced Study nohyungdhanyang. ac kr
fcpusnu. ac. kr
Abstract
It has been observed that the generalization performance of neural networks corre- lates with the sharpness of their loss landscape. Dinh et al. (2017) [8] have observed that existing formulations sharpness measures fail to be invariant with respect to scaling and reparametrization. While some ccale-invariant measures have recently been proposed, reparametrization- invariiant measures are still lacking. Moreover, they often do not provide any theoretical insights into generalization performance nor lead to practical use to improve the performance. Based on an information geometric analysis of the neural network parameter space, in this paper we propose a reparamettivaaiioniivvaa
sharpness measure that captures the change loss with respect changes in the probability distribution modeled by neural networks, rather than with respect to changes in the parameter values. We reveal some the- oretical connections of our measure to generalization performance. In particular, experiments confirm that using our measure as regularizer in neural network training significantly improves performance.
Introduction
From recent discoveries in deep learning, has been conjectured that the generalization performance of neural networks correlates with the sharpness of their loss landscape. In particular, lower general- ization performance of large-batch training of stochastic gradient descent (SGD) has been attributed to its convergence to sharper minima 22 49 and several optimization methods have shown better generalization performance by actively finding solutions with lower sharpness 19
12 24)
Minimum description length (MDL)- arguments [66] [17] and the generalization upper bound of the PAC-Bayes theory for neural networks [10] also suggest that the flatter the loss landscape, the better the generalization performance. Furthermore, in [21 32], measures based on the sharpness concept show better performance in predicting the generalization performance of models among various generalization measures considered in the past including the margin [5] and norm 34 33
Various measures for sharpness and flatness have been proposed in the literature. Hochreiter & Schmidhuber (1997) [17] propose the concept of 'flat minima,' in which the flatness of a minimum interpreted the volume of the connected region the parameter space over which the loss has approximately similar values. Similarly, the sharpness has often been measured by the maximum loss value (inside Euclidean ball) near the minimum 22 by the spectral norm of the Hessian the minimum 49
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

The above definitions of sharpness have some notable critical flaws. Dinh et al. (2017) [ point out that certain parameter scalings (e.g.. for neural networks with ReLU activation functions (3I) and reparametrizations may have no effect on the model output or overall generalization performance, and yet lead to wildly different values of sharpness Other sharpness measures have been proposed to address the lack of scale-invariance [26] 46 40 39,. but these measures fail to be invariant with respect to reparametrizations:; Section 5 of [8] offers several nonlinear reparametrization examples. The connections between sharpness and generalization performance also remain elusive.
In this paper we address the lack of scale- and reparametrization- anarrmmtrriaaiionnivvaaaanee of existing sharpness measures, as well as the problem that they often do not explain the generalization performance of deep learning networks nor lead practical use to improve the performance. Our approach rests on the observation and insight that sharpness should be measured taking into proper account the geometry of the underlying parameter space. For this purpose we draw upon tools from information geometry, paying attention to the fact that in most classification problems, probabilistic classification models are parametrized in terms of neural networks. In information geometry, the Fisher information matrix (FIM) serves natural Riemannian metric for the parameter space [3] and allows for measuring the change in probability density with respect to changes in the parameter values.
The eigenspectra of the FIM of a neural network are observed to have a small number of positive outliers (the number is usually equated with the number of classes) and bulk consisting of small eigenvalues [4] 37 36 48 [4]. This observation indicates that the number of principal varying components of the probability densities modeled by neural networks significantly lower compared to the number of parameters. Therefore, identically weighting every possible parameter change direction in the parameter space that is, applying Euclidean geometry - can be problematic, since places too much weight on parameter change directions that are meaningless with respect to changes in the probabilistic model, or equivalently, not enough weight is placed along meaningful directions.
We argue that one should consider these implications and use the change in probabilistic models as a 'ruler' when defining sharpness measure of neural network loss landscapes that does not depend on how the model is parametrized.
Building the above geometric amalysis of the neural network parameter space, in this paper propose a new sharpness measure based on information geometry. This measure is by definition reparametrization-innaaie
Additionally, we show that this sharpness measure simultaneously satis- fies the scale-invariance for neural networks with ReLU activation functions, since it is invariant for parameter transforms that do not change the model output. Hence measure is free from all the reparametrization- and scale-variance issues of previous sharpness measures raised by (8]
As second contribution of this paper, we also show that our geometric sharpness measure sheds an important insight on generalization performance. As detailed in Section B when the FIM is replaced with the Hessian in measure, the measure is in form similar to Takeuchi's 's information criterion (TIC), which quantifies the asymptotic bias of the big-liillhood value (usually the negative loss) evaluated at the maximum likelihood estimates hence corresponding the expectation of the generalization gap 444) TIC for neural networks has empirically shown a strong correlation with the generalization gap 45 One can apply the same argument to measure since the Hessian can be approximated by the FIM under mild conditions for deep neural networks (28). In addition, we show that our measure can be linked generalization in another geometrical way. The measure is associated with a margin defined in an information geometric sense, enabling the interpretation that a smaller sharpness measure indicates larger margins. We also demonstrate experimentally that using our measure as a regularizer to train neural networks can significantly improve generalization performance.
Our contributions can be summarized as follows:
We provide an information geometric analysis of the neural network parameter space by investigating the eigensubspace of the Fisher information matrix (FIM) of neural networks.
We propose a reparametrization- and inveriinva sharpness measure based on information geometry that is free from the problems of existing sharpness measures posed by [8) and discuss the relation between the measure and generalization performance.
II this paper, unlike other works where parameter scalings are often referred to as (linear) reparametrizations, following [8]. it is called a reparametrization to represent a model on a different parameter space obtained by applying a (nonlinear) bijection to the original parameters.

---

We use the proposed measure as regularizer when training neural networks, and demon- strate improved generalization performance.
We provide an information geometric analysis of the neural network parameter space in Section 2 Building on this analysis, Section 3 presents our reparametrization iiniiiiatiooniiviiantt invariant sharpness measure based information geometry and connects the proposed measure to generalization. In Section we perform numerical experiments using our measure as a regularizer to improve generalization performance in neural network training.
2 An information geometric analysis of the neural network parameter space
In this paper, we focus on the probabilistic classification models parametrized by neural networks. Let B a E IR denote data, and y E {L....,O, C} denote class labels with as the number of classes. Let p(a, y) and p(y| 0) respectively denote the data generating distribution and the parametric model of the probability density of classes given data. Here the parameter 0 E R" is the set of all the weight and bias parameters of neural networks. We consider neural network
function f(: 0) R RC x > (z; 0) = (fi (2; 0), ..... [x; 0)) where fj(: 0) RR R is a function that returns the j-th logit for j = 1..... C. Our parametric model is then represented as plyl p 0) = Σ, (x))) log p(y]
(x)(
and we denote the negative likeliiiihood by Ily, fa; 0))
Suppose data ((11,3)) ..., (a... yN)} are drawn i.i.d. from the data generating distribution p(r, v)- The cross-entropy loss, our training objective function throughout the paper, be written as follows
N N
(0)
(0)),
where (0) Iyy (x; 0)) - -log (1gs((ss 0).
2.1
The Fisher information matrix (FIM)
The Fisher information matrix (FIM) for the family of probability density functions p(a, y; 0) p(z))(y)a;; 0) parametrized by the neural network parameters 0 is defined as
(
[01)
0l
T
al
]]
{
T
al
F(0)= = Ep(a)
Eptvlen0))
Ep(r))
J(x) T Epirloe)
(af))
[(x)
00
00
0f
1 N
J. (diag(p:) p.P..) Js. 1=1
(2)
where the dependence of lly, f(x; 0)) to (2;0) is omitted for simplicity and J(x) = 0f (x) E
gCxm is the Jacobian of the logits with respect to the parameters. In deriving (2) the expectation with respect to p(x) is approximated by the finite sum over data points 21, , .y drawn i.i.d. from
(
p(z), J. [x,), and we use the fact that Eptvlra:))
0e 8f
diag(p.) - p.P." where
p: (p(1|2z;; 0), ..... p(C x O)T
RRC the model's prediction on the probability that x; belongs to each class and dpa)(p.) E R. Xxx is diagonal matrix whose (j.j) entry is (pi)j-
In this paper, among many of the important properties and applications of the FIM (e.g.. being the Cramer- Rao lower bound of estimator variances in estimation theory and statistics), we focus on the fact that the FIM can serve density models in information geometry. Differential geometrically speaking, the FIM acts as a natural Riemannian metric the statistical manifold, a space where the family of probability densities modeled with smoothly varying parameters 0 E IR' m is gathered. The FIM makes it possible to
metric measure the distance between the parametric probability
We discuss the applicability of our measure to the square loss in Appendix F
NNote that the family of probabilistic models from neural networks is usually not manifold in mathemati- cally rigorous sense due to the inherent singularities in the neural network parameter space (see Section 12.2 of 3). but this fact not crucial for discussion.
3

---

40
Elgenvalues
50
20
10
15
20
-2
(a) Top eigenvalues of (b) Perturbed decision (c) Perturbed decision (d) Perturbed decision F(0)
boundaries from A0 boundaries from 202 boundaries from 10
Figure 1: synthetic three-class classification example. For (b)-(d), the black, red, and blue lines correspond to decision boundaries of the neural network with the trained parameter values, and pa- rameter values perturbed along the k-th eigenvector 20,- R" (associated with the k-th eigenvalue) of F(0) with steps of 0.5 and -0.5, respectively.
measure geometric quantities such as length, angle, and volume on the manifold. (We refer the reader [ for the backgrounds on the Riemannian manifolds and differential geometry, and 3 35 28 for those on the information geometry and the Fisher information matrix.) Note that when the FIM is
full-rank, e.g., in the case of overparameterized neural networks with N, can be used as pseudo metric.
The FIM closely related to the KL-divergence, which is frequently used as information theoretic measure of discrepancy between probability density functions. The KL-divergence between two probability density functions with infinitesimal parameter difference d0 E I can be approximated
as
(L(p(x, y: + d0) 0)) = d F(0)d8.
(3)
2
2.2 An analysis of the eigensubspace of the FIM
This section discusses the characteristics of the eigensubspace of FIM of neural networks. In the analysis of the eigenspectra of deep neural network Hessians of (37. the FIM in 2 has been decomposed as follows:
N
N
1 N
1
F(O)=
=
J (diag(p.) -PiP.) J, J M, M,J.
(4)
N
where Mj = diag( (I- 1pT)) E RCx vpi (V(p)) (Di)i,.....
( E RRC and
=
(1, 1)T E T is a vector whose elements are all one.
According to the decomposition, the FIM can be thought of as non-centralized second moment of the (modified) logit gradients, i.e., each row of MiJ; E R CCm whose j-th row vector is represented
C E (F.))) #=1
-M = (in), (J
E Rm
(5)
where Ji, E RRxm is the j-th row of Jr. It has been observed that these logit gradients form a kind of hierarchical structure after sufficient training. the observation from (36), the gradients of the c' of c-th logit calculated from the data in class are gathered to form a cluster. A matrix obtained from averaging the outer products of the averaged logit gradients belonging to each cluster is then attributed to the outliers of the eigenspectra, and the outliers of the cigenvalues appear much the number classes. That is, the principal eigenvalues of the FIM (and possibly the eigenvectors) can be closely connected with distinguishing each class from the rest of the c' + classes (as can be implied from averaging (5) for # c and by assuming (p:)e c = 1). This becomes more evident in Appendix A which assumes the prediction is balanced as (p:) )er
for all d' # with < 1.
In order to more intuitively understand the characteristics of the eigensubspace of FIM implied from the above analysis, we provide classification example for two-dimensional synthetic data generated from mixtures of three Gaussians in Figure [] trained three-layer fully connected neural network with 70-dimensional hidden units (the number of parameters = 5,393).

---

Figure 1 (a) shows the top twenty eigenvalues of the FIM of the trained neural network. Note that there are few outliers among the eigenvalues in the entire 5.393-dimensional parameter space, with the remainder being almost O) The eigenvalues indicate the rate of change of the probability density (modeled by the neural network) measured by the KI.-divergence for unit change of parameter values along the corresponding eigenvectors. The existence of a few outliers means that the density change mainly occurs in only a few specific directions and that the amount of change in probability density can vary significantly according to the direction of change in parameter values.
This fact can also be observed in decision boundaries obtained from neural networks with the parameter values perturbed along some eigenvectors shown in Figure [ the values along the principal eigenvectors, a significant change appears in the decision boundary, increasing or decreasing the margin of certain classes. On the other hand, for eigenvectors with small eigenvalues, there is almost no change the decision boundary according to the same level of change in parameter values (in the Euclidean sense).
(b-d). When perturbing
These examples imply that it would be more meaningful reflect the different influences on the probabilistic model according to the change direction when we measure the quantities concerning changes in the parameter values, e.g., the sharpness of the loss landscape. By reflecting on these implications and using the unit change in probability distribution modeled by neural networks as our 'ruler' measure the sharpness. that is, applying the information geometry. the following section presents sharpness measure that does not rely on how the model is parametrized.
3 An information geometric sharpness measure
Reflecting the above analysis, we define an information geometric sharpness (IGS) measure as follows:
04: ae
04, T a0
IGS( = Σ
F(O)1
(6)
where F(O)+= R is the pseudo-inverse of F(0) and 4 Uys. [x;;; 0)) is the cross-entropy loss evaluated the i-th data (x,, y:).
Our measure evaluates the squared norm of the loss gradients calculated at each data point using the
(pseudo-inverse of) FIM and averages them for all data points Since span of the eigenvectors of F(0) with non-zero eigenvalues, our sharpness measure well-defined in the sense that there are not any components of (o neglected in measuring its information geometric norm (we elaborate on this in Appendix B
(L E R. belongs to the
20
Our sharpness measure in (6 is intrinsic quantity, i.c., coordinate invvariit (in differential geomet- ric terms). To see why, observe that under a local coordinate transformation (or reparametrization) R. -> R + 0' (0)), i.e., the model is represented with respect to different
parameter 6' as f(: 8²) = f(: (()) f(; f-; F and
(0.)
transform according the following
rules: (i) F F" = Φ F where Φ = J0
de
E mR (ii)
at
H.
where
30
20
20
it can be verified that the (
(t, F(O) (l. remains the same. Since the above measure gives the 868
(8))
same value regardless of which parametrization (e.g., 0 or 0') the statistical model is parametrized, it is free from the reparametrization- variincee issues raised in S
We can also define other reparametrization invariant measures by measuring the information geomet- ric norm of the gradients of the entire loss the mini-batch losses. For mini- of size b, the corresponding measure can be defined as follows:
a4 (
{
24 00
(G) = -Ep(B)
F(O)
I 20
FEEB
where Ep(B) [-] denotes the expectation with respect to some distribution p(B) of mini-batches
B C (a1,..., NN) of size [B] b. We discuss the relation between (6) and
7 Appendix C
Note
'The reason for defining the sharpness measure in 6 will be evident in Section 3.3 where we develop the relationship between our measure and the generalization.
5

---

that this definition can be linked to the concept of m-sharpness, which measures the sharpness of the loss landscape by averaging the sharpness of mini mini-batch losses and shows a better correlation the generalization as the batch size reduces [22
3.1
Transformation invariance
Neural networks composed of activation functions such as ReLU and leaky ReLU possess scale- invariant properties. In the case of layer-wise scaling, for consecutive layers of linear transforms and ReI.U, e.g., fx; [Wi.... Wr) = Wr. ReI.U(W,.-1 ReL,U(W,x)) where Wi E T24 xd:= for = 1..... I. with dn D and dr. C, if each weight Wi is multiplied by constant o >> 0 and the constants satisfy IT C: 1, there no change the outputs of the neural network under the same input values. A similar concept includes the node-wise scaling, multiplying weights entering node (or hidden variable in neural networks) by a positive constant and dividing weights out the node by the same constant.
These kinds of weight scaling can be viewed transformation (or mapping between identical parameter spaces) that ensures that the neural network model maintains the same probability density function, i.c., remains equivalent. a transformation differentiable and locally invertible without changing the probability density function that the neural network parameter models for all parameters on given neighborhood U S IR. of 6, our measures defined in (6) and 77) become invariant with respect to such a transformation. This fact can be expressed as the following proposition:
Proposition 3.1. Suppose there exist open subsets U, V.V. S R and an invertible and locally differentiable transformation g U - V that satisfies f(;; 0) (a; g()) for all RD and 9 E U with g(U) S V. The information geometric sharpness measures in (6) and (7) are invariant such a transformation, i.e., IGS(() (g)) (or (GS (gS)(((0))) for all 0 U.
The proof of Proposition 3.1 provided in Appendix D.I
Since the layer-wise and node-wise scalings considered for neural networks with ReL.U satisfy the assumptions of Proposition 3.1 our measures in (6) and ( are invariant to such scalings. Proposition 3.T is more general than the ocale-invariance, and Appendix(D.2)
examples of transformations other than parameter scalings that satisfy the assumptions in the proposition.
3.2 A comparison to some previous sharpness measures
After g pointed out the critical scale- and issues in the sharp-
10
reparametrizaaion-Varriie
ness measures considered in [17 22 49]. var-
Sharpowestre..
Sharpness.or).
102
ious scale-invariant sharpness measures have
10a
been proposed that suit to neural networks with activation functions satisfying the non-negative
10-2
10-1
homogeneity conditions (i.c., o a = a ((x 46 26] The
Ficher
Rangamani elal.
Petzka el
-Rau
for a > 0) such as ReLU 40
39
measure presented in 40 is based on the ge-
The ratio of sharpness measures evalu- ated for the reparametrized models those for the Sharpness (re.) original models Shuppness lod.) ). Note that the y-axis in the log scale. For the nonlinear reparametriza- tion, we use r) = (0) = (10 0 + b)" 6) + considered in g with several choices of (a. b). which are represented by different colors.
Figure 2:
ometry of the quotient manifold obtained from an equivalence class of neural networks, estab- lishing the scale-invariance depending on the considered equivalence types. The work in [39] considers a layer-wise flatness measure neu- ral networks and additionally explores the con-
ditions for the measure to explain the generaliza- tion well. Also, scale-invariant sharpness measure based the PAC-Bayes theory is developed in [46]. The Fisher-Rao norm is proposed as a capacity measure for neural networks 26 and defined as 0 F())O, which is scale-invariant.
However, the above measures do not satisfy the invariance to general nonlinear reparametrizatioos,, hence solving the problem raised by g only partially. That is, for a nonlinear reparametrizzatio
0 + 1 o0). the measures evaluated on the reparametrized model f(::n) = -(:: f(; (with respect to 7)) can arbitrarily vary from those evaluated on the original model [(; 0) (with respect
to 0). In Figure 2 we empirically demonstrate the existence of this problem in conventional sharpness

---

40) and [39 (respectively denoted measures such as the trace of the Hessian, the measures of 26), as Tr(H), Fisher-Raee Rao, Rangamani al., and Petzka et al. on the x x-axis of the figure; see Appendix G.1 for details). We can observe that the magnitudes of these measures can vary significantly.
Compared with the previous measures, only our measure (denoted IGS the x-axis of Figure is invariant to the considered reparametriizatoos.. These experiments empirically demonstrate that our measure satisfies the apaaramettizaaiionnnvaaaaaaan hence providing a solution that properly resolves the issues posed by g
3.3
Connections to the generalization
3.3.1 Connections to Takeuchi's information criterion (TIC)
Our measures can be linked to generalization in different aspects. When we train parametric models via the maximum likelihood estimation (this equivalent to minimizing the negative ongllikeldoodd of (T), the obtained maximum likelihood estimate is asymptotically unbiased, guaranteed theoretically by the asymptotic normality. However, the lig-likelihood value (or the negative loss) evaluated the obtained estimate does not enjoy such a property.
The well-known Akaike's information criterion (AIC) in the model selection literature calculates this bias under the assumption that the parametric model contains the actual data distribution M. Takeuchi's information criterion (TIC) a generalized form of the AIC by considering the case of
misspecified models, i.e., the parametric models do not contain the actual data distribution 44 9 45
The bias b(θ) of the log-likelihood value evaluated at the maximum likelihood estimate defined as follows:
plg (p(e,,,;0000 QD)) Ep(n.w) [logp( ys @(DD)]
b(0) EE
Σ (EDD
(8)
(D)
where D = vi)...., (IN, ]N is the set of N training data with (I;, Mi) p(æ,, y) and O(D) denotes the maximum likelihood estimate obtained from using DD [9]. Since our loss is the negative log-likelihood, the first and second terms inside Ep[[] correspond to the negative training and test losses, respectively. Consequently, the bias in (8) becomes the expectation of the parametric model's generalization gap (i.c., test loss training loss).
TIC calculates this bias under asymptotic assumptions, i.e., in the limit N oc, follows
1 TT(H((0)
TIC = lim b(0) N-
where 0, E Rm is a local maximum of the expected log- likeliioooodd, H (Oo) ((x0)
R ma is the Hessian
(
of the (expected) loss, and C(0p) = Ep(na))
mm is the
80
80
A=A0
non-centered covariance of the loss gradients. Here both H and C(0s) evaluated using the data generating distribution p(.. y).
The formulation in 9 is very similar to our information geometric sharpness measure when evaluated local minimum E _d of the loss in T.. Compared to TIC, our measure in (6) contains the FIM instead the Hessian, and the expectation for C(0) is taken with respect to the empirical distribution rather than the true distribution. After a model is sufficiently trained, the Hessian can be approximated
to the FIM 36 28 indicating that our sharpness measure is closely related to TIC.
It has been observed that TIC predicts the generalization gap of neural network models well when the expectation with respect to p(æ, y) is approximated by finite sum the integrands over the test data 45 These findings are also confirmed by experiments using our measures. In Figure 3 compared to the other sharpness measures (labeled the same as Figure 2 we can observe that our measure correlates better with the generalization gap. The experimental details are provided in Appendix G.2
7

---

ICS
Tried
Fisher-Rao
Rangamani at
Petzka at.
lacels, =; (.400))) log
Generalization gap
0.0425
CIFAR-10 MNIST
18-0.660
RwO, .00
R=0.363
Rw0.570
3.1
2.9
2.7
K-0.610
Re0.544
Awbr223
N-0
Rw0.555
2.5
Luca (1t, 2: log
Normalized sharpness measures
Figure 3: Plots for various sharpness measures (normalized to be in [0,1]) vs. the generalization gap. The top and bottom rows are obtained using MNIST and CIFAR-I0 data sets, respectively. In each subplot, we provide the correlation coefficient between the corresponding measure and the generalization gap.
Figure 4: Change in the decision boundary according to the pertur- bation in parameter values + 0 e...
3.3.2 Connections to the margin
Our measure can also be linked to generalization in terms of margin. Unlike the usual definition of the margin as the distance from the decision boundary to its closest data, we devise similar concept developed in the parametric model space rather than the input space.
Given model parameter value, if perturb the value that the model's decision boundary passes through the location of a nearby data shown in Figure 4 the difference between the initial and the perturbed parameter values would correspond to a sort of margin defined in the model parameter space. Since the parameter space dimension is high, we can perturb the model parameter value in various directions. Considering the perturbations of parameter values along the eigenvectors of the FIM results in an interesting connection between this margin and our sharpness measure. We now formalize this relationship for the binary classification case, while that for the multi case is derived in Appendix E
z; log pog 0) (1 =:) log(1 - 0)), where z; E (0, 1) denotes the binary class label for an input v, E R and p(a,;; 8) denotes the model's prediction on the probability that x; belongs class one. The condition for I; to lie on the decision boundary is p(r:) = 2. which is equal to 4(0) = log 2 regardless of the 2, value.
We define the binary cross-entropy loss (,(0) [nc z;
Suppose we perturb the neural network parameter value from I to + A0, where Adk E R is the k-th eigenvector of the FIM associated with the k-tth eigenvalue Å and G, is a scalar. For the decision boundary of the perturbed model to lie on F;, the scalar G₁ should satisfy log 2 = (0 + <1,)00k) s (0) + c
01
Ad,, where we apply the first-order Taylor expansion by
50
assuming small e, The scalar 61,6 can then be approximated as 6, RV
log 2.)
A sort of margin can be derived in information geometric sense by computing the squared norm of the perturbation vector E AO using the inner product based on the FIM as follows:
log 2 (0)
AO P(0)a02 =L, An
(10)
A
260
The summation over k = I,..., m' (with m' as the largest is with non-zero 2&) of the reciprocal of these squared norms (hence weighing more on small e²,, values which would reduce the error from approximations) can then be related to our sharpness measure as follows:
m
m'
A log - (())
0L
2
0L P90
F(O)
OL
Σ
(11)
A
(log 2 2
where we have used F(0)+ = = X measure at each data (x,, y1) appears in the numerator of (II). This indicates that the smaller our sharpness measure, the larger the squared norms (e, in the LHS, which means that the margin becomes larger in information geometric sense.
AT in deriving the last identity. Note that our sharpness

---

LOSE
W
12k
18k 20k
9-6
5k 12k
1sk
20k
Iterations:
Iterations
(a) Decision boundary from SGD
(b) Decision boundary from SGD with regular- ization
Loss
values
from
(d) Loss values from SGD with regularization
SGD
Figure 5: A synthetic three-clase classification example trained using SGD with and without regular- ization under the effect of poison data. The loss values are smoothed for better visualization.
Using the sharpness measure as a regularizer to train neural networks
We now apply our sharpness measure regularizer to train neural networks to see if regularizing the measure can improve the generalization performance. The corresponding loss function written
1 [(0) - N Σ + pG
(12)
where P is coefficient for the regularization term. We consider toy example and then consider image classification tasks involving MNIST and CIFAR-10/100 data sets. We also provide some tractable and possibly efficient ways regularize our sharpness measure. All the experiments have been performed using the PyTorch library (38
4.1 A toy example
We first apply our regularizer to a toy example. In [18]. they have trained neural networks on poison data with the wrong labels to hamper the generalization performance and make the loss landscape extremely sharp while fitting all the training data. We follow this experimental setting and check whether the obtained solution can avoid the undesirable generalization performance and loss landscapes when applying our regularizer (calculated on the training data). The experimental details, as well as how the IGS is approximated for this example, are provided in Appendix G.3
The experimental results shown in Figure 3 The results from SGD without regularization show very irregular decision boundaries with tiny margins, whereas relatively soft boundaries with larger margins appear when our regularizer is used, especially around the training data. Applying our regularizer also shows a lower generalization gap, i.c., the difference between the training and test losses, than the SGD without regularization.
4.2 MNIST and CIFAR-10/100
To check the effect of the regularizer more realistic settings, we apply our regularizer to the classifi- cation of MNIST and CIFAR-10/100 data sets. For this experiment, regularize the mini-hatch IGS defined in 77. For an efficient implementation of our regularizer, the following approximation on
perturbed loss functions with perturbation 8 pF(()t t
R (with small p 0) is useful:
89
01
0l F(0))
T
I(0 5) =o 1(0) +
= 1(0) + p. 08
(13)
00
00
Note that for 1 = E with mini-batch B of size [ b, minimizing minimizing stochastic version of (12) (with (G),, where the stochasticity comes from sampling a mini-batch B. Therefore we minimize (33 in our experiments with as tunable hyperparameter. To obtain 8, resort to EKFAC-based approximation for the natural gradients as detailed in Appendix G.4.I
(3) becomes
A three-layer fully connected neural network (3FCN) is used for the experiments using MNIST data, and various convolutional neural networks such as VGG 42, ResNet [15], and WideResNet [50) used for CIFAR- 10/100 experiments, with data augmentation, cosine annealing [27]. and label

---

Table I: Averages and standard errors of the test classification accuracies for SGD, GR. SAM, ASAM, and SGD with our regularization method on MNIST, CIFAR- 10, and CIFAR-100 data sets.
DATA SET
MODEL
SGD
GR
SAM
ASAM
OURS
MNIST
3FCN
98.13: 0.05
98.20
0.01
98.62
0.01
98.65 +: 0.07
98.65 + 0,03
VGGIL-BN
92.75
0.11
93.13
93.61
93.78 -
93.81 +
CIFAR-10
RESNET-56
94,08 0.177
94.40 = 0.25
95.22 4 0.11
95.36 1 0.06
94.96 1: 0.01
WRN-16-8
95.98 0.03
96.15 = 0.10
96.70 0.14
0.06 97.03
96.57 0.10
VGGII-BN
72.25 0.17
73.05 = 0.32
73.69 0.20
73.61 + 0.18
74.24 + 0.22
CIFAR-100
RESNET-56
73.14 0.10
74.67
75.90
75.87 -
76.07
WRN-16-8
80.55 0.33
81.13 0.06
82.20 + 0.11
82.38 + 0.09
82.44 + 0.29
smoothing [30] methods additionally applied. For comparison, consider the gradient regularization sharpness-aware minimization (SAM) method [12], and the adaptive SAM
(GR) method 4 43],
(ASAM) method 24] extension of SAM involve the scale-invariance. The detailed experimental
settings are explained in Appendix G.4.2
We provide the averages and standard errors of the test classification accuracies obtained from three runs of each method in Table As can seen from the table, one can confirm that the generalization performance of SGD is significantly improved with our regularizer. Furthermore, our method shows better performance than the GR method and comparable performance improvement with the SAM and ASAM methods.
5 Conclusion
Various empirical results observed from running deep learning algorithms such SGD have suggested that minima with a flatter loss landscape tend to show better generalization performance. However, the previous definitions of sharpness/flatness have faced a severe dependence on reparametrizations or parameter scalings that do not change the model output and the generalization performance (8).
In this paper, based have proposed an information geometric sharpness measure which is reparametrization- and scale- invariant, making it free from the issues posed by [8. We have also discussed the connection of the measure to generalization. In addition, a regularizer that reduces the suggested sharpness measure is proposed, showing a significant improvement in generalization performance under practical settings.
information geometric analysis of the neural network parameter space,
Unlike many other sharpness measures, the Fisher information matrix (FIM) plays a crucial role in our measure. Since it can be computationally demanding to calculate the FIM and our sharpness measure for large-scale neural networks, more efficient evaluation methods should be developed to apply our measure such networks. Concerning the regularization of our measure during training, exploring better ways to reduce the time complexity and increase generalization performance is left for future work. Furthermore, analyzing the generalization properties of models obtained from deep learning algorithms such as SGD or natural gradient descent using our measure would be another intriguing future work.
Acknowledgments and Disclosure of Funding
C. Jang and Y.-K. Noh were supported by IITP Artificial Intelligence Graduate School Program for Hanyang University funded by MSIT (Grant No. 2020-0-01373). S. Lee was supported by KIAS Individual Grant (AP083601) via the Center for AI and Natural Sciences at Korea Institute for Advanced Study. F. C. Park was supported in part by SRRC NRF grant 2016R (A..RRAA11988.. HTP-MSIT grant 2022-0-00480 (Training and Inference Methods for Goal-Oriented AI Agents), SNU- AIIS, SNU-IAMD, and the SNU Institute for Engineering Research. Y.K. Noh was partly supported by NREMSST (Grant No. 2018R AS7709544,, 2021M335D22A0101955 and IITP/MSIT (Grant
No. ITP-2021-0-02068)
10

---

References
[1] Akaike, H.: A new look at the statistical model identification. IEEE transactions on automatic control 19(6). 716-7266777 (1974)
[2] Amari, S.I.: Natural gradient works efficiently in learning. Neural computation 10(2), 251 (1998)
[3] Amari, S.i.: Information geometry and its applications, vol. 194. Springer (2016)
[4] Barrett, D.G., Dherin, B.: Implicit gradient regularization. arXiv preprint arXiv:2009. 11162 (2020)
[5] Bartlett, P.L., Foster, DJ., Telgarsky, M.J.: Spectrally
margin bounds for neural networks. Advances in Neural Information Processing Systems 30, 6240-6249 (2017)
[6] Boothby, W.M.: An introduction to differentiable manifolds and Riemannian geometry, Revised, vol. 120. Gulf Professional Publishing (2003)
[7] Chaudhari, P., Choromanska, A., Soatto, S., LeCun, Y., Baldassi, C., Borgs, C., Chayes, J., Sagun, L., Zecchina, R.: Est: Biasing gradient descent into wide valleys. Journal of Statistical Mechanies: Theory and Experiment 2019(12), 124018 (2019)
[8] Dinh, L., Pascanu, R., Bengio, S., Bengio, Y:: Sharp minima can generalize for deep nets. In: International Conference on Machine Learning, pp. 1019-1028. 1028, PMLR (2017)
[9] Dixon, M., Ward, T:: Takeuchi's information criteria as a form of regularization. arXiv preprint mrXiv:1803.04941 (2018)
[10] Dziugaite, G.K., Roy, D.M.: Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data. arXiv preprint arXiv:1703. 11008 (2017)
[11] Fomenko, A., Novikov, S., Dubrovin, B.: Modern Ceomeeryy-ethodde and Applications, Part I: The Geometry of Surfaces, Transformation Groups, and Fields. Springer, Berlin (1992)
[12] Foret, P., Kleiner, A., Mobahi, H., Neyshabur, B.: Sharpness-aware minimization for efficiently improving generalization. arXiv preprint arXiv:2010.01412 (2020)
[13] George, T., Laurent, C., Bouthillier, X., Ballas, N., Vincent, P:: Fast approximate natural gradient descent
in kronecker factored eigenbasis. Advances in Neural Information Processing Systems 31 (2018)
[14] Ghorbani, B., Krishnan, S., Xiao, Y.: An investigation into neural net optimization via hessian eigenvalue density. In: International Conference on Machine Learning, pp. 2,32-2241 PMLR (2019)
[15] He, K., Zhang. X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778 (2016)
[16] Hinton, G.E., Van Camp. D.: Keeping the neural networks simple by minimizing the description length of the weights. In: Proceedings of the sixth annual conference on Computational learning theory, pp. 5-13 (1993)
[17] Hachreiter, S., Schmidhuber, Flat minima. Neural computation 9(), 1-42 (1997)
[18] Huang, W.R., Emam, Z.. Goldblum. M., Fowl, L., Terry, J.K., Huang, F., Goldstein. T.: Understanding generalization through visualizations (2020)
[19] Izmailov, P., Podoprikhin, D., Garipov, T., Vetrov, D., Wilson, A.G.: Averaging weights leads to wider optima and better generalization. In: 34th Conference on Uncertainty in Artificial Intelligence 2018, UAI 2018, pp. 876-885. Association For Uncertainty in Artificial Intelligence (AUAT) (2018)
[20] Jastrzebski. S., Arpit, D.. Astrand, O., Kerg, G.B., Wang, H., Xiong, C.. Socher, R., Cho, K., Geras. K.J.: Catastrophic fisher explosion: Early phase fisher matrix impacts generalization. In: International Conference on Machine Learning, pp. 4772-4784. PMLR (2021)
[21] Jiang, Y., Neyshabur, B., Mobahi, H., Krishnan, D., Bengio, S.: Fantastic generalization measures and where to find them. In: International Conference on Learning Representations (2019)
[22] Keskar. N.S., Nocedal. J., Tang, P.T.P... Mudigere, D., Smelyanskiy, M.: On large-batch training for deep learning: Generalization gap and sharp minima. In: 5th International Conference on Learning Representations, ICLR 2017 (2017)
[23] Kunstner, F. Hennig, P., Balles, L.: Limitations of the empirical fisher approximation for natural gradient descent. Advances in neural information processing systems 32 (2019)
[24] Kwon, J., Kim, J., Park, H., Choi, L.K.: Asam: Adaptive sharpness- aware minimization for scale-invariant learning of deep neural networks. arXiv preprint arXiv:2102.1166 (2021)
[25] LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proceedings of the IEEE 86(11) 2278-2324 (1998)
[26] Liang. T., Poggio, T., Rakhlin, A., Stokes, J.: Fisher-rao metric, geometry, and complexity of neural networks. In: The 22nd International Conference on Artificial Intelligence and Statistics, pp. 888-896. PMLR (2019)
11

---

[27] Loshchilov, I. Hutter. F.: Sgdr: Stochastic gradient descent with warm restarts.
arXiv preprint
arXiv:1608.0308 0398330333338 (2016)
[28] Martens, J:: New insights and perspectives on the natural gradient method. Journal of Machine Learning
Research 21, 1-76 (2020)
[29] Martens, J., Grosse, R.: Optimizing neural networks with fronecker-factored approximate curvature. arXiv preprint arXiv:1503.05677 (2015)
[30] Müller, R., Komblith, S., Hinton, G.: When does label smoothing help? arXiv preprint arXiv:1906.0266 (2019)
[31] Nair, V., Hinton, G.E.: Rectified linear units improve restricted boltzmann machines. In: Ieml (2010)
[32] Neyshabur, B.. Bhojanapalli. S., Mcallester, D., Srebro, N.: Exploring generalization in deep learning. Advances in Neural Information Processing Systems 30, 5947-5956 (2017)
[33] Neyshabur, B., Salakhutdinov, R., Srebro, N.: Path-sgd: normalizzdd optimization in deep neural networks. Proceedings of the 28th International Conference on Neural Information Processing Systems-
Volume 2. pp. 2422-2430 (2015)
[34] Neyshabur, B., Tomioka, R., Srebro, N.: Norm-hased capacity control in neural networks. In: Conference Learning Theory, pp. 1376-1401. PMLR (2015)
[35] Nielsen, F: An elementary introduction to information geometry. Entropy 22((0), 1100 (2020)
[36] Papyan, V:: The full spectrum of deepnet hessians scale: Dynamics with sgd training and sample size. arXiv preprint arXiv:1811.007
(2018)
[37] Papyan, V.: Measurements of three-level hierarchical structure in the outliers in the spectrum of deepnet
hessians. In: International Conference Machine Learning, pp. 5012-5021. PMLR (2019)
[38] Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang. E., DeVito, Z., Lin, Z., Desmaison, A., Antiga. L.,
Lerer, A.: Automatic differentiation in pytorch (2017)
[39] Petzka, H., Kamp, M., Adilova, L.., Sminchisescu, C., Boley, M.: Relative flatness and generalization. Advances in Neural Information Processing Systems 34 (2021)
[40] Rangamani, A., Nguyen, N.H., Kumar, A., Phan, D., Chin, S.P.. Tran, T.D.: A scale invariant measure of
flatness for deep network minima. In: ICASSP 2021-2021 IEEE International Conference on Acousties, Speech and Signal Processing (ICASSP), pp. 1680-1684, IEEE (2021)
[41] Sagun, L., Evei, U., Guney, V.U., Dauphin, Y., Bottou, L.: Empirical analysis of the hessian of over- parametrized neural networks. arXiv preprint MiXiv:1706.045555 (2017)
[42] Simonyan, K. Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1409.1556 (2014)
[43] Smith, S.L., Dherin, B., Barrett, D.G., De, S.: On the origin of implicit regularization in stochastic gradient descent. arXiv preprint arXiv:2 10t::2111.12177 (2021)
[44] Takeuchi, K.: The distribution of information statistics and the criterion of goodness of fit of models. Mathematical Science 153, 12-18 (1976)
[45] Thomas, V., Pedregosa, F., Merriënboer, B., Manzagol, P.A., Bengio, Y., Le Roux, N.: On the interplay between noise and curvature and its effect optimization and generalization. In: International Conference on Artificial Intelligence and Statistics, pp. 3503-3513, PMLR (2020)
[46] Tsuzuku, Y., Sato, I., Sugiyama, M.: Normalized flat minima: Exploring scale invariant definition of
flat minima for neural networks using pac- bay-baye analysis. In: International Conference on Machine Learning, pp. 9636 9647, PMLR (2020)
[47] Wu, J.. Hu, W., Xiong, H., Huan, J.. Braverman, V., Zhu, Z. On the noisy gradient descent that generalizes as sgd. In: International Conference on Machine Learning, pp. 10367-10376. PMLR (2020)
[48] Yao, Z., Gholami, A., Keutzer, K., Mahoney, M.W.; W.: Pyhessian: Neural networks through the lens of the hessian. In: 2020 IEEE International Conference on Big Data (Big Data), pp. 581-590. IEEE (2020)
[49] Yao, Z., Gholami, A., Lei, Q., Keutzer, K., Mahoney, M. W.: Hessian- analysis large batch training and robustness to adversaries. Advances Neural Information Processing Systems (2018)
[50] Zagoruyko, S., Komodakis, N.: Wide residual networks. arXiv preprint
(2016)
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] The need to consider information geometry in mea- suring sharpness is discussed in Section 2 the definition of our sharpness measure, its reparametrizaaionninaaare invariance,, and its connection to generalization are explained in Section B and the experiments to use measure regularizer in neural network training are performed Section
(b) Did you describe the limitations of your work? [Yes] See Section S
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] We have read the ethics review guidelines and checked that our paper conforms them.
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Section 3.1
(b) Did you include complete proofs of all theoretical results? (Yes] See Appendix D.I
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes] See supplemen-
tal materials.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Appendix G
(c) Did you report error bars (c.g., with respect to the random seed after running experi- ments multiple times)? [Yes] See Tables and
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Appendix G.4.2
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] See Appendix G
(b) Did you mention the license of the assets? [Yes] See Appendix G
(c) Did you include any new assets either the supplemental material or as URL? [Yss]
See supplemental materials.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
13


Reviews:
[{"review_id": "uFsuizr8qQp", "reviewer": "Reviewer_uwH8", "summary": "The article proposes a novel flatness measure for probabilistic classification models, parametrized as neural networks. The measure is computed as combination of inverse Fisher information matrix and first order derivatives of the loss function (formula (6)). The measure is justified from the information geometry perspective, where FIM allows to measure the difference between distributions. The other parts of the flatness measure are justified by introducing a similarity to the Takeuchi's information criterion, that nicely correlates with generalization. The measure is provably indifferent to a general reparametrization, when the parameters of the model are mapped to another set of parameters, as compared to other proposed measures, that are invariant only to scale reparametrizations. It is empirically demonstrated that introducing the proposed measure as regularization to the training procedure improves generalization.", "questions": "1 - Is there a significant difference between the properties of the Hessian and FIM: the discussion in the introduction attracts attention to the different directions that contribute more or less to the change of the considered function, but it is as well property of the Hessian?\n2 - It seems that any modification of the parameters will not be changing the proposed flatness measure due to its form. Is it so and does it mean that even for different models the measure can be the same?\n\n----\nI thank the authors for the detailed answers in the rebuttal phase. Nevertheless, I still have my doubts about the reparametrizations used and the consequences of being insensitive to them, therefore I stay with my initial score.", "limitations": "It would be beneficial to address the limitations of the theoretical connection to generalization, in a sense of similarity to empirically proven indicators of generalization. Also, more detailed analysis of the limitations of the computation of the proposed measure would be in place.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The proposed measure addresses the general reparametrization problem, that is not restricted to the scaling reparametrizations of the ReLU networks. It is provably invariant and helps the training process when used as regularizer. \nThe idea is presented very clear with sufficient explanations and good reproducibility of the experiments.\nNevertheless, it is hard to pinpoint how exactly this flatness measure improves the understanding of the training procedure and helps to create a link to generalization. Different from Petzka et al. this work does not mathematically represent the test loss or generalization gap in terms of the proposed flatness measure, only the indicative connection to TIC and max-margin are provided. In the empirical sense, as a regularization, the measure seems to perform on par with other flatness based regularizers as SAM and ASAM (table2 in appendix).\nThe correlation with generalization is reported only in appendix and it is not clear if the proposed flatness can be used as an indicator of good generalization, or it is only improving the training process.\n\nImportant to the originality of the proposed measure is the definition of reparametrizations. It would be good to provide it in the introduction already - it is very common in the literature to call scalings as reparametrizations, while explicit bijection of parameter space is not so common. Also, it seems to be hard to define it: Dinh et al. paper is referenced for an example of the bijection, but a reparametrization proposed there in the Section5 is batch normalization. The reparametrization demonstrated in the paper experiments is most probably taken from the appendix of the Dinh et al. paper and it has modified formulas - I would find it very useful if it is demonstrated that this bijection is indeed not affecting the network output (and loss) and also more attention is put on the explanation and variety of such reparametrizations.\n\nIt is mentioned in the paper that FIM is hard to compute, especially for large models, but it was not quite clear explained how this was addressed in the experiments: 4.1 does not contain the way it was computed, 4.2 proposes approximation to the gradient, and the generalization correlation contains another approach.\n\nMinor:\nFigure 4 would benefit from the explanation of why (a) shows less regular boundaries than (b), because they look somehow similar.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "nD4mXH2mCzC", "reviewer": "Reviewer_Mksd", "summary": "This paper presents a sharpness measure based on the Fisher information matrix of the probability distribution parameterized by a neural network. This measure takes into account reparameterizations other than just the rescaling of the weight matrices, and extends the invariance to any continuous transformation of the neural network parameters. The authors provide comparisons between their sharpness measures and other scale-invariant measures. They also run experiments showing that adding their measure as a regularizer can improve generalization. ", "questions": "The entire development in the paper seems to be tied to the cross entropy loss. While that is standard for classification, the square loss is also widely used for classification. Is the IGS measure easily adaptable to that case?\n", "limitations": "Yes", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is well written and the sharpness measure is well motivated. The experiments are rigorous and the visualization of decision boundaries is informative.\n\nWeaknesses:\nThe comparison against other scale-invariant sharpness measures is illustrative but a little unfair since those measures were only designed to be invariant to scale (especially the one based on the quotient manifold). \n\nThe sections on theoretical connections to generalization only show how the IGS measure can be plugged into other criteria that are correlated with generalization. This paper does not answer the larger question of how sharpness can explain generalization - and whether this is a phenomenon unique to deep learning, or whether it explains generalization in other statistical learning problems.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "AxEsyuRevju", "reviewer": "Reviewer_q6RN", "summary": "The paper develops a new sharpness measure based on Fisher information. The proposed measure is invariant to scaling and reparametrization and is connected to generalization performance. Experiments shows that using the proposed sharpness measure as a regularizer significantly improves generalization for SGD, and achieves comparable performance as other sharpness-aware methods.", "questions": "- In addition to the time reported at the end of the appendix, is it possible to provide more details about the added time complexity of using the (approximated) Fisher information matrix? \n- Proposition 3.1 seems to be more general than scaling invariance. In different architectures, are there other mappings other than scaling that satisfy the assumptions in the proposition? \n", "limitations": "The authors have addressed limitations in the conclusion. \n", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The use of Fisher information in developing a scale- and reparametrization- invariant sharpness measure is novel and well motivated. \n- The paper is technically sound. The authors present the background and mathematical details in a way that is easy to read. \n- The connections to existing concepts in generalization are interesting and insightful. \n\nWeaknesses:\n- From Table 1, it is not clear that regularization using the proposed sharpness measure can outperform other sharpness-aware methods. Additionally, the proposed method is slower than SAM and ASAM, which makes its empirical advantage unclear. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "-1wnbegBSCT", "reviewer": "Reviewer_fVF7", "summary": "The paper investigates the spectrum of the Fisher Information Matrix (FIM) and empirically observes that the eigenspace of large eigenvalues is connected to large shifts of the decision boundary of the network. The authors then propose a novel sharpness measure that is based on the pseudo-inverse of the FIM and the loss gradients at each sample. In comparison to existing measures, the proposed measure is shown to be reparameterization-invariant (instead of just scale-invariant). A theoretical connection to generalization is provided by presenting the proposed measure as an approximation to Takeuchi’s information criterion, and showing that a measure of margin for a sample point can be formulated in terms of the proposed sharpness measure at this datapoint. Empirically, it is shown that using the sharpness measure as a regularization term during training, can improve generalization performance.\n\n", "questions": "Could the authors provide specifics in which sense they would argue their measure has a a theoretical connection to generalization that other sharpness measures are lacking? \n\nWhat are the theoretical advantages over the TIC in terms of its connection to generalization?\n\nCan the authors present evidence that their sharpness measure is experimentally advantageous to other measures?\n\nCan the authors present a (weight) reparameterization of a neural network (with fixed architecture), that is not covered by scale invariance? In other words, are the authors aware of a (classical deep) neural network architecture that allows two distinct weight configurations, not in linear dependence, that lead to the same network function ?", "limitations": "As explained in detail above, I don't think that the limitations of the proposed measure are fairly discussed. According to the presentation, the proposed measure does not provide a stronger theoretical connection to generalization than competing measures, neither is it shown to outperform other measures experimentally.\n", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n\n\\+ The paper proposes a reparameterization-invariant sharpness measure. As competing measures are only shown to be scale-invariant, this is a clear improvement over other measures.  While this theoretical observation would be sufficient, it is backed up by a clear empirical result.\n\n\\+ The FIM is presented as a natural candidate for a measure due to its theoretical properties. In addition, by investigating the spectrum of the FIM, the authors empirically observe that the eigenspace of large eigenvalues is connected to large shifts of the decision boundary of the network.  This helps to naturally place the proposed measure into existing theory.\n\n\\+ Moreover, the paper illustrates interesting connection to the margin and the Takeuchi information criterion, which further offers indications that the proposed measure could be related to generalization.\n\n\\+ The empirical evaluation (on MINST, CIFAR10 and CIFAR100) supports this theoretical observations by showing that by using the sharpness measure as a regularization term during training, the performance of networks is improved.\n\n\\+ The paper is very well-written and easy to follow. The supplements add all the details left out in the main article.\n\nWeaknesses:\n\n\\- The authors propose a novel sharpness measure with theoretical and empirical connections to generalization, but it remains unclear how the measure compares to competing measures. While the papers present their measure as largely superior in both theoretical and empirical properties, this is not a fair placement in related work:\n\n(i) Theory: The authors write that for sharpness measures,\"any theoretical insights into generalization performance“ (Line 5) are \"still lacking“ and that they address other measures “inability to explain generalization performance of deep learning networks“ (Line 39). However, other sharpness measures have been theoretically connected to generalization, for example in terms of generalization bounds. At the same time, the presented connection to generalization for the proposed measure is not that strong. Firstly, the proposed measure is an approximation to another measure (TIC) that has been connected to correlation empirically. If the authors aim to borrow the theoretical properties of the TIC for their connection to generalization, then the TIC itself (i.e. replace the FIM with the Hessian in the proposed measure) would be the superior measure (a measure that has been discussed in related work). Secondly, the connection to the margin requires a (possibly tailored) novel definition of margin .\n\n(ii) Experiments: The experiments successfully show that the proposed measure can be employed as a regularization term to improve generalization. However, the experiments do neither show that this approach outperforms existing approaches (SAM, ASAM), nor do they show that the proposed measure outperforms other sharpness measures. Other sharpness measures have been similarly applied to improve generalization, but no comparison is provided.  There is also no comparison of different sharpness measures in their ability to explain generalization. In particular, it would be important to compare to the TIC as used in the reference [44] to the proposed measure. \n\n(iii) Reparameterization-invariance: A clear improvement of the proposed measure over other sharpness measures is its reparameterization-invariance, which goes beyong scale-invariance. However, the benefit may not be as clear as it first seems. The reparameterizations that are considered as evidence for the failing of other measures, are not reparameterizations of the weights in a network, but they go beyond the (classical) setting of neural networks. So, to my knowledge, the linear reparameterizations (that competing measures are invariant to) may be the only reparameterizations that play a role in the setting of neural networks.\n\n\\- The motivation for introducing the novel sharpness measure is not well-explained. It seems plausible that the Takeuchi’s information criterion (TIC) and/or the calculations on the margin in 3.3.2 motivated the definition of the novel measure, but the paper presents no explicit motivation for the introduction of the measure (\"Based on the above analysis\" in line 169 does not point to anything that explains the explicit form of the measure)  and then presents a relation to the TIC and the margin.\n\n*****\n\nTaken together, the paper offers an interesting addition to the set of sharpness measures and their ability to explain generalization. However, the paper does not present clear evidence that the proposed measure is superior to other sharpness measures empirically or theoretically. Unless advantages and motivation could be worked out more clearly, the paper should not claim its superiority over related work. \n\n-----\nTypos:\n\n25: masures\n\n242: Possibly revise this statement, which says that \"When we perform the maximum likelihood estimation [..], we can obtain the maximum likelihood estimate [..]\"\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept