Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper presents a sharpness measure based on the Fisher information matrix of the probability distribution parameterized by a neural network. This measure takes into account reparameterizations other than just the rescaling of the weight matrices, and extends the invariance to any continuous transformation of the neural network parameters. The authors provide comparisons between their sharpness measures and other scale-invariant measures. They also run experiments showing that adding their measure as a regularizer can improve generalization. 
strengths_and_weaknesses: The paper is well written and the sharpness measure is well motivated. The experiments are rigorous and the visualization of decision boundaries is informative.

Weaknesses:
The comparison against other scale-invariant sharpness measures is illustrative but a little unfair since those measures were only designed to be invariant to scale (especially the one based on the quotient manifold). 

The sections on theoretical connections to generalization only show how the IGS measure can be plugged into other criteria that are correlated with generalization. This paper does not answer the larger question of how sharpness can explain generalization - and whether this is a phenomenon unique to deep learning, or whether it explains generalization in other statistical learning problems.
questions: The entire development in the paper seems to be tied to the cross entropy loss. While that is standard for classification, the square loss is also widely used for classification. Is the IGS measure easily adaptable to that case?

limitations: Yes
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper develops a new sharpness measure based on Fisher information. The proposed measure is invariant to scaling and reparametrization and is connected to generalization performance. Experiments shows that using the proposed sharpness measure as a regularizer significantly improves generalization for SGD, and achieves comparable performance as other sharpness-aware methods.
strengths_and_weaknesses: Strengths:
- The use of Fisher information in developing a scale- and reparametrization- invariant sharpness measure is novel and well motivated. 
- The paper is technically sound. The authors present the background and mathematical details in a way that is easy to read. 
- The connections to existing concepts in generalization are interesting and insightful. 

Weaknesses:
- From Table 1, it is not clear that regularization using the proposed sharpness measure can outperform other sharpness-aware methods. Additionally, the proposed method is slower than SAM and ASAM, which makes its empirical advantage unclear. 
questions: - In addition to the time reported at the end of the appendix, is it possible to provide more details about the added time complexity of using the (approximated) Fisher information matrix? 
- Proposition 3.1 seems to be more general than scaling invariance. In different architectures, are there other mappings other than scaling that satisfy the assumptions in the proposition? 

limitations: The authors have addressed limitations in the conclusion.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The article proposes a novel flatness measure for probabilistic classification models, parametrized as neural networks. The measure is computed as combination of inverse Fisher information matrix and first order derivatives of the loss function (formula (6)). The measure is justified from the information geometry perspective, where FIM allows to measure the difference between distributions. The other parts of the flatness measure are justified by introducing a similarity to the Takeuchi's information criterion, that nicely correlates with generalization. The measure is provably indifferent to a general reparametrization, when the parameters of the model are mapped to another set of parameters, as compared to other proposed measures, that are invariant only to scale reparametrizations. It is empirically demonstrated that introducing the proposed measure as regularization to the training procedure improves generalization.
strengths_and_weaknesses: The proposed measure addresses the general reparametrization problem, that is not restricted to the scaling reparametrizations of the ReLU networks. It is provably invariant and helps the training process when used as regularizer. 
The idea is presented very clear with sufficient explanations and good reproducibility of the experiments.
Nevertheless, it is hard to pinpoint how exactly this flatness measure improves the understanding of the training procedure and helps to create a link to generalization. Different from Petzka et al. this work does not mathematically represent the test loss or generalization gap in terms of the proposed flatness measure, only the indicative connection to TIC and max-margin are provided. In the empirical sense, as a regularization, the measure seems to perform on par with other flatness based regularizers as SAM and ASAM (table2 in appendix).
The correlation with generalization is reported only in appendix and it is not clear if the proposed flatness can be used as an indicator of good generalization, or it is only improving the training process.

Important to the originality of the proposed measure is the definition of reparametrizations. It would be good to provide it in the introduction already - it is very common in the literature to call scalings as reparametrizations, while explicit bijection of parameter space is not so common. Also, it seems to be hard to define it: Dinh et al. paper is referenced for an example of the bijection, but a reparametrization proposed there in the Section5 is batch normalization. The reparametrization demonstrated in the paper experiments is most probably taken from the appendix of the Dinh et al. paper and it has modified formulas - I would find it very useful if it is demonstrated that this bijection is indeed not affecting the network output (and loss) and also more attention is put on the explanation and variety of such reparametrizations.

It is mentioned in the paper that FIM is hard to compute, especially for large models, but it was not quite clear explained how this was addressed in the experiments: 4.1 does not contain the way it was computed, 4.2 proposes approximation to the gradient, and the generalization correlation contains another approach.

Minor:
Figure 4 would benefit from the explanation of why (a) shows less regular boundaries than (b), because they look somehow similar.

questions: 1 - Is there a significant difference between the properties of the Hessian and FIM: the discussion in the introduction attracts attention to the different directions that contribute more or less to the change of the considered function, but it is as well property of the Hessian?
2 - It seems that any modification of the parameters will not be changing the proposed flatness measure due to its form. Is it so and does it mean that even for different models the measure can be the same?

----
I thank the authors for the detailed answers in the rebuttal phase. Nevertheless, I still have my doubts about the reparametrizations used and the consequences of being insensitive to them, therefore I stay with my initial score.
limitations: It would be beneficial to address the limitations of the theoretical connection to generalization, in a sense of similarity to empirically proven indicators of generalization. Also, more detailed analysis of the limitations of the computation of the proposed measure would be in place.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper investigates the spectrum of the Fisher Information Matrix (FIM) and empirically observes that the eigenspace of large eigenvalues is connected to large shifts of the decision boundary of the network. The authors then propose a novel sharpness measure that is based on the pseudo-inverse of the FIM and the loss gradients at each sample. In comparison to existing measures, the proposed measure is shown to be reparameterization-invariant (instead of just scale-invariant). A theoretical connection to generalization is provided by presenting the proposed measure as an approximation to Takeuchi’s information criterion, and showing that a measure of margin for a sample point can be formulated in terms of the proposed sharpness measure at this datapoint. Empirically, it is shown that using the sharpness measure as a regularization term during training, can improve generalization performance.


strengths_and_weaknesses: Strengths: 

\+ The paper proposes a reparameterization-invariant sharpness measure. As competing measures are only shown to be scale-invariant, this is a clear improvement over other measures.  While this theoretical observation would be sufficient, it is backed up by a clear empirical result.

\+ The FIM is presented as a natural candidate for a measure due to its theoretical properties. In addition, by investigating the spectrum of the FIM, the authors empirically observe that the eigenspace of large eigenvalues is connected to large shifts of the decision boundary of the network.  This helps to naturally place the proposed measure into existing theory.

\+ Moreover, the paper illustrates interesting connection to the margin and the Takeuchi information criterion, which further offers indications that the proposed measure could be related to generalization.

\+ The empirical evaluation (on MINST, CIFAR10 and CIFAR100) supports this theoretical observations by showing that by using the sharpness measure as a regularization term during training, the performance of networks is improved.

\+ The paper is very well-written and easy to follow. The supplements add all the details left out in the main article.

Weaknesses:

\- The authors propose a novel sharpness measure with theoretical and empirical connections to generalization, but it remains unclear how the measure compares to competing measures. While the papers present their measure as largely superior in both theoretical and empirical properties, this is not a fair placement in related work:

(i) Theory: The authors write that for sharpness measures,"any theoretical insights into generalization performance“ (Line 5) are "still lacking“ and that they address other measures “inability to explain generalization performance of deep learning networks“ (Line 39). However, other sharpness measures have been theoretically connected to generalization, for example in terms of generalization bounds. At the same time, the presented connection to generalization for the proposed measure is not that strong. Firstly, the proposed measure is an approximation to another measure (TIC) that has been connected to correlation empirically. If the authors aim to borrow the theoretical properties of the TIC for their connection to generalization, then the TIC itself (i.e. replace the FIM with the Hessian in the proposed measure) would be the superior measure (a measure that has been discussed in related work). Secondly, the connection to the margin requires a (possibly tailored) novel definition of margin .

(ii) Experiments: The experiments successfully show that the proposed measure can be employed as a regularization term to improve generalization. However, the experiments do neither show that this approach outperforms existing approaches (SAM, ASAM), nor do they show that the proposed measure outperforms other sharpness measures. Other sharpness measures have been similarly applied to improve generalization, but no comparison is provided.  There is also no comparison of different sharpness measures in their ability to explain generalization. In particular, it would be important to compare to the TIC as used in the reference [44] to the proposed measure. 

(iii) Reparameterization-invariance: A clear improvement of the proposed measure over other sharpness measures is its reparameterization-invariance, which goes beyong scale-invariance. However, the benefit may not be as clear as it first seems. The reparameterizations that are considered as evidence for the failing of other measures, are not reparameterizations of the weights in a network, but they go beyond the (classical) setting of neural networks. So, to my knowledge, the linear reparameterizations (that competing measures are invariant to) may be the only reparameterizations that play a role in the setting of neural networks.

\- The motivation for introducing the novel sharpness measure is not well-explained. It seems plausible that the Takeuchi’s information criterion (TIC) and/or the calculations on the margin in 3.3.2 motivated the definition of the novel measure, but the paper presents no explicit motivation for the introduction of the measure ("Based on the above analysis" in line 169 does not point to anything that explains the explicit form of the measure)  and then presents a relation to the TIC and the margin.

*****

Taken together, the paper offers an interesting addition to the set of sharpness measures and their ability to explain generalization. However, the paper does not present clear evidence that the proposed measure is superior to other sharpness measures empirically or theoretically. Unless advantages and motivation could be worked out more clearly, the paper should not claim its superiority over related work. 

-----
Typos:

25: masures

242: Possibly revise this statement, which says that "When we perform the maximum likelihood estimation [..], we can obtain the maximum likelihood estimate [..]"



questions: Could the authors provide specifics in which sense they would argue their measure has a a theoretical connection to generalization that other sharpness measures are lacking? 

What are the theoretical advantages over the TIC in terms of its connection to generalization?

Can the authors present evidence that their sharpness measure is experimentally advantageous to other measures?

Can the authors present a (weight) reparameterization of a neural network (with fixed architecture), that is not covered by scale invariance? In other words, are the authors aware of a (classical deep) neural network architecture that allows two distinct weight configurations, not in linear dependence, that lead to the same network function ?
limitations: As explained in detail above, I don't think that the limitations of the proposed measure are fairly discussed. According to the presentation, the proposed measure does not provide a stronger theoretical connection to generalization than competing measures, neither is it shown to outperform other measures experimentally.
review_title: null

Meta-review: metareview: The expert reviewers appreciated the contributions and ideas in this paper and also liked their rebuttals.
It is also good to see new ideas being introduced to the important topic of flatness measures for deep learning.
Also, the presentation also enhances the paper's reputation.

In contrast, it is worth noting that their claims are somewhat excessive. The reviewers have pointed out specific modifications, hence I recommend to accept this paper on the condition that the points are modified.
confidence: Certain
Decision: Accept