Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper provides a metric to characterize local minima of deep network loss landscapes based on the Fisher information matrix of the model parameterized by the deep network. The authors connect the Fisher information to the curvature of the loss landscape (the loss considered is the negative loss likelihood) and obtain generalization bounds through PAC Bayes analysis. They further propose regularizing the training of deep networks using the local curvature of the loss as a regularizer. In the final experimental section of the paper, the relationship between the empirical measures and generalization is shown on a variety of networks.

This is an interesting paper, but I have a few concerns.

1. The information-theoretic measure that is proposed is essentially the (log) determinant of the hessian of the loss function.  If there are degenerate eigendirections (zero eigenvalues) then the proposed measure would not be able to distinguish between minima with different numbers of degenerate directions / same number of degenerate directions but different spectral norms of the hessians. If the authors contention is that there will be no zero eigenvalues, that suggests that local minima of deep networks are all strict, isolated minima, contrary to recent work on connected solutions (See Draxler et. al. 2018, Essentially No Barriers in Neural Network Energy Landscapes, ICML 2018).

2. I would like to see how the authors believe their measure deals with rescalings layer parameters in deep networks, ie the issue brought up by Dinh et. al. in "Sharp Minima can Generalize for Deep Networks" ICML 2017. While I can see that the log determinant is invariant, it is not clear that the proposed approximation will be invariant to rescaling of deep network layer parameters. If the parameters corresponding to the eigenvalues sampled in the approximation are rescaled, I believe the proposed measure will not be invariant.

3. The experiments regarding the local minima characterization are well constructed, though some details are missing such as how the authors decided that training had converged to a local minimum. As far as regularization based on the local curvature is concerned, I would like to see some more experiments that compare the proposed technique to adagrad/adam and other techniques that purport to condition the gradient based on local curvature. It would also be interesting to see whether the regularization indeed converges to flatter minima characterized by the proposed flatness measure. Since the claim is that the regularizer gets you flatter solutions, that information is important to decide whether the proposed technique is performing as advertised.

I am willing to update my score based on responses to these concerns.
review_title: Official Blind Review #3

final_score: rating: 1: Reject
review_content: Post-rebuttal update: I have just noticed the authors modified their summary post below and claimed "[my concerns] are all minor or resolved". This is not true. Here is my summary of unresolved concerns written after the discussion period.

This work has been substantially improved during the rebuttal process, and some of my concerns are addressed. But there are still major issues, as raised in my [last comment]( https://openreview.net/forum?id=BJlXgkHYvS&noteId=r1xAnokijS ), that remains unanswered. Specifically,

(A) the relation between this work and information theory

In the revision, the authors have make it very clear that the relation between FIA and their proposed regularized objective is very vague, relying on the crude approximation of expected Fisher information with observed Fisher information. Therefore the "information-theoretic" part in the title seems awkward and to some extent, misleading.

As Reviewer 1 has pointed out, it would have been better if the authors relate their theory and method to the observed FIM, instead of information theory, from the beginning. Since the observed FIM and the neural tangent kernel (NTK) share the same eigenspectrum, it would also be interesting to relate this work to the NTK.

(B) the different behavior of the proposed regularization (log det(I)) and its bound that is actually implemented (log tr(I))

This is the more important issue. My concern is that the observed FIM (or the NTK) is known to have fast decaying spectrum; (Karakida et al) has shown empirically that the decay can be exponential. Thus log det(I) would be dominated by the long tail (since after taking logarithm it is the sum (or average) of an arithmetic sequence), while log tr(I) would be dominated by the first largest few values. 

The authors claim that this is not an issue since they replaced the observed FIM with a subsampled, low-rank (<=10), version. It corresponds to consider a small submatrix of (the gram matrix of) the NTK. Denote this matrix as .
(a) This does not help with the problem, since we now have no chance of recovering the smaller eigenvalues that would have dominated log(det(I)), and it is impossible that the proposed regularizer has a similar behavior to log(det(I)).
(b) One could verify easily, using small feed-forward networks (or even simpler, computing a gram matrix using RBF kernels, since the FIM shares its eigenspectrum with NTK which is a p.d. kernel), that the new matrix  still has a fast-decaying eigenspectrum, so the behavior of  and \tilde{I} are still significantly different, even though this cannot be established by concentration bounds as the authors argue. While FFN and modern deep architectures can have different behaviors, I believe the above evidence suggests that a numerical experiment comparing the behavior of the two bounds is a must.

Following this argument we can see *another issue* of this work, namely the proposed generalization bound will be vacuous given the fast-decaying spectrum of the FIM, since it contains gamma=log(det(I)).

Reviewer 1 mentioned this work could enlighten future discussions on this subject. While I agree this paper presents interesting empirical observations (namely its final algorithm, which is vaguely connected to the proposed objective, leads to improved performance on CV tasks), I think this submission in its current form is a bit too misleading to serve this purpose well, and overall I believe it would be better to go through another round of revision.


Original Review
============================================

This paper presents a generalization bound based on Fisher information at a local optima, and proposes to optimize (an approximation to) it to get better generalization guarantees. There are issues in both parts, and I don't think it should be accepted. Specifically,

1. The definition of Fisher information is incorrect (for almost every parameter). The expectation should be taken w.r.t the model distribution p(c_x|x;w), instead of the data distribution S. 
2. Assumption (1) (loss locally quadratic) is not reasonable for DNNs, since local optimas will not be unique in their neighborhoods. See e.g. Section 12.2.2, "Information Geometry and Its Applications".
3. Regarding the approximation to the bound, approximating log det(I) with log trace(I) is not a good idea: adding a very small eigenvalue will lead to noticeable change in the former, but negligible change in the latter. This is particularly problematic for DNNs, since the spectrum of their Fisher information matrix varies in a wide range: see "Universal Statistics of Fisher Information in Deep Neural Networks: Mean Field Approach". 

(Edit 11.8:
* regarding point (1), there is a quantity called observed Fisher information in e.g. Grunwald (2007) that coincide with Eq (1) in the paper, but it is a function of the dataset instead of the model parameter, and can only used to study model parameters near the *global optima* (as it is applied in Grunwald (2007)); it cannot help with choosing between different local optimas as this work claims. Additionally, the FIA criterion, which is used in this paper to devleop the generalization bound, is defined using the standard form of Fisher information (i.e. taking expectation w.r.t model distribution), see Rissanen (1996). These facts lead me to believe this is a confusion on the authors' part.
* in point (2) I was referring to the authors' argument " Since L(S,w) is analytic and w_0 is the *only local minimum* of L(S,w) in M(w_0)", which is incorrect.)
review_title: Official Blind Review #2

final_score: rating: 8: Accept
review_content: This paper contributes to the deep learning generalization theory, mainly from the theoretical perspective with experimental verifications. The key proposition is given by the unnumbered simple equation in the middle of page 4 (please number it), where \mathcal{I} is the Fisher information matrix. According to the authors, this simple metric, which is the log-determinant of the Fisher information matrix, can characterize the generalization of a DNN.

Remarkably, this piece of work is well written in terms of English and formulations, and complete, with a rigorous theoretical analysis (section 5.1, 5.2), practical approximations (section 5.3) and empirical verifications (section 6).

On the theoretical side, this work builds upon Rissanen's formulation of the MDL principle, which has two parts (describing data given the model as well as the model complexity). Under rough approximations, the complexity term becomes the log-determinant of the Fisher information matrix evaluated at the local (global) optimum. This simple approximation is further proved to upper-bounds the generalization error as stated in theorem 1.

To make the criterion to be practically useful, the author used the Jensen inequality so that the metric simply depends on the trace of the Fisher information matrix.

The empirical study showed the usefulness of the proposed metric which can well approximate the testing error and a regularization term (based on the trace of the Fisher information matrix) that can improve generalization on real DNN experiments.

The reviewer has the following minor comments to further improve this contribution:

section 5.1, explain the abbreviation FIA

Regarding the choice of the neighborhood \mathcal{M}(w_0), what is the reason to define the model (neighbourhood of w_0) based on the loss? Why not simply take a coordinate neighborhood?

According to your metric, the smaller the scale of the Fisher information matrix, the better the generalization. In section 5.1, there has to be some remarks on the intuition and related works on the flatness of the local minimum that is related to generalization.

As this contribution is related to the spectral properties of the Fisher information matrix, the reviewer points the authors to "Universal Statistics of Fisher Information in Deep Neural Networks: Mean Field Approach. Karakida et al. 2018." and "Lightlike Neuromanifolds, Occam's Razor and Deep Learning. Sun and Nielsen. 2019", which deals with asymptotic cases and have similar MDL formulations expressed in terms of the spectrum of the Fisher information matrix.

review_title: Official Blind Review #1

Meta-review: This paper proposes using the Fisher information matrix to characterize local minima of deep network loss landscapes to indicate generalizability of a local minimum. While the reviewers agree that this paper contains interesting ideas and its presentation has been substantially improved during the discussion period, there are still issues that remain unanswered, in particular between the main objective/claims and the presented evidence. The paper will benefit from a revision and resubmission to another venue.
Decision: Reject