Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper investigates properties of the logit Hessian spectrum, how it relates to the top-eigenvalue of the Hessian (which they refer to as 'sharpness' though it can be debated), and implies a particular implicit regularization of the Jacobian during an early phase of training (which is shown to yield better generalization). They use the derived relations to 'explain' previous observations on sharpness and generalization.
main_review: **Strengths:**
- While not totally novel, it is still a nice, relatively unexplored, approach to look at the low-dimensional logit Hessian to derive some insights. 
- In particular, Theorem 1 provides an interesting relation between eigenvalues of (per-sample) logit Hessian and corresponding probability output. This also provides a way to get some bounds on the top eigenvalue of the logit-Hessian (impurity) in terms of the largest probability for a class. The corresponding figures are also quite nice and convey the point. 
- The sharpness-impurity-jacobian behaviour and its connection to oscillatory, catapult aspects and others are interesting. (But unfortunately, unclear how confidently we can take away the message, see below. Also, the fact that such a relation between sharpness-impurity-jacobian exists is not too far-fetched, given the form of the Gauss-Newton term.)


**Weaknesses:**
1. Theoretically unmotivated usage of the particular 'Jacobian norm': In Section 5, the authors propose an approximate relation between the Hessian spectral norm, 'impurity', and the 'Jacobian norm'. However, the Jacobian norm is not the usual Jacobian Frobenius norm -- as one would expect or as used in related work (Hoffmann et. al. 2019) -- but is the 'norm' with the particular vector all-one vector (normalized to unit length). This usage is far from being innocuous. It seems that, ideally, the authors would have liked to have the eigenvectors of the logit Hessian, and in particular, q^(1), for the spectral-norm, as also evident from the discussion in the appendix --- but this is perhaps hard to come by practically. Hence the issue is that, in turn, the particular normalized all-one vector used is nothing but the eigenvector corresponding to the zero eigenvalue of the logit Hessian. This is very strange given the particular relationship they want to derive. 
Thus, it seems to be an unmotivated heuristic which "surprisingly" seems to hold in practice as the figure 3 shows. (On that note, in Figure 3, the ||H||_\sigma is the spectral norm of the entire Hessian or the Gauss-Newton term G alone? Over how many samples is it computed?)

2. Explicit regularization: Here, the authors do not use their version of the 'Jacobian norm', but rather just the Frobenius norm. Yes, both are closely related and perhaps ultimately regularizing either might have similar effects. But, why is such a choice made in the first place, while not following their own derivation? What happens if you explicitly regularize by your particular Jacobian norm ||J 1||^2? Otherwise, as such, it currently makes the whole 'implicit Jacobian regularization' slightly fishy. 

3. Behaviour of the actual Jacobian Frobenius norm: In Figure 3 and 5, it would be very relevant to see how does the Jacobian (Frobenius) norm behaves, not just the particular version of the 'Jacobian norm' used by the authors. Can you please add these to the figures?

4. What does the active regularization period correspond to in terms of the progress in loss or performance? This implicit Jacobian regularization claimed by the paper is only active with a particular (early) phase of training but does not indicate the values of the loss there (please provide them) --- thus, it is hard to concretely understand how 'early' is this phase as well as the fact that why or if it suffices to be present only in this phase?

5. Applicability is restricted to only cross-entropy (CE) loss: An important downside of their approach is that this only applies for CE loss, not something like MSE --- where, the logit-Hessian-equivalent is just the identity matrix. Thus, the offered explanations -- if correct -- will not describe the overall behaviour of generalization in neural networks, only the specific case of CE. I think the authors should very explicitly mention this downside. 

6. Other puzzling empirical aspects:
- Regarding the connection between the Jacobian and Fisher penalty, at what point are get Figures in appendix 11 computed? When is this valid in general? Is this valid, at least in the entire 'active regularization period'?
- Figure 7: Here most of the performance gains, with explicit regularization, are 5-10% which, in my opinion, are pretty large to be true. So, i would like to understand this better: what are the particular networks considered (especially the ones on the bottom-left side)? are the networks trained long enough to a similar level of loss (not just fixed # of epochs) so as to render them comparable? I believe the "real" boost, if so, is represented by the point in the upper-right of Figure 7 (right) --- but I would love if my belief is proved wrong.
- Figure 6: If i read the plot correctly, the trend of impurity, as well as the 'Jacobian-norm', for increasing LR and decreasing batch sizes seems to be reversed --- which is not clear why?

7. Presentation of the Sharpness-impurity-jacobian relation: This relation is also vaguely/poorly expressed.
- In Fig 3, when you say \lambda^(1) and J are averaged over the training set, what do you precisely mean in the context of the 'Jacobian-norm'? Is it that the Jacobian J is averaged over the training set and then || E(J) 1||^2 is computed or is it the expectation of the norm over the training set, i.e., E(||J_x 1||^2)? Then why is one preferred over the other? Do they have similar behaviour?
- Given that you refer to \lambda^(1) as per-sample impurity in the previous section, and here implicitly the average over training set is considered. Thus, I ask the authors to properly delineate these two usages, say via the horizontal bar at the top. 

8. Unclear/incorrect statements:
- The authors say that Gauss-Newton approximation sometimes fails in the later phase of training. However, I think they are being imprecise here, and strictly speaking, Gauss-Newton approximation (for the Hessian as a whole) can be justified only towards this late phase or end of training. See Sagun et. al., 2017 page 5 "At a point close to a local minimum ....". I guess what the authors mean to say this, only in the context of the top-eigenvalue, that in the late-phase the top-eigenvalue is not clearly isolated from the bulk (which makes sense). So, the authors should really clarify that top-eigenvalue approximation from the Gauss-Newton is valid only during early phase, rather than (incorrectly) conflate explanations as presently done. 
- " However, the generalization gap keeps increasing, which provides another counter-example that flat minima generalizes poorly": Where are the corresponding train and test loss curves based on which you make this inference?
- Figure 3 caption is also a bit weird. Can you specify what is the default optimizer, dataset, network? otherwise only some mixed and partial information is present.

summary_of_the_review: The analysis of the logit Hessian spectrum, in itself, is very interesting. But other than that, most of the other contributions are not so far-fetched or novel, the claim about implicit regularization is not motivated theoretically, many of the empirical aspects are unexplained and questionable, the approach is very specific to cross-entropy loss and thus likely not a general explanation if so. As a result, overall it is unclear if their offered explanations are conclusive and not speculative.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper investigates the dynamics of leading eigenvalues in the so called logit-Hessian matrix and tries to correlate that with the convergence to a good local optima. The work also tracks the behavior of the Jacobian matrix for the pre-softmax outputs and introduces the notion of implicit Jacobian regularization.
main_review: - Theorem 1: What is novel in the claim and proof compared to prior work? In particular, I am referring here to the contributions from linear algebra by Golub (1973), Golub & van Loan, and Bunch et al. (1978). For example, Eq. (9) is known as the secular equation of the eigen-decomposition problem for rank-one modification of a diagonal matrix. Results (a-c) are for sure quite well-known in that literature (this might be a misrepresentation of the theoretical contributions). So, the novel contribution here might only be (d), which should then be formulated as a separate proposition and Theorem 1 should be cited with Golub and others.

- Section 4.2: In the introduction, there has been a lot of stress on eigenvalues of the Hessian, which should be matrix H. This section, on the other hand, states that the evolution of top eigenvalues of matrix M are tracked. This is the matrix of an exponential family model that completely ignores all the layers of a neural network apart from the last one where the softmax operates. The relation to prior work by Cohen et al (2021) is also unclear -- what would be a novel finding compared to that work? In what sense, the current work extends the estimation procedure from that work?

- Figure 3 and opening of Section 5 is unclear and too informal for a conference publication. The appendix is optional and findings should be described clearly in the main part of the paper.

- Section 5.1: Impurity here refers to the top eigenvalue of M? This notion is not defined, there is just a bound of the top eigenvalue in terms of Gini impurity.

- Eq. (11): I did not follow the derivation for this?

- Section 5.2: Could you please clarify the implication that weight norm increase leads to increase in the Jacobian norm? The Jacobian norm should be about the geometry of the loss landscape which can be independent of the actual parameter values? Alternatively, if you start with all-zero parameters then this is to be expected.

- The reasoning on the Jacobian regularization is unclear. Would not Jacobian norm need to decrease for GD to be able to converge?
summary_of_the_review: The paper is poorly written and too informal in a number of places. There are many claims which are presented as facts but are in reality hypotheses. I have expected an analysis based on the Hessian of a neural network and it turns out that the work focuses only on the Hessian matrix of the exponential family model, completely ignoring the network parameters up to the sofmax operator. I also fail to grasp the relevance of the Jacobian regularization and do not see why the drop in the magnitude of Jacobian norm would be of interest.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In this paper, the relationship between the Jacobian (the gradient of the final activation w.r.t parameters) and the Hessian is analyzed for the softmax cross-entropy loss. As a key tool, the approximation of the Hessian with the probability vector (the softmax output) is used, which suggests connections with several optimization (and generalization) concepts such as sharpness v.s. flatness of the loss landscape and the oscillation on the optimization path at GD/SGD. 
main_review: Strengths 

1. This study links three concepts in optimization and generalization (flat minima, oscillation, gradient regularization). It is interesting to see how the diversity (i.e. entropy) of the softmax outputs is connected to the sharpness of the loss landscape. 
2. The eigenvalues of the approximated Hessian (logit Hessian) are theoretically analyzed (Theorem 1).
3. There are experiments that are intended to support the theoretical analysis.


Weaknesses

1. This paper often lacks clear descriptions of technical details that are necessary to be contained. I read this paper three times from the beginning, but I'm still puzzled. My questions are:
- What is the experimental setting of Figure 3? Appendix C4 says CNN is trained on CIFAR10, but the caption says MNIST and Resnet are also used. 
- What is the model architecture of Figure 4?
- In Eq. (11), the expectation \hat{E}_D is separately applied to \lambda and J. Why you can do that? I concern this operation yields a significant approximation error. Indeed, in Figure 5, at the point around 3500 steps ||J1|| reaches 1000 and \lambda^{(1)} takes around 0.1, which means ||H|| should be around 100 according to (11). However, the actual value is 50. This gap looks large. 
- There is a sentence "This explains why the behavior of the sharpness in the early phase of training seems to impact the final generalization." in Section 5.2. Could you elaborate on this? Why the sharpness in the early phase influences the final performance?
- The same paragraph says "λ acts as a regularization coefficient". Please clarify this. Do you mean the objective is approximated as `original_loss + λ ||H||_σ`? If so, please derive this mathematically. 
2. Jacobian regularization is not significantly evaluated. This study suggests using the Frobenius norm of the gradient as a regularizer. However, its effect is not comprehensively examined in an empirical manner. How is it better than popular regularizers (e.g. weight decay, l1, gradient penalty, Jacobian regularizers described in [*])? How is the effect consistent for different architectures and datasets?
3. The motivation is not solid. As is described in Section 2, there are several studies working on the decomposition of the Hessian (e.g. Papyan (2019)). Why the decomposition H ~= JMJ should be studied in this paper?


[*] https://www.ejournals.eu/pliki/art/13926/
summary_of_the_review: This paper provides a new view that connects three notions in optimization and generalization (flat minima, oscillation, gradient regularization) with theoretical and empirical supports. However, the current manuscript lacks clear descriptions; without these, I cannot evaluate the correctness. Furthermore, the proposed method (Jacobian regularization) is not significantly evaluated. Also, the motivation of using the proposed decomposition is not grounded.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors study the largest eigenvalue and eigenvector of the Hessian of the loss function. The authors approximate the Hessian matrix by a low-dimensional matrix, which is a rank-one modification of a diagonal matrix. The eigendecomposition helps to explain how the sharpness influences the gradient descent method and the generalization properties. They show that GD has implicit regularization effects on the Jacobian norm weighted with the impurity of the probability output, which is also related to the Fisher information matrix. 
main_review: The paper is overall well written with clear mathematics. 

Cons:  Theorem 1 is enlightening. 

Weakness:  Can the author provide some analytical examples. E.g., how do the neural network structures and activation functions take effect in the logit Hessian matrix?  A  two-layer neural network example is needed.

In addition, there is essential literature on related information matrices, which are missed.

Li, Zhao, Wasserstein information matrix, arXiv:1910.11248
summary_of_the_review: The paper proposes to study an approximate Hessian operator of the loss function. This is helpful in understanding the fundamental non-convexity questions in learning. 

A more analytical example is needed to improve the current paper. The example should address how the simplified Hessian matrix depends on the neural network functions and architectures.
review_title: null

Meta-review: The paper studies an interesting question of the relationship between the eigenvalues of the Hessian matrix with the probability of the output in the logistic loss, and use this to propose a regularization that can improve the performance of the neural networks.

 All the reviewers agree that although the question is interesting, the paper lacks significantly in terms of representation and would benefit from another round of revision.
Decision: Reject