Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: # Summary
This work claims SGD regularizes the model through penalizing the trace of Fisher in the early phase. This claim is supported by the similar generalization behavior of SGD (with optimal learning rate) and Fisher penalization (for SGD with small learning rate). A series experiments are conducted to verify the understanding.

I find the paper writing is rather misleading. On the one hand, no mathematical justifications, and the logical chain is weak --- hard to claim which factor is the cause and which one is the effect. On the other hand, though written as "trace of Fisher" in the paper, in experiments they compute a different regularizer --- norm of expected gradient. Detailed questions come in the following.

# Questions

1. Abstract, "We highlight that in the absence of implicit or explicit regularization...". How do you get rid of the implicit regularization???

2. Eq.(2) is super misleading. Trace of Fisher is the expected gradient norm, but in Eq. (2) you compute norm of expected gradient. Statistically the latter has nothing to do with the former: one is the second moment, and the other is the squared first moment. Please justify.

3. I have a very simple explanation to your observed phenomenon. In the beginning, gradient is large, thus your version of "trace of fisher" is large, as the gradient decreases, the "trace of fisher" decreases. Now we look at large learning rate and small learning rate. With large learning rate, SGD converges faster, thus its gradients decrease faster, which causes the "trace of fisher" decreases faster. But with small learning rate, SGD converges slower, thus its gradient decrease slower, and the "trace of fisher" appears to be large in the beginning epochs.

Therefore I am not at all convinced by your arguments. 

4. From the above discussion, at least you need to normalize the "trace of fisher" by gradient norm --- which then becomes a measurement of gradient confusion. See following for references.

5. FP/GP. If I understand correctly, you need to compute gradient of gradient, which expands your computation graph at least twice. Can you report the GPU memory consumption? 

6. Finally, let us take about the practical role of FP. According to Table 1, the improvement of FP over GP is marginal. I cannot see a potential of FP. Not only in theory, but also in practice this paper is not satisfactory. 


# Missing Refs
Tons of theory paper should be discussed.  A few of them come to my brain are listed in below. Please do a more complete literature investigation.

Fisher
- Liang, Tengyuan, et al. "Fisher-rao metric, geometry, and complexity of neural networks." The 22nd International Conference on Artificial Intelligence and Statistics. 2019.
- Karakida, Ryo, Shotaro Akaho, and Shun-ichi Amari. "Universal statistics of Fisher information in deep neural networks: Mean field approach." The 22nd International Conference on Artificial Intelligence and Statistics. PMLR, 2019.

SGD regularization mechanism
- Daneshmand, H., Kohler, J., Lucchi, A., and Hofmann, T. Escaping saddles with stochastic gradients. arXiv preprint arXiv:1803.05999, 2018.
- Zhu, Zhanxing, et al. "The Anisotropic Noise in Stochastic Gradient Descent: Its Behavior of Escaping from Sharp Minima and Regularization Effects." arXiv preprint arXiv:1803.00195 (2018).
- Wu, Jingfeng, et al. "On the Noisy Gradient Descent that Generalizes as SGD." arXiv preprint arXiv:1906.07405 (2019).

Gradient confusion is highly related to the trace Fisher. See this one and its follow-ups.
- Sankararaman, Karthik A., et al. "The impact of neural network overparameterization on gradient confusion and stochastic gradient descent." arXiv preprint arXiv:1904.06963 (2019).

Adversarial regularization is also related to GP/FP:
- Miyato, Takeru, et al. "Virtual adversarial training: a regularization method for supervised and semi-supervised learning." IEEE transactions on pattern analysis and machine intelligence 41.8 (2018): 1979-1993.


  
review_title: Misleading and Even Wrong Writing

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary: This paper explores the relationship between the trace of the Fisher Information Matrix (FIM) and generalization performance of deep learning models. There are a few core insights which are verified empirically across a number of different models and datasets: (1) large learning rate/small batch sizes (which tends to lead to improved generalization) can be realized as implicitly penalizing the trace of Fisher and (2) how regularizing the trace of Fisher discourages memorization and guides optimization to "flatter minima". 

My overall assessment of this paper leans towards positive. While it is probably not surprising to deep learning optimization/generalization researchers that the trace of FIM is closely related to generalization performance, this paper does a good job from the empirical standpoint, the experiments are done in a clean and systematic way to verify the hypotheses. 

Some comments/remarks/questions:

- The manuscript is overall well-written; I have no problems following the logic and the experimental setups. Related works for the most part are discussed throughly and well-explained. A downside is that there is completely no theoretical arguments given in the paper, I am wondering if it's possible even in the case of a convex quadratic/extremely simple networks, the authors can show how a small Tr(F) leads to a tighter/better generalization bound? 

- Just to verify- the Fisher matrix which is computed throughout is the empirical estimate of the true Fisher and not the "empirical Fisher"- as the labels are sampled from the predictive distribution? This often results in a confusion; see [1] for more details. 

- In Section 3, the authors compared FP with GP which is a fair comparison. I am wondering if the authors compared their approach to even more closely-related methods such as input-output Jacobian regularization [2], regularizing by the Frobenius norm of Fisher; Tr(F^2) instead of Tr(F), and the Fisher-Rao norm which has a tight relationship with capacity/generalization of neural networks as shown in [3].

References:

[1] Kunstner, Frederik, Philipp Hennig, and Lukas Balles. "Limitations of the empirical Fisher approximation for natural gradient descent." Advances in Neural Information Processing Systems. 2019.

[2] Novak, Roman, et al. "Sensitivity and generalization in neural networks: an empirical study." arXiv preprint arXiv:1802.08760 (2018).

[3] Liang, Tengyuan, et al. "Fisher-rao metric, geometry, and complexity of neural networks." The 22nd International Conference on Artificial Intelligence and Statistics. 2019.
review_title: A through empirical study of the relationship between trace of Fisher matrix and deep learning generalization

final_score: rating: 6: Marginally above acceptance threshold
review_content: After the rebuttal, the authors added a section on large-batch training, which shows that the catastrophic Fisher explosion also occurs in large batch training. This makes the paper more convincing. However, the main concern that this paper lacks theoretical contribution still exists. Therefore, I keep the weak acceptance recommendation. 

Summary:

In this paper, the authors made a study on Fisher Information Matrix (FIM) in the training dynamics. By tracking the trace of FIM in the initial training process, the authors show that it can be strongly connected to a model’s generalization performance. This successfully explains why some hyper-parameter choices lead to significantly better generalization. Finally, empirical evaluations show that penalizing the trace FIM can reduce memorization and encourage wide minima solution, leading to better generalization.

Pros:

This paper takes one step forward to the question of why some hyper-parameter choices (small batch size or large learning rate) lead to better generalization. In many previous works, the implicit regularization effect of stochastic gradient descent (SGD) accounts for this observation (small batch size generalizes better). Most of the above works focused on the anisotropic noise in SGD. However, there is still a gap between the anisotropic noise and implicit regularization, i.e. why does anisotropic noise lead to implicit regularization hence better generalization. [1] explains this in a very general way: it helps to escape local minima. This paper tries to fill in the gap by leveraging FIM. 

FIM is also studied in the context of SGD regularization [2]. However, it only shows that the trace of FIM affects optimization performance. How it relates to the generalization performance still remains unknown. Section 4 and section 5 shed light on this question by drawing connection to memorization and final curvatures. 

Larger scale dataset such as Tiny-Imagenet is included in the experiments. This shows that the conclusion also stands on a more practical and noisy dataset. 

Cons:

Most of the empirical verifications in this paper are comparing large and small learning rates. To be more general, the authors can also include the study of batch size. The central claim in this paper would be more convincing if catastrophic Fisher explosion also occurs in large batch training. More importantly, reducing the trace can improve large batch training generalization. Additionally, the connection between FIM and noisy gradient descent [3] is worth mentioning. For example, how does the trace of FIM in noisy gradient descent compared to other methods?

One of the postulations is regularizing the trace of FIM reduces memorization. The paper verifies this by introducing noisy labels. This requires modification to the training dataset. Another way to verify this postulation is to make comparisons on out-of-distribution dataset (CIFAR10-C and CIFAR100-C). The method which reduces memorization should be more robust on out-of-distribution datasets.

A majority of the analysis comes from empirical evaluations. The claim is more convincing if the conclusion can also be analytically proved in the convex quadratic scenario. 
The experiments section of this paper is based on the empirical Fisher matrix (true Fisher matrix is expensive to compute). Although empirical Fisher is commonly used in practice to approximate true Fisher, the approximation is only accurate in the local minima [4]. The paper should have a discussion on the difference between true Fisher and empirical Fisher. 

Legends are missing in Figure 4 and Figure 6. 

[1]: Zhu, Zhanxing et al. “The Anisotropic Noise in Stochastic Gradient Descent: Its Behavior of Escaping from Sharp Minima and Regularization Effects.” ICML (2019).
[2]: Wen, Yeming et al. “An Empirical Study of Stochastic Gradient Descent with Structured Covariance Noise.” AISTATS (2020).
[3]: Wu, Jingfeng et al. “On the Noisy Gradient Descent that Generalizes as SGD.” ICML (2020)
[4]: Kunstner, Frederik et al. “Limitations of the empirical Fisher approximation for natural gradient descent.” NeurIPS (2019).
review_title: One step forward to uncover the implicit regularization effect of SGD.

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper empirically investigates the effect of the trace of the Fisher Information Matrix (FIM) early in training has on the generalization of SGD. Authors demonstrate that the effect of optimally chosen learning rate and batch size for SGD can be modeled as an implicit penalty on the trace of FIM. They argue that explicitly penalizing the trace of FIM discourages memorizing noisy labels, thus leading to better generalization. Furthermore, they experimentally show that the early low value of the trace of FIM may bias the optimization towards a flat optimum which has been observed to correlate well with good generalization.

*Positives:*
+ The paper is well-written and easy to follow. Authors convey their message clearly.
+ The paper's motivation is definitely relevant. Understanding the connection between the early stage loss landscape of neural networks and final generalization is crucial and has drawn significant attention recently.
+ The introduced Fisher-penalty is interesting and the authors provide insights into the mechanism how regularizing the trace of FIM might improve generalization.

*Negatives:*
- The exact mechanism how FP influences learning with noisy labels could be investigated a bit more rigorously. The discussion provided in the paper is not completely clear.
- The experiments are somewhat inconclusive in some cases (see below) and would require some extra comments/explanation.

Overall, the paper has some interesting contribution by demonstrating the positive effect of various gradient norm penalties on the generalization of SGD and it is tied well into existing literature. Furthermore, the experiments are comprehensive and in-depth. Therefore I would recommend acceptance if a couple of issues are addressed.

First, as seen in Fig. 4 generalization peaks if FP is applied starting after some positive number of epochs. The difference in validation accuracy between turning on FP from the very beginning and only turning it on somewhat later is not always negligible. This observation somewhat violates the 'earlier we regularize the better' idea. Can the authors comment on this discrepancy?

Second, the benefit of high early regularization is somewhat inconclusive based on Fig. 10. It appears that the best generalization belongs to low early regularization, whereas high early regularization leads to better average test accuracy. I would be interested to know how much FP might hinder optimization by discouraging the exploration of certain directions in the loss landscape. Can it even contribute to trapping the network in local optima by penalizing large gradients needed to escape such 'bad' optima?

Minor comment: there is a typo in the caption of Fig. 6.
review_title: Interesting contribution, extensive experiments, some questions

Meta-review: This is a tricky one, hence my low confidence rating.

The reviewers seem to agree that the paper is well written, easy to follow, and that it tests a relevant hypothesis that is of interest to the community. There was some disagreement as to whether the experiments are comprehensive, complete and/or conclusive enough, although on balance it seems reviewers were overall satisfied barring a few additional requests which the authors addressed in their feedback.

However, no reviewers support the paper strongly (borderline accepts) while R5 remains unconvinced and has raised a technical point in their review about the estimator of the Trace of the Fisher information matrix. The question R5 has raised is central to the paper's methods, arguments and conclusions. In a message to ACs and PCs the authors raised concerns about R5. I personally thought that while R5 could have worded their review more carefully and respectfully (as I pointed out in my respose) the concerns raised were otherwise motivated, the reviewer engaged in a discussion, and the arguments were laid out clearly. I side with R5 and I think that the paper should be rewritten with more clarity on this question - the problem R5 found is likely to trip up others who read or build on the paper.

The authors have raised that there are two parallel submissions closely related to this one, complicating the decision making somewhat:
[1] https://openreview.net/forum?id=rq_Qr0c1Hyo [2] https://openreview.net/forum?id=3q5IqUrkcF

Decision: Reject