Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: Summary:
This paper studies the gradient norm as a measure of generalization in deep learning. The authors first an approximation to the gradient norm (GN) that is the norm of the gradients for only fully connected layers (AGN). Then they empirically evaluate the correlation between AGN and GN as well as GN and the generalization error. In Section 2.1, the authors conclude that AGN is highly correlated with GN and both are correlated with generalization error. In Section 3, the authors conclude that the correlation between AGN and generalization error is not consistent in a wider family of models. In Section 4, authors propose to use AGN for model selection and conclude that AGN is not good for model selection unless the hyperparameter for mixing AGN with another metric is optimal.

Pros:
- Recently there has been huge interest in measures of generalization error. Studies of this sort even with mixed results can be helpful in better evaluation of individual proposed measures.
- Gradient norm is a measure proposed by prior published work (Li et al. (2020)) with theoretical justifications without extensive empirical evaluations which is done in this work.

Cons:
- The main limitation of this work is that the authors propose an approximate measure to gradient norm (AGN), then the results are mostly negative for AGN in Section 3 and 4. But this doesn’t necessarily mean gradient norm is a poor measure of generalization. The link in Section 2 is only empirically shown for a MNIST, Fashion-MNIST, and CIFAR-100 datasets and there is no theoretical reason to justify this. So I don’t find the results to be conclusive about gradient norm.
- Gradient norm can be computed efficiently in much less time than the claimed time in the paper. See [1] for a framework that computes gradient norm efficiently and [2] for general dot products between gradients.
- Empirical results in Fig 1 are the main support of the paper for using the approximate gradient norm in the next sections. It is not clear if the conclusions from MNIST, Fashion-MNIST, and CIFAR-100 datasets would hold for other datasets. Especially that the conclusion in future sections seems to be negative for the application of AGN.

Additional notes:
- In abstract given that only an approximation to gradient norm is computed the following statement cannot be made: “...gradient norm also fails to predict the generalization performance”.
- Fig 1, a-d: the norm of the gradient on MNIST should tend to zero by the end of the training as the loss tends to 0. Despite that, the range of values for AGN and GN in this figure is way above 100. In contrast, GN in Fig 1.e is below 1 which matches the expectation. Have models in a-d converged?
- The authors could also look into recent work on “robust” generalization measures [3]. AGN seems not to be a robust measure based on Figure 3. The authors could study better approximations to gradient norm and their robustness.

Typos:
- focuses -> focus
- some advanced measure -> measures
-  Thomas et al. (2019) did they derive TIC? Or only try it out?
- correlates to -> with
-  Thomas et al. (2019) is a posterior measure -> TIC is a
- our work make -> makes
- technical contributions -> contribution

[1] Dangel, F., Kunstner, F., & Hennig, P. (2019). BackPACK: Packing more into backprop. arXiv preprint arXiv:1912.10985.
[2] Faghri, F., Duvenaud, D., Fleet, D. J., & Ba, J. (2020). A Study of Gradient Variance in Deep Learning. arXiv preprint arXiv:2007.04532.
[3] Dziugaite, G. K., Drouin, A., Neal, B., Rajkumar, N., Caballero, E., Wang, L., ... & Roy, D. M. (2020). In Search of Robust Measures of Generalization. arXiv preprint arXiv:2010.11924.

============
After rebuttal:
I thank authors for their response. I share concerns with others reviewers and I highly encourage authors to consider answering questions suggested by Reviewer 3 at the end of their discussion. I believe a systematic study of gradient norm is interesting but this work does not provide a solid set of answers. As such, I'm reducing my rating to 4.

review_title: Gradient norm is interesting to be studied, but the specific approximation proposed does not seem to be a good.

final_score: rating: 4: Ok but not good enough - rejection
review_content: Authors of the paper conducted many numerical experiments on the relationship between generalization and their proposal on an approximated form of gradient norms. 

The motivation for their study is the article Li et al. (2020) which discovers deep relation between generalization and gradient norms. However, as far as I can understand, what the article Li et al. (2020) precisely studies is a noisy version of gradient descent (stochastic gradient Langevin dynamics), and without the noisy assumption their theoretical analysis is not valid. Based on these, I am not convinced that studying the relation between generalization and a *variant* of gradient norms with respect to the *true* SGD is a proper topic that the ML community should consider. Many experiments in the gap should be carried on.

Moreover, in 'contribution' (3) the authors make a digression discussing generalization and hyperparameter searching, and mark as one of the major contribution. The conclusion is that their 'approximated gradient norm' is not well-behaved and has many constrains in application, so I would mark these as merely observations rather than 'contributions'. I think this part is rather incomplete.

To me the article seems to be an experimental report on what has been observed during the numerical trials. It might be a good submission to workshop, but not qualify for ICLR main conference.
review_title: empty

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper empirically investigates the sum of gradient norms as a measure to determine the generalization abilities of a neural network. The approach is inspired by the theoretical work of Li, et al. 2020 which showed that the generalization gap can be upper bounded by a function of the sum of the full gradient norms of the training path. 

The paper empirically investigates the gradient norm as a measure to predict the generalization gap of neural networks. For that, the paper proposes an approximation of the sum of the full gradient norms that is more computationally efficient. The paper shows that this approximation is, albeit being in a different scale, strongly correlated with the full gradient measure. The paper then continues to openly investigate the properties of the measure and finds that it is only partially correlated with the generalization gap. When used for hyperparameter tuning, it can slightly improve the results. Lastly, the paper shows that the measure is not effective in predicting the generalization ability of DNNs with different architectures. 
The empirical evaluation is sound (although limited to only fairly simple image classification problems), the results are interesting, and I particularly regard the honesty about the negative results. I think that this is a valuable study. 

What is missing though is a discussion of these findings and a placement of these results into the wider context of generalization in deep learning. Do these results imply that the gradient norm is not a sufficient measurement? Or can the gradient norm be an effective measure in certain situations? The paper hints at that, saying that for well-trained networks the measure correlates more strongly with the generalization gap. 

A particularly interesting discussion here would be the relation to flatness measures which have been found to be strongly correlated with the generalization gap [1,2,3,4,5]. It seems, the gradient path norm does not necessarily outperform those measures but might be related. I.e., if a model is in a very flat minimum, at least for a part of its training path (the last part) it will have been in that flat region and thus gradient norms would be small. 

In summary, the paper presents an interesting empirical analysis and makes some technical contributions in the form of the proposed AGN measure. However, the missing in-depth discussion limits the contribution of this paper. Since such a discussion would go beyond the minor edits for a CRC, I am not convinced this paper is ready for publication, yet.

More out of interest than as a critique of this paper, I would like the authors to clarify how the gradient norm could be a meaningful measure to determine generalization in general. Li, et al. 2020 have shown a modified PAC-Bayesian bound that uses the gradient norm over the path, but it seems that most of the heavy lifting there is done by assuming a distribution dependent (i.e., perfect) prior - please correct me if I am wrong here. It seems to me then that the gradient norm can at best be a part of the explanation for good generalization (i.e., the one that relates to flatness and thus robustness). To illustrate my point, assume your data is drawn with x uniform in R and y = cos(v*x) + eps, where eps is some Gaussian noise. That is, the data is a noisy cosine function. Our model class consists of functions f_w (x) = cos(w*x) with a single parameter w. The optimal model has parameter w^*=v. Now the error surface of that model space if we use, for example, the squared loss, has lots of local minima of various depths and one global minimum at w^*=v (both in terms of empirical risk and true risk). If we now initialize our model randomly, it can start at a steep part of a local minimum and jump into the next and from there on to the next with fairly large gradients until it ends up (hopefully) in the global minium, where it will not escape (all also depending on the learning rate, of course). In this hypothetical (and arguably quite artificial) example, the path from most initializations to the global minimum would have large gradient norms. If we instead initialize close to a local minimum, the gradients will be small and if we start close enough to the minimum, the model will not escape. It will remain in that locally flat area with very small gradients. Thus, the gradient norm of the path is very small, yet the generalization error is large (the gap then depends on the actual sample and can be either small or quite large). My question now is: am I misunderstanding Li et al. or is my example too simple? Could the authors give some intuition on how the gradient norm could explain generalization?


[1] Jiang, et al. Fantastic generalization measures and where to find them. ICLR 2020.
[2] Keskar, et al. On large-batch training for deep learning: Generalization gap and sharp minima. ICLR, 2017.
[3] Petzka, et al. Relative Flatness and Generalization in the Interpolation Regime. arxiv prerpint, 2020.
[4] Neyshabur, et al. Exploring generalization in deep learning. NIPS, 2017.
[5] Tsuzuku, et al. Normalized flat minima: Exploring scale invariant definition of flat minima for neural networks using PAC-Bayesian analysis. arxive preprint, 2019.


------ After Discussion -------
I had an interesting discussion with the authors that clarified some important points. I came to the conclusion that the topic is interesting and the study is valuable, however, not yet conclusive. Consequently, the paper is not ready for publication, yet. Thus I have lowered my score by one. I hope the authors continue this work, since I'm convinced a complete empirical study on the impact of the gradient norm along the training path is insightful and a valuable contribution to the community.
review_title: Sum of Gradient Norms as Measure for Generalization

final_score: rating: 6: Marginally above acceptance threshold
review_content: In this paper, they provide the empirical studies  to understand the effectiveness and efficiency of the use of the gradient norm (induced by [the Li et al., 2020]) as the model selection criterion. To speed up the calculation process the of the gradient norm, they first propose an approximate gradient norm (AGN) based on the depth-wise, sample-wise and epoch-wise accelerations.  Their empirical studies find that the use of AGN can select the models with lower generalization error, but fails for bandit-based or population-based algorithms, and fails to predict the generalization performance of models based on different architectures.  In conclusion, they do not recommend using (approxiamte) gradient norm for model selection in practice.

Pros:

-1:  In this paper, they propose an approximate gradient norm (AGN) based on an accelerated approximation (Goodfellow, 2015) of gradient norm that only computes the loss gradient in the Fully-Connected Layers, which can significantly reduce the computation cost (200∼20,000 times faster) than the original one. They also find in empirical evaluations that AGN and GN behave identically with respect to empirical generalization gap.

-2: They carry out extensive experiments to validate the correlations between generalization performance and AGN, and find that, when the models are well-fitted, AGN well corresponds with the empirical generalization gap, but for the  under-fitted models, the correlations  between empirical generalization gap and AGN are not consistent.

-3: They use AGN  as an objective for  hyper-parameter selection under both blackbox optimization and bandit-based search, and find that,  AGN may help black-box optimization algorithms with an additional hyper-parameter, but fails to bandit-based or population-based algorithms, and fails to predict the generalization performance of models based on different architectures.

Cons:

-1: In Figure 1(a) and (c), we can find that for  Multi-Layer Preceptors (MLP), the trends of AGN behave identically with respect to the generalization gap, but, in Figure 3, they show that for ResNet-20, ResNet-56, ResNet-110 and DenseNet 100*24, the correlations between AGN and generalization gap are not consistent. These results confuse me, can we draw a new conclusion that, when the model is simple,  the  correlations  between AGN and generalization gap are consistent, but for complex models, the correlations are not consistent.  I want to see the additional experiments on the analysis of the MLP in Figure 3, I am surprised why ignoring the MLP in Section 3 and Section 4.

-2:  In Figure 1(e) and (f), one can see that the Pearson coeff  for epoch 81-120 are higher than that of the Epoch 1-80 and Epoch 121-160. This result seems a bit counterintuitive. Why the middle epoch can obtain the highest Pearson coeff?
review_title: Can We Use Gradient Norm as a Measure of Generalization Error for Model Selection in Practice

Meta-review: Dear authors,

Thank you for your submission. The reviewers all appreciated the direction of research and the message that GN can be a bad measure of generalization. That said, they all shared concerns regarding the strength of the conclusions that can be drawn from your work.

I encourage you to address their comments and submit a revised version to a later conference.

---------------------------------
Reviewer 1 wanted to update their review but couldn't so here is the update:


Some more details on my original concerns

Thank you for your detailed responses. I wanted to add more details to the ones not discussed by other reviewers.

- Regarding the speed of computing the gradient norm, I still don't agree that the computation cost is high. Figure 6 in the Backpack paper shows the cost of computing individual gradients at most 4x the cost of a single backprop not 100-1000x. In reference [2] that I gave, there is also a cheaper approximation discussed with computational costs detailed in Appendix B. As long as the computation of gradient norm is comparable with the cost of a single back-prop it should be cheap enough to run all your experiments.

- Regarding the conclusions in the paper. Thank you for giving more details. Adding those explanations to the paper would help. I personally missed some of those takeaway messages.

Overall, I strongly recommend either strengthening the link between GN and AGN or using better approximations. As well as better discussing the conclusions. Of course in addition to the suggestions by other reviewers.
Decision: Reject